| 1 |
<?xml version="1.0" encoding="ISO-8859-1"?> |
| 2 |
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" |
| 3 |
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| 4 |
<html xmlns="http://www.w3.org/1999/xhtml"> |
| 5 |
<head> |
| 6 |
<title>Eric S. Raymond's former Design notes on fetchmail</title> |
| 7 |
<link rev="made" href="mailto:esr@snark.thyrsus.com" /> |
| 8 |
<meta name="description" content="Design notes on fetchmail." /> |
| 9 |
<meta name="keywords" content="fetchmail, POP, POP2, POP3, IMAP, remote mail" /> |
| 10 |
<style type="text/css"> |
| 11 |
/*<![CDATA[*/ |
| 12 |
h1.c1 {text-align: center} |
| 13 |
/*]]>*/ |
| 14 |
</style> |
| 15 |
</head> |
| 16 |
<body> |
| 17 |
<table width="100%" cellpadding="0" summary="Canned page header"> |
| 18 |
<tr> |
| 19 |
<td width="30%">Forward to <a href="design-notes.html">Updated design |
| 20 |
notes</a></td> |
| 21 |
<td width="30%">Back to <a href="index.html">Fetchmail Home Page</a></td> |
| 22 |
<td width="30%" align="right">$Date$</td> |
| 23 |
</tr> |
| 24 |
</table> |
| 25 |
|
| 26 |
<hr /> |
| 27 |
<h1 class="c1">Eric S. Raymond's former Design Notes On Fetchmail</h1> |
| 28 |
|
| 29 |
<p>These notes are for the benefit of future hackers and |
| 30 |
maintainers. The following sections are both functional and |
| 31 |
narrative, read from beginning to end.</p> |
| 32 |
|
| 33 |
<h1>History</h1> |
| 34 |
|
| 35 |
<p>A direct ancestor of the fetchmail program was originally |
| 36 |
authored (under the name popclient) by Carl Harris |
| 37 |
<ceharris@mal.com>. I took over development in June 1996 and |
| 38 |
subsequently renamed the program `fetchmail' to reflect the |
| 39 |
addition of IMAP support and SMTP delivery. In early November 1996 |
| 40 |
Carl officially ended support for the last popclient versions.</p> |
| 41 |
|
| 42 |
<p>Before accepting responsibility for the popclient sources from |
| 43 |
Carl, I had investigated and used and tinkered with every other |
| 44 |
UNIX remote-mail forwarder I could find, including fetchpop1.9, |
| 45 |
PopTart-0.9.3, get-mail, gwpop, pimp-1.0, pop-perl5-1.2, popc, |
| 46 |
popmail-1.6 and upop. My major goal was to get a header-rewrite |
| 47 |
feature like fetchmail's working so I wouldn't have reply problems |
| 48 |
anymore.</p> |
| 49 |
|
| 50 |
<p>Despite having done a good bit of work on fetchpop1.9, when I |
| 51 |
found popclient I quickly concluded that it offered the solidest |
| 52 |
base for future development. I was convinced of this primarily by |
| 53 |
the presence of multiple-protocol support. The competition didn't |
| 54 |
do POP2/RPOP/APOP, and I was already having vague thoughts of maybe |
| 55 |
adding IMAP. (This would advance two other goals: learn IMAP and |
| 56 |
get comfortable writing TCP/IP client software.)</p> |
| 57 |
|
| 58 |
<p>Until popclient 3.05 I was simply following out the implications |
| 59 |
of Carl's basic design. He already had daemon.c in the |
| 60 |
distribution, and I wanted daemon mode almost as badly as I wanted |
| 61 |
the header rewrite feature. The other things I added were bug fixes |
| 62 |
or minor extensions.</p> |
| 63 |
|
| 64 |
<p>After 3.1, when I put in SMTP-forwarding support (more about |
| 65 |
this below) the nature of the project changed -- it became a |
| 66 |
carefully-thought-out attempt to render obsolete every other |
| 67 |
program in its class. The name change quickly followed.</p> |
| 68 |
|
| 69 |
<h1>The rewrite option</h1> |
| 70 |
|
| 71 |
<p>MTAs ought to canonicalize the addresses of outgoing non-local |
| 72 |
mail so that From:, To:, Cc:, Bcc: and other address headers |
| 73 |
contain only fully qualified domain names. Failure to do so can |
| 74 |
break the reply function on many mailers. (Sendmail has an option |
| 75 |
to do this.)</p> |
| 76 |
|
| 77 |
<p>This problem only becomes obvious when a reply is generated on a |
| 78 |
machine different from where the message was delivered. The two |
| 79 |
machines will have different local username spaces, potentially |
| 80 |
leading to misrouted mail.</p> |
| 81 |
|
| 82 |
<p>Most MTAs (and sendmail in particular) do not canonicalize |
| 83 |
address headers in this way (violating RFC 1123). Fetchmail |
| 84 |
therefore has to do it. This is the first feature I added to the |
| 85 |
ancestral popclient.</p> |
| 86 |
|
| 87 |
<h1>Reorganization</h1> |
| 88 |
|
| 89 |
<p>The second thing I did reorganize and simplify popclient a lot. |
| 90 |
Carl Harris's implementation was very sound, but exhibited a kind |
| 91 |
of unnecessary complexity common to many C programmers. He treated |
| 92 |
the code as central and the data structures as support for the |
| 93 |
code. As a result, the code was beautiful but the data structure |
| 94 |
design ad-hoc and rather ugly (at least to this old LISP |
| 95 |
hacker).</p> |
| 96 |
|
| 97 |
<p>I was able to improve matters significantly by reorganizing most |
| 98 |
of the program around the `query' data structure and eliminating a |
| 99 |
bunch of global context. This especially simplified the main |
| 100 |
sequence in fetchmail.c and was critical in enabling the daemon |
| 101 |
mode changes.</p> |
| 102 |
|
| 103 |
<h1>IMAP support and the method table</h1> |
| 104 |
|
| 105 |
<p>The next step was IMAP support. I initially wrote the IMAP code |
| 106 |
as a generic query driver and a method table. The idea was to have |
| 107 |
all the protocol-independent setup logic and flow of control in the |
| 108 |
driver, and the protocol-specific stuff in the method table.</p> |
| 109 |
|
| 110 |
<p>Once this worked, I rewrote the POP3 code to use the same |
| 111 |
organization. The POP2 code kept its own driver for a couple more |
| 112 |
releases, until I found sources of a POP2 server to test against |
| 113 |
(the breed seems to be nearly extinct).</p> |
| 114 |
|
| 115 |
<p>The purpose of this reorganization, of course, is to trivialize |
| 116 |
the development of support for future protocols as much as |
| 117 |
possible. All mail-retrieval protocols have to have pretty similar |
| 118 |
logical design by the nature of the task. By abstracting out that |
| 119 |
common logic and its interface to the rest of the program, both the |
| 120 |
common and protocol-specific parts become easier to understand.</p> |
| 121 |
|
| 122 |
<p>Furthermore, many kinds of new features can instantly be |
| 123 |
supported across all protocols by modifying the one driver |
| 124 |
module.</p> |
| 125 |
|
| 126 |
<h1>Implications of smtp forwarding</h1> |
| 127 |
|
| 128 |
<p>The direction of the project changed radically when Harry |
| 129 |
Hochheiser sent me his scratch code for forwarding fetched mail to |
| 130 |
the SMTP port. I realized almost immediately that a reliable |
| 131 |
implementation of this feature would make all the other delivery |
| 132 |
modes obsolete.</p> |
| 133 |
|
| 134 |
<p>Why mess with all the complexity of configuring an MDA or |
| 135 |
setting up lock-and-append on a mailbox when port 25 is guaranteed |
| 136 |
to be there on any platform with TCP/IP support in the first place? |
| 137 |
Especially when this means retrieved mail is guaranteed to look |
| 138 |
like normal sender- initiated SMTP mail, which is really what we |
| 139 |
want anyway.</p> |
| 140 |
|
| 141 |
<p>Clearly, the right thing to do was (1) hack SMTP forwarding |
| 142 |
support into the generic driver, (2) make it the default mode, and |
| 143 |
(3) eventually throw out all the other delivery modes.</p> |
| 144 |
|
| 145 |
<p>I hesitated over step 3 for some time, fearing to upset |
| 146 |
long-time popclient users dependent on the alternate delivery |
| 147 |
mechanisms. In theory, they could immediately switch to .forward |
| 148 |
files or their non-sendmail equivalents to get the same effects. In |
| 149 |
practice the transition might have been messy.</p> |
| 150 |
|
| 151 |
<p>But when I did it (see the NEWS note on the great options |
| 152 |
massacre) the benefits proved huge. The cruftiest parts of the |
| 153 |
driver code vanished. Configuration got radically simpler -- no |
| 154 |
more grovelling around for the system MDA and user's mailbox, no |
| 155 |
more worries about whether the underlying OS supports file |
| 156 |
locking.</p> |
| 157 |
|
| 158 |
<p>Also, the only way to lose mail vanished. If you specified |
| 159 |
localfolder and the disk got full, your mail got lost. This can't |
| 160 |
happen with SMTP forwarding because your SMTP listener won't return |
| 161 |
OK unless the message can be spooled or processed.</p> |
| 162 |
|
| 163 |
<p>Also, performance improved (though not so you'd notice it in a |
| 164 |
single run). Another not insignificant benefit of this change was |
| 165 |
that the manual page got a lot simpler.</p> |
| 166 |
|
| 167 |
<p>Later, I had to bring --mda back in order to allow handling of |
| 168 |
some obscure situations involving dynamic SLIP. But I found a much |
| 169 |
simpler way to do it.</p> |
| 170 |
|
| 171 |
<p>The moral? Don't hesitate to throw away superannuated features |
| 172 |
when you can do it without loss of effectiveness. I tanked a couple |
| 173 |
I'd added myself and have no regrets at all. As Saint-Exupery said, |
| 174 |
"Perfection [in design] is achieved not when there is nothing more |
| 175 |
to add, but rather when there is nothing more to take away." This |
| 176 |
program isn't perfect, but it's trying.</p> |
| 177 |
|
| 178 |
<h1>The most-requested features that I will never add, and why |
| 179 |
not:</h1> |
| 180 |
|
| 181 |
<h2>Password encryption in .fetchmailrc</h2> |
| 182 |
|
| 183 |
<p>The reason there's no facility to store passwords encrypted in |
| 184 |
the .fetchmailrc file is because this doesn't actually add |
| 185 |
protection.</p> |
| 186 |
|
| 187 |
<p>Anyone who's acquired the 0600 permissions needed to read your |
| 188 |
.fetchmailrc file will be able to run fetchmail as you anyway -- |
| 189 |
and if it's your password they're after, they'd be able to rip the |
| 190 |
necessary decoder out of the fetchmail code itself to get it.</p> |
| 191 |
|
| 192 |
<p>All .fetchmailrc encryption would do is give a false sense of |
| 193 |
security to people who don't think very hard.</p> |
| 194 |
|
| 195 |
<h2>Truly concurrent queries to multiple hosts</h2> |
| 196 |
|
| 197 |
<p>Occasionally I get a request for this on "efficiency" grounds. |
| 198 |
These people aren't thinking either. True concurrency would do |
| 199 |
nothing to lessen fetchmail's total IP volume. The best it could |
| 200 |
possibly do is change the usage profile to shorten the duration of |
| 201 |
the active part of a poll cycle at the cost of increasing its |
| 202 |
demand on IP volume per unit time.</p> |
| 203 |
|
| 204 |
<p>If one could thread the protocol code so that fetchmail didn't |
| 205 |
block on waiting for a protocol response, but rather switched to |
| 206 |
trying to process another host query, one might get an efficiency |
| 207 |
gain (close to constant loading at the single-host level).</p> |
| 208 |
|
| 209 |
<p>Fortunately, I've only seldom seen a server that incurred |
| 210 |
significant wait time on an individual response. I judge the gain |
| 211 |
from this not worth the hideous complexity increase it would |
| 212 |
require in the code.</p> |
| 213 |
|
| 214 |
<h2>Multiple concurrent instances of fetchmail</h2> |
| 215 |
|
| 216 |
<p>Fetchmail locking is on a per-invoking-user because |
| 217 |
finer-grained locks would be really hard to implement in a portable |
| 218 |
way. The problem is that you don't want two fetchmails querying the |
| 219 |
same site for the same remote user at the same time.</p> |
| 220 |
|
| 221 |
<p>To handle this optimally, multiple fetchmails would have to |
| 222 |
associate a system-wide semaphore with each active pair of a remote |
| 223 |
user and host canonical address. A fetchmail would have to block |
| 224 |
until getting this semaphore at the start of a query, and release |
| 225 |
it at the end of a query.</p> |
| 226 |
|
| 227 |
<p>This would be way too complicated to do just for an "it might be |
| 228 |
nice" feature. Instead, you can run a single root fetchmail polling |
| 229 |
for multiple users in either single-drop or multidrop mode.</p> |
| 230 |
|
| 231 |
<p>The fundamental problem here is how an instance of fetchmail |
| 232 |
polling host foo can assert that it's doing so in a way visible to |
| 233 |
all other fetchmails. System V semaphores would be ideal for this |
| 234 |
purpose, but they're not portable.</p> |
| 235 |
|
| 236 |
<p>I've thought about this a lot and roughed up several designs. |
| 237 |
All are complicated and fragile, with a bunch of the standard |
| 238 |
problems (what happens if a fetchmail aborts before clearing its |
| 239 |
semaphore, and how do we recover reliably?).</p> |
| 240 |
|
| 241 |
<p>I'm just not satisfied that there's enough functional gain here |
| 242 |
to pay for the large increase in complexity that adding these |
| 243 |
semaphores would entail.</p> |
| 244 |
|
| 245 |
<h1>Multidrop and alias handling</h1> |
| 246 |
|
| 247 |
<p>I decided to add the multidrop support partly because some users |
| 248 |
were clamoring for it, but mostly because I thought it would shake |
| 249 |
bugs out of the single-drop code by forcing me to deal with |
| 250 |
addressing in full generality. And so it proved.</p> |
| 251 |
|
| 252 |
<p>There are two important aspects of the features for handling |
| 253 |
multiple-drop aliases and mailing lists which future hackers should |
| 254 |
be careful to preserve.</p> |
| 255 |
|
| 256 |
<ol> |
| 257 |
<li> |
| 258 |
<p>The logic path for single-recipient mailboxes doesn't involve |
| 259 |
header parsing or DNS lookups at all. This is important -- it means |
| 260 |
the code for the most common case can be much simpler and more |
| 261 |
robust.</p> |
| 262 |
</li> |
| 263 |
|
| 264 |
<li> |
| 265 |
<p>The multidrop handing does <em>not</em> rely on doing the |
| 266 |
equivalent of passing the message to sendmail -t. Instead, it |
| 267 |
explicitly mines members of a specified set of local usernames out |
| 268 |
of the header.</p> |
| 269 |
</li> |
| 270 |
|
| 271 |
<li> |
| 272 |
<p>We do <em>not</em> attempt delivery to multidrop mailboxes in |
| 273 |
the presence of DNS errors. Before each multidrop poll we probe DNS |
| 274 |
to see if we have a nameserver handy. If not, the poll is skipped. |
| 275 |
If DNS crashes during a poll, the error return from the next |
| 276 |
nameserver lookup aborts message delivery and ends the poll. The |
| 277 |
daemon mode will then quietly spin until DNS comes up again, at |
| 278 |
which point it will resume delivering mail.</p> |
| 279 |
</li> |
| 280 |
</ol> |
| 281 |
|
| 282 |
<p>When I designed this support, I was terrified of doing anything |
| 283 |
that could conceivably cause a mail loop (you should be too). |
| 284 |
That's why the code as written can only append <em>local</em> names |
| 285 |
(never @-addresses) to the recipients list.</p> |
| 286 |
|
| 287 |
<p>The code in mxget.c is nasty, no two ways about it. But it's |
| 288 |
utterly necessary, there are a lot of MX pointers out there. It |
| 289 |
really ought to be a (documented!) entry point in the bind |
| 290 |
library.</p> |
| 291 |
|
| 292 |
<h1>DNS error handling</h1> |
| 293 |
|
| 294 |
<p>Fetchmail's behavior on DNS errors is to suppress forwarding and |
| 295 |
deletion of the individual message that each occurs in, leaving it |
| 296 |
queued on the server for retrieval on a subsequent poll. The |
| 297 |
assumption is that DNS errors are transient, due to temporary |
| 298 |
server outages.</p> |
| 299 |
|
| 300 |
<p>Unfortunately this means that if a DNS error is permanent a |
| 301 |
message can be perpetually stuck in the server mailbox. We've had a |
| 302 |
couple bug reports of this kind due to subtle RFC822 parsing errors |
| 303 |
in the fetchmail code that resulted in impossible things getting |
| 304 |
passed to the DNS lookup routines.</p> |
| 305 |
|
| 306 |
<p>Alternative ways to handle the problem: ignore DNS errors |
| 307 |
(treating them as a non-match on the mailserver domain), or forward |
| 308 |
messages with errors to fetchmail's invoking user in addition to |
| 309 |
any other recipients. These would fit an assumption that DNS lookup |
| 310 |
errors are likely to be permanent problems associated with an |
| 311 |
address.</p> |
| 312 |
|
| 313 |
<h1>IPv6 and IPSEC</h1> |
| 314 |
|
| 315 |
<p>The IPv6 support patches are really more protocol-family |
| 316 |
independence patches. Because of this, in most places, "ports" |
| 317 |
(numbers) have been replaced with "services" (strings, that may be |
| 318 |
digits). This allows us to run with certain protocols that use |
| 319 |
strings as "service names" where we in the IP world think of port |
| 320 |
numbers. Someday we'll plumb strings all over and then, if inet6 is |
| 321 |
not enabled, do a getservbyname() down in SocketOpen. The IPv6 |
| 322 |
support patches use getaddrinfo(), which is a POSIX p1003.1g |
| 323 |
mandated function. So, in the not too distant future, we'll zap the |
| 324 |
ifdefs and just let autoconf check for getaddrinfo. IPv6 support |
| 325 |
comes pretty much automatically once you have protocol family |
| 326 |
independence.</p> |
| 327 |
|
| 328 |
<h1>Internationalization</h1> |
| 329 |
|
| 330 |
<p>Internationalization is handled using GNU gettext (see the file |
| 331 |
ABOUT_NLS in the source distribution). This places some minor |
| 332 |
constraints on the code.</p> |
| 333 |
|
| 334 |
<p>Strings that must be subject to translation should be wrapped |
| 335 |
with GT_() or N_() -- the former in function arguments, the latter |
| 336 |
in static initializers and other non-function-argument |
| 337 |
contexts.</p> |
| 338 |
|
| 339 |
<h1>Checklist for Adding Options</h1> |
| 340 |
|
| 341 |
<p>Adding a control option is not complicated in principle, but |
| 342 |
there are a lot of fiddly details in the process. You'll need to do |
| 343 |
the following minimum steps.</p> |
| 344 |
|
| 345 |
<ul> |
| 346 |
<li>Add a field to represent the control in <code>struct |
| 347 |
run</code>, <code>struct query</code>, or <code>struct |
| 348 |
hostdata</code>.</li> |
| 349 |
|
| 350 |
<li>Go to <code>rcfile_y.y</code>. Add the token to the grammar. |
| 351 |
Don't forget the <code>%token</code> declaration.</li> |
| 352 |
|
| 353 |
<li>Pick an actual string to declare the option in the .fetchmailrc |
| 354 |
file. Add the token to <code>rcfile_l</code>.</li> |
| 355 |
|
| 356 |
<li>Pick a long-form option name, and a one-letter short option if |
| 357 |
any are left. Go to <code>options.c</code>. Pick a new |
| 358 |
<code>LA_</code> value. Hack the <code>longoptions</code> table to |
| 359 |
set up the association. Hack the big switch statement to set the |
| 360 |
option. Hack the `?' message to describe it.</li> |
| 361 |
|
| 362 |
<li>If the default is nonzero, set it in <code>def_opts</code> near |
| 363 |
the top of <code>load_params</code> in |
| 364 |
<code>fetchmail.c</code>.</li> |
| 365 |
|
| 366 |
<li>Add code to dump the option value in |
| 367 |
<code>fetchmail.c:dump_params</code>.</li> |
| 368 |
|
| 369 |
<li>For a per-site or per-user option, add proper |
| 370 |
<code>FLAG_MERGE</code> actions in fetchmail.c's optmerge() |
| 371 |
function. For a global option, add an override at the end of |
| 372 |
load_params; this will involve copying a "cmd_run." field to a |
| 373 |
corresponding "run." field, see the existing code for models.</li> |
| 374 |
|
| 375 |
<li>Document the option in fetchmail.man. This will require at |
| 376 |
least two changes; one to the collected table of options, and one |
| 377 |
full text description of the option.</li> |
| 378 |
|
| 379 |
<li>Hack fetchmailconf to configure it. Bump the fetchmailconf |
| 380 |
version.</li> |
| 381 |
|
| 382 |
<li>Hack conf.c to dump the option so we won't have a version-skew |
| 383 |
problem.</li> |
| 384 |
|
| 385 |
<li>Add an entry to NEWS.</li> |
| 386 |
|
| 387 |
<li>If the option implements a new feature, add a note to the |
| 388 |
feature list.</li> |
| 389 |
</ul> |
| 390 |
|
| 391 |
<p>There may be other things you have to do in the way of logic, of |
| 392 |
course.</p> |
| 393 |
|
| 394 |
<p>Before you implement an option, though, think hard. Is there any |
| 395 |
way to make fetchmail automatically detect the circumstances under |
| 396 |
which it should change its behavior? If so, don't write an option. |
| 397 |
Just do the check!</p> |
| 398 |
|
| 399 |
<h1>Lessons learned</h1> |
| 400 |
|
| 401 |
<h3>1. Server-side state is essential</h3> |
| 402 |
|
| 403 |
<p>The person(s) responsible for removing LAST from POP3 deserve to |
| 404 |
suffer. Without it, a client has no way to know which messages in a |
| 405 |
box have been read by other means, such as an MUA running on the |
| 406 |
server.</p> |
| 407 |
|
| 408 |
<p>The POP3 UID feature described in RFC1725 to replace LAST is |
| 409 |
insufficient. The only problem it solves is tracking which messages |
| 410 |
have been read <em>by this client</em> -- and even that requires |
| 411 |
tricky, fragile implementation.</p> |
| 412 |
|
| 413 |
<p>The underlying lesson is that maintaining accessible server-side |
| 414 |
`seen' state bits associated with Status headers is indispensible |
| 415 |
in a Unix/RFC822 mail server protocol. IMAP gets this right.</p> |
| 416 |
|
| 417 |
<h3>2. Readable text protocol transactions are a Good Thing</h3> |
| 418 |
|
| 419 |
<p>A nice thing about the general class of text-based protocols |
| 420 |
that SMTP, POP2, POP3, and IMAP belongs to is that client/server |
| 421 |
transactions are easy to watch and transaction code correspondingly |
| 422 |
easy to debug. Given a decent layer of socket utility functions |
| 423 |
(which Carl provided) it's easy to write protocol engines and not |
| 424 |
hard to show that they're working correctly.</p> |
| 425 |
|
| 426 |
<p>This is an advantage not to be despised! Because of it, this |
| 427 |
project has been interesting and fun -- no serious or persistent |
| 428 |
bugs, no long hours spent looking for subtle pathologies.</p> |
| 429 |
|
| 430 |
<h3>3. IMAP is a Good Thing.</h3> |
| 431 |
|
| 432 |
<p>Now that there is a standard IMAP equivalent of the POP3 APOP |
| 433 |
validation in CRAM-MD5, POP3 is completely obsolete.</p> |
| 434 |
|
| 435 |
<h3>4. SMTP is the Right Thing</h3> |
| 436 |
|
| 437 |
<p>In retrospect it seems clear that this program (and others like |
| 438 |
it) should have been designed to forward via SMTP from the |
| 439 |
beginning. This lesson may be applicable to other Unix programs |
| 440 |
that now call the local MDA/MTA as a program.</p> |
| 441 |
|
| 442 |
<h3>5. Syntactic noise can be your friend</h3> |
| 443 |
|
| 444 |
<p>The optional `noise' keywords in the rc file syntax started out |
| 445 |
as a late-night experiment. The English-like syntax they allow is |
| 446 |
considerably more readable than the traditional terse keyword-value |
| 447 |
pairs you get when you strip them all out. I think there may be a |
| 448 |
wider lesson here.</p> |
| 449 |
|
| 450 |
<h1>Motivation and validation</h1> |
| 451 |
|
| 452 |
<p>It is truly written: the best hacks start out as personal |
| 453 |
solutions to the author's everyday problems, and spread because the |
| 454 |
problem turns out to be typical for a large class of users. So it |
| 455 |
was with Carl Harris and the ancestral popclient, and so with me |
| 456 |
and fetchmail.</p> |
| 457 |
|
| 458 |
<p>It's gratifying that fetchmail has become so popular. Until just |
| 459 |
before 1.9 I was designing strictly to my own taste. The multi-drop |
| 460 |
mailbox support and the new --limit option were the first features |
| 461 |
to go in that I didn't need myself.</p> |
| 462 |
|
| 463 |
<p>By 1.9, four months after I started hacking on popclient and a |
| 464 |
month after the first fetchmail release, there were literally a |
| 465 |
hundred people on the fetchmail-friends contact list. That's pretty |
| 466 |
powerful motivation. And they were a good crowd, too, sending fixes |
| 467 |
and intelligent bug reports in volume. A user population like that |
| 468 |
is a gift from the gods, and this is my expression of |
| 469 |
gratitude.</p> |
| 470 |
|
| 471 |
<p>The beta testers didn't know it at the time, but they were also |
| 472 |
the subjects of a sociological experiment. The results are |
| 473 |
described in my paper, <a |
| 474 |
href="//www.catb.org/~esr/writings/cathedral-bazaar/">The |
| 475 |
Cathedral And The Bazaar</a>.</p> |
| 476 |
|
| 477 |
<h1>Credits</h1> |
| 478 |
|
| 479 |
<p>Special thanks go to Carl Harris, who built a good solid code |
| 480 |
base and then tolerated me hacking it out of recognition. And to |
| 481 |
Harry Hochheiser, who gave me the idea of the SMTP-forwarding |
| 482 |
delivery mode.</p> |
| 483 |
|
| 484 |
<p>Other significant contributors to the code have included Dave |
| 485 |
Bodenstab (error.c code and --syslog), George Sipe (--monitor and |
| 486 |
--interface), Gordon Matzigkeit (netrc.c), Al Longyear (UIDL |
| 487 |
support), Chris Hanson (Kerberos V4 support), and Craig Metz (OPIE, |
| 488 |
IPv6, IPSEC).</p> |
| 489 |
|
| 490 |
<h1>Conclusion</h1> |
| 491 |
|
| 492 |
<p>At this point, the fetchmail code appears to be pretty stable. |
| 493 |
It will probably undergo substantial change only if and when |
| 494 |
support for a new retrieval protocol or authentication method is |
| 495 |
added.</p> |
| 496 |
|
| 497 |
<h1>Relevant RFCS</h1> |
| 498 |
|
| 499 |
<p>Not all of these describe standards explicitly used in |
| 500 |
fetchmail, but they all shaped the design in one way or |
| 501 |
another.</p> |
| 502 |
|
| 503 |
<dl> |
| 504 |
<dt><a href="ftp://ftp.isi.edu/in-notes/rfc821.txt">RFC821</a></dt> |
| 505 |
|
| 506 |
<dd>SMTP protocol</dd> |
| 507 |
|
| 508 |
<dt><a href="ftp://ftp.isi.edu/in-notes/rfc822.txt">RFC822</a></dt> |
| 509 |
|
| 510 |
<dd>Mail header format</dd> |
| 511 |
|
| 512 |
<dt><a href="ftp://ftp.isi.edu/in-notes/rfc937.txt">RFC937</a></dt> |
| 513 |
|
| 514 |
<dd>Post Office Protocol - Version 2</dd> |
| 515 |
|
| 516 |
<dt><a href="ftp://ftp.isi.edu/in-notes/rfc974.txt">RFC974</a></dt> |
| 517 |
|
| 518 |
<dd>MX routing</dd> |
| 519 |
|
| 520 |
<dt><a href="ftp://ftp.isi.edu/in-notes/rfc976.txt">RFC976</a></dt> |
| 521 |
|
| 522 |
<dd>UUCP mail format</dd> |
| 523 |
|
| 524 |
<dt><a |
| 525 |
href="ftp://ftp.isi.edu/in-notes/rfc1081.txt">RFC1081</a></dt> |
| 526 |
|
| 527 |
<dd>Post Office Protocol - Version 3</dd> |
| 528 |
|
| 529 |
<dt><a |
| 530 |
href="ftp://ftp.isi.edu/in-notes/rfc1123.txt">RFC1123</a></dt> |
| 531 |
|
| 532 |
<dd>Host requirements (modifies 821, 822, and 974)</dd> |
| 533 |
|
| 534 |
<dt><a |
| 535 |
href="ftp://ftp.isi.edu/in-notes/rfc1176.txt">RFC1176</a></dt> |
| 536 |
|
| 537 |
<dd>Interactive Mail Access Protocol - Version 2</dd> |
| 538 |
|
| 539 |
<dt><a |
| 540 |
href="ftp://ftp.isi.edu/in-notes/rfc1203.txt">RFC1203</a></dt> |
| 541 |
|
| 542 |
<dd>Interactive Mail Access Protocol - Version 3</dd> |
| 543 |
|
| 544 |
<dt><a |
| 545 |
href="ftp://ftp.isi.edu/in-notes/rfc1225.txt">RFC1225</a></dt> |
| 546 |
|
| 547 |
<dd>Post Office Protocol - Version 3</dd> |
| 548 |
|
| 549 |
<dt><a |
| 550 |
href="ftp://ftp.isi.edu/in-notes/rfc1344.txt">RFC1344</a></dt> |
| 551 |
|
| 552 |
<dd>Implications of MIME for Internet Mail Gateways</dd> |
| 553 |
|
| 554 |
<dt><a |
| 555 |
href="ftp://ftp.isi.edu/in-notes/rfc1413.txt">RFC1413</a></dt> |
| 556 |
|
| 557 |
<dd>Identification server</dd> |
| 558 |
|
| 559 |
<dt><a |
| 560 |
href="ftp://ftp.isi.edu/in-notes/rfc1428.txt">RFC1428</a></dt> |
| 561 |
|
| 562 |
<dd>Transition of Internet Mail from Just-Send-8 to 8-bit |
| 563 |
SMTP/MIME</dd> |
| 564 |
|
| 565 |
<dt><a |
| 566 |
href="ftp://ftp.isi.edu/in-notes/rfc1460.txt">RFC1460</a></dt> |
| 567 |
|
| 568 |
<dd>Post Office Protocol - Version 3</dd> |
| 569 |
|
| 570 |
<dt><a |
| 571 |
href="ftp://ftp.isi.edu/in-notes/rfc1508.txt">RFC1508</a></dt> |
| 572 |
|
| 573 |
<dd>Generic Security Service Application Program Interface</dd> |
| 574 |
|
| 575 |
<dt><a |
| 576 |
href="ftp://ftp.isi.edu/in-notes/rfc1521.txt">RFC1521</a></dt> |
| 577 |
|
| 578 |
<dd>MIME: Multipurpose Internet Mail Extensions</dd> |
| 579 |
|
| 580 |
<dt><a |
| 581 |
href="ftp://ftp.isi.edu/in-notes/rfc1869.txt">RFC1869</a></dt> |
| 582 |
|
| 583 |
<dd>SMTP Service Extensions (ESMTP spec)</dd> |
| 584 |
|
| 585 |
<dt><a |
| 586 |
href="ftp://ftp.isi.edu/in-notes/rfc1652.txt">RFC1652</a></dt> |
| 587 |
|
| 588 |
<dd>SMTP Service Extension for 8bit-MIMEtransport</dd> |
| 589 |
|
| 590 |
<dt><a |
| 591 |
href="ftp://ftp.isi.edu/in-notes/rfc1725.txt">RFC1725</a></dt> |
| 592 |
|
| 593 |
<dd>Post Office Protocol - Version 3</dd> |
| 594 |
|
| 595 |
<dt><a |
| 596 |
href="ftp://ftp.isi.edu/in-notes/rfc1730.txt">RFC1730</a></dt> |
| 597 |
|
| 598 |
<dd>Interactive Mail Access Protocol - Version 4</dd> |
| 599 |
|
| 600 |
<dt><a |
| 601 |
href="ftp://ftp.isi.edu/in-notes/rfc1731.txt">RFC1731</a></dt> |
| 602 |
|
| 603 |
<dd>IMAP4 Authentication Mechanisms</dd> |
| 604 |
|
| 605 |
<dt><a |
| 606 |
href="ftp://ftp.isi.edu/in-notes/rfc1732.txt">RFC1732</a></dt> |
| 607 |
|
| 608 |
<dd>IMAP4 Compatibility With IMAP2 And IMAP2bis</dd> |
| 609 |
|
| 610 |
<dt><a |
| 611 |
href="ftp://ftp.isi.edu/in-notes/rfc1734.txt">RFC1734</a></dt> |
| 612 |
|
| 613 |
<dd>POP3 AUTHentication command</dd> |
| 614 |
|
| 615 |
<dt><a |
| 616 |
href="ftp://ftp.isi.edu/in-notes/rfc1870.txt">RFC1870</a></dt> |
| 617 |
|
| 618 |
<dd>SMTP Service Extension for Message Size Declaration</dd> |
| 619 |
|
| 620 |
<dt><a |
| 621 |
href="ftp://ftp.isi.edu/in-notes/rfc1891.txt">RFC1891</a></dt> |
| 622 |
|
| 623 |
<dd>SMTP Service Extension for Delivery Status Notifications</dd> |
| 624 |
|
| 625 |
<dt><a |
| 626 |
href="ftp://ftp.isi.edu/in-notes/rfc1892.txt">RFC1892</a></dt> |
| 627 |
|
| 628 |
<dd>The Multipart/Report Content Type for the Reporting of Mail |
| 629 |
System Administrative Messages</dd> |
| 630 |
|
| 631 |
<dt><a |
| 632 |
href="ftp://ftp.isi.edu/in-notes/rfc1894.txt">RFC1894</a></dt> |
| 633 |
|
| 634 |
<dd>An Extensible Message Format for Delivery Status |
| 635 |
Notifications</dd> |
| 636 |
|
| 637 |
<dt><a |
| 638 |
href="ftp://ftp.isi.edu/in-notes/rfc1893.txt">RFC1893</a></dt> |
| 639 |
|
| 640 |
<dd>Enhanced Mail System Status Codes</dd> |
| 641 |
|
| 642 |
<dt><a |
| 643 |
href="ftp://ftp.isi.edu/in-notes/rfc1894.txt">RFC1894</a></dt> |
| 644 |
|
| 645 |
<dd>An Extensible Message Format for Delivery Status |
| 646 |
Notifications</dd> |
| 647 |
|
| 648 |
<dt><a |
| 649 |
href="ftp://ftp.isi.edu/in-notes/rfc1938.txt">RFC1938</a></dt> |
| 650 |
|
| 651 |
<dd>A One-Time Password System</dd> |
| 652 |
|
| 653 |
<dt><a |
| 654 |
href="ftp://ftp.isi.edu/in-notes/rfc1939.txt">RFC1939</a></dt> |
| 655 |
|
| 656 |
<dd>Post Office Protocol - Version 3</dd> |
| 657 |
|
| 658 |
<dt><a |
| 659 |
href="ftp://ftp.isi.edu/in-notes/rfc1957.txt">RFC1957</a></dt> |
| 660 |
|
| 661 |
<dd>Some Observations on Implementations of the Post Office |
| 662 |
Protocol (POP3)</dd> |
| 663 |
|
| 664 |
<dt><a |
| 665 |
href="ftp://ftp.isi.edu/in-notes/rfc1985.txt">RFC1985</a></dt> |
| 666 |
|
| 667 |
<dd>SMTP Service Extension for Remote Message Queue Starting</dd> |
| 668 |
|
| 669 |
<dt><a |
| 670 |
href="ftp://ftp.isi.edu/in-notes/rfc2033.txt">RFC2033</a></dt> |
| 671 |
|
| 672 |
<dd>Local Mail Transfer Protocol</dd> |
| 673 |
|
| 674 |
<dt><a |
| 675 |
href="ftp://ftp.isi.edu/in-notes/rfc2060.txt">RFC2060</a></dt> |
| 676 |
|
| 677 |
<dd>Internet Message Access Protocol - Version 4rev1</dd> |
| 678 |
|
| 679 |
<dt><a |
| 680 |
href="ftp://ftp.isi.edu/in-notes/rfc2061.txt">RFC2061</a></dt> |
| 681 |
|
| 682 |
<dd>IMAP4 Compatibility With IMAP2bis</dd> |
| 683 |
|
| 684 |
<dt><a |
| 685 |
href="ftp://ftp.isi.edu/in-notes/rfc2062.txt">RFC2062</a></dt> |
| 686 |
|
| 687 |
<dd>Internet Message Access Protocol - Obsolete Syntax</dd> |
| 688 |
|
| 689 |
<dt><a |
| 690 |
href="ftp://ftp.isi.edu/in-notes/rfc2195.txt">RFC2195</a></dt> |
| 691 |
|
| 692 |
<dd>IMAP/POP AUTHorize Extension for Simple Challenge/Response</dd> |
| 693 |
|
| 694 |
<dt><a |
| 695 |
href="ftp://ftp.isi.edu/in-notes/rfc2177.txt">RFC2177</a></dt> |
| 696 |
|
| 697 |
<dd>IMAP IDLE command</dd> |
| 698 |
|
| 699 |
<dt><a |
| 700 |
href="ftp://ftp.isi.edu/in-notes/rfc2449.txt">RFC2449</a></dt> |
| 701 |
|
| 702 |
<dd>POP3 Extension Mechanism</dd> |
| 703 |
|
| 704 |
<dt><a |
| 705 |
href="ftp://ftp.isi.edu/in-notes/rfc2554.txt">RFC2554</a></dt> |
| 706 |
|
| 707 |
<dd>SMTP Service Extension for Authentication</dd> |
| 708 |
|
| 709 |
<dt><a |
| 710 |
href="ftp://ftp.isi.edu/in-notes/rfc2595.txt">RFC2595</a></dt> |
| 711 |
|
| 712 |
<dd>Using TLS with IMAP, POP3 and ACAP</dd> |
| 713 |
|
| 714 |
<dt><a |
| 715 |
href="ftp://ftp.isi.edu/in-notes/rfc2645.txt">RFC2645</a></dt> |
| 716 |
|
| 717 |
<dd>On-Demand Mail Relay: SMTP with Dynamic IP Addresses</dd> |
| 718 |
|
| 719 |
<dt><a |
| 720 |
href="ftp://ftp.isi.edu/in-notes/rfc2683.txt">RFC2683</a></dt> |
| 721 |
|
| 722 |
<dd>IMAP4 Implementation Recommendations</dd> |
| 723 |
|
| 724 |
<dt><a |
| 725 |
href="ftp://ftp.isi.edu/in-notes/rfc2821.txt">RFC2821</a></dt> |
| 726 |
|
| 727 |
<dd>Simple Mail Transfer Protocol</dd> |
| 728 |
|
| 729 |
<dt><a |
| 730 |
href="ftp://ftp.isi.edu/in-notes/rfc2822.txt">RFC2822</a></dt> |
| 731 |
|
| 732 |
<dd>Internet Message Format</dd> |
| 733 |
</dl> |
| 734 |
|
| 735 |
<!-- |
| 736 |
RFC2192 IMAP URL Scheme |
| 737 |
RFC2193 IMAP4 Mailbox Referrals |
| 738 |
RFC2221 IMAP4 Login Referrals |
| 739 |
--> |
| 740 |
|
| 741 |
<h1>Other useful documents</h1> |
| 742 |
|
| 743 |
<dl> |
| 744 |
<dt><a |
| 745 |
href="http://www.faqs.org/faqs/LANs/mail-protocols/">http://www.faqs.org/faqs/LANs/mail-protocols/</a></dt> |
| 746 |
|
| 747 |
<dd>LAN Mail Protocols Summary</dd> |
| 748 |
</dl> |
| 749 |
|
| 750 |
<hr /> |
| 751 |
<table width="100%" cellpadding="0" summary="Canned page footer"> |
| 752 |
<tr> |
| 753 |
<td width="30%">Back to <a href="index.html">Fetchmail Home Page</a></td> |
| 754 |
<td width="30%" align="right">$Date$</td> |
| 755 |
</tr> |
| 756 |
</table> |
| 757 |
|
| 758 |
<br clear="left" /> |
| 759 |
<address>Eric S. Raymond <a |
| 760 |
href="mailto:esr@thyrsus.com"><esr@snark.thyrsus.com></a></address> |
| 761 |
</body> |
| 762 |
</html> |