|
|
Home | Switchboard | Unix Administration | Red Hat | TCP/IP Networks | Neoliberalism | Toxic Managers |
| (slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix | |||||||
|
|
|
|
E-Mail was born in the decade before the ARPANET, by users on time sharing systems who wished to leave messages for one another. The earliest e-mail systems were fairly primitive affairs; each user's mailbox was a simple file, readable only by that user, to which a mail program would append messages. There were no mail reader programs, users simply waded through the raw text of their mailboxes, and so large mailboxes quickly became chaotic. The first of these systems was MAILBOX, and it was installed on the Compatible Time Sharing System, at MIT. Although comparatively crude, mail was so useful that it proliferated quickly.
The geographical reach of the ARPANET added the next piece to the e-mail puzzle. The first e-mail delivery to occur between two networked machines occurred in 1972. Ray Tomlinson, then an engineer at BBN delivered a mail message by using some inventive hacking to copy a mail message across a network link connecting two DEC PDP-10's. It was Ray Tomlinson, who, looking down at his keyboard, decided to use the '@' symbol to separate the user from the host part of an e-mail address.
By early 1973, three-quarters of the traffic on the ARPANET was e-mail. There were no mail specific protocols in place then - mail was piggybacked over FTP, which had commands specific to mail transfer. Mail delivery and tracking information (and pretty much everything else) was included in the mail headers, and one of the major problems with mail transfer was that there were no defined header standards. Mail programs that disagreed over formats would often refuse to talk to one another. For example, the famous '@' symbol was used as a line kill command on Multics systems, and Multics users tended to be somewhat 'mail-challenged' as a result of this...
The first major attempt to standardize mail headers resulted in RFC 680, which cleared up some of the chaos surrounding mail at the time. In 1977, RFC 680 was succeeded by RFC 724, which was greeted with a lukewarm reception from the ARPANET community. RFC 724 was revised soon after, and became RFC 733, but this still did little to stem the arguments surrounding mail header formats.
At the time of most of these events, TCP/IP had not yet appeared. The ARPANET used NCP as its core network protocol, and was utterly unable to communicate with any of the other packet networks in existence at the time. Into this environment, Eric Allman released delivermail, the ancestor of sendmail. Delivermail dealt with ARPANET mail using FTP over NCP, and its configuration was compiled into the program. The first version of delivermail shipped with 4.0 and 4.1 BSD in 1979.
As these events were taking place, Vint Cerf and Bob Kahn were working on a method to connect together the many packet networks in existence. Their idea, which was started around 1973 was to create a "network of networks", and the results of their work would become the TCP/IP protocol suite. The ARPANET made its official transition to TCP/IP in 1982. The widespread implementation of TCP/IP paved the way for the removal of mail services from the FTP protocol, and the creation of a new protocol : SMTP. Shortly after this, in 1986, the widespread adoption of DNS as a replacement for the old HOSTS file provided a way to extend the functionality of e-mail systems.
In response to the new SMTP protocol, and the functionality provided by DNS, Eric Allman evolved his delivermail program into sendmail. The first public version of sendmail shipped with 4.1c BSD. The new protocols extended sendmail's reach beyond the ARPANET, onto the many other packet networks comprising the nascent Internet.
Eric Allman stopped the development of sendmail in 1982, and did not go back to it until 1990. In 1998, Sendmail Inc. was formed, to create a commercial version of sendmail. Profits from the sale of this version would allow Allman to give sendmail the time it required, and the core distribution would remain free, allowing everyone to benefit from the new code. At the time of writing, the latest version of Sendmail is 8.9.2.
The sendmail program was originally written by Eric Allman while he was a student and staff member at the University of California at Berkeley. At the time, one campus machine (Ingres) was connected to the ARPAnet, and was home to the INGRES project where Eric was working. Another machine (Ernie CoVax) was home to the Berkeley UNIX project and had recently started using UUCP. These machines (as well as several others on campus) were connected by a low-cost network built by Eric Schmidt, called BerkNet. Software existed to move mail within ARPAnet, within UUCP, and within BerkNet, but none yet existed to move mail between these three networks.A sudden increase in protocol types, coupled with the anticipation of an explosion in the number of networks, motivated Eric to write delivermail - the precursor to sendmail. The delivermail program was shipped in 1979 with 4.0 and 4.1 BSD UNIX. Unfortunately, delivermail was not flexible enough to handle the changes in mail routing requirements that actually occurred. Perhaps its greatest weakness was that its configuration was compiled-in.
In 1980, ARPAnet began converting from NCP (Network Control Protocol) to TCP (Transmission Control Protocol). This change increased the number of possible hosts from 256 to over one billion. Another change converted from a "flat" host-name space (like MIT-XX) into a hierarchical name space (like XX.MIT.EDU). Prior to these changes, mail was transported using the ftp protocol (File Transfer Protocol). Afterward, a new protocol was developed for transporting mail called SMTP (Simple Mail Transfer Protocol). These developments were not instantaneous. Some networks continued to run NCP years after most others switched to TCP. SMTP itself underwent many revisions before finally settling into its present form.
Responding to these and other changes, Eric evolved delivermail into sendmail. To ensure that messages transferred between networks would obey the conventions required by those networks, Eric took a "liberal" approach - modifying address information to conform, rather than rejecting it. At the time, for example, UUCP mail often had no headers at all, so sendmail had to create them from scratch.
The first sendmail program was shipped with 4.1c BSD (the first version of Berkeley UNIX to include TCP/IP). From that first release to the present, [1] Eric has continued to enhance sendmail, first at UC Berkeley, then at Britton Lee, then back at UC Berkeley, and now with InReference Inc. The current version of sendmail is 8.x (or V8 for short). V8 is a major rewrite that includes many bug fixes and significant enhancements.
[1] With one long gap between 1982 and 1990.
But Eric wasn't the only one working on sendmail. In 1987, Lennart Lovstrand of the University of Linköping, Sweden, developed the IDA enhancements to BSD sendmail Version 5. IDA (which stands for "Institutionen för Datavetenskap") injected a number of improvements into sendmail (such as support for dbm files and separate rewriting of headers and envelopes) and fixed a number of bugs. As the '90s approached, two offspring of IDA appeared.
Neil Rickert (Northern Illinois University) and Paul Pomes (The University of Illinois) took over maintenance of IDA sendmail. With contributions from around the world, their version (UIUC IDA) represents a continuation of the work begun by Lennart Lovstrand. Neil focused on fixing and enhancing the configuration files into their current m4-based form. Paul maintained the code, continually adding enhancements and fixing bugs. In general, their version was large, ambitious, and highly portable. It succeeded in solving many complex mail routing problems.
A variation on IDA sendmail was also developed by Paul Vixie (while at Digital Equipment Corporation). Called KJS (for King James sendmail), it was a more conservative outgrowth of Lennart Lovstrand's last IDA release. The focus of KJS was on code improvement rather than changes to configuration files.
In addition to these major offshoots, many vendors have modified sendmail to suit their particular needs. Sun Microsystems made many modifications and enhancements to sendmail, including support for nis and nisplus maps. Hewlett Packard also contributed many fine enhancements including 8BITMIME support.
This explosion of sendmail versions has led to a great deal of confusion. Solutions to problems that work for one version of sendmail fail miserably with others. Beyond this, configuration files are not portable, and some features cannot be shared.
In 1994, Eric began work on V8.7 sendmail. The first major departure from tradition in years, V8.7 introduces multicharacter option and macro names, new interactive commands to use with -bt mode, and fixes many of the problems and limitations of earlier releases.
But, more important, V8.7 has officially adopted most of the good features from IDA, KJS, Sun, and HP's sendmail, and kept abreast of the latest standards from the Internet Engineering Task Force.
In 1996, Eric began work on V8.8 sendmail. This release continued the trend begun with V8.7, adding many requested new features and options, and tightening security. Since V8.8 is now the official release of sendmail, it is the one solely documented in this book.
Sendmail - An Internetwork Mail Router. Distributed Processing: Concepts and Structures, A.L. Ananda and B. Srinivasan, IEEE Computer Society Press. [Boissevain 1974] J. Boissevain. Friends of Friends - Networks, Manipulators, and Coalitions .Routing mail through a heterogenous internet presents many new problems. Among the worst of these is that of address mapping. Historically, this has been handled on an ad hoc basis. However, this approach has become unmanageable as internets grow. Sendmail acts a unified "post office " to which all mail can be submitted. Address interpretation is controlled by a production system, which can parse both domain-based addressing and oldstyle ad hoc addresses. The production system is powerful enough to rewrite addresses in the message header to conform to the standards of a number of common target networks, including old (NCP/RFC733) Arpanet, new (TCP/RFC822) Arpanet, UUCP, and Phonenet. Sendmail also implements an SMTP server, message queueing, and aliasing. Sendmail implements a general internetwork mail routing facility, featuring aliasing and forwarding, automatic routing to network gateways, and flexible configuration. In a simple network, each node has an address, and resources can be identified with a host-resource pair; in particular, the mail system can refer to users using a host-username pair. Host names and numbers have to be administered by a central authority, but usernames can be assigned locally to each host.
SENDMAIL -- An Internetwork Mail Router
Eric Allman*
University of California, Berkeley
Mammoth Project
ABSTRACT
Routing mail through a heterogenous internet pre-
sents many new problems. Among the worst of these
is that of address mapping. Historically, this has
been handled on an ad hoc basis. However, this ap-
proach has become unmanageable as internets grow.
Sendmail acts a unified "post office" to which all
mail can be submitted. Address interpretation is
controlled by a production system, which can parse
both domain-based addressing and old-style ad hoc
addresses. The production system is powerful enough
to rewrite addresses in the message header to con-
form to the standards of a number of common target
networks, including old (NCP/RFC733) Arpanet, new
(TCP/RFC822) Arpanet, UUCP, and Phonenet. Sendmail
also implements an SMTP server, message queueing,
and aliasing.
Sendmail implements a general internetwork mail routing
facility, featuring aliasing and forwarding, automatic rout-
ing to network gateways, and flexible configuration.
In a simple network, each node has an address, and
resources can be identified with a host-resource pair; in
____________________
*A considerable part of this work was done while under
the employ of the INGRES Project at the University of Cali-
fornia at Berkeley and at Britton Lee.
SENDMAIL -- An Internetwork Mail Router SMM:9-1
SMM:9-2 SENDMAIL -- An Internetwork Mail Router
particular, the mail system can refer to users using a host-
username pair. Host names and numbers have to be adminis-
tered by a central authority, but usernames can be assigned
locally to each host.
In an internet, multiple networks with different char-
acterstics and managements must communicate. In particular,
the syntax and semantics of resource identification change.
Certain special cases can be handled trivially by ad hoc
techniques, such as providing network names that appear
local to hosts on other networks, as with the Ethernet at
Xerox PARC. However, the general case is extremely com-
plex. For example, some networks require point-to-point
routing, which simplifies the database update problem since
only adjacent hosts must be entered into the system tables,
while others use end-to-end addressing. Some networks use a
left-associative syntax and others use a right-associative
syntax, causing ambiguity in mixed addresses.
Internet standards seek to eliminate these problems.
Initially, these proposed expanding the address pairs to
address triples, consisting of {network, host, resource}
triples. Network numbers must be universally agreed upon,
and hosts can be assigned locally on each network. The
user-level presentation was quickly expanded to address
domains, comprised of a local resource identification and a
hierarchical domain specification with a common static root.
The domain technique separates the issue of physical versus
SENDMAIL -- An Internetwork Mail Router SMM:9-3
logical addressing. For example, an address of the form
"[email protected]" describes only the logical organi-
zation of the address space.
Sendmail is intended to help bridge the gap between the
totally ad hoc world of networks that know nothing of each
other and the clean, tightly-coupled world of unique network
numbers. It can accept old arbitrary address syntaxes,
resolving ambiguities using heuristics specified by the sys-
tem administrator, as well as domain-based addressing. It
helps guide the conversion of message formats between dis-
parate networks. In short, sendmail is designed to assist a
graceful transition to consistent internetwork addressing
schemes.
Section 1 discusses the design goals for sendmail.
Section 2 gives an overview of the basic functions of the
system. In section 3, details of usage are discussed. Sec-
tion 4 compares sendmail to other internet mail routers, and
an evaluation of sendmail is given in section 5, including
future plans.
1. DESIGN GOALS
Design goals for sendmail include:
(1) Compatibility with the existing mail programs,
including Bell version 6 mail, Bell version 7 mail
[UNIX83], Berkeley Mail [Shoens79], BerkNet mail
SMM:9-4 SENDMAIL -- An Internetwork Mail Router
[Schmidt79], and hopefully UUCP mail [Nowitz78a,
Nowitz78b]. ARPANET mail [Crocker77a, Postel77]
was also required.
(2) Reliability, in the sense of guaranteeing that
every message is correctly delivered or at least
brought to the attention of a human for correct
disposal; no message should ever be completely
lost. This goal was considered essential because
of the emphasis on mail in our environment. It
has turned out to be one of the hardest goals to
satisfy, especially in the face of the many anoma-
lous message formats produced by various ARPANET
sites. For example, certain sites generate
improperly formated addresses, occasionally caus-
ing error-message loops. Some hosts use blanks in
names, causing problems with UNIX mail programs
that assume that an address is one word. The
semantics of some fields are interpreted slightly
differently by different sites. In summary, the
obscure features of the ARPANET mail protocol
really are used and are difficult to support, but
must be supported.
(3) Existing software to do actual delivery should be
used whenever possible. This goal derives as much
from political and practical considerations as
technical.
SENDMAIL -- An Internetwork Mail Router SMM:9-5
(4) Easy expansion to fairly complex environments,
including multiple connections to a single network
type (such as with multiple UUCP or Ether nets
[Metcalfe76]). This goal requires consideration
of the contents of an address as well as its syn-
tax in order to determine which gateway to use.
For example, the ARPANET is bringing up the TCP
protocol to replace the old NCP protocol. No host
at Berkeley runs both TCP and NCP, so it is neces-
sary to look at the ARPANET host name to determine
whether to route mail to an NCP gateway or a TCP
gateway.
(5) Configuration should not be compiled into the
code. A single compiled program should be able to
run as is at any site (barring such basic changes
as the CPU type or the operating system). We have
found this seemingly unimportant goal to be criti-
cal in real life. Besides the simple problems
that occur when any program gets recompiled in a
different environment, many sites like to "fiddle"
with anything that they will be recompiling any-
way.
(6) Sendmail must be able to let various groups main-
tain their own mailing lists, and let individuals
specify their own forwarding, without modifying
the system alias file.
SMM:9-6 SENDMAIL -- An Internetwork Mail Router
(7) Each user should be able to specify which mailer
to execute to process mail being delivered for
him. This feature allows users who are using spe-
cialized mailers that use a different format to
build their environment without changing the sys-
tem, and facilitates specialized functions (such
as returning an "I am on vacation" message).
(8) Network traffic should be minimized by batching
addresses to a single host where possible, without
assistance from the user.
These goals motivated the architecture illustrated
in figure 1. The user interacts with a mail generating
____________________________________________________________
+---------+ +---------+ +---------+
| sender1 | | sender2 | | sender3 |
+---------+ +---------+ +---------+
| | |
+----------+ + +----------+
| | |
v v v
+-------------+
| sendmail |
+-------------+
| | |
+----------+ + +----------+
| | |
v v v
+---------+ +---------+ +---------+
| mailer1 | | mailer2 | | mailer3 |
+---------+ +---------+ +---------+
Figure 1 -- Sendmail System Structure.
____________________________________________________________
SENDMAIL -- An Internetwork Mail Router SMM:9-7
and sending program. When the mail is created, the gen-
erator calls sendmail, which routes the message to the
correct mailer(s). Since some of the senders may be net-
work servers and some of the mailers may be network
clients, sendmail may be used as an internet mail gate-
way.
2. OVERVIEW
2.1. System Organization
Sendmail neither interfaces with the user nor
does actual mail delivery. Rather, it collects a mes-
sage generated by a user interface program (UIP) such
as Berkeley Mail, MS [Crocker77b], or MH [Borden79],
edits the message as required by the destination net-
work, and calls appropriate mailers to do mail deliv-
ery or queueing for network transmission1. This dis-
cipline allows the insertion of new mailers at minimum
cost. In this sense sendmail resembles the Message
Processing Module (MPM) of [Postel79b].
2.2. Interfaces to the Outside World
There are three ways sendmail can communicate
with the outside world, both in receiving and in send-
ing mail. These are using the conventional UNIX
____________________
1except when mailing to a file, when sendmail does the
delivery directly.
SMM:9-8 SENDMAIL -- An Internetwork Mail Router
argument vector/return status, speaking SMTP over a
pair of UNIX pipes, and speaking SMTP over an inter-
process(or) channel.
2.2.1. Argument vector/exit status
This technique is the standard UNIX method for
communicating with the process. A list of recipi-
ents is sent in the argument vector, and the mes-
sage body is sent on the standard input. Anything
that the mailer prints is simply collected and sent
back to the sender if there were any problems. The
exit status from the mailer is collected after the
message is sent, and a diagnostic is printed if
appropriate.
2.2.2. SMTP over pipes
The SMTP protocol [Postel82] can be used to
run an interactive lock-step interface with the
mailer. A subprocess is still created, but no
recipient addresses are passed to the mailer via
the argument list. Instead, they are passed one at
a time in commands sent to the processes standard
input. Anything appearing on the standard output
must be a reply code in a special format.
SENDMAIL -- An Internetwork Mail Router SMM:9-9
2.2.3. SMTP over an IPC connection
This technique is similar to the previous
technique, except that it uses a 4.2bsd IPC channel
[UNIX83]. This method is exceptionally flexible in
that the mailer need not reside on the same
machine. It is normally used to connect to a send-
mail process on another machine.
2.3. Operational Description
When a sender wants to send a message, it issues
a request to sendmail using one of the three methods
described above. Sendmail operates in two distinct
phases. In the first phase, it collects and stores
the message. In the second phase, message delivery
occurs. If there were errors during processing during
the second phase, sendmail creates and returns a new
message describing the error and/or returns an status
code telling what went wrong.
2.3.1. Argument processing and address parsing
If sendmail is called using one of the two
subprocess techniques, the arguments are first
scanned and option specifications are processed.
Recipient addresses are then collected, either from
the command line or from the SMTP RCPT command, and
a list of recipients is created. Aliases are
SMM:9-10 SENDMAIL -- An Internetwork Mail Router
expanded at this step, including mailing lists. As
much validation as possible of the addresses is
done at this step: syntax is checked, and local
addresses are verified, but detailed checking of
host names and addresses is deferred until deliv-
ery. Forwarding is also performed as the local
addresses are verified.
Sendmail appends each address to the recipient
list after parsing. When a name is aliased or for-
warded, the old name is retained in the list, and a
flag is set that tells the delivery phase to ignore
this recipient. This list is kept free from dupli-
cates, preventing alias loops and duplicate mes-
sages deliverd to the same recipient, as might
occur if a person is in two groups.
2.3.2. Message collection
Sendmail then collects the message. The mes-
sage should have a header at the beginning. No
formatting requirements are imposed on the message
except that they must be lines of text (i.e.,
binary data is not allowed). The header is parsed
and stored in memory, and the body of the message
is saved in a temporary file.
To simplify the program interface, the message
is collected even if no addresses were valid. The
SENDMAIL -- An Internetwork Mail Router SMM:9-11
message will be returned with an error.
2.3.3. Message delivery
For each unique mailer and host in the recipi-
ent list, sendmail calls the appropriate mailer.
Each mailer invocation sends to all users receiving
the message on one host. Mailers that only accept
one recipient at a time are handled properly.
The message is sent to the mailer using one of
the same three interfaces used to submit a message
to sendmail. Each copy of the message is prepended
by a customized header. The mailer status code is
caught and checked, and a suitable error message
given as appropriate. The exit code must conform
to a system standard or a generic message ("Service
unavailable") is given.
2.3.4. Queueing for retransmission
If the mailer returned an status that indi-
cated that it might be able to handle the mail
later, sendmail will queue the mail and try again
later.
2.3.5. Return to sender
If errors occur during processing, sendmail
returns the message to the sender for
SMM:9-12 SENDMAIL -- An Internetwork Mail Router
retransmission. The letter can be mailed back or
written in the file "dead.letter" in the sender's
home directory2.
2.4. Message Header Editing
Certain editing of the message header occurs
automatically. Header lines can be inserted under
control of the configuration file. Some lines can be
merged; for example, a "From:" line and a "Full-name:"
line can be merged under certain circumstances.
2.5. Configuration File
Almost all configuration information is read at
runtime from an ASCII file, encoding macro definitions
(defining the value of macros used internally), header
declarations (telling sendmail the format of header
lines that it will process specially, i.e., lines that
it will add or reformat), mailer definitions (giving
information such as the location and characteristics
of each mailer), and address rewriting rules (a lim-
ited production system to rewrite addresses which is
used to parse and rewrite the addresses).
____________________
2Obviously, if the site giving the error is not the orig-
inating site, the only reasonable option is to mail back to
the sender. Also, there are many more error disposition op-
tions, but they only effect the error message -- the "return
to sender" function is always handled in one of these two
ways.
SENDMAIL -- An Internetwork Mail Router SMM:9-13
To improve performance when reading the configu-
ration file, a memory image can be provided. This
provides a "compiled" form of the configuration file.
3. USAGE AND IMPLEMENTATION
3.1. Arguments
Arguments may be flags and addresses. Flags set
various processing options. Following flag arguments,
address arguments may be given, unless we are running
in SMTP mode. Addresses follow the syntax in RFC822
[Crocker82] for ARPANET address formats. In brief,
the format is:
(1) Anything in parentheses is thrown away (as a
comment).
(2) Anything in angle brackets ("<>") is preferred
over anything else. This rule implements the
ARPANET standard that addresses of the form
user name <machine-address>
will send to the electronic "machine-address"
rather than the human "user name."
(3) Double quotes ( " ) quote phrases; backslashes
quote characters. Backslashes are more power-
ful in that they will cause otherwise equiva-
lent phrases to compare differently -- for
SMM:9-14 SENDMAIL -- An Internetwork Mail Router
example, user and "user" are equivalent, but
\user is different from either of them.
Parentheses, angle brackets, and double quotes
must be properly balanced and nested. The rewriting
rules control remaining parsing3.
3.2. Mail to Files and Programs
Files and programs are legitimate message recipi-
ents. Files provide archival storage of messages,
useful for project administration and history. Pro-
grams are useful as recipients in a variety of situa-
tions, for example, to maintain a public repository of
systems messages (such as the Berkeley msgs program,
or the MARS system [Sattley78]).
Any address passing through the initial parsing
algorithm as a local address (i.e, not appearing to be
a valid address for another mailer) is scanned for two
special cases. If prefixed by a vertical bar ("|")
the rest of the address is processed as a shell com-
mand. If the user name begins with a slash mark ("/")
the name is used as a file name, instead of a login
name.
____________________
3Disclaimer: Some special processing is done after
rewriting local names; see below.
SENDMAIL -- An Internetwork Mail Router SMM:9-15
Files that have setuid or setgid bits set but no
execute bits set have those bits honored if sendmail
is running as root.
3.3. Aliasing, Forwarding, Inclusion
Sendmail reroutes mail three ways. Aliasing
applies system wide. Forwarding allows each user to
reroute incoming mail destined for that account.
Inclusion directs sendmail to read a file for a list
of addresses, and is normally used in conjunction with
aliasing.
3.3.1. Aliasing
Aliasing maps names to address lists using a
system-wide file. This file is indexed to speed
access. Only names that parse as local are allowed
as aliases; this guarantees a unique key (since
there are no nicknames for the local host).
3.3.2. Forwarding
After aliasing, recipients that are local and
valid are checked for the existence of a ".forward"
file in their home directory. If it exists, the
message is not sent to that user, but rather to the
list of users in that file. Often this list will
contain only one address, and the feature will be
used for network mail forwarding.
SMM:9-16 SENDMAIL -- An Internetwork Mail Router
Forwarding also permits a user to specify a
private incoming mailer. For example, forwarding
to:
"|/usr/local/newmail myname"
will use a different incoming mailer.
3.3.3. Inclusion
Inclusion is specified in RFC 733 [Crocker77a]
syntax:
:Include: pathname
An address of this form reads the file specified by
pathname and sends to all users listed in that
file.
The intent is not to support direct use of
this feature, but rather to use this as a subset of
aliasing. For example, an alias of the form:
project: :include:/usr/project/userlist
is a method of letting a project maintain a mailing
list without interaction with the system adminis-
tration, even if the alias file is protected.
It is not necessary to rebuild the index on
the alias database when a :include: list is
changed.
SENDMAIL -- An Internetwork Mail Router SMM:9-17
3.4. Message Collection
Once all recipient addresses are parsed and veri-
fied, the message is collected. The message comes in
two parts: a message header and a message body, sepa-
rated by a blank line.
The header is formatted as a series of lines of
the form
field-name: field-value
Field-value can be split across lines by starting the
following lines with a space or a tab. Some header
fields have special internal meaning, and have appro-
priate special processing. Other headers are simply
passed through. Some header fields may be added auto-
matically, such as time stamps.
The body is a series of text lines. It is com-
pletely uninterpreted and untouched, except that lines
beginning with a dot have the dot doubled when trans-
mitted over an SMTP channel. This extra dot is
stripped by the receiver.
3.5. Message Delivery
The send queue is ordered by receiving host
before transmission to implement message batching.
Each address is marked as it is sent so rescanning the
SMM:9-18 SENDMAIL -- An Internetwork Mail Router
list is safe. An argument list is built as the scan
proceeds. Mail to files is detected during the scan
of the send list. The interface to the mailer is per-
formed using one of the techniques described in sec-
tion 2.2.
After a connection is established, sendmail makes
the per-mailer changes to the header and sends the
result to the mailer. If any mail is rejected by the
mailer, a flag is set to invoke the return-to-sender
function after all delivery completes.
3.6. Queued Messages
If the mailer returns a "temporary failure" exit
status, the message is queued. A control file is used
to describe the recipients to be sent to and various
other parameters. This control file is formatted as a
series of lines, each describing a sender, a recipi-
ent, the time of submission, or some other salient
parameter of the message. The header of the message
is stored in the control file, so that the associated
data file in the queue is just the temporary file that
was originally collected.
3.7. Configuration
Configuration is controlled primarily by a con-
figuration file read at startup. Sendmail should not
SENDMAIL -- An Internetwork Mail Router SMM:9-19
need to be recomplied except
(1) To change operating systems (V6, V7/32V, 4BSD).
(2) To remove or insert the DBM (UNIX database)
library.
(3) To change ARPANET reply codes.
(4) To add headers fields requiring special pro-
cessing.
Adding mailers or changing parsing (i.e., rewriting)
or routing information does not require recompilation.
If the mail is being sent by a local user, and
the file ".mailcf" exists in the sender's home direc-
tory, that file is read as a configuration file after
the system configuration file. The primary use of
this feature is to add header lines.
The configuration file encodes macro definitions,
header definitions, mailer definitions, rewriting
rules, and options.
3.7.1. Macros
Macros can be used in three ways. Certain
macros transmit unstructured textual information
into the mail system, such as the name sendmail
will use to identify itself in error messages.
SMM:9-20 SENDMAIL -- An Internetwork Mail Router
Other macros transmit information from sendmail to
the configuration file for use in creating other
fields (such as argument vectors to mailers); e.g.,
the name of the sender, and the host and user of
the recipient. Other macros are unused internally,
and can be used as shorthand in the configuration
file.
3.7.2. Header declarations
Header declarations inform sendmail of the
format of known header lines. Knowledge of a few
header lines is built into sendmail, such as the
"From:" and "Date:" lines.
Most configured headers will be automatically
inserted in the outgoing message if they don't
exist in the incoming message. Certain headers are
suppressed by some mailers.
3.7.3. Mailer declarations
Mailer declarations tell sendmail of the vari-
ous mailers available to it. The definition speci-
fies the internal name of the mailer, the pathname
of the program to call, some flags associated with
the mailer, and an argument vector to be used on
the call; this vector is macro-expanded before use.
SENDMAIL -- An Internetwork Mail Router SMM:9-21
3.7.4. Address rewriting rules
The heart of address parsing in sendmail is a
set of rewriting rules. These are an ordered list
of pattern-replacement rules, (somewhat like a pro-
duction system, except that order is critical),
which are applied to each address. The address is
rewritten textually until it is either rewritten
into a special canonical form (i.e., a (mailer,
host, user) 3-tuple, such as {arpanet, usc-isif,
postel} representing the address "postel@usc-
isif"), or it falls off the end. When a pattern
matches, the rule is reapplied until it fails.
The configuration file also supports the edit-
ing of addresses into different formats. For exam-
ple, an address of the form:
ucsfcgl!tef
might be mapped into:
[email protected]
to conform to the domain syntax. Translations can
also be done in the other direction.
3.7.5. Option setting
There are several options that can be set from
the configuration file. These include the
SMM:9-22 SENDMAIL -- An Internetwork Mail Router
pathnames of various support files, timeouts,
default modes, etc.
4. COMPARISON WITH OTHER MAILERS
4.1. Delivermail
Sendmail is an outgrowth of delivermail. The
primary differences are:
(1) Configuration information is not compiled in.
This change simplifies many of the problems of
moving to other machines. It also allows easy
debugging of new mailers.
(2) Address parsing is more flexible. For example,
delivermail only supported one gateway to any
network, whereas sendmail can be sensitive to
host names and reroute to different gateways.
(3) Forwarding and :include: features eliminate the
requirement that the system alias file be
writable by any user (or that an update program
be written, or that the system administration
make all changes).
(4) Sendmail supports message batching across net-
works when a message is being sent to multiple
recipients.
SENDMAIL -- An Internetwork Mail Router SMM:9-23
(5) A mail queue is provided in sendmail. Mail
that cannot be delivered immediately but can
potentially be delivered later is stored in
this queue for a later retry. The queue also
provides a buffer against system crashes; after
the message has been collected it may be reli-
ably redelivered even if the system crashes
during the initial delivery.
(6) Sendmail uses the networking support provided
by 4.2BSD to provide a direct interface net-
works such as the ARPANET and/or Ethernet using
SMTP (the Simple Mail Transfer Protocol) over a
TCP/IP connection.
4.2. MMDF
MMDF [Crocker79] spans a wider problem set than
sendmail. For example, the domain of MMDF includes a
"phone network" mailer, whereas sendmail calls on pre-
existing mailers in most cases.
MMDF and sendmail both support aliasing, cus-
tomized mailers, message batching, automatic forward-
ing to gateways, queueing, and retransmission. MMDF
supports two-stage timeout, which sendmail does not
support.
SMM:9-24 SENDMAIL -- An Internetwork Mail Router
The configuration for MMDF is compiled into the
code4.
Since MMDF does not consider backwards compati-
bility as a design goal, the address parsing is sim-
pler but much less flexible.
It is somewhat harder to integrate a new channel5
into MMDF. In particular, MMDF must know the location
and format of host tables for all channels, and the
channel must speak a special protocol. This allows
MMDF to do additional verification (such as verifying
host names) at submission time.
MMDF strictly separates the submission and deliv-
ery phases. Although sendmail has the concept of each
of these stages, they are integrated into one program,
whereas in MMDF they are split into two programs.
4.3. Message Processing Module
The Message Processing Module (MPM) discussed by
Postel [Postel79b] matches sendmail closely in terms
of its basic architecture. However, like MMDF, the
MPM includes the network interface software as part of
its domain.
____________________
4Dynamic configuration tables are currently being consid-
ered for MMDF; allowing the installer to select either com-
piled or dynamic tables.
5The MMDF equivalent of a sendmail "mailer."
SENDMAIL -- An Internetwork Mail Router SMM:9-25
MPM also postulates a duplex channel to the
receiver, as does MMDF, thus allowing simpler handling
of errors by the mailer than is possible in sendmail.
When a message queued by sendmail is sent, any errors
must be returned to the sender by the mailer itself.
Both MPM and MMDF mailers can return an immediate
error response, and a single error processor can cre-
ate an appropriate response.
MPM prefers passing the message as a structured
object, with type-length-value tuples6. Such a con-
vention requires a much higher degree of cooperation
between mailers than is required by sendmail. MPM
also assumes a universally agreed upon internet name
space (with each address in the form of a net-host-
user tuple), which sendmail does not.
5. EVALUATIONS AND FUTURE PLANS
Sendmail is designed to work in a nonhomogeneous
environment. Every attempt is made to avoid imposing
unnecessary constraints on the underlying mailers. This
goal has driven much of the design. One of the major
problems has been the lack of a uniform address space, as
postulated in [Postel79a] and [Postel79b].
____________________
6This is similar to the NBS standard.
SMM:9-26 SENDMAIL -- An Internetwork Mail Router
A nonuniform address space implies that a path will
be specified in all addresses, either explicitly (as part
of the address) or implicitly (as with implied forwarding
to gateways). This restriction has the unpleasant effect
of making replying to messages exceedingly difficult,
since there is no one "address" for any person, but only
a way to get there from wherever you are.
Interfacing to mail programs that were not initially
intended to be applied in an internet environment has
been amazingly successful, and has reduced the job to a
manageable task.
Sendmail has knowledge of a few difficult environ-
ments built in. It generates ARPANET FTP/SMTP compatible
error messages (prepended with three-digit numbers [Nei-
gus73, Postel74, Postel82]) as necessary, optionally gen-
erates UNIX-style "From" lines on the front of messages
for some mailers, and knows how to parse the same lines
on input. Also, error handling has an option customized
for BerkNet.
The decision to avoid doing any type of delivery
where possible (even, or perhaps especially, local deliv-
ery) has turned out to be a good idea. Even with local
delivery, there are issues of the location of the mail-
box, the format of the mailbox, the locking protocol
used, etc., that are best decided by other programs. One
SENDMAIL -- An Internetwork Mail Router SMM:9-27
surprisingly major annoyance in many internet mailers is
that the location and format of local mail is built in.
The feeling seems to be that local mail is so common that
it should be efficient. This feeling is not born out by
our experience; on the contrary, the location and format
of mailboxes seems to vary widely from system to system.
The ability to automatically generate a response to
incoming mail (by forwarding mail to a program) seems
useful ("I am on vacation until late August....") but
can create problems such as forwarding loops (two people
on vacation whose programs send notes back and forth, for
instance) if these programs are not well written. A pro-
gram could be written to do standard tasks correctly, but
this would solve the general case.
It might be desirable to implement some form of load
limiting. I am unaware of any mail system that addresses
this problem, nor am I aware of any reasonable solution
at this time.
The configuration file is currently practically
inscrutable; considerable convenience could be realized
with a higher-level format.
It seems clear that common protocols will be chang-
ing soon to accommodate changing requirements and envi-
ronments. These changes will include modifications to
the message header (e.g., [NBS80]) or to the body of the
SMM:9-28 SENDMAIL -- An Internetwork Mail Router
message itself (such as for multimedia messages [Pos-
tel80]). Experience indicates that these changes should
be relatively trivial to integrate into the existing sys-
tem.
In tightly coupled environments, it would be nice to
have a name server such as Grapvine [Birrell82] inte-
grated into the mail system. This would allow a site
such as "Berkeley" to appear as a single host, rather
than as a collection of hosts, and would allow people to
move transparently among machines without having to
change their addresses. Such a facility would require an
automatically updated database and some method of resolv-
ing conflicts. Ideally this would be effective even
without all hosts being under a single management. How-
ever, it is not clear whether this feature should be
integrated into the aliasing facility or should be con-
sidered a "value added" feature outside sendmail itself.
As a more interesting case, the CSNET name server
[Solomon81] provides an facility that goes beyond a sin-
gle tightly-coupled environment. Such a facility would
normally exist outside of sendmail however.
ACKNOWLEDGEMENTS
Thanks are due to Kurt Shoens for his continual cheer-
ful assistance and good advice, Bill Joy for pointing me in
the correct direction (over and over), and Mark Horton for
SENDMAIL -- An Internetwork Mail Router SMM:9-29
more advice, prodding, and many of the good ideas. Kurt and
Eric Schmidt are to be credited for using delivermail as a
server for their programs (Mail and BerkNet respectively)
before any sane person should have, and making the necessary
modifications promptly and happily. Eric gave me consider-
able advice about the perils of network software which saved
me an unknown amount of work and grief. Mark did the origi-
nal implementation of the DBM version of aliasing, installed
the VFORK code, wrote the current version of rmail, and was
the person who really convinced me to put the work into
delivermail to turn it into sendmail. Kurt deserves acco-
lades for using sendmail when I was myself afraid to take
the risk; how a person can continue to be so enthusiastic in
the face of so much bitter reality is beyond me.
Kurt, Mark, Kirk McKusick, Marvin Solomon, and many
others have reviewed this paper, giving considerable useful
advice.
Special thanks are reserved for Mike Stonebraker at
Berkeley and Bob Epstein at Britton-Lee, who both knowingly
allowed me to put so much work into this project when there
were so many other things I really should have been working
on.
REFERENCES
[Birrell82] Birrell, A. D., Levin, R., Needham, R. M.,
and Schroeder, M. D., "Grapevine: An Exercise
in Distributed Computing." In Comm. A.C.M.
25, 4, April 82.
[Borden79] Borden, S., Gaines, R. S., and Shapiro, N.
Z., The MH Message Handling System: Users'
Manual. R-2367-PAF. Rand Corporation.
October 1979.
[Crocker77a] Crocker, D. H., Vittal, J. J., Pogran, K. T.,
and Henderson, D. A. Jr., Standard for the
Format of ARPA Network Text Messages. RFC
733, NIC 41952. In [Feinler78]. November
1977.
[Crocker77b] Crocker, D. H., Framework and Functions of
the MS Personal Message System. R-2134-ARPA,
Rand Corporation, Santa Monica, California.
1977.
[Crocker79] Crocker, D. H., Szurkowski, E. S., and Far-
ber, D. J., An Internetwork Memo Distribution
Facility -- MMDF. 6th Data Communication
Symposium, Asilomar. November 1979.
SMM:9-30 SENDMAIL -- An Internetwork Mail Router
SENDMAIL -- An Internetwork Mail Router SMM:9-31
[Crocker82] Crocker, D. H., Standard for the Format of
Arpa Internet Text Messages. RFC 822. Net-
work Information Center, SRI International,
Menlo Park, California. August 1982.
[Metcalfe76] Metcalfe, R., and Boggs, D., "Ethernet: Dis-
tributed Packet Switching for Local Computer
Networks", Communications of the ACM 19, 7.
July 1976.
[Feinler78] Feinler, E., and Postel, J. (eds.), ARPANET
Protocol Handbook. NIC 7104, Network Infor-
mation Center, SRI International, Menlo Park,
California. 1978.
[NBS80] National Bureau of Standards, Specification
of a Draft Message Format Standard. Report
No. ICST/CBOS 80-2. October 1980.
[Neigus73] Neigus, N., File Transfer Protocol for the
ARPA Network. RFC 542, NIC 17759. In [Fein-
ler78]. August, 1973.
[Nowitz78a] Nowitz, D. A., and Lesk, M. E., A Dial-Up
Network of UNIX Systems. Bell Laboratories.
In UNIX Programmer's Manual, Seventh Edition,
Volume 2. August, 1978.
[Nowitz78b] Nowitz, D. A., Uucp Implementation Descrip-
tion. Bell Laboratories. In UNIX
SMM:9-32 SENDMAIL -- An Internetwork Mail Router
Programmer's Manual, Seventh Edition, Volume
2. October, 1978.
[Postel74] Postel, J., and Neigus, N., Revised FTP Reply
Codes. RFC 640, NIC 30843. In [Feinler78].
June, 1974.
[Postel77] Postel, J., Mail Protocol. NIC 29588. In
[Feinler78]. November 1977.
[Postel79a] Postel, J., Internet Message Protocol. RFC
753, IEN 85. Network Information Center, SRI
International, Menlo Park, California. March
1979.
[Postel79b] Postel, J. B., An Internetwork Message Struc-
ture. In Proceedings of the Sixth Data Com-
munications Symposium, IEEE. New York.
November 1979.
[Postel80] Postel, J. B., A Structured Format for Trans-
mission of Multi-Media Documents. RFC 767.
Network Information Center, SRI Interna-
tional, Menlo Park, California. August 1980.
[Postel82] Postel, J. B., Simple Mail Transfer Protocol.
RFC821 (obsoleting RFC788). Network Informa-
tion Center, SRI International, Menlo Park,
California. August 1982.
SENDMAIL -- An Internetwork Mail Router SMM:9-33
[Schmidt79] Schmidt, E., An Introduction to the Berkeley
Network. University of California, Berkeley
California. 1979.
[Shoens79] Shoens, K., Mail Reference Manual. Univer-
sity of California, Berkeley. In UNIX Pro-
grammer's Manual, Seventh Edition, Volume 2C.
December 1979.
[Sluizer81] Sluizer, S., and Postel, J. B., Mail Transfer
Protocol. RFC 780. Network Information Cen-
ter, SRI International, Menlo Park, Califor-
nia. May 1981.
[Solomon81] Solomon, M., Landweber, L., and Neuhengen,
D., "The Design of the CSNET Name Server."
CS-DN-2, University of Wisconsin, Madison.
November 1981.
[Su82] Su, Zaw-Sing, and Postel, Jon, The Domain
Naming Convention for Internet User Applica-
tions. RFC819. Network Information Center,
SRI International, Menlo Park, California.
August 1982.
[UNIX83] The UNIX Programmer's Manual, Seventh Edi-
tion, Virtual VAX-11 Version, Volume 1. Bell
Laboratories, modified by the University of
California, Berkeley, California. March,
SMM:9-34 SENDMAIL -- An Internetwork Mail Router
1983.
Society
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
Quotes
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
Classic books:
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|
|
You can use PayPal to to buy a cup of coffee for authors of this site |
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.
Last modified: March 12, 2019
