Sendmail performance tuning

Most people consider email to be the Internet "killer application". In a large corporation a typical Sendmail server often serves as mail router in DMZ and typically deals with volumes of (spam-filtered) 7000-10000 emails per minute at peak (max rate does not mean that that many emails are processed; peak can be just injection of 2K emails in 15 sec interval)

Peak email server loads is limited to relatively short period (rush hours). The latter are typically between 8AM and 11 AM. There can be several spikes before that, for example, if you are using Postini for spam filtering -- it sends emails to each user of the system. I think that by default it does it at 5 am.

Average email load per 24 hours on workdays (Mn-Fr) is much lower. Typically between 100-300 emails per minute. But that number hides high diversity email length (distribution that has fat tails).

Weekends typically have email traffic that is three-five times lower (again we are talking about spam-filtered rates; raw rate are higher as San, Sun are favorite days for spammers). A typical problem is distribution of company email bulletins when "stupid" list distribution software injects into email stream, say 10K messages in a very short period of time.

Sendmail performance depends on several factors, but the key is the disk system performance. Generally there can be other reasons although this happens very, very rarely.

Disk bottlenecks

This is most typical problem for Sendmail. It demonstrates itself that during peak mail load uptime shoots far above number_of_CPUs * 4 multiple. It does not make since to use Sendmail servers with more then one CPU so for all practical purposes number of CPUs is now 4. That means uptime above 16 is unhealthy even if it is transitory and occurred only at peaks.

Sendmail has internal thresholds for high uptime value and stop accepting emails at some point. By default, Sendmail:

• Switches to queue-only delivery mode when the system load is 8 * number of processors.
• Switches to rejecting MTA/MSA connections when the system load is 12 * number of processors

Those setting are configurable and can be changed. Reaction of mail storm by-and-large determined by the type of storage used:

• Single pair dedicated mirrored 15K RPM local disks allows approximately the max mail rate of 1.5K emails/minute before uptime shoots above the "choking threshold".
• SAN improves this to approximately 3K emails per minute. I think that twelve 15K RPM local disks with caching controller that has 1GB of cache can give SAN a run for the money.
• TMPFS more then doubles that. Moving emails filesystem to TMPFS (easy both on Solaris and Linux) usually gives tremendous improvement over magnetic media. The drawback is that if power is lost and there is no UPS some mails will be lost. ioDrive or SSI drives are helpful too but more expensive.

Over the years, I/O subsystem in servers and operating systems have grown increasingly sophisticated but basic remain the same. It is important to remember that two mirrored disks are inadequate. You need at least 6 disks per controller to get adequate performance. Raid 10 can improve performance further with the cost of doubling the number of disks. Disks are cheap those days and you can use small individual disks this way, for example twelve 73GB or 146GB 15 RPM drives is an acceptable starting combination. Dell Seagate ST3146855SS 146GB 15K SAS is around $200 on Amazon so the whole subsystem will cost you less then $3K.

Today, several levels of data caches are used.

• Operating system caches file so the more RAM you have the bigger is cache. Those days it does not make much sense to run enterprise mail server with less then 16GB of RAM.
• Disk controller caches data and one gigabyte controller cache is strongly recommended.
• Each harddrive also has small (8M or so) cache. So the more harddrive you have the better.

In case of SAN the cache can be quote large (several gigabytes) and this better performance then individual pair of local disks. I suspect that difference in performance between SAN and a number of local disk with, say, 1GB cache of controller is not that great.

The problem with SAN is that it is too expensive to have it dedicated and it is shared, you are depending on the strangers for the level of performance of I/O subsystem. That's never good for mission critical severs like enterprise Sendmail. If you want to use SAN and have money to do so. its preferable to buy a smaller dedicated small unit or use a shared a unit that you know for sure is undersubscribed (have hew other connections). SAN behaviors is dependent of the number of other subscribers and the load they experience; you generally does not want that effect to be prominent.

But again usually it does not pay to use actual storage. TMPS with a couple of local UPS for the server provides much, much better performance.

On Linux the syslog daemon fsync() log files after each entry is written to them. If a server handles, say, 6000 emails per minute (100 mailed per second) in addition to writing file to mail queue, it writes up to 300 log entries (3 for each email) per second. On 15K RPM disks this load is not substantial, especially if /var subsystem is on a separate pair of hard drives. Generally is easy to detect from iostat data for the /var partition. In such cases moving /var/log on a separate harddrive (the same as swap), usage of SAN storage for OS including /var partition is a good idea.

Usual non-issues

• CPU-Bound Systems. With modern CPUs that's rarely happens so you need to ensure that you are looking at the right bottleneck. Most commonly CPU bottleneck really masks another problem like insufficient memory — for example, the system may be working very hard to move processes in and out of swap space. You need to correlate CPU utilization with I/O utilization. Sendmail can run many processes simultaneously to handle multiple requests.
• Memory-Bound Systems. As we’ve already learned, while Sendmail does not consume too much memory by itself, decent enterprise Sendmail server should have 16GB or more of memory to allow the use of TMPS for mail queue (8GB is more then enough for a typical server). With such amount memory bottleneck should not ever occur. A typical sign of memory bottleneck is the use of swap. On modern production servers it generally should be zero.

TMPS as the Quick Fix for overloaded server

If server uses local storage and gets into a jam the best quick fix is to move mail query to TMFS. This can be done in a couple of minute by creating TMFS filesystem, mounting it and then modifying Sendmail configuration and restarting Sendmail. After that old messages can wait until the jam is over and then delivered. They can also can be delivered from a different server if the infrastructure has several received headers will be wrong in this case.

One also could change the RefuseLA parameter in the sendmail.cf file and restart sendmail. Too high value usually does not help. Sometimes you can lower it.

But the truth is that generally short-term fixes rarely help. In most cases it's best to just let the server work its way out of a jam and then reconfigure it to TMFS to void the repetition of the situation the next day. on a 16GB Linux server it is enough to allocate 8GB to TMFS.

Using mounting of a queue to SAN and TMFS as an optimization tool

Sendmail will use by default a single mail queue. For high traffic mail servers it is useful to split the queue over several directories, as the queue sequentially can slow down the system.

Frist you need to create actual directories as sendmail will not do this by default:

mkdir /var/spool/mqueue/q{1,2,3,4,5,6,7,8}

Permissions should be something like smmta:smmsp

chown -R smmta:smmsp /var/spool/mqueue/q*

Next, we need to enable the multiple queues in the sendmail configuration. To do this you need to edit sendmail.mc and append one line:

define(`QUEUE_DIR', `/var/spool/mqueue/q*')dnl

m4 sendmail.mc > /etc/mail/sendmail.cf

After restarting sendmail, you can check the results by running mailq (or sendmail -bp) that will output each of the queues:

mailq 
/var/spool/mqueue/q6 is empty
/var/spool/mqueue/q4 is empty
...
Total requests: 0

Note:

• if you want to increase the number of folders all you have to do is to create the respective folders, set the appropriate permissions and restart sendmail.
• Unique across queues so you can move the items among queues if you have to.

The Runners= (R=) queue-group equate tells sendmail how many queue processors to launch each queue-processing interval. The queues are serviced in round-robin order. So, for example, if your queue group has three queues, you can set R= to 1, 2, 3, or 4, respectively,

The Runners= queue-group equate is declared like the following:

Runners=12 #  12 per queue run
Runners=0 #  no limit, so one per queue each queue run 
Runners=none #  the same as R=0 

If the number of queue-group runners specified by this equate is more than the number of queue children allowed by the MaxQueueChildren option (§24.9.71 on page 1049), the number of queue-group runners is reduced to that amount, and the following error is logged and printed:

Q=queuegroup: R=number exceeds MaxQueueChildren=limit,set to MaxQueueChildren

If the MaxQueueChildren option is set to zero, there is no limit to how many queuegroup runners you can declare.

Additional possibilities for optimization arise from the fact that you can mount different queries to different partitions and prescribe usage of certain queries for certain senders.

Here is the quote from Sendmail book about queue group feature that allow assign mail to different queues based on the sender :

How to Declare Queue Groups with the m4 Technique

You declare queue groups inside your mc configuration file withth e QUEUE_GROUP mc configuration macro. As you saw in the previous sections, it is used like this:

QUEUE_GROUP(`group name', `equates')

The queue group name can contain any characters except a comma or a whitespace
character (a space or a tab).* It must not be surrounded (inside the quotes) with whitespace characters. The equates form the second argument to the QUEUE_GROUP mc configuration macro. The equates are described in 11.4.2 on page 409. To illustrate, consider the following QUEUE_GROUP mc configuration macro declaration:

QUEUE_GROUP(`slowmail', `P=/var/spool/mqueues/slowqueue')

Here, the name of the queue group is set to slowmail. The second argument is a single equate, the P= queue-group equate, which defines the queue directory or directories to be used by this queue group.

If you want to define which queue group to use for certain delivery agents, you can use the Q= delivery-agent equate (see 20.5.12 on page 750) as set, for example, with the LOCAL_MAILER_QGRP mc macro. For example, the following tells sendmail to queue all local mail in the /queues/lq queue directory:

QUEUE_DIR(`/queue') 
QUEUE_GROUP(`localgroup', `P=/queue/lq') 
define(`LOCAL_MAILER_QGRP', `localgroup')<- must be before MAILER(local) 
MAILER(`local')
	

In the first line we set the default queue directory. In the second line we define the queue group localgroup, and set its queue directory to be /queue/lq. In the third line we declare that the Q= equate for the local delivery agent will be:

		Q=localgroup
	

The fourth line declares support for the local delivery agent. Note that the definition of LOCAL_MAILER_QGRP must precede the MAILER(local); otherwise, that definition will be silently ignored. Those four lines cause all mail for local users to be queued in the /queue/lq directory. Note that you can dedicate queue groups for other delivery agents. See 20.5.12 on page 750 for a full description of this process.

The FEATURE(queuegroup) and the access Database

The easiest way to select queue groups based on recipient addresses or recipient domains is by using the FEATURE(queuegroup). It is declared in your mc configuration file like this:

FEATURE(`queuegroup')  
FEATURE(`queuegroup', `default group')

The first line causes the queue group to default to mqueue if a queue group in the access database is missing or nonexistent. The second line allows you to set a different default queue group. For example, consider the following lines from an mc file:

QUEUEGROUP(`localgroup', `/queue/lq') 
FEATURE(`queuegroup', `localgroup')

This causes sendmail to use the group named localgroup instead of mqueue as the default if a queue group in the access database is missing or nonexistent. Once you have enabled the FEATURE(queuegroup), the next step is to add lines such as the following to the source file for your access database:

QGRP:slow-poke.com slowgroup  
QGRP:[email protected] fastgroup  
QGRP:your.domain localgroup

Each line that selects queue groups must begin with the literal expression:

QGRP:

This prefix tells sendmail that you wish to map recipient addresses or domains to queue groups. The first line causes mail to the slow-poke.com domain to use the queue group called slowgroup. This shows that you can list just a domain in the lefthand column and it will work just as expected. The second line causes mail to the specific recipient [email protected] to use the queue group named fastgroup. This line demonstrates that mail to an individual can be used in the lefthand column. The third line illustrates your local domain, which shows that mail to your domain, your.domain, will use the queue group named localgroup.

Running the second instance of sendmail for non critical mail

You can have two instances of sendmail. One that uses NAS and process all regular mail and the second that uses TMFS and processes all non-critical mail such as billeting, etc.

Tools

We will use GNU version in this discussion.

Notes:

• the script command can be used to capture output of commands like iostat and vmstat

• There is a side effect of monitoring but on modern servers is close to non-existent unless you really go overboard:

  it is impossible to measure a system without affecting it. Just by running a tool we necessarily change the behavior of the very computer we’re monitoring. These utilities consume memory and CPU time, they open sockets and files, and they read data off disks. Therefore, we can never be entirely sure that a problem that we observe on a system isn’t at least partially influenced by the fact that we’re monitoring it.

  Continually running top each 10 sec or recoding uptime each 10 sec usually do not have adverse affected on modern servers.

• Uptime. The memory information displayed by top is slightly misleading. Due to caching, we should expect the amount of free memory listed to stay very near zero. On contemporary operating systems, any RAM that goes unused by processes will be allocated to caching some data. Thus, just because there is very little memory free, it doesn’t mean that the system is memory starved. On some operating systems, top will show more memory information, such as how much RAM is allocated to filesystem caches; if this number drops near zero, it would likely indicate that the server would use additional RAM effectively.
  
• iostat. The iostat is similar to vmstat, except that it measures system I/O rather than virtual memory statistics. Typically, iostat reports its data as kilobytes per second or transfers per second. Some versions also show how long the average transfer takes, svc_t in this example, which can be very useful metric for determining loading. If this number starts going up, it indicates that the device is heavily loaded. On Solaris and some recent versions of Linux, the -x flag gives even more valuable information, as in this example, including the average amount of time each request spends in the wait queue and the percentage of time I/O requests are waiting to be serviced by the disk device. These numbers represent some of the best indicators of disk contention in the absence of a baseline, but they’re no substitute for one. A disk can be 100% busy and yet the system can still provide adequate service. Knowing that a disk always has requests sitting in the wait queue doesn’t explain why a change in server behavior has occurred. If kilobytes per second increases while tps remains constant, it would indicate that we’re dealing with larger requests, which may alert us to a temporary or permanent change in the type of email flowing through the system. Because email servers so often become I/O bound, iostat may be the single most important utility in the email administrator’s toolkit.
  
• vmstat The first column, labeled r, indicates the number of runnable processes, which provides a snapshot of the system load average. In this example, a number of processes want to run but can’t because they have no CPU time slice available to them. The second column, labeled b, gives the number of processes that are blocked from proceeding because they are waiting for I/O. If a significant number of processes are listed in this column, the system is likely I/O bound. In our example, we occasionally see a blocked process, but this event is rare, giving us an indication that this system isn’t I/O bound.
  

• netstat. The third tool in the “*stat” trio is netstat. As one would expect, netstat provides information about system networking. It can display either a snapshot of very detailed information about nearly every conceivable network parameter (netstat -s) or periodic data like that found with vmstat or iostat (e.g., netstat -w 5 on BSD systems, netstat -i 5 on Solaris, or netstat -c on Linux).

• Obviously, in its periodic mode, some of the parameters provided by netstat that we want to carefully observe include the number of packets per second and the number of bytes per second. Both statistics, and especially trends in them, can provide the most direct information on the objective external load on a system, so they should be tracked. How the ratio of input to output statistics might change can also be highly informative.
  
• sar On System V-derived UNIX versions, you can run the System Activity Reporter (sar) program in the background to gather statistics and accounting information, including much of the data reported by the tools that have already been mentioned in this section. It is an excellent baselining tool, and collecting data every 1 minutes on a system via sar and archiving those data is something that every server administrator should seriously consider. One problem is that sar is uncable to measure uptime each 10 sec.
  
• ps ps can tell you the number of sendmail processes...
  
• The syslog package was originally developed as a part of the sendmail distribution to help maintain the information it would log. It was later adopted by the rest of Berkeley UNIX around the BSD 4.1 timeframe, and from there spread to other UNIX versions. Now it has become so ubiquitous that the fact that its origin is tied to sendmail has been largely forgotten. During an SMTP message reception, sendmail logs the sender information at the end of the message transaction, whether the message is actually sent or not. If the message is accepted, then the log entry occurs after the end of the DATA phase. If the message is rejected, then the log entry is made immediately after the rejection. During a successful SMTP message reception, sendmail logs the recipient information at the end of the session, although the precise timing can vary depending on sendmail’s delivery mode. At the conclusion of each failed delivery attempt, that attempt is also logged. At least two log entries for each successful delivery and one log entry for each unsuccessful attempt, plus various other entries for start-up, encountering errors, STARTTLS information, and so on, will be made. The two types of log entries mentioned initially make up the bulk of the log messages generated, however, and are the two that occur as a result of a successful delivery. All of these log messages can result in a lot of data, and this information can provide significant insight into what is happening on the server.
  
• Other. Sometimes utilities like pstat, lsof, and pstack can be useful. But that happens rarely.

Removing Bottlenecks

Here are points from "Sendmail Performance Tuning" by Nick Christenson that are most relevant:

1. On a high-performance or mission-critical server, make even small changes with extreme caution. It you misdiagnose the case you can make the situation worse not better.
2. Apply one change at a time and thorously document each.
3. When something is going wrong, the reasons may not be evident, but misbehaviors always has some internal logic. At the end of the day computers are deterministic objects.
4. Test, test, and test again before deploying changes to important production system like the primary company mailserver. Make the test as close to “real life” as possible. There’s never enough time to be as thorough as one would like, but if there isn’t enough time to test at least some extreme cases, there isn’t enough time to do it right.

Baselining the system

The most useful troubleshooting technique is preventive in nature—baselining the system. To understand what’s going wrong when the system behaves poorly, it is crucial to understand to collect the data about its behaviors in "normal" mode. On a critical Sendmail server, an administrator should tar the content of /etc directory with each change so that he can compare it later if problem will be detected.

Sar is also useful tool as you can keep records for a long time and they are pretty compact.

It is important it is to distinguish information related to a present problem from incidental information. Without a baseline, it can be difficult—if not impossible—to tell whether a given piece of information is even out of the ordinary. When something goes wrong, while looking for the source of the problem we’ve all encountered something unexpected and asked ourselves, “Was this always like that?” Baselining reduces the number of times this uncertainty will arise in a crisis, which should lead to faster problem resolution.

Run baseline tests periodically and compare their results against previous test runs. Going the extra mile and performing a more formal trend analysis can prove very valuable, too. It offers two benefits. First, it enables one to spot situations that slowly are evolving into problems before they become noticeable. Of course, not all changes represent problems waiting to happen, but trend analysis can also spot secular changes in the way a server operates, which may indicate new patterns in user behavior or changes in Internet operation.

Sar data analysis allows administrators to become more familiar with the servers they are charged with maintaining, which is a very positive side effect. Better understanding of the server and its workload helps to resolve problems sooner and quicker.

Top Visited					Switchboard
					Latest
					Past week
					Past month

NEWS CONTENTS

• 20110928 : MTA Performance Tuning Tips for Sendmail - Documentation - Confluence ( MTA Performance Tuning Tips for Sendmail - Documentation - Confluence, Sep 28, 2011 )
• 20110923 : Sendmail Performance Tuning for Large Systems by Brad Knowles ( Old but still good advice. Actually deeper understanding of issues then in Nick Christenson's Sendmail Performance Tuning Reviews )
• 20101029 : Maximum Performance Rate of Sendmail ( Maximum Performance Rate of Sendmail, Oct 29, 2010 )
• 20101029 : QueueLA and RefuseLA ( QueueLA and RefuseLA, )

Old News ;-)

[Sep 28, 2011] MTA Performance Tuning Tips for Sendmail - Documentation - Confluence

If you are looking for online resources with regards to this subject, see Nick's slides for his talk entitled "Performance Tuning Sendmail Systems" (see http://www.jetcafe.org/~npc/doc/performance_tuning.pdf). Then there are the slides for the talk by Brad Knowles entitled "Sendmail Performance Tuning for Large Systems" (see http://www.shub-internet.org/brad/papers/sendmail-tuning/). You may also be interested in Brad's paper "MTA Performance Comparison: sendmail, postfix, & exim on *BSD" (see http://www.shub-internet.org/brad/papers/mtacomparison/).

For papers that are specific to the issue of tuning sendmail for use with mailing lists, read Rob Kolstad's paper "Tuning Sendmail for Large Mailing Lists" (see http://www.usenix.org/publications/library/proceedings/lisa97/21.kolstad.html), and Strata Chalup's paper "Drinking from the Fire(walls) Hose: Another Approach to Very Large Mailing Lists" (see http://www.usenix.org/publications/library/proceedings/lisa98/chalup.html).

---

Other comments (taken from the mailing lists):

  http://mail.python.org/pipermail/mailman-users/2001-May/011257.html
  http://mail.python.org/pipermail/mailman-developers/2001-August/009329.html

If you're running anything less than sendmail 8.11 – upgrade, and spend some time configuring subdirectories to reduce I/O load and contention in the directory structures (QueueDirectory=/var/spool/mqueue/q* where * = at least 5 directories, with appropriate df/qf/xf subdirs).

If you aren't running DNS on that local machine, set up a caching-only name server. You'll see a significant increase in your performance by removing the network interactions, even if you're dealing with a DNS server on your local LAN.

---

I know that tweaking sendmail.cf files gives many people hives, and so people aren't likely to do it. It's also unneccesary.

You can do this without modifying your sendmail files at all. Instead, in your startup script, add:

        /usr/sbin/sendmail -bd -ODeliveryMode=defer \
                -ODaemonPortOptions=Name=MSA,Port=NNNN,M=E,Addr=127.0.0.1

Where NNNN is some port number not otherwise used (you can test if something's in use by doing "telnet localhost NNNN" – if it's refused, there's no daemon listening)

This sets up a sendmail process listening to the alternate port, in DEFER mode, but set to talk only to the localhost interface, so it's not accessible by anyoneother than your local machine: no open relay problems.

To make mailman access that port, add this to your mm_cfg.py:

 # define alternate SMTP port
 SMTPPORT = NNNN

I've been running this fine for about a week, and I'm quite comfortable with it. It works with sendmail 8.10 and later and doesn't require a sendmail god to implement.

---

Both of these answers culled from messages by Chuq Von Rospach [email protected]

Last changed on Tue Sep 5 22:58:45 2006 by Brad Knowles

[Sep 23, 2011] Sendmail Performance Tuning for Large Systems by Brad Knowles

Old but still good advice. Actually deeper understanding of issues then in Nick Christenson's Sendmail Performance Tuning Reviews

[Oct 29, 2010] Maximum Performance Rate of Sendmail

PHWinfo

Andrzej Adam Filip

Re: Maximum Performance Rate of Sendmail ashish <[email protected]> wrote:

> I did search for this topic but was not able to find the specific
> answer which I am looking for so thought of asking this group. For my
> project we are using sendmail as an intermediate message store of
> messages generated by our application so that we get storage of
> messages and sendmail then just forwards that to the final mail server
> to be delivered outside. So it is like: Application -> Submits message
> to sendmail -> Sendmail submits to mail server
>
> The rate of message generated by our application is around 200
> messages (emails) per second. What we are witnessing is that sendmail
> is not able to handle this load and is resulting in messages getting
> queued in sendmail and also due to the response time for each message
> from sendmail to our application the overall message rate of the
> application is getting affected.
>
> Our application logic is like: Submit the message to mail server (in
> this case sendmail), only when submission is successful (so that we
> are sure that email will not be lost and is being held by sendmail)
> move ahead to do rest of the job. Hence if the message takes longer to
> get submitted to sendmail the entire cycle of our application gets
> affected.
>
> Now there are can be two reasons for this:
> 1) What is the maximum message rate (both submission to sendmail and
> sending out from sendmail) which sendmail can handle
> 2) We have not configured sendmail optimally
>
> Before I see the second one, I wanted to verify that what is the
> official performance figure (per second) of sendmail and the platform
> in which it was done. I could not find this answer so far. Would be
> great if someone can point me to the right direction for point-2 as
> well.
>
> The details of the hardware and software are :
> -> Sendmail version: 8.13.1-2
> -> OS: Red Hat Enterprise Linux AS release 4 (Nahant Update 2)
> -> CPU: Dual-Core AMD Opteron(tm) Processor (64 bit). Number of CPU's
> - 2
> -> RAM: 8 GB
>
> Our application and sendmail runs on the same box.
>
> Appreciate the .

Your options:
a) use multiple parallel SMTP session to submit messages [ application side universal solution ]
b) use one SMTP session to "fine tuned" sendmail process e.g. to "sendmail -odd -bs" on custom port started by inetd daemon
-bs - accept SMTP session over stdin/stdout
-odd - queue accepted messages, do no DNS lookups (main delay)

I would recommend option "b" with some extra fine tuning of sendmail daemon configuration to assure that sendmail will start delivery attempts no later than a few minutes after messages are queued.

--
[pl>en Andrew] Andrzej Adam Filip : [email protected] : [email protected]
Breakpoint 1, main (argc=1, argv=0xbffffc40) at main.c:29
29 printf ("Welcome to GNU Hell!\n");
-- "GNU Libtool documentation"

David F. Skoll

ashish wrote:

> 1) What is the maximum message rate (both submission to sendmail and
> sending out from sendmail) which sendmail can handle

This is a very complex question. I strongly recommend reading
"Sendmail Performance Tuning" by Nick Christenson.

http://books.google.ca/books?hl=en&i...result#PPR7,M1

He has a chapter on configuring Sendmail for sending large volumes of e-mail.

Regards,

David.

Rob Kouwenberg

ashish <[email protected]> wrote:

> to be delivered outside. So it is like: Application -> Submits message
> to sendmail -> Sendmail submits to mail server

Why not use your local 'mail' program ? mailx on solaris and mail on
linux*.

Next have several mail sweepers scan and send out x amount of messages.

200 messages a second equals 17 million emails per day. Are you usa.net
or hotmail.com ? Heck if you send that amount of messages you can afford
a consultant (hint hint).

Regards Rob
koneco.nl

QueueLA and RefuseLA

By default, sendmail will reject MTA/MSA connections when the system load is 12 * number of processors, and operate in queue-only delivery mode when the system load is 8 * number of processors.

$ egrep "QueueLA|RefuseLA" /etc/mail/sendmail.cf
#O QueueLA=8
#O RefuseLA=12

From cf/README:

confQUEUE_LA QueueLA [varies] Load average at which
queue-only function kicks in.
Default values is (8 * numproc)
where numproc is the number of
processors online (if that can be
determined).
confREFUSE_LA RefuseLA [varies] Load average at which
incoming SMTP connections are
refused. Default values is (12 *
numproc) where numproc is the
number of processors online (if
that can be determined).

For dedicated mail servers, the value of RefuseLA should be greater than the value of QueueLA. Mail should not be accepted faster than it can be processed, as mentioned in sendmail/TUNING:

It is important not to accept mail
faster than it can be delivered otherwise the system will be
overwhelmed. Hence RefuseLA should be lower than QueueLA, the number
of daemon children should probably be lower than the number of queue
runnners (MaxChildren vs. MaxQueueChildren).

Also, read this comp.mail.sendmail post.

Recommended Links

Sendmail perfomance - Application Development Forum

sendmail.cf
O MaxDaemonChildren=5000
O ConnectionRateThrottle=150

#O Timeout.ident=0

define(`confTO_IDENT', `0s')

make sure your Greetpause is not set to high, a value of 5000 is ample

What is the response time in off peak hours?

[ Are you *SURE* the delays are (mostly) load related? ]

URL(s):
http://www.sendmail.org/documentation/books
[...]
sendmail Performance Tuning
Authored by Nick Christenson
http://www.jetcafe.org/~npc/book/sendmail/
Date: 2003 (September, 2002) Pages: 256

Tuning Configuration Parameters (Mail Administration Guide) - Sun Microsystems

Linux Mail-Queue mini-HOWTO

Sendmail Performance Tuning for Large Systems

MTA Performance Tuning Tips for Sendmail - Documentation - Confluence

Nick Christenson's Sendmail Performance Tuning Reviews

• Author: Nick Christenson
• Pages: 256
• Publisher: Addison Wesley
• Rating: 9
• Reviewer: Andy Murren
• ISBN: 0321115708
• Summary: Compact but well-constructed book; shows you ways to cost-effectively improve your sendmail setup, and much of the rest of your system along the way.

• His web site [gangofone.com]
• His email paper [gangofone.com]
• His Usenet paper [gangofone.com]

I agree with the opinion of user xchino on Slashdot:

I didn't notice anything particularly useful for experienced sendmail admins, but it did look like a good head start for someone with little or no sendmail experience. There are several tuning guides online, and all material in this book is pretty much googleable.

Slashdot Sendmail Performance Tuning

by xchino (591175) on Wednesday January 08 2003, @12:12PM (#5040646)

I've thumbed through this book, and I didn't notice anything particularly useful for experienced sendmail admins, but it did look like a good head start for someone with little or no sendmail experience. There are several tuning guides online, and all material in this book is pretty much googleable.

And on a side rant to those who bitch about sendmail, If you don't like it.. don't use it. All of the gripes I've seen about it seem to derive from the posters ignorance on how to set it up. Granted, .cf is hellspawned, but using the macro file makes changes quick and easy.

Sendmail has a history of being insecure simply because it has a history. It's the grandaddy of all other MTA's and it can still bend them over it's knee and spank them.

Andy Murren review of Nick Christenson's Sendmail Performance Tuning

The first thing that I noticed about this book was how relatively thin it is at only 223 pages. This is compared to the O'Reilly Sendmail book by Bryan Costales and Eric Allman (a.k.a. the 'Bat Book') which weights in at 1232 pages. In this case, thin is good.

The second thing that I noticed about the book is the quote from Eric Allman emblazoned across the top that proclaims 'This book is great.' That's rather ringing endorsement from the original creator of sendmail. But great what? A great technical reference? Companion to the bat book? Kindling? Now here is my sound bite about this book: 'One damned fine technical book.'" Read on for the rest of Andy's review.

This book was very easy to read and kept moving along. While I did not find it as much of a page turner as Douglas Adams' Hitchhikers Guide, it did keep me interested. One word of warning though: do not read this when you are tired. You will miss some really good information and have to re-read parts.

I was surprised and very pleased with how much I learned about so many things about Unix, networking and hardware while learning more about sendmail. There is a wide range of information presented that readers at almost any skill level would be able to use. The writing style and use of language was easy to read. The wealth of information packed into the pages of this book I found immediately usable on my Linux boxes and for my job.

I think the only drawback to the book was that there was not more specific sendmail information. Due to the nature of the topic, a lot of the book is devoted to how Unix systems work, more than specific sendmail configuration tasks. Time and detail is spent on other more important considerations such as logging, disk performance, test planning and file systems.

Chapter 1 gets us started with an overview of tuning in general. One of the more important themes of the book is established up front: It is that a cost/benefits analysis should be done for each step.

Sections 1.5, "Tuning Isn't Always Necessary" and 1.6, "Not So Fast ..." establish baseline considerations for making a decision on what, if any, tuning efforts should be made. Throughout the book Christenson reminds the reader to decide on a path that is most cost effective for his organization. Is it the most cost-effective use of the company money and time to have the IT staff hand-craft solutions, or is it better to throw some hardware at the problem?

Let's face it -- if you are running sendmail for a small company and only move a few hundred emails a week, how much performance tuning do you need? If, however, you are running an ISP, a mailing list server or a medium- (or even a large-) sized company mail server, then you need to tune your mail server. This is the book for you. The information in this book, while oriented for sendmail, is actually applicable for tuning any Unix based Mail Transport Agent (MTA) server.

Each solution is an individual matter, that is wholly dependent on several factors. Some of these factors are: volume of email, what the main use of the email system is, how the end users interact with the system, what hardware is being used, how much bandwidth you have and much time and money you have to throw at the problem. Of course, what management considers important is the overriding factor in all decisions.

Chapter 2 is a ten-page introductory overview to sendmail, covering versions, obtaining the (Open Source) code and building sendmail. One of the important things covered is the queue and message spool layouts and permissions. This is helpful for making sense of things later in the book.

A few very important pages are spent on creating the .cf file and why you should use M4, the macro language, for managing the configuration files. Having hand-crafted a .cf file myself several years ago then having to deal with maintaining it, I can vouch for the wisdom of using M4.

The maintainers of sendmail update the M4 macros for new features and changes. The 200 - 300 lines of M4 macro files are converted into a 1500to 2000 line configuration file. While it may be easier to figure out the configuration file to make changes, those changes may not be valid from one version of sendmail to the next.

Christenson admits that he does not always use M4 when in a rush or to test some things. What he does is copy the working configuration file to sendmail.cf.REAL before making changes and updating the .mc file afterward.

The next chapter, Chapter 3 'Tuning Email Relaying' starts with an overview of the email relaying sequence. Most of the discussion in this chapter is not sendmail specific. The importance of data synchronization is emphasized here. Section 6.1 of RFC 2821 is quoted, where it states the email server 'MUST NOT lose the message'. Once that is stated and understood all of the requirements that are discussed in the rest of the chapter are clear.

The next 17 pages are spent discussing how file systems, networking and effective use of file space support, and can detract from, meeting the edict of the RFC. For me, these sections are some of the most interesting, filled with information presented in a concise, readable and detailed manner. I learned a lot about what impact some very basic decisions have on email performance. I also learned how much better I could have made the email servers I have worked on. I will be turning to this chapter the next time I am putting together a box, be it a web server, mail server, file server or even a workstation or laptop. This is a great chapter that can help with any system configuration.

Email reception is covered in chapter 4. Different strategies for verifying recipient, tuning POP and IMAP are covered. Effective use of Local Delivery Agents (LDAs), including procmail, is covered here also.

Additionally, an excellent discussion of storage systems, including disks and solid-state disks, is in this chapter. The sections covering RAID levels, benchmarking and use are well written and informative. Available options on drives, ATA vs. SCSI, Solid State, are just as well done. Section 4.4.2, 'Stupid Disk Drive Tricks,' has some nifty information about how to set up disk drives for even better performance.

Sending email is the next chapter. Here we see some more sendmail-specific information. Tuning of mailing lists and mass mailing is part of the discussion here. One of the more important sections is 'Draining Queues.' How backups are caused and what to do to recover are discussed in this section. This has good information that can be used with any MTA.

One of the most important chapters for me is Chapter 6 'Configuration, Security and Architecture.' Sendmail specific configuration and tuning options are discussed. Section 6.1 covers configuration and is in many ways the heart of the book. This is where sendmail directives that can directly impact performance are covered. How a system's architecture (and DNS) is laid out can have a significant impact on performance.

The section on security is good, but brief. Most of the discussion is on privacy and stopping spam. The use of Transport Layer Security (TLS) is covered in less than a page. With a smattering of security in other parts of the book, this is the sum total of the security discussion of the book. Considering some of the problems with sendmail in the past, I would have liked some more information on this topic.

The next two chapters, 'Finding and Removing Bottlenecks' and 'Load Generation and Testing,' are good, solid, well-written sections that are applicable to most any email system. We are given some effective ways of making systems run better, and how to prove that the system actually does work better. This is where we justify to the boss that the work we have done is really cost effective.

Chapter 9, 'Conclusion' is basically a very brief wrap up and a list of books Christenson thinks we should read.

My own conclusion is that I have learned and relearned a lot of things about Unix and email. Even if you do not use sendmail, I recommend this book without reservation. It is an excellent reference on general system performance tuning, with information on making your sendmail installation run better.

Table of contents

1. Introduction
   • 1.1 Performance Tuning Examples
   • 1.2 sendmail Versions Covered
   • 1.3 Definitions
   • 1.4 Email Server Tasks
   • 1.5 Tuning Isn't Always Necessary
   • 1.6 Not So Fast...
   • 1.7 Email System Profiling
   • 1.8 General Tuning Ideas
   • 1.9 Summary
2. Sendmail Introduction
   • 2.1 Obtaining Sendmail
   • 2.2 Building Sendmail
   • 2.3 Creating a .cf file
   • 2.4 Why Use M4?
   • 2.5 System Setup
   • 2.6 Summary
3. Tuning Email Relaying
   • 3.1 What Happens During Relaying
   • 3.2 Synchronization
   • 3.3 File Systems
   • 3.4 File Space
   • 3.5 Networking
   • 3.6 Summary
4. Tuning Email Reception
   • 4.1 What Happens During Email Reception
   • 4.2 Recipient Verification
   • 4.3 Storage Systems
   • 4.4 Disks
   • 4.5 Solid State Disks
   • 4.6 POP Tuning Specifics
   • 4.7 Message Storage Hashing
   • 4.8 IMAP Tuning Specifics
   • 4.9 Summary
5. Tuning Email Sending
   • 5.1 Mailing Lists
   • 5.2 Command-Line Message Generation
   • 5.3 Draining Queues
   • 5.4 Another Mailing List Strategy
   • 5.5 SMTP PIPELINING
   • 5.6 More Notes on Mass Mailing
   • 5.7 Summary
6. Configuration, Security and Architecture
   • 6.1 Configuration
   • 6.2 Security and Performance
   • 6.3 Other General Strategies
   • 6.4 Summary
7. Finding and Removing Bottlenecks
   • 7.1 Kernel Parameters Run Amok
   • 7.2 The Quick Fix
   • 7.3 Tools
   • 7.4 syslog
   • 7.5 Removing Bottlenecks
   • 7.6 Summary
8. Load Generation and Testing
   • 8.1 Test System Setup
   • 8.2 Testing Tools
   • 8.3 Load Testing Pitfalls
   • 8.4 Summary
9. Conclusion

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019