|Contents||Bulletin||Scripting in shell and Perl||Network troubleshooting||History||Humor|
|Recommended Links||eMail Security||Sending email attachments from command line||spam||Phishing||Postfix Connection Refused Problem|
|Mutt||mailx||Etiquette||MTA Log Analysers||Forwarding||Pipes in ~/.forward File|
|MTA||Sendmail||Sendmail performance tuning||Sendmail Log Formats||Email Overload||Invoking procmail from sendmail||Sendmail Disgnostic Interpretation||Sendmail file permissions|
|Postfix||Postfix Connection Refused Problem||smtpd_recipient_restrictions||Postfix Troubleshooting||Decoding Mime Attachments||Mail aliases||MUA|
|Perl Mail Processing Scripts||Perl mail daemons||MX records checking||Mail Relay Testing||uucp|
|The Unix Hater’s Handbook||DNS||Listservers||Horror Stories||History||Humor||Etc|
SMTP protocol is one thing, but its interpretation of particular mail program is neither. This interesting fact is aptly demonstrated by early history of Sendmail. Fragment below is extracted from Chapter 4 of The Unix Hater’s Handbook
Date: Wed, 15 May 1991 14:08-0400 From: Christopher Stacy
To: UNIX-HATERS Subject: harder!faster!deeper!unix
Remember when things like netmail used to work? With UNIX, people really don’t expect things to work anymore. I mean, things sorta work, most of the time, and that’s good enough, isn’t it? What’s wrong with a little unreliability with mail? So what if you can’t reply to messages? So what if they get dropped on the floor? The other day, I tried talking to a postmaster at a site running sendmail. You see, whenever I sent mail to people at his site, the headers of the replies I got back from his site came out mangled, and I couldn’t reply to their replies. It looked like maybe the problem was at his end—did he concur? This is what he sent back to me:Date: Mon, 13 May 1991 21:28 EDT From: firstname.lastname@example.org (Stephen J. Silver)1 To: mit-eddie!STONYBROOK. SCRC.Symbolics.COM!CStacy@EDDIE.MIT.EDU Subject: Re: mangled headers
No doubt about it. Our system mailer did it. If you got it, fine. If not, how did you know? If you got it, what is wrong? Just does not look nice? I am not a sendmail guru and do not have one. Mail sorta works, most of the time, and given the time I have, that is great. Good Luck.
Writing a mail system that reliably follows protocol is just not all that hard. I don’t understand why, in 20 years, nobody in the Unix world has been able to get it right once.
Date: Tue, 12 Oct 93 10:31:48 -0400 From: email@example.com To: UNIX-HATERS Subject: sendmail made simple
I was at a talk that had something to do with Unix. Fortunately, I’ve succeeded in repressing all but the speaker’s opening remark:
I’m rather surprised that the author of sendmail is still walking around alive.
The thing that gets me is that one of the arguments that landed Robert Morris, author of “the Internet Worm” in jail was all the sysadmins’ time his prank cost. Yet the author of sendmail is still walking around free without even a U (for Unixery) branded on his forehead.
Sendmail is the standard Unix mailer, and it is likely to remain the standard Unix mailer for many, many years. Although other mailers (such as MMDF and smail) have been written, none of them simultaneously enjoy sendmail’s popularity or widespread animosity.
endmail was written by Eric Allman at the University of Berkeley in 1983 and was included in the Berkeley 4.2 Unix distribution as BSD’s “internetwork mail router.” The program was developed as a single “crossbar” for interconnecting disparate mail networks. In its first incarnation, sendmail interconnected UUCP, BerkNet and ARPANET (the precursor to Internet) networks. Despite its problems, sendmail was better than the Unix mail program that it replaced: delivermail.
In his January 1983 USENIX paper, Allman defined eight goals for sendmail:
1. Sendmail had to be compatible with existing mail programs.
2. Sendmail had to be reliable, never losing a mail message.
3. Existing software had to do the actual message delivery if at all possible.
4. Sendmail had to work in both simple and extremely complex environments.
5. Sendmail’s configuration could not be compiled into the program, but had to be read at startup.
6. Sendmail had to let various groups maintain their own mailing lists and let individuals specify their own mail forwarding, without having individuals or groups modify the system alias file.
7. Each user had to be able to specify that a program should be executed to process incoming mail (so that users could run “vacation” programs).
8. Network traffic had to be minimized by batching addresses to a single host when at all possible.
(An unstated goal in Allman’s 1983 paper was that sendmail also had to implement the ARPANET’s nascent SMTP (Simple Mail Transport Protocol) in order to satisfy the generals who were funding Unix development at Berkeley.)
Sendmail was built while the Internet mail handling systems were in flux. As a result, it had to be programmable so that it could handle any possible changes in the standards. Delve into the mysteries of sendmail’s unreadable sendmail.cf files and you’ll discover ways of rewiring sendmail’s insides so that@#$@$^%<<<@#) at @$%#^!
is a valid e-mail address.
That was great in 1985. In 1994, the Internet mail standards have been decided upon and such flexibility is no longer needed. Nevertheless, all of sendmail’s rope is still there, ready to make a hangman’s knot, should anyone have a sudden urge.
Sendmail is one of those clever programs that performs a variety of different functions depending on what name you use to invoke it. Sometimes it’s the good ol’ sendmail; other times it is the mail queue viewing program or the aliases database-builder. “Sendmail Revisited” admits that bundling so much functionality into a single program was probably a mistake: certainly the SMTP server, mail queue handler, and alias database management system should have been handled by different programs (no doubt carrying through on the Unix “tools” philosophy). Instead we have sendmail, which continues to grow beyond all expectations.Date: Sun, 6 Feb 94 14:17:32 GMT From: Robert Seastrom
To: UNIX-HATERS Subject: intelligent? friendly? no, I don’t think so...Much to my chagrin, I’ve recently received requests from folks at my site to make our mailer non-RFC821-compliant by making it pass 8- bit mail. Apparently, the increasingly popular ISO/LATIN1 encoding format is 8-bit (why? last I checked, the Roman alphabet only had 26 characters) and messages encoded in it get hopelessly munged when the 8th bit gets stripped off. I’m not arguing that stripping the high bit is a good thing, just that it’s the standard, and that we have standards for a reason, and that the ISO people shouldn’t have had their heads so firmly implanted in their asses. But what do you expect from the people who brought us OSI?
So I decided to upgrade to the latest version of Berzerkly Sendmail (8.6.5) which reputedly does a very good job of not adhering to the standard in question. It comes with an FAQ document. Isn’t it nice that we have FAQs, so that increasingly incompetent Weenix Unies can install and misconfigure increasingly complex software, and sometimes even diagnose problems that once upon a time would have required one to
read the source code! One of the books it recommends for people to read if they want to become Real Sendmail Wizards is:Have you seen this book? It has more pages than War and Peace. More pages than my TOPS-10 system calls manual. It will stop a pellet fired from a .177 air pistol at point-blank range before it penetrates even halfway into the book (.22 testing next weekend). It’s probably necessary to go into this level of detail for some of the knuckle-draggers who are out there running machines on the Internet these days, which is even more scary. But I digress.
Costales, Allman, and Rickert, Sendmail. O’Reilly & Associates.
Then, below, in the actual “Questions” section, I see:
Q: Why does the Costales book have a bat on the cover?
A: Do you want the real answer or the fun answer? The real answer is that Bryan Costales was presented with a choice of three pictures, and he picked the bat because it appealed to him the most. The fun answer is that, although sendmail has a reputation for being scary, like a bat, it is really a rather friendly and intelligent beast.
Friendly and intelligent? Feh. I can come up with tons of better answers to that one. Especially because it’s so patently wrong. To wit:
- The common North American brown bat’s diet is composed principally of bugs. Sendmail is a software package which is composed principally of bugs.
- Sendmail and bats both suck.
- Sendmail maintainers and bats both tend to be nocturnal creatures, making “eep eep” noises which are incomprehensible to the average person.
- Have you ever watched a bat fly? Have you ever watched Sendmail process a queue full of undelivered mail? QED.
- Sendmail and bats both die quickly when kept in captivity.
- Bat guano is a good source of potassium nitrate, a principal ingredient in things that blow up in your face. Like Sendmail.
- Both bats and sendmail are held in low esteem by the general public.
- Bats require magical rituals involving crosses and garlic to get them to do what you want. Sendmail likewise requires mystical incantations such as:R<$+>$*$=Y$~A$* $:<$1>$2$3?$4$5 Mark user portion. R<$+>$*!$+,$*?$+ <$1>$2!$3!$4?$5 is inferior to @ R<$+>$+,$*?$+ <$1>$2:$3?$4 Change src rte to % path R<$+>:$+ <$1>,$2 Change % to @ for immed. domain R<$=X$-.UUCP>!?$+ $@<$1$2.UUCP>!$3 Return UUCP R<$=X$->!?$+ $@<$1$2>!$3 Return unqualified R<$+>$+?$+ <$1>$2$3 Remove '?' R<$+.$+>$=Y$+ $@<$1.$2>,$4 Change do user@domain
- Farmers consider bats their friends because of the insects they eat. Farmers consider Sendmail their friend because it gets more college educated people interested in subsistence farming as a career.
I could go on and on, but I think you get the idea. Stay tuned for the .22 penetration test results!
A mail system must perform the following relatively simple tasks each time it receives a message in order to deliver that message to the intended recipient:
Sendmail manages to blow every step of the process.
This is easy for humans. For example, take the following message:
Date: Wed, 16 Oct 91 17:33:07 -0400 From: Thomas Lawrence
To: firstname.lastname@example.org Subject: Sidewalk obstruction
The logs obstructing the sidewalk in front of the building will be used in the replacement of a collapsing manhole. They will be there for the next two to three weeks.We have no trouble figuring out that this message was sent from “Thomas Lawrence,” is meant for the “msgs” mailing list which is based at the MIT Media Lab, and that the body of the message is about some logs on the sidewalk outside the building. It’s not so easy for Unix, which manages to produce:
Date: Wed, 16 Oct 91 17:29:01 -0400 From: Thomas Lawrence
Subject: Sidewalk obstruction To: email@example.com Cc: The@media-lab.media.mit.edu, logs.obstructing.the.sidewalk.in.front.of.the.building. firstname.lastname@example.org
On occasion, sendmail has been known to parse the entire body of a message (sometimes backwards!) as a list of addresses:
Subject: Returned Mail: User Unknown 69 Date: Thu, 13 Sep 90 08:48:06 -0700 From: MAILER-DAEMON@Neon.Stanford.EDU Comment: Redistributed from CS.Stanford.EDU Apparently-To: <Juan ECHAGUE e-mail:email@example.com tel:76 57 46 68 (33)> Apparently-To: <PS:I’ll summarize if interest,etc.@Neon.Stanford. EDU> Apparently-To: <Juan@Neon.Stanford.EDU> Apparently-To: <Thanks in advance@Neon.Stanford.EDU> Apparently-To: <for temporal logics.Comments and references are welcomed.@ Neon.Stanford.EDU> Apparently-To: <I’m interested in gentzen and natural deduction style axiomatizations@Neon.Stanford.EDU>
When Joe Smith on machine A wants to send a message to Sue Whitemore on machine B, he might generate a header such as Sue@bar!B%baz!foo.uucp. It’s up to sendmail to parse this nonsense and try to send the message somewhere logical.
At times, it’s hard not to have pity on sendmail, since sendmail itself is the victim of multiple Unix “standards.” Of course, sendmail is partially responsible for promulgating the lossage.
If sendmail weren’t so willing to turn tricks on the sender’s behalf, maybe users wouldn’t have been so flagrant in the addresses they compose. Maybe they would demand that their system administrators configure their mailers properly. Maybe netmail would work reliably once again, no matter where you were sending the mail to or receiving it from.
Just the same, sometimes sendmail goes too far:
Date: Wed, 8 Jul 1992 11:01-0400 From: Judy Anderson
To: UNIX-HATERS Subject: Mailer error of the day.
I had fun with my own mailer-error-of-the-day recently. Seems I got mail from someone in the “.at” domain. So what did the Unix mailer do with this address when I tried to reply? Why it turned “at” into “@” and then complained about no such host! Or was it invalid address format? I forget, there are so many different ways to lose.
…Or perhaps sendmail just thinks that Judy shouldn’t be sending e-mail to Austria.
Sending mail to QUICHE-EATERS then results in mail being dropped into three mailboxes. Aliases files are a natural idea and have been around since the first electronic message was sent. Unfortunately, sendmail is a little unclear on the concept, and its alias file format is a study in misdesign.
We’d like to say something insulting, like “it’s from the dark ages of computing,” but we can’t: alias files worked in the dark ages of computing. It is sendmail’s modern, up-to-date alias files that are riddled with problems. Figure 1 shows an excerpt from the sendmail aliases file of someone who maintained systems then and is forced to use sendmail now.
Sendmail not only has a hopeless file format for its alias database: many versions commonly in use refuse to deliver mail or perform name resolution, while it is in the processing of compiling its alias file into binary format.
Subject: Returned Mail: User Unknown Date: Thu, 11 Apr 91 13:00:22 EDT From: Steve Strassmann
To: UNIX-HATERS Subject: pain, death, and disfigurement ############################################################### # # READ THESE NOTES BEFORE MAKING CHANGES TO THIS FILE: thanks! # # Since aliases are run over the yellow pages, you must issue the # following command after modifying the file: # # /usr/local/newaliases # (Alternately, type m-x compile in Emacs after editing this file.) # # [Note this command won't -necessarily- tell one whether the # mailinglists file is syntactically legal -- it might just silently # trash the mail system on all of the suns. # WELCOME TO THE WORLD OF THE FUTURE.] # # Special note: Make sure all final mailing addresses have a host # name appended to them. If they don't, sendmail will attach the # Yellow Pages domain name on as the implied host name, which is # incorrect. Thus, if you receive your mail on wheaties, and your # username is johnq, use "johnq@wh" as your address. It # will cause major lossage to just use "johnq". One other point to # keep in mind is that any hosts outside of the "ai.mit.edu" # domain must have fully qualified host names. Thus, "xx" is not a # legal host name. Instead, you must use "xx.lcs.mit.edu". # WELCOME TO THE WORLD OF THE FUTURE # # # Special note about large lists: # It seems from empirical observation that any list defined IN THIS # FILE with more than fifty (50) recipients will cause newaliases to # say "entry too large" when it's run. It doesn't tell you -which- # list is too big, unfortunately, but if you've only been editing # one, you have some clue. Adding the fifty-first recipient to the # list will cause this error. The workaround is to use:include # files as described elsewhere, which seem to have much larger or # infinite numbers of recipients allowed. [The actual problem is # that this file is stored in dbm(3) format for use by sendmail. # This format limits the length of each alias to the internal block # size (1K).] # WELCOME TO THE WORLD OF THE FUTURE # # Special note about comments: # Unlike OZ's MMAILR, you -CANNOT- stick a comment at the end of a # line by simply prefacing it with a "#". The mailer (or newaliases) # will think that you mean an address which just so happens to have # a "#" in it, rather than interpreting it as a comment. This means, # essentially, that you cannot stick comments on the same line as # any code. This also probably means that you cannot stick a comment # in the middle of a list definition (even on a line by itself) and # expect the rest of the list to be properly processed. # WELCOME TO THE WORLD OF THE FUTURE # ###################################################################
FIGURE 1. Excerpts From A sendmail alias file
Date: Thu, 11 Apr 91 13:00:22 EDT From: Steve Strassmann <firstname.lastname@example.org> To: UNIX-HATERS Subject: pain, death, and disfigurement
Sometimes, like a rare fungus, Unix must be appreciated at just the right moment. For example, you can send mail to a mailing list. But not if someone else just happens to be running newaliases at the moment.
You see, newaliases processes /usr/lib/aliases like so much horse meat; bone, skin, and all. It will merrily ignore typos, choke on perilous whitespace, and do whatever it wants with comments except treat them as comments, and report practically no errors or warnings.
How could it? That would require it to actually comprehend what it reads. I guess it would be too hard for the mailer to actually wait for this sausage to be completed before using it, but evidently Unix cannot afford to keep the old, usable version around while the new one is being created. You see, that would require, uh, actually, it would be trivial. Never mind, Unix just isn’t up to the task. As the alias list is pronounced dead on arrival, what should sendmail do? Obviously, treat it as gospel.
If you send mail to an alias like ZIPPER-LOVERS which is at the end of the file, while it’s still gurgitating on ACME-CATALOG-REQUEST, sendmail will happily tell you your addressee is unknown. And then, when it’s done, the new mail database has some new bugs, and the old version—the last known version that actually worked—is simply lost forever. And the person who made the changes is not warned of any bugs. And the person who sent mail to a valid address gets it bounced back. But only sometimes.
Practically everybody who has been unfortunate enough to have their messages piped through sendmail had a special message sent to the wrong recipient. Usually these messages are very personal, and somehow uncanningly sent to the precise person for whom receipt will cause the maximum possible damage.
On other occasions, sendmail simply gets confused and can’t figure out where to deliver mail. Other times, sendmail just silently throws the mail away. Few people can complain about this particular sendmail mannerism, because few people know that the mail has been lost. Because Unix lies in so many ways, and because sendmail is so fragile, it is virtually impossible to debug this system when it silently deletes mail:
Subject: Returned Mail: User Unknown 73 Date: Tue, 30 Apr 91 02:11:58 EDT From: Steve Strassmann
To: UNIX-HATERS Subject: Unix and parsing
You know, some of you might be saying, hell, why does this straz guy send so much mail to UNIX-HATERS? How does he come up with new stuff every day, sometimes twice a day? Why is he so filled with bile? To all these questions there’s a simple answer: I use Unix. Like today, for example. A poor, innocent user asked me why she suddenly stopped getting e-mail in the last 48 hours. Unlike most users, with accounts on the main Media Lab machine, she gets and reads her mail on my workstation.
Sure enough, when I sent her a message, it disappeared. No barf, no error, just gone. I round up the usual suspects, but after an hour between the man pages for sendmail and other lossage, I just give up. Hours later, solving another unrelated Unix problem, I try “ps -ef” to look at some processes. But mine aren’t owned by “straz,” the owner is this guy named “000000058.” Time to look in /etc/ passwd.
Right there, on line 3 of the password file, is this new user, followed by (horrors!) a blank line. I said it. A blank line. Followed by all the other entries, in their proper order, plain to you or me, but not to Unix. Oh no, whoever was fetching my name on behalf of ps can’t read past a blank line, so it decided “straz” simply wasn’t there. You see Unix knows parsing like Dan Quayle knows quantum mechanics. But that means—you guessed it. Mailer looks in /etc/passwd before queuing up the mail. Her name was in /etc/passwd, all right, so there’s no need to bounce incoming mail with “unknown user” barf. But when it actually came down to putting the message someplace on the computer like /usr/mail/, it couldn’t read past the blank line to identify the owner, never mind that it already knew the owner because it accepted the damn mail in the first place. So what did it do? Handle it the Unix way: Throw the message away without telling anyone and hope it wasn’t important!
So how did the extra blank line get there in the first place? I’m so glad you asked. This new user, who preceded the blank line, was added by a well-meaning colleague using ed 3 from a terminal with some non-standard environment variable set so he couldn’t use Emacs or vi or any other screen editor so he couldn’t see there was an extra blank line that Unix would rather choke dead on than skip over. That’s why.
The problem with sendmail is that the sendmail configuration file is a rule-based expert system, but the world of e-mail is not logical, and sendmail configuration editors are not experts.Beyond blowing established mail delivery protocols, Unix has invented newer, more up-to-date methods for ensuring that mail doesn’t get to its intended destination, such as mail forwarding. Suppose that you have changed your home residence and want your mail forwarded automatically by the post office. The rational method is the method used now: you send a message to your local postmaster, who maintains a centralized database. When the postmaster receives mail for you, he slaps the new address on it and sends it on its way to its new home. There’s another, less robust method for rerouting mail: put a message near your mailbox indicating your new address. When your mailman sees the message, he doesn’t put your mail in your mailbox. Instead, he slaps the new address on it and takes it back to the post office. Every time.
—David Waitzman, BBN
The flaws in this approach are obvious. For one, there’s lots of extra overhead. But, more importantly, your mailman may not always see the message— maybe it’s raining, maybe someone’s trash cans are in front of it, maybe he’s in a rush. When this happens, he misdelivers your mail into your old mailbox, and you never see it again unless you drive back to check or a neighbor checks for you. Now, we’re not inventing this stupider method: Unix did. They call that note near your mailbox a .forward file. And it frequently happens, especially in these distributed days in which we live, that the mailer misses the forwarding note and dumps your mail where you don’t want it.
“Ed is the standard Unix editor.” —Unix documentation (circa 1994).
75 Date: Thu, 6 Oct 88 22:50:53 EDT From: Alan Bawden To: SUN-BUGS Cc: UNIX-HATERS Subject: I have mail?
Whenever log into a Sun, I am told that I have mail. I don’t want to receive mail on a Unix, I want my mail to be forwarded to “Alan@AI.” Now as near as I can tell, I don’t have a mailbox in my home directory on the Suns, but perhaps Unix keeps mailboxes elsewhere? If I send a test message to “alan@wheaties” it correctly finds its way to AI, just as the .forward file in my home directory says to do. I also have the mail-address field in my inquir entry set to “Alan@AI.” Nevertheless, whenever I log into a Sun, it tells me that I have mail. (I don’t have a personal entry in the aliases file, do I need one of those in addition to the .forward file and the inquir entry?)The next day, Alan answered his own query:
So could someone either:
A. Tell me that I should just ignore the “You have mail” message, because in fact I don't have any mail accumulating in some dark corner of the file system, or
B. Find that mail and forward it to me, and fix it so that this never happens again.
Date: Fri, 7 Oct 88 14:44 EDT From: Alan Bawden
To: UNIX-HATERS Subject: I have mail? Date: Thu, 6 Oct 88 22:50:53 EDT From: Alan Bawden
… (I don’t have a personal entry in the aliases file, do I need one of those in addition to the .forward file and the inquir entry?) …
Apparently the answer to this is “yes.” If the file server that contains your home directory is down, the mailer can’t find your .forward file, so mail is delivered into /usr/spool/mail/alan (or whatever). So if you really don’t want to learn how to read mail on a Unix, you have to put a personal entry in the aliases file. I guess the .forward file in your home directory is just a mechanism to make the behavior of the Unix mailer more unpredictable.
I wonder what it does if the file server that contains the aliases file is down?
Every society has rules to prevent chaos and to promote the general welfare. Just as a neighborhood of people sharing a street might be composed of people who came from Europe, Africa, Asia, and South America, a neighborhood of computers sharing a network cable often come from disparate places and speak disparate languages. Just as those people who share the street make up a common language for communication, the computers are supposed to follow a common language, called a protocol, for communication. This strategy generally works until either a jerk moves onto the block or a Unix machine is let onto the network. Neither the jerk nor Unix follows the rules. Both turn over trash cans, play the stereo too loudly, make life miserable for everyone else, and attract wimpy sycophants who bolster their lack of power by associating with the bully.
We wish that we were exaggerating, but we’re not. There are published protocols. You can look them up in the computer equivalent of city hall— the RFCs. Then you can use Unix and verify lossage caused by Unix’s unwillingness to follow protocol.
For example, an antisocial and illegal behavior of sendmail is to send mail to the wrong return address. Let’s say that you send a real letter via the U.S. Postal Service that has your return address on it, but that you mailed it from the mailbox down the street, or you gave it to a friend to mail for you. Let’s suppose further that the recipient marks “Return to sender” on the letter. An intelligent system would return the letter to the return address; an unintelligent system would return the letter to where it was mailed from, such as to the mailbox down the street or to your friend.
That system mimicking a moldy avocado is, of course, Unix, but the real story is a little more complicated because you can ask your mail program to do tasks you could never ask of your mailman. For example, when responding to an electronic letter, you don’t have to mail the return envelope yourself; the computer does it for you. Computers, being the nitpickers with elephantine memories that they are, keep track not only of who a response should be sent to (the return address, called in computer parlance the “Reply-to:” field), but where it was mailed from (kept in the “From:” field). The computer rules clearly state that to respond to an electronic message one uses the “Reply-to” address, not the “From” address. Many versions of Unix flaunt this rule, wrecking havoc on the unsuspecting. Those who religiously believe in Unix think it does the right thing, misassigning blame for its bad behavior to working software, much as Detroit blames Japan when Detroit’s cars can’t compete.
For example, consider this sequence of events when Devon McCullough complained to one of the subscribers of the electronic mailing list called PAGANISM4 that the subscriber had sent a posting to the e-mail address PAGANISM-REQUEST@MC.LCS.MIT.EDU and not to the address PAGANISM@MC.LCS.MIT.EDU:
From: Devon Sean McCullough
This message was sent to PAGANISM-REQUEST, not PAGANISM. Either you or your ‘r’ key screwed up here. Or else the digest is screwed up. Anyway, you could try sending it again.
The clueless weenie sent back the following message to Devon, complaining
that the fault lied not with himself or sendmail, but with the PAGANISM
Date: Sun, 27 Jan 91 11:28:11 PST From:
To: Devon Sean McCullough
From my perspective, the digest is at fault. Berkeley Unix Mail is what I use, and it ignores the "Reply-to:" line, using the "From:" line instead. So the only way for me to get the correct address is to either back-space over the dash and type the @ etc in, or save it somewhere and go thru some contortions to link the edited file to the old echoed address. Why make me go to all that trouble? This is the main reason that I rarely post to the PAGANISM digest at MIT.The interpretation of which is all too easy to understand:
Date: Mon, 28 Jan 91 18:54:58 EST From: Alan Bawden <email@example.com> To: UNIX-HATERS Subject: DepressingNotice the typical Unix weenie reasoning here: “The digestifier produces a header with a proper Reply-To field, in the expectation that your mail reading tool will interpret the header in the documented, standard, RFC822 way. Berkeley Unix Mail, contrary to all standards, and unlike all reasonable mail reading tools, ignores the Reply-To field and incorrectly uses the From field instead.”
“The digestifier is at fault.”
Frankly, I think the entire human race is doomed. We haven’t got a snowball’s chance of doing anything other than choking ourselves to death on our own waste products during the next couple hundred years.
It should be noted that this particular feature of Berkeley Mail has been fixed; Mail now properly follows the “Reply-To:” header if it is present in a mail message. On the other hand, the attitude that the Unix implementation is a more accurate standard than the standard itself continues to this day. It’s pervasive. The Internet Engineering Task Force (IETF) has embarked on an effort to rewrite the Internet’s RFC “standards” so that they comply with the Unix programs that implement them.
>From Unix, with Love
We have laws against the U.S. Postal Service modifying the mail that it delivers. It can scribble things on the envelope, but can’t open it up and change the contents. This seems only civilized. But Unix feels regally endowed to change a message's contents. Yes, of course, it’s against the computer law. Unix disregards the law.
For example, did you notice the little “>” in the text of a previous message? We didn’t put it there, and the sender didn't put it there. Sendmail put it there, as pointed out in the following message:
79 Date: Thu, 9 Jun 1988 22:23 EDT From: firstname.lastname@example.org To: UNIX-HATERS Subject: mailer warts
Did you ever wonder how the Unix mail readers parse mail files? You see these crufty messages from all these losers out in UUCP land, and they always have parts of other messages inserted in them, with bizarre characters before each inserted line. Like this:From Unix WeenieIn your last post you meant to flame me but you clearly don’t know what your talking about when you say
Date: Tue, 13 Feb 22 12:33:08 EDT From: Unix Weenie To: net.soc.singles.sf-lovers.lobotomies.astronomy.laserlovers. unix.wizards.news.group> >> %> $> Received: from magilla.uucp by gorilla.uucp > >> %> $> via uunet with sendmail > >> %> $> …so think very carefully about what you say when you post >From your home machien because when you sent that msg it went to all the people who dont want to read your falming so don’t do it ):-(Now! Why does that “From” on the second line preceding paragraph have an angle bracket before it? I mean, you might think it had something to do with the secret codes that Usenet Unix weenies use when talking to each other, to indicate that they're actually quoting the fifteenth preceding message in some interminable public conversation, but no, you see, that angle bracket was put there by the mailer. The mail reading program parses mail files by looking for lines beginning with “From.” So the mailer has to mutate text lines beginning with “From” so’s not to confuse the mail readers. You can verify this for yourself by sending yourself a mail message containing in the message body a line beginning with “From.”
This is a very important point, so it bears repeating. The reason for “>From” comes from the way that the Unix mail system to distinguishes between multiple e-mail messages in a single mailbox (which, following the Unix design, is just another file). Instead of using a special control sequence, or putting control information into a separate file, or putting a special header at the beginning of the mail file, Unix assumes that any line beginning with the letters F-r-o-m followed by a space (“ ”) marks the beginning of a new mail message.
Using bits that might be contained by e-mail messages to represent information about e-mail messages is called inband communication, and anybody who has ever taken a course on telecommunications knows that it is a bad idea. The reason that inband communication is bad is that the communication messages themselves sometimes contain these characters. For this reason, sendmail searches out lines that begin with “From ” and changes them to “>From.”
Now, you might think this is a harmless little behavior, like someone burping loudly in public. But sometimes those burps get enshrined in public papers whose text was transmitted using sendmail. The recipient believes that the message was already proofread by the sender, so it gets printed verbatim. Different text preparation systems do different things with the “>” character. For example, LaTeX turns it into an upside question mark (¿). If you don't believe us, obtain the paper “Some comments on the assumptioncommitment framework for compositional verification of distributed programs” by Paritosh Pandya, in “Stepwise Refinement of Distributed Systems,” Springer-Verlag, Lecture Notes in Computer Science no. 430, pages 622–640. Look at pages 626, 630, and 636—three paragraphs start with a “From” that is prefixed with a ¿.
Sendmail even mangles mail for which it isn’t the “final delivery agent”— that is, mail destined for some other machine that is just passing through some system with a sendmail mailer. For example, just about everyone at Microsoft uses a DOS or Windows program to send and read mail. Yet internal mail gets goosed with those “>Froms” all over the place. Why? Because on its hop from one DOS box to another, mail passes through a Unix-like box and is scarred for life.
So what happens when you complain to a vendor of electronic mail services (whom you pay good money to) that his machine doesn’t follow protocol— what happens if it is breaking the law? Jerry Leichter complained to his vendor and got this response:
Date: Tue, 24 Mar 92 22:59:55 EDT From: Jerry Leichter <email@example.com> To: UNIX-HATERS Subject: That wonderful >From From: <A customer service representative>5 From: <MAILER-DAEMON@berkeley.edu>
I don’t and others don’t think this is a bug. If you can come up with an RFC that states that we should not be doing this I’m sure we will fix it. Until then this is my last reply. I have brought this to the attention of my supervisors as I stated before. As I said before, it appears it is Unix’s way of handling it. I have sent test messages from machines running the latest software. As my final note, here is a section from rfc976:
I won’t include that wonderful quote, which nowhere justifies a mail forwarding agent modifying the body of a message—it simply says that “From” lines and “>From” lines, wherever they might have come from, are members of the syntactic class From_Lines. Using typical Unix reasoning, since it doesn’t specifically say you can’t do it, and it mentions that such lines exist, it must be legal, right? I recently dug up a July 1982 RFC draft for SMTP. It makes it clear that messages are to be delivered unchanged, with certain documented exceptions. Nothing about >’s. Here we are 10 years later, and not only is it still wrong—at a commercial system that charges for its services—but those who are getting it wrong can’t even SEE that it’s wrong.
I think I need to scream.
NOTE: This message was returned to a UNIX-HATER subscriber by a technical support representative at a major Internet provider. We’ve omitted that company’s name, not in the interest of protecting the guilty, but because there was no reason to single out this particular company: the notion that "sendmail is always right" is endemic among all of the Internet service providers.
You can tell those who live on the middle rings of Unix Hell from those on lower levels. Those in the middle levels know about >From lossage but think that uuencode is the way to avoid problems. Uuencode encodes a file that uses only 7-bit characters, instead of 8-bit characters that Unix mailers or network systems might have difficulty sending. The program uudecode decodes a uuencoded file to produce a copy of the original file. A uuencoded file is supposedly safer to send than plain text; for example,
">From" distortion can’t occur to such a file.Unfortunately, Unix mailers have other ways of screwing users to the wall:
Date: Tue, 4 Aug 92 16:07:47 HKT From: Olin G. Shivers
To: UNIX-HATERS Subject: Need your help.
Anybody who thinks that uuencode protects a mail message is living in a pipe dream. Uuencode doesn’t help. The idiot program uses ASCII spaces in its encoding. Strings of nuls map to strings of blanks. Many Unix mailers thoughtfully strip trailing blanks from lines of mail. This nukes your carefully–encoded data. Well, it’s Unix, what did you expect?
Of course you can grovel over the data, find the lines that aren’t the right length, and re-pad with blanks—that will (almost certainly?) fix it up. What else is your time for anyway, besides cleaning up after the interactions of multiple brain-damaged Unix so-called “utilities?” Just try and find a goddamn spec for uuencoded data sometime. In the man page? Hah. No way. Go read the source—that’s the “spec.” I particularly admire the way uuencode insists on creating a file for you, instead of working as a stdio filter. Instead of piping into tar, which knows about creating files, and file permissions, and directories, and so forth, we build a half-baked equivalent functionality directly into uuencode so it’ll be there whether you want it or not. And I really, really like the way uuencode by default makes files that are world writable.
Maybe it’s Unix fighting back, but this precise bug hit one of the editors of this book after editing in this message in April 1993. Someone mailed him a uuencoded PostScript version of a conference paper, and fully 12 lines had to be handpatched to put back trailing blanks before uudecode reproduced the original file.
550 chiarell... User unknown: Not a typewriter 550
... User unknown: Address already in use 550 firstname.lastname@example.org... User unknown: Not a bicycle 553 abingdon I refuse to talk to myself 554 "| /usr/new/lib/mh/slocal -user $USER"... unknown mailer error 1 554 "| filter -v" ... unknown mailer error 1 554 Too many recipients for no message body
“Not a typewriter” is sendmail’s most legion error message. We figure that the error message “not a bicycle” is probably some system administrator’s attempt at humor. The message “Too many recipients for no message body” is sendmail’s attempt at Big Brotherhood. It thinks it knows better than the proletariat masses, and it won’t send a message with just a subject line.
The conclusion is obvious: you are lucky to get mail at all or to have messages you send get delivered. Unix zealots who think that mail systems are complex and hard to get right are mistaken. Mail used to work, and work highly reliably. Nothing was wrong with mail systems until Unix came along and broke things in the name of “progress.”
Date: Tue, 9 Apr 91 22:34:19 -0700 From: Alan Borning
To: UNIX-HATERS Subject: the vacation program
So I went to a conference the week before last and decided to try being a Unix weenie, and set up a “vacation” message. I should have known better.
The vacation program has a typical Unix interface (involving creating a .forward file with an obscure incantation in it, a .vacation.msg file with a message in it, etc.) There is also some -l initialization option, which I couldn’t get to work, which is supposed to keep the vacation replies down to one per week per sender. I decided to test it by sending myself a message, thinking that surely they would have allowed for this and prevented an infinite sending of vacation messages. A test message, a quick peek at the mail box, bingo, 59 messages already. Well. It must be working.
However, the really irksome thing about this program is the standard vacation message format. From the man page:
From: email@example.com (Eric Allman) Subject: I am on vacation Delivered-By-The-Graces-Of: the Vacation program
Depending on one’s theology and politics, a message might be delivered by the grace of some god or royal personage — but never by the grace of Unix. The very concept is an oxymoron.
For example, before sendmail will accept a message (by returning exit status or sending a response code) it insures that all information needed to deliver that message is forced out to the disk. In this way, sendmail has “accepted responsibility” for delivery of the message (or notification of failure). If the message is lost prior to acceptance, it is the “fault” of the sender; if lost after acceptance, it is the “fault” of the receiving sendmail. This algorithm implies that a window exists where both sender and receiver believe that they are “responsible” for this message. If a failure occurs during this window then two copies of the message will be delivered. This is normally not a catastrophic event, and is far superior to losing a message.
This design choice to deliver two copies of a message rather than none at all might indeed be far superior in most circumstances. Certainly, lost mail is a bad thing. On the other hand, techniques for guaranteeing synchronous, atomic operations, even for processes running on two separate computers, were known and understood in 1983 when sendmail was written.
Date: Thu, 09 May 91 23:26:50 -0700 From: Erik E. Fair (Your Friendly Postmaster)
To: firstname.lastname@example.org, email@example.com, [...] Subject: Case of the Replicated Errors: An Internet Postmaster’s Horror Story
This Is The Network: The Apple Engineering Network.
The Apple Engineering Network has about 100 IP subnets, 224 AppleTalk zones, and over 600 AppleTalk networks. It stretches from Tokyo, Japan, to Paris, France, with half a dozen locations in the U.S., and 40 buildings in the Silicon Valley. It is interconnected with the Internet in three places: two in the Silicon Valley, and one in Boston. It supports almost 10,000 users every day.
When things go wrong with e-mail on this network, it’s my problem.
My name is Fair. I carry a badge.
6Erik Fair graciously gave us permission to reprint this message which appeared on the TCP-IP, UNICODE, and RISKS mailing lists, although he added: “I am not on the UNIX-HATERS mailing list. I have never sent anything there personally. I do not hate Unix; I just hate USL, Sun, HP, and all the other vendors who have made Unix FUBAR.”
[insert theme from Dragnet]
The story you are about to read is true. The names have not been changed so as to finger the guilty.
It was early evening, on a Monday. I was working the swing shift out of Engineering Computer Operations under the command of Richard Herndon. I don’t have a partner.
While I was reading my e-mail that evening, I noticed that the load average on apple.com, our VAX-8650, had climbed way out of its normal range to just over 72.
Upon investigation, I found that thousands of Internet hosts7 were trying to send us an error message. I also found 2,000+ copies of this error message already in our queue.
I immediately shut down the sendmail daemon which was offering SMTP service on our VAX.
I examined the error message, and reconstructed the following sequence of events:
We have a large community of users who use QuickMail, a popular Macintosh based e-mail system from CE Software. In order to make it possible for these users to communicate with other users who have chosen to use other e-mail systems, ECO supports a QuickMail to Internet e-mail gateway. We use RFC822 Internet mail format, and RFC821 SMTP as our common intermediate r-mail standard, and we gateway everything that we can to that standard, to promote interoperability. The gateway that we installed for this purpose is MAIL*LINK SMTP from Starnine Systems. This product is also known as GatorMail-Q from Cayman Systems. It does gateway duty for all of the 3,500 QuickMail users on the Apple Engineering Network. Many of our users subscribe, from QuickMail, to Internet mailing lists which are delivered to them through this gateway. One such 7Erik identifies these machines simply as “Internet hosts,” but you can bet your cookies that most of them were running Unix. Apple Computer’s Mail Disaster of 1991 user, Mark E. Davis, is on the firstname.lastname@example.org mailing list, to discuss some alternatives to ASCII with the other members of that list. Sometime on Monday, he replied to a message that he received from the mailing list. He composed a one paragraph comment on the original message, and hit the “send” button.
Somewhere in the process of that reply, either QuickMail or MAIL*LINK SMTP mangled the “To:” field of the message.
The important part is that the “To:” field contained exactly one “<” character, without a matching “>” character. This minor point caused the massive devastation, because it interacted with a bug in sendmail. Note that this syntax error in the “To:” field has nothing whatsoever to do with the actual recipient list, which is handled separately, and which, in this case, was perfectly correct.
The message made it out of the Apple Engineering Network, and over to Sun Microsystems, where it was exploded out to all the recipients of the email@example.com mailing list.
Sendmail, arguably the standard SMTP daemon and mailer for UNIX, doesn’t like “To:” fields which are constructed as described. What it does about this is the real problem: it sends an error message back to the sender of the message, AND delivers the original message onward to whatever specified destinations are listed in the recipient list.
This is deadly.
The effect was that every sendmail daemon on every host which touched the bad message sent an error message back to us about it. I have often dreaded the possibility that one day, every host on the Internet (all 400,000 of them8) would try to send us a message, all at once.
On Monday, we got a taste of what that must be like. I don’t know how many people are on the firstname.lastname@example.org mailing list, but I’ve heard from Postmasters in Sweden, Japan, Korea, Aus- 8There are now more than 2,000,000 hosts. —Eds.
tralia, Britain, France, and all over the U.S. I speculate that the list has at least 200 recipients, and about 25% of them are actually UUCP sites that are MX’d on the Internet.
I destroyed about 4,000 copies of the error message in our queues here at Apple Computer.
After I turned off our SMTP daemon, our secondary MX sites got whacked. We have a secondary MX site so that when we’re down, someone else will collect our mail in one place, and deliver it to us in an orderly fashion, rather than have every host which has a message for us jump on us the very second that we come back up. Our secondary MX is the CSNET Relay (relay.cs.net and relay2.cs.net). They eventually destroyed over 11,000 copies of the error message in the queues on the two relay machines. Their postmistress was at wit’s end when I spoke to her. She wanted to know what had hit her machines.
It seems that for every one machine that had successfully contacted apple.com and delivered a copy of that error message, there were three hosts which couldn’t get ahold of apple.com because we were overloaded from all the mail, and so they contacted the CSNET Relay instead.
I also heard from CSNET that UUNET, a major MX site for many other hosts, had destroyed 2,000 copies of the error message. I presume that their modems were very busy delivering copies of the error message from outlying UUCP sites back to us at Apple Computer. This instantiation of this problem has abated for the moment, but I’m still spending a lot of time answering e-mail queries from postmasters all over the world.
The next day, I replaced the current release of MAIL*LINK SMTP with a beta test version of their next release. It has not shown the header mangling bug, yet.
The final chapter of this horror story has yet to be written. The versions of sendmail with this behavior are still out there on hundreds of thousands of computers, waiting for another chance to bury some unlucky site in error messages.
Apple Computer’s Mail Disaster of 1991 89
Are you next?
[insert theme from “The Twilight Zone”]
just the vax, ma’am,
Erik E. Fair
Feb 21, 2017 | economistsview.typepad.comRC AKA Darryl, Ron : , February 20, 2017 at 04:09 AMMy Kaspersky Internet Security blocked the No Hesitations link for phishing. I trust Kaspersky more between the two.cm -> RC AKA Darryl, Ron... , February 20, 2017 at 11:04 AMProbably a false positive, or the site may run ads/third party code that was flagged, which are not controlled by the author. Ads with their intrusiveness have been well-known malware vectors. Ad funded sites (or which "monetize" their brand by selling ad space) typically outsource the mechanics of inserting ads, user tracking etc. to third parties, and necessarily give them control over what content is delivered to readers. In part due to high overheads and low margins, there is not a lot of "vetting" what content gets on the page.cm -> RC AKA Darryl, Ron... , February 20, 2017 at 11:25 AM
And of course intrusive tracking code may be legitimately viewed as malware in itself, even if it doesn't try to "infect" your system.
No reason to distrust the authors.
Even if you use an ad blocker, the schemes they use are necessarily heuristic and reactive in nature - basically block/avoid accessing content by known domain name, URL patterns, common "standard" image sizes and other tells, known suspicious patterns in programmable content, etc.
It comes with the usual heuristic-detection/classification problem - you can increase the result rate only by increasing the error, and you can only choose whether you want to err on the side of more false positives or false negatives (misses). This is because of the forced binary outcome (block/let through, or flag/don't flag).
Usually the algorithm computes some sort of confidence score that is used to detect (and possibly reject) "inconclusive" results. If the algorithm is any good, there is a region where it accurately detects (presence or absence of the targeted feature) with high confidence. To simplify, the choice is then to map low confidence scores to "detect" (more security) or "not detect" (more convenience).
Feb 20, 2017 | www.tecmint.com
mail is part of the mailutils (On Debian ) and mailx (On RedHat ) package and it is used to process messages on the command line.$ sudo apt-get install mailutils # yum install mailx
Now its time to send an email attachment using mail command a shown.$ echo "Message Body Here" | mail -s "Subject Here" email@example.com -A backup.zip
In the above command, the flag:
-s– specifies the message subject.
-A– helps to attach a file.
You can as well send an existing message from a file as follows:$ mail -s "Subject here" -t firstname.lastname@example.org -A backup.zip < message.txt2. Using mutt Command
mutt is a popular, lightweight command line email client for Linux .
If you do not have it on your system, type the command below to install it:$ sudo apt-get install mutt # yum install mutt
You can send an email with attachment using the mutt command below.$ echo "Message Body Here" | mutt -s "Subject Here" -a backup.zip email@example.com
where the option:
-s– indicates the message subject.
-a– identifies the attachment(s).
Read more about Mutt – A Command Line Email Client to Send Mails from Terminal3. Using mailx Command
mailx works more like the mutt command and it it also a part of mailutils (On Debian) package.$ sudo apt-get install mailutils # yum install mailx
Now send the attachment mail from the command-line using mailx command.$ echo "Message Body Here" | mailx -s "Subject Here" -a backup.zip firstname.lastname@example.org. Using mpack Command
mpack encodes the named file in one or more MIME messages and sends the message to one or more recipients, or writes it to a named file or set of files, or posts it to a set of newsgroups.$ sudo apt-get install mpack # yum install mpack
To send a message with attachment, run the command below.$ mpack -s "Subject here" file email@example.com
That's all! Do you have in mind any other methods of sending emails with attachment from the Linux terminal, that are not mentioned in the list above? Let us know in the comments.
Jan 12, 2017 | www.nakedcapitalism.comMatthew G. Saroff , January 12, 2017 at 2:56 pm
I read all my email on via SSH on a shell server.
If someone can hack my machine through a text window, they deserve to control my machine.
Nov 11, 2016 | www.nbcnews.com
A widespread problem
In the last few years, the Federal Trade Commission has sued more than dozen debt relief companies. "They simply lie to consumers," says the FTC's Alice Hrdy.
FTC ad IRS investigators have also found some counseling services that claim to be non-profit when they are actually a for-profit company. The non-profit pitch can make a potential client feel confident about signing up for the service. "They're preying on the consumer's trust," Hrdy says.
Some of the bad apples in this industry mislead people about their charges. "They either say there are no fees involved or just a small fee," Hrdy explains. Sometimes, they don't mention fees at all.
Bruce, who lives near Seattle, signed up with a company that promised to lower his interest rates. He was told to send them a check for $265.
"It was my clear understanding that money was going to pay off my credit card bills," Bruce told me. It turned out to be a "referral fee" to find him a company that would supposedly help him.
"It was a nasty experience," Bruce says. "They basically stole my money."
Warning: Debt settlement programs
Some companies now claim they can negotiate a one-time settlement with all of your creditors that will reduce your principal by as much as 50 to 70 percent. By doing this, they say, your monthly payments will drop dramatically.
"That is virtually impossible under any circumstances," says Travis Plunkett, Legislative Director of the Consumer Federation of America. That's why CFA warns consumers not to use debt settlement programs. "They are promising something they can't deliver," Plunkett says.
Credit counselors - a better option
Charles Helms, president of Consumer Counseling Northwest, sees a lot of people who have been burned by these phony debt relief programs. "It's horrible," he says. Because most of them have a large up-front fee, they'll take anyone who can pay.
"Their goal is to get you to sign up, not to successfully complete the program," Helms says. "So here's someone who is financially damaged to begin with and then these companies just go out and take the last of their resources and kill any hope they have of getting out of that situation."
With a legitimate credit counselor, there is no right answer for everyone. They sit down with you and give you a free and objective assessment of your financial situation. At Credit Counseling Northwest, they saw 6,000 people last year and found that debt management was the right option for only 19 percent of them. The rest were given a plan to work things out on their own.
With a customized consolidated payment plan you should be able to pay off your credit card debt in 3 to 5 years. You write the counseling agency one check each month and they pay all your creditors.Advertise Advertise Advertise
Do your homework
Facing mounting bills can be frightening, but getting debt relief is not a decision that should be based on hearing a radio commercial or getting a sales call. You want to find an organization that will design a debt relief plan specifically for you.
Shop around. Compare a couple of services and get a feel for how they operate. The credit counselor should spend at least 20 to 30 minutes with you in order to get a complete picture of your finances. If they don't do that, you're not really getting any counseling.
Ask a lot of questions and get those answers in writing. Find out about the fees. The Consumer Federation of America says you shouldn't pay more than $50 for the set-up fee and no more than a $25 monthly maintenance fee. If the agency is vague or reluctant to talk about fees, go someplace else.
Don't rely on names or the claim of a non-profit status. Check them out with the Better Business Bureau or your local consumer protection office.
By doing your homework you should be able to find a service that doesn't over-charge or over-promise. Here's a good place to start: The National Foundation for Credit Counseling . They'll help you find a certified counselor near you.
Nov 03, 2016 | www.weeklystandard.comPresenting...the Clinton IT Department! This has not been an especially ennobling election. Or a rewarding one. Or even entertaining. Pretty much everything about 2016 has been boorish and grotesque. But finally it is time to laugh.
This has not been an especially ennobling election. Or a rewarding one. Or even entertaining. Pretty much everything about 2016 has been boorish and grotesque. But finally it is time to laugh.
Ladies and gentlemen, I present the Clinton IT department.
Over the weekend we finally found out how Clinton campaign honcho John Podesta's emails were hacked. But first a couple disclaimers:
1) Yes, it's unpleasant to munch on the fruit of the poisoned tree. But this isn't a court of law and you can't just ignore information that's dragged into the public domain.
2) We're all vulnerable to hackers. Even if you're a security nut who uses VPNs and special email encryption protocols, you can be hacked. The only real security is the anonymity of the herd. Once a hacker targets you, specifically, you're toast.
I'm a pretty tech-savvy guy and if the Chinese decided to hack my emails tonight, you'd have everything I've ever written posted to Wikileaks before the sun was up tomorrow.
But that is … not John Podesta's situation.
What happened was this: On March 19, Podesta got what looked--kind of, sort of--like an email from Google's Gmail team. The email claimed that someone from the Ukraine had tried to hack into Podesta's Gmail account and that he needed to change his password immediately.
This is what's called a "phishing" scam, where hackers send legitimate-looking emails that, when you click on the links inside them, actually take you someplace dangerous. In Podesta's case, there was a link that the email told him to click in order to change his password.
This was not an especially good bit of phishing. Go have a look yourself. The email calls Podesta by his first name. It uses bit.ly as a link shortener. Heck, the subject line is the preposterous "*someone has your password*". Why would Google say "someone has your password?" They wouldn't. They'd say that there had been log-in attempts that failed two-step authentication, maybe. Or that the account had been compromised, perhaps. If you've spent any time using email over the last decade, you know exactly how these account security emails are worded.
And what's more, you know that you never click on the link in the email. If you get a notice from your email provider or your bank or anyone who holds sensitive information of yours saying that your account has been compromised, you leave the email, open your web browser, type in the URL of the website, and then manually open your account information. Again, let me emphasize: You never click on the link in the email!
But what makes this story so priceless isn't that John Podesta got fooled by an fourth-rate phishing scam. After all, he's just the guy who's going to be running Hillary Clinton's administration. What does he know about tech? And Podesta, to his credit, knew what he didn't know: He emailed the Clinton IT help desk and said, Hey, is this email legit?
And the Clinton tech team's response was: Hell yes!
No, really. Here's what they said: One member of the team responded to Podesta by saying "The gmail one is REAL." Another answered by saying "This is a legitimate email. John needs to change his password immediately."
It's like the Clinton IT department is run by 90-year-old grandmothers. I half-expect the next Wikileaks dump to have an email from one Clinton techie to another asking for help setting their VCR clock.
As the other guy likes to say, "only the best people."
Here are some easy ways the Clinton team could have avoided getting hacked and might prevent it in the future.
There is probably no one more acutely aware of the importance of good cybersecurity right now than Hillary Clinton's campaign chairman John Podesta, whose emails have been laid bare by WikiLeaks, are being mined for news by journalists (including at The Intercept), and are available for anyone with internet access to read.
So as a public service to Podesta and everyone else on Clinton's staff, here are some email security tips that could have saved you from getting hacked, and might help you in the future.
Use a strong password
There's a method for coming up with passwords that are mathematically unfeasible for anyone to ever guess by brute force, but that are still possible for you to memorize. I've written about it before, in detail, including an explanation of the math behind it.
But in short: You start with a long list of words and then randomly select one (by rolling dice), then another, and so on, until you end up with something like: "slinging gusty bunny chill gift." Using this method, called Diceware, there is a one in 28 quintillion (that is, 28 with 18 zeros at the end) chance of guessing this exact password.
For online services that prevent attackers from making very many guesses - including Gmail - a five-word Diceware password is much stronger than you'll ever need. To make it super easy, use this wordlist from the Electronic Frontier Foundation.
.... ... ...
Use a unique password for each application
The same day that WikiLeaks published Podesta's email, his Twitter account got hacked as well. How do you think that happened? I have a guess: He reused a password that was exposed in his email, and someone tried it on his Twitter account.
... ... ...
Turn on two-factor authentication
Last year, when I asked National Security Agency whistleblower Edward Snowden what ordinary people could do to improve their computer security, one of the first pieces of advice he gave was to use two-factor authentication. If Podesta had enabled it on his Gmail account, you probably wouldn't be reading his email today.
Google calls it "2-Step Verification" and has an excellent website explaining why you need it, how it works, and how it protects you. In short: When you log in to your account, after you type in your password you'll need one more piece of information before Google will allow you to proceed. Depending on how you set it up you might receive this uniquely generated information in a text message, a voice call, or a mobile app, or you could plug in a special security key into your USB port.
Once you start using it, hackers who manage to trick you into giving up your password still won't be able to log in to your account - at least not without successfully executing a separate attack against your phone or physically stealing your security key.
Watch out for phishers
... ... ...
Well-crafted spear-phishing emails can be incredibly hard to spot, but if you ever end up on a website asking you for a password, you should be skeptical. Check the URL and make sure you're at a legitimate login page before typing in your password, or navigate to the login page directly.
Encrypt your email
.... ... ...
To get started, check out the Electronic Frontier Foundation's Surveillance Self-Defense guide for using email encryption for Windows, Mac OS X, and Linux. If enough people in your organization use encrypted email, consider using our newly released tool GPG Sync to make it somewhat simpler.
Don't listen to the wrong people
... ... ...
Sep 28, 2016 | ftalphaville.ft.com
Izabella Kaminska joined FT Alphaville in October 2008. Before that she worked as a producer at CNBC, a natural gas reporter at Platts and an associate editor of BP's internal magazine.If your email provider suffered a security breach would you:
a) prefer to be informed about it as soon as possible so as to take evasive action?
b) prefer not to be informed until years later, by which time any evasive actions may have become pointless?
On the basis you chose the first option and a security breach happened, would you:
a) appreciate the warning and the password reset nudge, dismiss the incident to a Smeg happens scenario and continue using the service provider because at least they're vigilant about security?
b) Recoil in disgust at the very idea your email provider's security systems were lax enough to allow this to happen and immediately defect to a rival provider?
On the basis you would have chosen the first option and then the first option again (and then a security breach happened), how would you then react if your email provider determined that a) it was better to keep you in the dark about it and b) this was because they anticipated you would defect?
To wit, here's a nice insight from Nicole Perlroth and Vindu Goel at the New York Times for the legacy loyal yahoo email users still out there (h/t @melaniehannah):
Mr. Stamos, who departed Yahoo for Facebook last year, declined to comment. But during his tenure, Ms. Mayer also rejected the most basic security measure of all: an automatic reset of all user passwords, a step security experts consider standard after a breach. Employees say the move was rejected by Ms. Mayer's team for fear that even something as simple as a password change would drive Yahoo's shrinking email users to other services.
Two points on the back of that.
As a yahoo email user, I can testify to the fact that being continuously told by friends and family that: "Hey there, I think your email may have been hacked" is incentive enough to defect to an alternative provider.
Second, when I tried to download our complete email history so as to shutter the account formally, we found that this was in fact impossible unless we had the time and temperament to forward up to 20 years worth of email individually to a new account.
To date I am yet to get a reply from the Yahoo service team with respect to how I might get my hands on my own data in a more practical manner.
Speaking of frictions, here's another relevant snippet from the article:
The "Paranoids," the internal name for Yahoo's security team, often clashed with other parts of the business over security costs. And their requests were often overridden because of concerns that the inconvenience of added protection would make people stop using the company's products.
All of which suggests the crux of Mayer's Yahoo strategy was focused on maximising the security/access paradox to her own benefit. Namely, maximising access to the detriment of user security if it helped to bolster Yahoo's user numbers, but minimising user access to their own data if it helped to maximise the security of yahoo's own stock valuation.
Nice. This entry was posted by Izabella Kaminska on Wednesday September 28th, 2016 17:02 . Tagged with cyber security , yahoo .
The choice between security and ease of access is a difficult one, and shouldn't be trivialized. Password policies are a good example - overly loose, and hackers will be able to guess users' passwords; overly strict (e.g., requiring a password change every month), and users will resort to passwords on sticky notes stuck to their monitors. If you make things too difficult for users, they will find ways to ease the burden, and some of those ways will actually make security significantly worse.
That's not to say that Yahoo made the right decision, but it is to say that it isn't as easy as assuming that more security is always better.
I have managed to use the web for 20 years without ever visiting yahoo.com - by intention. I got the impression that they try to imprison their users rather than empower them.
I assume their e-mail service was 'free'. If so their users got exactly what they paid for.
In an ideal world each e-mail would cost the sender a cent. This would solve the problem of spam, and generate funds to develop and promote better web security.
Oooh, you had a Yahoo email account? You've just lost a big chunk of credibility.
I mean I have a Yahoo account (as well as a Netscape account and a Hotmail, sorry, whatever they call it) plus one or two others. Every time a new email provider has popped up I check their tech credentials and migrate to the provider that seems to hire the best techies. They get the sensitive mail. I keep the old accounts and use them for spam-associated registrations and whatnot.
Presently Google and Proton are my principal providers. Anyone who carried on with Yahoo for sensitive mail has nobody to blame other than him/herself.
@ Simple Simon
Settle down. Changing email accounts is a hassle, particularly for one's contacts.
@izabellakaminska - setup up your yahoo account and your new email account on an email client like mac mail or microsoft outlook- make sure they are both setup as an IMAP account. Wait for all the yahoo email to download and then simply select all messages and drag them across to your new account.
@ OBA Better yet, just leave the digital past...proud achievements and baggage alike...and step into the future with a clean slate.
@ OBA Thank you, this is a great suggestion. I've been trying to figure out how to backup my Yahoo! account - I only use it for signing up for things where I might get spam, but still wanted an easy way to back it up. I already used an e-mail client to get e-mails for one of my other accounts, I don't know why it never occurred to me to do the same for Yahoo!.
OpenDKIM is method to digitally sign & verify emails on the mail servers using public & private keys. In other words opendkim implements the DKIM (DomainKeys Identified Mail) standard for signing and verifying email messages on a per-domain basis.
- Set Up DKIM On Postfix With dkim-milter (CentOS 5.2)(Apr 06, 2009)
- Sign emails with OpenDkim + Postfix + Ubuntu 10.04 64x(Oct 04, 2010)
- iRedMail: Build A Full-Featured Mail Server On CentOS 5.x(Apr 22, 2009)
- Set Up DKIM (DomainKeys Identified Mail) Working With Postfix On CentOS Using OpenDKIM(Nov 16, 2012)
- Using OpenDKIM to Sign Postfix Mails on Debian(May 16, 2013)
- Configure DomainKeys (OpenDKIM) with Postfix on CentOS 7(Dec 03, 2015)
- Configure An Email Gateway With Scrollout F1 Anti-Spam & DLP(Jul 26, 2012)
Oct 13, 2015 | Daily Mail Online
Datto Inc has been revealed to have stored Hillary Clinton's emails - which contained national secrets - when it backed up her private server
- It claims it runs 'data fortresses' monitored by security 24 hours a day, where only a retinal or palm scan allows access to its facilities
- But its building in Bern Township, Pennsylvania, doesn't have a perimeter fence or security checkpoints and has two reception areas
- Dumpsters at the site were left open and unguarded, and loading bays have no security presence
- Clinton faces first Democratic debate tonight amid falling poll numbers and growing questions
The congressional committee is focusing on what happened to the server after she left office in a controversy that is dogging her presidential run and harming her trust with voters.
In the latest developments it emerged that hackers in China, South Korea and Germany tried to gain access to the server after she left office. It has also been reported that hackers tried to gain access to her personal email address by sending her emails disguised parking violations which were designed to gain access to her computer.
Daily Mail Online has previously revealed how a former senior executive at Datto was allegedly able to steal sensitive information from the company's systems after she was fired.
Hackers also managed to completely take over a Datto storage device, allowing them to steal whatever data they wanted.
Employees at the company, which is based in Norwalk, Connecticut, have a maverick attitude and see themselves as 'disrupters' of a staid industry.
On their Facebook page they have posed for pictures wearing ugly sweaters and in fancy dress including stereotypes of Mexicans.
Its founder, Austin McChord, has been called the 'Steve Jobs' of data storage and who likes to play in his offices with Nerf guns and crazy costumes.
Nobody from Datto was available for comment.
Daily Mail Online
Investigation by the Associated Press reveals that the clintonemail.com server lacked basic protections
- Microsoft remote desktop service she used was not intended for use without additional safety features - but had none
- Government and computer industry had warned at the time that such set-ups could be hacked - but nothing was done to make server safer
- President this weekend denied national security had been put at risk by his secretary of state but FBI probe is still under way
... ... ...
Clinton's server, which handled her personal and State Department correspondence, appeared to allow users to connect openly over the Internet to control it remotely, according to detailed records compiled in 2012.
Experts said the Microsoft remote desktop service wasn't intended for such use without additional protective measures, and was the subject of U.S. government and industry warnings at the time over attacks from even low-skilled intruders.
.... ... ...
Records show that Clinton additionally operated two more devices on her home network in Chappaqua, New York, that also were directly accessible from the Internet.
" That's total amateur hour. Real enterprise-class security, with teams dedicated to these things, would not do this" -- Marc Maiffret, cyber security expert
- One contained similar remote-control software that also has suffered from security vulnerabilities, known as Virtual Network Computing, and the other appeared to be configured to run websites.
- The new details provide the first clues about how Clinton's computer, running Microsoft's server software, was set up and protected when she used it exclusively over four years as secretary of state for all work messages.
- Clinton's privately paid technology adviser, Bryan Pagliano, has declined to answer questions about his work from congressional investigators, citing the U.S. Constitution's Fifth Amendment protection against self-incrimination.
- Some emails on Clinton's server were later deemed top secret, and scores of others included confidential or sensitive information.
- Clinton has said that her server featured 'numerous safeguards,' but she has yet to explain how well her system was secured and whether, or how frequently, security updates were applied.
'That's total amateur hour,' said Marc Maiffret, who has founded two cyber security companies. He said permitting remote-access connections directly over the Internet would be the result of someone choosing convenience over security or failing to understand the risks. 'Real enterprise-class security, with teams dedicated to these things, would not do this,' he said.
The government and security firms have published warnings about allowing this kind of remote access to Clinton's server. The same software was targeted by an infectious Internet worm, known as Morta, which exploited weak passwords to break into servers. The software also was known to be vulnerable to brute-force attacks that tried password combinations until hackers broke in, and in some cases it could be tricked into revealing sensitive details about a server to help hackers formulate attacks.
'An attacker with a low skill level would be able to exploit this vulnerability,' said the Homeland Security Department's U.S. Computer Emergency Readiness Team in 2012, the same year Clinton's server was scanned.
Also in 2012, the State Department had outlawed use of remote-access software for its technology officials to maintain unclassified servers without a waiver. It had banned all instances of remotely connecting to classified servers or servers located overseas.
The findings suggest Clinton's server 'violates the most basic network-perimeter security tenets: Don't expose insecure services to the Internet,' said Justin Harvey, the chief security officer for Fidelis Cybersecurity.
Clinton's email server at one point also was operating software necessary to publish websites, although it was not believed to have been used for this purpose.
Traditional security practices dictate shutting off all a server's unnecessary functions to prevent hackers from exploiting design flaws in them.
In Clinton's case, Internet addresses the AP traced to her home in Chappaqua revealed open ports on three devices, including her email system.
Each numbered port is commonly, but not always uniquely, associated with specific features or functions. The AP in March was first to discover Clinton's use of a private email server and trace it to her home.
Mikko Hypponen, the chief research officer at F-Secure, a top global computer security firm, said it was unclear how Clinton's server was configured, but an out-of-the-box installation of remote desktop would have been vulnerable.
Those risks - such as giving hackers a chance to run malicious software on her machine - were 'clearly serious' and could have allowed snoops to deploy so-called 'back doors.'
The U.S. National Institute of Standards and Technology, the federal government's guiding agency on computer technology, warned in 2008 that exposed server ports were security risks.
It said remote-control programs should only be used in conjunction with encryption tunnels, such as secure VPN connections.
November 01, 2006
Last week, a handful of employees at Dekalb Medical Center in Decatur, Ga., received e-mails saying they were being laid off. The subject line read "Urgent - employment issue," and the sender listed on the message was at dekalb.org, which is the domain the medical center uses. The e-mail contained a link to a Web site that claimed to offer career-counseling information.
And so a few employees, concerned about their employment status and no doubt miffed about being laid off via e-mail, clicked on the link to learn more and unwittingly downloaded a keylogger program that was lurking at the site.
Score another one for spammers.
Called targeted spam or spear phishing , this type of spam that's currently on the rise is particularly vexing because the spammer is able to "spoof" the sending e-mail address to make it look like it's coming from within the organization of the recipient, making it difficult for spam filters to catch. And, unlike traditional spam that is sent in the thousands, spammers are sending just handfuls of these messages at a time, again making it difficult for antispam technology to detect.
Tip of the Trade: Sendmail's Greet_Pause
By Carla Schroder
Slamming is a popular spammer tactic in which the spammer quickly fires off SMTP messages without waiting for responses from the receiving server. A poorly behaved MTA will then accept traffic from the spammer, instead of rejecting it as it should. But even well-behaved MTAs are affected because of the sheer volume of traffic with which they are forced to deal. The venerable sendmail, as of version 8.13, has a nifty feature called "greet_pause" that not only rejects incorrect SMTP transactions, but also discourages re-sends.
In a normal SMTP transaction, the client first connects and the server is supposed to send back a "220" greeting, something like:$ telnet mail.foo.org 25 Trying 188.8.131.52... Connected to foo.com. Escape character is '^]'. 220-host6.foo.org ESMTP Sendmail 8.13.6/8.13.6; Wed, 14 Jun 2006 18:04:49 -0600 220-We do not authorize the use of this system to transport unsolicited, 220 and/or bulk e-mail.Then, the client says "ehlo" or "helo," and the transaction continues. When the client is an impatient spammer and sends more commands without listening, the greet_pause feature detects this, marks the connection bad, and responds to anything else that tries to come over that connection with a 554 (transaction failed) message. It works by pausing briefly before sending out its 220 messages.
The pause interval is configurable, so you can tune it as needed.
Interestingly, you'll probably find that your total spam attempts drop significantly after implementing greet_pause, possibly because the spammer's software thinks it's hitting a bad server or bad addresses, or otherwise getting stuck somehow. It's an ingenious and simple method with a low-overhead that discourages significant amounts of spam.
As always, be sure to whitelist all of your important addresses. Visit sendmail.org/doc/ to learn more.
Open WebMail is a webmail system written with Perl. It is designed to manage very large mail folder files in a memory efficient way.
It also provides a range of features to help users migrate smoothly from Microsoft Outlook to Open WebMail.
Open WebMail has the following features: multiple languages, multiple iconset/styles, strong MIME support, SMTP relaying, virtual hosting, user aliases, pure virtual user, per user based capability, multiple authentication modules, PAM support, folder/message management, draft folder, confirm reading support, full content search, spellchecking, auto reply, mail filter, webdisk, calendar, event reminder, POP3 support, online password changing, message count preview, user history, and persistant running support.
smash - "Secure Mail Shell (or smash for short) is a utility for securely executing commands on a remote system via email."
You can find it at: http://www.karl.jorgensen.com/smash/
About: Email Security through Procmail provides methods to sanitize e-mail, removing obvious exploit attempts and disabling the channels through which exploits are delivered. Facilities for detecting and blocking Trojan Horse exploits and worms are also provided.
Changes: Improved the sender address checking to better avoid notifying forged sender addresses, and fixed the quoting of unquoted attachment filenames that have embedded semicolons. Improved active HTML defanging a bit, improved customizability, and made other minor improvements and bugfixes.
Mailing list archive:
Download as Tarball (be sure to completely read the README.TXT file and configwm.pm.dist template when using from tarball)
Download RH7 RPM (SRPM)
Download a tar of RH7 RPMS of PERL modules needed (to install, do a tar xvf nswm-packages.tar, then a rpm -ivh perl-*)
Sources can also be retrieved from anonymous cvs:
cvs -d:pserver:firstname.lastname@example.org:/cvsroot/nswm login
cvs -z3 -d:pserver:email@example.com:/cvsroot/nswm co nswm
You can also browse the CVS.
Ask your questions to the NS Forum.
NikoSoft WebMail is a set of PERL scripts working as CGIs, on any platform (Linux being the development platform), and released under NSPL (a slightly modified GPL). It is a web-based application allowing your users to retrieve and send mails using SMTP and POP3. Mails are encoded using MIME standards, and downloading or uploading attachments is possible. It has be designed to be "light and simple", and therefore:
And still feel free to submit bugs: use the "forum"
Subscribe to the NikoSoft Development mailing list: mailto:firstname.lastname@example.org?subject=subscribe
Messaging in all its forms makes up the heart of the Internet. Like our human hearts, we rarely give it much notice until something starts to break. Then suddenly, we realize we can't live without it. During the dark hours of September 11th in New York City and Washington, the telephone system couldn't handle the load. So where did people go to communicate? To the Internet and e-mail.
All this is very comforting, but in an exclusive interview with ConsultingTimes, Greg Olson, the Chairman of Sendmail, Inc., warns that e-mail is fast approaching its own communications and administrative crunch. To solve the problem, Sendmail has rolled out a new set of products to ramp up transaction volumes and simplify e-mail administration for large enterprises and service providers.E-mail's 30th Birthday
First, a couple of bytes of history. E-mail unceremoniously had its 30th birthday last week. Ray Tomlinson, the Cambridge, Massachusetts engineer regarded as the "father of e-mail," doesn't recall the contents of the first e-mail he authored, or who he sent it to. Bit prior to his invention, you could only send messages to another person on the same computer. His ingenious 200-line program let you send messages to anyone on a computer network. He even devised to use the now ubiquitous "at" symbol to route messages to their designated recipients.
A decade later, Eric Allman, a computer scientist at the University of California at Berkeley, wrote Deliver Mail, the precursor of sendmail, the first universal mail transfer agent to reformat and forward messages across network boundaries. The sendmail code was freely distributed, and quickly became a staple item on UNIX servers. As Allman's invention evolved, it drove the development of Internet mail standards.
With the subsequent explosive growth of the Internet, the core sendmail team had a hard time keeping up with the challenge of maintaining, enhancing, and supporting the technology used by the majority of mail systems. In looking for help to fund his project, Eric turned to an old friend, Greg Olson. Together they formed Sendmail, Inc., a hybrid enterprise which sponsors the development of the free sendmail, even as it develops commercial products for larger enterprises.Interview with Greg Olson
What follows is a verbatim transcript of our interview with Sendmail's chairman. E-mail systems are so poorly understood within the enterprise, and the issues faced are so critical, that it's worth listening to his every word.The Need for a Commercial Product
CT: Can you tell us a little about the origins of commercial Sendmail?
Olsen: Yes, there's a tremendous amount of interesting stuff happening in e-mail. It's actually the part of the Internet that people use most, but it's not the high profile part, because it isn't real glitzy. Underneath the surface, it turns out that there's a tremendous amount of change, upheaval, and challenge.
E-mail grew a lot beyond its original research community bounds. In the mid-90s, Eric and his team of volunteers who developed and maintained sendmail just couldn't keep up with the load for support requests and new feature requests. At that point Eric and I got together to look at how sendmail can step up to exploding Internet demands.
The research community had been growing by a few guys every year. What we found very quickly was that the explosion was happening at the commercial sites. So I went out and talked to a lot of companies using sendmail, and they had requirements that were -- not surprisingly -- a little different from those of the research community.
These guys wanted standard, packaged, push button installs, and warrantied distribution. They wanted tools that made sendmail easy to administer, particularly for system administrators that had not attained guru status yet.
Quite frankly, sendmail, because it's incredibly powerful, is quite tricky to configure.
CT: That's sendmail with a small "s"?
Olsen: With a small "s" -- the open source. And the third thing people really, really needed was service and support -- tech support, training, and consulting services.
In order to meet those needs, we decided it would take a company. We formed a commercial company called Sendmail, Inc., built on top of the open source, added new features and, since then, about 20 other things that had been on the commercial users' wish lists in the areas of security, availability, and management tools.
So that's the model. Sendmail as a company continues to do, invest in, and release the open source technology. That's an important part of why the company exists. The investors don't mind that we pump resources, because they see it as the world's best developer program.
CT: Does Sendmail with a large "s" employ the open source sendmail?
Olsen: Yes. The baseline code is the same.Combining Commercial and Open Development
CT: What determines what portions remain in the commercial application and what gets fed back to the sendmail open source?
Olsen: We do, and it's an interesting problem. Let me tell you how we deal with it.
The hybrid model deals with this using fairly traditional product line thinking. In the open source we use product lines for the development community. The commercial line is a product line for people who use this to run businesses. We've got some real smart product managers who argue about what goes in which product line. That's not unique. In fact, any company that has multiple product lines has this problem. Basically, we argue on the basis of what the target users in these segments need.
Now, there are a couple of other rules. Anything that has to do with standards has to go everywhere, of course. Anything that's contributed by the community of open source users must be fed back to that community as part of the open source. We sometimes put some of the commercial innovations into the open source because we want to spur open source development. For instance, the APIs are developed commercially to do very efficient content filtering. We published that in the open source, and that encourages the development community to come up with new, imaginative ways to do mail filtering.
We've also developed commercial filters -- both on a consulting basis and on a commercial product basis -- that are part of the commercial product line. But they can be plugged into the open source because the APIs are compatible.
CT: So the open source community could replicate the commercial products if they chose to?
Olsen: Yes, I you want to do that much programming. Of course. For the average commercial user, it isn't cost effective to try to do this yourself.
CT: Is the Sendmail company more oriented to services than to the code itself?
Olsen: No. In that sense, we're unlike some of the other open source hybrids. The original model, for instance, done by Cygnus [the maker of open source systems and development tools, acquired by Red Hat] was that the software itself was always free, but you paid for service.
Services are an important part of our business -- about 40% of our revenues last quarter were from services -- but the packaged products that contain all the enhancements from the commercial guys' wish lists are very much the focus of business here. In that sense, Sendmail, Inc. looks very much like a traditional software business.
CT: Do the people on the sendmail.org side see any conflicts with the commercial side?
Olsen: No, they've been very enthusiastic. Sendmail, Inc. is a sponsor and a responsible member of the open source community. We host an annual conference of the key contributors. We fly them in from around the world. They spend time brainstorming, sharing their ideas, and charting new developments for the next year as to where Internet mail needs to go.
We also employ full time programmers to contribute to the open source. Since we've done that the rate of innovation in the sendmail open source distributions has gone up quite a bit.
The third thing is very interesting to the community in general. It's not technical, but has to do with PR. Everyone who works on open source stuff is a little irritated by the fact that Microsoft comes up with the most gratuitous bell or whistle and plasters it over every publication in the universe. But when somebody in the open source community comes up with a ground breaking improvement to the usefulness of the Internet, for instance, nobody hears about it unless they happen to read the right newsgroup. We have the resources to inform people about what's going on in the mail space and make sure they hear about it.
Adding Commercial Tools
CT: What kind of reception has the company enjoyed?
Olsen: We launched Sendmail in March of 1998, and largely because sendmail is so widely used we've had a pretty fast ramp. We've got about 30 thousand commercial servers out there -- a little more than ten thousand companies have become customers.
We've also added more pieces to the product line besides the mail routing agent. That includes mailbox servers and web and wireless servers. I'll talk a little about that in the context of architecture. Sendmail is the most recognized part of the mail architecture, so most people thought sendmail was also supposed to host mailboxes, although they were probably using something like QPOP. People thought we already did that and wanted a complete solution, so it made sense to offer that as part of the commercial package.
CT: So the lower case sendmail doesn't do mailbox management?
Olsen: That's right. There are a couple of solutions that are widely used in open source.
CT: What else distinguishes the commercial products?
Olsen: We've added things like graphical management tools, and distributed management tools. Scalability is also very important for commercial customers. If you want to run a million clients or ten million clients, that's pretty hard to do with open source stuff that's off the net. We've added a number of features to make that pretty straightforward and give you very high availability.
The E-mail Explosion
CT: Is it time to get more background on the mail explosion itself?
Olsen: Yes. The key trend in e-mail is volume. It's continuing to explode. Last year there were about 3.1 trillion messages over the Internet, and that adds up to about 30 petabytes of data, which is a thousand terabytes or a million gigabytes.
For business users, Gartner tells us that you're seeing a triple geometric growth. There are about 40% more mailboxes each year, the number of messages per mailbox is growing at about 40% per year, and the size of the messages themselves is growing at about 40% per year. So there's a geometric growth somewhere in the vicinity of 250% to 275% a year for companies running e-mail.
CT: That's phenomenal.
Olsen: Yes. So what this means is that everybody's mail system is going to break, if not this year, next year. Or, if you really had a lot of foresight and engineered the hell out of it, you might get by ‘till the year after. But with this kind of geometric growth, everything is breaking.
CT: Literally breaking?
Olsen: Yes, it runs out of capacity. Now what most people see are administrative headaches. As the system gets more and more overloaded, the administrators have more and more trouble.
This is particularly true for groupware. The solution for groupware is "So, we're getting overloaded? Add another server." The administrative headache is a cross product of the number of nodes that you have to manage. It's not bad when you've got one Exchange server. Ten is a lot more complicated. A hundred is a nightmare -- your administrative environment just goes crazy.
While the architecture is at the root of the problem, what people will see is that the mail gets unreliable. In the case of Exchange servers, they actually lock up, and you lose mail when you restart them. The Linux technology is typically more resilient, but the mail really slows down, or you start getting big delays in the delivery of the mail. The cost of managing more servers goes through the roof because it's complicated.
The Criticality of E-mail
The next point of pain is the criticality of e-mail. E-mail in the business environment used to be a neat thing to play with, even as little as three years ago. But it's now become essential to business communications. All manner of business documents are typically sent by e-mail now, as opposed to mail or FedEx, because it's instant, and nobody wants to wait.
The other trend is that companies are using it as their fundamental means of communication, which means everybody in the company has access to e-mail. So now e-mail has to be as reliable as the telephone system. We also have to look after security and control and the fact as you start to use e-mail for business transactions, very often legal requirements come into the picture. This is particularly true in the financial services industry and the health care industries where there are, in fact, regulatory requirements that have to be met.
The Need for Content Management
Because e-mail is essential, you really need to control the environment. That brings in a new category of function called "content management". It's pretty obvious in the case of, say, virus filtering. You don't want any mail to contaminate your whole company in about ten minutes. Other content management issues have to do with the inappropriate use of the mail system for, say, harassment. We've got customers who are checking and screening for employees using profanity in mail going out to their customers. These are the sorts of things you need to start looking at, and that's hard to do that with the kind of mail systems people had in place three years ago.Wireless Mail
The next thing that's creating a lot of pressure is web and wireless mail. E-mail is, in fact, the lead wireless data application -- one of the reasons people get it. And when people are wireless, they want realtime access to their mail -- the same mail they get on their desktop -- they don't want it delayed. That's why they're using such wireless technology -- to be up to the second.
Web mail itself is the fastest growing form of mail in the enterprise environment. Particularly when you want to give mail to all the employees, like factory workers or retail workers, web kiosks are an economical way to do that.
This is causing a lot of pain in organizations, just because of the variety of devices, the volume of mail, and the way you have to splice it into the existing system.Geometrically Rising Costs
The other thing is costs. Cost reduction seems to be the top priority right now. For service providers, it's a matter of survival. If you have to add more servers and more administration, costs go up at least as geometrically as the volume is going up. The costs of adding capabilities like wireless mail, for instance, are daunting. Those are the real hot buttons in the environment now.
CT: So the mail usage has gotten has gotten ahead of the systems in place?
Olsen: In most organizations, this is true. And the most common problem in all of this comes down to architecture. One thing that we've found here, being in this business of helping commercial companies to do mail right, is that it's challenging because very, very few people understand how mail works.The Mysteries of E-mail
CT: We all hear about it, we all use it ...
Olsen: But how does it work? It's funny. When we set up Sendmail I went out and asked those who used systems in corporate environments what they wanted, and we put that right into the product. So we figured that everybody would buy it the first week. But it turned out, they can't. They can't buy it because they don't understand what they need and what it's supposed to do.
This is why we built the services organization and why it's an important part of the business. When we go to a company, it's usually because they have one particular problem, like the mail system is breaking for too much volume, or they need to add archiving because there are regulatory requirements, or they need virus filtering because the latest virus just brought their whole company down.
We'll start with them and say "How many nodes -- how many MTAs do you have at present on the Internet?" "Oh, gee, anybody know around here? I don't know." That's easy, because the way the Internet works we can get on the net and poll and tell them the answer in about 30 seconds. But, the next question is, "Now how do those mail servers connect into all the users and mailbox servers in your company? What is your mail architecture?" That one also draws the some blank. The answer is typically "Well, we don't know how this works because the guy who set it up three years ago isn't here anymore, and none of us know how to configure the cf files. But it hasn't broken, so we don't mess with it." This is Fortune 100 companies, not just small companies.Defining the E-mail Architecture
CT: When you talk about a triple 40% increase in volume each year, that's just going to blindside you.
Olsen: Right. Or one of those new requirements comes in. Suppose you're a brokerage company, and all of a sudden you realize the SEC requires you to archive any mail with the customer about trades. Well, gee, how do I put that into the system when I don't even know how the system works?
Quite frankly, we usually have to sit down with them and do an archeology assignment first, to understand how their mail system works today. Then we have to say "Well, what do you want it to do over the next five years?" Design them an architecture, and then help them implement it. The real problem is that they don't know how it works today, and they don't know how they need it to work tomorrow.
CT: And you're talking about systems administrators, in the corporation?
Olsen: Oh yeah, the people who are responsible for mail in the corporate environment. This is true for small companies, up to the very largest.
What people are aware of is they've got the Internet, they've got some mail servers around, like Exchange, or Domino, or maybe some iPlanet, or some of the Linux technology. They have a lot of clients around -- POP clients, IMAP clients, maybe some web clients.
Let me tell you about the pieces of this architecture and what we do. The first thing is the presence on the public net. These are called mail routers -- MTAs.
CT: And sendmail is one of those.
Olsen: Sendmail is the most common technology used for that on the Internet. The key aspect here is that we want to provide a very secure, reliable Internet presence. Because this is, in fact, an application-specific firewall. The e-mail port on the Internet is basically going through this so it has a very important security function.
What we've done is extend that with our plugin architecture which allows you to plug in filters to look for viruses, spam, or other kinds of prohibited content. Some companies don't like to take GIFs off the Internet mail 'cause it's usually pornography. You're crazy if you take vbs' off the public net, right? So these filters help you to do that, and also allow you to set up archive functions, based on policy. So if it's on a list of customers, you can archive it because it's probably required if you're trading with them.
Filters come in two kinds: inbound and outbound. People don't think so much about virus filtering on an outbound basis, but as it turns out, that's real important. I don't know if you've ever had to call up someone and say "You know that e-mail I sent you 30 minutes ago...? Please don't open it!" Then the guy says, "Oh gee, I'm sorry, I forwarded it to everybody in my department."
CT: That's all you want to hear.
Olsen: If that was your best customer, you just blew it. There was actually a case in Japan where one of the leading consumer products companies ran a web site. Associated with that they e-mailed bulletins and updates. They sent viruses to 35 thousand customers.
So you really want to have outbound filtering as well. The advantage of doing this at the routing backbone is that you can set it up as a policy that applies to the whole company. There's no easy way to get around it and it's easy to administer on the central backbone. We provide the distributed management console that lets you manage and monitor the whole architecture as a single, coherent system. It's graphical, it's got wizards, it runs on Linux ...Managing E-mail from One Location
CT: That's very nice. So you can manage all your servers from one location?
Olsen: Right. It's actually quite sophisticated. It allows you to set up different classes of servers with different kinds of parameters. You can maintain, propagate, and manage them as classes, as opposed to individual machines.
Then it's got monitoring tools. You can look at the traffic and see if things are getting overloaded. It also contains alerts so, for instance, if the queues are filling up on one machine, you can page the operator to take a look.
CT: Can you re-rout traffic, if necessary?
Olsen: Yes. That can be configured dynamically. Basically what it does is make the management of one of these complex networks as easy as possible. It's also secure. So if it's two in the morning and you're at home, you can, using TLS encrypted protocols, safely manage that system from outside the firewall.The World's Biggest Mailbox Server
The next element is the mailbox hosting. This is where the mail drops to and where the individual interacts with it to read and post new mail. Through our distribution routers, we can tie in things like Exchange, Domino, and iPlanet, and make them more reliable and manageable, with all these policies and filters in place.
We also provide our own mailbox hosting servers. The ones that we offer focus very much on providing the most reliable, cost-effective mailbox solution. You get the largest number of users for any given hardware. We've really tuned it to the optimal that way -- it's a great combination with Linux. In fact, on the high-end Linux, like for the IBM zSeries, we recently announced the world's biggest mail server. We can get two million users on a single system.
CT: That's an incredible number.
Olsen: It is. It's a little big for your average corporate user, but for a service provider, it's wonderful.Scaling Sendmail
CT: Can that server handle mail from multiple locations?
Olsen: Absolutely. And the system is designed to be easy to scale. Typically our systems are cost effective starting at about a thousand users, up to 10 million -- which is the largest one we've got deployed now. The scaling is incremental -- you can add processors, memory -- it's never disruptive and it's always a low incremental cost to go to the next level.
CT: This is essentially Linux running on various hardware?
Olsen: Right. It goes from unit processor Linux up to multiprocessor Linux, clustered with RAID storage, then clusters of Linux servers. That's how you go from one thousand to ten million without any discontinuity.
CT: It will run on various IBM servers, including the largest now?
Olsen: Yes. Actually, we support all the major platforms, not just Linux. IBM AIX, Solaris -- NT is our highest volume platform, believe it or not.The Economies of Consolidation
CT: But basically you're saying that a large enterprise facing this crunch needs to go to a mainframe solution?
Olsen: Right. This gets you out of managing lots and lots of little departmental servers. This thing will scale as big as you want -- you save a lot of money when you consolidate the mailboxes on one managed server.
For companies that already have a mainframe to run, say, their accounting system, dropping in a Linux workload -- since the system is already paid for and already managed -- is extremely cost effective, even if the system is pretty small.
CT: Right. That's the Integrated Facility for Linux.
Olsen: The other thing is for service providers or large companies. We just signed up a global consulting company that is going to consolidate all of their worldwide mail systems onto a zSeries mainframe. They're saving a ton of money because they're consolidating servers -- in this case they were Sun servers that were all over the globe. That saves a lot of money. Not just the hardware -- the killer is the day to day administrative costs.
I talked about mailbox hosting. The other thing that's very important here is the new web and wireless access. We provide servers to do that, and it's completely standard layers on top of the mailbox hosting. It gives you a very scalable way to tie in WAP, i-mode, and web clients.
The other thing that's important there is that it's extremely customizable. It can be tailored so you can have your own company look and feel for web mail, WAP mail, or whatever. It's all templated, so you can do that quite easily.
These are building blocks. The truth is, if you've got mail in there, you do have an architecture with all these elements in it. It's probably organized in a way that's a lot messier than what I just talked about -- a simple, layered architecture where everything between the layers is standards based.
We typically go to a customer and say "What are your requirements?," and then we can draw from the menu of building blocks an architecture that will meet their requirements for some time into the future, and give them an easy way to scale as they grow.
The other thing that's very important is our consulting services. We've got guys here that have put in more of these than you can find anywhere else. We've done about six hundred enterprise mail architectures now.A Goldmine for Consultants
Quite frankly, in terms of your readership, I'll bet a lot of your people are Linux consultants. Boy, is this a hot area! Because everybody is dying and nobody has a clue. An interesting call to action is that people who understand something about how mail works, and can help companies sort out their mail architectures, have limitless opportunity out there right now. I can't hire enough people in that category, and, quite frankly, I don't want to hire them all.
The right way to get involved, particularly at the small and medium sized business level, is with local, service-oriented entities. This is a great place for somebody to get oriented and get plugged in. Because of this incredible growth rate and all these new requirements that are dropping in as mail becomes part of the critical business -- the demand for this stuff is fairly infinite.
CT: Networking was the hot job. Now you're saying that handling the mail crunch within that is even more so?
Olsen: Right. It's a subset that right now has infinite demand because this thing is continuing to grow wildly. Incidentally, we see absolutely no recession at all in e-mail. One of the guys that's on the board is Andy Bechtolsheim, the founder of Sun. He's now one of the CTOs at Cisco. The way he puts it is, "You know, it's unbelievable. Cisco's business is way down. Our partners' business is way down. Our vendors' business is shrinking. But I'm still getting more e-mail every day."
CT: And it's interesting because companies are cutting back on airline travel while using e-mail more. E-mail is driving a reorientation of the way people work.
Olsen: That's exactly right. It's going to mean a lot more Powerpoints get mailed. People are doing more distributed work. The fact is that this stuff works so well -- it's so useful -- that the trend seems to be unstoppable.
CT: So the crunch is going to get worse before it gets better?
Olsen: That's right. It's the geometric growth. It started when the Internet began to be plugged in commercially, about the mid-90s. Someday, we'll all get saturated with e-mail, but right now there's so much new use coming in every day that it's not going to slow down.
CT: So it seems that Sendmail is well-positioned as a company. You're not going to face recession in terms of demand?
Olsen: No, we're still growing this year. Not a fast as last year, but primarily because I don't have the capital resources to do as much as we'd like.
CT: So it's not for lack of demand.
Olsen: No, it's not market limited at all. It's just that capital is harder to come by.Sendmail and IBM
CT: Sounds very promising. And your mainframe offerings are relatively new?
Olsen: Yes, we just announced those at LinuxWorld at the end of August.
CT: IBM has put Linux on the mainframe, which is great. But then you say "Well... what are you going to run on it? Where are my critical applications for the mainframe?" It sounds like Sendmail is one of those.
Olsen: Absolutely. What's really interesting is when I first heard this -- when the IBM folks first approached me and said "We want you to put this stuff on the mainframe" -- I had to wonder, because it's not what you think of as an Internet server.
It turns out though it's almost a perfect server for e-mail systems. E-mail is an I/O bound application. Mainframes do not have the highest CPU MIPS capability, but they have massive I/O capability. The other thing about e-mail is that people want it to never go down. Reliability is one of the most important things in the new environment. Mainframes are as good as you can get. MTBF on a 390 is about 60 years now.
CT: Sixty years, wow!
Olsen: They've optimized it for 30 years. They just don't go down. Everything's redundant, auto failover -- it's the benchmark for reliability.
The other thing is that it comes with all these management tools that make it easy to manage in a way that's reliable. Quite frankly, the thing that brings systems down now is not usually hardware failure -- it does happen -- but it's usually operator error. The staffs that run the mainframes are people who know how to keep systems up all the time.
So it turns out to be a really, really good match. The scalability is amazing. We've proven that we can put two million users on the platform, and that's using pretty conservative measures. It's very cost effective to add it to workload if you've already got a mainframe running something else. It's very efficient, and IBM's price for Linux dropped very aggressively.
CT: It certainly has. Do you find that IBM is bringing customers to you?
Olsen: Yeah, actually it works both ways. They are bringing customers to us and IBM is pretty enthusiastic about the idea of selling mainframes for new applications. "New workloads," they call it. This hasn't happened for a while, so it's pretty exciting to them.
The other thing that's happened is that now that we have a handle on where the cost advantages are compelling on the mainframe, we've actually brought IBM into situations where people were struggling with the costs of the mail system, and telling them that they can do it much more cost effectively than they thought. That's been a pleasant surprise. People don't think of mainframes as cheap, but it's a lot cheaper than running half a dozen Sun Enterprise Servers, for instance. They're not cheap either. And vastly cheaper than running a little warehouse full of Intel boxes.
CT: Particularly not just the hardware, but the administrative costs.
Olsen: Yes, it's actually the administrative costs that kill people because it grows as a geometric with the number of nodes.
Quite frankly, reliability is a problem too. If you have a warehouse of Intel boxes, keeping everything up is just impossible. So the mainframe's great. It's got it's place particularly for the larger areas where you can consolidate, or for companies that are already running critical stuff on mainframes.Solutions for Smaller Enterprises
For all of the other places there are a lot of other platforms. We're real impressed with Linux on the latest Intel hardware, particularly as you get into the 64 bit stuff on Itanium. The numbers that we're seeing out of the lab are so good that we didn't think that the benchmarks were run right.
CT: So for a reasonably small company, this could be very cost effective.
Olsen: Yes, absolutely. The thing about mail is that everybody needs it, so it has to go from small up to big.
CT: Do all of your products run on Windows and Linux?
CT: So you're basically platform-agnostic?
CT: You'll go with whatever is most cost effective for the customer?
Olsen: Yes, and quite frankly there's some arbitrariness there. If you've got a shop where everybody knows NT and nothing else, that's going to be the most cost effective for you. In general, we try to support all the platforms that people need. Open source sendmail, incidentally, is supported on, I think, eighty-eight platforms.
CT: So open source sendmail scales the same way, but just doesn't have the administrative tools?
Olsen: It has some of the same scaling capability, since it architecturally has the same base code, but, quite frankly, a lot of what you need to scale effectively is in the tools environment. The biggest problem for scalability is the administrative environment that lets you gang these things together in large numbers and still run it as a coherent system.
CT: And that's your switching product?
Olsen: Yes, Sendmail Switch is routing product line, and the big investment here has primarily been on the management tools, above the open source.
CT: It's very interesting ... and very promising. You gave us a lot to go on.
Apr 19, 2001 | Linux Journal
(130 reads) (1 talkbacks) (Posted by kreichard)
"In this article we will address a basic installation procedure (sort of a recipe for a quick set up of your mail server) for the average user. Assuming the system is a home or small company network with a Linux machine running Sendmail as the mail server, Sendmail's functions will be to receive mail messages from machines on the internal network, deliver local messages to their respective users and deliver to the Internet messages for external destinations. Additionally, the server will receive mail from the Internet."
"Under Linux there are several MTAs including sendmail, the most common across most forms of UNIX; and D.J. Bernstein's qmail and Wietse Venema's Postfix. These accept and relay mail. This sounds quite simple, but in practice it can be quite complex. There are a number of routing and masquerading options that can be set by administrative policy --- and these amount to programming languages that filter and modify the headers of each message as it is relayed. In addition the process of routing mail and finding user mail boxes (mail stores) can involve arbitrarily complex interactions with various directory services (DNS, passwd files, NIS, LDAP alias/dbm files, and all manner of custom databases)."
"These days MTAs also have to implement anti-spam features that amount to access control lists and rules about the address formats (to and from headers) that are allowed from specific domains and address ranges. (Those generally also involve queries on tables or directory services, including those like Paul Vixie's RBL (real-time blackhole list: or MAPS, mail abuse prevention system) and it's ilk, like Dorkslayer/ORBS. Recently, MTAs are being increasing required to enforce other policies and implement anti-virus/anti-worm features."
The code itself can be found at sendmail.org or from their FTP sever. A complete list of changes in sendmail 8.10.0 is available on sendmail.net.
Re:Sendmail ... (Score:2, Funny)
by SuperJ on Wednesday March 08, @08:34AM EST (#14)
(User Info) http://www.mbhs.edu/~josborn
We had sendmail running on one of our Linux machines in the the computer lab. A sysop came up to us and said "What? You don't need sendmail, shut that down." I said, "You gotta have sendmail, what if you forget the root password? You gotta be able to find a bug in sendmail and hack root!"
Re:Sendmail ... (Score:1, Informative)
by Anonymous Coward on Wednesday March 08, @09:16AM EST (#33)
Here's something to put security holes in perspective.
Re:Sendmail ... (Score:1)
by Molina the Bofh (email@example.com) on Wednesday March 08, @05:13PM EST (#82)
More people are running it in production environments than any other MTA.
More people are running Win 95/98 in production environments than any other OS. More people run wu-ftpd than any other ftpd. More people watch TV than read newspapers.
sendmail's bugs tend to get found very quickly, publicized immediately, and fixed very quickly.
They have a quick response because they're already used to it. And, besides, a quick response for a software bug is common practice in the open source community, specially if security-related. But the point is: a well designed MTA wouldn't have that many bugs.
Re:qmail vs. sendmail (Score:2, Informative)
by goodEvans (firstname.lastname@example.org) on Wednesday March 08, @08:35AM EST (#17)
Dahling, if you want Qmail eeeasily, then you really must try e-smith Server and Gateway.
Installs in an hour, add addresses via a web interface and so much more, it's really quite exhilarating....;-)
sendmail versus qmail. (Score:2, Informative)
by mrsam (email@example.com) on Wednesday March 08, @08:26AM EST (#11)
(User Info) http://www.geocities.com/SiliconValley/Peaks/5799/etrouble/
"it would seem that qmail has gotten the upper hand as far as features were concerned."
You have to be kidding, right?
At least as of a couple of weeks ago (haven't checked recently), Qmail hasn't been updated in three years. Here are some features in sendmail that are nowhere to be found in Qmail:
ESMTP AUTHentication/some kind of SASL support
RFC 1894 Delivery Status Notifications
Any kind of spam filtering
Qmail is still incapable of batching recipients for the same domain into one transaction
And there's more where that's came from. I suppose DJB has been a bit occupied, the last couple of years, fighting the US Commerce Dept on the crypto issue, so Qmail has gotten a bit moldy.
Re:sendmail versus qmail. (Score:2)
by arivanov on Wednesday March 08, @08:38AM EST (#21)
You are correct. You also forgot that sendmail is a swiss army knife. You can configure it to do almost anything short of dry cleaning and laundry. The only pending rival here may be the new exim with perl-like capabilities in the config.
But at the same time, Qmail still rips the guts out of sendmail as performance. Qmail does not have the record of the second most security-troubled sofwtare after Washington University Qmail still has more flexible local delivery support which sendmail gets only via various external delivery agents. Qmail as is does not have SPAM filtering. If you want to kill SPAM you can
- easily integrate it into the local delivery.
- Modify the RBL patches to refer to your own antispam database. This is elementary. Been there done that (not myself, by one of my colleges, I did the sendmail rulesets ;-). As a result you can get network wide synchronized SPAM filtering. As you probably deduced it can be done with sendmail as well ;-)
@*** Baker's Law *** Misery no longer loves company. Nowadays it insists on it.
Re:sendmail versus qmail. (Score:2)
by mrsam (firstname.lastname@example.org) on Wednesday March 08, @01:05PM EST (#69)
(User Info) http://www.geocities.com/SiliconValley/Peaks/5799/etrouble/
"Qmail still rips the guts out of sendmail as performance."
Only in low/medium bandwidth situation. A properly tuned sendmail will beat the pants off Qmail in high volume mailings, mostly because of sendmail's ability to batch recipients to the same domain, and ability to recycle SMTP sessions.
I agree on the remaining points.
Re:"One of the primary people"? (Score:0)
by Anonymous Coward on Wednesday March 08, @02:35PM EST (#74)
There are many people who work on sendmail. Greg Shapiro and Claus Aßmann are probably the primary contributors. You should check out the RELEASE_NOTES which lists some contributions and who provided them.
Sendmail.net (Score:4, Informative)
by noeld on Wednesday March 08, @08:28AM EST (#12)
(User Info) http://rootprompt.org
For lots of really good information on Sendmail 8.10 checkout Sendmail.net
RootPrompt.org -- Nothing but Unix
News and information for Unix Sysadmins
Multiple Queue Support (Score:4, Informative)
by EraseMe on Wednesday March 08, @08:35AM EST (#16)
(User Info) http://www.elapsed.net/
I found this to be interesting:
Support multiple queue directories. To use multiple queues, supply a QueueDirectory option value ending with an asterisk. For example, /var/spool/mqueue/q* will use all of the directories or symbolic links to directories beginning with 'q' in /var/spool/mqueue as queue directories. Keep in mind, the queue directory structure should not be changed while sendmail is running. Queue runs create a separate process for running each queue unless the verbose flag is given on a non-daemon queue run. New items are randomly assigned to a queue. Contributed by Exactis.com, Inc.
This could be great for my Solaris box with 50,000+ active SMTP connections, as we may be able to segregate the mail queue onto seperate partitions! :)
Still cannot fix this broken machine. - Trent Reznor, 1992
Re:Sendmail Sucks (Score:2, Informative)
by mbyte on Wednesday March 08, @09:22AM EST (#35)
(User Info) http://www.sambahq.de/
read that "bat book" from o'reilly ...and look at those m4 files.
You don't need to edit a .cf file in order to configure sendmail ... using the m4 files is very easy ... want to use cyrus deliver ?
Thats it ! :)
I think sendmail is quite EASY to configure ... :)
(and its still FAR more configurable than qmail or postfix :)
Have you tried Linuxconf? (Score:1, Interesting)
by Anonymous Coward on Wednesday March 08, @11:15AM EST (#54)
I realize that Linuxconf is only available for mail servers running Linux, but if you are, I can really recommend it. Linuxconf will give you a web-, X-, or ncurses-based menu driven interface for configuring sendmail (and lots of other stuff). While it may not be a usability dream come true, it sure beats hand-editing sendmail's configuration files. And, if you need to customize your sendmail configuration beyond what Linuxconf offers, it will let you do that too, using your favourite command line tools.
Re:Sendmail Sucks (Score:3, Insightful)
by garver on Wednesday March 08, @11:51AM EST (#58)
I can speak for qmail with a little larger number of users. I have qmail running for a small ISP with 3000+ accounts. The same machine is handling authentication, file serving, POP, etc.
The machine is bored and its a low-end PC. You could build it for $1500 today. We push 15000+ messages a day.
We switched from sendmail/qpopper to qmail. I got tired of administrating sendmail, not having real virtual email account support, watching qpopper slam my disk by copying the user's mail file everytime they popped, etc, etc. sendmail just has too much baggage and isn't elegantly designed in the first place.
qmail is built very modular, tiny programs to handle every stop of the MTA process. This makes it more secure, setuid'ing whenever it can, reducing the amount of code that ever sees root permissions. Also, it is very easy to extend. I have qmail-pop authenticating from a SQL database, just by replacing the the checkpassword program.
After using it, Maildir support is a must. In a Maildir, each message is a file. It sounds like a waste of inodes, and it is, but the performance benefits are incredible. Now when a user POPs, they don't have to lock their mailbox, and only touch the messages that they want. Before qmail, qpopper was causing my server (then running 1000 users) to write 4 GB/sec on my little 4 GB drive. In addition, my secondary mail server can deliver into the same mailboxes without locking, etc.
I will give you that qmail can be a pain to administer by hand since its configuration is kind of distributed, with .qmail files in user's homedirs, redirecting their mail, etc. But I built a management system on top of it. This is where qmail really sings for us. We can change damn near anything just by twiddling some files, no restart, rebuilding config files, etc.
And the best part, in my opinion, I have been using qmail for 1 year and I'm still using the same version. It does what it does and is rock solid stable and secure.
How's that for a testimonial?
Why sendmail worked for us when qmail didn't. (Score:1)
by Deven (email@example.com) on Thursday March 09, @01:24AM EST (#91)
(User Info) http://www.ties.org/deven/
Okay, I wrote a long, detailed article on this topic, but Netscape crashed on me just when I was about ready to post it after an hour of composing it. Sigh. (I'll try to keep it shorter this time.)
We were trying to make a scalable, reliable, efficient and nearly fault-tolerant mail platform based on a strategy of cheap servers clustered around more expensive (but stable) NetApp filers. The inspiration for this architecture came from the following excellent Earthlink papers:
- A Scalable News Architecture on a Single Spool
- A Highly Scalable Electronic Mail Service Using Open Systems
We wanted to use Maildir format to avoid NFS locking issues on the shared mail spool. (The locking problems seemed to be the main trouble that Earthlink had, using Unix mailbox format.) At the insistence of a new hire, we tried using qmail instead of sendmail as the MTA. (My preference was sendmail, since I know it well; qmail was interesting but an unknown quantity, and we were under a tight deadline.)
Unfortunately, in our attempts to move to the intended server architecture, we ran into a number of assumptions in qmail which are hardcoded and scattered through the "modular" qmail code:
... ... ...
- qmail assumes its queue directory is local and plays games with the inode numbers (we wanted to experiment with an NFS-mounted queue for fault-tolerance, although the performance tradeoff may have proven unacceptable)
- qmail assumes there is only one queue runner, so of course no locking is done on the queue (we wanted to experiment with a shared queue so multiple servers could drain a single queue in parallel and distribute load better)
After fighting with qmail for several weeks, we ended up tossing all that work and starting over with sendmail when the new hire abruptly quit the company. In three days, we had most of the code written and working in sendmail that we fought with qmail for weeks trying to get it to do what we wanted.
In my experience, the core qmail code is nearly incomprehensible, totally unmaintainable, and the much tauted "security" seems to be mostly through obscurity. The code is filled with idioms unique to qmail, and riddled with cross-dependencies between the ridiculous number of separate source files (many of which are one line long). While it may be easy to extend in certain ways envisioned by the author, modifying the core code can be a nightmare.
Sendmail, on the other hand, is very clean. The code is well-modularized with clear interfaces. (I added a new map type to the sendmail source easily, in less than a day with very few lines of the original source modified.) The MDA functions are clearly separated from MTA functions, and the MTA doesn't make unwarranted assumptions. (It often doesn't even make warranted assumptions, but that's a different topic of discussion.) Making a Maildir version of "mail.local" was a breeze. Even modifying the arcane "sendmail.cf" file wasn't nearly as as hard as trying to work with the qmail source code!
In summary, qmail has a niche it fills well -- small, simple user communities on a single server. If you have more than about 5,000 users, you may start finding that the single server no longer can handle the load, and that's when you'll start to stumble across qmail's limitations. If you want to run a serious mail platform under heavy load, sendmail is a better choice.
"Simple things should be simple, and complex things should be possible."
Re:Why sendmail worked for us when qmail didn't. (Score:1)
by nxsy on Thursday March 09, @03:33AM EST (#92)
(User Info) http://moria.org/~nbm/
Some of your points above saying that 'qmail assumes' are simply what you assume that qmail assumes.
qmail does not assume that all users have entries in the passwd file, nor does it assumes all users have different UIDs, not in fact does it assume that each user has a home directory.
Just take a look at what vpopmail does to simply provide hints to qmail as to how to handle mail. All the stuff vpopmail does is easy to do manually, and all is easily understood from the available documentation. In fact, before I knew about vpopmail, I created a utility that did basically the exact same function in an hour or two.
Your other two points are true, although maybe not valid, and they're specific assumptions made by the code. You personally may think that a shared queue with multiple queue runners is the only way to work, but I would like to know whether you tried it qmail's way before deciding that was the way to do it. Admittedly, perhaps qmail isn't flexible in that area, but then again, perhaps qmail does it for a reason. djb is well-known for restricting people from shooting themselves in the foot, even if they might want to aim in the general direction of their foot, and are sure they won't hit it.
That said, I currently have no preference in MTAs between exim, postfix, and qmail, since they all seem to be very good products. I haven't had the time and inclination both at the same time to learn sendmail yet, but I'm sure I'll give it appropriate time before making any judgement.
Re:User Authentication or Roaming Profiles (Score:2, Informative)
by kzanol (firstname.lastname@example.org) on Wednesday March 08, @10:46AM EST (#52)
Sendmail 8.10 Supports SMTP Authentication; it'l interoperate with Outlook Express, Netscape, Eudora etc. To avoid reinventing the wheel, it uses the cyrus SASL Library to supply the authentication funnctions. See ftp://ftp.andrew.cmu.edu/pub/cyrus-mail for the current libraries. To install sendmail with Authentication, look for sasl in the src/README file in the distribution.
you have moved your mouse, please reboot to make this change take effect
Re:Other MTAs? (Score:2)
by Mark F. Komarinski (markkATcgipcDOTcom) on Wednesday March 08, @09:17AM EST (#34)
(User Info) http://www.cgipc.com/
Qmail is a bit more logical in it setup than Sendmail, but that's not saying much. Setup is fairly simple (an hour at most after reading the docs).
There are two places where Qmail really shines for me:
1) Security. There was a $1,000 reward to anyone who could find a bug in Qmail that would allow access to the host. The deadline was a year (IIRC) and it came and went without being paid. Sure, it's not as gone over as Sendmail, but in three years, none has reported a security bug of this nature.
2) Mailing Lists. There's a package for mailing lists called ezmlm that really works. Normal users can create their own mailing lists as a part of their name (like email@example.com) with all the regular features of Majordomo - automated sub/unsub, digests, etc. Creation is two or three commands - no editing files, no running "newaliases". It's available immediately.
I'm not sure how it handles big loads, but I have it on a few smaller boxes and I've never had trouble with it.
Re:Other MTAs? (Score:2, Interesting)
by Molina the Bofh (firstname.lastname@example.org) on Wednesday March 08, @10:29AM EST (#49)
>I'm not sure how it handles big loads, but I have it on a few smaller boxes and I've never had trouble with it.
Actually Qmail is way much faster than Sendmail and requires a lighter load with the same ammount of traffic.
I don't even think of using Sendmail. Why would one want to use a monolithic, buggy system like this? Sendmail has been designed WRONG from the very beggining (it's a monolithic program running as root most of the time). That's why so many security holes appeared. OTOH, a program whose compromise is with security (i.e. Qmail) runs as root the less time possible. No root account has been compromised via Qmail. The only problem that appeared is a possible DoS.
I sincerely can't understand why people go for crappy software. Another very popular example is wu-ftpd. Sorry to say that folks, but IMO wu-ftpd sucks. Have you ever tried to chroot an user using wu-ftpd ? Gee... Not only it's a pain in the ass, it's also messy. How many bugs have been reported to wu-ftpd ? It's also historically insecure. There are much better ftp daemons. My favorite is ncftpd (yes, this one is commercial).
So I just want to understand: Why are wu-ftpd and sendmail so popular ?
Re:Other MTAs? (Score:0)
by Anonymous Coward on Wednesday March 08, @01:02PM EST (#68)
Why are wu-ftpd and sendmail so popular ?
For the same reason that Unix and C and X-Windows and MS-Windows are popular. Because worse is better.
Re:Other MTAs? (Score:0)
by Anonymous Coward on Wednesday March 08, @01:55PM EST (#72)
I'm not sure how it handles big loads,
We are using it on a list server and deliver about 20 million emails per month using qmail/ezmlm. Our current setup handles up to 50 remote connections per second, so theoretically we could deliver abour 4M emails/day (or 120M/month, for the non-math people).
Re:Other MTAs? EXIM (Score:1)
by 13013dobbs (email@example.com) on Wednesday March 08, @11:17PM EST (#87)
Check out Exim. http://www.exim.org A very simple drop-in replacement for sendmail. Easy to install and powerful.
QMail -- Flame War (Score:0)
by Anonymous Coward on Wednesday March 08, @09:13AM EST (#31)
Its is only 2 seconds work to create a flame war between any two or more rivals. In this case, Sendmail, Qmail and Postfix could be argued about for years. My experience is with qmail, and I have found that with the goods available at inter7.com, there is no easier way to manage mail for a couple of 1000 users and a 1000 domains. Just my thoughts.
Re:The best of the new sendmail ... (Score:2, Informative)
by timmartin on Wednesday March 08, @12:29PM EST (#65)
Alexey from Messaging direct has been keeping lists of all things that support SASL. I'm not sure if the sites moved but here's a cached copy http://www.google.com/search?q=cache:www.taxxi.com/homerus/mail/SASL_ClientRef.html
Hopefully you'll be able to add mozilla to that list shortly too.
Re:The best of the new sendmail ... (Score:1)
by bodin on Thursday March 09, @12:47AM EST (#90)
(User Info) http://x42.com/
qmail has a patch for SMTP AUTH since a while back. Check out http://www.qmail.org/
Re:Can postfix and qmail handle multiple domains? (Score:3, Informative)
by Hazard Class (firstname.lastname@example.org (no class, Fat Albert)) on Wednesday March 08, @10:37AM EST (#51)
(User Info) http://risse.tierranet.com/asprin/books/thieves_world.html
Yes. Easily. qmail with the vpopmail addon from Inter7 will make you wonder why you ever bothered to try and configure Sendmail.
qmail is different than Sendmail, considerably so. But once you understand how it works, I think it's design is far superior to that of Sendmail. It's much more unixy, IMNSHO. There is ample evidence that qmail is considerably faster and less resource intensive than Sendmail, but what really made the difference for me was the security focus of qmail.
As I said, qmail is different from Sendmail, but there is a lot of contributed documentation available as well as commercial support. The qmail community is large, capable and very motivated. They do have one problem though, they don't have a 4-inch-thick O'Reilly book dedicated to their MTA...
...hmmm, maybe there's a reason for that!
Game Over Man! --Aliens
Virtuser table (Score:3, Interesting)
by autechre on Wednesday March 08, @12:30PM EST (#66)
(User Info) http://wmbc.umbc.edu
I have a server which is doing 3, soon to be 5 virtual domains. Apache configuration is simple. Sendmail was also very easy to configure. All you need to do is this:
1. Have support for a sendmail.cw file, so that it will accept mail for all the hostnames. Put the hostnames in that file :)
2. Add in support for virtusertable, which is similar to /etc/mail/aliases, but a bit different. This allows you to redirect, say, webmaster@host1 to a different place than webmaster@host2, redirect all mail for 1 domain to one place, etc.
I have the O'Reilly book, but I didn't actually need it; I found all the info I needed on www.sendmail.org. It took about 1/2 hour. In case you're wondering, I'm a college student who's been using Linux for about 2 years, not a 60-year-old UNIX guru.
Soto la panche, la capra crepa Sopra la panche, la capra campa
Comments from a Sendmail developer (Score:3, Interesting)
by cying on Wednesday March 08, @11:54AM EST (#60)
(User Info) http://www.csua.berkeley.edu/~chucky/
<CYA>Mini-disclaimer: I work for Sendmail, Inc., am one of Sendmail Switch's developers, but my opinions are not necessarily representative of those of Sendmail, Inc.</CYA>
Sendmail Switch isn't open source software, it's commercial software. It does many sophisticated management thingies besides configuring sendmail.
That being said, OS sendmail configuration got much easier since m4 configuration files came about. And while it's not an Apache-style configuration, etc., it's on the same level in terms of difficulty.
The OS sendmail developers work pretty much orthogonal to the commercial component developers. Feature sets of OS sendmail are driven by the OS community. They are aware of the inherent difficulty of configuring sendmail, and consider it to be quite a shortcoming of OS sendmail, independent of whether management components exist in a commercial software product.
You will probably see OS sendmail become easier to use somewhere down the line.
One final note, Sendmail Switch was built using open source technology. It's not apparent to people outside the company, but if you bought the product you'd see we use open source technology extensively in the product. The commercial component developers also believe in OS principles, which is why our products use open source technology where possible.
Sendmail Switch is commercial software. But buying it supports the company. Supporting the company supports the OS developers - giving a secure "home" and dedicated resources to OS sendmail development. Benchmarking, compatibility labs, food, and clothing are examples of such.
Hope that gives a small view from the inside.
Re:Sendmail upgrade caused slower performance? (Score:3, Informative)
by Anonymous Coward on Wednesday March 08, @04:25PM EST (#78)
IDE drives were never "fairly happening." They were a cheap, low performance technology to keep desktop prices low. They were never high performance or designed for servers.
Mail (and mail) is usually fairly IO bound (it must commit messages to disk per RFC 82(1|2) before passing them on). Get good disk and you'll go faster.
That said, I've been told that sendmail can't do more than a couple messages a second by "experts". Fortunately, my machines which ran a typical 30,000 messages/hour with bursts to 50 or 60k per hour didn't know about these "experts."
- Run SCSI. For more, run SCSI RAID. For more, run high performance SCSI RAID.
- Use tools like bulk_mailer on lists.
- Sort lists first by domain (better to send all the messages to hotmail over one single connection that tear down/start up the connections each time).
- Ponder hiring someone with experience to reviews your setup at the least. Common questions are answered on comp.mail.sendmail, but if you've got an unusual setup, or need someone to come in and help you, it's often cheaper in the end to hire a consultant with a clue to help for a couple days than to spend three weeks learning how to do it yourself.
Rob Kolstad wrote a paper for Usenix on tuning for lists a few years ago. If you're a member, you can find it. If not, join and find it.
8.10 (and the commercial product that uses it) allows multiple queues. This means that you can have 6 queues (each on a separate spindle) running mail for you. This should fill a T1 quite handily.
A big sendmail advantage is that you can get consulting and support. A company I did work for had those guys make some recommendations and help them and they seemed to benefit a lot. I figure if email is a production service, then buying support for it is a Good Thing. If the authors of Sendmail provide that, then great, money well spent - give back to the people who gave it to you (and these clients pay Sun a LOT for 24x7 hardware support).
Much of the tuning that can be done applies to any mailer. Sendmail, by default, is fairly "nice" to the machine. You can tune it a thousand ways so that it runs on machines from a 12MHz Sun 3 with 8MB RAM to a 128 way SGI at peak performance. If you want to tune it to chug out 120,000 message per hour and destroy the bandwidth of a 10baseT network, that can be done with some experience. If you don't have it, you can hire that experience.
Will 8.10 make a huge difference? Well it's been out for what, 15 hours? Beta for a while, but this has diffs from Beta12, so I don't think we know yet.
RE: the qmail/postfix rants. Showing release notes of security fixes of Beta releases doesn't offer that there was a hole that was exploited. It shows that the code has been reviewed (in beta and alpha, largely) and that potential problems have been removed. I thought that's was beta was for.
Adobe PDF (1,912 KB)
Zipped PDF (1,068 KB)
A new version of NIST Special Publication (SP) 800-45, Guidelines on Electronic Mail Security, is now available for public comment. The draft document, SP 800-45A, is a revision of the 2002 guideline and structured similarly, although a good deal of the material has been rewritten and augmented with new information. The guide is intended to aid organizations in the installation, configuration, and maintenance of secure mail servers and mail clients. Administrators of electronic mail and other infrastructure services are encouraged to provide feedback on all or part of the document. NIST requests submission of public comments on the draft on or before October 6, 2006. Comments may be sent to SP800-45A@nist.gov.
Softpanorama hot topic of the month
***** sendmail.net resources
Open Directory - Computers Software Internet Servers Mail -- recommended primary site
RFCs for the Rest of Us: Introduction
RFC 2554: SMTP Authentication
RFC 2476: Message Submission Agent
RFC 2553: Basic Socket Interface Extensions for IPv6
RFC 2034: SMTP Service Extension for Returning Enhanced Error Codes
RFC 1777: Lightweight Directory Access Protocol (LDAP)
RFC 2505: Anti-Spam Recommendations
Turning off Sendmail Forever
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.
ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least
Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.
Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info|
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Last modified: January 15, 2017