Recent Features
| |||||||||||||
LCA: Lessons from 30 years of SendmailLCA: Lessons from 30 years of SendmailPosted Feb 3, 2011 21:05 UTC (Thu) by dskoll (subscriber, #1630)In reply to: LCA: Lessons from 30 years of Sendmail by HelloWorld Parent article: LCA: Lessons from 30 years of Sendmail
Did I claim this was the case? No, I didn't. But it does help, and not just because more modern languages are memory-safe. They also tend to be slower (often a lot slower) and more memory-hungry. Unfortunately, if you want the ultimate in performance on a UNIX-like system, you're stuck with assembly, C or C++. Assembly is clearly the wrong tool for most applications, and IMO C++ is a monstrosity that adds a ton of garbage to C without really making it any safer. I can see writing most things in safer languages. But your MTA has to be fast and efficient, which is why we don't see many (any?) widely-used MTAs written in anything but C or C++.
(Log in to post comments)
LCA: Lessons from 30 years of Sendmail Posted Feb 3, 2011 21:34 UTC (Thu) by foom (subscriber, #14868) [Link]
> But your MTA has to be fast and efficient
I'd dispute that: I bet you could write an MTA in *Ruby* and it would be perfectly good on today's computers for all but the largest sites. There's not that much mail volume, and computers have gotten a lot faster...
LCA: Lessons from 30 years of Sendmail Posted Feb 3, 2011 21:44 UTC (Thu) by dskoll (subscriber, #1630) [Link] I bet you could write an MTA in *Ruby* and it would be perfectly good on today's computers for all but the largest sites. Go for it. Report back when done.
LCA: Lessons from 30 years of Sendmail Posted Feb 3, 2011 22:06 UTC (Thu) by foom (subscriber, #14868) [Link]
No thanks. I have no interest in writing a new MTA, nor in writing any Ruby code. :)
But I will point out that many large mailing list installations run on mailman, which is written completely in Python. That seems to be performant enough.
And spam filtering of email content is frequently done with spamassassin -- written in Perl.
LCA: Lessons from 30 years of Sendmail Posted Feb 3, 2011 22:40 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
Mailman is not a MTA.
it manages the list of users that the MTA will send mail to (a low overhead activity)
it validates messages sent to the list (it handles each message once, before it gets multiplied by potentially several orders of magnitude as it's sent out)
LCA: Lessons from 30 years of Sendmail Posted Feb 4, 2011 11:08 UTC (Fri) by paravoid (subscriber, #32869) [Link]
Have a look at Lamson, then.
LCA: Lessons from 30 years of Sendmail Posted Feb 4, 2011 19:42 UTC (Fri) by dskoll (subscriber, #1630) [Link] From the "About Lamson" page: However, as great as Lamson is for processing email intelligently, it isnt the best solution for delivering mail. There is 30+ years of SMTP lore and myth stored in the code of mail servers such as Postfix and Exim that would take years to replicate and make efficient. Being a practical project, Lamson defers to much more capable SMTP servers for the grunt work of getting the mail to the final recipient.
LCA: Lessons from 30 years of Sendmail Posted Feb 5, 2011 0:17 UTC (Sat) by dskoll (subscriber, #1630) [Link] And spam filtering of email content is frequently done with spamassassin -- written in Perl. Indeed so. We use Perl (including SpamAssassin) to filter our mail. In terms of memory size, SpamAssassin on our system is about 70MB per instance vs. about 9MB for Sendmail. (To be fair, we use a lot of other Perl modules apart from SpamAssasssin in our filter.) And when it comes to performance, SpamAssassin is so slow relative to Sendmail that Sendmail becomes completely negligible. Bolting anything in Perl onto Sendmail is like towing a five-ton truck with a motorcycle. :)
LCA: Lessons from 30 years of Sendmail Posted Feb 5, 2011 0:43 UTC (Sat) by foom (subscriber, #14868) [Link]
Yes. Perl is an order of magnitude slower than C. Furthermore, spam filtering is inherently a harder job than mail routing. Yet, Spamassassin is *still* fast enough! Thus my claim: MTAs don't need to be written in C.
But writing a new MTA from scratch now is pretty pointless, no matter what language it's in.
LCA: Lessons from 30 years of Sendmail Posted Feb 5, 2011 17:13 UTC (Sat) by dskoll (subscriber, #1630) [Link] Thus my claim: MTAs don't need to be written in C. That's probably true, 99% of the time. But again, MTA authors tend to worry a lot about performance and tend to write their software to cope with huge amounts of mail. I believe that's the correct approach because even a small site can suddenly get a huge spike in traffic for various reasons (eg, a spammer does a massive joe-job spam run.) You don't really want your email to fall over.
LCA: Lessons from 30 years of Sendmail Posted Feb 26, 2011 13:07 UTC (Sat) by job (guest, #670) [Link]
Spam filtering is very different from general mail routing. You might find a better example in qpsmtpd.
It is (almost) pure Perl and in use at quite a few large installations including the Apache project.
LCA: Lessons from 30 years of Sendmail Posted Feb 3, 2011 21:58 UTC (Thu) by HelloWorld (guest, #56129) [Link] Assembly is clearly the wrong tool for most applications [...]But your MTA has to be fast and efficient, which is why we don't see many (any?) widely-used MTAs written in anything but C or C++.When C was invented, compilers were primitive, and it was trivial to produce assembly code that would run faster than the equivalent C code. Yet, people decided in the 1970s to port the single most performance-critical piece of code - the UNIX kernel - to C, because that made it portable and generally easier and more pleasant to program. Nowadays, machines are faster by orders of magnitude, and there are actually quite a few safe programming languages that allow one to produce efficient code: Go, D, ATS and many others. More are being developed, such as Rust. Yet, people nowadays refuse to accept a negligible overhead over C (say, 20%, which is the goal set by the Go language), and this seems just bizarre to me. I believe that the main reason for sticking with C is simply inertia. "We've always done it that way!"
LCA: Lessons from 30 years of Sendmail Posted Feb 3, 2011 23:38 UTC (Thu) by dskoll (subscriber, #1630) [Link] Yet, people nowadays refuse to accept a negligible overhead over C (say, 20%, which is the goal set by the Go language), and this seems just bizarre to me. I believe that the main reason for sticking with C is simply inertia. "We've always done it that way!" I don't think that's the reason for sticking with C (it's not my reason, anyway.) Here are my reasons:
LCA: Lessons from 30 years of Sendmail Posted Feb 4, 2011 2:51 UTC (Fri) by HelloWorld (guest, #56129) [Link] I have a lot of experience with C and I like it. There's certainly something to be said for familiarity.I don't mean to insult you, but this is exactly the mindset I mean. It's just another way of saying "because I always did it that way", really. C is old and well-tested.Being well-tested helps against bugs. It doesn't help one bit against design mistakes (except if you change the design, but that never happened to a meaningful extent for the C language (I guess that's what you meant with "keeping C true to the original C spirit")) There are many C libraries and tools available, far more than for newer languages.Any decent language has a foreign function interface for calling C functions. And at least some of them, like D, make it absolutely trivial to create bindings. In some cases, a 20% performance hit isn't worth it. An MTA on a busy mail system is one of those cases. While it's true that most mail systems are not that busy, MTA authors rightly attempt to make their MTA as fast and reliable as possible.No, they don't. C won't give you the fastest possible result, assembly will (portability is not an excuse, use a portable assembly language like LLVM assmbly). Yet, people seem to think that C is somehow the be-all and end-all of programming languages when it comes to performance - it's preposterous!
LCA: Lessons from 30 years of Sendmail Posted Feb 4, 2011 3:19 UTC (Fri) by dskoll (subscriber, #1630) [Link] I don't mean to insult you, but this is exactly the mindset I mean. Meh... everyone tends to like tools he/she is familiar with. That's just human nature. Maybe some Perl or Ruby fanatics will eventually write an MTA in their language... Any decent language has a foreign function interface for calling C functions. But... but... then you're back in dangerous territory, no? No, they don't. C won't give you the fastest possible result, assembly will (portability is not an excuse, use a portable assembly language like LLVM assmbly) OK, now I know you're just arguing for the sake of argument. :) That's totally ridiculous and you know it. Anyway... go ahead. Write a UNIX MTA in something other than C or C++ and see how it fares. The proof is in the pudding.
LCA: Lessons from 30 years of Sendmail Posted Feb 4, 2011 12:59 UTC (Fri) by james (subscriber, #1325) [Link] Using qpsmtpd for traps.spamassassin.org describes how SpamAssassin had been using Postfix on a donated server for their spamtraps.Unsurprisingly, they were getting a lot of spam on that server, and postfix was having trouble keeping up. So they switched to qpsmtpd, "a flexible smtpd daemon written in Perl" which uses "Danga Interactive / Six Aparts insanely scalable event-driven asynchronous socket class". Justin reports: "results have been great we now have a pure-perl system handling heavy volumes without breaking a sweat, certainly compared to the previous system."
LCA: Lessons from 30 years of Sendmail Posted Feb 4, 2011 16:07 UTC (Fri) by dskoll (subscriber, #1630) [Link] I read that article. The problem with the Postfix solution was not Postfix, but the fact that they were filtering from a procmail-invoked Perl script. That's a huge fail. We run our Perl spam-scanner in conjunction with Sendmail, but we use the Milter interface to keep persistent Perl scanners running. The bottleneck in this system is by far the spam-scanning; Sendmail doesn't even show up on the radar.
LCA: Lessons from 30 years of Sendmail Posted Feb 5, 2011 1:23 UTC (Sat) by HelloWorld (guest, #56129) [Link] Meh... everyone tends to like tools he/she is familiar with. That's just human nature.This is another way of saying "I do it that way because I always did, but that's OK because everybody else does too.". But... but... then you're back in dangerous territory, no?Using a C library and writing the rest of the program in another language is still better than writing it all in C. OK, now I know you're just arguing for the sake of argument. :) That's totally ridiculous and you know itIt's not, you just didn't understand my point. You said that MTA authors attempt to make their software as efficient as possible, which, together with the fact that most MTAs are written in C, implies that C is the language you can write the most efficient software in. But this simply isn't the case: C is just one spot among many, many, many others on the efficiency-vs.-comfort curve. More specifically, it is not on the performance maximum of that curve, since that's where assembly language is. Yet, many (most?) people seem to believe that C somehow hit the perfect spot for "systems" programming. Even if that was true in 1970, people should start getting comfortable with the idea that it's not any longer.
LCA: Lessons from 30 years of Sendmail Posted Feb 5, 2011 1:51 UTC (Sat) by mjg59 (subscriber, #23239) [Link]
Yet, despite C++ having been available for so long, there's still a distinct lack of widely-used operating systems written in it. Good system programmers are often tend to be the ones familiar with the kernel underlying their code, and that means having to have a good grasp of C regardless of what you'd prefer to code in. There's a natural selection pressure in favour of C even if there are arguably better choices.
(The first significant codebase I worked on was C++, and I've probably still written more Perl than I have C)
LCA: Lessons from 30 years of Sendmail Posted Feb 5, 2011 17:10 UTC (Sat) by dskoll (subscriber, #1630) [Link] But this simply isn't the case: C is just one spot among many, many, many others on the efficiency-vs.-comfort curve. More specifically, it is not on the performance maximum of that curve, since that's where assembly language is. It's not a smooth curve. The transition from C to assembly involves a *huge* increase in the difficulty curve with a relatively modest increase in the efficiency curve. C is a just-high-enough/just-low-enough level language to hit a sweet spot in efficiency-vs-difficulty. But you already know this and are just arguing for the sake of it.
LCA: Lessons from 30 years of Sendmail Posted Feb 6, 2011 1:37 UTC (Sun) by HelloWorld (guest, #56129) [Link]
> But you already know this and are just arguing for the sake of it.
No, I merely disagree with you. C doesn't sit in a sweet spot, as there are many, many useful features that could be added without compromising efficiency, like templates, modules, proper macros and many more. Anyway, discussing this with you is obviously pointless by now.
LCA: Lessons from 30 years of Sendmail Posted Feb 4, 2011 7:34 UTC (Fri) by yoe (subscriber, #25743) [Link]
I find it interesting that you detract from C as an unsafe language by advocating for assembler.
LCA: Lessons from 30 years of Sendmail Posted Feb 4, 2011 23:45 UTC (Fri) by giraffedata (subscriber, #1954) [Link] I find it interesting that you detract from C as an unsafe language by advocating for assembler. He didn't advocate for assembler. It was reductio ad absurdum -- if fast is the only consideration, you would use assembler. Since you don't use assembler, fast is not the only consideration. Since fast is not the only consideration, maybe you should consider safety. Or lower development cost (with which you can probably pay for the extra hardware you need compared to the C or assembler implementation at the same speed).
LCA: Lessons from 30 years of Sendmail Posted Feb 5, 2011 2:07 UTC (Sat) by nybble41 (subscriber, #55106) [Link]
Actually, even if speed is your only consideration you are still likely to be better off using C rather than assembler where possible. A decent compiler can generate better-optimized code from C source than all but the best assembler programmers could write by hand.
That is not to say that there are no other relevant considerations, but other languages are not necessarily any safer or easier to develop in than C; it varies depending on the project. Sometimes there really is no other viable alternative. For example, the only other realistic systems programming language I know of is D, which (while otherwise a great language) is not as portable as C (yet), nor nearly as well established so far as libraries and tool support are concerned.
LCA: Lessons from 30 years of Sendmail Posted Feb 26, 2011 13:16 UTC (Sat) by job (guest, #670) [Link]
You are being inflammatory. The choice of programming language is clearly not the most important part of secure coding. Otherwise I'd expect you to run a Python based sshd on your system, no?
C is good because it is trivial and easy to read when done properly. Token expansion or resource exhaustion won't creep up on you from the underlying libraries.
There are of course other aspects to it. Given the choice between a C daemon and a Python one, without knowing the program I too would probably pick the latter one, but it all boils down to prejudice.
|
|||||||||||||