Weekly Edition
Return to the Development pageRecent Features
| |||||||||||||
Stresstesting IMAP clients (Hackvalue)
Over at the Hackvalue weblog, Armijn Hemel compares IMAP client performance. He created 12,000 test emails and then looked at how well several clients (Evolution, KMail, Thunderbird, and Claws Mail) could download the messages as well as how they performed moving them between accounts. "In the past I have had to move quite a bit of mail around between IMAP servers, which usually worked just fine, apart from taking some resources. Other people I talked to had different experiences, including crashes and suspicions of emails being lost. This triggered the idea of testing a few graphical email programs for IMAP performance: how would they behave if you had a huge amount of mails that you wanted to move to a different account?"
(Log in to post comments)
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 14:44 UTC (Mon) by vivo (subscriber, #48315) [Link]
12000 is far far away from huge
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 18:54 UTC (Mon) by drag (subscriber, #31333) [Link]
Yet it was enough to confirm the uselessness of Kmail and Thunderbird for
many people.
The state of GUI programs for support of IMAP is pretty sad affair,
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 23:29 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
12K messages is a fair number. it's enough to cause grief for mail clients that don't really use IMAP (i.e. they use IMAP as POP3 to fetch messages and then do everything locally), but it's not enough to cause problems if you have an IMAP server that uses ext2/3 and stores all the messages as seperate files (other filesystems handle this much better)
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 0:35 UTC (Tue) by nix (subscriber, #2304) [Link]
ext3/4 have no problems with that sort of thing either if you turn on
dir_index to use hashed directories.
I used to use reiserfs for my nnml spools and news tradspool. I've
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 0:58 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]
I haven't tried ext4 yet (it's design changes give it a good chance of performing well), but my testing with ext3 and dir_index have not given nearly the performance that I can get from XFS
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 15:05 UTC (Mon) by ber (subscriber, #2142) [Link] KMail has some known performance problems for quite a while now. This is one of the reasons for developing a new mailreader application based on Akonadi. For Akonadi there is already a new imap library (see this post to kolab-devel) and an Kontact-Prototype-E5 including and email reader using it. (Full disclosure: I am helping to coordinate the Kolab team working on this client.)For very large quantities of emails I guess using a specialized migration tool like imapsync could help in some cases as well.
offlineimap Posted Oct 5, 2009 23:16 UTC (Mon) by dmarti (subscriber, #11625) [Link] Need to put in a plug for offlineimap here. Really fast and reliable, although I haven't raced it against a MUA's built-in IMAP.
offlineimap Posted Oct 6, 2009 3:37 UTC (Tue) by k8to (subscriber, #15413) [Link]
I use offlineimap in a 2 minute looping shellscript. I love it to death.
I use pycmail to filter the mail afterwards, and mutt to read it.
it's about 50,000 times better than any mail gui client I've ever used.
offlineimap Posted Oct 6, 2009 10:26 UTC (Tue) by job (guest, #670) [Link]
Just curious, how is it better than using mutt for IMAP directly?
mswatch Posted Oct 7, 2009 0:00 UTC (Wed) by ChrisFrost (subscriber, #61105) [Link]
If you sometimes wish that you were notified of new mail before the next two
minute polling period or sometimes wish the polling used less bandwidth, you might be interested in http://mswatch.sourceforge.net/. mswatch invokes mail synchronization programs when it detects changes to the local or remote mailbox. (Bias note: I wrote mswatch.)
mswatch Posted Oct 7, 2009 0:13 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
1. IMAP probably doesn't need anything like this as the protocol supports efficient checking of messages
2. if you avoid polling, how do you find out that the remote mailbox has changed?
mswatch Posted Oct 7, 2009 0:26 UTC (Wed) by ChrisFrost (subscriber, #61105) [Link]
1. If the IMAP account contains hundreds or thousands of folders, the client
and server would need that many open sockets (for the IDLE command). Additionally, the server may have multiple clients, each doing this. Also, I don't happen to know of any clients that maintain an IMAP connection for each folder (Mail.app, Thunderbird, mutt, and mbsync, at least). mswatch improves on the IMAP protocol by using one socket, and only one inotify file descriptor, for an arbitrary number of folders.
2. mswatch asks the kernel to inform mswatch when files change, using the
mswatch Posted Oct 9, 2009 13:10 UTC (Fri) by job (guest, #670) [Link]
The IMAP protocol needs to be fixed so many IDLE commands can be active simultaneously or given the ability to IDLE over several mailboxes, whatever is best protocol-wise. This should be fixed in the most common implementations. IMHO, kludges will only make that work more difficult.
mswatch Posted Oct 9, 2009 14:48 UTC (Fri) by ChrisFrost (subscriber, #61105) [Link]
FYI, the mswatch page discusses all these points that you have raised, in
its "Why mswatch" section :).
"The author of IMAP, Mark Crispin, does not see this capability [multi-
FYI, the Lemonade working group is working on such a protocol. But this is
mswatch Posted Oct 10, 2009 2:35 UTC (Sat) by foom (subscriber, #14868) [Link]
> The author of IMAP, Mark Crispin, does not see this capability [multi-
> mailbox IDLE] as belonging in IMAP
Be that as it may,
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 15:26 UTC (Mon) by dwmw2 (subscriber, #2063) [Link] "The conclusion from this test is fairly obvious: Evolution and Claws Mail outperform both KMail and Thunderbird when faced with the task of moving large amounts of mail between IMAP accounts."That's all very well, but isn't really what I use a MUA for — especially a graphical one. If I want to move a large amount of mail around, I'll usually use rsync or scp. Maildir makes that nice and easy. And if I really had to use IMAP, I'd probably use pine. A more interesting test would have been how long these MUAs take to check for new mail in a set of a few dozen 'active' folders. That's something that happens quite a lot more frequently, and often before a graphical MUA will even start redrawing its windows properly on startup! I very much doubt that Evolution would perform well in that test. It used to be fairly OK, and would use the STATUS command that IMAP provides for just this purpose. But about 3½ years ago, it started doing something really stupid instead — it started fetching all the headers for every mail in the folders, just so it could tell you how many unseen messages there are. And then it got really slow. This is GNOME bug #336076.
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 16:52 UTC (Mon) by ken (subscriber, #625) [Link]
Then run evolution on an nfs mounted home directory and it positively crawls. it is many times slower than on a local hard disk. hundreds of megabyte of data goes over the network I have no idea what the application is doing.
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 0:17 UTC (Tue) by nix (subscriber, #2304) [Link]
Agreed. Evolution manages to take longer to read a single mailbox with
5000 mails in it (over NFS over GigE) than Gnus takes to get new mail in a couple of dozen mailboxes totalling perhaps 50,000 mails. And Gnus is not exactly renowned for speed.
One of these days I must sysprof it (now that sysprof has learnt to use
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 12:28 UTC (Tue) by xav (guest, #18536) [Link]
I also wanted to sysprof (or oprofile) it for a while. Nowadays Evolution is dead slow at startup, with a lot of communication with the server for just a few new mails.
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 0:28 UTC (Tue) by nix (subscriber, #2304) [Link]
Ye gods. It takes *hours* now? It's been that bad for *three flipping
years*?
And the developers say "this bug can't be fixed"? Well, Evolution can't be
("Evolution has features those other mailers don't." Evo has features Gnus
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 1:08 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]
the thing is that the result (how many unread messages are there) can be found very efficiently through the IMAP protocol (if nothing else, search for all messages with the 'new' flag and count them, although I suspect that there are even better ways)
the behavior you are describing for evolution is a perfect example of a POP3 mail client being modified to 'support' IMAP by using the IMAP protocol to fetch everything and then processing it locally.
My dad uses pegasus mail (under wine) and mulberry, he was doing some testing of searches recently and found that pegasus (really a pop client with imap commands) can take hours to search his mailboxes. mulberry takes seconds to do the same search.
pegasus fetches all the messages (in every folder) to do the search, mulberry sends a few IMAP commands:
thread <list of messages returned by the search commands> (returns messages in threaded order)
for each message that would fit on the screen, fetch the header info for that message.
I'll bet evolution would do the same thing as pegasus (in this case fetch a few hundred thousand messages over several hours)
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 11:10 UTC (Tue) by kelvin (guest, #6694) [Link]
> if nothing else, search for all messages with the 'new' flag and count them
Evolution used to use this method, but the problem is that deleted messages might still be flagged as "new". Combined with IMAP's braindead handling of deleted messages, this method causes discrepancies in the unread-count. There were bugs reported against these discrepancies.
Deleting messages in IMAP doesn't actually delete them, they are only flagged as deleted and you then have to manually "expunge" the folder. While this might have made sense to a computer-savvy user familiar with the mbox storage model, it is quite unintuitive to any user accustomed to modern GUI's (or MS Exchange/Outlook for that matter).
One solution might be for Evolution to "break" the IMAP spec by moving all deletion-flagged messages to the Trash-folder, including messages flagged by other mail agents. This would solve the problem for Evolution, and should be fairly non-disruptive for other mail agents.
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 17:28 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]
so change the search to be all messages with the new flag set and not the deleted flag set. it's still one IMAP command
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 22:37 UTC (Tue) by cortana (subscriber, #24596) [Link]
The hilarious thing is that since Evolution 2.26, these discrepancies have existed, even though Evolution does the stupid thing and treats IMAP like POP.
Day by day, I draw closer to the following conclusions:
1. IMAP is a horribly complex protocol
It's enough to drive one to gmail!
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 23:25 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]
IMAP has lots of features, true
but it's not _that_ complex (web UIs are fare more complex, especially once you add AJAX), but it does require a very different mindset from pop clients to do well. the first thing that you have to accept is the idea that mailbox access is not exclusive and dedicated to one system.
once you start thinking that you may be accessing the same mailbox from your laptop, desktop, phone, etc _at the same time_ so that no one app has full control of the mailbox (the other apps may make changes that you are notified of) then you are most of the way there, and the rest of the decisions will probably be pretty good.
there are a couple of clients that work well with IMAP, and a couple of good servers. this isn't that different from other areas
how many good HTTP clients and HTTP servers are there? not that many (and on the client side, you could argue that most of the common ones in use are not among the short list of good ones ;-)
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 22:53 UTC (Tue) by foom (subscriber, #14868) [Link]
Ummm. So I have this mail program. Apple Mail. Guess what it does to delete messages? Flags them
as deleted. They stay in their mailboxes, hidden, unless you select "Show deleted messages" from the menu. It works great, and is quite intuitive.
There is an option "Move deleted messages to Trash mailbox" if you want to copy Outlook-style
But neither of those options require that you download the entire header list just to be notified of
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 12:27 UTC (Tue) by niner (subscriber, #26151) [Link]
> the thing is that the result (how many unread messages are there) can be
found very efficiently through the IMAP protocol (if nothing else, search for all messages with the 'new' flag and count them, although I suspect that there are even better ways)
It's even much simpler: the number of unseen messages is part of the
Regarding the deleted-problem: just move the messages to a trash folder or
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 17:30 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]
you don't want to expunge immediatly after delete
for one thing expunge is frequently a fairly expensive (i.e. slow) command on the server
for another, it's not uncommon for IMAP users to have multiple clients open to the same mail server, auto-expunge can cause surprises for the user (especially if the delete is done as part of a filter of some sort with the user actually looking at another client)
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 8:14 UTC (Tue) by eru (subscriber, #2753) [Link] If I want to move a large amount of mail around, I'll usually use rsync or scp. Maildir makes that nice and easy.Not an option for most users of a graphical mail clients: The data typically is somewhere on some server, managed by an ISP or corporate IT support, that cannot be accessed by normal users, except by IMAP or some other mail protocol. And a less-sophisticated user will want to do the task with his familiar mail client, not some "obscure" command line tool. So I think the test is a quite realistic use case, but obviously not the only performance measure (and the original article admits this).
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 10:30 UTC (Tue) by dwmw2 (subscriber, #2063) [Link] I also gave an option for the case where I was forced to use IMAP. But mostly, I ensure that I'm not forced to use IMAP. I want my mail folders to be greppable. That's what fetchmail/getmail is for.You're right that the naïve user will want to do it with his familiar mail client — in which case the benchmark isn't particularly relevant either. Unless you're suggesting that said user (or his sysadmin) should choose an MUA to specifically optimise for this use case. Rather than the things that the user will be doing every day? That's why I mentioned the 'check for new mail' performance of Evolution, instead of the 'start up for first time ever' performance. (Which was a number of hours when I filed the above-mentioned bug. By switching to a GigE connection all the way to the mail server, I managed to reduce it to closer to half an hour.) Going back to the original test, was it confirmed that the messages were copied intact to the new store? Evolution used to corrupt whitespace when it did that. Still does, AFAICT. Here's (part of) an original...
X-Spam-Report: SpamAssassin version 3.2.5 on casper.infradead.org summary:
Content analysis details: (4.6 points, 5.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
2.0 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in bl.spamcop.net
[Blocked - see <http://www.spamcop.net/bl.shtml?81.91.235.239>]
0.0 HTML_MESSAGE BODY: HTML included in message
0.0 BAYES_50 BODY: Bayesian spam probability is 40 to 60%
[score: 0.5000]
1.4 MIME_QP_LONG_LINE RAW: Quoted-printable line longer than 76 chars
1.2 ADVANCE_FEE_2 Appears to be advance fee fraud (Nigerian 419)
And here's what it looks like after I copy it from one mailstore to another with Evolution...
X-Spam-Report: SpamAssassin version 3.2.5 on casper.infradead.org summary:
Content analysis details: (4.6 points, 5.0 required) pts rule name
description ---- ----------------------
-------------------------------------------------- 2.0
RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in bl.spamcop.net [Blocked
- see <http://www.spamcop.net/bl.shtml?81.91.235.239>] 0.0 HTML_MESSAGE
BODY: HTML included in message 0.0 BAYES_50 BODY:
Bayesian spam probability is 40 to 60% [score: 0.5000] 1.4
MIME_QP_LONG_LINE RAW: Quoted-printable line longer than 76 chars 1.2
ADVANCE_FEE_2 Appears to be advance fee fraud (Nigerian 419)
Can you read the Evolution-mangled version? I can't.
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 12:08 UTC (Tue) by eru (subscriber, #2753) [Link] Unless you're suggesting that said user (or his sysadmin) should choose an MUA to specifically optimise for this use case. Rather than the things that the user will be doing every day?I agree the user will not do this often, but when he needs to do it, the operation should work correctly and in a finite time. According to the original article, this mass of mails caused all but Evolution and Claws to fail to work. Can you read the Evolution-mangled version? I can't. Actually, I haven't touched Evolution, after it did worse mangling to my mails years ago...
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 14:12 UTC (Tue) by nix (subscriber, #2304) [Link]
Good grief. Are they *trying* to make every catastrophic mistake Outlook
made, only make it worse?
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 18:41 UTC (Mon) by pj (subscriber, #4506) [Link]
Some of this is why I wrote MHI (http://code.google.com/p/mhi/) ; it's about as lightweight as you can get, though it needs some work. Contributions welcome!
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 20:11 UTC (Mon) by maks (subscriber, #32426) [Link]
/me misses mutt measurement in aboves list :)
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 20:24 UTC (Mon) by armijn (subscriber, #3653) [Link]
That's because it was a test of graphical mail clients. I am planning on testing other clients as well in the next few months.
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 22:00 UTC (Mon) by lbt (subscriber, #29672) [Link]
Useful - thanks.
A few years back I wondered how cyrus/thunderbird would cope with "quite a few" emails - you know... just to see.
I was subscribed to LKML and a few other lists... so I let them accumulate
Total email count in 1 imap/thunderbird acc last week : 873,473
Search is quick although flicking between folders sometimes takes a second or 5.
Up until the latest thunderbird/icedove 2.0.22 things were fine; sadly I purged them down to 95k a couple of days ago as the newest version seemed to leak and grew to about 1.4Gb/800Mb resident and takes minutes to start. Purging and compacting didn't help though reverting to the lenny version did make a slight difference.
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 23:26 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
you should also check mulberry ( http://mulberrymail.com/ )
it is now opensource (although getting it to compile on linux is currently black-magic, blocking development there)
while it has some headaches (and in my opinion needs a chainsaw taken to it to reorginize much of it's GUI), it is a solid IMAP client
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 8:08 UTC (Tue) by BlueLightning (subscriber, #38978) [Link]
Mulberry may be great I'm sure; but I ask you, what kind of GUI application has a website without a single screenshot?!
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 9:20 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]
on that is opensource without much of a community (unfortunantly)
it used to be a commercial product (with a very good reputation as I understand it), the developer sold it to a company that let it die, a couple of years ago he bought back the rights to the code and released it.
unfortunately he primarily develops on windows and the mac and linux versions were forks, not ports and he used some 'interesting' proprietary tools to compile it. It is slowly getting cleaned up (I understand the forks have been merged at this point), but nobody has documented how to do a compile from a clean linux box yet (it can be done if there is the leftover garbage from prior compiles with the proprietary tools around) this puts a significant crimp in the development community around the product. the windows and OSX ports do not have this compile problem, just the linux port.
one of the annoying problems with it in linux is that it's possible to have it get confused about the size of the fonts it is using, and not allocate enough screen space for the text it is displaying (at which point the 'extra' text gets cut off) the suspicion is that this is caused by using a different dpi display than the developer, but without the ability to easily recompile it, it's not easy to troubleshoot.
but that being said (and the above problems are significant) it does work pretty well at efficiantly handling IMAP mail.
the developers of these other programs would do well to learn from mulberry
and if a experienced linux developer or two could get mulberry unstuck it could then learn from the advances in GUIs over the last 5 years or so and get prettied up enough to give these programs a real challenge.
Stresstesting IMAP clients (Hackvalue) Posted Oct 5, 2009 23:43 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
doing a test of 'downloading 12000 e-mails' is not really relavent for a good IMAP implementataion.
you won't do that, you will view the e-mail headers, read some (or all) of the messages, but unless you are trying to support offline mode you won't download them all.
the fact that many/most of these clients _do_ download all of the messages when you first connect is an indication that they are broken by design
moving messages to a different account is also a fairly unusual event.
better tests would be
how long does it take to connect to the server for the first time after the 12000 new messages appear
move the messages to a different folder on the same server
sort the messages
thread the messages
add some new messages and thread the messages again.
how long to search for a particular subject, from address, etc
can you tag an entire thread for action
note that for all of these tests to be valid, you need to make sure that your IMAP server supports all the features, I don't know if dovecot supports all of these.
I use cyrus as my mail server (running on XFS) with pine and mulberry as my clients and they easily handle accounts with hundreds of folders with some folders having 60K+ messages
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 9:06 UTC (Tue) by kruemelmo (subscriber, #8279) [Link]
mod parent up!
Having said that... I always thought thunderbird downloads all mail to perform its spam filter check. It seems strange that it *still* needs the server connection to display the mails.
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 9:42 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]
that may be an additional reason, but it was downloading the entire mail evenbefore it gained spam filters and does so even if spam filters are disabled (say you have server-level spam filters)
some/many IMAP servers have very extensive mail filtering capabilities (including the ability to have per-user filter rules on the server)
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 5:23 UTC (Tue) by SimonKagstrom (subscriber, #49801) [Link]
Nice to see the good results for Claws mail. I have to disagree with the authors dislike of its "ugly" UI though, I think it's simple and functional.
I was also very impressed when I realised that you could select Mew/Wanderlust-style keyboard shortcuts, which made transition from Wanderlust easier than I would have expected. At least for me, that counts as part of an excellent UI!
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 13:53 UTC (Tue) by drag (subscriber, #31333) [Link]
Man, I've tried using Clawsmail several times in the past, but all it ends
up doing is making me want to claw my eyes out. And it does not have a simple UI.. it has a horribly over-engineered UI.
To me mutt is far easier to use, even though I tried claws first... I am
* Need a email client that does not treat html email as spawn of satan.
* Need something that works properly with imap
* Need something that works properly with mapi.
* Need something that doesn't crash all the time.
* Need something that has good performance.
* Need something that works properly with encryption.
So far Evolution is the best GUI that I have found. Balsa email seems to be
Evolution is probably the best thing going forward really. Especially as
Really.. unless you can view corporate document folders, schedual a
But for the time being I'll keep farting around with mutt.
Stresstesting IMAP clients (Hackvalue) Posted Oct 6, 2009 14:06 UTC (Tue) by SimonKagstrom (subscriber, #49801) [Link]
Well to each their own. To me the UI is simple (and really not that different from other GUI mailers). I appreciate configurability (keyboard shortcuts) and that it generally stays out of my way in formatting mails etc (patches is no problem for example). Filtering is also very simple to use from the GUI.
I view HTML mails without problems in claws mail by the way, but agree that calendar stuff is a bit lacking. It's not a problem in practice for me though, but I understand that other people might be better off with evolution.
Earlier versions were a bit unstable, but since the last couple of versions I've not had problems with crashes.
Deletion times Posted Oct 7, 2009 13:44 UTC (Wed) by rich0 (guest, #55509) [Link]
Thunderbird drives me nuts when I try to delete email. It can take 30s or
more to delete a message. Once I've deleted a message then further deletes in the same session go fairly quickly. Other mail clients access in the same IMAP store don't have this issue, so I doubt it is a server issue.
I suspect that Thunderbird is keeping a local cache of the mail spool -
The thing is that deletes should be completely asynchronous. Why should
Unfortunately I don't know of many other threaded mail clients that cover
|
|||||||||||||