Stresstesting IMAP clients (Hackvalue) [LWN.net]

Stresstesting IMAP clients (Hackvalue)

Posted Oct 6, 2009 1:08 UTC (Tue) by dlang (guest, #313)
In reply to: Stresstesting IMAP clients (Hackvalue) by dwmw2
Parent article: Stresstesting IMAP clients (Hackvalue)

the thing is that the result (how many unread messages are there) can be found very efficiently through the IMAP protocol (if nothing else, search for all messages with the 'new' flag and count them, although I suspect that there are even better ways)

the behavior you are describing for evolution is a perfect example of a POP3 mail client being modified to 'support' IMAP by using the IMAP protocol to fetch everything and then processing it locally.

My dad uses pegasus mail (under wine) and mulberry, he was doing some testing of searches recently and found that pegasus (really a pop client with imap commands) can take hours to search his mailboxes. mulberry takes seconds to do the same search.

pegasus fetches all the messages (in every folder) to do the search, mulberry sends a few IMAP commands:

one or more 'search for X' commands (returns a list of messages), one per folder

thread <list of messages returned by the search commands> (returns messages in threaded order)

for each message that would fit on the screen, fetch the header info for that message.

I'll bet evolution would do the same thing as pegasus (in this case fetch a few hundred thousand messages over several hours)

to post comments

Stresstesting IMAP clients (Hackvalue)

Posted Oct 6, 2009 11:10 UTC (Tue) by kelvin (guest, #6694) [Link] (4 responses)

> if nothing else, search for all messages with the 'new' flag and count them

Evolution used to use this method, but the problem is that deleted messages might still be flagged as "new". Combined with IMAP's braindead handling of deleted messages, this method causes discrepancies in the unread-count. There were bugs reported against these discrepancies.

Deleting messages in IMAP doesn't actually delete them, they are only flagged as deleted and you then have to manually "expunge" the folder. While this might have made sense to a computer-savvy user familiar with the mbox storage model, it is quite unintuitive to any user accustomed to modern GUI's (or MS Exchange/Outlook for that matter).

One solution might be for Evolution to "break" the IMAP spec by moving all deletion-flagged messages to the Trash-folder, including messages flagged by other mail agents. This would solve the problem for Evolution, and should be fairly non-disruptive for other mail agents.

Stresstesting IMAP clients (Hackvalue)

Posted Oct 6, 2009 17:28 UTC (Tue) by dlang (guest, #313) [Link]

so change the search to be all messages with the new flag set and not the deleted flag set. it's still one IMAP command

Stresstesting IMAP clients (Hackvalue)

Posted Oct 6, 2009 22:37 UTC (Tue) by cortana (subscriber, #24596) [Link] (1 responses)

The hilarious thing is that since Evolution 2.26, these discrepancies have existed, even though Evolution does the stupid thing and treats IMAP like POP.

Day by day, I draw closer to the following conclusions:

1. IMAP is a horribly complex protocol
2. Because of 1, clients that implement the protocol correctly are non-existent
3. Because of 1, servers that implement the protocol correctly are rare

It's enough to drive one to gmail!

Stresstesting IMAP clients (Hackvalue)

Posted Oct 6, 2009 23:25 UTC (Tue) by dlang (guest, #313) [Link]

IMAP has lots of features, true

but it's not _that_ complex (web UIs are fare more complex, especially once you add AJAX), but it does require a very different mindset from pop clients to do well. the first thing that you have to accept is the idea that mailbox access is not exclusive and dedicated to one system.

once you start thinking that you may be accessing the same mailbox from your laptop, desktop, phone, etc _at the same time_ so that no one app has full control of the mailbox (the other apps may make changes that you are notified of) then you are most of the way there, and the rest of the decisions will probably be pretty good.

there are a couple of clients that work well with IMAP, and a couple of good servers. this isn't that different from other areas

how many good HTTP clients and HTTP servers are there? not that many (and on the client side, you could argue that most of the common ones in use are not among the short list of good ones ;-)

Stresstesting IMAP clients (Hackvalue)

Posted Oct 6, 2009 22:53 UTC (Tue) by foom (subscriber, #14868) [Link]

Ummm. So I have this mail program. Apple Mail. Guess what it does to delete messages? Flags them
as deleted. They stay in their mailboxes, hidden, unless you select "Show deleted messages"
from the menu. It works great, and is quite intuitive.

There is an option "Move deleted messages to Trash mailbox" if you want to copy Outlook-style
stupidity. Then it will actually move (well...copy; IMAP doesn't have a move command) the messages
to a separate Trash mailbox. A *REAL* Trash mailbox. I don't use that option, because I *like*
having my deleted messages segmented by origin mailbox (aka: the way IMAP was designed to
work), but it's there.

But neither of those options require that you download the entire header list just to be notified of
new mail. Only the method that Evolution decided upon has that requirement. And they call
it a feature! Crazy!

Stresstesting IMAP clients (Hackvalue)

Posted Oct 6, 2009 12:27 UTC (Tue) by niner (guest, #26151) [Link] (1 responses)

> the thing is that the result (how many unread messages are there) can be
found very efficiently through the IMAP protocol (if nothing else, search
for all messages with the 'new' flag and count them, although I suspect
that there are even better ways)

It's even much simpler: the number of unseen messages is part of the
response to the STATUS command.

Regarding the deleted-problem: just move the messages to a trash folder or
expunge immediately after delete. The typical GUI mail client user would
not know what the flagging/expunging is about anyway.

Stresstesting IMAP clients (Hackvalue)

Posted Oct 6, 2009 17:30 UTC (Tue) by dlang (guest, #313) [Link]

you don't want to expunge immediatly after delete

for one thing expunge is frequently a fairly expensive (i.e. slow) command on the server

for another, it's not uncommon for IMAP users to have multiple clients open to the same mail server, auto-expunge can cause surprises for the user (especially if the delete is done as part of a filter of some sort with the user actually looking at another client)