LWN: Comments on "The notmuch mail client"

The notmuch mail client

xoddam — Wed, 25 Nov 2009 23:12:25 +0000

> we've run into a lot of problems with heavily multithreaded access to sqlite.

Try delegating writes to a dedicated thread (per db handle) and allowing only read access from other threads.

Language wars

Baylink — Tue, 24 Nov 2009 20:35:31 +0000

I call your attention to the Go! wars, in another thread.

Interpreted and compiled languages

rwmj — Sun, 22 Nov 2009 23:56:13 +0000

Python is a really poor example. It's slow and uses loads of memory.

Look at a real, static, compiled language with type inference. The current leaders in this area are all
functional languages. FWIW I started writing something similar to notmuch/sup in OCaml (a few
months ago, and without knowing about sup -- I'll probably abandon it for something better
supported now like notmuch).

Rich.

The notmuch mail client

endecotp — Thu, 19 Nov 2009 23:27:57 +0000

I was subscribed to an IMAP list at one time but didn't discuss this much; my opinion of IMAP is rather low but I'm too polite to jump into someone else's list and tell them what I really think...

There are certainly mismatches between IMAP and SQL. For example, anything that implements virtual folders on the server will have to try hard to make the IMAP message sequence numbers work right. To be honest, I have been using Decimail for about five years and it "just works" for me, and I have now forgotten (or blocked the memories!) of the implementation challenges.

Phil.

The notmuch mail client

krake — Thu, 19 Nov 2009 12:18:49 +0000

It is somewhat similar but then also quite different.

Akonadi's server access protocol is similar to IMAP in terms of syntax and
grammar. Contents of commands and responses can be different and there are
additional commands for things usually not found in IMAP.

As far as I can tell there is also a different usage pattern for the
database, i.e. permanent storage vs. transient storage (Akonadi uses its
database mainly for caching, potentially only of parts of items, e.g.
envelope of mails).

The notmuch mail client

lysse — Thu, 19 Nov 2009 05:54:34 +0000

The arrogance of rank ignorance is petegn's special gift to the world.

The notmuch mail client

bojan — Wed, 18 Nov 2009 23:23:57 +0000

> You don't like Thunderbird?

No support for Exchange. Cannot edit LDAP address books, AFAIK.

The notmuch mail client

aleXXX — Wed, 18 Nov 2009 22:46:35 +0000

Is that the same principle as what Akonadi does ?
They use also a database and IMAP is also involved AFAIK...

Alex

The notmuch mail system

aleXXX — Wed, 18 Nov 2009 22:45:00 +0000

And I thought I have a lot of mails on my system...

I checked, and currently I have around 85000 mails here, with the biggest
folder containing 17000 mails.
Until now there are absolutely no performance issues when using kmail for
this amount of mails.

Alex

The notmuch mail client

raf — Wed, 18 Nov 2009 22:22:12 +0000

I'm doing something similar, using PostgreSQL at the back end, PXTL for a web-based front end, and Twisted's IMAP4 component as the basis of the IMAP server.

I wanted it in PostgreSQL because I want to link it up with my GnuCash data (whose beta versions support PostgreSQL again) and a personal wiki (currently MediaWiki).

Anyway, nothing worth releasing yet.

Smoke test

man_ls — Wed, 18 Nov 2009 21:49:04 +0000

To get the best of both worlds you just run an integrated test suite instead of or in addition to compiling. This way you can catch both type errors and other functional errors. It's not that hard to do, really, and it often takes less than compiling a lot of code. Just be sure to make it run real fast.

The notmuch mail client

jonabbey — Wed, 18 Nov 2009 21:46:38 +0000

Very neat! Did you ever talk with other IMAP server implementers about your experience with using a SQL back-end?

I had gathered from various comp.mail.imap discussions over the years that certain features of IMAP were rather difficult to do efficiently with a relational database, though I never went about implementing anything server-side with IMAP.

Jack of many trades

man_ls — Wed, 18 Nov 2009 19:00:35 +0000

I use it at work, and "happy" is not the word. "Constantly bothered by it, but not sufficiently annoyed to actively fight it" is more like it. And the bloody calendar is not that bad if you have, say, 10--20 meetings a week. Yeah, my life sucks :(

Many people fail to understand the fundamental truth behind your wise statement: if your software doesn't do X, corporate buyers will not even consider it. It can be badly done, not scale and generally be poorly engineered, but do whatever they want it to do and suddenly you are eligible. Sounds reasonable, right? The vast majority of people are not able to see beyond the "what" and into the "how".

The notmuch mail client

wstephenson — Wed, 18 Nov 2009 17:25:42 +0000

That's just a packaging mistake then. What are these debian cowboys like? </joke>.

Contrary to what you've heard Akonadi doesn't require a configured mysql server running on the machine. It spawns a per-session instance of mysql running a custom config that is tweaked, minimal and secure. We started with mysql-embedded but after running into weird problems with it, went for the standalone process - it doesn't create significantly more overhead and we think we have the process management sorted.

You do still have to have an installed mysql though. Postgresql and sqlite support are being worked on, but we've run into a lot of problems with heavily multithreaded access to sqlite.

The notmuch mail client

jond — Wed, 18 Nov 2009 16:57:55 +0000

Thanks for pointing this out. I'm also going to have to have a proper look
:)

The notmuch mail client

jond — Wed, 18 Nov 2009 16:42:50 +0000

I think I've heard of this akonadi. That's the thing that isn't actually a
DB, but backends to one, right? Iirc, it's the cause of some ire in Debian
because the package depends on the mysql-server package, meaning any KDE
desktop user needs an installed and configured mysql server on their
machine.

Anyway, I think xapian is more fit-for-purpose for this problem than a
generic relational database.

The notmuch mail client

nye — Wed, 18 Nov 2009 14:19:52 +0000

This look awesome.

I hope to find time to look at it properly at some point, thanks.

The notmuch mail client

lab — Wed, 18 Nov 2009 13:14:58 +0000

"Evolution is the only one that has the features that typical users expect nowadays"

You don't like Thunderbird?

The notmuch mail client

halla — Wed, 18 Nov 2009 13:11:32 +0000

I'm getting a few thousand mails a day in three accounts -- about half of
them spam, the rest ordinary mail and a lot of mailing list mails. KMail
handles that just fine, including the spam filtering.

But I'm using pop: not sure whether imap would would work as well. My
archives aren't that big though, I throw away a lot of mailing list mail
after having skimmed it, and my archives don't go back to 1993 anymore
after a stupid crash.

And I'm going to try mailody soon, which uses Akonadi and is designed for
handling large amounts of mail over imap.

The notmuch mail client

endecotp — Wed, 18 Nov 2009 12:20:47 +0000

Nice to see that there is innovation going on in email, still.

A few years ago I had some spare time and decided to "scratch an itch" by implementing my own "ideal" email system. My requirements were:

- Accessible via a web client, at least as a secondary interface.
- No "lock in" to a particular front end.
- Fast search and message categorisation.

I ended up by putting all of the messages in a PostgreSQL database, with full text indexing, and then writing an IMAP interface to it. I have "virtual folders" that are implemented by SQL queries, with new ones created in response to IMAP actions. So in my IMAP client (any IMAP client) I can create a folder called "Keywords/Beer" and it immediately contains all messages containing that keyword, or I can create a folder called "People/Fred" and drag one message from Fred into it, and it immediately contains all messages from (and maybe to) Fred.

It works reasonably well with ~100,000 messages. The weak part is IMAP, which I found to be a rather nasty protocol. It is GPLed and available at http://decimail.org/ - however, it has never had much external interest and has suffered from bit rot, so I don't really encourage anyone to try to use it unless very keen.

Phil.

The notmuch mail client

nye — Wed, 18 Nov 2009 11:26:54 +0000

I would normally expect 'typical users' to mean 'home desktop users', in which case MAPI support is irrelevant - who runs their own Exchange server?

I presume you are thinking rather about the sort of features that might encourage large-scale corporate deployment?

I kind of think this is leaving the realm of mail clients though. MAPI clients (without using nasty hacks, are there any other than Outlook?) aren't really the same thing as mail clients to my mind.

While Outlook is *technically* a mail client, it's about the worst you're ever likely to find (with the exception of Evolution once you start paying attention to things like performance and stability), but it's not really meant to be its function, per se, just one of its features. (It's interesting that Outlook does everything badly - but it does several things all together, which makes people happy)

The notmuch mail client

cworth — Wed, 18 Nov 2009 10:03:39 +0000

Thanks for the idea.

I've intentionally put the most interesting pieces of notmuch into
what I think is a clean and well-documented library interface. I'm
hoping that people will take the code and integrate it into lots of
things I haven't imagined yet. (And integration with dovecot is
definitely something I hadn't imagined.)

Things that I have imagined are a web-based client, and using notmuch
to get good archives of mailing lists. (The fact that mailman's
default archiver, pipermail breaks threads at arbitrary boundaries
likes months has driven me crazy for years.)

-Carl

The notmuch mail system

cworth — Wed, 18 Nov 2009 09:59:45 +0000

Hi Alex,

You've got Keith's take on our project so far. Let me try to add a bit
of my own.

First, I like to call this project the "notmuch mail system" rather
than the "notmuch mail client". Our emacs client is just one (tiny)
piece of the system and I know that not everybody will want that,
(probably most people won't).

But a more interesting thing that Notmuch has is a really nice, clean
C library interface to all of the interesting pieces, (mail indexing,
fast search, tagging, etc.), thanks to Xapian of course.

So I would be quite happy if the developers of existing graphical
email clients took a look at Notmuch from the point of view of
integrating with the library for search. Personally, I've got about
600 thousand emails in my collection and I've yet to find a graphical
client which can search even a small fraction of that very quickly.

But with Notmuch, even the largest searches can return results almost
instantly, and that's really nice.

Then again, if nobody uses it but me, then I'll still be happy. :-)

-Carl

The notmuch mail client

jospoortvliet — Wed, 18 Nov 2009 09:57:47 +0000

Makes sense, you need a database for that many emails I suppose. Something like Akonadi I would say :D

Would certainly be an option for the notmuch mail client...

The notmuch mail client

dmk — Wed, 18 Nov 2009 09:12:09 +0000

hm... make that 0-100msgs probably... because at that point thunderbird starts to hurt already pretty much...

The notmuch mail client

dmk — Wed, 18 Nov 2009 09:10:05 +0000

to adress the 'kmail and thunderbird' comment:

there's a difference between using mail with 0-400 msgs per day and using mail with >400 msgs per day.

the former is the category where the mainstream emailagents are ok. but for larger number of mails there are are performance problems and mail-handling-inefficencies that criple one's ability to handle mail fast. (these inefficiencies are mostly by design and not bugs)

i for one use claws-mail, but am in the <400msgs per day category.

cheers,
dmk

p.s.: and plz! don't mention outlook, because most people here don't even know what it looks like.

Interpreted and compiled languages

mjthayer — Wed, 18 Nov 2009 08:47:57 +0000

> My other complaint about Ruby/python/perl/misc is fairly simple -- I expect the computer to detect and complain about my programming errors as soon as possible. Static typing errors can be detected during compilation and need not wait until execution time.

This is probably a really silly thought, but more often than not, Python code is compiled (into pycode, but still compiled). Sometimes on the fly, usually in advance for distributed packages. I can picture Python scripts containing additional type information for variables, perhaps as specially formatted comments or docstrings, and the compiler printing warnings at compile time, lint-style, when something looked fishy - perhaps not for on-the-fly compiles, but only for advance compiles, or then again perhaps for all.

I don't know if Python has any other special issues (apart from the white space thing and the global namespace).

The notmuch mail client

magfr — Wed, 18 Nov 2009 06:47:09 +0000

You mean like this?

The notmuch mail client

drag — Wed, 18 Nov 2009 02:52:21 +0000

Oh, and thanks for bring up seive. That's something I've been wanting for a
while now.

The notmuch mail client

Darkmere — Wed, 18 Nov 2009 02:40:18 +0000

1.45 Million emails at least cause it to churn a bit. And that amount isn't really usable on a normal filesystem with maildir unless you start using year/month/ sorting for various lists.

The notmuch mail client

roc — Wed, 18 Nov 2009 02:32:57 +0000

Define "sufficiently large"?

The notmuch mail client

drag — Wed, 18 Nov 2009 02:23:02 +0000

> I'm not sure what you include as "features typical users expect nowadays",
> but I find Kmail to be quite featureful, and there certainly others.

Well MAPI support would be a big one.

The notmuch mail client

rsidd — Wed, 18 Nov 2009 02:18:46 +0000

Uh... that's Keith Packard you're talking about. Look him up.

The notmuch mail client

nix — Wed, 18 Nov 2009 01:09:09 +0000

kmail and thunderbird both choke when faced with sufficiently large
amounts of email. xapian doesn't. (Oh, and 'why emacs' is simple: if you
use it as your text editor anyway, it makes for a bloody good environment
to implement the whole UI in. This is not an isolated belief: rmail
(historical interest only), VM, Mew, and Gnus attest that this is an itch
that a lot of people like to scratch. notmuch looks seriously nice to me,
because the one thing that's clumsy and slow in Gnus is searching.)

The notmuch mail client

petegn — Tue, 17 Nov 2009 23:28:32 +0000

Yes Kmail is perfect i have used it for ages with no problems at all dunno about this Imap stuff never tried it dont need it , Then there is what is it Evolution on Gnome donu use gnome so not too sure .

I think what people are actually trying to say without actually saying as much is they only know about outlook and cant find outlook (scuse me whilst i barf ) for Linux boy am i glad of that Outlook Distress is just that Distressing..

Language wars

NAR — Tue, 17 Nov 2009 22:34:07 +0000

I expect the computer to detect and complain about my programming errors as soon as possible. Static typing errors can be detected during compilation and need not wait until execution time.

Don't forget that you have to execute the code anyway to test it. Now the real question is that which is faster: compile (and compile and compile...) C/C++/Java source and execute, or compile a perl/python/ruby/erlang/etc. source, execute, compile, execute, etc. It depends on the problem domain, but the later could be faster in the long run.

Type errors are probably found earlier with C (although the automatic numeric conversions could be tricky), but program logic errors might be found later. I work with a dynamically typed language and sometimes miss the compiler help with types - but certainly don't miss the "type make, then go for a coffee" days of C++ programming.

And I haven't even mentioned buffer overflows plaguing C programs since forever, where the compiler won't help much.

Language wars

epa — Tue, 17 Nov 2009 22:06:31 +0000

Thanks for the explanation. I agree with your point about dynamically typed languages; I would also like the compiler to do more work for me, and I enjoy hacking things in languages like Haskell (and occasionally C# or even C++) and getting that solid 'clunk' sound when all the code compiles successfully into an object file with no possibility of runtime type errors. And yet my preferred language for hacking something up in a hurry remains perl.

The notmuch mail client

aleXXX — Tue, 17 Nov 2009 19:38:52 +0000

Hmm, what's wrong with kmail or Thunderbird ?
And they fix that (?) by introducing an email client for use in
emacs ????

Alex

linux GUI IMAP client

dlang — Tue, 17 Nov 2009 18:55:58 +0000

take a look at mulberry http://mulberrymail.com/

it does a good job of properly using IMAP (including sieve for server-side filtering), although most of the code is old so it may not have all of 'the features that people expect today'

it was released as opensource a year or so ago, but compiling it has been a horrible pain so very little progress has been made in that time. however in the last week or two there have been a lot of commits to SVN that make it much easier to compile, so I expect a lot more changes to start happening now.

The notmuch mail client

rfunk — Tue, 17 Nov 2009 18:21:40 +0000

Sieve is supposed to make it easier for clients to mess with server-side
filters, but unfortunately support for it isn't quite universal.
http://sieve.info/

I'm not sure what you include as "features typical users expect nowadays",
but I find Kmail to be quite featureful, and there certainly others.