The notmuch mail client [LWN.net]

The notmuch mail client

Posted Nov 17, 2009 15:01 UTC (Tue) by johill (subscriber, #25196) [Link] (16 responses)

They need to combine forces with Timo Sirainen (of dovecot fame) and actually make this a server-side feature.

The notmuch mail client

Posted Nov 17, 2009 15:45 UTC (Tue) by ptman (subscriber, #57271) [Link] (14 responses)

I'd really like for mail to be like dvcs: fast and local with syncing. While
I don't want to say no to server-side xapian search, I believe it has to be
possible without a connection.

The notmuch mail client

Posted Nov 17, 2009 16:44 UTC (Tue) by rfunk (subscriber, #4054) [Link] (13 responses)

Theoretically, disconnected IMAP is what you want, e.g. with OfflineIMAP.

I haven't tried it though.

The notmuch mail client

Posted Nov 17, 2009 17:40 UTC (Tue) by drag (guest, #31333) [Link] (12 responses)

I used to buy into the superiority of IMAP as a given, but lately I've decided that downloading the email and processing it on the client-side is actually a good way to do things. Just because it's a pain to have to log into a remote system to do fairly mundane things combined with the lack of a really good GUI Imap client for Linux.

But sync issues is what makes IMAP worth it.. (as in lack of good sync for the desktop)

The notmuch mail client

Posted Nov 17, 2009 17:58 UTC (Tue) by rfunk (subscriber, #4054) [Link] (10 responses)

Um, nothing about IMAP prevents client-side processing. In fact that's
exactly the use I was talking about.
http://software.complete.org/software/projects/show/offli...

Not to mention that fetchmail has explicitly recommended IMAP over POP
forever, and it's inherently about downloading the email and processing it
locally.

Hooray for offlineimap

Posted Nov 17, 2009 18:06 UTC (Tue) by dmarti (subscriber, #11625) [Link]

I use offlineimap, mairix, and mutt, but just pulled notmuch and will try that too. Offlineimap works great for the sync up part -- I'd rather do an explicit sync and move stuff around in local folders in between syncs.

(Thinking about re-running spam filters at IMAP time, to handle newly-caught bad Received: IP addresses.)

The notmuch mail client

Posted Nov 17, 2009 18:12 UTC (Tue) by drag (guest, #31333) [Link] (8 responses)

Yes, but when I put lots of effort into making custom rules and sorting
stuff then that does not get translated to any other client.

If I want to make, say, a mailing list filter I have to do it on the server
if I want the changes to automatically propagate to other clients. Otherwise
it starts getting messy.. some clients will properly update the local
folders and on the server, while others will not.. all depending on what I
happen to be using at the time and whether not I remember or have enough
time to manually copy rules around.

It's all a PITA and things should be better then this.

Not to also mention that email clients on "Desktop Linux" suck. Evolution is
the only one that has the features that typical users expect nowadays, but
it's a not programmed well.

The notmuch mail client

Posted Nov 17, 2009 18:21 UTC (Tue) by rfunk (subscriber, #4054) [Link] (5 responses)

Sieve is supposed to make it easier for clients to mess with server-side
filters, but unfortunately support for it isn't quite universal.
http://sieve.info/

I'm not sure what you include as "features typical users expect nowadays",
but I find Kmail to be quite featureful, and there certainly others.

The notmuch mail client

Posted Nov 18, 2009 2:23 UTC (Wed) by drag (guest, #31333) [Link] (3 responses)

> I'm not sure what you include as "features typical users expect nowadays",
> but I find Kmail to be quite featureful, and there certainly others.

Well MAPI support would be a big one.

The notmuch mail client

Posted Nov 18, 2009 6:47 UTC (Wed) by magfr (subscriber, #16052) [Link]

You mean like this?

The notmuch mail client

Posted Nov 18, 2009 11:26 UTC (Wed) by nye (subscriber, #51576) [Link] (1 responses)

I would normally expect 'typical users' to mean 'home desktop users', in which case MAPI support is irrelevant - who runs their own Exchange server?

I presume you are thinking rather about the sort of features that might encourage large-scale corporate deployment?

I kind of think this is leaving the realm of mail clients though. MAPI clients (without using nasty hacks, are there any other than Outlook?) aren't really the same thing as mail clients to my mind.

While Outlook is *technically* a mail client, it's about the worst you're ever likely to find (with the exception of Evolution once you start paying attention to things like performance and stability), but it's not really meant to be its function, per se, just one of its features. (It's interesting that Outlook does everything badly - but it does several things all together, which makes people happy)

Jack of many trades

Posted Nov 18, 2009 19:00 UTC (Wed) by man_ls (guest, #15091) [Link]

I use it at work, and "happy" is not the word. "Constantly bothered by it, but not sufficiently annoyed to actively fight it" is more like it. And the bloody calendar is not that bad if you have, say, 10--20 meetings a week. Yeah, my life sucks :(

Many people fail to understand the fundamental truth behind your wise statement: if your software doesn't do X, corporate buyers will not even consider it. It can be badly done, not scale and generally be poorly engineered, but do whatever they want it to do and suddenly you are eligible. Sounds reasonable, right? The vast majority of people are not able to see beyond the "what" and into the "how".

The notmuch mail client

Posted Nov 18, 2009 2:52 UTC (Wed) by drag (guest, #31333) [Link]

Oh, and thanks for bring up seive. That's something I've been wanting for a
while now.

The notmuch mail client

Posted Nov 18, 2009 13:14 UTC (Wed) by lab (guest, #51153) [Link] (1 responses)

"Evolution is the only one that has the features that typical users expect nowadays"

You don't like Thunderbird?

The notmuch mail client

Posted Nov 18, 2009 23:23 UTC (Wed) by bojan (subscriber, #14302) [Link]

> You don't like Thunderbird?

No support for Exchange. Cannot edit LDAP address books, AFAIK.

linux GUI IMAP client

Posted Nov 17, 2009 18:55 UTC (Tue) by dlang (guest, #313) [Link]

take a look at mulberry http://mulberrymail.com/

it does a good job of properly using IMAP (including sieve for server-side filtering), although most of the code is old so it may not have all of 'the features that people expect today'

it was released as opensource a year or so ago, but compiling it has been a horrible pain so very little progress has been made in that time. however in the last week or two there have been a lot of commits to SVN that make it much easier to compile, so I expect a lot more changes to start happening now.

The notmuch mail client

Posted Nov 18, 2009 10:03 UTC (Wed) by cworth (subscriber, #27653) [Link]

Thanks for the idea.

I've intentionally put the most interesting pieces of notmuch into
what I think is a clean and well-documented library interface. I'm
hoping that people will take the code and integrate it into lots of
things I haven't imagined yet. (And integration with dovecot is
definitely something I hadn't imagined.)

Things that I have imagined are a web-based client, and using notmuch
to get good archives of mailing lists. (The fact that mailman's
default archiver, pipermail breaks threads at arbitrary boundaries
likes months has driven me crazy for years.)

-Carl

The notmuch mail client

Posted Nov 17, 2009 15:38 UTC (Tue) by rfunk (subscriber, #4054) [Link] (10 responses)

Which doesn't change the state of GUI mail applications one bit.

Interesting that this is mostly a rewrite of Sup in C (rather than Ruby),
except with no UI. I'd be quite amused if Sup just adopted notmuch as its
back-end.

The notmuch mail client

Posted Nov 17, 2009 16:21 UTC (Tue) by epa (subscriber, #39769) [Link] (8 responses)

Some of the reasons they give are good, but some are mere language bigotry. Twice they claim that program X is good at what it does, but is obviously flawed because it is written in language L - without even a nod to the possibility that writing it in L might be one reason why it was possible to produce such good software.

'waiting for ruby is not on my list of desired activities' is a reasonable sentiment, but they haven't shown any examples where sup is too slow; since the heavy lifting of search and database storage is done with C and C++ libraries, performance of the thin Ruby layer on top is unlikely to be a problem. So why the prejudice?

The notmuch mail client

Posted Nov 17, 2009 16:39 UTC (Tue) by rfunk (subscriber, #4054) [Link]

Yeah, the language bigotry bothered me too. You've stated it well.

But I mostly left C and its memory-management issues behind years ago, and
have preferred Ruby for the past few years, so I could be accused of my own
language bigotry.

Language wars

Posted Nov 17, 2009 18:16 UTC (Tue) by keithp (subscriber, #5140) [Link] (6 responses)

The big performance issue is in adding new messages to the database; Xapian isn't terribly zippy, but I was able to average about 80 messages/sec for 700000 messages using notmuch while sup managed only about 20; profiling showed that most of that delay was in running the ruby message parsing code.

The Ruby UI is also sluggish when presented with a long list of messages or a large message to display; sitting there for seconds much of the time.

For me, it was the Apple Newton that demonstrated why I never want to write any 'mission critical' applications in an interpretive language. The Newton shipped with most of the applications running in an interpreter. I'm sure it was easier to develop them as a result, but the end product was mocked in large part due to poor performance.

My other complaint about Ruby/python/perl/misc is fairly simple -- I expect the computer to detect and complain about my programming errors as soon as possible. Static typing errors can be detected during compilation and need not wait until execution time. Any language which passes the burden of this mechanical process on to the programmer is one where programs will have numerous latent bugs, and sup is no exception -- it crashed many times each day with various type errors. Yes, I know, dynamic typing is all the rage; I guess I'm just old fashioned.

In any case, I program in a lot of different languages as I believe that one should choose the tool appropriate to the job at hand. It may be that sup was best written in Ruby, I don't frankly know as notmuch is *not* a sup clone -- the bulk of sup is the UI, which is really quite good and notmuch tries to present a clone of that through emacs while re-implementing the Xapian glue code in a much faster (and type safe) language.

Language wars

Posted Nov 17, 2009 22:06 UTC (Tue) by epa (subscriber, #39769) [Link] (1 responses)

Thanks for the explanation. I agree with your point about dynamically typed languages; I would also like the compiler to do more work for me, and I enjoy hacking things in languages like Haskell (and occasionally C# or even C++) and getting that solid 'clunk' sound when all the code compiles successfully into an object file with no possibility of runtime type errors. And yet my preferred language for hacking something up in a hurry remains perl.

Language wars

Posted Nov 24, 2009 20:35 UTC (Tue) by Baylink (guest, #755) [Link]

I call your attention to the Go! wars, in another thread.

Language wars

Posted Nov 17, 2009 22:34 UTC (Tue) by NAR (subscriber, #1313) [Link] (1 responses)

I expect the computer to detect and complain about my programming errors as soon as possible. Static typing errors can be detected during compilation and need not wait until execution time.

Don't forget that you have to execute the code anyway to test it. Now the real question is that which is faster: compile (and compile and compile...) C/C++/Java source and execute, or compile a perl/python/ruby/erlang/etc. source, execute, compile, execute, etc. It depends on the problem domain, but the later could be faster in the long run.

Type errors are probably found earlier with C (although the automatic numeric conversions could be tricky), but program logic errors might be found later. I work with a dynamically typed language and sometimes miss the compiler help with types - but certainly don't miss the "type make, then go for a coffee" days of C++ programming.

And I haven't even mentioned buffer overflows plaguing C programs since forever, where the compiler won't help much.

Smoke test

Posted Nov 18, 2009 21:49 UTC (Wed) by man_ls (guest, #15091) [Link]

To get the best of both worlds you just run an integrated test suite instead of or in addition to compiling. This way you can catch both type errors and other functional errors. It's not that hard to do, really, and it often takes less than compiling a lot of code. Just be sure to make it run real fast.

Interpreted and compiled languages

Posted Nov 18, 2009 8:47 UTC (Wed) by mjthayer (guest, #39183) [Link] (1 responses)

> My other complaint about Ruby/python/perl/misc is fairly simple -- I expect the computer to detect and complain about my programming errors as soon as possible. Static typing errors can be detected during compilation and need not wait until execution time.

This is probably a really silly thought, but more often than not, Python code is compiled (into pycode, but still compiled). Sometimes on the fly, usually in advance for distributed packages. I can picture Python scripts containing additional type information for variables, perhaps as specially formatted comments or docstrings, and the compiler printing warnings at compile time, lint-style, when something looked fishy - perhaps not for on-the-fly compiles, but only for advance compiles, or then again perhaps for all.

I don't know if Python has any other special issues (apart from the white space thing and the global namespace).

Interpreted and compiled languages

Posted Nov 22, 2009 23:56 UTC (Sun) by rwmj (subscriber, #5474) [Link]

Python is a really poor example. It's slow and uses loads of memory.

Look at a real, static, compiled language with type inference. The current leaders in this area are all
functional languages. FWIW I started writing something similar to notmuch/sup in OCaml (a few
months ago, and without knowing about sup -- I'll probably abandon it for something better
supported now like notmuch).

Rich.

The notmuch mail client

Posted Nov 17, 2009 17:04 UTC (Tue) by tzafrir (subscriber, #11501) [Link]

The UI is Emacs.

The notmuch mail client

Posted Nov 17, 2009 17:31 UTC (Tue) by dlang (guest, #313) [Link]

if you use a good IMAP server (like Cyrus) you can have fast text searches, arbitrary labeling of messages, but not have to have everything stored locally on each client machine that you use (so that you could access it effectively from a lightweight client on a slow network like a phone)

The notmuch mail client

Posted Nov 17, 2009 19:38 UTC (Tue) by aleXXX (subscriber, #2742) [Link] (15 responses)

Hmm, what's wrong with kmail or Thunderbird ?
And they fix that (?) by introducing an email client for use in
emacs ????

Alex

The notmuch mail client

Posted Nov 17, 2009 23:28 UTC (Tue) by petegn (guest, #847) [Link] (2 responses)

Yes Kmail is perfect i have used it for ages with no problems at all dunno about this Imap stuff never tried it dont need it , Then there is what is it Evolution on Gnome donu use gnome so not too sure .

I think what people are actually trying to say without actually saying as much is they only know about outlook and cant find outlook (scuse me whilst i barf ) for Linux boy am i glad of that Outlook Distress is just that Distressing..

The notmuch mail client

Posted Nov 18, 2009 2:18 UTC (Wed) by rsidd (subscriber, #2582) [Link] (1 responses)

Uh... that's Keith Packard you're talking about. Look him up.

The notmuch mail client

Posted Nov 19, 2009 5:54 UTC (Thu) by lysse (guest, #3190) [Link]

The arrogance of rank ignorance is petegn's special gift to the world.

The notmuch mail client

Posted Nov 18, 2009 1:09 UTC (Wed) by nix (subscriber, #2304) [Link] (6 responses)

kmail and thunderbird both choke when faced with sufficiently large
amounts of email. xapian doesn't. (Oh, and 'why emacs' is simple: if you
use it as your text editor anyway, it makes for a bloody good environment
to implement the whole UI in. This is not an isolated belief: rmail
(historical interest only), VM, Mew, and Gnus attest that this is an itch
that a lot of people like to scratch. notmuch looks seriously nice to me,
because the one thing that's clumsy and slow in Gnus is searching.)

The notmuch mail client

Posted Nov 18, 2009 2:32 UTC (Wed) by roc (subscriber, #30627) [Link] (5 responses)

Define "sufficiently large"?

The notmuch mail client

Posted Nov 18, 2009 2:40 UTC (Wed) by Darkmere (subscriber, #53695) [Link] (4 responses)

1.45 Million emails at least cause it to churn a bit. And that amount isn't really usable on a normal filesystem with maildir unless you start using year/month/ sorting for various lists.

The notmuch mail client

Posted Nov 18, 2009 9:57 UTC (Wed) by jospoortvliet (guest, #33164) [Link] (3 responses)

Makes sense, you need a database for that many emails I suppose. Something like Akonadi I would say :D

Would certainly be an option for the notmuch mail client...

The notmuch mail client

Posted Nov 18, 2009 16:42 UTC (Wed) by jond (subscriber, #37669) [Link] (2 responses)

I think I've heard of this akonadi. That's the thing that isn't actually a
DB, but backends to one, right? Iirc, it's the cause of some ire in Debian
because the package depends on the mysql-server package, meaning any KDE
desktop user needs an installed and configured mysql server on their
machine.

Anyway, I think xapian is more fit-for-purpose for this problem than a
generic relational database.

The notmuch mail client

Posted Nov 18, 2009 17:25 UTC (Wed) by wstephenson (guest, #14795) [Link] (1 responses)

That's just a packaging mistake then. What are these debian cowboys like? </joke>.

Contrary to what you've heard Akonadi doesn't require a configured mysql server running on the machine. It spawns a per-session instance of mysql running a custom config that is tweaked, minimal and secure. We started with mysql-embedded but after running into weird problems with it, went for the standalone process - it doesn't create significantly more overhead and we think we have the process management sorted.

You do still have to have an installed mysql though. Postgresql and sqlite support are being worked on, but we've run into a lot of problems with heavily multithreaded access to sqlite.

The notmuch mail client

Posted Nov 25, 2009 23:12 UTC (Wed) by xoddam (subscriber, #2322) [Link]

> we've run into a lot of problems with heavily multithreaded access to sqlite.

Try delegating writes to a dedicated thread (per db handle) and allowing only read access from other threads.

The notmuch mail client

Posted Nov 18, 2009 9:10 UTC (Wed) by dmk (guest, #50141) [Link] (2 responses)

to adress the 'kmail and thunderbird' comment:

there's a difference between using mail with 0-400 msgs per day and using mail with >400 msgs per day.

the former is the category where the mainstream emailagents are ok. but for larger number of mails there are are performance problems and mail-handling-inefficencies that criple one's ability to handle mail fast. (these inefficiencies are mostly by design and not bugs)

i for one use claws-mail, but am in the <400msgs per day category.

cheers,
dmk

p.s.: and plz! don't mention outlook, because most people here don't even know what it looks like.

The notmuch mail client

Posted Nov 18, 2009 9:12 UTC (Wed) by dmk (guest, #50141) [Link]

hm... make that 0-100msgs probably... because at that point thunderbird starts to hurt already pretty much...

The notmuch mail client

Posted Nov 18, 2009 13:11 UTC (Wed) by halla (subscriber, #14185) [Link]

I'm getting a few thousand mails a day in three accounts -- about half of
them spam, the rest ordinary mail and a lot of mailing list mails. KMail
handles that just fine, including the spam filtering.

But I'm using pop: not sure whether imap would would work as well. My
archives aren't that big though, I throw away a lot of mailing list mail
after having skimmed it, and my archives don't go back to 1993 anymore
after a stupid crash.

And I'm going to try mailody soon, which uses Akonadi and is designed for
handling large amounts of mail over imap.

The notmuch mail system

Posted Nov 18, 2009 9:59 UTC (Wed) by cworth (subscriber, #27653) [Link] (1 responses)

Hi Alex,

You've got Keith's take on our project so far. Let me try to add a bit
of my own.

First, I like to call this project the "notmuch mail system" rather
than the "notmuch mail client". Our emacs client is just one (tiny)
piece of the system and I know that not everybody will want that,
(probably most people won't).

But a more interesting thing that Notmuch has is a really nice, clean
C library interface to all of the interesting pieces, (mail indexing,
fast search, tagging, etc.), thanks to Xapian of course.

So I would be quite happy if the developers of existing graphical
email clients took a look at Notmuch from the point of view of
integrating with the library for search. Personally, I've got about
600 thousand emails in my collection and I've yet to find a graphical
client which can search even a small fraction of that very quickly.

But with Notmuch, even the largest searches can return results almost
instantly, and that's really nice.

Then again, if nobody uses it but me, then I'll still be happy. :-)

-Carl

The notmuch mail system

Posted Nov 18, 2009 22:45 UTC (Wed) by aleXXX (subscriber, #2742) [Link]

And I thought I have a lot of mails on my system...

I checked, and currently I have around 85000 mails here, with the biggest
folder containing 17000 mails.
Until now there are absolutely no performance issues when using kmail for
this amount of mails.

Alex

The notmuch mail client

Posted Nov 18, 2009 12:20 UTC (Wed) by endecotp (guest, #36428) [Link] (7 responses)

Nice to see that there is innovation going on in email, still.

A few years ago I had some spare time and decided to "scratch an itch" by implementing my own "ideal" email system. My requirements were:

- Accessible via a web client, at least as a secondary interface.
- No "lock in" to a particular front end.
- Fast search and message categorisation.

I ended up by putting all of the messages in a PostgreSQL database, with full text indexing, and then writing an IMAP interface to it. I have "virtual folders" that are implemented by SQL queries, with new ones created in response to IMAP actions. So in my IMAP client (any IMAP client) I can create a folder called "Keywords/Beer" and it immediately contains all messages containing that keyword, or I can create a folder called "People/Fred" and drag one message from Fred into it, and it immediately contains all messages from (and maybe to) Fred.

It works reasonably well with ~100,000 messages. The weak part is IMAP, which I found to be a rather nasty protocol. It is GPLed and available at http://decimail.org/ - however, it has never had much external interest and has suffered from bit rot, so I don't really encourage anyone to try to use it unless very keen.

Phil.

The notmuch mail client

Posted Nov 18, 2009 14:19 UTC (Wed) by nye (subscriber, #51576) [Link]

This look awesome.

I hope to find time to look at it properly at some point, thanks.

The notmuch mail client

Posted Nov 18, 2009 16:57 UTC (Wed) by jond (subscriber, #37669) [Link]

Thanks for pointing this out. I'm also going to have to have a proper look
:)

The notmuch mail client

Posted Nov 18, 2009 21:46 UTC (Wed) by jonabbey (guest, #2736) [Link] (1 responses)

Very neat! Did you ever talk with other IMAP server implementers about your experience with using a SQL back-end?

I had gathered from various comp.mail.imap discussions over the years that certain features of IMAP were rather difficult to do efficiently with a relational database, though I never went about implementing anything server-side with IMAP.

The notmuch mail client

Posted Nov 19, 2009 23:27 UTC (Thu) by endecotp (guest, #36428) [Link]

I was subscribed to an IMAP list at one time but didn't discuss this much; my opinion of IMAP is rather low but I'm too polite to jump into someone else's list and tell them what I really think...

There are certainly mismatches between IMAP and SQL. For example, anything that implements virtual folders on the server will have to try hard to make the IMAP message sequence numbers work right. To be honest, I have been using Decimail for about five years and it "just works" for me, and I have now forgotten (or blocked the memories!) of the implementation challenges.

Phil.

The notmuch mail client

Posted Nov 18, 2009 22:22 UTC (Wed) by raf (guest, #35151) [Link]

I'm doing something similar, using PostgreSQL at the back end, PXTL for a web-based front end, and Twisted's IMAP4 component as the basis of the IMAP server.

I wanted it in PostgreSQL because I want to link it up with my GnuCash data (whose beta versions support PostgreSQL again) and a personal wiki (currently MediaWiki).

Anyway, nothing worth releasing yet.

The notmuch mail client

Posted Nov 18, 2009 22:46 UTC (Wed) by aleXXX (subscriber, #2742) [Link] (1 responses)

Is that the same principle as what Akonadi does ?
They use also a database and IMAP is also involved AFAIK...

Alex

The notmuch mail client

Posted Nov 19, 2009 12:18 UTC (Thu) by krake (guest, #55996) [Link]

It is somewhat similar but then also quite different.

Akonadi's server access protocol is similar to IMAP in terms of syntax and
grammar. Contents of commands and responses can be different and there are
additional commands for things usually not found in IMAP.

As far as I can tell there is also a different usage pattern for the
database, i.e. permanent storage vs. transient storage (Akonadi uses its
database mainly for caching, potentially only of parts of items, e.g.
envelope of mails).