|
|
Log in / Subscribe / Register

Stephenson: hackweek9: Lightweight KDE Desktop project

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 17, 2013 11:00 UTC (Wed) by jospoortvliet (guest, #33164)
In reply to: Stephenson: hackweek9: Lightweight KDE Desktop project by anselm
Parent article: Stephenson: hackweek9: Lightweight KDE Desktop project

So this is what is going to happen if you rewrite KMail.

You create a graphical mail application. Of course you need Calendaring, too. And contact management. And Todo. So, you develop these 4 applications. Maybe someone helps out, trows in a feed reader or such. You happily add features where needed but not too much, you want to keep it slim, hey!

Of course, you need to have access to the contacts in the mail application. The mail app can directly read the contacts that the contacts app has access to, too but to avoid clashes with the contact mini-app you use DBus so they talk to each other. And you want to be able to have the contacts in the Calendar so you have appointments with people. Again, IPC to the rescue. You build on and people ask if you can show the online status in chat in the mail app - so you don't have to mail them but talk to them by mail. That makes sense. Now performance becomes a bit of an issue, so you cache the contacts in the mail app, the calendar app and well, you need todo access in the calendar app so you cache the todo's there too. Oh and todo also needs to access contacts. But let's be clear: you're not doing anything weird, this stuff needs to be simple, understandable (for the newcomers) but also scale - that's why you need cache and some database stuff. After all, with today's email load of 6-8 GB of mail and hundreds of contacts, it has to have some serious performance.

By now, you have everything 3-4 times in memory and in 3-4 different databases and caches. So, as a good programmer, you decide to re-engineer this: you need a common storage for this data. That will allow you to do the same thing but a lot better.

So, you make a great plan, a proper new architecture and you get to work. Unfortunately, you depend on a group of volunteers. This work is difficult so only the most 'core' members can do it, and time moves on - people get busy with other things so you lose people. Meanwhile you have to maintain the old applications too, as the new architecture is still far from ready. So you device a way to slowly migrate things over, and hope people can either bite through the birthing pains or come and help you out. But they don't, they don't get why this was needed, claim that Claws does mail just fine, and you get little help. Meanwhile, you've missed every possible deadline, it is 2013, your mail client (now just a for-two-years neglected GUI) is getting old and has issues of its own and everybody yells at you that you suck for even TRYING to fix the architecture behind their apps.

Somehow, you ended up with Akonadi. sjees. And I hope you will have learned that, surprise, doping PIM properly is HARD. That's why KMail/Akonadi/KDEPIM has issues: they are trying to build a PROPERLY engineered solution which actually WORKS for what people in 2013 want to do and that is not easy. It requires a solution which is complicated, so it needs a lot of testing, bugfixing, looking for corner cases. And with just a handful of volunteers, that is not easy to do.

There are surely some 'basic' use cases out there where you can use an app here and an app there. It will work for you. Might need the occasional sacrificial chicken and handwaving and it won't work that great with your company infrastructure, but hey, it works for you.

If you want calendaring, contacts, todo and all that scale-able and with multiple accounts and identities and grouping and proper security and performance - say, the stuff a modern company will need for its 2000 employees, KDE PIM is the only game in town. Or, of course, Microsoft Outlook. Groupwise, maybe.

That is why Kolab, which has as its mission to finally fix the final pillar of Free Software on the Corporate Desktop (PIM), is now working on KDEPIM. They are testing and stabilizing it as their customers will need it. So a year or two from now, it'll be quite good for what you want to do. Until then, help is welcome. And Boudewijn's attitude is far better for getting it finally done than yours - uninformed ranting never got far.

Have a nice day.


to post comments

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 17, 2013 12:33 UTC (Wed) by anselm (subscriber, #2796) [Link] (19 responses)

I don't disagree that KDEPIM based on Akonadi will be a fine piece of software when it is done. When it is done. I don't disagree that the problem to be solved, like incidentally many others of considerable interest to the community at large, is a difficult one that takes skill and experience.

However, the point is that you don't use something like that to replace outright what was there and worked (for all its shortcomings) when the new thing is only half-baked, and hope that the new thing will finally be all right six versions later once enough people have volunteered to fix the bugs and add the features that the previous version used to have and that the original developers couldn't be bothered to put in before inflicting the new code on the previous version's users as a non-optional replacement. (I consider myself a fairly competent developer but I'm not that interested in doing KDE stuff; I have my own projects to take care of and wouldn't have the time to figure out KDEPIM even if I wanted to. Does that disqualify me from using KDEPIM because I'm not in a position to help with developing it further?)

Arguably the way to do this sort of thing properly is to write a new program (or set of programs) based on the new architecture. You can eventually EOL the old program once the new program actually has all its features and there are well-debugged tools to assist with a migration from the old one etc., but in the meantime those people who are interested in the benefits of the new version can move over if and when they want. Let them use the old inefficient stuff for as long as they can stand it but don't force them to move to something that is only half-finished and doesn't work for them, just for your own convenience.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 17, 2013 13:09 UTC (Wed) by niner (guest, #26151) [Link] (18 responses)

So who exactly _forced_ you to abandon your well working setup and upgrade to one that doesn't? And why exactly do you thing you can demand from other people that they tend to a codebase for which they have no interest or not time when you are not prepared to contribute in any way?

FYI: the kmail 1 codebase is still out there, it's still free software. kdepim developers did what you want them to: they wrote a new program based on the new architecture. If people still want to use the old, _nothing_ is keeping them from it. It's your choice. But if you do, it's also your responsibility to do the maintenance work or find someone who does. I don't expect anyone else to clean my place either. Or if I do, I'm well prepared to pay them for it.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 17, 2013 13:42 UTC (Wed) by anselm (subscriber, #2796) [Link] (17 responses)

So who exactly _forced_ you to abandon your well working setup and upgrade to one that doesn't?

I'm tracking KDE in Debian. At some point the Kmail I'm using forced me to run a MySQL server just to cache some mail headers (I don't use the KDE addressbook program). I can live with that if I must but it's not exactly what I would consider great engineering.

And why exactly do you thing you can demand from other people that they tend to a codebase for which they have no interest or not time when you are not prepared to contribute in any way?

For the record, I do contribute to the free software community in enough different ways to not have to have a bad conscience because I don't want to also hack C++ code for KDE.

Having said that, I don't »demand« anything from other people. I respect the KDEPIM developers as clever programmers even if I don't agree with all the choices they make. Some of these choices mean that I don't respect them as great engineers but that is my privilege, just like it is their privilege, which I am not going to contest in any way, to do with their code whatever they please.

FYI: the kmail 1 codebase is still out there, it's still free software.

I know. I'm using it now. When Kmail 2.x lands in Debian I'll have a look to see whether it still works for me and if it doesn't I may start using something else that does.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 17, 2013 13:55 UTC (Wed) by niner (guest, #26151) [Link] (16 responses)

If this post were all you ever said on the topic, there wouldn't even have been a need for a discussion. But since we're at it: as a programmer, I think it's far better engineering to use a proven and solid storage component like MySQL than hand rolling it. And experience with the flaky hand rolled mail indexes in kmail 1 seems to confirm this. One could argue about which DBMS exactly would be better for the job and personally, I'd probably have never picked MySQL for the job but for now the existing solution just makes me a happy akonadi user. FWIW my initial reaction to this design decision was pretty similiar to yours. But the reliability and performance of the current version convinced me otherwise.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 17, 2013 21:36 UTC (Wed) by dlang (guest, #313) [Link] (15 responses)

Why do you need to cache all the headers locally anyway?

A good IMAP client knows how to query the server to leverage it's power (and local indexing)

For local files you will need to do something, but MySQL is a rather heavy thing to have to install just for this, using dbm or sqlite would seem to be much better fits for this type of task.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 17, 2013 23:00 UTC (Wed) by anselm (subscriber, #2796) [Link] (14 responses)

The KDE party line says that (a) the caching infrastructure must work even if you don't get your mail via IMAP, and (b) when Akonadi got started, SQLite couldn't cope with its requirements as far as concurrent access was concerned, hence Akonadi defaults to MySQL. There now appear to be PostgreSQL and SQLite backends in addition to the MySQL backend; the SQLite backend is said to be working »with limitations« and is mostly used on mobile devices.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 1:10 UTC (Thu) by HelloWorld (guest, #56129) [Link] (7 responses)

SQLite is a piece of crap anyway. You can specify all kinds of constraints when you create a table and it won't enforce many of them, not even the types. Really, what were they thinking? Developers write database schemas for a reason!

I'd also like to see some concrete evidence that SQLite is actuallylight-weight. I suspect that people think that essentially because of the cleverly-chosen name.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 1:23 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

Compare the size of SQLite library to the size of MySQL embedded library. That's the answer.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 21:45 UTC (Thu) by HelloWorld (guest, #56129) [Link] (5 responses)

In 2013 there are still people who think *that's* a sensible metric? Oh come on...

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 22:14 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

Yep. There is a difference between 1MLOC and 100kLOC. In maintenance and debugability, at least.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted May 4, 2013 11:19 UTC (Sat) by jospoortvliet (guest, #33164) [Link] (3 responses)

Ok, so THAT is what lightweight means? Lines of code? It has nothing to do with features versus size or metrics like memory usage and performance? I say you're ignoring a few 'details' here.

As Martin Grasslin wrote in an angry blog post a while ago - the term lightweight is both heavily under-defined and over-used, mostly by and through ignorance.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted May 4, 2013 19:57 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

Yes, number of lines of code is a good proxy. A small number means that a project is focused on one thing and nothing else.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted May 6, 2013 7:20 UTC (Mon) by jospoortvliet (guest, #33164) [Link] (1 responses)

A small scope is just that - a small scope. Useless in most situations.

I'm all for keeping things small and simple and for me, a lower SLOC is certainly a feature - but just ONE feature. Performance and efficiency are other rather important ones. And the ability to, you know, do stuff I need, that also scores high.

In case of mail clients, doing one thing and nothing else (mail) was great in 1995. It's 2013 now and even your phone shows you twitter status updates in the contact app & grabs contact details and image from Facebook.

In case of SQL - a library which doesn't scale or perform in many real-world scenarios (despite its small SLOC count) is useless for those scenarios. And I mean - on mobile phones, even. I mean, it is great if you don't need concurrency (who has a multicore CPU or more than one app running, right?) or more than a small number of entries in the db.

Right tool for the right job - lightweight continues to be an empty concept. I'm happy to exploit the widespread love for the word but I expect smart people to know better than take it serious.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted May 6, 2013 8:32 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

Quite a lot of projects don't NEED to scale to multiple CPUs and have esoteric features.

For example, SQLite is used to keep my contact book on my phone. It works fine even with a corporate address book with thousands of entries. Why would I need SQLite to scale to thousands of cores?

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 2:41 UTC (Thu) by dlang (guest, #313) [Link] (5 responses)

My point is that you should not need to have a mandatory caching layer at all. A good IMAP server will not need it, and the cost can be crippling under some conditions.

It's all well and good to offer one for non-IMAP systems, or to offer it as an option for IMAP system (not all IMAP servers are good)

But it shouldn't be mandatory for all collections.

I have mail folders with over 300,000 messages in them. having the mail client decide that it needs to download headers from every message when I just want to look at the last 30 messages in the folder is just unacceptable (and part of the reason I don't use kmail and other clients that act similarly)

Especially if you have poor or expensive connectivity, the idea that you _should_ have a local cache of this sort of thing is just wrong. It's just to expensive to populate.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 6:06 UTC (Thu) by anselm (subscriber, #2796) [Link] (4 responses)

The whole thing probably comes from the KDE project's love of abstraction layers.

I'm probably not the best person to defend the idea because I'm not convinced that it makes great sense, either. Then again I use offlineimap to sync all my mail to a local Dovecot instance because Kmail's client-side IMAP support used to really suck. For all I know it may be better now but I'm staying with this arrangement because it works well for me; »one tool for the job« etc. and no way am I allowing Kmail to go out on the Internet. As you say, a caching layer for e-mail metadata is superfluous in this case but this observation is apparently lost on the KDEPIM guys, or they would at least have introduced a »light-weight passthrough« option that made the API work but didn't actually cache anything within Akonadi.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 6:38 UTC (Thu) by dlang (guest, #313) [Link] (3 responses)

abstraction layers can be a good thing, but they key is picking the right ones.

In this case, instead of making the abstraction layer be a local database that holds info on all mail, local, remote, or IMAP and accessing that database directly, a better approach would have been to have the abstraction be a very thin layer over IMAP, and implement the needed indexing/caching only for the data sources that don't have the IMAP features (close to just implementing a local IMAP server, but without the need to handle all IMAP commands and corner cases, only what they use)

for those who don't know, good IMAP servers implement fast searching, sorting, threading on the server, so copying this data to a local database instead is falling into the trap of maintaining multiple copies of the same data unnecessarily (so you are guaranteed to have them get out of sync), and it means that if you have thousands or hundreds of thousands of messages in one folder (not uncommon on good IMAP servers), the client can spend a lot of time downloading headers when the user connects before the user can actually do anything.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 6:58 UTC (Thu) by niner (guest, #26151) [Link] (2 responses)

What you're missing is, that I want to access my IMAP mailbox even when I have no internet connection. So it can make sense to cache all the data locally even with IMAP.

Also fast searching and threading on the server are relatively recent IMAP features. At least they are only recently supported by IMAP servers. But you're right in that they are really awesome and we make good use of them in CiderWebmail which ironically does not do any caching at all. But since it's a webmail, having an internet connection is somewhat mandatory for use anyway :)

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 7:46 UTC (Thu) by dlang (guest, #313) [Link] (1 responses)

> What you're missing is, that I want to access my IMAP mailbox even when I have no internet connection. So it can make sense to cache all the data locally even with IMAP.

That is what things like offline IMAP are written for.

I don't dispute that you want to do that, and I don't say that it's a bad option to have.

However, forcing ALL IMAP users to suffer the pain of copying all their folders to each device just because some people want to work without a network connection is throwing away a lot of the advantage of IMAP.

I couldn't do that if I wanted to, my IMAP folders are large enough they wouldn't fit on some of my devices.

> Also fast searching and threading on the server are relatively recent IMAP features. At least they are only recently supported by IMAP servers.

depends on what you consider recent :-) Cyrus has supported them for quite a while.

Thanks for the link to CiderWebmail, I've been on the lookout for a good IMAP webmail client, I'll have to give it a try

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted May 4, 2013 11:20 UTC (Sat) by jospoortvliet (guest, #33164) [Link]

Akonadi doesn't cache all your data if you have an on-line IMAP account. only disconnected imap caches a lot of data. So this whole discussion is rather moot...

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 17, 2013 12:47 UTC (Wed) by HelloWorld (guest, #56129) [Link] (6 responses)

Nobody is arguing against a properly engineered PIM solution. But the fact that you're doing something that is right in principle doesn't mean that it actually works well. As I said earlier, kio_imap4 often just seems to hang indefinitely, imap IDLE doesn't work (even though it's supposed to), and afaik Akonadi still doesn't work with an NFS home directory. Oh, and kmail still isn't able to handle folder names with Umlauts. And sure, I could report all that. Or I can just use something that does what *I* need.

And that doesn't include most of the stuff that you just listed. I don't "talk" to people via email, that's what XMPP is for. And I honestly don't understand why one would even need a calendar application these days when everybody uses their smartphone to manage appointments with (and contact/appointment synchronisation is one area where the open source desktop still sucks badly). And even if I saw a use for that, why would my calender need to access my contacts? I can have appointments with people just fine: Joe will send me an email with Subject "Lunch tomorrow at 13? (eom)", I'll reply with "OK (eom)" and type "Lunch with Joe" into my phone. And as you can see, my email program doesn't need to know about my calendar either, whoa! I have a feeling that things become much simpler when you do them this way, and I think most people do.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 17, 2013 13:00 UTC (Wed) by niner (guest, #26151) [Link] (4 responses)

So, _you_ don't need the functionality, so nobody else is allowed to? News for you: you are not the center of the world.

I _want_ my email program to have access to my calendar, so when I get a meeting invitation as ICAL attachment, I can click on accept and have it in my calendar. I do not want to have to type anything on a smartphone that I don't even have. And when I set up a meeting I want to click in my calendar and just add my business contacts and not have to type in a list of email addresses, thank you very much.

kio_imap4 hangs indefinitely and IMAP IDLE doesn't work? Ever thought about upgrading to current akonadi based kmail2 where these things are fixed?

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 17, 2013 17:03 UTC (Wed) by HelloWorld (guest, #56129) [Link] (3 responses)

> So, _you_ don't need the functionality, so nobody else is allowed to?
I never said anything remotely like that. Heck, I wouldn't even know which part of my comment could be misunderstood that way. I'm just saying that the Akonadi developers shouldn't be surprised when the people who don't need that kind of functionality leave for something that does a better job at what they do need.

> kio_imap4 hangs indefinitely and IMAP IDLE doesn't work? Ever thought about upgrading to current akonadi based kmail2 where these things are fixed?
I have the most recent version of kmail from the Fedora repositories installed (version 4.10.1 according to the about dialog) and the relevant server does support IMAP IDLE. So yes, it's supposed to be fixed, and no, it doesn't actually work for me.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 11:17 UTC (Thu) by nye (guest, #51576) [Link]

>I never said anything remotely like that. Heck, I wouldn't even know which part of my comment could be misunderstood that way.

I don't think anything you said could be even plausibly construed to mean that by anyone not specifically intending to pick a fight.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 11:50 UTC (Thu) by niner (guest, #26151) [Link] (1 responses)

It's words like "I honestly don't understand why one would even need" or "even if I saw a use for that" that can convey such a meaning.

You wrote about kio_imap4 which is not used by kmail 2. Hence I got the impression that you were still using kmail 1. Especially since you cited problems that are supposed to be fixed in current Akonadi. As they are still there for you, I strongly suggest reporting them on bugs.kde.org. The developers really try to fix them but there are obviously many different problems leading to the same symptoms.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 18, 2013 22:40 UTC (Thu) by HelloWorld (guest, #56129) [Link]

kio_imap4 in fact isn't used any longer, sorry about that. Nevertheless, IMAP IDLE doesn't work for me. Anyway, my personal experience with bug reporting is that it hardly ever helps. I have other fish to fry and simply use another email client now.

Stephenson: hackweek9: Lightweight KDE Desktop project

Posted Apr 23, 2013 12:38 UTC (Tue) by nix (subscriber, #2304) [Link]

I believe it is strigi that doesn't work with an NFS home directory. Indexing, not caching. (Why you can't just run the indexer on the server and have the client query it over this 'network' thing we seem to have these days is not clear to me, but apparently you can't. Ah well, as of KDE 4.10 strigi can hardly index any file types anyway, so it's not much of a loss.)


Copyright © 2026, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds