Interview with three OpenChange project developers

Julien Kerihuel: The OpenChange MAPI library (libmapi) provides interoperability with Microsoft Exchange servers, which means being able to communicate with Exchange in the same way that Outlook does. Our primary objective in providing the MAILOOK release is to make our implementation available to beta testers so we can validate the design, evaluate libmapi code scalability and reliability; and thus fix possible inconsistencies before we go to much further in the development process.

This release is a very important step in the project roadmap. It is the first time that we've provided code which can be used in stand-alone applications and which is generic enough to perform most of the messaging operations that final users would deal with, such as:

• fetch emails (including those with attachments)
• send emails (including those with attachments
• delete emails
• set multiple recipients to, cc, bcc
• UTF8, Unicode and HTML content support
• folder creation/deletion

In addition, if developers take a closer look at the repository version (subversion trunk), they will find alpha code for other Exchange items support:

• fetch/create calendar events for appointment/meetings screenshot 1, screenshot 2
• fetch/create contacts screenshot
• fetch/create tasks screenshot
• fetch/create sticky notes

These features will be included in the next library release, planned for early June 2007.

Julien Kerihuel: I am the project leader. I founded the project in 2003 as my final study project at EPITECH (French computer sciences school). After graduation, I started a company (OpenChange SARL) devoted to Open Source development and Education. I dedicate most of my time to OpenChange and provide MAPI consulting services, education, software development and web services for a living. The project team is mostly composed of students or graduate students from EPITECH. The current team is composed of:

• Ali Mdidech: Network and System Administrator
• Pauline Khun and Fabien Lementec: Core Developers
• Remi L'Ecolier, Jean-Baptiste Arnoult Clement Jamot and Thomas Chalmel: Evolution plugin

We are also tightly linked with the Samba team who contribute to the project on a daily basis.

OpenChange SARL was created in 2006 for several reasons:

• 1. Provides services, support and consulting when OpenChange 1.0 is out
• 2. Support full time developers (for the moment, only me)
• 3. Support parts of the project: server renting, communication material and so on.

Julien Kerihuel: We want OpenChange to be a complete Exchange replacement framework and the client library is just the first step in our development road map. Our primary objective has always been the server replacement, but we needed to go through the MAPI library development first:

1. If we want interoperability, we need to supply material so people can replace either their Outlook clients or their Exchange servers.

2. Developing the MAPI library is the best way to understand MAPI internals and how the server should handle data depending on clients requests.

We have already started the server development and have a working Exchange Message Store Address Book Provider (EMSABP). This part of the server is used by Outlook to perform name resolution and Exchange account setup. Our implementation works on top of Samba4 and deals with Samba4 Active Directory.

We plan to have a server "hacking preview" release in late June 2007.

Julien Kerihuel: While the library is still alpha, we are working on design improvements so it can be used by third party applications. Our main objective is to provide a common API which can be used by Gnome Evolution, KMail, Thunderbird and Sunbird. This is a challenging issue that will require quite some time and development, but this is the final objective.

So far, we have started the implementation of command line tools so we could validate libmapi ability to get used in stand-alone applications:

• openchangeclient: MAPI command line messaging tool designed to send/receive mails
• exchange2mbox: convert an Exchange mailbox to a mbox file, support attachment and mime types.

At the same time, part of the team has been working on a evolution plugin using libmapi. The current implementation provides:

• account creation wizard
• Exchange name resolution
• fetching emails with/without attachments
• sending emails with/without attachments to Exchange users
• deleting emails
• flag management: attachment, important, and date

Last but not least, libmapi provides a development environment for the OpenChange server test cases.

Julien Kerihuel: The Samba team involvement into OpenChange has played a key role since OpenChange early days. I remember in 2003 when I discussed for the first time with Andrew Tridgell on IRC. He has been the first one to believe in OpenChange at a time when we didn't even have a single line of IDL (Interface Definition Language). He gave us his time to explain Samba4 internals, how to write IDLs etc.

OpenChange soon became a Samba4 fork and we had quite some troubles maintaining the code between Samba4 which was alpha and changing all the time and our own code.

One of the most important step in OpenChange development has been the amazing work Andrew Bartlet and Jelmer Vernooij did in adding dynamic library support to Samba4. At this point we have been able to move from a nice fork to a Samba4 independent plugin and push OpenChange code on its own repository.

We are working on a daily basis with most of the team:

• Jelmer Vernooij who is actively contributing to OpenChange, its build system and Debian packaging.
• Simo Sorce when we have LDB/TDB databases related questions,
• Andrew Bartlet and Stefan Metzmacher who give us valuable advice based on their Samba development experience.

As you may imagine, OpenChange is strongly relying on Samba4:

• Transport and network layers
• Storage layers and Active Directory
• Authentication
• Memory Allocation

Julien Kerihuel: For the client side, OpenChange is like a mille-feuilles (en-us: Napoleon, en-uk: Vanilla slice). We provide a low level API that closely matches the Exchange protocol, but we also attempt to provide a high level API that can be easy to use and understand by developers.

MAPI is a RPC (Remote Procedure Call) protocol which means any operation is performed on the server and results are sent back to the clients. Exchange is using objects and tables (like a spreadsheet), which means that the client can either ask for a new or existing Exchange object or opens a MAPI view (columns and rows) with the results it expects. All the communication on a given object is then done using its handle as a reference. What we have tried to do in OpenChange is to provide a sufficient layer of abstraction so any of these objects could be handled with an opaque structure and be used in any of the MAPI library public functions.

OpenChange also provides MAPI profile support, so developers don't have to deal with credentials or any other "offline" information that would be needed during Exchange communication.

For the server side and its existing implementation (the EMSABP provider) OpenChange is fetching information from Samba4 Active Directory and sending information back to the client with the MAPI transport layer.

Julien Kerihuel: We got through a very long analysis process. We developed numerous tools other the years to help in analyzing network captures and have also been working with Wireshark and the Samba4 hacking test suite, such as ndrdump to validate the IDL and smbtorture to test the implementation and find unknown IDL fields.

The OpenChange tools include:

• rpcextract: built on tshark (Wireshark command line tool), rpcextract splits a communication file in ndrdump file we can next process with ndrdump or mapitrace.
• rpcsniff: a windows tool designed to understand MAPI calls over the wire.
• mapitrace: an experimental Perl tool that parses ndrdump output. It provides a way to analyze MAPI communications, generates GraphViz graphs, has a basic error report and also a statistics module. It offers a more convenient way to see how much of a given scenario we are able to decode and how MAPI calls are linked one to the other with a graphical or console view.
• The KBPortal. This is a Web portal designed to help development. It can store scenarios, share knowledge, adding comments, generate PDF reports. Due to a lack of maintainers, we decided to stop supporting it. The problem is that the code is far from being maintainable, and we would need several developers to rework it from scratch so we can use it again properly and plan extensions. The tool has been developed over 3 years and I think it's a shame we had to stop using it.

Julien Kerihuel: I wouldn't use "have been" but "is" cause we are still far from any final code. More than parts, I would be tempted to say the exact moment when you finish the design and implementation and you have a working test for the very first time. You first look at it like an accomplishment, a visible result for all the time you have spent doing research and network analysis. A couple of seconds later, you start to imagine all the improvements you can add and define a roadmap made of new challenges.

Now we start to have some MAPI and Exchange protocols background, we can focus on many tasks we could not consider before.

Fabien Lementec: Opensource community interaction is the very interesting part. Network analysis and protocol understanding are challenging tasks, given the fact there is very few documentation about the protocols used.

Julien Kerihuel: The research process needed to find MAPI calls. Fabien has played a key role with the development of the rpcsniff windows tool which offered the first framework we needed to achieve this task.

In the meantime, I think we have not yet started the most difficult parts! The Message Store on the server side is really the most challenging part of the project. It will rely on Samba4's Active Directory and offer abstraction layers so people can link it with their existing messaging server (such as Postfix or sendmail).

Julien Kerihuel: Due to the recent advances and beta testers feedback, I think we now have a better overview of what libmapi needs to look like and the major tasks we need to focus on. We originally plan to have basic and full Exchange items support for Mid-August, but we are so close to this result that we want to raise the stakes.

First we have to remove MAILOOK release limitations and particularly ensure that we can send emails to external recipients. There is the notifications issue that is quite interesting and I would like libmapi to support for next release. Last but not least, internationalization and Unicode support will be key to making libmapi generic.

We will certainly soon provide an intermediate release with new Exchange items support (calendar, tasks, contacts, notes) and MAILOOK limitations fixes.

We will next work on a simple MAPI API for high level programming which will make abstraction of most of MAPI internals, but this process can only be successful if other open source communities get involved and share with us their needs and expectations.

We do need beta testers with different Exchange servers, so we can identify the numerous limitations libmapi still encounter. We also need people to help on the project documentation, maintaining a FAQ, review the existing documentation and make it sound "more English".

Ali Mdidech: Another aspect of making MAILOOK available for everyone is to provide a release under most common platforms. Having a port on FreeBSD, a package for *BSD, an RPM for Linux, a Solaris version and much more should be a real opportunity for having testers. But this requires a lot of work and we would be really happy if some folks want to get involved in it.

Finally, we are really interested in providing bindings for other languages, but we have limited human resources. If people are interested in developing PHP, Python or Perl bindings, they are truly welcome!