Development issues part 1: Project communication

Free software projects, like all projects, live and die by their communications; developers must be able to talk to each other easily so that a consistent, coherent result emerges. But developers have differing ideas about what methods to use. A discussion on the Emacs development list provides a nice contrast between two of the main communications methods used by projects today.

Traditionally, developer communications have been handled by the venerable mailing list, but that is changing, at least for some projects. Internet relay chat (IRC) has become the tool of choice for newer projects, which may leave those who are not inclined towards realtime communication out of the loop. Development methodologies are evolving, and some are adopting the new ways more quickly than others – some may never adopt them at all.

The difference between communicating in IRC or via a mailing list is in some ways like the difference between text messaging and email. Email has its advantages, in that the recipient chooses the time to read and respond to the message, but it is often seen as slow. Text messaging or IRC have the advantage of speed; people receive a message and generally respond immediately. But that speed comes at a cost – interrupting the recipient. It also requires a full-time internet connection.

While email archives are somewhat cumbersome to use, they are usable. IRC logs are exceedingly painful as they are not subject-based; they just cover a specific time span of all conversation on the channel. Email conversations may play out over days or weeks, but they are generally easier to follow compared to the multiple interleaved conversations that occur on IRC channels. It is in the nature of the medium: IRC conversations are meant to be used immediately, not reread weeks later.

It is, in some ways, a culture clash. Younger developers tend to be more inclined towards realtime communications, while older hackers tend to be more comfortable with mailing lists. In what would seem to be an uphill battle, Eric S. Raymond has been advocating a more "modern" development style for GNU Emacs. His messages, appearing on Emacs-devel, champion a development style that includes IRC communication, a bug tracking system, and a version control system (VCS) more advanced than CVS.

Raymond's experiences working with the Battle for Wesnoth development team exposed him to some of the newer techniques used in project communication, particularly IRC. He reached a somewhat surprising conclusion about IRC:

The Wesnoth project uses IRC for all day-to-day design and development decisions, leaving the mailing list for more complicated discussions and white papers. This has the effect of excluding interested developers who are not able or willing to monitor an IRC channel throughout their day, but that is unlikely to be the intent. The reverse is also true: the perceived slow pace of mailing-list only projects has the effect of excluding those with a strong preference for a faster style of development. As Raymond shows, though, there is hope that members of one school can retrain – if they wish – for the other.

While decision making by IRC does not seem to be in the cards any time soon for Emacs, an upgrade to something other than CVS seems to have gained more traction. Richard Stallman has been asking a lot of questions about git while other developers discuss other distributed version control systems (DVCS), like darcs, monotone, arch, and Mercurial. Raymond is working on a survey of the VCS landscape that, once completed, he and others hope will guide the project into a better VCS choice.

One of the main DVCS features that seems of interest to Stallman is the "offline" capabilities. Having the entire history of a project and being able to do commits of work in progress while being disconnected from the internet are features that CVS does not have. Stallman is adamant that the tools used to develop Emacs be usable by those who are not always connected to the net which makes a DVCS rather attractive.

The Emacs project is one of the oldest free software projects in existence; it is, like its founder, fairly resistant to change. While Emacs itself is used by hackers everywhere, it is increasingly falling behind its competitors, at least partially because of the slow pace at which it is developed. Raymond's belief is that by upgrading the tools used to take advantage of advances made since CVS and mailman were new, the time between Emacs releases could be reduced to something more sane. Doing that could go a long way towards making Emacs more relevant to younger hackers:

It is unlikely that just some tool changes will be enough to resurrect the flagging popularity of Emacs, but there are hopeful signs. Some of Raymond's suggestions met a warmer reception than one might have expected. It is clear that a fair number of Emacs fans and developers are frustrated with the current state of affairs. It may be that "just some tool changes" are enough to reinvigorate the project to a point where it attracts more developers and users. That can only be a good thing for Emacs.

>One of the main DVCS features that seems of interest to Stallman is the "offline"
capabilities. Having the entire history of a project and being able to do commits of work in
progress while being disconnected from the internet are features that CVS does not have.

Did he hear of Git yet? Also, "distributed" itself already screams for offline working, but I
am welcome to hear counter-examples.

The article says, "Richard Stallman has been asking a lot of questions 
about git"....

> The reverse is also true: the perceived slow pace of mailing-list only projects has the
effect of excluding those with a strong preference for a faster style of development. As
Raymond shows, though, there is hope that members of one school can retrain – if they wish –
for the other.

No, this is often not only the issue of will. It's related to the style of life. For a
majority of high-tech full-job occupations, IRC is impossible during the work hours. Of
course, DVCS and specialized bug-tracking software are OK.

Well it depends if it's part of your job. At my current place of work we have internal IRC
chat rooms for each project where it's not uncommon for someone to pop up and ask a question
about X and be pointed in the right direction. The use of trigger phrases and sounds mean you
don't actually have to read every line of dialog ever said.

Careful choice of IRC bots can also be useful. Once such hand crafted "multibot" provides a
great deal of useful script like functionality to all the engineers in the room at the same
time allowing idle observers to see stuff someone is looking at.

As noted above, IRC is problematic for many people during work hours.

However IRC is also inconvenient when developers are all over the world in completely
different time zones.

Eyal.

>This has the effect of excluding interested developers who are not able or willing to monitor
an IRC channel throughout their day, but that is unlikely to be the intent.

There is #emacs on irc.freenode.net, where actual usage discussions of emacs are the
exception, and development discussion is rare, AFAIK.  But it's good for a belly laugh.

There may be a middle ground for serious development discussion on IRC that is then edited and
posted somewhere.  Taking the transcript, formatting it, and posting to
http://www.emacswiki.org, for example, would lessen the impact of the timezone and firewall
issues, though it would require either a) some discipline amongst the developers (it could
happen), or b) some hapless fellow to edit (but hey, it's on a wiki, so finding someone to
filter it and cross-link it should be less of a challenge).

The question of IRC vs. Email list is only on the surface a culture clash. But below, it's the
difference between people who can work full-time on a project (or are, at least, full-time
reachable for communication on a project) and people who do so in their spare time. Since more
and more people can afford to make a living out of Open Source software, the first class of
contributors gets larger.

Now, these full-time contributors have already a big advantage when it comes to shape a
project; not only code-wise, but also in architectural and design decisions. This is already
visible on mailing lists, where some people can always respond within the hour (and thus shape
the future discussion) and others answer that evening or the following day. Moving
communication to IRC as some ``standard'' communication medium that's expected leaves those in
the cold who can work only in the evening (or live in other parts of the world and not just in
another US time zone, but ESR seems to be oblivious to that point). And, as noted already, to
read up such discussions later, is much harder with IRC than with email.

Moving day-to-day communication to IRC has the effect of building a cabal for a project, it
forwards the creation of an `inner circle', loosing the bazaar effect that ESR says is so
important. That said, IRC is a very good medium to have project meetings at preset times, to
discuss things that would take too long over email. 

For me, the difference boils down to the questions: ``Are main developers expected to be on
IRC the whole day through?'' vs. ``Why don't we use IRC as an additional communication medium
for project meetings?'' The first is bad, the second is good.

Cheers, Joachim

PS: I'm the CEO of a small consulting company. My staff can spent 20% of their time on Open
Source projects of their choosing. But I would not allow them to be on IRC full-time for the
rest of their working time; after all they have projects to do that bring in the money for
those 20% free work.

I've spent a bit over 4 years working on Gentoo, a project with a major IRC component. I'm
logged into IRC 24 hours a day via screen session running on a server, but I only check in for
messages, similar to email. When I have a message, I can check whether the sender's present
and immediately have a realtime conversation and get things done quickly. As someone already
mentioned, just because you're logged into IRC doesn't mean you're staring at its window all
day. The same is true for both email and IRC -- you can waste hours on it, or you can be
efficient.

Another thing I've seen on IRC is growth of a tighter community, where people actually get to
know other people instead of merely have technical discussions with them. I've also seen that
far fewer flamewars happen on IRC.

> When those Eclipse fans pointed and laughed because we're still stuck on
> CVS

Eclipse is still stuck on CVS:
http://dev.eclipse.org/viewcvs/index.cgi/

Some sub-projects are apparently using SVN, but the main development is done using CVS.  And
if you want to use SVN from inside of Eclipse, you have to use third-party plugins for that.

Looking at the big picture, emacs releases are glacial. The tight control of the lead
maintainer is legendary and has been responsible for many talented hackers giving up on emacs
and taking their hacking talents elsewhere.

It is criminal that support for decent type in emacs on free software systems has not present
in official, released versions for the past two or three years. Whereas other, non-free
systems don't put up with crap like this and have had decent type for years. 

Instead, most die-hard emacs users on GNU systems are forced to use:

http://www.emacswiki.org/cgi-bin/wiki/XftGnuEmacs

Status still uncertain.

There's packages for ubunutu that package a cvs build with decent font support.

http://peadrop.com/blog/2007/01/06/pretty-emacs/

You could blame Emacs, or the awfulness which is flattish QWERTY 
keyboards. I know my RSI vanished as soon as I moved from QWERTY to a 
decent contoured keyboard, without needing to torment myself and ruin my 
productivity by abandoning XEmacs :)

Even if Emacs has no bug tracker, it has incredibly responsive developers. The five odd bugs I
ran into (using CVS-Emacs) were usually resolved within hours by RMS and other developers. 

Clearly much better than any bug tracker. 

That is great news for the bugs you found, but there is more to it than that.

If the developers had been busy at the time and couldn't work on it immediately, are you sure
that they wouldn't have lost your report?

When the developers are preparing a release, do they know what problems are fixed?  Do they
know what problems haven't been fixed?

When a new issue is reported, do the developers have a way of deciding how important it is
relative to the other open issues?

These sorts of things are very useful when deciding what problems to fix or making a release.

Wow, I really find this series interesting. Please keep them coming.

In the past, I've found many older software projects throw up many barriers to entry. Be them
annoying mailing lists, autotool-build systems, or centralized source control systems (that
make doing your own local hacks before you have write access very annoying). The quicker to
allow developers to contribute (wikis!) the better, its just good for moral.