LinuxCon: Clay Shirky on collaboration

Author and keen technology observer Clay Shirky came to LinuxCon in Vancouver to impart his vision of how large-scale collaboration works—and fails to work. In an energetic and amusing—if not always 100% historically accurate—talk, Shirky likened collaboration to "structured fighting" and looked at how and why that is. It is, he said, the structure that makes all the difference.

Shirky started things off with his "favorite bug report ever" (Firefox bug #330884), which starts with the line: "This privacy flaw has caused my fiancé and I to break-up after having dated for 5 years." Because of the way Firefox was recording information about sites that were blocked from ever storing the password, the woman who filed the bug found out that her intended was still visiting dating sites. What was interesting, Shirky said, was that the responses in the bug report not only included technical advice, but also relationship advice that was presented as if it were technical information. The report is proof that we can never really "disentangle the hard technical stuff from the squishy human stuff", he said.

He then put up a picture of the "most important Xerox machine in the world" as it was the one that was sent to Richard Stallman's lab without any source code for a driver. In "an epic fit of pique", Stallman wrote a driver and has devoted the following 25 years of his life to fighting the strategy of releasing software without the corresponding source code. But GNU projects were tightly managed, and it wasn't until another project came along, Linux, that the full power of large-scale collaboration was unlocked. Eric Raymond had this idea that the talent pool for a project was the entire world. Linus Torvalds took that idea and ran with it, he said. (That probably isn't quite the order of those events the rest of us remember, but Shirky's point is still valid.) One of the things that open source has given to the world, is the "amazing" ability to manage these large-scale collaborations.

Cognitive surplus

It goes well beyond software, he said. If you look at the "cognitive surplus" that is available for collaborative projects, it is truly a huge resource. A back-of-the-envelope calculation in 2008 came up with 100 million hours to create all of Wikipedia, including the talk pages, revisions, and so on. But that pales in comparison to television watching which takes up an estimated 1 trillion hours per year. There is an "enormous available pool of time and attention" that can be tapped since people are now all connected to the same grid, Shirky said.

As an example, he pointed to the Red Balloon Challenge that DARPA ran last year. They wanted to test new collaboration models, so they tethered ten weather balloons in locations across the US. The challenge was to gather a list of all ten and their latitude/longitude to within a mile.

An MIT team won the challenge by saying they would share the prize money with anyone who gave them information about the locations. But they also took a cue from Amway, he said, and offered a share of the prize to people who found a person that could give them location information. That led to a network effect, where people were asking their friends if they had seen any of the balloons. In the end, the MIT team solved the problem in nine hours, when DARPA had allocated 30 days for the challenge. "That's the cognitive surplus in action", Shirky said.

"When the whole world is potentially your talent pool, you can do amazing things", Shirky said. lolcats is one of those things and a "goodly chunk of cognitive surplus" goes into creating them, which leads to criticism of the internet. But that always happens with new media, he said, pointing out that the first erotic novel was written shortly after the invention of the printing press but that it took 150 years to think of using the invention for a scientific journal.

He showed several quotes from people reacting to new media like the telegraph, telephone, and television at the time each was introduced. The introduction of the television led the commenter to believe that world peace would occur because it would allow us to better connect with and understand other cultures. "Here's a hint of what happens with new media—it's not world peace", he said. More people communicating actually leads to more fighting, and the challenge is to figure out how to structure that fighting.

Shirky believes that the transition from alchemy to chemistry was fueled by the "decision to add structure to what the printing press made possible". Instead of performing and recording experiments in secret as alchemists did, the rise of the scientific journal changed the focus to publishing results that others could test for themselves—or argue about. The difference between the two is that alchemists hid their discipline, while chemists published, he said.

Three observations

Three observations about collaboration rounded out the rest of Shirky's talk. While its not a canonical list, he said, there are useful lessons from the observations. The first is that "many large-scale collaborations actually aren't". If you look at the page for Linux on Wikipedia, there have been some 10,000 edits from 4,000 different people. That equates to 2.5 edits per person, which is a pretty standard rate for Wikipedia pages.

That might appear to be a very large-scale collaboration, but it's not, he said. If you graph the contributions, you soon see that the most active contributors are doing the bulk of the work, with the top contributor doing around 500 edits of their own. The tenth highest contributor did 100 edits, and the 100th did 10 edits. Around 75% of contributors did only one edit ever.

That same pattern shows up in many different places, he said, including Linux kernel commits. These seemingly large-scale collaboration projects are really run by small, tight-knit groups that know each other and care about the project. That group integrates lots of small fixes that come from the wider community. Once we recognize that, we can plan for it, Shirky said.

Shirky's second observation was that many of the people who want to collaborate shouldn't be allowed to, at least easily. He pointed to stackoverflow and the related StackExchange sites as embodying some of this philosophy. StackExchange was spun off from stackoverflow to handle additional topic areas beyond just programming that the latter covers. Possible question and answer topics are "anything that is geeky enough to have a right answer" and that people want to argue about, Shirky said. But creating new Q&A sites on StackExchange does not follow the model that many other internet sites have used: "just let people do what they want and see what sticks". Instead, it is difficult to start a new site, which ensures that there is enough real interest in the topic.

The sites are "taking karma really seriously", and are "stretching both ends of the karmic equations". New users are not allowed to post either questions or answers right away, but must build up karma by reading the site first. Net etiquette always said that new users should do that, but "no one did it". At the other end of the spectrum, users can build up enough karma that they get sysop-like powers. These sites are an "attempt to say that we don't have to treat all people the same", he said.

Technology and human patterns need to match up, Shirky said, as his third observation. This goes back to the bug report at the beginning of his talk. It has taken a long time to align the two, he said, because code dominated free software communities for so long.

As an example, he pointed to the saga of Linux kernel source code management, which started out as tarballs and patches. Then BitKeeper showed up, and then went away, which (Shirky said) caused Torvalds to go back to tarballs and patches. Basically, Torvalds chose not to use any source code manager than use one whose functionality did not embrace the ideals of the GPL, Shirky said. He was not making a licensing argument here, after all Torvalds had been using the decidedly non-GPL BitKeeper, but instead was arguing (perhaps somewhat inaccurately) that Torvalds chose BitKeeper, and later Git, because the way they operate is in keeping with GPL ideals. Git "lives up to the promise of the GPL", because it decentralizes repositories and allows easy forking. Merging code should always be a community decision, which Git also embodies, he said.

Once Git was released, there were other interesting developments. Source code management systems had been around for decades, but were never used for anything but source code, Shirky said. Because Git matches people's mental model of how collaboration should work, it spawned things like github. But it doesn't stop there, he said, noting that there are Git repositories for novels, and that someone had checked in their genome to a public repository. The latter, of course, spawned an immediate pull request for 20 upgrades. A joke, but one that eventually resulted in a scholarly discussion about caffeine sensitivity that had participants from organizations like the National Institutes of Health.

There is also an effort called Open Knesset [Hebrew] that is attempting to use Git to help people better understand what they agree and disagree about. Essentially it takes laws proposed in the Israeli Knesset and checks them into Git, then tells people to fork the law and write it the way they would like to see it. "That will show where the arguments are", Shirky said. It is "audacious enough" that it probably won't work, but he also noted that "audacity beats predictability over the long haul". He believes we will see more of this kind of thing in the future.

One way to look at large-scale collaboration is that it is more people pooling more ideas, and that's true he said, but he would make an addition: "after arguing about it for a really long time". Taking this "structured argument approach" that free software (and other) communities have and moving it into other areas of our world will be beneficial. Applying some of the lessons learned from communities like StackExchange, Open Knesset, and the Linux kernel, as well as lessons from things like Mozilla bug entries will provide a means to take argumentation to the next level—and actually make it matter.

[ I would like to thank the Linux Foundation for travel assistance to attend LinuxCon. ]