Notes from an online free-software conference

The first thing the organizers of a conference that is threatened in this way must do is to face the problem early and start thinking about how to respond. Your editor has talked with organizers from a few other events who didn't (or couldn't) do that; the result is that they were not fully prepared and the attendee experience suffered. In the case of LPC, planning started in March, fully six months ahead of the event — well before the decision to cancel the in-person event had been made. We needed every bit of that time.

Which platform?

An online event requires an online platform to host it. The Linux Foundation, which supports LPC in a number of ways, offered a handful of possibilities, all of which were proprietary and expensive. One cannot blame the Linux Foundation for this; the events group there was under great pressure with numerous large events going up in flames. In such a situation, one has to grasp at whatever straws present themselves. We, though, had a bit more time and a strong desire to avoid forcing our attendees onto a proprietary platform, even if the alternative required us to build and support a platform ourselves.

Research done in those early days concluded that there were two well-established, free-software systems to choose from: Jitsi and BigBlueButton. Either could have been made to work for this purpose. In the end, we chose BigBlueButton for a number of reasons, including better-integrated presentation tools, a more flexible moderation system, and a more capable front-end system (though, as will be seen, we didn't use that part).

BigBlueButton worked out well for LPC, but it must be said that this system is not perfect. It's a mixture of highly complex components from different projects glued together under a common interface; its configuration spans literally hundreds of XML files (and some in other formats). It only runs on the ancient Ubuntu 16.04 distribution. Many features are hard to discover, and some are outright footguns: for moderators, the options to exit a meeting (leaving it running) and to end the meeting (thus kicking everybody else out, disposing of the chat session, and more) are adjacent to each other on the menu and look almost identical. Most worryingly, BigBlueButton has a number of built-in scalability limitations.

The FAQ says that no BigBlueButton session should have more than 100 users — a limitation that is certain to get the attention of a conference that normally draws around 600 people. A lot of work was done to try to find out what the real limitations of the platform were; these included automated testing and running a couple of "town hall" events ahead of the conference. In the end, we concluded that BigBlueButton would do the job if we took care not to stress it too hard.

"Taking care" meant a number of things, starting with the elimination of plenary sessions. Such sessions have never been a big part of the event; LPC is focused on smaller groups where serious discussions happen and real work gets done, so this decision did not require major changes. An opening welcome message was recorded by 2020 LPC chair Laura Abbott and made available on YouTube. Not putting 500 or more people into a single "room" side-stepped many of the scalability concerns.

Beyond that, attendees were asked to keep their webcams turned off except when they were actively participating in the discussion; one of the quickest ways to find the scalability limits on both the server and client sides is to turn a lot of cameras on. Use of the "listen-only" mode was encouraged to reduce server load; this was probably unnecessary and also runs counter to the "everybody participates" goal of the event. Live streams were set up through YouTube as a way of further reducing load and providing a backup channel if everything went up in flames. The YouTube streams turned out to be useful for a number of people who were unable to register for the conference or just found it easier to watch that way.

The smoothness with which the actual conference ran made it clear that we need not have worried quite so much. LPC sessions routinely held well over 100 people, with the kernel development in Rust session drawing over 170 participants. The servers were never seriously stressed. Things only got bumpy during the "show off your beer" session held after the official close of the event, when 20 or more attendees had their cameras on at the same time; BigBlueButton just does not work well under that sort of load.

Creating the conference experience

BigBlueButton nicely handles the task of hosting videoconferences, but does not solve the whole problem of creating an online conference event. It is meant to run behind a front-end system that provides a landing page for incoming guests, controls access to rooms, and more. It ships with a system called Greenlight that handles these tasks, but Greenlight is not really meant for hosting conferences. It cannot, for example, present attendees with a set of available rooms and tell them what is happening therein. Thus, we ended up writing our own front-end system to take care of these tasks; it fetched the schedule from the main LPC web site and produced a version that allowed attendees easy access to the appropriate rooms. The front end also provided some management screens allowing the organizers to get a picture of what was going on. For the curious, the resulting code has been released at git://git.lwn.net/lpcfe.git.

Anybody who has been to both an in-person conference and a videoconference knows that the two are not equivalent. The organizing committee worried that we have all had more than our share of videoconferences in the past few months, and that enthusiasm for a conference that looked like yet another one would be limited. We knew that we would never be able to replicate the experience of an in-person event, but we wanted to get as close as we could.

Part of that was working on a set of practices to make the sessions as productive and interactive as possible. The "cameras off except when discussing" protocol worked well in that regard. It allows attendees to focus on the people who are actually talking, and popping up a video window is a clear signal that somebody has something to add. The chat and polling systems built into BigBlueButton also helped to increase interactivity and were quite heavily used.

In-person events often create conversations that continue in the hallway, a separate breakout room, or in a pub afterward. We were unable to provide the pub experience, but could try to provide some of the rest. Several "hack rooms" were set aside for any group that wanted to use them for side discussions; those, too, were heavily used. We also encouraged the scheduling of birds-of-a-feather sessions at the last minute for topics needing further discussion; those sessions saw a lot of real work done during the conference.

In an attempt to create a "hallway" for other discussions, a separate chat forum was set up using Rocket.Chat. This, too, proved to be popular, and it also served as a support channel for attendees during the event.

An online event must also address the problem that attendees are spread across time zones around the planet. LPC ran some polls to try to find an optimal time window, but ended up running four-hour days starting at 14:00 UTC — a slot that works reasonably well for the Americas and Europe, but which is rather less comfortable for attendees in much of Asia and Oceania. The Android microconference ran an extended BOF session outside of the normal schedule as a way of accommodating some of those folks; the conference as a whole probably could have benefited from more sessions at other times.

The decision to go with four-hour days was the only rational choice, though, even though it meant turning LPC into a five-day event. Videoconferences are tiring, and few people have the patience to sit through a full day even if the time window is favorable.

All told, nobody would ever confuse an online LPC for an in-person gathering, but we were able to preserve many of the aspects that has made LPC a productive gathering over the years. Attendees seemed happy about their experience. A number of them asked for there to be some sort of video presence mode available in the future, even after in-person gatherings become possible again.

Other points of interest

Some conferences have responded to the pandemic by simply canceling for this year; these include the Linux Storage, Filesystem, and Memory-Management Summit, Kernel Recipes, and the GNU Tools Cauldron. The Cauldron, though, subsequently accepted an invitation from LPC to run a track there. The result was an intermixing of GNU toolchain developers and their users on a scale that has rarely been seen; one can only hope that it will be possible to run other events together in the future.

One need not attend many in-person events to have the dubious pleasure of watching a presenter desperately trying to get their laptop working with the projector while the session's scheduled time ticks away. Putting all of those presenters onto a new and unfamiliar platform threatened to raise that experience to a whole new level. In an attempt to avoid that, committee members put together a set of documents on how to use BigBlueButton in both the presenter and moderator roles. Then, a set of training sessions was run to drive home the basics of operating the platform and running sessions. The fact that the sessions themselves ran quite smoothly is a testament to the effectiveness of this documentation and training.

We have all had painful videoconference experiences by now where some attendees simply cannot be seen or heard. In an attempt to make LPC a little better, the committee chose, for the traditional speaker gift, a kit consisting of a headset, camera, and lights, along with a few other goodies. The kits were appreciated and heavily used with, one hopes, a corresponding improvement in audio and video quality.

The LPC committee struggles every year with the question of how many tickets should be made available. LPC is a working conference; letting it get too big makes it harder for people to find each other and runs counter to the overall goals of the event. The question takes a bit of a different form for an online event, though; the decision was made to allow more than the usual number of attendees for a number of reasons. In the end, nearly 950 people registered, making this by far the largest LPC event ever — and we still ended up turning people away.

Some people were surprised by this, clearly thinking that an online event could allow entry to an unlimited number of people. The registration cap existed for two reasons: concerns about having too many people in the "room" and the above-mentioned scalability worries. In the end, we could have allowed more people in. More people in the "room" do not inhibit the discussion, and the problem of pressing through a crowded hallway to reach the coffee or find a collaborator are not present. The servers handled the load just fine. But we had no way to know how things would work out ahead of time.

Of the nearly 950 people who registered, just over 800 attended at least one session during the event. The growing body of wisdom about online events suggests that no-show rates of up to 50% can be expected, so this is a relatively high turnout. Even for an online event, where travel is not necessary and the registration fee is minimal, people who register for LPC actually want to attend.

For the curious, the LPC online infrastructure ended up consisting of 17 virtual machines. Six of those were BigBlueButton servers; they had 32 dedicated CPUs each, which turned out to be ridiculously oversized. Half as many CPUs would still have been more than the system could use. For each BigBlueButton server there was a machine dedicated to creating the live YouTube stream for the principal room in that server. One machine held the Rocket.Chat server and one ran the front end; there was one more running an LDAP directory server providing login information to both. There was a coturn server for NAT traversal, and a monitoring host running Zabbix.

All told, the experience of running LPC as a virtual event was exhausting, but also rewarding. Despite our worries, we were able to pull off a successful event using 100% free software. Hopefully, by the time LPC 2021 comes around, it will be possible to meet in person again; should that not be the case, we will be in a much better position to run things virtually. One way or another, we'll get our community together to work out how to make free software even better.

Finally, your editor wrote this article and is fully responsible for any mistakes in it, but the work described above — along with the many other tasks required to run an event like LPC (developing the program, managing registrations and speaker passes, working with sponsors, etc.) — was done by the organizing committee as a whole. That committee was Laura Abbott, Elena Zannoni, Kate Stewart, James Bottomley, Christian Brauner, Jonathan Corbet, Guy Lunardi, Paul McKenney, Ted Ts'o, Steve Rostedt, and David Woodhouse. Thanks are due to the Linux Foundation for its support as well. The biggest thanks, of course, are owed to the attendees of LPC, without whom there would be no interesting conversations at all.