Notes from the LF End User Summit

The End User Summit draws a different crowd than any other event. Well-known Linux developers are present, certainly, but they do not form the majority of the crowd; they are, instead, strongly outnumbered by representatives of banks, insurance companies, and financial firms. Old conference T-shirts are far outnumbered by suits and ties in this crowd. The End User Summit, in other words, caters to enterprise distribution customers and others who are using Linux in high-stakes situations - even a major stock exchange which has based its operation on Gentoo. It makes for an interesting combination of people and a unique set of conversations.

One speaker was Brian Clark from the New York Stock Exchange. NYSE's systems run under high pressure and tight constraints. They process some three billion transactions per day - more than Google does - and those transactions need to execute in less than one millisecond. Customers can switch to competing exchanges instantly and for almost no cost, so if NYSE's systems are not performing, its customers will vanish. A typical trading day involves the processing of 1.5TB of data; some 8 petabytes of data are kept online. And this whole operation runs on Linux.

NYSE is highly concerned with software quality and security; they are subject to thousands of attacks every day. Downtime is to be limited to 90 seconds per year. All told, Linux has worked very well in this setting. NYSE had some requests, though, including the increasingly common desire for a way to move everything except a specific application off of a given core. Brian requested a way to lock a process's memory in place - a functionality which mlock() would appear to have provided for many years. He would also like a non-disruptive way to measure latencies, especially in the network stack.

In the end, he says, NYSE likes Linux because of the community which stands behind it - an interesting position given NYSE's rather low profile in that community. One place where it was suggested NYSE could help would be to advise the developers on the best placement of tracepoints into the network stack to yield the sort of latency measurements they would like to see.

Al Gillen of IDC is a common presence at this sort of event; he gave a chart-heavy talk on how IDC expects things to go in the server marketplace. The outlook for Linux server shipments would appear to be bright. One interesting tidbit from the talk: Linux server shipments will be growing strongly in the coming years, while Unix will be declining. That means that, in 2013, the Linux market looks likely to reach half the revenue value of the Unix server market. Unix may be suffering, but there's still a lot of money being spent on it.

Anthony Golia of Morgan Stanley discussed the use of Linux there; Morgan Stanley has been heavily using the operating system for several years now, and is running it on tens of thousands of systems. It was, he says, a bit of a rough start, but Morgan Stanley learned that the community "lends itself well to partnership." The company figured out how to send fixes back upstream and has experience the "warm fuzzy feeling" that comes with getting fixes merged. In recent times they are finding far fewer bugs and are quite happy with the choice to go with Linux.

Anthony had some requests too, beginning with support for TCP offload engines. What Morgan Stanley really needs, though, is shorter network latencies. Trades are dependent on getting orders in quickly in response to events, and latencies work against that goal. They would like a way to generate long-term statistics of a process's memory use, mostly as a way of knowing whether it's safe to load more work onto a specific server. There was also a request for better coordination between distributors and hardware manufacturers, yielding support for new hardware as soon as that hardware is available.

Jeffrey Birnbaum of the Bank of America led a session on shortcomings he sees with Linux at this time. In particular, Jeffrey anticipates a future dominated by increasing availability of fast CPUs and the growing influence of solid-state storage devices. The world is changing, and he worries that Linux is not changing quickly enough to keep up with it. Technology is improving quickly, he says, and the kernel is holding users back.

Specific problems include latency in the network stack and the ability of networking to make use of large numbers of CPUs. TCP, he says, is not scalable, but it wasn't clear where the problems are. One request that was clear was a means by which messages could be sent to multiple destinations with a single system call - something akin to the proposed sendmmsg() system call. He suggested that the time has come to move beyond POSIX interfaces - he is a fan of Ulrich Drepper's event interface proposal - and that the use of protocols like SATA to talk to solid-state storage is a mistake. There was also some discussion about difficulties getting a scalability problem with the epoll_wait() system call fixed.

Perhaps the clearest point to emerge from this session is that users like Jeffrey need a solid channel to communicate with the development community about their needs and frustrations. One would think that this would be an ideal role for enterprise distribution vendors to fill; indeed, in the following session, Novell's Carlos Montero-Luque described the session as a great advertisement for commercial distributions. But, for whatever reason, those distributions do not appear to be filling that role in this case.

Carlos, along with Red Hat's Brian Stevens, talked about the future as the distributors see it. There was lots of talk on the value of Linux on mainframes, which seems to be of great interest to this user community currently. Interestingly, Brian noted that Red Hat is not entirely sure that the success which has been achieved with Linux can be replicated at other levels; the JBoss development community, for example, is nearly 100% Red Hat employees.

On the subject of unpaid Linux, Brian claimed that these deployments were "fantastic." Anything which grows the overall market can only be good for the participants therein. Carlos had some darker comments about how unpaid Linux is not "free," and that it will always be paid for in some other way.

[PULL QUOTE: Everybody was afraid of being sued and ending up on the front page of the Wall Street Journal, so outright prohibitions on the use of open source were common. END QUOTE] Tim Golden is a manager at a high-profile American bank; in his talk on "the changing role of enterprise open source," though, he was clear to point out that he was speaking only for himself. This talk started with the relatively early days, when companies like banks saw open source as being far too risky to use. Everybody was afraid of being sued and ending up on the front page of the Wall Street Journal, so outright prohibitions on the use of open source were common.

There were a couple of intermediate steps, including one where managers came to the radical conclusion that the submission of bug fixes did not deprive a company of its Valuable Intellectual Property. During this time, fears about the use of open source faded considerably, and companies increasingly decided that they could tolerate whatever risk remained - at least in "high value" situations.

The current situation is heavily affected by the financial crisis; financial companies have realized that they must find a way to be competitive with far less money. This understanding has helped to usher in the "open source software as a strategy" era, with companies setting up formalized management programs for open source. An interesting thing is happening in some companies as they go through this process, though: executives are figuring out that it's hard to drive open-source projects from the back seat. They are also coming to the conclusion that participation in development projects is not as disruptive as they had once thought.

So now these companies are beginning to dip their toes in the water and look at ways to participate. There are lots of options, ranging from simple cash contributions - which don't create any real linkage with the community - through to investments in companies and "intellectual property contributions." Eventually, says Tim, we'll start to see something that was once unthinkable: development projects being run by end users.

That last statement maybe reveals something about how these companies see free software. To them, projects run by end users are a new, scary, and exotic thing. But your editor would submit that almost every development project of interest is run by end users. The developers who came together to create the Linux kernel weren't working for others. The group that pulled together their patches and released "a patchy" server were planning to deploy that server (now "Apache") themselves. As end users in the financial industry start to run projects aimed at meeting their own needs, some of those projects, at least, should prove equally successful.

There is no need to convince the financial industry that free software can benefit its operation; they have understood that for a few years now. Convincing this industry that contributing to the software it uses makes sense has been somewhat harder. It would appear that this message is starting to be heard, and companies in this industry are beginning to look for ways to reach out to the development community. Events like the End User Summit seem like an ideal way to facilitate communication between the existing development community and its future members; it is a learning experience for everybody involved.