Rob Savoye on the Cygnal rich media server

On June 11, 2009 Rob Savoye spoke to the Boulder Linux Users Group about the Cygnal rich media server project.

Cygnal is part of the Gnash SWF movie player project and development is being funded by Open Media Now (OMN). Open Media Now projects include Gnash, Cygnal and Ming, an SWF output library and PHP module. OMN is particularly focused on performing legal, clean-room reverse engineering of proprietary protocols. Supporters of the project include the Electronic Frontier Foundation, the Software Freedom Law Center, the Free Software Foundation, and notable individuals such as John Gilmore, Bob Young, and Mark Shuttleworth, along with a community of developers.

Rob discussed the underlying concept of rich media, also known as interactive media. Unlike basic streaming media, rich media is interactive, involving the bidirectional transport of complex objects. Typical uses include video conferencing, electronic whiteboards, and other types of groupware.

Cygnal is loosely aimed at being a replacement for Adobe's commercial Flash Media Server, with enhancements. The motivation behind Cygnal was to create a 100% free video conferencing server. Cygnal development was started because there were no really good free software rich media server projects available. Compared to the commercial offering, the project aims to be more secure, lack a central point of control, and have a greater respect for user privacy.

Cygnal supports a variety of patent-free and proprietary CODECs and protocols including HTTP, Adobe's proprietary Real Time Messaging Protocol (RTMP), RTMPT, and ActionScript protocols as well as the free Ogg Vorbis audio and Ogg Theora video protocols from the Xiph.org project. Cygnal also aims to support PHP and Python inside of a server sandbox environment.

Some of the Cygnal features that Rob highlighted include built-in clustering and load balancing and support for Oggz chopping so that one could, for example, edit video on a distant server by remote control from a cell phone. For a complete feature list, consult the Cygnal web site. Rob pointed out inefficiencies with streaming video servers, such as YouTube, that a rich media server can greatly improve upon. If you want to move around a video randomly, or play just the last few seconds, with YouTube it is necessary to load the entire video first. The bidirectional capability of a rich media server would allow a finer level of control over which data is sent over the network.

Cygnal supports statistics gathering for tuning and optimizing its behavior. It is possible to connect to the server's system console via tcp, and the plan is to be able to remotely dump statistics while the server is operating. Rob emphasized that there was a fine line between gathering statistics and invading privacy, so the project would aim to protect privacy first.

A large amount of work has been done on the legal clean-room reverse engineering of the RTMP protocol and the ActionScript class library. RTMP has a lot of advanced capabilities including the transfer of ActionScript objects, the ability to drop frames when network congestion is detected, and the ability to seek within a video stream. All of this functionality makes decoding the protocol a fairly difficult job. RTMP is not publicly documented, so reverse engineering is the only way to gain access to its inner workings.

A number of special techniques and limitations were used in the reverse engineering process in order to keep everything legal. It is critical that people doing the reverse engineering don't install any Adobe software so as to avoid agreeing to the EULA. The EULA forces the user to agree not to redistribute any software relating to decoding Adobe's proprietary protocols. It is legal to have a third party install the commercial software and agree to the EULA, then have a developer sniff network packets and sift through the voluminous hex dumps. Adobe has already shown their legal teeth, they recently sent a DMCA letter to SourceForge concerning the rtmpdump utility (documented on FlashComGuru). Rob described the reverse engineering process as sneaky, brutal, and fun.

At the heart of Cygnal is the network engine. It must support multiple threads for incoming connections. As soon as the maximum number of threads has been reached, the engine multiplexes the i/o operations through the existing threads. This is currently an area of experimentation and flux within the project; time and experience will eventually reveal the most efficient approach to achieving good operation. Rob emphasized the network engine's use of zero-copy techniques; messages are put into a queue and pointers are passed around, resulting in very good performance. Interestingly, Rob mentioned that the PowerTop utility was one of the more useful tools that he used for optimizing the code.

Rob pointed out a few weaknesses in the Linux kernel and libraries that required new implementations of existing software. He pointed out that the Linux Asynchronous I/O library uses three threads per connection and was a source of much inefficiency. Also, using jemalloc [pdf] (A Scalable Concurrent malloc Implementation for FreeBSD) instead of malloc can improve the server's speed by up to 15% on multi-core machines.

The Cygnal project is in need of a wide variety of help including performing language translations, testing and feedback, bug reporting, documentation, build farm maintenance, and donations of cash (and beer). Volunteers are encouraged to help out, or, at least, to send Rob some free beer.