An introduction to projectM

Many people have seen music visualizations before, whether in a music player on their computer, at a live concert, or possibly on a home stereo system. Those visualizations may have been generated using the open-source music-visualization software library that is part of projectM. Software-based abstract visualizers first appeared along with early MP3 music players as a sort of nifty thing to watch along with listening to your MP3s. One of the most powerful and innovative of these was a plugin for Winamp known as MilkDrop, which was developed by a Nullsoft (and later NVIDIA) employee named Ryan Geiss. The plugin was extensible by using visualization equation scripts (also known as "presets").

Sometime later, a project to implement a cross-platform, MilkDrop-compatible, open-source (LGPL v2.1) music visualizer began: projectM. The main focus of the project is a library (libprojectM) to perform visualizations on audio data in realtime—using the same user-contributed script files as MilkDrop—along with reference implementations for various applications and platforms. The project, which began in 2003 and was first released in 2004, is of interest to many for its creative and unique visuals, its use by media-player projects, and its interesting design and features. After years of development and contributions, the project stalled, but now there are efforts to rejuvenate and modernize the code.

How it works

LibprojectM is written in C++ and has a simple interface. The host application is responsible for creating an OpenGL context for the library to draw in, and then feeds in PCM audio data. From the audio data, the library extracts bass, mid, and treble amplitudes using the standard fast Fourier transform (FFT) that all audio visualizers use; it also attempts to perform beat detection to make the visuals synchronize with the music better. Each frame, the host application simply asks projectM to render its current visualization; it does that by drawing into the current OpenGL context. Here is the basic idea:

    renderFrame() {
	glClear(...);

	projectM->pcm()->addPCMfloat(pcmData, 512);
	projectM->renderFrame();

	flipOpenGLBuffers(...);
    }

When projectM renders a frame, it does so by passing the features extracted from the audio data to equations drawn from the currently selected MilkDrop preset file. There are two sets of equations (described in detail here), one set is evaluated per-frame and describes the shape, rotation, size, and colors of the waveform being drawn from the FFT data, and the other set is per-vertex equations for more complex transformations and deformations. These per-vertex equations are calculated for each point in a mesh (the points are the vertices), and then interpolated to the current screen display size, allowing less-powerful computers to reduce the number of vertices in order to sacrifice some accuracy for speed. MilkDrop preset files can also contain DirectX GPU shader programs, allowing for more complex programs and faster computation.

There are a number of other features for projectM, in varying degrees of completeness, including the ability to render to an OpenGL texture, better support for GPU shaders, "native" presets written in C++, preset playlists, the ability to control the speed of transitions between presets, and text overlays with Freetype GL.

Work needed

While the core of libprojectM works well and performs its task dutifully, there is a great deal of work still needed to make it a modern and usable piece of software. The development stalled for a period of several years until renewed interest restarted efforts to continue improving it.

ProjectM's OpenGL drawing is done with what are known as legacy "immediate-mode" calls. This OpenGL interface is one where the application transfers vertex information for each frame via API calls to the GPU. Drawing a triangle looks something like this:

    glBegin(GL_TRIANGLES);  // we are going to feed in color and geometry to generate triangles

    glColor3f(1.0f, 0.0f, 0.0f);  // draw red
    glVertex2f(0.0f, 0.0f);  // draw three vertices in object space
    glVertex2f(25.0f, 25.0f);
    glVertex2f(50.0f, 0.0f);

    glEnd();  // finished drawing triangles

This is inferior to the newer API where vertices are uploaded to the GPU in the form of vertex buffer objects (VBOs); in contrast, the immediate-mode calls need to send the geometry every frame from the CPU with numerous API calls. The geometry of a VBO only needs to be sent to the GPU once if the geometry doesn't change. VBOs can be colored and deformed through the use of shaders, which are GPU programs that operate on vertices and pixel colors.

The modest number of immediate-mode calls that projectM makes need to be converted to VBO code. This will not only improve performance but it is also needed for compatibility with newer OpenGL implementations, namely OpenGL ES, which is the specification for embedded systems. Until the OpenGL calls are updated, it is unlikely an open-source projectM will ever be able to run on an embedded system, as many have desired. It's worth noting that this work was done by one of the main authors of projectM and a previous maintainer, but he has not chosen to share his modifications with the project.

MilkDrop was heavily Windows-based, implemented with DirectX, Win32 APIs, and assembler. ProjectM did a good job of replicating the functionality in a cross-platform manner but one DirectX-specific piece remains: the shader code in the preset files. Some presets can contain GPU shader programs as mentioned previously. Because they were written for MilkDrop, they are in HLSL, a shader language for DirectX. Support for HLSL was provided in projectM by NVIDIA's Cg toolkit, but that has long been deprecated and is unsupported. Either manual or automatic conversion (possibly using something along the lines of HLSL2GLSL for Unity) needs to be added along with code to compile and upload the shaders. This would greatly increase performance and capabilities, enable the most advanced presets, and drop the dependency on an out-of-date and unsupported proprietary framework.

Building with GNU Autotools

Work was done recently to replace the problematic CMake-based build system with GNU Autotools. This was done partly because of the limitations and complexity of CMake coupled with its less-ubiquitous status; Autotools is more widely known. The core library and SDL2-based example host application are building for macOS, Linux with GCC and Clang, and FreeBSD 11 without incident using a well-known build command:

    $ ./configure && make

The other reference implementations that come with the project have not been updated yet to build with Autotools so some small amount of work is needed there. It is unclear which implementations (if any) are actually in use by anyone. The switch to Autotools has caused some small disturbances with downstream package maintainers who may lump the library itself together with the reference implementations; the difference in build systems can lead to confusion about the state of the project.

There were some compile-time feature flags exposed by CMake, such as render to texture and Freetype GL support; these need to be added to the Autotools-based build. Anyone with GNU Autotools experience who wants to help finish the migration for the various implementations and feature flags is sought. Support for building on Windows would be useful as well.

Reference implementations

projectM's source repository contains many examples of host applications and libraries making use of libprojectM, including XMMS, JACK, PulseAudio, macOS iTunes, Qt, screensavers, and libsdl2. Many of these are in a questionable state, some have not been updated in approximately a decade. Work needs to be done to go through the various implementations and either update them or determine if they should get the axe. The two most recently updated were the macOS iTunes plugin, which enjoyed some recent popularity until an update caused it to stop functioning properly, with no cause determined as of yet.

The most useful and up-to-date reference implementation is based on libsdl2, which is a cross-platform media library used in many media applications and games; it is currently supported in large part by Valve. This program can read in audio from a capture device, which is a feature only recently added to libsdl2. If improvements were made, such as proper configuration file support (already implemented in libprojectM), more keyboard-input support, and adding some sort of basic user interface, the projectM SDL application would make a nice cross-platform standalone audio-visualization program.

Higher-level language bindings, such as for Python, could be also useful.

Community involvement

Of course, projectM is an open-source project like any other. It's been put together over the years by contributions from developers who like the project and find it useful, either for getting stoned and watching trippy visuals with their dubstep MP3s, for integrating into media players, or for projecting at live music shows. There is interest in the project, but more developers and contributors are needed to modernize and polish it. That will make it easier for other open-source projects to include a powerful and versatile music-visualization library.

Contributors are welcome to come file issues and send pull requests at the GitHub repository. There is a legacy SourceForge page that is still up, but it is not where the development is happening. While there are plenty of useful features inside of projectM there is still lots of work to be done to make the code base usable by end users as well as downstream maintainers, portable to new platforms such as embedded and web, and keeping pace with changing libraries and APIs.

[Mischa Spiegelmock is the current maintainer and a small-time contributor to projectM.]