Development [LWN.net]

Using Linux to manage a large audio collection

Recently, your author decided to dig into a long-delayed project, the consolidation of a large collection of music onto a hard drive-based archive. Over the years, a large collection of music has been built up, the majority of which is live concert recordings from various tape trading networks. Prices of hard drives in the several hundred gigabyte size range have been steadily falling, making an online audio archive possible and affordable.

Music recording technology has gone through an amazing number of technology changes in the last 30 years. First there were reel-to-reel tapes, then audio cassettes. VHS-HiFi was a short-lived medium that offered improvements over cassettes until the audio DAT tape showed up. After that was CD-R media. Add to that vinyl records, commercial CDs, sound tracks from video tapes, and live recordings from multi-track digital recorders. Recently, a large number of BitTorrent sites have been good sources of live music recordings.

Your author is somewhat picky about audio standards, although he routinely shuns audiophile audio hardware. The Microsoft .wav file format seemed like the universal standard to use as an archive medium. The 44.1K sampling rate was chosen due to its compatibility with audio CDs. Compressed formats like Ogg Vorbis and MP3 have their uses for portable players, but for archiving purposes, .wav is for the most part, the best sounding and most universally workable standard.

Taking that one step further, .wav files can be bit-for-bit converted to and from FLAC, the Free Lossless Audio Codec. FLAC typically squashes music by about 50% of its original size, and .flac files can be played directly with music players such as XMMS. It is trivial to re-convert .flac files to .wav for burning CD-Rs.

The standard Linux filesystem was chosen as way of storing the archive. It offers a wealth of handy command-line utilities for management, and GUI-based interfaces for those who prefer the that mode of operation. Some conventions were chosen for representing the music from a typical live concert. Each concert gets stored in a unique directory named ArtistYYYYMMDD. Individual songs were stored in their own .wav files with a similar naming scheme: ArtistYYYYMMDDd#T##.wav where YYYYMMDD is the date, d# is the (optional) disk number from a multi-CD set and T## is the track number. Additionally, a file called ArtistYYYYMMDD.txt can be included to contain a textual description of the audio source, processing information, song lists, and other information. It would be nice to have a meta-data file such as an XML file that contains more information in a computer readable format, but that's for the future.

A big collection of CD-burning source material was recently rescued from an old computer system and a big pile of backup tapes from the same machine. The majority of this data consisted of 1-2 GB .wav files that were created by copying audio DATs into a CMI8732-based sound card with a lossless S/PDIF digital audio interface. The audio stream was converted to .wav files with the obsolete SoundRecorder utility, or Cinelerra. Sound from analog sources was fed into the computer using an external Flying Cow Analog to S/PDIF converter.

Many of the audio DATs were recorded with a 48Khz audio sampling rate; those were converted to 44.1Khz files with SoX. Most of the source material was stereo volume normalized with Ecasound and some custom audio scripts. Volume normalization is a bit of an art, it usually works best on large parts of the source .wav file, broken up by concert sets, or where the person running the recorder tweaked the recording level. Normalizing groups of songs gets rid of annoying volume changes from song to song.

Although Cinelerra is primarily a video editor, it has the critical ability of being able to digest a 2GB audio file without choking. It also has the ability to mark audio edit points throughout the source material, and bulk-render individual .wav files as marked by the edit points.

After all of that, the music archive is showing many signs of improvement. All of the long gaps and audience chatter has been removed from the source material. The volume of the .wav files is fairly constant from concert to concert. The songs are now accessible individually. Your author was lucky in that he discovered how poor the CD-R medium is for long time archiving before he deleted his source .wav files. CD-R media degrades with heat, dirt, and repeated handling.

Now that the music archive is coming together, some big advantages are beginning to surface. It is possible to copy the entire archive to another computer with a one line ssh/tar command. This is extremely powerful for backing up the data or copying sections to a friend's computer. A spare computer can easily be retrofitted with a large hard drive, then turned into a music library audio appliance. Although not as portable as an mp3 player, the entire archive can be copied to a laptop for listening away from home.

This is very much a work in progress, it is also a process that will never be finished. The archive is up to 45Gb and is growing daily. Some software remains to be written. A random music player should be easy to write with just a few lines of Python code. That could be extended to include more advanced features such as noting song groups that should always be played together, and skipping files that contain short spoken segments such as a band introduction. In the source material, files are occasionally split into two pieces due to the editing out of a bad section in the middle, your author is still searching for a way to join two .wav files into one to fix that problem.

Linux and the wide variety of open-source tools have made this entire process a breeze, if somewhat time consuming. Audio recording has gone through a series of diverging technologies, this distillation effort has reversed that trend.

Comments (19 posted)

System Applications

Database Software

MySQL 5.1.7-beta has been released

Version 5.1.7-beta of the MySQL database is available for testing. "This is the first published Beta release in the 5.1 series. All attention will continue to be focused on fixing bugs and stabilizing 5.1 for later production release."

Full Story (comments: none)

pgAdmin III v1.4.2 Released

Version 1.4.2 of pgAdmin has been released. "The pgAdmin Development Team are pleased to announce the release of version 1.4.2 of pgAdmin, the Open Source administration and development platform for PostgreSQL 7.3 and above. pgAdmin can be run on Linux, FreeBSD, Mac OSX, Solaris and Windows."