The Grumpy Editor's guide to audio stream grabbers [LWN.net]

This article is part of the LWN Grumpy Editor series.

Your editor is fortunate enough to live in a town with an excellent radio station. It is a public station, funded (mostly) by its listeners and operated (mostly) by volunteers. It is a nearly 30-year-old application of many free software concepts to the airwaves; appropriately, its name is KGNU. For those who do not live in the area, or who find the reception problematic here on the edge of the mountains, KGNU makes a set of streams available over the net; there is even an Ogg stream.

KGNU airs an incredible variety of music and public affairs programming; much of what is heard there is available nowhere else in the area. Unfortunately, some of the most interesting programs are not broadcast at times when it is convenient for your editor to listen to them. Some of the best music is late at night, and the public affairs programs broadcast during the day tends to be incompatible with the need to write LWN articles.

As a result, your editor has a strong desire to record shows of interest and listen to them at a later time. This is, of course, a classic, legal exercise of fair use rights. For years, this activity has been performed using a DAT deck, which will happily record a three-hour show without breaks. Unfortunately, this solution (1) requires somebody to push the "record" button at the right time, and (2) depends on the continued operation of an aging piece of audio equipment whose reliability was not the greatest even when it was new. It would make a lot of sense to, instead, simply record the audio stream from the net. Recording could be automated, and the result could be moved to a portable player for convenient listening.

It is not surprising that proprietary players for streaming media lack a "record" option. But, one would think, free players would provide such an obvious bit of functionality. As it turns out, however, most of the free players which can tune in network streams also lack recording capability. Whether this omission is simply a matter of other development priorities coming first or is, instead, a capitulation to the entertainment industry is not clear. Regardless of why, a Linux user who has fired up totem, amarok, or xmms to play an audio stream will not readily find a "record" option there.

There are, however, a number of options available for those who would record audio streams on a Linux system. Here are a few that your editor has found.

Recording through the sound system

Audio streams passing through the ALSA sound system are generally available to applications via a capture interface. So, in fact, almost any free recording application can be used to grab the stream as it passes through the kernel. A simple example can be made with arecord:

    arecord -f cd -d 7200 stream.wav

This command will record a stream in WAV format, automatically stopping after two hours. Other recording applications (ecasound, ardour, etc.) can also be used.

There are some downsides to this approach. Recording in this way occupies the sound system, making it impossible to listen to anything else. Changes to mixer settings can affect the recording. Depending on the sound hardware in use, the system might have trouble simultaneously playing an audio stream and recording it. And, regardless of other problems, this solution involves several transformations to the audio stream between the network interface and its eventual resting place on the disk. Your editor would rather store the stream as it was received from the source.

ogg123

If the stream of interest is in the Ogg Vorbis format, the ogg123 tool can be used to capture it. A command like this will do:

    ogg123 -d wav -f stream.wav http://stream-url

With a second option (-d oss), ogg123 can simultaneously play the stream and record it to the disk file. There is an option for specifying the duration of the recording (useful for grabbing shows via a cron job), but it did not work properly on your editor's system.

For whatever reason, ogg123 lacks the ability to save an Ogg stream directly to disk - it must convert it to the uncompressed WAV format first. One can always re-encode the stream - at recording time using a pipe, even - but putting an audio stream through a second round of lossy encoding cannot do it any good. It would be much nicer to just save the stream directly to disk.

wget

If something exists on the net, there is a way to tell wget to fetch it. Audio streams are no exception; running:

    wget http://stream-url

will do the trick. No transformations will be applied to the stream - it will be saved as received from the source, which is as it should be. On the other hand, wget is not really designed with streams in mind. In particular, it lacks an option for setting the recording period, making it a bit harder to run in an automated mode - though a couple lines of shell scripting suffice to take care of that problem.

mplayer

While most streaming media players lack a record option, mplayer is a notable exception. A stream can be recorded with a command like:

    mplayer -dumpstream -dumpfile stream.ogg http://stream-url

Of course, streams in just about any format can be recorded in this manner; mplayer will save the stream as it receives it.

The list of options understood by mplayer easily qualifies as one of the longest for any application anywhere on the planet. A definitive study could require some months, but, as far as your editor can tell, none of those options tell mplayer how long it should run. As with wget, that omission makes mplayer a little harder to use in an automated mode.

Some distributions are more enthusiastic about including mplayer than others. Packages for almost any distribution are readily available, however, to those who search for them.

streamripper

The definitive tool for capturing streams may well be streamripper. This utility will grab a stream and store it to disk, possibly splitting it into separate tracks as it goes. It can function as a relay, making it possible to listen to a stream as it is being recorded - or to distribute a stream around an internal network. In its simplest form, streamripper is run as:

    streamripper http://stream-url

Options exist to limit recording time, control separation into tracks, establish a stream relay, and automatically discard advertisements. There are graphical frontends for GNOME (streamtuner) and KDE (KStreamRipper). There is also an amarok plugin available.

To conclude

From your editor's point of view, streamripper is the right tool for this job. It is the only one which was designed for the purpose of capturing audio streams in their original format. In a pinch, wget will do the job, as will mplayer. Employing a huge tool like mplayer for this purpose feels somewhat like using a nail gun to hang a calendar, however.

For now, we are lucky in that there are quite a few high-quality streams which can be time-shifted and enjoyed in this manner. Unfortunately, the future looks to be made up of DRM-encrusted streams and no access for users of free software. No fair use rights. If we want to live in a world where broadcast streams are accessible with free tools and developers of stream players are not afraid to add "record" buttons, we need to ensure that the legal climate does not become more hostile than it is already. Otherwise, finding a good stream capture tool could become much harder than it is today.

(Log in to post comments)

mplayer - for those "proprietary" media formats

Posted May 10, 2006 11:35 UTC (Wed) by dowdle (subscriber, #659) [Link]

Ok, this article was geared for AUDIO but let's not forget that much of it applies to VIDEO too.

A lot of media you will find out there is in RealPlayer, Windows Media Player and various other formats... often with an rtsp:// or an mms:// type URL. I have found mplayer to be very handy for capturing almost every stream you can find... assuming you have installed all of those questionable codecs from questionable sources.

Play Back: Once you have that proprietary media, how are you going to play it? With mplayer, xine, or vlc of course.

Getting the URL: mplayerplug-in's "Copy URL" feature (or however it is worded) is very handy when pages attempt to hide the real URL of the media. I often start the media with the browser plugin, copy the URL and them dumpstream it with mplayer.

Re-encoding: For converting those proprietary media formats into something more desirable, mencoder and/or ffmpeg are very handy. Getting to know some of mencoder's and ffmpeg's command line options and then making a cheat sheet keeps one from going crazy trying to read those techie man pages. There are various command line tricks for joining multiple media files into one... which is very handy when something like... say... PBS Frontline breaks their shows into 4 to 6 pieces. Ever want to extract the audio from a video and make it into an .mp3 or .ogg? You can once you learn the command line flags. avidemux2 is handy for converting to ogm as well as performing various operations (cropping, FPS change, etc) in a reasonable GUI... although avidemux2 can be a little picky on poorly encoded original files.

modified recorder with simultaneous encoding and mp3

Posted May 18, 2006 14:57 UTC (Thu) by nealmcb (subscriber, #20740) [Link]

Thanks, Stevan. My version of mplayer doesn't seem to take the
":file" modifier to the "-ao pcm" option, so I used the "-aofile"
option instead.

I also didn't like the idea of filling up my disk with the .wav
data until the recording was done, so I changed it to run the
encoder in the background. I couldn't do that via a pipeline
since mplayer doesn't seem to be able to write to stdout, so I
had to connect them via a named pipe. Here is my modification,
which also produces mp3 rather than ogg since that is what my
nokia tablet knows how to play by default.

----
#!/bin/sh
# record a stream with mplayer, e.g. realaudio, saving it as an ogg vorbis file

# todo: trap terminate signal and kill encoding process, remove named pipe
# consider option for dumping raw format via -dumpstream -dumpfile,
# and later conversion

# based on Stevan's script at http://lwn.net/Articles/182954/
# See also scheck's 'srec' script there

usage="$0 url name"

URL=$1
NAME=`echo $2 | tr " " _`.mp3
FILE=$RANDOM
QUALITY=48

mknod $FILE.wav p

(lame --quiet -V6 --vbr-new --resample 22 -m m $FILE.wav $FILE.mp3) &

# (oggenc --quiet -b $QUALITY $FILE.wav) &

mplayer -cache 32 -nojoystick -nolirc -prefer-ipv4 -playlist $1 -ao pcm -aofile $FILE.wav -vc dummy -vo null

wait # wait for encoding process to finish

mv $FILE.mp3 $NAME
rm $FILE.wav

modified recorder with simultaneous encoding and mp3

Posted Mar 7, 2007 13:22 UTC (Wed) by devasura (guest, #43937) [Link]

some mplayer options seem to be decprecated, use:
mplayer -cache 32 -nojoystick -nolirc -prefer-ipv4 -playlist $1 -ao pcm:file= $FILE.wav -vc null -vo null

modified recorder with simultaneous encoding and mp3

Posted Mar 7, 2007 13:28 UTC (Wed) by devasura (guest, #43937) [Link]

ignore the above comment, didnt read the full article
sorry

modified recorder with simultaneous encoding and mp3

Posted Mar 8, 2007 10:27 UTC (Thu) by devasura (guest, #43937) [Link]

i replaced oggenc line by:
(cat $FILE.wav | oggenc --quiet -b $QUALITY - | tee $FILE.ogg | mplayer -cache 32 - > /dev/null) &
to enable playing while the stream is being recorded

A bloated script for time-scheduled audio stream recordings

Posted May 18, 2006 6:15 UTC (Thu) by scheck (subscriber, #4447) [Link]

I wrote a small script for time-scheduled stream recordings and bloated it by extensive use of comments and a small "station database", which holds my favourite German radio stations.

A mere echo 'srec myrockstation 60 "Iron Maiden Live in Lederhosen"' | at 22:00 Sat would take care of your desired one-hour-show on coming Saturday at 10pm while you go out with friends who listen to Madonna... ;-)

BTW: if you adapt the mplayer command and the assignment of extensions, the script might also work with videos...

#!/bin/sh
#
# srec - stream recording
# v0.01 2006-03-21 torsten.scheck@gmx.de
#
# Copyright is hereby assigned to the Public Domain.
#
# This script relies on wget or mplayer, depending on
# the stream format.

# parameter defaults
station=${1:-"NoStation"}
length=$((${2:-60} * 60))
name=${3:-"Unnamed"}

# time period in seconds, after which the dumping process is checked
checktime=10

# choose station from database and build station list
url=''
list=''
add() {
  list="$list"$(printf "%-8s %-3s %-25s %s" "$list_station" "$list_ext" \
               "$list_info" "$list_name")"\n"
  if [ "$station" == "$list_station" ]; then
    ext=$list_ext
    info=$list_info
    url=$list_url
  fi
}
############################################################
list_station="dlf"
list_ext="ogg"
list_name="Deutschlandfunk"
list_info="96 kbps, variable, stereo"
list_url="http://dradio-live.ogg.t-bn.de/dlf_high.ogg"
add
list_station="dlfmp3"
list_ext="mp3"
list_name="Deutschlandfunk"
list_info="48 kbps, constant, mono"
list_url="http://dradio-live.mp3.t-bn.de/dlf_live"
add
list_station="dlfwma"
list_ext="wma"
list_name="Deutschlandfunk"
list_info="48 kbps, constant, stereo"
list_url="mms://dradio-live.wm.t-bn.de/live/dlf/dlf"
add
list_station="dlrk"
list_ext="ogg"
list_name="Deutschlandradio Kultur"
list_info="96 kbps, variable, stereo"
list_url="http://dradio-live.ogg.t-bn.de/dkultur_high.ogg"
add
list_station="contra"
list_ext="ra"
list_name="SWR Cont.Ra"
list_info="44 kbps, stereo"
list_url="rtsp://213.254.239.61/farm/*/encoder/swr/contra/livestream.rm"
add
############################################################

filename="${name}_${station}_$(date +%Y%m%d_%H%M).${ext}"
logname="${filename%.*}.log"

# display help when station name is not found or wasn't provided
if [ -z "$url" ]; then
  echo "Usage: $0 STATION [LENGTH in minutes] [NAME]
Saves an audio stream of a given internet radio STATION to a file in 
the current working directory. 

LENGTH defaults to 60 minutes and NAME defaults to 'Unnamed', so the 
resulting file name looks like this: ${filename}xxx
The download process is logged to a separate file using the suffix 'log'.

This script is best used with the Unix tool 'at' for time-controlled
recordings. For example, try this to record the 55-minutes-show 
'Querkoepfe' on the station 'Deutschlandfunk' next Wednesday at 21:05
including some 'safety minutes' before and after the show:
> echo 'srec dlf 60 Querkoepfe' | at 21:03 Wed
Mind your time zone and get the list of your locale's supported 
abbreviations for the time specification with:
> locale -c LC_TIME

Available stations:" >&2
  echo -ne "$list" | sort >&2
  exit 1
fi

# use 'wget' as default tool and 'mplayer' for special streaming formats
case "$ext" in
  ra|wma) util="mplayer";;
  *) util="wget"
esac
# quit if needed tool is not installed
type "$util" >/dev/null 2>&1 || { echo "Please install '$util'."; exit 1; }

# all output is redirected to logfile for easier 'at' usage
exec >> "$logname"
exec 2>> "$logname"

until
  # start background process to dump stream to file 
  if [ "$util" == "mplayer" ]; then
    mplayer -vo null -vc dummy -dumpstream -dumpfile "$filename" "$url" &
  else
    wget -nv -O "$filename" "$url" &
  fi
  # keep process id of bg process
  mypid=$!
  # in case of SIGINT and SIGTERM: kill bg process before exit
  trap 'kill $mypid; exit;' 2 15
  # wait until bg process figured out if stream is available
  # e.g. "ERROR 504: Server Full" ends the bg process
  sleep $checktime
  # check if process is still here, i.e. the stream is being downloaded,
  # or if the recording length is reached 
  kill -0 $mypid >/dev/null 2>&1 || [ $length -le 0 ]
do
  echo "Server problem, another try: $(date '+%Y%m%d %H%M%S')" 
  # decrease recording length by checktime; in the case of a transient
  # problem the remaining length of the stream will be saved
  length=$((length - checktime))
  # retry download with another until loop
done

if [ $length -gt 0 ]; then 
  # wait recording length
  sleep $length
  # kill bg process and wait for its termination
  kill $mypid 
  wait 
else
  # the recording length had been reached by trying just to get 
  # the download started
  # an unrecoverable problem prevented the desired download
  echo "Aborted." 
fi
echo "$filename END: $(date '+%Y%m%d %H%M%S')"