|
|
Log in / Subscribe / Register

The Grumpy Editor's guide to audio stream grabbers

This article is part of the LWN Grumpy Editor series.
Your editor is fortunate enough to live in a town with an excellent radio station. It is a public station, funded (mostly) by its listeners and operated (mostly) by volunteers. It is a nearly 30-year-old application of many free software concepts to the airwaves; appropriately, its name is KGNU. For those who do not live in the area, or who find the reception problematic here on the edge of the mountains, KGNU makes a set of streams available over the net; there is even an Ogg stream.

KGNU airs an incredible variety of music and public affairs programming; much of what is heard there is available nowhere else in the area. Unfortunately, some of the most interesting programs are not broadcast at times when it is convenient for your editor to listen to them. Some of the best music is late at night, and the public affairs programs broadcast during the day tends to be incompatible with the need to write LWN articles.

As a result, your editor has a strong desire to record shows of interest and listen to them at a later time. This is, of course, a classic, legal exercise of fair use rights. For years, this activity has been performed using a DAT deck, which will happily record a three-hour show without breaks. Unfortunately, this solution (1) requires somebody to push the "record" button at the right time, and (2) depends on the continued operation of an aging piece of audio equipment whose reliability was not the greatest even when it was new. It would make a lot of sense to, instead, simply record the audio stream from the net. Recording could be automated, and the result could be moved to a portable player for convenient listening.

It is not surprising that proprietary players for streaming media lack a "record" option. But, one would think, free players would provide such an obvious bit of functionality. As it turns out, however, most of the free players which can tune in network streams also lack recording capability. Whether this omission is simply a matter of other development priorities coming first or is, instead, a capitulation to the entertainment industry is not clear. Regardless of why, a Linux user who has fired up totem, amarok, or xmms to play an audio stream will not readily find a "record" option there.

There are, however, a number of options available for those who would record audio streams on a Linux system. Here are a few that your editor has found.

Recording through the sound system

Audio streams passing through the ALSA sound system are generally available to applications via a capture interface. So, in fact, almost any free recording application can be used to grab the stream as it passes through the kernel. A simple example can be made with arecord:

    arecord -f cd -d 7200 stream.wav

This command will record a stream in WAV format, automatically stopping after two hours. Other recording applications (ecasound, ardour, etc.) can also be used.

There are some downsides to this approach. Recording in this way occupies the sound system, making it impossible to listen to anything else. Changes to mixer settings can affect the recording. Depending on the sound hardware in use, the system might have trouble simultaneously playing an audio stream and recording it. And, regardless of other problems, this solution involves several transformations to the audio stream between the network interface and its eventual resting place on the disk. Your editor would rather store the stream as it was received from the source.

ogg123

If the stream of interest is in the Ogg Vorbis format, the ogg123 tool can be used to capture it. A command like this will do:

    ogg123 -d wav -f stream.wav http://stream-url

With a second option (-d oss), ogg123 can simultaneously play the stream and record it to the disk file. There is an option for specifying the duration of the recording (useful for grabbing shows via a cron job), but it did not work properly on your editor's system.

For whatever reason, ogg123 lacks the ability to save an Ogg stream directly to disk - it must convert it to the uncompressed WAV format first. One can always re-encode the stream - at recording time using a pipe, even - but putting an audio stream through a second round of lossy encoding cannot do it any good. It would be much nicer to just save the stream directly to disk.

wget

If something exists on the net, there is a way to tell wget to fetch it. Audio streams are no exception; running:

    wget http://stream-url

will do the trick. No transformations will be applied to the stream - it will be saved as received from the source, which is as it should be. On the other hand, wget is not really designed with streams in mind. In particular, it lacks an option for setting the recording period, making it a bit harder to run in an automated mode - though a couple lines of shell scripting suffice to take care of that problem.

mplayer

While most streaming media players lack a record option, mplayer is a notable exception. A stream can be recorded with a command like:

    mplayer -dumpstream -dumpfile stream.ogg http://stream-url

Of course, streams in just about any format can be recorded in this manner; mplayer will save the stream as it receives it.

The list of options understood by mplayer easily qualifies as one of the longest for any application anywhere on the planet. A definitive study could require some months, but, as far as your editor can tell, none of those options tell mplayer how long it should run. As with wget, that omission makes mplayer a little harder to use in an automated mode.

Some distributions are more enthusiastic about including mplayer than others. Packages for almost any distribution are readily available, however, to those who search for them.

streamripper

The definitive tool for capturing streams may well be streamripper. This utility will grab a stream and store it to disk, possibly splitting it into separate tracks as it goes. It can function as a relay, making it possible to listen to a stream as it is being recorded - or to distribute a stream around an internal network. In its simplest form, streamripper is run as:

    streamripper http://stream-url

Options exist to limit recording time, control separation into tracks, establish a stream relay, and automatically discard advertisements. There are graphical frontends for GNOME (streamtuner) and KDE (KStreamRipper). There is also an amarok plugin available.

To conclude

From your editor's point of view, streamripper is the right tool for this job. It is the only one which was designed for the purpose of capturing audio streams in their original format. In a pinch, wget will do the job, as will mplayer. Employing a huge tool like mplayer for this purpose feels somewhat like using a nail gun to hang a calendar, however.

For now, we are lucky in that there are quite a few high-quality streams which can be time-shifted and enjoyed in this manner. Unfortunately, the future looks to be made up of DRM-encrusted streams and no access for users of free software. No fair use rights. If we want to live in a world where broadcast streams are accessible with free tools and developers of stream players are not afraid to add "record" buttons, we need to ensure that the legal climate does not become more hostile than it is already. Otherwise, finding a good stream capture tool could become much harder than it is today.


to post comments

The Grumpy Editor's guide to audio stream grabbers

Posted May 10, 2006 16:35 UTC (Wed) by b7j0c (guest, #27559) [Link]

very informative! i didn't even know many of these tools existed.

The Grumpy Editor's guide to audio stream grabbers

Posted May 10, 2006 16:35 UTC (Wed) by MenTaLguY (guest, #21879) [Link] (2 responses)

Hmm, the way most players support recording is by providing a "disk writer" output plugin which writes to a .WAV file instead of the sound card. I'm fairly certain XMMS ships with one, for example.

The Grumpy Editor's guide to audio stream grabbers

Posted May 10, 2006 20:55 UTC (Wed) by havardk (subscriber, #810) [Link]

Hmm, the way most players support recording is by providing a "disk writer" output plugin which writes to a .WAV file instead of the sound card. I'm fairly certain XMMS ships with one, for example.
Yes but that's not very practical for timeshifting.

XMMS can also dump ogg/vorbis and mp3 streams in original format, and that might be a good option, atleast if you want to listen to the stream at the same time.

Not. Want. WAV.

Posted May 11, 2006 11:17 UTC (Thu) by smurf (subscriber, #17840) [Link]

It's a lossily-compressed stream. You don't want to store that in uncompressed form (takes too much disk space, even these days...), and sound quality is very unlikely to be improved by re-compression.

mplayer - for those "proprietary" media formats

Posted May 10, 2006 17:35 UTC (Wed) by dowdle (subscriber, #659) [Link]

Ok, this article was geared for AUDIO but let's not forget that much of it applies to VIDEO too.

A lot of media you will find out there is in RealPlayer, Windows Media Player and various other formats... often with an rtsp:// or an mms:// type URL. I have found mplayer to be very handy for capturing almost every stream you can find... assuming you have installed all of those questionable codecs from questionable sources.

Play Back: Once you have that proprietary media, how are you going to play it? With mplayer, xine, or vlc of course.

Getting the URL: mplayerplug-in's "Copy URL" feature (or however it is worded) is very handy when pages attempt to hide the real URL of the media. I often start the media with the browser plugin, copy the URL and them dumpstream it with mplayer.

Re-encoding: For converting those proprietary media formats into something more desirable, mencoder and/or ffmpeg are very handy. Getting to know some of mencoder's and ffmpeg's command line options and then making a cheat sheet keeps one from going crazy trying to read those techie man pages. There are various command line tricks for joining multiple media files into one... which is very handy when something like... say... PBS Frontline breaks their shows into 4 to 6 pieces. Ever want to extract the audio from a video and make it into an .mp3 or .ogg? You can once you learn the command line flags. avidemux2 is handy for converting to ogm as well as performing various operations (cropping, FPS change, etc) in a reasonable GUI... although avidemux2 can be a little picky on poorly encoded original files.

The Grumpy Editor's guide to audio stream grabbers

Posted May 10, 2006 17:35 UTC (Wed) by kwoot (guest, #134) [Link]

isn't vlc capable of capturing and saving streams. It worked for me in the past.

greets,

The Grumpy Editor's guide to audio stream grabbers

Posted May 10, 2006 17:39 UTC (Wed) by maniax (subscriber, #4509) [Link]

When I use mencoder/mplayer to edit/cut video files, there's the -frames option, which multiplied by the fps gives you the time the app will run. I just tested this with an audio-only stream, and it works ok, but you'll probably need to do some testing to be sure how long is a frame in the stream that you're recording.

non-free streams

Posted May 10, 2006 22:12 UTC (Wed) by rfunk (subscriber, #4054) [Link] (1 responses)

I think this article aims too low. There are many ways to do this with an Ogg stream, or even MP3. But RealAudio or Windows Media streams are more of a challenge, and this article only hints at that issue.

I've done this sort of time-shifting on a RealAudio stream using trplayer.

mplayer (with the right codecs installed) may be the best tool for capturing Windows Media streams.

non-free streams

Posted May 11, 2006 9:29 UTC (Thu) by cortana (subscriber, #24596) [Link]

I have had success with 'mimms' in the past for capturing Windows Media MSS streams.

The Grumpy Editor's guide to audio stream grabbers

Posted May 11, 2006 3:49 UTC (Thu) by jmorris42 (guest, #2203) [Link]

> Regardless of why, a Linux user who has fired up totem, amarok, or xmms to
> play an audio stream will not readily find a "record" option there.

Ok, xmms doesn't have a nice simple record button, but it does allow saving of streams. And while it doesn't have a built in way to set the length that isn't a problem for a UNIXhead since xmms is scriptable.

I have been known to use "at" to have xmms start a playlist a few minutes after the alarm clock as a backup when I really have to get up early. No reason the same idea couldn't be used to start and stop a recording.

mplayer: -endpos

Posted May 11, 2006 4:38 UTC (Thu) by ewen (subscriber, #4772) [Link]

MPlayer (mplayer and mencoder) has an -endpos argument, which will tell it long it should read/encode from a file or stream. It takes a value in hours:minutes:seconds (and optionally milliseconds).

I've never actually tried this while dumping a network stream to disk (to my recollection), but it definitely works fine for timed duration recording off a TV capture card, and playback/transcoding from a file. If you already have MPlayer installed it may be worth experimenting with that.

Ewen

mimms

Posted May 11, 2006 8:45 UTC (Thu) by jneves (guest, #2859) [Link]

Simple, specifically for the mms protocol.

XMMS saves Vorbis

Posted May 11, 2006 9:23 UTC (Thu) by zooko (guest, #2589) [Link]

I save copies of vorbis radio while listening to it with XMMS. There is a checkbox somewhere in the audio output ui options.

The Grumpy Editor's guide to audio stream grabbers

Posted May 11, 2006 15:32 UTC (Thu) by stevan (guest, #4342) [Link] (4 responses)

May I humbly submit a little script that I use for recording BBC output,
especially BBC7 stuff from their "Listen Again" page. BBC output is in
Realaudio format, but I've used the same script for other formats too.

You need the URL of the stream. This can usually be pasted from the
clipboard, but is sometimes hidden when you use the BBC RadioPlayer front
end. However, you can find the URL in your squid logs!

The script accepts the URL and a final file name as arguments. Of course,
the final result is an ogg file. I hope someone will improve it so that
it becomes clickable app under KDE.


Stevan
------



#!/bin/sh

FILE=$RANDOM
QUALITY=48

mplayer -nojoystick -nolirc -prefer-ipv4 -playlist $1 -ao \
pcm:file=~/Desktop/$FILE.wav -vc dummy -vo null

cd ~/Desktop
oggenc -b $QUALITY $FILE.wav
rm $FILE.wav
mv $FILE.ogg `echo $2 | tr " " _`.ogg

modified recorder with simultaneous encoding and mp3

Posted May 18, 2006 20:57 UTC (Thu) by nealmcb (guest, #20740) [Link] (3 responses)

Thanks, Stevan. My version of mplayer doesn't seem to take the
":file" modifier to the "-ao pcm" option, so I used the "-aofile"
option instead.

I also didn't like the idea of filling up my disk with the .wav
data until the recording was done, so I changed it to run the
encoder in the background. I couldn't do that via a pipeline
since mplayer doesn't seem to be able to write to stdout, so I
had to connect them via a named pipe. Here is my modification,
which also produces mp3 rather than ogg since that is what my
nokia tablet knows how to play by default.

----
#!/bin/sh
# record a stream with mplayer, e.g. realaudio, saving it as an ogg vorbis file

# todo: trap terminate signal and kill encoding process, remove named pipe
# consider option for dumping raw format via -dumpstream -dumpfile,
# and later conversion

# based on Stevan's script at http://lwn.net/Articles/182954/
# See also scheck's 'srec' script there

usage="$0 url name"

URL=$1
NAME=`echo $2 | tr " " _`.mp3
FILE=$RANDOM
QUALITY=48

mknod $FILE.wav p

(lame --quiet -V6 --vbr-new --resample 22 -m m $FILE.wav $FILE.mp3) &

# (oggenc --quiet -b $QUALITY $FILE.wav) &

mplayer -cache 32 -nojoystick -nolirc -prefer-ipv4 -playlist $1 -ao pcm -aofile $FILE.wav -vc dummy -vo null

wait # wait for encoding process to finish

mv $FILE.mp3 $NAME
rm $FILE.wav

modified recorder with simultaneous encoding and mp3

Posted Mar 7, 2007 20:22 UTC (Wed) by devasura (guest, #43937) [Link] (1 responses)

some mplayer options seem to be decprecated, use:
mplayer -cache 32 -nojoystick -nolirc -prefer-ipv4 -playlist $1 -ao pcm:file= $FILE.wav -vc null -vo null

modified recorder with simultaneous encoding and mp3

Posted Mar 7, 2007 20:28 UTC (Wed) by devasura (guest, #43937) [Link]

ignore the above comment, didnt read the full article
sorry

modified recorder with simultaneous encoding and mp3

Posted Mar 8, 2007 17:27 UTC (Thu) by devasura (guest, #43937) [Link]

i replaced oggenc line by:
(cat $FILE.wav | oggenc --quiet -b $QUALITY - | tee $FILE.ogg | mplayer -cache 32 - > /dev/null) &
to enable playing while the stream is being recorded

It Seems To Me

Posted May 13, 2006 20:32 UTC (Sat) by Baylink (guest, #755) [Link]

that the underlying reason for the subject of your complaint is that we're only recently starting to enter an environment wherein there are continuously streamed audio or video source from which one might *want* to record a times subsegment.

(And let's remember that since streaming tends to be a regime of uncertain delays, any utility which *does* attempt to do this should start early and run long...)

Let's find GUI players with a "record" button

Posted May 18, 2006 9:35 UTC (Thu) by jpetso (guest, #36230) [Link]

Kaffeine has (in the "File" menu) a "Record stream" action. That may not
qualify for time-scheduled recordings, but should do well for audio- as
well as video-streams.

A bloated script for time-scheduled audio stream recordings

Posted May 18, 2006 12:15 UTC (Thu) by scheck (guest, #4447) [Link] (1 responses)

I wrote a small script for time-scheduled stream recordings and bloated it by extensive use of comments and a small "station database", which holds my favourite German radio stations.

A mere echo 'srec myrockstation 60 "Iron Maiden Live in Lederhosen"' | at 22:00 Sat would take care of your desired one-hour-show on coming Saturday at 10pm while you go out with friends who listen to Madonna... ;-)

BTW: if you adapt the mplayer command and the assignment of extensions, the script might also work with videos...

#!/bin/sh
#
# srec - stream recording
# v0.01 2006-03-21 torsten.scheck@gmx.de
#
# Copyright is hereby assigned to the Public Domain.
#
# This script relies on wget or mplayer, depending on
# the stream format.

# parameter defaults
station=${1:-"NoStation"}
length=$((${2:-60} * 60))
name=${3:-"Unnamed"}

# time period in seconds, after which the dumping process is checked
checktime=10

# choose station from database and build station list
url=''
list=''
add() {
  list="$list"$(printf "%-8s %-3s %-25s %s" "$list_station" "$list_ext" \
               "$list_info" "$list_name")"\n"
  if [ "$station" == "$list_station" ]; then
    ext=$list_ext
    info=$list_info
    url=$list_url
  fi
}
############################################################
list_station="dlf"
list_ext="ogg"
list_name="Deutschlandfunk"
list_info="96 kbps, variable, stereo"
list_url="http://dradio-live.ogg.t-bn.de/dlf_high.ogg"
add
list_station="dlfmp3"
list_ext="mp3"
list_name="Deutschlandfunk"
list_info="48 kbps, constant, mono"
list_url="http://dradio-live.mp3.t-bn.de/dlf_live"
add
list_station="dlfwma"
list_ext="wma"
list_name="Deutschlandfunk"
list_info="48 kbps, constant, stereo"
list_url="mms://dradio-live.wm.t-bn.de/live/dlf/dlf"
add
list_station="dlrk"
list_ext="ogg"
list_name="Deutschlandradio Kultur"
list_info="96 kbps, variable, stereo"
list_url="http://dradio-live.ogg.t-bn.de/dkultur_high.ogg"
add
list_station="contra"
list_ext="ra"
list_name="SWR Cont.Ra"
list_info="44 kbps, stereo"
list_url="rtsp://213.254.239.61/farm/*/encoder/swr/contra/livestream.rm"
add
############################################################

filename="${name}_${station}_$(date +%Y%m%d_%H%M).${ext}"
logname="${filename%.*}.log"

# display help when station name is not found or wasn't provided
if [ -z "$url" ]; then
  echo "Usage: $0 STATION [LENGTH in minutes] [NAME]
Saves an audio stream of a given internet radio STATION to a file in 
the current working directory. 

LENGTH defaults to 60 minutes and NAME defaults to 'Unnamed', so the 
resulting file name looks like this: ${filename}xxx
The download process is logged to a separate file using the suffix 'log'.

This script is best used with the Unix tool 'at' for time-controlled
recordings. For example, try this to record the 55-minutes-show 
'Querkoepfe' on the station 'Deutschlandfunk' next Wednesday at 21:05
including some 'safety minutes' before and after the show:
> echo 'srec dlf 60 Querkoepfe' | at 21:03 Wed
Mind your time zone and get the list of your locale's supported 
abbreviations for the time specification with:
> locale -c LC_TIME

Available stations:" >&2
  echo -ne "$list" | sort >&2
  exit 1
fi

# use 'wget' as default tool and 'mplayer' for special streaming formats
case "$ext" in
  ra|wma) util="mplayer";;
  *) util="wget"
esac
# quit if needed tool is not installed
type "$util" >/dev/null 2>&1 || { echo "Please install '$util'."; exit 1; }

# all output is redirected to logfile for easier 'at' usage
exec >> "$logname"
exec 2>> "$logname"

until
  # start background process to dump stream to file 
  if [ "$util" == "mplayer" ]; then
    mplayer -vo null -vc dummy -dumpstream -dumpfile "$filename" "$url" &
  else
    wget -nv -O "$filename" "$url" &
  fi
  # keep process id of bg process
  mypid=$!
  # in case of SIGINT and SIGTERM: kill bg process before exit
  trap 'kill $mypid; exit;' 2 15
  # wait until bg process figured out if stream is available
  # e.g. "ERROR 504: Server Full" ends the bg process
  sleep $checktime
  # check if process is still here, i.e. the stream is being downloaded,
  # or if the recording length is reached 
  kill -0 $mypid >/dev/null 2>&1 || [ $length -le 0 ]
do
  echo "Server problem, another try: $(date '+%Y%m%d %H%M%S')" 
  # decrease recording length by checktime; in the case of a transient
  # problem the remaining length of the stream will be saved
  length=$((length - checktime))
  # retry download with another until loop
done

if [ $length -gt 0 ]; then 
  # wait recording length
  sleep $length
  # kill bg process and wait for its termination
  kill $mypid 
  wait 
else
  # the recording length had been reached by trying just to get 
  # the download started
  # an unrecoverable problem prevented the desired download
  echo "Aborted." 
fi
echo "$filename END: $(date '+%Y%m%d %H%M%S')" 

curl has timed downloads, unlike wget

Posted May 18, 2006 21:05 UTC (Thu) by nealmcb (guest, #20740) [Link]

I've often wanted a timer for wget also. Then I learned that the popular 'curl' program and library has one....

curl -m 3600 -s -S -o $f $url

FM

Posted May 18, 2006 22:44 UTC (Thu) by yem (guest, #1138) [Link]

If it's FM you're after, a cheap USB FM tuner and a little scripting goes a long way.

The national public radio station in NZ has only recently added online streams but a) it's not real time and b) it is of poor quality - think 24-48kbit mono wma format. I have some perl-fu that scrapes the online program listing, consults a PVR style config file of programs to record, schedules them with atd and takes care of encoding, file naming and archiving. Works a treat.

The Grumpy Editor's guide to audio stream grabbers

Posted Oct 12, 2006 7:38 UTC (Thu) by anonymous21 (guest, #30106) [Link]

What you really would like (I think) is a program called Screamer Radio. It does not (unfortunately) have timed startup, but it does have these features:

1) Record button
2) Mute button (so you can mute the audio stream while recording, and listen to some other audio using another player. Also great for manually muting annoying commercials during playback!).
3) *** 5 minute audio buffer ***, so when you click "Record", it actually starts recording from five minutes prior to that moment, assuming you've been playing the stream for that long. So when you realize, two minutes into a program, that you wish you'd started recording it from the start, you now have that option (again, assuming you had actually started listening that far back - it can't record what it never had the chance to buffer!)

The problem, of course, is that it only runs really well under Windows. You CAN install and run it under Wine, but if you do it will only play mp3 (and, I think, ogg) streams. Specifically, it will not play streams in Windows Media format, even though it plays those fine under Windows. Beyond that, under Wine it has a nasty tendency to freeze up unexpectedly.

If someone would clone this program for use under Linux (or figure out how to make it work 100% under Wine, although I'd rather have a native Linux app), I for one would be appreciative.


Copyright © 2006, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds