Wgetting bytes from random.org [LWN.net]

Wgetting bytes from random.org

Posted May 22, 2008 23:22 UTC (Thu) by pr1268 (guest, #24648)
In reply to: Appropriate sources of entropy by cpeterso
Parent article: Appropriate sources of entropy

I'll happily share my technique for grabbing some random bytes from Random.org (run as root, obviously):

#!/bin/sh # Grab 4 unsigned bytes from random.org's HTTP interface, # parse out just the numbers (filtering the tab chars out), # create a 4-byte (32-bit) hexdump, and finally, # write output to /dev/random. wget -o /dev/null -O - \ "http://www.random.org/integers/?num=4&min=0&max=255&col=4&base=16&format=plain&md=new" \ | sed -e 's/\t//g' \ | xxd -r -p \ > /dev/random

Feel free to use or derive from the above; while I'm the author, I do not imply any copyright on it. Comments, suggestions, and criticism are most certainly welcome.

to post comments

Disclaimers and correction to above wget script

Posted May 23, 2008 2:22 UTC (Fri) by pr1268 (guest, #24648) [Link]

Standard disclaimers and a correction:

I claim no liability whatsoever for whatever happens if anyone runs the above script on their computer. Granted, I have tested it on Slackware 12.0, and it appears to work fine.
Random.org imposes a quota (based on client IP address) for fetching entropy bits from it. Please be mindful of this, as I do not want to blamed for others being denied access due to overuse.
I seem to have misspelled the last query parameter name. According to Random.org's HTTP interface specification, it should be rnd= instead of md=.

The value of the num= query parameter specifies how many bytes are provided. I'm unsure whether 8 bytes (64 bits) would be preferred for 64-bit Linux users (anyone 64-bit user out there wish to comment on that?).

Finally, my apologies for sounding like a Random.org shill, but there are some really good articles and links on entropy and randomness there.

Wgetting bytes from random.org

Posted May 24, 2008 0:09 UTC (Sat) by jch (guest, #51929) [Link] (6 responses)

> wget ... | xxd -r ... > /dev/random

I believe that this will mix new data into the entropy pool, but not actually increase the
entropy estimate.  In other words, any process blocking on /dev/random will remain blocked
until some other, accounted for, entropy is added.

I believe you'll need to ioctl(RNDADDENTROPY) in order to fix that.

Wgetting bytes from random.org

Posted May 24, 2008 1:31 UTC (Sat) by pr1268 (guest, #24648) [Link] (5 responses)

I believe that this will mix new data into the entropy pool, but not actually increase the entropy estimate.

But isn't that what's happening all the time anyway, with the "environmental noise from device drivers and other sources" mentioned in the random(4) man page? What I was attempting to do is add another, external entropy source. Actually, I don't use this script hardly at all; it was more an attempt to make something useful while honing my shell scripting skills. I did learn xxd(1).

In other words, any process blocking on /dev/random will remain blocked until some other, accounted for, entropy is added.

I don't think the entropy pool has ever been drained completely (i.e., /dev/random blocked for reading) on my workstation. Except for the time I did that myself from a shell console. A few keystrokes, mouse movements, and network traffic bytes later (30 seconds or so), the pool had been refilled, according to /proc/sys/kernel/random/entropy_avail.

I believe you'll need to ioctl(RNDADDENTROPY) in order to fix that.

But this is supposed to be a shell script, not a C program! ;-)

I do appreciate your feedback and comments, thanks! Understand that I learn a lot reading our editors' stuff and others' comments.

Wgetting bytes from random.org

Posted May 24, 2008 2:12 UTC (Sat) by jch (guest, #51929) [Link] (4 responses)

> But isn't that what's happening all the time anyway, with the "environmental noise from
device drivers and other sources"

No.  The various sources of entropy add entropy to the random pool and increase the entropy
estimate by some small value.

You may want to see for yourself in the kernel sources -- they end up calling
add_timer_randomness (drivers/char/random.c line 571) which calls add_entropy_words and then
credit_entropy_store.  This increases the value of entropy_count, and may cause processes
blocking on /dev/random to wake up.

Wgetting bytes from random.org

Posted May 24, 2008 3:52 UTC (Sat) by pr1268 (guest, #24648) [Link] (3 responses)

So how is me writing bytes to /dev/random via a script any different than what my distro (Slackware 12.0) does in rc.S (also a script) when the "carry-over" entropy file (/etc/random-seed) is written to the random device? Other than a difference in quantity of bytes, I see no difference.

My intuition was that /dev/random is (root) writable so that the sysadmin can incorporate additional sources of entropy. I don't mean to sound like I'm arguing, just sincerely interested... Thanks!

Wgetting bytes from random.org

Posted May 24, 2008 5:53 UTC (Sat) by dlang (guest, #313) [Link]

the sysadmin can add randomness, but the system will not trust that it is random (without
tweaking things via the ioctl), even if the sysadmin sends completely predictable data to
/dev/random it won't do any harm.

Wgetting bytes from random.org

Posted May 24, 2008 16:26 UTC (Sat) by jch (guest, #51929) [Link] (1 responses)

> My intuition was that /dev/random is (root) writable so that the sysadmin can incorporate
additional sources of entropy.

The distinction between mixing in new data into the random pool and adding to the entropy
estimate is what the article is about.

The in-kernel RNG maintains a pool of random data and an estimate of how much entropy is in
the pool.

When you read from /dev/(u)random, the entropy estimate is reduced.  When it reaches 0, reads
from /dev/random will block.  That's the easy part.

The difficult part is deciding when to increase the entropy estimate.  When you write 100
bytes to /dev/random, unless the 100 bytes are perfectly random, they should not add 800 bits
to the entropy estimate, but some lower value that only the person who generated the data is
able to choose reasonably.

For that reason, merely writing to /dev/random does not add to the entropy estimate; you need
to explicitly increase it by using the ioctl.

> So how is me writing bytes to /dev/random via a script any different than what my distro
(Slackware 12.0) does in rc.S (also a script) when the "carry-over" entropy file
(/etc/random-seed) is written to the random device?

It's no different.  Your distribution is mixing the old data into the random pool, but not
increasing the entropy estimate.  This way if the carry-over data is not truly random, no
serious security vulnerability will ensue.

Thanks for the replies

Posted May 26, 2008 2:23 UTC (Mon) by pr1268 (guest, #24648) [Link]

Thanks to those who replied to my questions. Coincidentally, Ted T'so recently had a related explanation (to dlang's and jch's above) on the LKML.

Wgetting bytes from random.org - copyright

Posted May 24, 2008 17:40 UTC (Sat) by giraffedata (guest, #1954) [Link]

while I'm the author, I do not imply any copyright on it.

You should change your wording for this kind of thing. In most places, copyright is implied by law, whether you do something to imply you want it or not. You have to explicitly disclaim it. "I disclaim any copyright" would work. "I contribute this to the public domain" is the usual wording.

Wgetting bytes from random.org

Posted May 30, 2008 10:12 UTC (Fri) by Duncan (guest, #6647) [Link]

[reposted as the initial submission timed out without confirmation]

Viewing the previous replies to your script posting, I'd say it's a good 
thing you don't increase the entropy count based on that script.  After 
all, you're using an unencrypted http: connection.  As such, it'd be a 
(relatively) simple matter for an attacker, indeed, even a remote 
attacker, to do a MitM attack and substitute whatever he wanted into 
the "response from random.org", which for all you know is anything but.

So yes, in line with the theme of the article, adding the bits shouldn't 
do any harm, as long as you don't count it as added entropy.  However, it 
certainly can't be counted on to /help/ either, since you've really no 
idea where the data is coming from or how predictable it might be, so it's 
a good thing your script does /not/ have the system count it as added 
entropy.

Of course, the first instinct would then be to use an encrypted/ssl 
connection.  However, I believe that'd be defeating the purpose to some 
extent, since creating the encrypted connection will (I assume, I'm no 
authority and really haven't a clue, only a guess) consume entropy in the 
first place.  Assuming it's allowed, one could then grab more entropy from 
random.org than was consumed, but there'd still need to be some entropy 
available initially or the encrypted connection itself would be suspect.

I'm really surprised nobody else noted this in their replies... <shrug>

Duncan