fallbacks for testing and development [LWN.net]

fallbacks for testing and development

Posted Apr 21, 2008 15:24 UTC (Mon) by Jel (guest, #22988)
Parent article: Benchmarking Linux filesystems on software RAID 1 (Lone Wolves)

I seem to recall trying LVM on loopback files, and being shocked to find 
out that it's not supported.

I never got why LVM doesn't allow this.  Isn't that the whole unix 
philosophy -- to treat everything as a file?  To allow for unusual 
scripting and unexpected setups?

Wouldn't it be perfect for both benchmarkers and developers to be able to 
flexibly test these things on top of a single hard drive (or even a ram 
drive) which has known performance characteristics?

I think every linux feature that uses hardware should have software or 
filesystem fallbacks.  That way IM client developers, media player 
developers could write for all sorts of webcam and TV card and genlocker 
and surround sound setups without actually having that hardware.

I only need to barely mention the security researchers setting up honey 
pots, the admins doing deployments, etc. for people to see that there's a 
real need for this sort of flexibility.  Just think of how many times 
you've mounted a filesystem as loopback, simulating a physical drive.  
This should be possible across the whole platform, with every device.  I 
know that's a tall order, but we should recognise the principle and move 
towards it, at least.

(Log in to post comments)

No LVM on loopback

Posted Apr 22, 2008 1:40 UTC (Tue) by pr1268 (subscriber, #24648) [Link]

I never got why LVM doesn't allow this. Isn't that the whole unix philosophy -- to treat everything as a file? To allow for unusual scripting and unexpected setups?

I'm curious about that, too. I thought that the whole VFS kernel subsystem was to separate all the layers of the file system to enable all sorts of different HW and FS combinations¹. But, I do take comfort in knowing that, if it doesn't exist currently, then there's probably a good reason why it doesn't exist.

¹ I'm still amazed at how easy it was to set up a Linux kernel software RAID-0 on two IDE disks running off of separate ribbon cables attached to a Promise ATA controller card that came free with my purchase of a WDC 200GB hard drive four years ago. As unusual as I consider this setup to be, Linux gives me absolutely no grief about it (and this virtual drive hauls ass!). ;-)

No LVM on loopback

Posted Apr 23, 2008 12:55 UTC (Wed) by daniel (subscriber, #3181) [Link]

I have set out to rewrite Linux LVM (lvm3 project, a merge of device mapper capabilities with
the generic block layer) and I will certainly ensure that the result works properly on
loopback files.  Anybody have more wishes?

Daniel

No LVM on loopback

Posted Apr 23, 2008 19:55 UTC (Wed) by nix (subscriber, #2304) [Link]

Useful snapshots: i.e. >32 of them, snapshots where snapshotting the root 
filesystem works... ideally a frontend command like chroot but which 
snapshots everything below some tree, kicks up a shell in it, and zaps all 
the snapshots when the shell exits.

(Why? So I can back my whole system up via snapshots, reliably. I might 
well have >32 filesystems: I obviously want to back up / without running a 
deadlock risk: and the chroot-in-snapshot command is just obviously 
useful.)

So far LVM snapshots have enough limitations that I haven't found a single 
use for them in practice... it would be nice to have snapshots that work. 
(Of course we might all go to btrfs and get them for free!)

No LVM on loopback

Posted Apr 23, 2008 21:01 UTC (Wed) by zlynx (subscriber, #2285) [Link]

I use LVM on a Fedora Core system at work.  I made a little script to snap each of the
filesystems, including the root fs, and then mount each snap in a snapshot directory.  That
script runs just before the nightly backup job which reads off the nightly snapshot.

The script also keeps a weekly snapshot.  The two snapshots are very convenient for grabbing
quick copies of old files.

I also took advantage of the ability to assign new LVs (which a snapshot is) to particular PVs
so as to distribute the write load across several disks.  Because, when a snapshot source is
written to, copies of the old blocks have to be written into the snapshots before the old
block can be overwritten on the source.  That sped things up a lot.

In a couple years of doing this I've never had that system lock up during a backup job, so the
risk of deadlock must be very low.

No LVM on loopback

Posted Apr 24, 2008 20:32 UTC (Thu) by nix (subscriber, #2304) [Link]

It is, but still... the problem is apparently that the lvm tools don't 
lock themselves in memory, and there's a window where I/O to the snapshot 
origin is disabled. If the OS needs to page in the LVM tools at that 
point, deadlock.

(And that's only the known deadlock.)

No LVM on loopback

Posted Apr 24, 2008 5:07 UTC (Thu) by muwlgr (guest, #35359) [Link]

I would wish :
 - raid1/raid5 facilities for separate LVs, unlike current situation
      when you have first to assemble the whole RAID PV 
      in underlying layer by mdadm.
 - online filesystem resizing, like jfs/jfs2 in AIX LVM.
      There was fsadm script in LVM1, but in LVM2 it has gone.

In short, the final wish really looks like ZFS/Btrfs and of course Zumastor :>