| |||||||||||||
EC2 (local) instance storageEC2 (local) instance storagePosted Dec 9, 2012 1:18 UTC (Sun) by man_ls (subscriber, #15091)In reply to: EC2 (local) instance storage by dlang Parent article: Optimizing stable pages New instances should not see the contents of uninitialized (by them) disk sectors. That is the point of the recent discussion about FALLOC_FL_NO_HIDE_STALE. The kernel will not allow one virtual machine to see the contents of another's disk, or at least that is what I understand. The AWS console has an option to reboot a machine, between "Terminate" and "Stop". You can also do it programmatically using EC2 commands, e.g. if the machine stops responding.
(Log in to post comments)
EC2 (local) instance storage Posted Dec 9, 2012 1:50 UTC (Sun) by dlang (✭ supporter ✭, #313) [Link]
Thanks for the correction about the ability to reboot an instance.
I don't think this is what FALLOC_FL_NO_HIDE_STALE is about. FALLOC_FL_NO_HIDE_STALE is about not zeroing something that this filesystem has not allocated before, but if you have a disk that has a valid ext4 filesystem on it and plug that disk into another computer, you can just read the filesystem.
When you delete a file, the data remains on the disk and root can go access the raw device and read the data that used to be in a file.
by default, when a filesystem allocates a block to a new file, it zeros out the data on that block, it's this step that FALLOC_FL_NO_HIDE_STALE lets you skip.
the If you really had raw access to the local instance storage without the hypervisor doing something, then you could just mount whatever filesystem the person before you left there. To avoid this Amazon would need to wipe the disks, and since it takes a long time to write a TB or so of data (even on SSDs), I'm guessing that they do something much easier, like doing some sort of encryption to make it so that one instance can't see data written by a prior instance.
EC2 (local) instance storage Posted Dec 9, 2012 12:04 UTC (Sun) by man_ls (subscriber, #15091) [Link] When Amazon EC2 creates a new instance, it allocates a new instance storage with its own filesystem. This process includes formatting the filesystem, and sometimes copying files from the AMI (image file) to the new filesystem. So any previous filesystems are erased. It is here that zeroing unallocated blocks from the previous filesystem comes into place, which is what FALLOC_FL_NO_HIDE_STALE would mess up.I don't know how Amazon (or the hypervisor) prevents access to the raw disk, where unallocated sectors might be found and scavenged even if the filesystem is erased. I guess they do something clever or we would have heard about people reading Zynga's customer database from a stale instance.
EC2 (local) instance storage Posted Dec 9, 2012 16:01 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]
Uhm. Nope.
Amazon doesn't care about your filesystem. AMIs are just dumps of block devices - Amazon simply unpacks them onto a suitable disk. You're free to use any filesystem you want (there might be problems with the bootloader, but they are not insurmountable).
You certainly can access the underlying disk device.
EC2 (local) instance storage Posted Dec 10, 2012 1:07 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
Amazon doesn't put a filesystem on the device, you do.
> I don't know how Amazon (or the hypervisor) prevents access to the raw disk, where unallocated sectors might be found and scavenged even if the filesystem is erased. I guess they do something clever or we would have heard about people reading Zynga's customer database from a stale instance.
This is exactly what I'm talking about.
There are basically three approaches to doing this without the cooperation of the OS running on the instance (which you don't have)
1. the hypervisor zeros out the entire drive before the hardware is considered available again.
2. the hypervisor does encryption of the blocks with a random key for each instance, loose the key and reading the blocks just returns garbage
3. the hypervisor tracks what blocks have been written to and only returns valid data for those blocks.
I would guess #1 or #2, and after thinking about it for a while would not bet either way
#1 is simple, but it takes a while (unless the drive has direct support for trim and effectively implements #3 in the drive, SSDs may do this)
#2 is more expensive, but it allows the system to be re-used faster
EC2 (local) instance storage Posted Dec 10, 2012 2:55 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
They are using #3. The raw device reads on initialized areas return zeroes.
EC2 (local) instance storage Posted Dec 10, 2012 3:27 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
that eliminates #2, but it could be #1 or #3
It seems like trying to keep a map of if this block has been written to would be rather expensive to do at the hypervisor level, particularly if you are talking about large drives.
Good to know that you should get zeros for uninitialized sectors.
EC2 (local) instance storage Posted Dec 10, 2012 3:30 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
#1 is unlikely because local storage is quite large (4Tb on some nodes). It's not hard to keep track of dirtied blocks, they need it to support snapshots on EBS volumes anyway.
EC2 (local) instance storage Posted Dec 10, 2012 6:18 UTC (Mon) by bjencks (subscriber, #80303) [Link]
Just to be clear, there are two different ways of initializing storage: root filesystems are created from a full disk image that specifies every block, so there are no uninitialized blocks to worry about, while non-root instance storage and fresh EBS volumes are created in a blank state, returning zeros for every block.
It's well documented that fresh EBS volumes keep track of touched blocks; to get full performance on random writes you need to touch every block first. That implies to me that they don't even allocate the block on the back end until it's written to.
Not sure how instance storage initialization works, though.
EC2 (local) instance storage Posted Dec 10, 2012 6:34 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
EBS storage is not simple disks, the size flexibility and performance you can get cannot be supported by providing raw access to drives or drive arrays.
As you say, instance local storage is different.
|
|||||||||||||