EC2 (local) instance storage

Posted Dec 8, 2012 23:56 UTC (Sat) by man_ls (guest, #15091)
In reply to: Optimizing stable pages by dlang
Parent article: Optimizing stable pages

I am not sure what "dies" means in this context. If the instance is stopped or terminated, then the instance storage is lost. If the instance is rebooted then the same instance storage is kept. Usually you reboot machines which "die" (i.e. crash or oops), so you don't lose instance storage.

In short: any new EC2 instance will of course get a new instance storage, but the same instance will get the same instance storage.

I understand your last paragraph even less. Why do transparent encryption? Just use regular filesystem options (i.e. don't use FALLOC_FL_NO_HIDE_STALE) and you are good. I don't get what a journal has to do with it.

Again, keep in mind that many instance types keep their root filesystem on local instance storage. Would you run / without a journal? I would not.

EC2 (local) instance storage

Posted Dec 9, 2012 1:11 UTC (Sun) by dlang (guest, #313) [Link] (10 responses)

If you don't do any sort of encryption, then when a new instance mounts the drives, it would be able to see whatever was written to the drive by the last instance used it.

I would absolutely run / without a journal if / is on media that I won't be able to access after a shutdown (a ramdisk for example)

I don't remember seeing anything in the AWS management console that would let you reboot an instance, are you talking about rebooting it from inside the instance? If you can do that you don't need a journal because you can still do a clean shutdown. I don't consider the system to have crashed. I count a crash as being when the system stops without being able to do any cleanup (kernel hang or power off on traditional hardware)

EC2 (local) instance storage

Posted Dec 9, 2012 1:18 UTC (Sun) by man_ls (guest, #15091) [Link] (9 responses)

New instances should not see the contents of uninitialized (by them) disk sectors. That is the point of the recent discussion about FALLOC_FL_NO_HIDE_STALE. The kernel will not allow one virtual machine to see the contents of another's disk, or at least that is what I understand.

The AWS console has an option to reboot a machine, between "Terminate" and "Stop". You can also do it programmatically using EC2 commands, e.g. if the machine stops responding.

EC2 (local) instance storage

Posted Dec 9, 2012 1:50 UTC (Sun) by dlang (guest, #313) [Link] (8 responses)

Thanks for the correction about the ability to reboot an instance.

I don't think this is what FALLOC_FL_NO_HIDE_STALE is about. FALLOC_FL_NO_HIDE_STALE is about not zeroing something that this filesystem has not allocated before, but if you have a disk that has a valid ext4 filesystem on it and plug that disk into another computer, you can just read the filesystem.

When you delete a file, the data remains on the disk and root can go access the raw device and read the data that used to be in a file.

by default, when a filesystem allocates a block to a new file, it zeros out the data on that block, it's this step that FALLOC_FL_NO_HIDE_STALE lets you skip.

the If you really had raw access to the local instance storage without the hypervisor doing something, then you could just mount whatever filesystem the person before you left there. To avoid this Amazon would need to wipe the disks, and since it takes a long time to write a TB or so of data (even on SSDs), I'm guessing that they do something much easier, like doing some sort of encryption to make it so that one instance can't see data written by a prior instance.

EC2 (local) instance storage

Posted Dec 9, 2012 12:04 UTC (Sun) by man_ls (guest, #15091) [Link] (7 responses)

When Amazon EC2 creates a new instance, it allocates a new instance storage with its own filesystem. This process includes formatting the filesystem, and sometimes copying files from the AMI (image file) to the new filesystem. So any previous filesystems are erased. It is here that zeroing unallocated blocks from the previous filesystem comes into place, which is what FALLOC_FL_NO_HIDE_STALE would mess up.

I don't know how Amazon (or the hypervisor) prevents access to the raw disk, where unallocated sectors might be found and scavenged even if the filesystem is erased. I guess they do something clever or we would have heard about people reading Zynga's customer database from a stale instance.

EC2 (local) instance storage

Posted Dec 9, 2012 16:01 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

Uhm. Nope.

Amazon doesn't care about your filesystem. AMIs are just dumps of block devices - Amazon simply unpacks them onto a suitable disk. You're free to use any filesystem you want (there might be problems with the bootloader, but they are not insurmountable).

You certainly can access the underlying disk device.

EC2 (local) instance storage

Posted Dec 10, 2012 1:07 UTC (Mon) by dlang (guest, #313) [Link] (5 responses)

Amazon doesn't put a filesystem on the device, you do.

> I don't know how Amazon (or the hypervisor) prevents access to the raw disk, where unallocated sectors might be found and scavenged even if the filesystem is erased. I guess they do something clever or we would have heard about people reading Zynga's customer database from a stale instance.

This is exactly what I'm talking about.

There are basically three approaches to doing this without the cooperation of the OS running on the instance (which you don't have)

1. the hypervisor zeros out the entire drive before the hardware is considered available again.

2. the hypervisor does encryption of the blocks with a random key for each instance, loose the key and reading the blocks just returns garbage

3. the hypervisor tracks what blocks have been written to and only returns valid data for those blocks.

I would guess #1 or #2, and after thinking about it for a while would not bet either way

#1 is simple, but it takes a while (unless the drive has direct support for trim and effectively implements #3 in the drive, SSDs may do this)

#2 is more expensive, but it allows the system to be re-used faster

EC2 (local) instance storage

Posted Dec 10, 2012 2:55 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

They are using #3. The raw device reads on initialized areas return zeroes.

EC2 (local) instance storage

Posted Dec 10, 2012 3:27 UTC (Mon) by dlang (guest, #313) [Link] (3 responses)

that eliminates #2, but it could be #1 or #3

It seems like trying to keep a map of if this block has been written to would be rather expensive to do at the hypervisor level, particularly if you are talking about large drives.

Good to know that you should get zeros for uninitialized sectors.

EC2 (local) instance storage

Posted Dec 10, 2012 3:30 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

#1 is unlikely because local storage is quite large (4Tb on some nodes). It's not hard to keep track of dirtied blocks, they need it to support snapshots on EBS volumes anyway.

EC2 (local) instance storage

Posted Dec 10, 2012 6:18 UTC (Mon) by bjencks (subscriber, #80303) [Link] (1 responses)

Just to be clear, there are two different ways of initializing storage: root filesystems are created from a full disk image that specifies every block, so there are no uninitialized blocks to worry about, while non-root instance storage and fresh EBS volumes are created in a blank state, returning zeros for every block.

It's well documented that fresh EBS volumes keep track of touched blocks; to get full performance on random writes you need to touch every block first. That implies to me that they don't even allocate the block on the back end until it's written to.

Not sure how instance storage initialization works, though.

EC2 (local) instance storage

Posted Dec 10, 2012 6:34 UTC (Mon) by dlang (guest, #313) [Link]

EBS storage is not simple disks, the size flexibility and performance you can get cannot be supported by providing raw access to drives or drive arrays.

As you say, instance local storage is different.