You don't have to distribute any keys. Just make it possible that the owner of the device can build and run modified software. Let him push his own keys or images to the device if he has physical access. It's not that hard. Well, unless you build your device to specifically deny full access to the owner.