If you don't deploy using IPMI then you clearly have physical access during initial configuration. And if you've left the IPMI network connected to the rest of your network, you're doing it very, very wrong. But since nobody in the server market seems to be talking about shipping with Secure Boot enabled by default, you can do your key installation in any way you want.