There seem to be three issues:
1. Atomic replacement of an executable (by name)
2. Removing permissions from potentially hard linked old copies (by inode)
3. Reporting possible nefarious activities by hard linkers.
I think this sequence deals with them all:
1. Hard link canonical name of old executable to first temporary name.
2. Create new executable with second temporary name and appropriate privs.
3. rename() second temporary name to canonical name (this is atomic).
4. Chown/chmod first temporary name to remove any special privileges, execute bits, etc.
5. stat() first temporary name and report if there is more then one link
(Should be just one from first temporary name at this point).
6. Unlink first temporary name.
At the end of the sequence, you will have atomically replaced the executable, removed any
special privileges from any links people may have hidden away, and reported (to the extent
possible without a full search of the filesystem) that somebody has been linking to
Of course, the sendmail example is a bad one as typically there is more then one valid link.
'sendmail' and 'newaliases were traditionally the same program. I know of no way to atomically
make a bunch of links to one
program so the best I can think of would be to make a bunch of temporary names for the new
execuable and then rename() them all into place sequentially in steps 2 and 3. Steps 4-6
could remain the same and would still be able to accurately report improper links while at the
same time making them useless.