I think this is a sensible idea. It should be possible to make the transition relatively painless:
- add a new inheritable process capability, 'BADFILENAMES', without which processes can't see or create files with bad names.
- add a command 'access_bad_filenames' which creates a shell with the capability.
- /bin/ls also needs the capability, but should not display bad filenames
unless an additional option is passed.
That way, most processes can run happily in the ignorance of any bad filenames. If you need to access one, you run the commands you need to access it with under the special shell.