LWN: Comments on "Safename: restricting "dangerous" file names"

Safename: breaking compatibility beteen system

mathstuf — Wed, 01 Jun 2016 22:52:05 +0000

While I agree, for software building, it fits the bill just fine. For weird things like triple-pass LaTeX builds or batch automation, keep writing Makefiles :) .

Safename: breaking compatibility beteen system

nix — Wed, 01 Jun 2016 20:08:51 +0000

Ninja is explicitly not intended as a general-purpose make replacement. :)

Safename: breaking compatibility beteen system

madscientist — Thu, 26 May 2016 12:52:50 +0000

You don't need to replace make. You can just set the make SHELL variable to point to whatever shell replacement you prefer.

Safename: restricting "dangerous" file names

tao — Wed, 25 May 2016 20:55:59 +0000

Honestly, that kind of sounds like you did a bad port. The purpose of a port isn't to copy all stupid things from the source platform, but to adapt it to run well on the target system...

In the same manner I wouldn't limit filenames to 8.3 when porting from DOS to Linux, or allow backslash in filenames when porting the other way around...

Safename: breaking compatibility beteen system

mathstuf — Wed, 25 May 2016 18:12:00 +0000

Why not Ninja?

https://ninja-build.org/manual.html

Safename: restricting "dangerous" file names

Wol — Wed, 25 May 2016 17:19:31 +0000

> Just adjust your shell so that * and .* globs expand to ./* and ./.*, respectively

And then you do what we did, and blow up your system (well, we didn't exactly, but our backup system went mad...)

We installed a system ported across from Pr1mos. It used * everywhere as part of a filename ...

(* at the start of a name was sort of the equivalent of .exe in Windows. I'm not even sure it was stored in the directory tree on Pr1mos, but because they couldn't do whatever they did on Pr1mos, they put it in the directory tree on nix ...)

Cheers,
Wol

Safename: breaking compatibility beteen system

nix — Wed, 25 May 2016 17:07:07 +0000

You would also need to replace make(1) et al (and what with? ant? no thank you very much.)

Safename: breaking compatibility beteen system

mathstuf — Sat, 21 May 2016 23:23:04 +0000

I just use UTF-8 and say "boo hoo" to things which don't support it. Which means I send a Tarbell to the phone and extract music files there because MTP is encoding-stupid. And it sends as a single file which is much faster.

Safename: restricting "dangerous" file names

Wol — Sat, 21 May 2016 20:30:34 +0000

Especially when it's deliberate :-)

PI-Open used to create directories (which the OS was not supposed to muck about with the contents of) which contained a whole bunch of files, whose name format was <space><backspace><number>.

It did that because Pr1mos (from which it was ported) had "segmented directories". Basically, a directory with no space for the filename, so files had to be referenced by offset number. For the most part, those directories were meant for programs - each file was mapped to a memory segment when the program was loaded (hence "segmented directory"), but as programmers will, plenty of us found other good uses for them :-)

Cheers,
Wol

Safename: restricting "dangerous" file names

Wol — Fri, 20 May 2016 22:11:19 +0000

> I wonder what the world would look like if there was a special built-in 'glob' for enabling globbing.

This sounds like Pr1mos ... :-)

The Pr1mos shell had expansion and completion, and it also had ways for programs to communicate to/from the shell. It's so long ago, I've forgotten the details, but there was "-verify" and "-confirm" or something like that. So if I typed the command

DELETE @@ -NO_CONFIRM

the @@ would expand to all the files in the directory. DELETE told the shell that its defaults were both verify and confirm, so as the shell expanded the @@, it would ask me to verify the expansion. But because I'd said no_confirm, the shell would then execute the command without asking.

Okay, it relied on the guy who wrote DELETE to get it right, but because you set flags in the executable that the shell picked up, it was pretty flexible. Obviously, something non-dangerous like COPY would default to no_verify no_confirm.

I've always felt that Unix completion is crippled compared to Pr1mos. But then, it is Eunuchs - a castrated Multics. Pr1mos was a Multics-derivative too :-)

Cheers,
Wol

Safename: breaking compatibility beteen system

mathstuf — Fri, 20 May 2016 17:07:01 +0000

Form we already have this between different filesystems (e.g., vfat, mtp, etc.)?

Safename: breaking compatibility beteen system

mathstuf — Fri, 20 May 2016 16:53:34 +0000

Almost a haiku there.

Let us see here now.
No accented characters?
No, thank you, kind sir.

Safename: breaking compatibility beteen system

jezuch — Fri, 20 May 2016 12:34:00 +0000

> In my experience using accented characters always creates problems down the line. There's the problem of different encodings, so if the file was created using ISO-8859-2

Yeah, yeah, yeah. It used to be like that... in the '90s. I mean, I live in a country where a proper encoding is vital to all our ą's and ź's. It's all in the past now, dead and buried. (Except for an occasional system made by occasional clueless uni-lingual Americans ;) )

Case 3=2b

farnz — Fri, 20 May 2016 08:15:19 +0000

Exactly; if I meant it to be a filename, bash can't tell the application anything that hints that that was my intent. Equally, if I meant it to be an option, bash can't tell the application anything that hints that that was my intent.

Thus, this is currently an insoluble problem, without going back in time and changing the idioms for filenames and options such that filenames *always* began . or / (which then makes log-{user.system}.txt clearly not a filename, as it starts "l"), and options always began -; then, you reserve all other characters for parameters that are neither filenames nor options (e.g. PIDs and IP addresses). This is about 40 years too late now (I wasn't even born when the decisions were being made), so it's an insoluble problem because any shell trying to enforce this needs to cope with the legacy that's already out there.

Safename: breaking compatibility beteen system

NAR — Fri, 20 May 2016 08:06:23 +0000

In my experience using accented characters always creates problems down the line. There's the problem of different encodings, so if the file was created using ISO-8859-2 encoding, but the terminal/application uses Unicode, the characters won't show properly. If the file was created using Unicode, it won't show properly when the application uses Code Page 852. If the file was created using Code Page 852, it won't show properly when the terminal is set to ISO-8859-2. Sometimes the language-specific encoding is not even available (or badly configured), so users get "inventive" and use õ or ô instead of ő and of course it won't show properly in ISO-8859-2. And then there's the problem when the given computer does not provide ways to enter accented characters, so one can't even type the name of that file.

Accented characters are useful for e-mail, web pages, formatted text files where the encoding can be specified, but the filenames lack this information.

Safename: breaking compatibility beteen system

jem — Fri, 20 May 2016 07:35:33 +0000

>Users want to have spaces on their filenames. And accented chars (especially in non-english languages, that need lots of accented vowels). And commas and parentheses. All of those are forbidden by POSIX.

Hear, hear! The proposal to enforce stricter limits on file names would severly limit, say, how a user can name documents produced with a word processor. And yet it is not the word processor's fault, but mostly because of an unrelated program: the shell. We need to go to the root of the problem: what we need is a new shell. A shell that can handle file names as a collection of names without the risk of the names being split up just because they contain spaces. A shell where the collection of names is just that, and won't be mixed up with command options. A shell where zero is as good a number as any other number, i.e. an empty collection of names does not get special treatment (replaced with "*", for instance.)

Another solution would be to invent some new storage for documents which is off limits for the shell.

Case 3=2b

gmatht — Fri, 20 May 2016 04:22:28 +0000

Bash doesn't care whether log-user.txt is a file or an option. That is the application's job to decide. Scanning the file system wouldn't even help, for example: `tar x x`.

We don't know that log-{user,system}.txt is an option. We do know that log-{user,system}.txt is a fixed expansion, that doesn't directly depend on any untrusted filenames. So either way we can pass it directly to the application and it handle this ambiguity without worrying too much about the existence of files with malicious names tricking the application.

Safename: breaking compatibility beteen system

hummassa — Thu, 19 May 2016 18:17:28 +0000

> and there isn't really a compelling reason not to, even if you're only targeting Linux

Users want to have spaces on their filenames. And accented chars (especially in non-english languages, that need lots of accented vowels). And commas and parentheses. All of those are forbidden by POSIX.

Not only that: even if a certain user (like me) does not like extraordinarily strange chars in his filenames, it will encounter tarballs/zips with such filenames on it, or use a mainstream web browser that appends the non-extension part of a filename with space-openparentheses-number-closeparentheses when a file with the same name already exists on the download directory (if 'file.txt' already exists, it will try to create 'file (1).txt'). Etc, etc.

Safename: breaking compatibility beteen system

micka — Thu, 19 May 2016 17:40:53 +0000

Let's take a look.
No accented character.
No, thank you.

Simple fix: Only add ./ if bash know is it a file

tao — Thu, 19 May 2016 12:55:06 +0000

As an additional bonus this will yield different results depending on whether you run it on a basic POSIX-compliant shell or using bash.

touch a b
dash$ ls {a,b}
ls: cannot access '{a,b}': No such file or directory
bash$ ls {a,b}
a b

Simple fix: Only add ./ if bash know is it a file

farnz — Thu, 19 May 2016 11:31:01 +0000

Indeed; hence me saying that this isn't actually fixable now. The chance to insist that paths, other strings, and options were distinct entities (with a - hint as first character of an option, and a . hint as the first character of a path, thus not needing a binary protocol to provide the hints) has long since gone away, especially since there are now programs like ps which use the presence or absence of the - to choose between different option parsers.

Simple fix: Only add ./ if bash know is it a file

anselm — Thu, 19 May 2016 11:18:29 +0000

Of course brace expansion, by definition, has nothing to do with existing files. There are lots of legimitate cases for brace expansion where the expansion results are file names that a file system scan won't uncover, because they don't exist. Consider

$ mkdir -p quarterly-results/201{5,6,7}q{1,2,3,4}

Doing a file system scan here to “validate” the expansion results would be completely pointless if not counterproductive.

Safename: restricting "dangerous" file names

anselm — Thu, 19 May 2016 11:11:50 +0000

The shell knows a few things about replacing bits of text with other bits of text. It doesn't really know (or care) what these bits of text mean, and what little knowledge it does have is only used for constructing command lines, not for making sense of them once they're there, which is what this discussion seems to be about.

Safename: breaking compatibility beteen system

anselm — Thu, 19 May 2016 11:09:06 +0000

Basic compatibility between systems is covered by POSIX, and POSIX makes very restrictive assumptions as far as file names go. As long as you adhere to those (and there isn't really a compelling reason not to, even if you're only targeting Linux), you should be safe.

Simple fix: Only add ./ if bash know is it a file

farnz — Thu, 19 May 2016 09:59:46 +0000

You missed case 3 - Bash did not have to scan the file system, but the user's intent was to match a file. For example rm log-{user,system}.txt. There's no way for Bash to detect that sanely without adding in a file system scan - but then the file system scan can cause bash to do the wrong thing for someone who does rm --{force,recursive}.

Simple fix: Only add ./ if bash know is it a file

gmatht — Thu, 19 May 2016 09:57:47 +0000

It it seems that there are only two cases
1) Bash knows that the glob is the file, because it had to scan the file system
2) Bash didn't have to scan the file system, so it safe to omit the './'

There are other ways adding ./ could break existing scripts though. For example, the following would no longer give you a list of naughty users:

cd /home; find * -name naughty.jpg | sed s,/.*,,g | sort -u

I guess "bash -n" could warn about unsafe use of "*", warn interactive users, and we could scan all packages for unsafe use of *.

Safename: restricting "dangerous" file names

nix — Wed, 18 May 2016 22:24:57 +0000

(Not, of course, that banning such names would help in the least: it would just stop me untarring the trees at all! And, of course, this proposal doesn't ban spaces in filenames by default in any case.)

Safename: restricting "dangerous" file names

nix — Wed, 18 May 2016 22:24:11 +0000

I didn't have one, in particular: I was just pointing out that not all applications *can* use proper quoting, so you're reduced to hoping that people don't use names containing spaces, which seems a forlorn hope: I have multiple source trees on this system alone that contain such names (mostly thanks to MacOS X developers, I think).

Safename: breaking compatibility beteen system

mjg59 — Wed, 18 May 2016 22:00:19 +0000

This is true of many security features - policy will vary between systems.

Safename: breaking compatibility beteen system

ballombe — Wed, 18 May 2016 19:40:49 +0000

This proposal will break break basic compatibility between systems by having each of them have different rules whether to accept or reject a filename.

Safename: restricting "dangerous" file names

nybble41 — Wed, 18 May 2016 18:14:35 +0000

> Actually the shell has some kind of this knowledge, used for completion like at https://github.com/scop/bash-completion.

The shell provides a framework for programmable completion as an aid to interactive users. It doesn't have any of that knowledge built in, the database may be incomplete or inaccurate, and even in an interactive setting the programmable completion feature may not be enabled. (I personally find it more annoying than helpful and prefer the more predictable, command-independent traditional completion, but tastes vary.) It also isn't generally available to scripts, which is where the problem lies.

Safename: restricting "dangerous" file names

flussence — Wed, 18 May 2016 18:07:20 +0000

I'd like it if more programs made use of filehandles and envvars. Having a different API as the boundary between input types instead of parsing magic tokens in argv just seems saner.

Just as one example, imagine `find` having an -execv [envvar_name] option instead of having to deal with its horrible inline syntax...

Safename: restricting "dangerous" file names

tao — Wed, 18 May 2016 13:05:59 +0000

Command line options is something that has always been broken, and is unlikely to ever unbreak.

tar, ar, ps, etc. don't even prefix their options with a dash (though you can).
Some commands require -- for long options, others don't.
Some commands enforce (or at least warn about) the ordering of options (find, for instance), others don't.

Safename: restricting "dangerous" file names

NAR — Wed, 18 May 2016 12:22:18 +0000

The shell doesn't (and shouldn't) have any knowledge of what these external commands do, or how they interpret their arguments.

Actually the shell has some kind of this knowledge, used for completion like at https://github.com/scop/bash-completion.

Safename: restricting "dangerous" file names

farnz — Wed, 18 May 2016 10:48:06 +0000

Unfortunately, because this has historically worked, it's now expected behaviour; I've worked with people who consider it the "definitive" style for options, and who prefer to do things like rm --{force,recursive,verbose} because they think that's clearer than rm --force --recursive --verbose. If that's deep in a script, my modified shell is going to break.

Hence coming back to my original point; had early UNIX authors foreseen this gotcha, they could have required relative paths to begin ./, and thus avoided all this pain down the line, because rm would never have been passed a filename without the ./. Now, though, it's too late - too much legacy to fix.

Safename: restricting "dangerous" file names

itvirta — Wed, 18 May 2016 10:39:33 +0000

> ...scripts containing things like rm -{f,r,v} and expect it to work.

Uh, that's hideous. Luckily rm -frv usually works, and even rm -f -r -v is probably easier to write than that.
(Exactly the same amount of characters too.) So hopefully there's no need for that.

Safename: restricting "dangerous" file names

farnz — Wed, 18 May 2016 09:59:30 +0000

That's not enough; you need anything that the shell creates as a "filename" to expand to ./result; otherwise rm -* (typo) is also ambiguous. Plus, you need to do this before the current behaviour gets set in historic tradition, so that people don't write scripts containing things like rm -{f,r,v} and expect it to work.

Safename: restricting "dangerous" file names

dlang — Tue, 17 May 2016 20:51:50 +0000

just make sure you don't change foo* to foo./* in the process.

Safename: restricting "dangerous" file names

hummassa — Tue, 17 May 2016 20:44:07 +0000

You inadvertently came up with a nice and neat (IMHO) solution.

Just adjust your shell so that * and .* globs expand to ./* and ./.*, respectively

Safename: restricting "dangerous" file names

farnz — Tue, 17 May 2016 19:12:36 +0000

I didn't. -fr is both a legitimate filename and an option to rm. The user can disambiguate them for rm by using a ./ prefix to say definitely a filename, but there is no equivalent for options, and the prefix is not required (nor is it put in there by shell expansions).

Thus the user can choose to be unambiguous, but if they are ambiguous, rm can't tell what they really meant - did I get -fr into argv[1] via glob expansion or tab completion (probably a filename) or by typing a literal -fr (probably an option).