Quotes of the week

Posted Aug 23, 2007 8:15 UTC (Thu) by AnswerGuy (guest, #1256) [Link]

All of this can be easily demonstrated using commands like:

/bin/echo $(python -c 'print "foo " * 10 ** 5')

... (try to use an external echo command to on an argument list consisting of a 100,000 instances of the word "foo ")

... or with something like:

/bin/echo /* /*/* /*/*/* /*/*/*/*

... (which will try to echo the names of all the files and directory down to about four-levels deep on your system's file-tree)

... and that this is coupled to the environment is demonstrated by:

export FOO="$(python -c 'print "a" * 10 ** 8')

... followed by any normal external command (such as ''ls'').

In both cases you will get "Argument list to long" and if you were to strace these you'd find that the -E2BIG error was being returned by the execve() system call. (The fact that the environment is copied in with the argument list is a little surprising to some --- but not to those of us who have seen the old, pre-K&R prototype for int main (int argc, char **argv, char **env) --- a pointer to the environment is being passed along with the argument count and the argument vector. It's still being passed to modern C programs under Unix/Linux --- it's just that direct use of that third argument/pointer is deprecated in favor of using the environ.h defined functions in libc ... which are intended to prevent corruption of your environment).

The fact that rm cannot normally remove some arbitrarily long list of files is simply a consequence of how UNIX shells expand glob patterns (on the one hand) and how most UNIX kernels have traditionally implemented execve() --- and this limitation does currently apply to Linux.

The fact that MS-DOS' "DEL" command can, theoretically, handle an unlimited number of files is a consequence their command line handling APIs. Specifically MS-DOS "wildcard patterns" (which are not precisely the same as UNIX glob patterns --- in ways that normal people need not concern themselves). MS-DOS programs make a library/API call to the "find first" function (don't remember the exact spelling/punctuation and character case for its name and don't feel like looking it up). Thereafter such programs go into a loop calling the "find next" until that returns an error or sentinel indicating that there are no more "next" matches.

Naturally one *could* implement a similar API under UNIX/Linux. One *could* have users call a command (perhaps naming it 'DEL' which if one passed it (quoted) wildcard patterns (quoted to avoid having the shell mistake them for glob patterns). Such a library or program *could* then use opendir() and implement a "findfirst()" with "findnext()" semantics. One could even write a shell named COMMAND.COM and have it implement these semantics for all it's built-in commands and assume that such semantics would be followed by any external commands.

None of this points to any fundamental deficiency in UNIX nor Linux (nor even in MS-DOS, for that matter). They are simply differences.

The quote, while amusing as a joke doesn't withstand a moment's worth of competent technical scrutiny. Any sysadmin worth his salt knows how to use find ... | xargs rm or the less efficient find ... -exec 'rm' '{}' \; or the more exotic perl -e "unlink <$GLOB>;"

(If a prospective intermediate to advanced Linux/UNIX sysadmin, can't tell me at least one way to handle "argument list too long" in an interview, then he or she probably won't get my recommendation for the job. An *advanced* one should be able to tell me *why* it's that way as well as some way to work around it).

JimD

Posted Aug 23, 2007 6:31 UTC (Thu) by pebolle (guest, #35204) [Link]

rename .cpp .cc *.cpp

Posted Aug 23, 2007 6:34 UTC (Thu) by jimparis (subscriber, #38647) [Link]

$ for i in *.cpp; do mv $i ${i%cpp}.cc ; done

If you don't use bash, you could use something like sed:

$ for i in *.cpp; do mv $i $(echo $i | sed s/.cpp$/.cc/) ; done

Yeah, for some things, the DOS way of handling it is a lot easier. But Unix is also a lot more powerful -- you could put literally anything inside that sed command. That's what I like about the Unix command line -- it's not pretty, but once you start learning the patterns, my interactions all feel like a straightforward "let me clearly specify what I want to do" regardless of how complicated they are.

Posted Aug 23, 2007 6:40 UTC (Thu) by asamardzic (guest, #27161) [Link]

In situations like this I have resorted to a for-loop that calls a small self-written program (for $name in *.oldsuffix; do mv $name `suffrep $name .newsuffix`; done), but is there more standard way?

for name in *.oldsuffix; do mv $name ${name%%.oldsuffix}.newsuffix; done

Posted Aug 23, 2007 8:41 UTC (Thu) by AnswerGuy (guest, #1256) [Link]

The simplest and most practical alternative is to rethink the problem and use a subdirectory. For example instead of COPY *.TXT *.BAK use ''mkdir ./bak; cp *.txt ./bak'' Often this will actually work better.

However, for cases where you really have a set of files which have to be renamed or copied based on some pattern replacement, you'd generally think in terms of "sed" (you are substituting some substring in the existing name with some text, which is precisely the most common use of sed).

So you use commands like:

for i in *.cxx; do mv $i $( echo $i | sed -e 's/\.cxx$/.cpp/'); done

However, that looks ugly and verbose. You can often use simpler constructs like:

for i in *.cxx; do mv $i $(basename $i .cxx).cpp; done

(Using GNU basename's ability to strip off filename extensions in addition to any leading pathname components).

You can also use Bash "parameter substitution" operators like ${x##...} or ${x%%...} or the relatively new ${x//...} glob substitution stuff.

For example:

for i in *.cxx; do mv $i ${i#.cxx}.cpp; done

(You can read more about these ${variable*} parameter substitution expressions in the GNU bash man pages --- and Korn shell and even some
simpler Bourne shells like ash, dash etc support some of them as well).

Of course all of these look more complicated than simple REN *.CXX *.CPP,
but that's, in part, because MS-DOS has such a constrained notion of what a filename looks like. The interpretation of *.CXX *.CPP to mean replace the characters CXX with CPP is rather idiomatic and makes no sense in a system which doesn't constrain filenames to a single dot followed by up to three characters for an extension.

The fact that UNIX supports free form filenames necessitates greater precision when trying to perform these sorts of patterned name replacements.

JimD

Posted Aug 23, 2007 11:50 UTC (Thu) by cortana (subscriber, #24596) [Link]

$ foo=bar.baz
$ echo ${foo%.baz}.qux
bar.qux

This may be a bashism of course. :)

Posted Aug 23, 2007 15:33 UTC (Thu) by chrish (subscriber, #351) [Link]

mmv '*.cpp' '#1.cc'

(There's also mcp, mln, and even "mad", which appends to the target.)

Posted Aug 24, 2007 6:39 UTC (Fri) by mingo (subscriber, #31122) [Link]

The concept itself (of making exec() arguments unlimited) has been in the works for years literally - this latest (and finally successful!) implementation from Ollie Wild (@Google) was written and first posted last year, it went through several iterations until it got merged upstream more than a month ago:

 commit b6a2fea39318e43fee84fa7b0b90d68bed92d2ba
 Author: Ollie Wild < aaw@google.com>
 Date:   Thu Jul 19 01:48:16 2007 -0700

    mm: variable length argument support
    
    Remove the arg+env limit of MAX_ARG_PAGES by copying the strings directly from
    the old mm into the new mm.
    
    We create the new mm before the binfmt code runs, and place the new stack at
    the very top of the address space.  Once the binfmt code runs and figures out
    where the stack should be, we move it downwards.
    
    It is a bit peculiar in that we have one task with two mm's, one of which is
    inactive.

That indeed makes the flamewa^H^H discussion useless and moot, on a cosmic scale ;-)

The code itself is quite nontrivial, as it touches a very critical (and complex) piece of kernel code, and because assumptions around the old limit were deeply embedded in the exec() code:

 15 files changed, 554 insertions(+), 453 deletions(-)

which explains why this enhancement took years. Kudos to Ollie!