Python for system administrators (developerWorks)

Posted Sep 6, 2007 19:08 UTC (Thu) by njs (subscriber, #40338)
In reply to: Python for system administrators (developerWorks) by IkeTo
Parent article: Python for system administrators (developerWorks)

Well, we aren't going to resolve the language wars here :-), so just two more quick comments:

a regex is usually the clearest way to express what you want to do

If this is true for you, then compared to me you have either very different things to do, or a very different brain.

for example, "line = ....readline();" followed by something like "while line:" followed by something like "line.chomp()" (not that I think there is such thing as "chomp" in Python) and remember to add "line = ....readline();" before ending the while loop.

For the record, the way to do something like this in python is the not-particularly-clunky:

for line in myfile:
    line = line.strip()
    # do other stuff with line here

(Or if you want the perl <> magic, then import fileinput and use 'for line in fileinput.input():'.)

Python for system administrators (developerWorks)

Posted Sep 7, 2007 6:08 UTC (Fri) by madscientist (subscriber, #16861) [Link] (7 responses)

> > a regex is usually the clearest way to express what you want to do

> If this is true for you, then compared to me you have either very different
> things to do, or a very different brain.

I wonder if there's truth to this. I find regex's incredibly elegant and almost always preferable to any other possible method of string manipulation. Just the other day I was looking at some C++ code that parsed some strings and I threw it all away and replaced it with a trivial invocation of the C runtime library's regcomp()/regexec(), and that's a pretty weak version of regex. Why wouldn't you?

On the other hand I know some people who run screaming from the room at the site of anything more complex than a shell globbing expression.

Maybe there's something basically different about RE lovers and RE haters.

Python for system administrators (developerWorks)

Posted Sep 7, 2007 7:51 UTC (Fri) by oak (guest, #2786) [Link] (1 responses)

I don't understand what's the problem in using regrexps in Python.

Yes, they need a bit more code around them than in Perl (in Python you
explicitly compile your regexps for better performance and you get "match"
objects from the regexp matches), but at least they don't create
side-effects behind your back.

I would choose on any day a language which requires writing a bit more,
but when there's a problem, it's obvious why it happens (Perl: write 10
minutes, debug 1 hour; Python: write 15 minutes, debug 5 minutes).

As to which one is a system administrators' language, for example many
RedHat and Ubuntu system administrator tools are written in Python. I
guess if you would count the lines of code in the normal desktop install,
there could be more Python than Perl (or Php :)).

Python for system administrators (developerWorks)

Posted Sep 7, 2007 15:37 UTC (Fri) by madscientist (subscriber, #16861) [Link]

> I don't understand what's the problem in using regrexps in Python.

I didn't say there was one: this threadlet is discussing whether REs in general constitute unclear coding constructs and should be avoided. I personally believe that REs are almost always the best way to manage string manipulation: they are succinct but straightforwardly understandable, and I was saying that I even use them in non-scripting languages such as C or C++ when I need to parse strings. The question asked was if there really were two classes of people: those who grok REs and those who don't. I don't know but it's an interesting question.

> I would choose on any day a language which requires writing a bit more,
> but when there's a problem, it's obvious why it happens (Perl: write 10
> minutes, debug 1 hour; Python: write 15 minutes, debug 5 minutes).

Needless to say I disagree with your assertion. Using -w and "use strict;" mode in Perl, along with a bit of discipline to avoid egregiously bizarre constructs at the expense of a bit of typing (which is not at all difficult, really--and needed in virtually any language, as the article being referenced here obviously shows!!), and Perl is just as straightforward to read and write as any other language. I've written HUGE packages in Perl, using lots of Perl OO, complex data structures, etc. and they were no more complex to read, and took no longer to create, than code of the same complexity in any other language (and less than many).

I like Perl because I like Lisp, or at least Lisp-like thinking, and I can write Perl in that mode and it's still easily understandable to people who don't have a Lisp background. I can't remember the last time I wrote a loop in Perl that used an iterator variable, for example. I find map{} to be a perfect operator: it's immensely powerful while at the same time getting out of your way syntactically and letting you understand what the code is doing, instead of the details of how it's done. I guess my brain is wired more closely to Larry Wall's than to Guido van Rossum's :)

Anyway, this is not the place for a language war, as others have said. I think Python is great and I think you can be extremely productive in it. I use Python all the time and I've written my fair share of Python scripts. I just don't think Perl is as horrible as many Python programmers seem to think that it is, and I have no problem at all creating maintainable, readable Perl even for large, complex systems.

Python for system administrators (developerWorks)

Posted Sep 9, 2007 14:43 UTC (Sun) by gravious (guest, #7662) [Link] (4 responses)

Any reason you didn't use Boost.Regex here? Just wondering... Is it because regex.h was built-in and to hand?

Python for system administrators (developerWorks)

Posted Sep 9, 2007 16:55 UTC (Sun) by madscientist (subscriber, #16861) [Link] (3 responses)

Yes, and Boost isn't available in our environment. That's a good suggestion, though: Boost has some great stuff in it. I would love Boost much more if it were autotooled (and I don't mean just a small "configure" wrapper on the outside). I know many people don't like autoconf etc., and I don't want to argue about that, but the Boost build environment is (was) a huge pain. Our environment is embedded and so we have to do cross-compilation and get output for lots of different targets, and the last time I tried to integrate Boost into this (admittedly this was 3 years or so ago) it was extremely difficult. If Boost were based on make then at least we could easily override the compiler, etc. settings from the command line but doing that with Boost's customized version of "jam" seems annoyingly difficult.

Autoconf/automake/libtool are baroque but they are founded on a lot of hard-won, difficult to come by knowledge about different types of user environments, including building for many targets from the same source, cross-compilation, building outside of the tree (from read-only source trees), etc. that, for the end user, mean so much.

Python for system administrators (developerWorks)

Posted Sep 10, 2007 1:44 UTC (Mon) by bronson (subscriber, #4806) [Link] (1 responses)

I agree 100%, trying to build Boost is utter torture. I tried to write robust deploy scripts for vertical apps that link against Boost and mostly failed.

Happily, though, everything my apps needed was header-only. With only minor tweaks, I deployed the apps without actually building Boost!

So, definitely try #including the Boost files you're interested in, but don't link against any Boost libs. There's a very good chance that this will work.

Regexes are a notable exception -- you definitely need to build the libraries for those.

Python for system administrators (developerWorks)

Posted Sep 19, 2007 20:11 UTC (Wed) by nix (subscriber, #2304) [Link]

I managed it. It took a couple of hundred lines of patches just to
reliably set the CFLAGS, LDFLAGS, and install prefix.

Compared to autoconf or cmake, that's crazy.

Python for system administrators (developerWorks)

Posted Sep 19, 2007 13:54 UTC (Wed) by gravious (guest, #7662) [Link]

Thanks for the heads up you Mad Scientist you, I was curious - consider my curiosity sated. You're basically saying your non-use is a function of Boost's build system in general and not the quality of the Boost.Regex code in particular.

Python for system administrators (developerWorks)

Posted Sep 7, 2007 8:24 UTC (Fri) by IkeTo (subscriber, #2122) [Link] (3 responses)

> > a regex is usually the clearest way to express what you want to do

> If this is true for you, then compared to me you have either very different
> things to do, or a very different brain.

Or perhaps, you have seen too many horribly complex regex. In a scripting language, if people want a bit of performance they do a bit of regex hack, since that is easier than to cerate a C library. Those are going to be hard to read!

One example to remind you that simple things exists: if I have a string like "The document is owned by (owner) written at (date), he/she own (other) other document(s) as well" extracted from a file (e.g., allow customization of strings, ordering or things in parentheses), and we have a hash like { owner => 'IkeTo', date => '2007-09-07', time => '10:00 GMT', other => '3' }, what is your clearest way to combine them to "The document is owned by IkeTo written at 2007-09-07, he/she own 3 other document(s) as well"? I would do it like this:

  $hash = { owner => 'IkeTo', date => '2007-09-07', time => '10:00 GMT',
	    other => '3' };
  $string = ...; # load the string
  $string =~ s{ \(
                (
                  \w+
                )
                \) }
              { defined($hash->{$1}) ? $hash->{$1} : "($1)" }gex;
  print $string, "\n";

Which my brain processes it to say "I'd search globally for things starting with '(', followed by a word, followed by a ')', and replace that with the content of the element indexed by that word in the hash if that exists, or just the original if otherwise. Now try to do it without regex and see whether you end up with something more elegant and clear.

> (Or if you want the perl <> magic, then import fileinput and use 'for line
> in fileinput.input():'.)

Thanks for your suggestion. I should read the docs more! =) My real point is that the $_ in Perl (the "line" in the corresponding Python code) is the default and, and it is good to have this as an option because this means the brain doesn't have to process it. Sorry for making that point unclear.

Python for system administrators (developerWorks)

Posted Sep 7, 2007 13:01 UTC (Fri) by azazel (guest, #4363) [Link] (2 responses)

In python, with a slight markup change:

dic = {owner: 'IkeTo', date: '2007-09-07', time: '10:00 GMT', other: '3'}

message = "The document is owned by %(owner)s written at %(date)s, he/she own %(other)s other document(s) as well"

print message % dic

Python for system administrators (developerWorks)

Posted Sep 7, 2007 13:18 UTC (Fri) by azazel (guest, #4363) [Link] (1 responses)

Ooops, i forgot to quote dic's keys, the right is:

dic = {'owner': 'IkeTo', 'date': '2007-09-07', 'time': '10:00 GMT', 'other': '3'}

another example, maintaining the same markup of the Perl example:

dic = {'owner': 'IkeTo', 'date': '2007-09-07', 'time': '10:00 GMT', 'other': '3'}

message = "The document is owned by (owner) written at (date), he/she own (other) other document(s) as well"

from pyparsing import Word, Suppress, alphas

token = Suppress('(') + Word(alphas)('label') + Suppress(')')

token.setParseAction(lambda tok: dic.get(tok.label, tok.label))

print token.transformString(message)

Python for system administrators (developerWorks)

Posted Sep 8, 2007 7:10 UTC (Sat) by IkeTo (subscriber, #2122) [Link]

Your first solution is uninteresting. My input is not tailored to Perl, your input is specifically tailored to Python. There are a lot of further customization that can be done in the original Perl code, there is essentially none that can still be done in the Python code.

Your second solution is much more interesting (even though it is not perfectly correct: you forgot to add the parentheses in case of hash miss). And thanks for letting me know about PyParsing.

Yes, regex is a simple parser. Indeed it doesn't quite have the power of PyParsing, and in complex cases regex looks real ugly. Of course, Perl people will use a library to counter it (like Parse::RecDescent), but that's not the best solution. If the language support some feature it should be support the feature well. Hope that Perl6 get traction soon, which replace regex by grammar and provide full recursive decent parser power in the language.