Python for system administrators (developerWorks)
Python for system administrators (developerWorks)
Posted Sep 6, 2007 7:58 UTC (Thu) by IkeTo (subscriber, #2122)In reply to: Python for system administrators (developerWorks) by mrons
Parent article: Python for system administrators (developerWorks)
The author is clueless. Just look at how he use a catch-all exception handler, doing nothing apart from hiding what exactly happened (it is much easier and better to just do nothing and let exception bail out to tell the admin what happened). If this is not enough, see his use of commands.getoutput() and then split() to makes sure shell special characters breaks the program, with the additional virtue to waste a big chunk of memory if the output is long (it would be better to use subprocess.Popen and readline).
But then, I don't think Python is the system administrators' language. It belongs to the programmers who want to implement their algorithms quickly and flexibly without worrying about running time. All its neat features like list comprehension, generator expression, generator functions and metaclasses are far too much to learn by system admins. It does provide a large library and a clean syntax, but for the former the system admins are better off learning Perl, for the latter they are better off learning how to write Perl in a clean way. Better to learn Perl because (1) at least the shell-like backticks work, and (2) the library is much more feature rich so they are less likely to hit a limitation of the library (something like "you cannot get the exit code from commands.getoutput()").
Posted Sep 6, 2007 8:09 UTC (Thu)
by njs (subscriber, #40338)
[Link] (16 responses)
(I don't do a whole lot of sysadminning per se these days, but I do do lots of quick data munging to glue stuff together -- one of perl's traditional home grounds -- and have mostly moved over to python for that. It's not just for high-falutin' algorithmics.)
Posted Sep 6, 2007 8:55 UTC (Thu)
by IkeTo (subscriber, #2122)
[Link] (15 responses)
I had been working for a company which have more than half of the code in Perl (although I'm in a developer's role rather than a system admin's role). At the start of the job I think in the same way as you do. But after a year I no longer believe it is true. You can write really ugly code in Perl. Many people do write really ugly code in Perl. But it doesn't mean that Perl is ugly. It just means that Perl is easy enough that many clueless people who cannot write pretty code can still write in Perl (in other language they would have to be re-educated). Throw those people to another language they will write ugly Java, ugly Python, ugly C++, etc., until educated. On the other hand, when properly educated, Perl has an unsurpassed expressive power, and is a DWIMer's (DWIM = Do What I Mean) dream.
As for the comment about one-off hard-to-read command, they exist, but there is no gun pointing to your head forcing you to write them. I personally never do.
Posted Sep 6, 2007 12:31 UTC (Thu)
by njs (subscriber, #40338)
[Link] (14 responses)
FWIW, I actually went the other way -- I spent years writing in perl, resisting friends' suggestions that python was worth looking at, and making the same argument you do ("perl doesn't obfuscate programs, programmers obfuscate programs").
The problem is that we never actually have infinite effort to spend on a program, so there's no use in comparing the best possible perl to the best possible python; if you do compare, then yeah, they're pretty much indistinguishable. But my experience is that for any given level of effort, python will be more readable. So this actually matters the most when I'm hacking up a quick script, like the sysadmin use case -- I'm writing really ugly python, but at least it has nice indentation, unambiguous syntax, and minimal regex mind-bendingness. We all have our weak moments; in python I regret mine a lot less.
DWIM, btw, is a paragon bad idea when you care about reading your code later. It's all about optimizing for writing -- I know what I want, I want the minimal code that matches it. Code reading is the opposite problem: I know what the code says, I'm trying to figure out what the person was thinking (insert expletives as necessary). The more magic the interpreter does, the more thinking I have to do to invert it in my head.
Posted Sep 6, 2007 13:27 UTC (Thu)
by IkeTo (subscriber, #2122)
[Link] (13 responses)
An example of what I'll call "good" DWIM feature is the use of $_ by default for many calls. After something like "while (<>) {", if I say "chomp" neither the writer nor the reader should think I'm chomping something else. Writing like that is clearer than saying, for example, "line = ....readline();" followed by something like "while line:" followed by something like "line.chomp()" (not that I think there is such thing as "chomp" in Python) and remember to add "line = ....readline();" before ending the while loop. An example of what I'll call "bad" DWIM feature is the use of parenthesis to influence the operator precedence. It is probably not helpful for the writer nor for a code reviewer if "print 'Hello', ' ', 'world'" works fine but "print ('Hello' . ' '), 'world'" doesn't.
So in Perl I had to choose, but if I choose correctly I get my message across very efficiently. In Python I can't choose. I can get my message across efficiently, but never "very" efficiently. Preference? I think I have told you at the beginning, I do both.
Posted Sep 6, 2007 19:08 UTC (Thu)
by njs (subscriber, #40338)
[Link] (12 responses)
a regex is usually the clearest way to express what you want to do
If this is true for you, then compared to me you have either very different things to do, or a very different brain.
for example, "line = ....readline();" followed by something like "while line:" followed by something like "line.chomp()" (not that I think there is such thing as "chomp" in Python) and remember to add "line = ....readline();" before ending the while loop.
For the record, the way to do something like this in python is the not-particularly-clunky:
(Or if you want the perl <> magic, then import fileinput and use 'for line in fileinput.input():'.)
Posted Sep 7, 2007 6:08 UTC (Fri)
by madscientist (subscriber, #16861)
[Link] (7 responses)
> If this is true for you, then compared to me you have either very different
I wonder if there's truth to this. I find regex's incredibly elegant and almost always preferable to any other possible method of string manipulation. Just the other day I was looking at some C++ code that parsed some strings and I threw it all away and replaced it with a trivial invocation of the C runtime library's regcomp()/regexec(), and that's a pretty weak version of regex. Why wouldn't you?
On the other hand I know some people who run screaming from the room at the site of anything more complex than a shell globbing expression.
Maybe there's something basically different about RE lovers and RE haters.
Posted Sep 7, 2007 7:51 UTC (Fri)
by oak (guest, #2786)
[Link] (1 responses)
Yes, they need a bit more code around them than in Perl (in Python you
I would choose on any day a language which requires writing a bit more,
As to which one is a system administrators' language, for example many
Posted Sep 7, 2007 15:37 UTC (Fri)
by madscientist (subscriber, #16861)
[Link]
I didn't say there was one: this threadlet is discussing whether REs in general constitute unclear coding constructs and should be avoided. I personally believe that REs are almost always the best way to manage string manipulation: they are succinct but straightforwardly understandable, and I was saying that I even use them in non-scripting languages such as C or C++ when I need to parse strings. The question asked was if there really were two classes of people: those who grok REs and those who don't. I don't know but it's an interesting question.
> I would choose on any day a language which requires writing a bit more,
Needless to say I disagree with your assertion. Using -w and "use strict;" mode in Perl, along with a bit of discipline to avoid egregiously bizarre constructs at the expense of a bit of typing (which is not at all difficult, really--and needed in virtually any language, as the article being referenced here obviously shows!!), and Perl is just as straightforward to read and write as any other language. I've written HUGE packages in Perl, using lots of Perl OO, complex data structures, etc. and they were no more complex to read, and took no longer to create, than code of the same complexity in any other language (and less than many).
I like Perl because I like Lisp, or at least Lisp-like thinking, and I can write Perl in that mode and it's still easily understandable to people who don't have a Lisp background. I can't remember the last time I wrote a loop in Perl that used an iterator variable, for example. I find map{} to be a perfect operator: it's immensely powerful while at the same time getting out of your way syntactically and letting you understand what the code is doing, instead of the details of how it's done. I guess my brain is wired more closely to Larry Wall's than to Guido van Rossum's :)
Anyway, this is not the place for a language war, as others have said. I think Python is great and I think you can be extremely productive in it. I use Python all the time and I've written my fair share of Python scripts. I just don't think Perl is as horrible as many Python programmers seem to think that it is, and I have no problem at all creating maintainable, readable Perl even for large, complex systems.
Posted Sep 9, 2007 14:43 UTC (Sun)
by gravious (guest, #7662)
[Link] (4 responses)
Posted Sep 9, 2007 16:55 UTC (Sun)
by madscientist (subscriber, #16861)
[Link] (3 responses)
Autoconf/automake/libtool are baroque but they are founded on a lot of hard-won, difficult to come by knowledge about different types of user environments, including building for many targets from the same source, cross-compilation, building outside of the tree (from read-only source trees), etc. that, for the end user, mean so much.
Posted Sep 10, 2007 1:44 UTC (Mon)
by bronson (subscriber, #4806)
[Link] (1 responses)
Happily, though, everything my apps needed was header-only. With only minor tweaks, I deployed the apps without actually building Boost!
So, definitely try #including the Boost files you're interested in, but don't link against any Boost libs. There's a very good chance that this will work.
Regexes are a notable exception -- you definitely need to build the libraries for those.
Posted Sep 19, 2007 20:11 UTC (Wed)
by nix (subscriber, #2304)
[Link]
Compared to autoconf or cmake, that's crazy.
Posted Sep 19, 2007 13:54 UTC (Wed)
by gravious (guest, #7662)
[Link]
Posted Sep 7, 2007 8:24 UTC (Fri)
by IkeTo (subscriber, #2122)
[Link] (3 responses)
> > a regex is usually the clearest way to express what you want to do
> If this is true for you, then compared to me you have either very different
Or perhaps, you have seen too many horribly complex regex. In a scripting language, if people want a bit of performance they do a bit of regex hack, since that is easier than to cerate a C library. Those are going to be hard to read!
One example to remind you that simple things exists: if I have a string like "The document is owned by (owner) written at (date), he/she own (other) other document(s) as well" extracted from a file (e.g., allow customization of strings, ordering or things in parentheses), and we have a hash like { owner => 'IkeTo', date => '2007-09-07', time => '10:00 GMT', other => '3' }, what is your clearest way to combine them to "The document is owned by IkeTo written at 2007-09-07, he/she own 3 other document(s) as well"? I would do it like this:
Which my brain processes it to say "I'd search globally for things starting with '(', followed by a word, followed by a ')', and replace that with the content of the element indexed by that word in the hash if that exists, or just the original if otherwise. Now try to do it without regex and see whether you end up with something more elegant and clear.
> (Or if you want the perl <> magic, then import fileinput and use 'for line
Thanks for your suggestion. I should read the docs more! =) My real point is that the $_ in Perl (the "line" in the corresponding Python code) is the default and, and it is good to have this as an option because this means the brain doesn't have to process it. Sorry for making that point unclear.
Posted Sep 7, 2007 13:01 UTC (Fri)
by azazel (guest, #4363)
[Link] (2 responses)
dic = {owner: 'IkeTo', date: '2007-09-07', time: '10:00 GMT', other: '3'}
message = "The document is owned by %(owner)s written at %(date)s, he/she own %(other)s other document(s) as well"
print message % dic
Posted Sep 7, 2007 13:18 UTC (Fri)
by azazel (guest, #4363)
[Link] (1 responses)
dic = {'owner': 'IkeTo', 'date': '2007-09-07', 'time': '10:00 GMT', 'other': '3'}
another example, maintaining the same markup of the Perl example:
dic = {'owner': 'IkeTo', 'date': '2007-09-07', 'time': '10:00 GMT', 'other': '3'}
message = "The document is owned by (owner) written at (date), he/she own (other) other document(s) as well"
from pyparsing import Word, Suppress, alphas
token = Suppress('(') + Word(alphas)('label') + Suppress(')')
token.setParseAction(lambda tok: dic.get(tok.label, tok.label))
print token.transformString(message)
Posted Sep 8, 2007 7:10 UTC (Sat)
by IkeTo (subscriber, #2122)
[Link]
Your second solution is much more interesting (even though it is not perfectly correct: you forgot to add the parentheses in case of hash miss). And thanks for letting me know about PyParsing.
Yes, regex is a simple parser. Indeed it doesn't quite have the power of PyParsing, and in complex cases regex looks real ugly. Of course, Perl people will use a library to counter it (like Parse::RecDescent), but that's not the best solution. If the language support some feature it should be support the feature well. Hope that Perl6 get traction soon, which replace regex by grammar and provide full recursive decent parser power in the language.
Perl is definitely better for writing programs up to, say, 80 characters in length, and that covers an awful lot of sysadmin programs. The problem is that what starts as a quick one-off command often will start growing into a more general tool, and python is *way* better when you need to read and modify the program later (IMHO, but of course MHO is *objectively* correct!). You never know which "one-off" programs are going to grow into reusable ones, so best to just use python all the time -- and in fact, this may encourage you to reuse these programs more...Python for system administrators (developerWorks)
> Perl is definitely better for writing programs up to, say, 80 characters inPython for system administrators (developerWorks)
> length, and that covers an awful lot of sysadmin programs. The problem is
> that what starts as a quick one-off command often will start growing into a
> more general tool, and python is *way* better when you need to read and
> modify the program later (IMHO, but of course MHO is *objectively*
> correct!).
>At the start of the job I think in the same way as you do. But after a year I no longer believe it is true.Python for system administrators (developerWorks)
I have written Python (even before entering the company I've mentioned), and I still do it some times. I no longer work for that company, but I still write Perl some times. I do not really "regret" using either. Python enforces indentation, which is a good thing, in Perl I'd just ask Emacs to indent everything I write, which is also a good thing. In Python I tend not to use regular expression at all because whenever I use it I need to look into the docs, in Perl I use it more extensively but never to the point where it is hard to know my meaning (well... at least, not without adding a line of comment around). I don't think "making sure you have thought of alternatives before using regex" should be a virtue of a language: it should be a virtue of a programming team, after all a regex is usually the clearest way to express what you want to do. As for your comment about DWIM, I think different people have to differ here, but I believe DWIM means the thoughts and the code are actually as close as one want. It should benefit both the writer and the reader, otherwise it is not DWIM, at least not "good" DWIM. And it should mean both sides should be able to work more efficiently. YMMV, of course.Python for system administrators (developerWorks)
Well, we aren't going to resolve the language wars here :-), so just two more quick comments:
Python for system administrators (developerWorks)
for line in myfile:
line = line.strip()
# do other stuff with line here
> > a regex is usually the clearest way to express what you want to doPython for system administrators (developerWorks)
> things to do, or a very different brain.
I don't understand what's the problem in using regrexps in Python.Python for system administrators (developerWorks)
explicitly compile your regexps for better performance and you get "match"
objects from the regexp matches), but at least they don't create
side-effects behind your back.
but when there's a problem, it's obvious why it happens (Perl: write 10
minutes, debug 1 hour; Python: write 15 minutes, debug 5 minutes).
RedHat and Ubuntu system administrator tools are written in Python. I
guess if you would count the lines of code in the normal desktop install,
there could be more Python than Perl (or Php :)).
> I don't understand what's the problem in using regrexps in Python.Python for system administrators (developerWorks)
> but when there's a problem, it's obvious why it happens (Perl: write 10
> minutes, debug 1 hour; Python: write 15 minutes, debug 5 minutes).
Any reason you didn't use Boost.Regex here? Just wondering... Is it because regex.h was built-in and to hand?Python for system administrators (developerWorks)
Yes, and Boost isn't available in our environment. That's a good suggestion, though: Boost has some great stuff in it. I would love Boost much more if it were autotooled (and I don't mean just a small "configure" wrapper on the outside). I know many people don't like autoconf etc., and I don't want to argue about that, but the Boost build environment is (was) a huge pain. Our environment is embedded and so we have to do cross-compilation and get output for lots of different targets, and the last time I tried to integrate Boost into this (admittedly this was 3 years or so ago) it was extremely difficult. If Boost were based on make then at least we could easily override the compiler, etc. settings from the command line but doing that with Boost's customized version of "jam" seems annoyingly difficult.Python for system administrators (developerWorks)
I agree 100%, trying to build Boost is utter torture. I tried to write robust deploy scripts for vertical apps that link against Boost and mostly failed.Python for system administrators (developerWorks)
I managed it. It took a couple of hundred lines of patches just to Python for system administrators (developerWorks)
reliably set the CFLAGS, LDFLAGS, and install prefix.
Thanks for the heads up you Mad Scientist you, I was curious - consider my curiosity sated. You're basically saying your non-use is a function of Boost's build system in general and not the quality of the Boost.Regex code in particular.Python for system administrators (developerWorks)
Python for system administrators (developerWorks)
> things to do, or a very different brain.
$hash = { owner => 'IkeTo', date => '2007-09-07', time => '10:00 GMT',
other => '3' };
$string = ...; # load the string
$string =~ s{ \(
(
\w+
)
\) }
{ defined($hash->{$1}) ? $hash->{$1} : "($1)" }gex;
print $string, "\n";
> in fileinput.input():'.)
In python, with a slight markup change:Python for system administrators (developerWorks)
Ooops, i forgot to quote dic's keys, the right is:Python for system administrators (developerWorks)
Your first solution is uninteresting. My input is not tailored to Perl, your input is specifically tailored to Python. There are a lot of further customization that can be done in the original Perl code, there is essentially none that can still be done in the Python code.Python for system administrators (developerWorks)