Creating Kubernetes distributions

Posted Dec 7, 2019 18:48 UTC (Sat) by epa (subscriber, #39769)
In reply to: Creating Kubernetes distributions by marcH
Parent article: Creating Kubernetes distributions

Python's subprocess module has come a long way but still requires boilerplate and relatively complex error handling code

Can you give an example of how to do error checking and handling correctly in a shell script? It seems to require at least as much boilerplate as Python or Perl if you want to write | pipelines or do control flow while at the same time checking the exit status of each subprocess and perhaps checking whether anything was written on standard error too.

Creating Kubernetes distributions

Posted Dec 7, 2019 21:50 UTC (Sat) by Jandar (subscriber, #85683) [Link]

> It seems to require at least as much boilerplate as Python or Perl if you want to write | pipelines or do control flow while at the same time checking the exit status of each subprocess and perhaps checking whether anything was written on standard error too.

If you use #!/bin/bash "set -o pipefail" gives you checking of exit status of every part of a pipe.

Python as a shell replacement

Posted Dec 7, 2019 23:22 UTC (Sat) by marcH (subscriber, #57642) [Link] (3 responses)

Right, error handling is not easy in shell either. There are a number of things you can do but nothing is bullet proof, agreed. As a start, every single script of mine starts with set -e. I also use a lot of: "do_the_work_func || die ...", "while do_work; do..." etc. (pro tip: you generally don't need ret=$?)

BTW C has similar error handling behaviors, most likely not a coincidence.

I repeat: any shell program longer than a few thousands lines or with some serious data structures is probably a mistake. This being out of the way, let me try to rephrase and clarify what I meant earlier:

1. Python-as-a-shell adds extra code and significant overhead; you can't start prototyping by just throwing your .bash_history into a file anymore.
2. Python has a built-in and pretty good exception system that you generally don't even have to think about.

So why did the migration overhead I paid in 1. didn't magically give me 2. for free? Why do I have to think so much about error handling when I use the subprocess module? In _short_ shell scripts good error handling is the only thing I was missing! So where did my migration money go?

The Python people are very smart, so I guess there must be good technical reasons for that, yet these excuses still don't make Python a desirable replacement for short shell scripts (unless you absolutely need to support Windows). Actually, I'm worried these justifications may not be Python specific and may preclude _any_ general purpose language as a shell replacement...

Insightful interview with Steve Bourne: https://www.arnnet.com.au/article/279011/a-z_programming_...

Python as a shell replacement

Posted Dec 13, 2019 17:23 UTC (Fri) by BenHutchings (subscriber, #37955) [Link] (2 responses)

I also use "set -e" by habit, but it doesn't do exactly what you probably want. When you check the result of a command, that completely suppresses its effect inside the command. For example:

set -e
f() {
    false
    echo "continued"
}
f || echo "failed"

prints:

continued

Python as a shell replacement

Posted Dec 13, 2019 22:59 UTC (Fri) by marcH (subscriber, #57642) [Link] (1 responses)

Yes, set -e is absolutely not a silver bullet. It does not catch all errors, so you must lower your expectations. It does catch many of them, which has saved me and many others a lot of time, routinely, for years.

Some influent and vocal experts seem to have decided that, short of catching "all errors", catching "no error" is better than "many errors". I've read all their essays and I still couldn't make sense of their logic https://mywiki.wooledge.org/BashFAQ/105

"Works for us".

PS: besides 105 and a couple others, https://mywiki.wooledge.org/BashFAQ is the best.

Python as a shell replacement

Posted Dec 14, 2019 0:08 UTC (Sat) by karkhaz (subscriber, #99844) [Link]

I occasionally use a combined shell script/makefile if I care about catching errors on each command:

#!/bin/sh
# vim:set syntax=make:set ft=make:

MAKEFILE_START_LINE=$(\
  grep -nre makefile_starts_here "$0" \
  | tail -n 1 \
  | awk -F: '{print $1}')

TMP=$(mktemp)
tail -n+${MAKEFILE_START_LINE} "$0" > "${TMP}"

make -f "${TMP}"
SUCCESS=$?

rm -f "$TMP"
exit "$SUCCESS"

makefile_starts_here:
	command-1
	command-2
	command-3

This prints out everything below and including "makefile_starts_here" to a Makefile and then runs make on it, executing the commands one at a time. This is especially nice if I want built-in parallelism etc, it's actually even better than just using the shell (just ensure to print out "MAKEFLAGS=-j" at the top of the file).