Creating Kubernetes distributions

Posted Dec 5, 2019 19:14 UTC (Thu) by marcH (subscriber, #57642)
Parent article: Creating Kubernetes distributions

> Among the Kubernetes release scripts that date back to 2016 is anago, which is an 1,800-line bash script for releasing Kubernetes. Anago imports three separate libraries, each with another 500 lines of shell code. "It's time to not do that anymore," Augustus said.

Was any alternative suggested? Didn't find any in the Powerpoint.

A few thousands lines of shell script is too high but not crazy high IMHO.

Unix shell scripting shows its age but I haven't seen anything coming close for smaller programs (say a few hundred lines) interacting with files and gluing other programs together. Nothing as concise, dynamic and high-level. For instance using functions as parameters is trivial - not too bad!

One major drawback is incompatibility with Windows but hey, who implements release management and QA on that? ;-) Most Windows people I run into look like they haven't even heard of PowerShell yet. Click, click, click... or straight to WSL.

Python's subprocess module has come a long way but still requires boilerplate and relatively complex error handling code, people seem to get that wrong every time (error handling code being of course never tested).
Maybe Perl would have been a good candidate if it hadn't committed suicide by optimizing itself for "write-only" usage?

Is there anything else?

Creating Kubernetes distributions

Posted Dec 6, 2019 23:27 UTC (Fri) by IanKelling (subscriber, #89418) [Link]

I've been on the lookout for alternatives for years now and I don't see any. With shellcheck and proper error handling, bash can go pretty far.

Creating Kubernetes distributions

Posted Dec 7, 2019 18:48 UTC (Sat) by epa (subscriber, #39769) [Link] (5 responses)

Python's subprocess module has come a long way but still requires boilerplate and relatively complex error handling code

Can you give an example of how to do error checking and handling correctly in a shell script? It seems to require at least as much boilerplate as Python or Perl if you want to write | pipelines or do control flow while at the same time checking the exit status of each subprocess and perhaps checking whether anything was written on standard error too.

Creating Kubernetes distributions

Posted Dec 7, 2019 21:50 UTC (Sat) by Jandar (subscriber, #85683) [Link]

> It seems to require at least as much boilerplate as Python or Perl if you want to write | pipelines or do control flow while at the same time checking the exit status of each subprocess and perhaps checking whether anything was written on standard error too.

If you use #!/bin/bash "set -o pipefail" gives you checking of exit status of every part of a pipe.

Python as a shell replacement

Posted Dec 7, 2019 23:22 UTC (Sat) by marcH (subscriber, #57642) [Link] (3 responses)

Right, error handling is not easy in shell either. There are a number of things you can do but nothing is bullet proof, agreed. As a start, every single script of mine starts with set -e. I also use a lot of: "do_the_work_func || die ...", "while do_work; do..." etc. (pro tip: you generally don't need ret=$?)

BTW C has similar error handling behaviors, most likely not a coincidence.

I repeat: any shell program longer than a few thousands lines or with some serious data structures is probably a mistake. This being out of the way, let me try to rephrase and clarify what I meant earlier:

1. Python-as-a-shell adds extra code and significant overhead; you can't start prototyping by just throwing your .bash_history into a file anymore.
2. Python has a built-in and pretty good exception system that you generally don't even have to think about.

So why did the migration overhead I paid in 1. didn't magically give me 2. for free? Why do I have to think so much about error handling when I use the subprocess module? In _short_ shell scripts good error handling is the only thing I was missing! So where did my migration money go?

The Python people are very smart, so I guess there must be good technical reasons for that, yet these excuses still don't make Python a desirable replacement for short shell scripts (unless you absolutely need to support Windows). Actually, I'm worried these justifications may not be Python specific and may preclude _any_ general purpose language as a shell replacement...

Insightful interview with Steve Bourne: https://www.arnnet.com.au/article/279011/a-z_programming_...

Python as a shell replacement

Posted Dec 13, 2019 17:23 UTC (Fri) by BenHutchings (subscriber, #37955) [Link] (2 responses)

I also use "set -e" by habit, but it doesn't do exactly what you probably want. When you check the result of a command, that completely suppresses its effect inside the command. For example:

set -e
f() {
    false
    echo "continued"
}
f || echo "failed"

prints:

continued

Python as a shell replacement

Posted Dec 13, 2019 22:59 UTC (Fri) by marcH (subscriber, #57642) [Link] (1 responses)

Yes, set -e is absolutely not a silver bullet. It does not catch all errors, so you must lower your expectations. It does catch many of them, which has saved me and many others a lot of time, routinely, for years.

Some influent and vocal experts seem to have decided that, short of catching "all errors", catching "no error" is better than "many errors". I've read all their essays and I still couldn't make sense of their logic https://mywiki.wooledge.org/BashFAQ/105

"Works for us".

PS: besides 105 and a couple others, https://mywiki.wooledge.org/BashFAQ is the best.

Python as a shell replacement

Posted Dec 14, 2019 0:08 UTC (Sat) by karkhaz (subscriber, #99844) [Link]

I occasionally use a combined shell script/makefile if I care about catching errors on each command:

#!/bin/sh
# vim:set syntax=make:set ft=make:

MAKEFILE_START_LINE=$(\
  grep -nre makefile_starts_here "$0" \
  | tail -n 1 \
  | awk -F: '{print $1}')

TMP=$(mktemp)
tail -n+${MAKEFILE_START_LINE} "$0" > "${TMP}"

make -f "${TMP}"
SUCCESS=$?

rm -f "$TMP"
exit "$SUCCESS"

makefile_starts_here:
	command-1
	command-2
	command-3

This prints out everything below and including "makefile_starts_here" to a Makefile and then runs make on it, executing the commands one at a time. This is especially nice if I want built-in parallelism etc, it's actually even better than just using the shell (just ensure to print out "MAKEFLAGS=-j" at the top of the file).