A shell in Rust could be really cool

Posted Feb 12, 2025 15:07 UTC (Wed) by koverstreet (✭ supporter ✭, #4296)
In reply to: A shell in Rust could be really cool by Cyberax
Parent article: Rewriting essential Linux packages in Rust

And yet, we keep using our interactive shells for writing general-purpose code...

A shell in Rust could be really cool

Posted Feb 12, 2025 15:12 UTC (Wed) by dskoll (subscriber, #1630) [Link] (19 responses)

Yes, because it's so (too?) convenient and guaranteed to exist on pretty much any UNIX-like system.

It's a tough habit to break, but I generally switch to a real scripting language (Perl is my favorite... don't judge) if a shell script gets longer than about 15 lines.

A shell in Rust could be really cool

Posted Feb 12, 2025 16:45 UTC (Wed) by wtarreau (subscriber, #51152) [Link] (8 responses)

On the opposite I wrote some complex programs (thousands of lines to manage network configurations) in shell for the sole reason that they had to be maintainable *and fixable* in the field by their own users. This is one often underestimated value of the shell. Every user knows a little bit of it, can easily test manually in their terminal, adapt the changes to the script and try again.

A long time ago when I begun with unix, my default shell was tcsh. I didn't understand why some of my scripts didn't work the same way on the command line. I thought this had to do with interactive vs non-interactive. When I realized that the shell was different, I switched to bash and lost a lot of stuff (it was 1.14 by then), but I started to write working scripts. When 2.05b arrived scripting abilities improved a lot! I once tried to switch to zsh because it was compatible with bourne, but admins hated it due to the way it displays completion and moves the screen up. And one difficulty was that you quickly get used to features that again don't work in your portable scripts. So I went back to bash, the (in)famous bug-again-shell that everyone loves to hate. But in the end, scripts written with it work basically everywhere. And if you need to get rid of it because you end up with a real bourne (like dash), that's not that dramatic, you spend a few hours getting rid of bashisms and that's done. Or the user installs bash or ksh, since their compatible subset is quite large.

That's why people continue to write (sometimes long) shell scripts: that's easy, portable and transferable to someone else to take over them. And they come with basically 0 dependencies (aside external programs) so that makes them future-proof contrary to many interpreted languages whose interpreter breaks compatibility, or whose libraries love to change API all the time by adding a new argument to a function...

A shell in Rust could be really cool

Posted Feb 13, 2025 9:09 UTC (Thu) by taladar (subscriber, #68407) [Link] (7 responses)

One thing that shouldn't be underestimated, however, is how easily shell scripts can break with changes in language and locale settings and it is a pain to debug.

A shell in Rust could be really cool

Posted Feb 13, 2025 12:50 UTC (Thu) by pizza (subscriber, #46) [Link] (6 responses)

> One thing that shouldn't be underestimated, however, is how easily shell scripts can break with changes in language and locale settings and it is a pain to debug.

This turned out to be the root cause of an infuriating intermittent bug I was trying to figure out -- nobody could give me a reproducible test case. Turns out they were using something other than English locales.

A shell in Rust could be really cool

Posted Feb 14, 2025 15:35 UTC (Fri) by taladar (subscriber, #68407) [Link] (5 responses)

Another "fun" bug to consider with shell (and similar text-based languages I guess) is what I mentally call the August/September bug that can happen when you using leading zeroes on stuff like months and aren't aware that you used it in a context (e.g. shell arithmetic) where leading zeroes signify octal.

A shell in Rust could be really cool

Posted Feb 14, 2025 23:51 UTC (Fri) by himi (subscriber, #340) [Link] (3 responses)

The constraints of shell arithmetic are a thing that has bitten me far too often (I've resorted to feeding stuff through `bc -l`, and occasionally even running calculations through python one-liners), but this is one I've never managed to run into - how exactly does this manifest?

A few simple experiments with bash suggest "08" used in an arithmetic expansion will error ('bash: 08: value too great for base (error token is "08")') - pretty much what I'd expect, though it'd take me by surprise the first time it happened (at least the error is nice and informative). Are you talking about something like iterating through 0-padded numeric months and hitting errors once you hit August?

A shell in Rust could be really cool

Posted Feb 15, 2025 0:16 UTC (Sat) by Wol (subscriber, #4433) [Link]

Sounds like dates in Excel. If you paste *text* into a cell, and then try to interpret it as a date, it barfs when fed a "wrong" date. I've just yesterday had to fix that problem ...

Copy a column of date-times from one spreadsheet to another. Format the date-time as a time. Save as a csv. If you're not in America the csv will contain a "random" mix of times and date-times. But all the date-times will not be a valid American date, for example today - 15/02/2025 - will be a date-time, while fast-forward a fortnight to 01/03/2025 and it will be correctly converted to a time.

The fix is easy - explicitly format/force the text into a European date, and everything then works, but it's a pain in the arse. Especially if every time you hit the problem, you've forgotten the previous time ...

Cheers,
Wol

A shell in Rust could be really cool

Posted Feb 17, 2025 13:42 UTC (Mon) by taladar (subscriber, #68407) [Link] (1 responses)

You get exactly that error but if you e.g. write a script in e.g. March and test it only with dates occurring while you write it (and maybe 01 and 12 for over/underflow edge cases) you can easily end up with a script that breaks with that error in August or September (month 08 and 09).

The solution is easy once you know about it, you can simply prefix every variable in the arithmetic expression with 10# to force base 10 but you need to know about.

A shell in Rust could be really cool

Posted Feb 18, 2025 21:46 UTC (Tue) by himi (subscriber, #340) [Link]

Ah, thanks for that explanation, it makes sense, and adds one more thing to the list of gotchas for me to keep in mind with shell code . . .

I try to use unix timestamps ($(date +%s)) whenever I'm dealing with times and dates of any sort in shell scripts, and then use $(date -d @...) to convert to something human readable - not particularly portable, but I'm fortunate enough to only really deal with Linux systems (aside from an occasional Mac, which has caused me serious headaches at times).

I do sometimes use sub-second precision (e.g. $(date +%s.%N)) for higher resolution timing, which then necessitates using bc -l or similar for any calculations. That recently bit me, in combination with scoping issues - I was mixing second and sub-second precision and the (perfectly logical) variable name 'now', and getting errors from arithmetic expansions when a sub-second precision value was still active where I thought it should be second precision . . . Fixing that just required fixing the scoping and naming issues ('local' is one of my favourite keywords), but it took *far* to long to spot the bug.

And to bring this back on topic, that's one monstrosity of a shell script that *absolutely* needs to be rewritten - though probably not in Rust, sadly, since I'm the only person in my team who'd have any hope of working on a Rust version . . .

A shell in Rust could be really cool

Posted Feb 17, 2025 21:12 UTC (Mon) by raven667 (subscriber, #5198) [Link]

...And 8am, I just fixed a bug in a cron job that was rotating some archives and would truncate the timespec where I want the leading zero (eg 0600-0859) for the filename `timenow=$( date +%H )`, just looked it up and I fixed it by referencing the previous hour as `$(( ${timenow#0} - 1 ))` which forced the evaluation to not be octal. good times.

A shell in Rust could be really cool

Posted Feb 13, 2025 2:21 UTC (Thu) by interalia (subscriber, #26615) [Link]

*judging intensifies*

A shell in Rust could be really cool

Posted Feb 13, 2025 6:01 UTC (Thu) by marcH (subscriber, #57642) [Link] (8 responses)

No shell script should be longer than some number of lines and that threshold is... "it depends" but it's always longer than 15 lines....

Error handling is a real pain in shell (just like in... C) but "set -e" and "trap EXIT" mitigate a bit.

The truth is: the shell ability to "glue" processes, files and pipes is still unmatched. Why? Simply because that's exactly why it was designed for. Everyone who has ever used Python's "subprocess" knows that. Also, "shellcheck" is incredibly good; saved me hours and hours of code reviews thanks to: "please run shellcheck, it will tell you what's wrong with this line". Last but not least: the shell is a decent... functional language! You can put a function in a variable which is amazing and very useful for a language that limited otherwise.

A shell in Rust could be really cool

Posted Feb 13, 2025 7:13 UTC (Thu) by himi (subscriber, #340) [Link] (7 responses)

> Also, "shellcheck" is incredibly good; saved me hours and hours of code reviews thanks to: "please run shellcheck, it will tell you what's wrong with this line".

I would like to second, third and fourth this (and then some) - shellcheck is amazing, it makes creating maintainable code in shell massively more practical.

The biggest issue I have with shellcheck is actually that it's /too/ good - it dramatically raises the threshold where my gut starts screaming at me that it's essential to rewrite this thing (some random hack grown into a monstrosity) in a "real" language, leaving me with constant cost/benefit anxiety. I've caught myself more times than I'd like to admit thinking "This really needs to be redone in Python . . . . but the incremental cost of adding X feature to the shell code is less than the cost of completely rewriting 1000+ lines of shell code from scratch, so . . . "

Without shellcheck making such a massive improvement in the quality, consistency and coherence of my shell code that would never have become a thing for me - it truly is both a blessing and a curse.

A shell in Rust could be really cool

Posted Feb 13, 2025 9:11 UTC (Thu) by taladar (subscriber, #68407) [Link] (6 responses)

I will never understand why people even consider Python when rewriting a shell script considering shell runs basically everywhere and any given Python version runs basically nowhere in terms of distros over time.

A shell in Rust could be really cool

Posted Feb 13, 2025 10:01 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (1 responses)

Because I don't need my quick and dirty script to run everywhere. I need it to run on the one system that is sitting right in front of me. That system has a reasonably recent Python, and Python is far less painful to program in than bash or other shells, so I will write scripts in Python.

A shell in Rust could be really cool

Posted Feb 13, 2025 12:48 UTC (Thu) by pizza (subscriber, #46) [Link]

> Because I don't need my quick and dirty script to run everywhere.

"It's only temporary if it doesn't work"

A shell in Rust could be really cool

Posted Feb 13, 2025 12:48 UTC (Thu) by pizza (subscriber, #46) [Link]

> I will never understand why people even consider Python when rewriting a shell script considering shell runs basically everywhere and any given Python version runs basically nowhere in terms of distros over time.

...And given how perl goes through great pains to be "bugward" compatible indefinitely.

(I have literally decades-old perl scripts that still JustWork(tm). During the python2->python3 migration debacle I decided to just rewrite a lot of p2 (and early p3) into perl instead, and it's not needed to be touched since then. Ran faster, too)

A shell in Rust could be really cool

Posted Feb 13, 2025 18:43 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

If you just stick to the built-in Python library (subprocess and the general IO), then you can get reasonable coverage across modern distros.

A shell in Rust could be really cool

Posted Feb 13, 2025 22:53 UTC (Thu) by himi (subscriber, #340) [Link] (1 responses)

Because my quick and dirty hack that grew into a thousand line monstrosity isn't portable? Either in the sense of the tooling and configuration it uses, or in applicability.

It may have been moderately portable when it really was a quick hack, but even then it was probably only used in exactly one location and probably wasn't very useful anywhere else. The growth pattern that turned it into a monstrosity certainly didn't increase portability - it made it into something with even more detailed ties to the environment it's used in.

Shell /is/ good for quick and dirty hacks that are moderately portable, far better than Python. But quick and dirty hacks tend to keep getting hacked on, and the point of the whole thread here is where you stop hacking on the shell code and rewrite it in something that's a bit more amenable to proper software engineering practises . . .

A shell in Rust could be really cool

Posted Feb 14, 2025 2:05 UTC (Fri) by marcH (subscriber, #57642) [Link]

> But quick and dirty hacks tend to keep getting hacked on, and the point of the whole thread here is where you stop hacking on the shell code and rewrite it in something that's a bit more amenable to proper software engineering practises . . .

Note this is not so specific to shell. How many times have we looked at some piece of code and thought "This has grown organically, it should really be rewritten". Of course the bar tends to be much higher when a language switch is required, notably because the _incremental_ change possibilities are much more limited. But at a very high level it's the same https://wiki.c2.com/?PlanToThrowOneAway logic.

A shell in Rust could be really cool

Posted Feb 12, 2025 15:25 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Who "we"? Shell overuse is really a phenomenon seen in low-level packages, where people for some reason don't want to reach out to perl/python/whatever.

A shell in Rust could be really cool

Posted Feb 12, 2025 16:10 UTC (Wed) by stijn (subscriber, #570) [Link]

In bioinformatics there are many pipelines where the units are processes/scripts/programs. They might be sequence aligners, statistics packages, clustering programs and a whole lot more. Typically these do not come as libraries attached to a language but as separate programs. These days they are usually orchestrated with tools such as Nextflow, Snakemake, Cromwell and others. There is still a fair bit of use for shell scripting when manipulating the outputs (which are usually big dataframes). It gets tiresome very quickly to write a python script with parameters that loads a large dataframe into memory when all you want to do is some stream data processing. The downside is the marshalling/unmarshalling of data. I wouldn't mind something like Apache Arrow formatted data that I could compose with unix pipes in a shell-like environment. It might exist, I haven't checked as there is more than just one obstacle.