Boiling the ocean. [LWN.net]

Boiling the ocean.

Posted May 18, 2011 18:02 UTC (Wed) by tshow (subscriber, #6411)
In reply to: See also by nye
Parent article: Announcing TermKit

That's the problem with all of these schemes; they're "boil the ocean" projects. You can't just replace the shell and the terminal, you also have to replace a good chunk of people's tools, and supply APIs (and hopefully patches) to get everyone else onto your system. Anything less than that, and people are going to stick with bash.

I've had a plan for a system kind of like this for a while; the ideas are similar, though I think the scheme I've got is substantially more radical. But again, success for my project implies displacing bash, the terminal emulator, context-aware quicklaunchers, the file browser and emacs/vi. Which is... um... ambitious enough that I've never pushed the idea past the formative stages.

I should really get back to it. The main problem is the surrounding framework; making everything that has a manpage play nice in it. That said, it would be worth it...

to post comments

Boiling the ocean.

Posted May 18, 2011 23:33 UTC (Wed) by elanthis (guest, #6227) [Link] (1 responses)

Consider this project more of a proof of concept.

Right now, for instance, it requires two processes to even work: a server and a viewer. That's pretty insane.

Really, all this system actually needs to work is a way to communicate some meta-data about the streams. You can then write a nice GUI shell (WebKit or not) that can use that meta-data, a nice TTY that uses it, or even just run plain ol' Bash and stock UNIX tools and let them ignore the extra meta-data.

You could do this with a relatively simple UNIX-y protocol. Set a couple environment variables and open a fourth bi-directional socket for meta-data communication. Something like SHELL_METADATA_PID which serves the purpose of indicating that the metadata socket exists and making sure that only the target process tries to read/write to it (and not sub-processes, which would have a different PID). Tools that support the new metadata and which see that the env var is set and the right PID can read RFC2822 style meta data from socket 3 and write output metadata to socket 3. The main shell is responsible for creating these sockets and sanitizing inputs/outputs between pipeline stages. If the shell is just Bash or something, the env var is not set (or even if it was set by the parent of Bash, it would never match any of the children, so they act as if it was unset) and so those new tools just assume their input is untyped data and don't bother writing out any extra metadata. Wrap up the basic code for that in a small library and then write a shell that can use the extra metadata for final output streams and there you go.

A more complex -- but more powerful -- protocol could also be devised that allows things like negotiation between stages. Maybe ls can output some nice structured format, but if the next process in the pipeline doesn't want to deal with that or doesn't know how, it should get the classical text output of ps. That requires that either ps knows what output is desired, or requires the shell to transparently translate the rich output into the classical output between the pipeline stages. Either of these approaches require some way to get the accepted inputs of subsequent pipeline stages, and the shell-based translation additionally means that an enchanced ps would only be able to work with classical tools while running in an enhanced shell (so I'd suggest making the output format selection ps's job). This is a tougher problem to solve, but it's still worth it.

Boiling the ocean.

Posted May 19, 2011 15:43 UTC (Thu) by tshow (subscriber, #6411) [Link]

The scheme I've cooked up actually works in "screen" functionality as well, (as in, detachable terminals), so it does require multiple processes to work properly. Part of the idea is to simply extend the terminal escape codes so that a legacy tool will just work, but tools written for the new system can request the additional functionality.

Some of the additional functionality is there anyways; since a terminal is essentially a discrete series of execution sessions, I visually separate them by placing (subtle) bubbles around them. Rather than "backgrounding" tasks, things just keep running in their bubble, and you can tear a bubble off the terminal window and put it elsewhere on the screen. You get that whether it's a legacy shell program or not.

The new functionality allows things like the ability to create a progress bar that will sit in the terminal window where it was created, and receive updates from the program. The updates are all through the terminal escape commands, so the program doesn't have to care where things wind up on the screen. It also means a single program can have an arbitrary number of widgets in the output stream.

It also allows tagging things, to allow for saying things like "this segment of the output is mouse-active, poke me with this ID if the mouse hits it".

There's a lot more to the overall idea; once you've an explicit logical split between the execution environments, you can start allowing them to persist. Give them names, and they become sources and sinks for data.

So, say you do "ls", and you get a list of 10000 files. In my scheme, that list winds up in a mouse-active named buffer that has a scrollbar, named (say) ls-5. You can then use the contents of the buffer as the source for another command:

> grep foo < $ls-5

Or you can append data to it.

> ls bar/baz >> $ls-5

So, the execution bubbles wind up being emacs-like buffers. Programs that understand the new terminal/shell can create their own buffers as well, and do split output.

Anyways, this is just scratching the surface. I'm sure every programmer has a boil-the-ocean crazy project, and this one is mine... :)