LWN: Comments on "Hacking the planet with Notcurses"

Hacking the planet with Notcurses

nix — Wed, 08 Apr 2020 22:44:16 +0000

> at least for this whole thread, you'll see many people complaining & comparing their performance figures not about some resource-intensive activities, but about emacs and like.

Emacs is an application manager as much as it is a text editor. Forking off performance-sensitive and/or memory-hungry subprocesses at the drop of a hat is downright common. (Even if you don't use magit. :P )

Hacking the planet with Notcurses

nix — Wed, 08 Apr 2020 22:42:50 +0000

I run multiple Emacsen: usually one for Gnus and one for work, under different uids. They use about 10GiB RAM each these days (Gnus is a bit of a pig and so is EDE/CEDET and I often have many hundreds of buffers open at once). My desktop doesn't have enough memory for that. They also like to talk to subprocesses which are better not "ssh BLAH" because the semantics of signal-handling across ssh sessions is screwed by the remoteness: you can't ctrl-Z one process inside an ssh you forked if you're that ssh's parent! Many of those subprocesses are things like compilations, and dammit I'd like to compile on the remote box with 20 cores and 128GiB ECCRAM and terabytes of SSD-cached storage, not the local box with 16GiB non-ECCRAM (half of which is eaten by Chrome/Firefox) and only two slow cores optimized for low power usage. (The local box has fast graphics, which is what matters for a desktop.)

The response latencies caused by doing this stuff over remote X are completely imperceptible. There's a reason I have 10GbE on the desktop! but the response times were completely ignorable even when it was only 100MbE. X is latency-sensitive above all else, and Ethernet is low latency.

Hacking the planet with Notcurses

nix — Wed, 08 Apr 2020 22:35:36 +0000

I don't think that's any more cheating, or any more "not SF", than, say, any of Greg Egan's novels which feature entirely fictional laws of physics.

I mean this means you'd have to define Greg Egan's _Dichronauts_ as fantasy, since it has multiple things that just cannot happen in the real universe and a physics which is so weird that Egan doesn't describe it at microscopic scales because everything goes completely to hell, or rather, to infinity down there. But it's definitely SF, even if -- and this is not a spoiler because it's on about the third page -- in that universe turning your body by more than 45 degrees is as impossible as exceeding the speed of light, and there is a pair of borders on the land surface which you cannot approach because it's where the surface of the sun intersects the surface of the world! But the story is *all about* exploring the possibilities of that deeply whacko alternate physics, and dammit that is not what fantasy does. The latter part of Anathem is similarly less about the social structures of the worldbuilding and more about exploring what would happen *if* platonism were true -- that it's not true of our world is neither here nor there, that's why they call it fiction.)

What makes SF SF is not physical plausibility. It's about the tropes and tactics the author expects the reader to employ when reading it, and by that standard, _Anathem_ is pure SF.

Hacking the planet with Notcurses

jwilk — Thu, 02 Apr 2020 21:21:25 +0000

Mouse reporting was added in 0.0.7: https://bugs.debian.org/923181
Beware there's apparently no way to opt out from this "feature".
I switched back to gpm, which just works.

Hacking the planet with Notcurses

mgedmin — Wed, 01 Apr 2020 12:08:26 +0000

AFAIU consolation only implements copy/paste, but doesn't provide any API for terminal applications, unlike gpm (or xterm).

Hacking the planet with Notcurses

jezuch — Mon, 30 Mar 2020 06:08:57 +0000

> a kind of ritualised university system, where a large group of people would learn and become experts in a specialised field and teach the next generation, over and over for thousands of years

Cool, so he's describing medieval Europe :)

Hacking the planet with Notcurses

NYKevin — Wed, 25 Mar 2020 21:53:59 +0000

As I recall, Anathem also had the "problem" that the outside world would, on a fairly regular but infrequent basis (multiple generations or longer), sack the university and force them to start over.

On a more boring note, Stephenson's approach to formalism and platonism is deeply alarming to me. Firstly, he portrays formalism with a very crude and oversimplified explanation that makes it sound like he's confused it with fictionalism (contrast large portions of Gödel, Escher, Bach); then he jumps straight to platonism without even acknowledging intermediate positions such as nominalism or intuitionism. But the really evil thing that Stephenson does here is switching the genre halfway through the novel. We abruptly change from a low-tech sci-fi to pure fantasy, because platonism doesn't actually work in the real world and Stephenson wanted to use it anyway. If you view the book as an argument, rather than a story, this is basically cheating. And if you don't view the book as an argument, then in my opinion it does a poor job of dissuading the reader from that interpretation.

Hacking the planet with Notcurses

Seegras — Wed, 25 Mar 2020 11:51:03 +0000

> If anything, technological progress allows making backups easier.

Don't underestimate the political gatekeepers. They would have built a restrictions management into that cube of knowledge that makes this impossible.

You only have to look at phones, tablets, kindle etc. to see that "copying" and "backups" have enemies.

Hacking the planet with Notcurses

Seegras — Wed, 25 Mar 2020 11:41:17 +0000

I'm heavily using Midnight Commander, on urxvt. Yes, it's not possible to change the bottom bar depending on modifier, but everything else, mouse, resize +-key and other numpad-keys and so on work nicely. it's using the SLang-library which is responsible for most of it.

Hacking the planet with Notcurses

rodgerd — Mon, 23 Mar 2020 20:32:48 +0000

> A recent fire in the Russian Library ( https://www.theguardian.com/world/2015/jan/31/fire-russia... ) destroyed many books and documents that existed only as a single copy.

And that's just accidental destruction. During World War I, when German invaded Belgium, they were so incensed by Belgian resistance that they destroyed - amongst other things - a library that contained what was then around a third of the world's extant medieval manuscripts and books.

After the war, other collections donated some of their works to rebuild the collection. During World War II, Germany again violated Belgian neutrality, and decided to punish the Belgians for objecting by obliterating the same library.

(Source: Barbara Tuchman's 1914)

urwid for Python

ber — Mon, 23 Mar 2020 19:56:35 +0000

In Python urwid seems to be a nice choice for terminal based applications.

Hacking the planet with Notcurses

Cyberax — Mon, 23 Mar 2020 18:56:56 +0000

I have backups from my phones since 2010 to about 2014. At that time I stopped caring about losing my phone, since all the data was in the cloud anyway and I don't particularly care about losing a handful of average-quality photos. Meanwhile, my important data (financial docs, personal writings, etc) is now backed up on multiple NASes.

Moreover, it's not like books are somehow immune to destruction. A recent fire in the Russian Library ( https://www.theguardian.com/world/2015/jan/31/fire-russia... ) destroyed many books and documents that existed only as a single copy.

Digitalization would make this much less likely. And the amount of data to store most of the world's historical books is pretty much trivial by today's standards. Suppose that you want to store 130 million books (the estimated total amount), and each book is scanned in b&w without any OCR, taking around 10 megabytes. That's around 1.3PB in total - less than the capacity of an average enterprise NAS. OCRing texts would also bring that down by an order of magnitude, into the realms accessible to home users (20Tb hard drives are already here).

It's also unlikely that you'd want to store ALL of the literature, so subsets of it (like pre 20-th century English-language books) would be small enough to fit on a single SD card.

Hacking the planet with Notcurses

NYKevin — Mon, 23 Mar 2020 18:28:48 +0000

For the average Regular Person (TM) in a developed, western country,* smartphone backups work in one of two ways:

You log into your iCloud account on your new iDevice, and It Just Works.
You log into your Google account on your new Android device, and It Just Works.

Yes, I know, there are numerous issues with that strategy. But bear in mind, the average Regular Person (TM) does not care about most of those issues, and is entirely unaware of many of them. The It Just Works factor is much more important to these people, since users are generally terrible at taking backups manually.

* This is an important qualifier. Phones in non-developed and non-western countries are very different from phones in countries that are both developed and western. In some places, the same argument applies, possibly for different values of "iCloud" and "Google." In others, the idea of backing up an entire phone to "the cloud" is ludicrous because mobile data is that expensive and home internet is either nonexistent or unreliable.

Hacking the planet with Notcurses

ecree — Mon, 23 Mar 2020 18:28:44 +0000

> RDP, Terminal Services and, yes, even VNC are far less hassle than any protocol that HAS to be tunneled to be safe (certificate-checking, etc. is performed by SSH, but in RDP it's a part of the protocol).

Isn't this a question of Do One Thing Well?

Having "secure tunnel" and "network-transparent GUI" handled by two different processes is proper Unix design.

(Fwiw, my current remote-work setup goes vncviewer → ssh -L → x11vnc → Xvfb. It's *like* pipelines, but with sockets.)

Hacking the planet with Notcurses

NYKevin — Mon, 23 Mar 2020 18:20:18 +0000

There is one other mode in which it is useful: Letting (remote) TUI applications interact with the (local) system clipboard. See for example Vim. This works well even over the public internet, because latency is basically irrelevant. A yank command taking an extra few hundred milliseconds is barely perceptible to me, and my keystrokes are buffered anyway.

X11, xcb, and round trip

NYKevin — Mon, 23 Mar 2020 18:13:37 +0000

It's entirely conceivable that this problem is (to some extent) specific to Emacs.

TL;DR: It does double-buffering by asking X11 to double-buffer, instead of doing it itself. As a result, it talks to the X server whenever it wants to draw to the back buffer, which is frequently (and not necessarily related to GUI events, because Emacs was designed like a TUI application - it draws when it damn well feels like drawing). If the back buffer could somehow live on the remote side, it would probably be a lot more efficient.

Ironically, that brings us full circle. If Emacs had been designed like a GUI application from the start, it wouldn't now be losing the SSH game to TUI applications. (Or maybe it still would. Hard to say what a "GUI from the start" Emacs would even look like to begin with.)

X11, xcb, and round trip

pizza — Mon, 23 Mar 2020 11:34:09 +0000

I don't know about cake, but ecn is on. This side a DSL link running PPPoE, with the modem configured as a dumb bridge. The other side is municipal fiber rate-limited to a only 100Mbps.

However, when I ran those tests the link was nearly idle, so I don't expect bufferbloat to have been a factor. (When the link is saturated, especially the upstream side, it is _very_ noticeable)

emacs is was by far the most pathological application I tried, though bog-simple fixed-font xterm didn't fare much better.

Oddly enough, GIMP was the most responsive (and very nearly usable!) perhaps because it mostly just slings pixmaps everywhere instead of relying on X primitives.

Tk-like toolkit for Curses

eru — Mon, 23 Mar 2020 09:13:58 +0000

Some X11 applications work quite nicely over even a slower line, some others are totally unusable because they keep redrawing everything constantly. The latter kind can be used remotely with VNC, which transmits only the actually changed pieces of the display.

X11, xcb, and round trip

mtaht — Mon, 23 Mar 2020 04:40:33 +0000

um, er, I have found a lot of the reason for bad x11 performance over tcp to be bufferbloat. tcp fills the pipe, then overbuffers & collapses.. your 40mbit link running cake? got ecn on on your tcp?

Hacking the planet with Notcurses

Kamilion — Mon, 23 Mar 2020 01:35:44 +0000

Oh good, something that I can use other than bearlibterminal.

http://foo.wyrd.name/en:bearlibterminal

You (Nick Black, Author of NOTCURSES) might have benefitted from a simple to grasp presentation like this:

http://foo.wyrd.name/en:bearlibterminal:design

although I'm reading the draft PDF book with interest so far.

bearlibterminal's weak point is that it's based on top of opengl. NOTCURSES seems more suitable for Vulkan-like behavior.

Quite a while ago, I cobbled together a little roguelike game using bearlibterminal.
https://files.sllabs.com/files/storage/games/pyrl/shnow_c...

Perhaps I might look into trying to port it to NOTCURSES as a distraction from the enfolding world events.
But, back to reading the pdf book...

X11, xcb, and round trip

anton — Sun, 22 Mar 2020 17:35:02 +0000

Then it's good that I did not "hold it up as the end-all-be-all universal solution for everyone while dismissing the alternative approaches that work better in those situations."

What I fear is that claims like your "unusable" will make software maintainers care even less for remote X performance than they do now.

Ideally there would better developer support by having a version of the X libraries that simulates a configurable remote latency also for local testing, and provides some profiling to point out where the latency arises.

Or, for your alternative approaches, make them work automatically across ssh. Ideally, both.

Hacking the planet with Notcurses

evgeny — Sun, 22 Mar 2020 17:13:50 +0000

> you can avoid needing to install and configure the program on each client,

which is a one-time and (typically) trivial task. Commercial software with a per-client licensing is a separate story, of course. But, and also responding to your next comment,

> and for resource-intensive activities a beefier remote system can be better.

at least for this whole thread, you'll see many people complaining & comparing their performance figures not about some resource-intensive activities, but about emacs and like.

Hacking the planet with Notcurses

pizza — Sun, 22 Mar 2020 16:00:19 +0000

(I should point out that x2go is a full remote desktop solution, not unlike vnc or rdp, as opposed to "just-launch-applications-in-a-ssh-session" tunneled X11...)

At my last gig, we experimented with x2go (and others) but ultimately settled on VNC. I don't recall the selection criteria (this was going on 7 years ago) but the applications were heavyweight EDA tools running on RHEL6 with all clients (other than mine) running Windows.

X11, xcb, and round trip

pizza — Sun, 22 Mar 2020 15:41:26 +0000

I'm saying "tunneled X11 may work for you, but it does not work for me (and others in my very-common situation) so you can't hold it up as the end-all-be-all universal solution for everyone while dismissing the alternative approaches that work better in those situations."

*shrug*

Hacking the planet with Notcurses

pabs — Sun, 22 Mar 2020 14:41:08 +0000

A comment from one of the Debian x2go maintainers on IRC:

x2goserver is in Debian oldstable-bpo, stable and testing/unstable. As a multi-session client, people could try pyhoca-gui (it performs less as everything is done in Python)

X11, xcb, and round trip

oldtomas — Sun, 22 Mar 2020 14:10:59 +0000

To be honest, and just as a reality check -- I went over your older posts and their quintessence is "it doesn't work for me, so it must be broken".

When presented with counterexamples, you preferred to stay where you are instead of investigating the discrepancies.

So such an answer (although not intended by the OP, as far as I can see) would seem like par for the course.

Just sayin'

Hacking the planet with Notcurses

sorpigal — Sun, 22 Mar 2020 13:43:20 +0000

> Why not run the GUI app locally and provide instead remote access to the data (files)?

A few reasons: you can avoid needing to install and configure the program on each client, and for resource-intensive activities a beefier remote system can be better.

Hacking the planet with Notcurses

HelloWorld — Sun, 22 Mar 2020 13:37:38 +0000

There was also systemd-consoled. However it was ripped out of systemd when the original developer stopped working on it and nobody else stopped up.

Hacking the planet with Notcurses

evgeny — Sun, 22 Mar 2020 10:50:21 +0000

I really don't understand why people use X remotely if the response times etc are intolerable. Even within the same LAN it's annoying sometimes. Why not run the GUI app locally and provide instead remote access to the data (files)?

E.g., I mount (user-level, with sshfs) a remote directory I'm working on, and have the files appear as local ones to the locally running GUI apps. Then fusermount -u, and that's all. The only remote traffic takes place when I hit the "save" or "load" buttons, and since the file sizes are typically small, this is almost unnoticeable. Or at least predictable.

X11, xcb, and round trip

anton — Sat, 21 Mar 2020 18:18:44 +0000

It works for me across the internet (not just in a LAN), so it may be unusable for you, but not in general.

Using it in a hotel with lots of computer scientists competing for WLAN and Internet access borders on unusable for me, too, though; however, even there my experience is not a delay for every character I type, but heavy lags in unpredictable intervals (probably due to packet loss).

Hacking the planet with Notcurses

tpo — Sat, 21 Mar 2020 17:54:38 +0000

>> (naturally they didn't keep backups)
>
> That doesn't sound reasonable. If anything, technological progress allows making backups easier.

You are not being sarcastic right? Because I have not ever been able to backup a Smartphone (== progress) in its entirety.

X11, xcb, and round trip

pizza — Sat, 21 Mar 2020 17:32:33 +0000

....are you seriously saying "it works for me and my setup, so it obviously must be good enough for everyone else?"

I've also been using emacs via tunneled X11 off and on since the late 90s, and still do so over LANs on at least a weekly basis. It works quite well in that context.

But across the public internet, with the latencies that I am forced to endure, emacs via tunneled X11 is completely unusable -- It's not the startup time; rather it's the multi-second delays that occur when interacting with the UI, neither of which occurs when using it (or other GUI applications) over VNC/RDP.

X11, xcb, and round trip

anton — Sat, 21 Mar 2020 16:38:02 +0000

I use emacs running on a remote machine over X over ssh on a daily basis, and have done so since 1998, so it's obviously usable. I also use other X applications, in particular, xrn, but also firefox. Applications tend to become less usable remotely over time, particularly firefox, where somewhat recent versions tend to produce heavy lags even on a LAN.

Emacs tends to be quite responsive in my daily usage, except that mouseover highlights tend to cause hangs (interruptible with Ctrl-g), and I have not managed to get rid of all of them. Startup can take long these days, and first usage of a font also seems to produce lags. Anyway, let's use the startup time as a benchmark:

 7s xrn over ssh
 8s emacs 21.4 over ssh with intermediate proxy
10s emacs 24.2 over ssh with intermediate proxy
13s emacs 24.5 over ssh
13s emacs 25.2 over ssh with intermediate Proxy
10s emacs 26.1 over ssh with intermediate Proxy

It seems to me that the startup time depends much on which fonts are used.

Hacking the planet with Notcurses

excors — Sat, 21 Mar 2020 16:02:57 +0000

> I'm reminded of one particular sci-fi short story about the sum total of humanity's knowledge being stored in a single library, [...]

Anathem (by Neal Stephenson) featured a more plausible way of preserving humanity's knowledge: a kind of ritualised university system, where a large group of people would learn and become experts in a specialised field and teach the next generation, over and over for thousands of years while civilisations rose and fell around them. If humanity faced some catastrophe while civilisation was at a low point, this system would provide a pool of experts ready to find and interpret the archived knowledge needed to tackle it. An archive by itself is worthless; you need people who've spent a lifetime learning how to make use of it.

(In software terms, documentation and source code by itself is much less valuable than an active community that knows how to work with the software and is willing to teach others. The important part of open source is the people, not the code.)

Hacking the planet with Notcurses

ballombe — Sat, 21 Mar 2020 15:50:58 +0000

The advantage of remote X over vnc etc.is that it does not require the server to run a windows system at all.
It just need to have the X libraries installed.

Hacking the planet with Notcurses

dowdle — Sat, 21 Mar 2020 15:35:03 +0000

For those complaining about the speed of X11 when tunneled through ssh, for goodness sakes try x2go. x2go doesn't provide support for 3D stuff (which includes some desktop environments) but generally speaking it is really good over a LAN and pretty darn usable over a WAN. x2go is a fork of NoMachine's NX 3 protocol. It uses ssh so no need to worry about opening up additional firewall ports. One shortcoming though is that the client is only geared toward one connection so multiple connections require multiple clients and windows. It would be so nice if the client got updated for multiple connections and a tabbed GUI. One very nice feature provided by x2go is ssh proxy which is handled in the GUI and easy to use.

Fedora and EPEL package both the x2go client and server. Debian (and it's derivatives) package the client but not the server (last I checked). The x2go project does provided repos/PPAs for Debian/Ubuntu. I'm not sure why Debian doesn't package the server?!?

Hacking the planet with Notcurses

Tov — Sat, 21 Mar 2020 14:17:16 +0000

I realize you are just venting frustration - and can somewhat sympathize with that.

However, it doesn't really make sense, that you ramble against *both* layers of indirection *and* dedicated (proprietary) interfaces.

What is your proposed solution for saving humanity? :-)

Hacking the planet with Notcurses

Tov — Sat, 21 Mar 2020 14:02:51 +0000

Agree. Using X forwarding over wifi with intermittent drop-outs would be a particularly risky experience...

X forwarding had its place in history and is probably only still useful for old-timers using old applications. (I still "fondly" remember using a CAD program spending maybe ten minutes downloading fonts when started on a Sun X terminal on our university 10Mbit LAN. However, when running it worked fine.).

These days, I will rather use a stateless, bitmap oriented protocol.

Hacking the planet with Notcurses

dezgeg — Sat, 21 Mar 2020 12:05:30 +0000

ChromeOS has frecon, not sure how reusable it is in general distros...

Hacking the planet with Notcurses

whot — Sat, 21 Mar 2020 10:21:53 +0000

> is that there is absolutely nothing to configure or startup

I daresay this is the *only* reason people keep talking about X11 over the network. No-one actually cares that it's X11 that's sent over the network. Maybe the energy celebrating network-transparent X could better be invested in helping to develop a smarter replacement with the same zero configuration entry threshold.