LWN: Comments on "Entering the mosh pit"

Entering the mosh pit

flussence — Mon, 22 May 2017 11:36:12 +0000

Last time I used urxvt, any codepoint above 0xFFFF displayed as a missing character box. It's one of several bugs that the author acknowledged and refused to fix (interpreting 24-bit color escapes and outputting garbage instead of ignoring them is another); for a terminal with “unicode” in its name that was enough for me to go elsewhere.

Entering the mosh pit

mathstuf — Mon, 22 May 2017 03:36:09 +0000

I haven't noticed Unicode issues with urxvt (at least inside tmux). What have you seen?

Entering the mosh pit

rsidd — Mon, 22 May 2017 02:18:13 +0000

Other than the scroll thing, the annoying thing about mosh is if your client goes down (laptop reboot) you cannot reconnect to the server. Mosh is useful sometimes, eg if you're working from a laptop and frequently changing internet connections but your client machine will stay up for the not-too-long duration. But if I want a truly persistent session I use screen. (Sometimes mosh+screen.)

Entering the mosh pit

flussence — Mon, 22 May 2017 01:55:17 +0000

Bad Unicode support is one deal-breaker; another for me is that it doesn't recognise the Delete key (in $TERM=st — which I'm using because urxvt has similar unfixed Unicode problems :-/)

Entering the mosh pit

TRS-80 — Sat, 20 May 2017 15:41:08 +0000

I gave up on mosh once I realised it was eating eomji. The relevant bug report is long, but the problem is mostly to do with mosh having to know the width of every character, which is hard when you compare the speed of unicode to update latency of system libraries.

You may think this is a pretty ridiculous reason to stop using it, and I would agree with you, however when reading Twitter with oysttyer sadly emoji being silently removed has a significant impact on comprehension. Having to press up, enter, re-enter my password, then up and enter again (for screen -xU) is far less of a problem.

Entering the mosh pit

nix — Sat, 20 May 2017 11:41:41 +0000

Oh, excellent! Now all we have to do is get it merged :)

Entering the mosh pit

obonaventure — Sat, 20 May 2017 07:54:44 +0000

Apple has been using MPTCP for Siri on all iPhones, iPads and now MacOS devices without observing severe problems caused by middleboxes deployed on the Internet. MPTCP works well through those middleboxes.

Entering the mosh pit

niner — Sat, 20 May 2017 07:28:17 +0000

No idea if mosh does it, but since it transfers very little data and may be used over networks with large packet loss, it could even opportunistically retransmit. Just send each package twice or even more to increase the chance that at least one of them arrives.

Entering the mosh pit

falstaff — Thu, 18 May 2017 16:37:24 +0000

I love mosh for its roaming capability: I can put my laptop into sleep at work, open at home and seamlessly continue working on the server. I had mosh sessions running for multiple weeks, even across continents! First, I almost stopped using mosh due to the scrollback issue, but it forced me to give tmux a try. Now that I use tmux to work around the scrollback issue, I also started making use of its muxing capabilities, so I only have to open one connection to the server. tmux furthermore helps when (involuntarily) rebooting the laptop since mosh is not able to reconnect to an existing session: Just create a new mosh session, kill the old mosh server and attach tmux again. I am really happy with this setup and don't look back to my old multiple gnome-terminal tabs with individual ssh sessions setup.

Entering the mosh pit

flussence — Thu, 18 May 2017 15:30:05 +0000

ChaCha20 is a bit faster than AES (4-15cpb), which is one of the reasons it's the default in OpenSSH now.

(Not fast enough to make me give up using HPN-SSH, mind you...)

Entering the mosh pit

caliloo — Thu, 18 May 2017 14:49:03 +0000

I daily manage remote systems that are either 1 or 2 geostationnary satellite hops from where I sit. I commonly have latencies of 1-2 seconds and single digits % packet drop, doing any kind of work without mosh would be much harder. Keep up the good work :)

Entering the mosh pit

swmike — Thu, 18 May 2017 09:09:06 +0000

I love mosh. I have been using it for years, and I'm very happy with it.

The only thing lacking is proper IPv4/IPv6 dual stack support, so one can switch between address families seamlessly.

Entering the mosh pit

luto — Thu, 18 May 2017 07:08:48 +0000

Even without AES acceleration, you're not going to notice the latency -- AES is quite fast even in software, and mosh is meant to be used by humans.

Instant update of typed characters

epa — Thu, 18 May 2017 06:04:46 +0000

But the password prompt drops into raw mode. I mean that the shell would switch to cooked mode for ordinary command input. If only cooked mode were a little bit smarter so it had provision for flipping back and forth between cooked and raw in the same line of input, as when Tab is pressed...

None of this would help much if you are running some full-screen text program inside the mosh session (and perhaps your command prompt inside that, as Emacs's shell-mode). I guess the heuristic used by mosh is the best we can do -- and the older line disciplines, while cleaner in principle, are just too inflexible. It's like remote graphical sessions where (with the demise of X11) we no longer send primitives to draw individual lines and rectangles, or render a device-independent character glyph, but instead send the individual pixels with a moderately intelligent compression layer. A good example of "worse is better".

Entering the mosh pit

Cyberax — Wed, 17 May 2017 23:02:03 +0000

> Using UDP and reinventing own flow control and retransmit is foolish.
Nope. Mosh is essentially a media application - it transmits the live video of the remote terminal.

So it's much better to transmit the whole screen state anew if an update packet is lost rather than wait for TCP retries to deliver all the intermediary packets.

For the same reason voice/video conferences use UDP - it's better to simply drop a small update to compensate for it later rather than cause the whole screen to stutter while the stream recovers.

>"Those who do not understand TCP are doomed to reinvent it"
Those who think that TCP is The Answer are foolish in the extreme.

Entering the mosh pit

renox — Wed, 17 May 2017 22:42:14 +0000

Your comment lacks substance, I could also say those who understand TCP try to replace it!
For example I hate having to type sleep 30 to prevent me from trying "too soon" to restart an application which then fail because the TCP connection fail.

Entering the mosh pit

itvirta — Wed, 17 May 2017 21:24:10 +0000

> Using UDP and reinventing own flow control and retransmit is foolish.

Perhaps, but it does have the advantage that it works without any kernel support.
Which helps a lot in getting your software to work across different operating systems.

In mosh's case, it also has the upside that it knows more about the underlying data
than a general purpose stream transmission algorithm does. Namely, changes to the
screen can be mercilessly discarded when the client isn't looking.

Entering the mosh pit

cgull — Wed, 17 May 2017 21:00:37 +0000

JuiceSSH is indeed fairly excellent, but there is an issue that causes screen corruption with a server running mosh 1.2.6 or 1.3.0. Fixed in Git master and the next release (coming soon), or you can fall back to mosh 1.2.5.

Entering the mosh pit

cgull — Wed, 17 May 2017 20:29:58 +0000

Mosh's crypto is pretty simple, it's just a textbook application of using OCB with a prearranged secret for sending messages back and forth (the secret is arranged over the initial SSH connection, so of course you must trust that).

There's nothing particularly special about OCB for interactive use, it has no particular advantage for that over any other AE mode.

Entering the mosh pit

cgull — Wed, 17 May 2017 20:07:42 +0000

You're worrying too much. Mosh's bandwidth needs are quite small, and even on a system without AES hardware, the crypto computation cost is pretty much in the noise compared to the virtual terminal emulation and state keeping. It's been a while since I benchmarked this, but I recall crypto being at most 2% of CPU usage in a profile.

Instant update of typed characters

cgull — Wed, 17 May 2017 20:00:05 +0000

And in particular, there are certain things that you do *not* want echoed even tentatively, such as the password you type at a password prompt. The two clever things about Mosh's predictive echo are that it won't echo until it sees some from the remote system, and that it cleans up errors in the echo once the real echo is received from the remote system.

Entering the mosh pit

cgull — Wed, 17 May 2017 19:53:00 +0000

Nice article, thanks! I think you have a pretty good handle on what Mosh is good for.

Mosh is a virtual terminal emulator that runs on both server and client. It transmits terminal output from server to client by sending updates from a previous state of the terminal screen to a newer one. It fundamentally maintains a rectangle of character cells for you to look at, rather than sending the complete character stream to update a terminal emulator on the client. So an application can update the screen quickly between two client updates, and character output that would go into scrollback on a normal terminal is simply never sent by Mosh, only enough data to construct an accurate display as of the second update.

So that's the weakness and strength of Mosh-- you don't get the full stream of data that you would with SSH, but the state synchronization protocol allows Mosh to skip intermediate transmitted state and blithely function on astoundingly bad networks. The predictive local echo is also important here. I've used Mosh on wi-fi networks with 50% packet loss and it's been noticeable but entirely usable. (Of course, when a network is this bad, it's a pretty short and steep slope from there to a network that drops all packets. Not even Mosh can fix that.)

Entering the mosh pit

shemminger — Wed, 17 May 2017 17:24:56 +0000

Using UDP and reinventing own flow control and retransmit is foolish.
"Those who do not understand TCP are doomed to reinvent it"

Entering the mosh pit

jengelh — Wed, 17 May 2017 16:16:57 +0000

>the client has to figure out when not to echo — when a screen-mode application is running, say, or when a password is being typed. Your editor did not have access to a network slow enough to exercise this feature;

Easy enough to fix that, without waiting for an airplane:

tc qdisc add dev lo root netem delay .3s

Entering the mosh pit

farnz — Wed, 17 May 2017 16:07:41 +0000

Yeah - AES acceleration instructions mean that your latency is dominated by network latency, not compute latency, by a significant margin - a realistic implementation of AES decrypt using the acceleration instructions is going to add under a microsecond of latency, which is less than the minimum Ethernet latency you're going to see on a real 10G network.

Entering the mosh pit

epa — Wed, 17 May 2017 15:59:34 +0000

Thanks. I am still living in the past, when I had to patch the openssh source code to add a 'no encryption' mode to get decent latency on my collection of ancient i386 machines...

Ten clock cycles (or even ten thousand) is plenty fast enough to display a keystroke instantly.

Entering the mosh pit

epa — Wed, 17 May 2017 15:55:59 +0000

Yes, the ssh connection is encrypted and has high latency. The one megabyte block of random data can be sent as a one-off on connection, and topped up in the 'background' when the terminal session is idle for a second or two. It doesn't matter that there is high latency for sending that data since it isn't needed immediately. This scheme would be inappropriate for a connection with lots of data going over it all the time, but it could work pretty well if what you are sending is tiny updates (individual keystrokes, or at most a screenful of text), which come at fairly infrequent intervals (human typing is slow), but they need to be processed with as little latency as possible when they do happen.

I suppose that if the UDP packets have a sequence number and a fixed length, you can still use the one-time pad to encrypt them (if a packet is lost, then that bit of the one-time pad is wasted too).

Others have pointed out how on modern CPUs AES is fast, so it may be a non-issue. (Although I would point out there is a difference between the average speed for decrypting a large block of data, and the speed if you are doing a single byte at a time. I doubt that decrypting just one byte on its own can be done in two clock cycles. But even if it takes a hundred thousand cycles that's still fast enough, on a modern CPU, to display the keystroke 'instantly' to a human user.)

Entering the mosh pit

MatejLach — Wed, 17 May 2017 15:32:02 +0000

Just tried it and unlike mosh, it does support scrolling. On the other hand, typing is much more laggy than with 'mosh --predict=experimental', so I'd say it's a supplement to mosh, rather than a complete replacement, at least for now.

Entering the mosh pit

itvirta — Wed, 17 May 2017 15:25:39 +0000

The possibility of packets getting lost is quite severe when the connection is done over UDP, _and_ when one of
the useful use-cases is that of roaming or sleeping laptop clients. Besides, what would you do when your megabyte runs out?
As for the latency of AES, that's what your SSH connections are encrypted with, too.

Entering the mosh pit

Felix.Braun — Wed, 17 May 2017 13:57:43 +0000

There is a fairly nice Android client for mosh (JuiceSSH) that makes remote administration even over EDGE connections quite bearable. I would not like to miss the possibility of accessing my server on the go with nothing more than my smart phone.

MPTCP

corbet — Wed, 17 May 2017 13:03:25 +0000

MPTCP is rather carefully designed to sneak through middleboxes; that's where a lot of the complexity in the protocol comes from. Some details are in my 2013 article on the subject.

Unfortunately, the work to upstream this code seems to have stalled completely.

Entering the mosh pit

farnz — Wed, 17 May 2017 12:53:11 +0000

Bear in mind that modern CPUs have dedicated AES assist instructions - this set of slides suggests that you end up using around 2 clock cycles per byte with an optimal implementation, and a latency on the order of 10 clock cycles from first byte in to first byte out. Given the clock speeds on modern CPUs, I suspect that you're going to hit network limits before AES is too slow.

Entering the mosh pit

smadu2 — Wed, 17 May 2017 12:47:17 +0000

I am eternally grateful for mosh, with out which I may not have had an opportunity to work remotely. Its not always possible/practical for a company to ship the infrastructure necessary to setup a dev environment to be able to work remotely. In such cases maintaining sane ssh sessions to your remote dev environment is God send. I have been working with mosh with latencies 300ms (on a good day) to ~1 sec with ~40% packet losses on worst days.

Entering the mosh pit

nix — Wed, 17 May 2017 12:42:14 +0000

Downside: the middleboxes presumably have to support MPTCP or at least not block it. The time before a random satellite Internet provider supports *that* is probably measured in the centuries :(

Entering the mosh pit

epa — Wed, 17 May 2017 12:41:05 +0000

I was thinking you could generate a megabyte of random data, send it across the ssh link, and use it as a one-time pad. That would cut latency further since you just need to XOR the incoming bytes rather than run AES decryption. As long as any packets don't get lost, I suppose. When the one-time pad is about to run out you can make a new one and again send it via the open ssh connection.

Or am I worrying too much about the latency imposed by AES?

Entering the mosh pit

pabs — Wed, 17 May 2017 12:30:25 +0000

Have the patches been submitted for mainline Linux?

Entering the mosh pit

1ace — Wed, 17 May 2017 10:59:01 +0000

> Your editor did not have access to a network slow enough to exercise this feature

You can use Linux's `tc` (traffic control) to simulate any network condition, including latency, bandwidth, packet loss, etc.
See https://wiki.archlinux.org/index.php/Advanced_traffic_con... and http://lartc.org/lartc.html#LARTC.QDISC ("9.2.2.2. Sample configuration" in particular) for instructions.

Entering the mosh pit

HenrikH — Wed, 17 May 2017 10:38:59 +0000

The nonce in OCB mode needs not be secret and can thus be a counter which makes it work well for situations like this. So you simply use the packet-number as the nonce and thus it's no problem if packets are missing (the encryption is not chained) or are received out of order.

Entering the mosh pit

flussence — Wed, 17 May 2017 10:15:31 +0000

It uses AES-OCB (https://tools.ietf.org/html/rfc7253) — this is the same encryption scheme Mumble uses for voice data, so I guess there's something to it that makes it good for interactive use.

Entering the mosh pit

obonaventure — Wed, 17 May 2017 09:37:13 +0000

Instead of changing the application to support mobility, another solution is to change the underlying transport protocol. When ssh is used over Multipath TCP, all the mobility features provided by mosh come for free thanks to the underlying Multipath TCP. All the features of ssh, including port forwarding, continue to work. If TCP reset attacks are a concern, Multipath TCP can cope with such attacks by recreating subflows and it would be possible to use ssh implementations to better interact with Multipath TCP. Patches are available for various versions of the Linux kernel from http://www.multipath-tcp.org