De Raadt: Important SSH patch coming soon [LWN.net]

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 15:56 UTC (Thu) by dougg (guest, #1894) [Link] (5 responses)

As an interim measure Theo de Raadt suggests adding the undocumented
"UseRoaming no" to ssh_config. I was hoping that would be to sshd_config
but when I tried with "openssh-server 1:6.0p1-4+deb7u2" (Debian 7)
I got:

sshd_config: line 30: Bad configuration option: UseRoaming

So is this really a client side interim fix?

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 15:59 UTC (Thu) by rsidd (subscriber, #2582) [Link] (3 responses)

>So is this really a client side interim fix?

Yes. Did you read the full message?

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 16:01 UTC (Thu) by rsidd (subscriber, #2582) [Link] (2 responses)

ps -- sorry for the snark, by "full message" I meant including corbet's update, but it may have been posted in the three minute window between your post and mine.

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 18:41 UTC (Thu) by bronson (subscriber, #4806) [Link] (1 responses)

I don't see Corbet's update...? just two two-line paragraphs.

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 19:06 UTC (Thu) by bronson (subscriber, #4806) [Link]

Ah, I see it, on the front page. But now the blurb on the front page looks nothing like this article. Strange.

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 16:25 UTC (Thu) by compudj (subscriber, #43335) [Link]

Add the config line to ssh_config, not sshd_config. ssh_config is the client configuration file, whereas sshd_config is the server configuration file.

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 17:13 UTC (Thu) by nye (subscriber, #51576) [Link] (9 responses)

Hmm, this undocumented feature that I've never heard of sounds great, in theory - I feel like I learn about more interesting feature via security advisories than any other means.

Anyway, it's weird that this option is supported and enabled by default in OpenSSH clients, but not supported at all in OpenSSH servers. Is it there because it's supported by other servers, or because it's planned for the future and they wanted to start getting supporting versions out there in advance, or what? Anyone know anything about SSH roaming?

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 17:20 UTC (Thu) by sytoka (guest, #38525) [Link]

Use mosh and byobu ;-)

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 18:03 UTC (Thu) by Lennie (subscriber, #49641) [Link] (7 responses)

All I know is:

The OpenSSH client did get an implementation, the OpenSSH server code wasn't made yet.

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 19:26 UTC (Thu) by bronson (subscriber, #4806) [Link] (6 responses)

From De Raadt's commit:

"Disable experimental client-side roaming support. Server side was
disabled/gutted for years already, but this aspect was surprisingly
forgotten. Thanks for report from Qualys"

Sounds like the server code was written but never shipped, or shipped and then backed out.

Shipping useless code on the client is an uncharacteristic slipup.

De Raadt: Important SSH patch coming soon

Posted Jan 16, 2016 10:13 UTC (Sat) by epa (subscriber, #39769) [Link] (5 responses)

I wonder if you can find dead code using the test suite - delete one or more lines of code at a time, at random, and of those which still compile, run them against the test suite. If some lines of code can be removed and tests still pass, then either that is dead code, or you have a gap in the test coverage.

De Raadt: Important SSH patch coming soon

Posted Jan 16, 2016 10:30 UTC (Sat) by karkhaz (subscriber, #99844) [Link] (1 responses)

> I wonder if you can find dead code using the test suite - delete one or more lines of code at a time, at random, and of those which still compile, run them against the test suite. If some lines of code can be removed and tests still pass, then either that is dead code, or you have a gap in the test coverage.

Even if this feature were working, I'm not sure how it could have been handled by a test case, thus I don't see that one could be written for it...looking at [1], it seems to be a combination of unit tests and black-box tests where the actual command-line programs are run to try out various aspects of the protocol.

To test roaming, one of these black box tests would presumably have to disconnect from the network and then reconnect to a different network...this arguably goes beyond the scope of what a test script should be doing :)

So given a complex protocol like SSH, I don't see how you could cover 100% of the codebase with black-box tests that exercise the _protocol_ (as opposed to unit tests for individual functions, which would be useless for detecting this problem anyway).

[1] https://github.com/openssh/openssh-portable/tree/master/r...

De Raadt: Important SSH patch coming soon

Posted Jan 18, 2016 17:06 UTC (Mon) by epa (subscriber, #39769) [Link]

Well, quite. There is no part of the test suite which exercises the roaming feature. By randomly removing chunks of code and rerunning the test suite, you would find out this fact. That would be an indication of a 'bad smell' coming from the roaming code, which might be dead code, or might need some minimal test coverage, or an explicit exclusion from the check. (FWIW, if you have code that is intended to run when you disconnect from the network and then reconnect, then yes, you do need test scripts which mock up such an environment to check the code works.)

De Raadt: Important SSH patch coming soon

Posted Jan 16, 2016 10:41 UTC (Sat) by JGR (subscriber, #93631) [Link] (1 responses)

You could probably delete some calls to free() or remove some of the diagnostics/debug output for example and still pass even a very rigorous test suite. That doesn't necessarily mean that they're dead code or that adding tests for them would be useful relative to the amount of time it would take.

De Raadt: Important SSH patch coming soon

Posted Jan 18, 2016 14:34 UTC (Mon) by epa (subscriber, #39769) [Link]

Your point is true in general. For the particular case of free() calls, you can usually run the test suite under some memory sanitizer (a debug version of the C library, or something more heavyweight like valgrind) which will report unfreed memory on exit, or a memory leak detector (such as the Boehm garbage collector used just to detect unreachable storage). But yes you are right that lots of exceptions would have to be added, not just comments and blank lines, but also things like free() or assertions or safety checks for 'this can never happen but check it anyway out of paranoia' conditions.

De Raadt: Important SSH patch coming soon

Posted Jan 17, 2016 16:40 UTC (Sun) by dezgeg (subscriber, #92243) [Link]

This technique is a subset of mutation testing - see http://pitest.org/ for an implementation for Java.

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 18:04 UTC (Thu) by Lennie (subscriber, #49641) [Link]

Important information.

This applies to these OpenSSH client versions: 5.4 up to 7.1

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 18:40 UTC (Thu) by bronson (subscriber, #4806) [Link] (2 responses)

Apparently roaming is an unfinished feature to allow resuming previous connections: http://www.gossamer-threads.com/lists/openssh/dev/49018?d... (not sure why anyone would want this when screen exists... work around XWindows maybe?)

Sounds like OpenBSD is having a Heartbleed moment.

De Raadt: Important SSH patch coming soon

Posted Jan 14, 2016 19:07 UTC (Thu) by bronson (subscriber, #4806) [Link]

Good info on this: http://undeadly.org/cgi?action=article&sid=2016011414...

De Raadt: Important SSH patch coming soon

Posted Jan 18, 2016 11:24 UTC (Mon) by nye (subscriber, #51576) [Link]

>not sure why anyone would want this when screen exists

Because SSH is not used exclusively (or perhaps even predominantly) for interactive terminal sessions.

De Raadt: Important SSH patch coming soon

Posted Jan 15, 2016 1:07 UTC (Fri) by alkadim (guest, #104623) [Link] (2 responses)

> https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-...

Both vulnerabilities prevented or mitigated by further non-security bugs. :V

De Raadt: Important SSH patch coming soon

Posted Jan 15, 2016 13:28 UTC (Fri) by welinder (guest, #4699) [Link] (1 responses)

> However, we eventually identified three reasons why, in our experiments,
> we were able to partially or completely retrieve the OpenSSH client's
> private keys through this information leak (depending on the client's
> version, compiler, operating system, heap layout, and private keys):

That does not read like any kind of mitigation to me.

De Raadt: Important SSH patch coming soon

Posted Jan 15, 2016 14:22 UTC (Fri) by alkadim (guest, #104623) [Link]

Well, go on and read "Mitigating Factors".

For example, they had to use openssh 6.4 for the POC to work (for comparison,
Debian stable is on 6.7).

Sure some versions and configurations are effectively exploitable and with work
others could probably be too. I was not trying to belittle the importance of
the vulnerability, just reflect at the sorry state of software engineering in
general when it comes to security: bugs saving us from bugs and such, and this
from the OpenBSD guys who exercise outstanding coding practices.

De Raadt: Important SSH patch coming soon

Posted Jan 16, 2016 10:01 UTC (Sat) by aggelos (subscriber, #41752) [Link] (17 responses)

I can't help but ask: are there any technical reasons for the canonical ssh implementation to be written in C these days?

Honest question, not necessarily implying that it needs to be rewritten.

De Raadt: Important SSH patch coming soon

Posted Jan 16, 2016 22:09 UTC (Sat) by khim (subscriber, #9252) [Link] (2 responses)

It's used on many [small] devices (like routers) thus hundred-megabytes runtime is no-go. Even modest one like Python would be too much. Also: it's often used on some strange CPUs where there are only C is supported and more modern languages (like rust) are not supported.

We could probably have two different "canonical implementations", but few guys care to support another one given the fact that OpenSSH exists...

De Raadt: Important SSH patch coming soon

Posted Jan 17, 2016 11:16 UTC (Sun) by aggelos (subscriber, #41752) [Link]

That's pretty encouraging, in that there are many, many, platforms where those considerations do not apply. Also, in that the memory restrictions would be less of an issue for a reimplementation in, say, rust.

I.e. the issues that you mention apply to most systems software written in C and an SSH implementation seems like an ideal case for using a language in which space and time memory safety violations are inexpressible...

On runtime no no's

Posted Jan 17, 2016 13:43 UTC (Sun) by roblucid (guest, #48964) [Link]

A runtime shared by much of the system processes would likely save not cost space, it can be designed to be shareable and runnable from RO storage. This goes back to the old UNIX idea of designing a language for your implementation, or early micro's where interpreters like Forth, UCSD Pascal, Mac Draw ROMs and better system scripts than shell like Perl4 accelerated common sysadmin processes.

Something like SSH, could dynamically link in performance critical parts to allow good throughput on encrypted file copies and such with the runtime getting out the way, but most code processing options/sockets/files would be easier to write correctly and be more compact with higher level common operations, shared between many servers and system processes which do similar things. Reducing executable load sizes, page faults & cache misses, would actually improve not reduce performance.

The problem is this kind of re-architecting the system and design, is difficult by small evolutionary steps; most ppl "KNOW"object code is faster as benchmarks operate with narrow focus and warm caches rather than a more hollistic system wide approach.

De Raadt: Important SSH patch coming soon

Posted Jan 18, 2016 17:24 UTC (Mon) by ballombe (subscriber, #9523) [Link] (10 responses)

You need a language which give you sufficient control of the memory so that data like passwords and keys are not leaked to the address space.

De Raadt: Important SSH patch coming soon

Posted Jan 19, 2016 16:48 UTC (Tue) by pkern (subscriber, #32883) [Link]

I guess given the Qualys piece that means no C, right? Compilers optimizing away even somewhat defined behavior like a memset to zero out sensitive buffers.

De Raadt: Important SSH patch coming soon

Posted Jan 21, 2016 13:03 UTC (Thu) by aggelos (subscriber, #41752) [Link] (8 responses)

That is assuming that the attacker can violate the memory safety of a program and trying to provide a (usually probabilistic) defense against the scenario where they're limited to arbitrary reads. So not really relevant here, I'd say.

De Raadt: Important SSH patch coming soon

Posted Jan 21, 2016 15:05 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (7 responses)

What you're really trying to protect against by keeping secrets in specific (typically mlock()'d) parts of the address space is the possibility that the secrets may be written out to a swap partition or swap file, or even just an unused portion of RAM, where an attacker may be able to uncover them later. For this to work the attacker needs either physical access or root, but the process itself does not need to be active. Storing secrets only in locked memory helps to ensure that the data doesn't hang around in the system long after the process is finished with it.

De Raadt: Important SSH patch coming soon

Posted Jan 21, 2016 19:07 UTC (Thu) by aggelos (subscriber, #41752) [Link] (6 responses)

I'm aware, the post I was referring to talks specifically about things being "leaked to the address space" (which is something that commenters, if not the article, seem to get hung up on on the newer post on the subject [https://lwn.net/Articles/672465]).

Leaks to swap are a different thing; my personal strategy so far is to disable swap in all systems where the drives might be sent out to be repaired, etc. But I agree that mlock'ing is desirable, of course.

De Raadt: Important SSH patch coming soon

Posted Jan 21, 2016 19:44 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (5 responses)

> post I was referring to talks specifically about things being "leaked to the address space" ... Leaks to swap are a different thing ...

Are they? I read the original "leaked to the address space" comment to mean that higher-level languages make it much more difficult to keep data confined to a specific area of memory; for example, a reserved area that had been mlock()'d to prevent it from being written to swap. However, I'll admit that I've never seen this "leaked to the address space" phrase used anywhere else, so it could mean something else entirely.

De Raadt: Important SSH patch coming soon

Posted Jan 23, 2016 12:37 UTC (Sat) by aggelos (subscriber, #41752) [Link] (4 responses)

Much more difficult? In all FFIs I've seen, it takes effort to expose the foreign data to the garbage collector (assuming the language you're using depends on one. E.g. Rust doesn't).

Ah, unless the argument is that one would read the key data into a managed byte buffer (equivalent to using fread(), I guess :)). In that case it takes a bit of care not to do that, yes. Is that really such a big deal?

De Raadt: Important SSH patch coming soon

Posted Jan 23, 2016 18:39 UTC (Sat) by nybble41 (subscriber, #55106) [Link] (3 responses)

> In all FFIs I've seen, it takes effort to expose the foreign data to the garbage collector...

I considered that, but unless you intend to only manipulate the data through FFI calls and never access it with native code, there is a risk that the data will be copied outside the locked memory region. This isn't just a matter of garbage collection, either; the native code may make copies for its own use at the compiler's discretion. Limiting the high-level code to treating the data as an opaque reference and accessing it only through FFI calls is a viable option, but it implies that all the security-relevant code has been written in a separate low-level language.

De Raadt: Important SSH patch coming soon

Posted Jan 24, 2016 1:59 UTC (Sun) by aggelos (subscriber, #41752) [Link] (2 responses)

All the security-relevant code? I don't see why. AFAIU, it's all the crypto-code that would be written in a non-GC'd language. This is an important distinction (exhibit A: TLS (example: ocaml-tls)).

I'm not sure what you mean by "the native code may make copies for its own use at the compiler's discretion", could you explain? Are you talking about object code or native as in "in the language that is calling the foreign code"?

De Raadt: Important SSH patch coming soon

Posted Jan 24, 2016 3:40 UTC (Sun) by nybble41 (subscriber, #55106) [Link] (1 responses)

I meant native as in "in the language that is calling the foreign code". Not native as in machine code.

By "security-relevant code" I meant the code which is relevant to the security of the specific data in question (private keys and sensitive cleartext) from the sort of attacks we were discussing. There are, of course, other aspects of security which can be managed by high-level code, and the use of safer languages for certain portions of the code, like parsers, offers security benefits in its own right.

De Raadt: Important SSH patch coming soon

Posted Jan 24, 2016 12:37 UTC (Sun) by aggelos (subscriber, #41752) [Link]

By "security-relevant code" I meant the code which is relevant to the security of the specific data in question (private keys and sensitive cleartext) from the sort of attacks we were discussing.

Can I draw attention back to the fact that the private data in question was not put in danger by the code directly manipulating it? It was made accessible by a memory safety vulnerability in unrelated code which invalidated any programmer intentions on which code will be accessing the data.

So, IMHO any code written in an unsafe language which is exposed to potentially malicious input is relevant to the security of the sensitive data (whether it resides in the same address space or not, depending on the specifics).

De Raadt: Important SSH patch coming soon

Posted Jan 20, 2016 10:46 UTC (Wed) by Sesse (subscriber, #53779) [Link] (2 responses)

At the very least, the crypto bits need to be in C—good luck mitigating timing and cache attacks in anything far from the metal. :-)

De Raadt: Important SSH patch coming soon

Posted Jan 20, 2016 12:13 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

It's complicated even in C (see the recent 'boringcc' thread).

I know that people are looking at LLVM to fix this. It turns out that it's not as impossible as thought of earlier.

De Raadt: Important SSH patch coming soon

Posted Jan 20, 2016 12:46 UTC (Wed) by Sesse (subscriber, #53779) [Link]

Yes, I didn't say it was easy in C :-) But at least it's not in the realm of “solid impossible”.

From:		Theo de Raadt <deraadt-AT-openbsd.org>
To:		misc-AT-openbsd.org, tech-AT-openbsd.org
Subject:		Important SSH patch coming soon
Date:		Thu, 14 Jan 2016 07:05:36 -0700
Message-ID:		<29041.1452780336@cvs.openbsd.org>
Archive‑link:		Article