LWN: Comments on "Fallout from the Python certificate verification change"

Fallout from the Python certificate verification change

Cyberax — Fri, 11 Dec 2015 03:36:39 +0000

> If there is no cert verification the encryption is worthless because anyone can intercept it.
Not intercept, but actually do a full-scale MITM. And it's much harder to do it covertly inside a local network than you think.

Fallout from the Python certificate verification change

toyotabedzrock — Fri, 11 Dec 2015 03:12:46 +0000

If there is no cert verification the encryption is worthless because anyone can intercept it.

They should be creating a way to whitelist the self signed certificates they want to use. That is a valid way to authenticate it within a it shop.

Fallout from the Python certificate verification change

noxxi — Fri, 04 Dec 2015 21:31:38 +0000

> self-signed certificates which are pinned,...

I'm not against self-signed as long as they are pinned or otherwise explicitly marked trusted. I'm also not against private CA. I don't favor the current model of public CA's where you have to buy certificates. But is is the only one which actually scales because without it everybody would need to explicitly trust each of the https sites without having any kind of idea what this actually means.

Writing secure programs is not possible if you expect each programmer to know every possible attack vector. Just look at the web with all the Cross Site Scripting and CSRF attacks which all can be in theory fixed if the programmers would understand the main problem and also all the small browser incompabilities. Instead the design and the defaults should provide security and robustness by default. That's why it is important to enable certificate validation by default and also make validation just work by default using the system of public CA's. There will be cases were the defaults are not appropriate and need to be adapated, but it is still much better than to expect all developers to switch on the security themselves.

Fallout from the Python certificate verification change

raven667 — Fri, 04 Dec 2015 21:02:31 +0000

> TLS is used to make the connection secure

Encryption and authentication can be two separate issues, self-signed certificates which are pinned, which is the SSH key management model, are still useful, even completely unauthenticated encryption is useful to detect errors and prevent passive monitoring even if it doesn't prevent an active attack. For far too long the idea that Authorities should Certify identity and that anything else was invalid, insecure and should generate big scary errors has retarded the deployment of our available tools. If anything unencrypted should generate warnings, encrypted but not authenticated by an authority should be normal and encrypted plus authenticated should flag the higher level of confidence.

Fallout from the Python certificate verification change

noxxi — Fri, 04 Dec 2015 20:56:44 +0000

> Security is not binary and even unvalidated self-signed certificates are way better than cleartext.

I agree. But if there is the option to do it properly one should do it. I think it can be done better than just randomly generating certificates and completely switching off certificate validation. For example one could create a list of all generated certificates and then explicitly mark these as trusted.

Nobody would (hopefully) suggest that the browsers should disable all validation just because there are some self-signed certificates out there. Instead browsers offer a way to add exceptions just for these special cases. But in other cases it should be ok to switch all validation for the application (environment variable) or even for the full system (special file)?

> .. literal IP addresses can't be signed!

It is true that public CA's will no longer do it. But private CA's can still create such certificates and they are accepted by browsers and in the programming languages. And if you use an IP address instead of hostname you will probably use a private CA anyway.

> Forcing half-baked TLS validation enforcement crap was one of the most ass-headed decisions of Python developers.

In my opinion it was actually fairly well done. Compared to PHP (no usable default CA store on windows) or Java (own CA store with comparable few root CA) they actually managed the transition with most applications still working. Of course, if one was implicitly expecting that no validation was done instead of explicitly disabling validation then the program broke. But when just connecting to a public site like the browser does it mostly worked because they also added support for SNI at the same time.

I think the main problems were actually caused by programmers which did not understand the concept of validation and when it was needed and when not and why it should be explicitly disabled when not needed. There is not really a good way to get secure defaults if the users of the software don't understand the concepts of security. And without the secure defaults we have know we had much bigger problems, because certificates were not checked even though most users implicitly expected it.

Unfortunately security has the big problem that it costs and is in a way without showing any obvious benefit. It is the same as with backups - you only realize how important they were when your data are lost. But then it is too late.

Fallout from the Python certificate verification change

jwilk — Fri, 04 Dec 2015 19:53:11 +0000

https://docs.python.org/2/library/warnings.html

Fallout from the Python certificate verification change

Cyberax — Fri, 04 Dec 2015 18:27:13 +0000

> I don't really accept this as a valid use case.
Security is not binary and even unvalidated self-signed certificates are way better than cleartext. You can't just sniff TLS using a purely passive listening device (an infected phone on a corporate WiFi) and an active MITM attack requires more resources to mount and is infinitely more conspicuous.

And of course, in some cases you can not even generate a real certificate - literal IP addresses can't be signed!

Forcing half-baked TLS validation enforcement crap was one of the most ass-headed decisions of Python developers. I had stopped using or advocating Python for anything more than a 10-line script after that.

Fallout from the Python certificate verification change

noxxi — Fri, 04 Dec 2015 18:04:32 +0000

> Randomly generated self-signed certificates are one case

I don't really accept this as a valid use case. TLS is used to make the connection secure and not just to let it look secure. This requires validation of the certificate, because otherwise a man in the middle attack or similar is possible. Blindly accepting random generated certificates is for me the wrong way to deal with the problem caused by automatical installations - instead it should be that the installation should setup the certificates properly, i.e. create a CA, add it to the trust store, issue various certificates from the CA etc.

And even though it is only used for testing - I would not recommend to ship anything with TLS enabled where the TLS part is not tested properly.

> dealing with expired certificates ... either turn off all certificates, or turn them on. Having a more limited exception mechanism would be helpful.

At least with Perl you can simple use SSL_fingerprint to accept specific certificates. This works similar to the exceptions you can make inside the browser. The underlying idea is not hard and it should be easy to add something like this to python and other languages too. This way you can trust specific certificates, no matter if they are expired, revoked, self-signed or for the wrong hostname. It is much more safe than to simply switch off verification.

I agree with you that most environments make it too hard to make *secure* adjustments to the validation. But the current way to solve this problem by making it easy to switch validation off is wrong. Instead it should be made easy to adjust validation in a secure way.

Fallout from the Python certificate verification change

jhhaller — Fri, 04 Dec 2015 15:58:45 +0000

The essence of the problem here is that there are certain common use cases which cause problem when certificate checking is enabled. Randomly generated self-signed certificates are one case, dealing with expired certificates where one doesn't easily control renewing said certificate is another. Unfortunately, the tools are our disposal to deal with this are binary, either turn off all certificates, or turn them on. Having a more limited exception mechanism would be helpful. One example of limited exceptions is how browsers allow support of deprecated SSL authentication, by configuring the browser with site or domain exceptions. A more limited exception mechanism which only allows certain IP addresses or domains to be exempted from certificate checking would allow minimizing the damage of globally disabling certificate settings to a subset of addresses. This would allow configuring only access to test systems to bypass certificate validation, without worrying that the setting would leak into production.

For test systems, it may be better to update the procedures to allow pulling the upstream signed certificate into the local test environment. This has the possibility of that certificate and it's signing certificate to leak into production, so must still be carefully handled, so as to not repeat Dell's experience. You are generating both a certificate authority certificate and host certificates, all signed with the signing certificate, right?

Of course, Stack Exchange will still be full of descriptions on how to globally disable certificate checking, but hopefully someone will give a better answer.

Fallout from the Python certificate verification change

epa — Fri, 04 Dec 2015 15:50:41 +0000

Does Python have a cultural equivalent to Perl's "warnings"? I've always felt this is something that should be handled better, with standard error from programs going into some kind of log which the system administrator can then review. Currently, for desktop applications, the stderr is usually ignored, and the lack of attention usually causes it to fill up with junk messages if it contains anything at all.

Fallout from the Python certificate verification change

kleptog — Fri, 04 Dec 2015 08:37:55 +0000

Just yesterday we ran into related problems with SSL validation. As a developer you're testing everything locally and with Docker you're essentially running an identical configuration to production, except of course you don't have valid certificates for all the internal components, they all have randomly generated self-signed certificates for testing. So every where you end up making flags on all the components to disable the certificate checking to you can test everything.

An environment variable is a very easy way to handle this, but you have to be very very careful to not have it sneak into a production build. What you need is a global flag that says "we're in production, no shortcuts!" but with all the virtualisation these days machines/images have no idea where they're running any more...

I'm open to better suggestions though.

Fallout from the Python certificate verification change

noxxi — Thu, 03 Dec 2015 03:48:19 +0000

I don't think it is a good idea to have a simple environment variable to switch off validation, especially if it has an innocent name like PYTHONHTTPSVERIFY. It should at least be something scary like "I_FULLY_UNDERSTAND_THAT_DISABLING_VALIDATION_IS_BAD" so users will think twice before using it.

It is not that I have a really good answer what should be done, but I can share what worked within Perl and what not: In Perl IO::Socket::SSL (where I'm the maintainer) has traditionally also defaulted to no validation. In 11/2012 a fat warning was added if the default of no validation was used (i.e. no explicit enabling or disabling) and in 07/2013 the default was changed to enable validation all the time. It was also added to use the usual CA path on UNIX and in 03/2014 some way to have a usable CA path on Windows too was added.

Unfortunately the result was that several code was upated to explicitly disable validation since the warnings were just annoying. And when looking at stackoverflow this is often one of the first things people suggest when dealing with SSL problems, i.e. there are lots of NullHostnameVerifier for Java and similar suggestions.

Thus I consider making it easy for users to disable validation a very bad thing, because they will just do it to get rid of problems and then forget about the impact, if they ever understood it. And even worse than the proposed environment variable is the idea of having a global file to switch validation off for all python code.

Incidentally IO::Socket::SSL has an official hook to replace any SSL settings done by the calling code. Thus by simply loading a specific module using this hook all validation could be switched off. But it is also used in practice to enable validation again for code which wrongly disabled it because the author did not fully understand the implications.