polarssl: denial of service

A bug in the logic of the parsing of PEM encoded certificates in x509parse_crt() can result in an infinite loop, thus hogging processing power.

While parsing a Certificate message during the SSL/TLS handshake, PolarSSL extracts the presented certificates and sends them on to be parsed. As the RFC specifies that the certificates in the Certificate message are always X.509 certificates in DER format, bugs in the decoding of PEM certificates should normally not be triggerable via the SSL/TLS handshake.

Versions of PolarSSL prior to 1.1.7 in the 1.1 branch and prior to 1.2.8 in the 1.2 branch call the generic x509parse_crt() function for parsing during the handshake. x509parse_crt() is a generic functions that wraps parsing of both PEM-encoded and DER-formatted certificates. As a result it is possible to craft a Certificate message that includes a PEM encoded certificate in the Certificate message that triggers the infinite loop.

This bug and code path will only be present if PolarSSL is compiled with the POLARSSL_PEM_C option. This option is enabled by default.