botan: three vulnerabilities

CVE-2016-2194 - (denial of service) - The ressol function implements the Tonelli-Shanks algorithm for finding square roots could be sent into a nearly infinite loop due to a misplaced conditional check. This could occur if a composite modulus is provided, as this algorithm is only defined for primes. This function is exposed to attacker controlled input via the OS2ECP function during ECC point decompression.

CVE-2016-2195 - (arbitrary code execution) - The PointGFp constructor did not check that the affine coordinate arguments were less than the prime, but then in curve multiplication assumed that both arguments if multiplied would fit into an integer twice the size of the prime. The bigint_mul and bigint_sqr functions received the size of the output buffer, but only used it to dispatch to a faster algorithm in cases where there was sufficient output space to call an unrolled multiplication function. The result is a heap overflow accessible via ECC point decoding, which accepted untrusted inputs. This is likely exploitable for remote code execution. On systems which use the mlock pool allocator, it would allow an attacker to overwrite memory held in secure_vector objects. After this point the write will hit the guard page at the end of the mmap’ed region so it probably could not be used for code execution directly, but would allow overwriting adjacent key material.

CVE-2016-2196 - (arbitrary code execution) - The P-521 reduction function would overwrite zero to one word following the allocated block. This could potentially result in remote code execution or a crash.