squashfs-tools: two vulnerabilities

CVE-2015-4645: The first problem overflows the bytes variable, so that the allocation of fragments_bytes[] has an erroneous size.

int bytes = SQUASHFS_FRAGMENT_BYTES(sBlk.s.fragments);
...
fragment_table = malloc(bytes);

CVE-2015-4646: If we fix this by making the variable size_t, we run into an unrelated problem in which the stack VLA allocation of fragment_table_index[] can easily exceed RLIMIT_STACK.