Truly understanding programming

The interesting thing is at least some NN architectures are equivalent to Turing machines (AIUI). In principle a model on such an architecture could be 'trained' to implement any extant algorithm and even /discover/ new algorithms (provided you know what output you need, so you can train it).

Given that (if that claim about Turing equivalence is true, as I've read), then there is a way to take that algorithm and describe it - both in terms of a more "normal" computer language and human language.