"It's the norm to have to rebuild binaries for different ARM sub-arches."
Maybe that is true of the microcontroller variants of ARM and of kernel code but userland code on linux tends to keep running fine on newer versions of arm. Debian armel will run happilly on everything from V4t right through to v7.