Indeed. Some of the time, staged builds can be made cross-safe (flex and bison can usually run on the build machine), but not always. For instance, a program that extracts struct member offsets from data structures for use in assembly routines. Or unit tests - fragments of a program that are run during or right after the build to make sure that they work correctly.
Perhaps most of these obstacles can be overcome by interposing an emulator into the build system for the execution of target code, but even with a fully-functional emulator, such an undertaking requires substantial effort and expertise for every package. And running the entire build inside an emulator is likely to be no faster than using native hardware.
I'm not fond of autoconf and its associated tools, but mainly because of their cruftiness (shell scripts, m4 macros) and how they seem to be designed for solving yesterday's problems. A modern software configuration and build system that makes it easy to do cross-buildable packages would be useful.