I have a nice little SheevaPlug based on the same Marvell CPU you mentioned. It's an ARM device with 512MB of RAM and 512MB of flash. That CPU and 512MB of RAM is more than enough for for building. But 512MB of flash is practically nothing if you also want to keep a build environment around.
I actually run it from a 4GB SD card. This gives a bit more space for a build system, but still, not as much as I'd like.
Being able to build packages on a stronger x86 system I have around is also useful.