What I have under my desk at work (and I'm quite happy with it) is the Dell T3500 Precision Workstation, which supports up to 24GB of ECC or non-ECC memory. It's not a mini-ATX desktop, but it's definitely not a server, either.
I really like how quickly I can build kernels on this machine. :-)
I'll grant it's not "cheap" in absolute terms, but I've always believed that skimping on a craftsman's tools is false economy.