The logic design of a piece of hardware is also math, and should not be patentable for the same reasons.
Now, if you were to design a novel hardware *component* for use in actual, physical circuits, that would be be an entirely different matter. In that case you've invented a novel way to use the physical properties of the universe toward a specific goal. If anything justifies a patent, that would. To get to this point you have to reduce your abstract ideas to practice, i.e. actually build something, or at least show that in principle the component could be constructed from real materials.
The problem with software patents in particular is that once you remove the part you didn't actually invent (the general-purpose computer), you're left with something which doesn't qualify for a patent (the software algorithm), which is pure math, which--for reasons which should be obvious--is an explicitly excluded subject.
If you were to invent a device specifically for the purpose of evaluating the algorithm you're trying to patent, you could probably get a patent on that specific device. However, that would be equivalent--putting aside the fact that the hard work was done long ago by someone else--to patenting a specific type of general-purpose computer running *specific code*. Just as you can accomplish the same goal (e.g. evaluating a given algorithm) in a *different way* without violating a normal, non-computer-related patent, even minor changes to the architecture of the computer or the low-level binary code should get around the computer+software patent. Real software patents, however, are far broader than their non-software equivalents, covering all possible ways to evaluate the patented algorithm on a general-purpose computer, not just a specific reduction of the algorithm to practice.
 However, nothing can justify artificial, aggression-based monopolies, so there is no real difference so far as natural law and just property rights are concerned.