How about a scheme where you would include the source IP into the crypto calculation and
derive a port knocking order which would then open the service port to the source IP?
AFAICS this would beat someone listening in (unless he can connect from the same source
address) and would make a distributed brute force attack more difficult (If you manage to
force the service open, you still have to connect within the time window from the successfull