Well, yes. Although you could scan the channels in the right order, i.e. the ones that you'd expect the same network (SSID) on first. Say your corporate installation -- it's probably only going to use a handful of channels (1,6,11,36,149 or so), so if you know from "experience" that (almost) all of the APs for the network are there, you can scan those first.
But ultimately you're right, in the general case you have to scan all channels and that simply takes a while. It shouldn't be more than a few seconds since you're not connected; here it takes ~3.5 seconds to scan all the channels my card supports, but I know that it can be (much) slower depending on the device.