It's important to analyse this correctly: how much of the workload is
- first-time reads (at boot, or new data)
- cached reads
- in-principle-cached reads (if you had enough RAM)
In terms of speeding up a Linux desktop system, the key benefit is going to arise from lots of RAM (8GB or more) to make perfect use of the filesystem cache. This won't speed up boot, but after a few day's uptime, all the apps will be starting from memory. USB flash disks really won't get you much. After that, the next important thing is to use ext4 (with relatime) rather than ext3.
USB flash devices are great for low-cost, low-power, small size, silent systems. But they are still terrible for writes. Also, swap isn't something for "normal" use anymore: it's there so that when you run right out of RAM, your system doesn't OOMkill an app, but just gets very slow instead. So don't worry about it.