I may be wrong, but I guess in your case much of the problem lies in the fact that it's a usb soud-card. Just don't do an apt-get while listening to music. Yes, that's obviously not a solution, but why would you pretend to have perfect audio with an USB soundcard while doing heavy IO? For example, if you've got to burn a dvd with an external usb dvd+rw, is it a good idea to start doing heavy IO while burning the dvd? I guess not. Why is this any different?