Even in the ideal world where the other OSes were free software (maybe they're working on FreeBSD and OpenSolaris drivers, too) and the entire driver development were in the open with no worries about undocumented features, this situation is not handled particularly well. If they have a fix to the driver, they want it to go into the largely-equivalent code that exists on all OSes. Meanwhile Linux refactorings should be Linux-only.
How do you structure your repositories to do that? I don't think git will handle this situation well, today.