GEM is the kernel-level interface. And in fact, other drivers _also_ use it. Except Intel implements GEM directly and other drivers use TTM (Translation Tables Mapping) instead.
There's a similar story with Mesa - there's a 'classic mesa' (drivers directly implement OpenGL spec) and Gallium3D (common infrastructure is used for OpenGL and drivers simply translate the resulting TGSI stream). And Intel is the only major remaining 'classic' Mesa user.