"I want to add more about kernel in general, former Torvalds told 'move as much as possible out to userspace!'"
Linus has never taken that position. All other things being equal, he favors that things stay in user space. But if there is good argument for moving it into the kernel, he's always been OK with that. In fact, he has been very explicit on the point that the goal should not be to keep the kernel small, but to make the right decisions as to what goes in and what does not.
The GGI guys wanted to move the whole graphics driver nightmare into the kernel. Linus put his foot down on that idea. But DRM, memory management, and kernel mode-setting were the peices of Linux graphics which were always fundamentally broken under the old "all graphics in userspace" model. Do you know that switching from X to a text console and back has never been guaranteed not to hard lock your machine? Surely, if you've been around long enough, you've encountered such console crashes. And when that happened, it was likely not a bug, per se. It was the result of a fatally flawed design. Only with the inclusion of KMS is your distro's graphics system not fatally flawed at a fundamental level.
Regarding your concerns about the size of the kernel, what does it matter? The vast majority of kernel code expansion is in additional drivers. And those are all modules. Modules whose code-bases are orthogonal to each other.
In comparison, DRM, KMS, and GEM are not particularly large. I'd never want to go back to the bad old days of our former "Green Acres" graphics architecture.