Are you /sure/ you can skip it? Do you disable SCTP in your builds? If you don't, then it may be that the bug can be tripped by a remote attacker despite the fact that you "don't use" SCTP. And that's just the first example that comes to mind.
It would be relatively easy to carve out patches that only touch code in a specific area of the tree (for example, the SCTP implementation). But the recent e1000e bug illustrates beautifully that bugs aren't very good at respecting modularisation of design, since they're not designed but are instead unintentional.
If you do want to try this, as I said, it's fairly possible, Git makes it very practical for you to knock together a script that lists only those patches which touch your subset of the kernel, assuming you have the technical knowledge to understand how the kernel is laid out, and you have a complete list of the components you do or don't use, plus the time to examine architectural changes that might alter that list.
One thing that's going on is that there are many different constituencies who think that the kernel developers ought to make /their/ lives easier by doing just a minute or two's extra work. Since the kernel developers are a very precious resource I suggest that wherever possible you should look for ways to make things better without asking the developers to do more paperwork for you.