not all patches that have security implications are known at the time they are created
so if you only install fixes that were tagged as security fixes, you will miss other fixes that have security implications because those implications were not known at the time they were written, and so they were not tagged.
it's not that fixing a security bug decreases security. what decreases security is the attitude that if it's not tagged as being a security fix, then it doesn't have security implications.
tagged as a security fix guarantees security implications
not tagged as a security fix does not guarantee that there are no security implications.
And if even you are making the mistake that tagging known security fixes means that other fixes don't need to be applied (on the basis that they don't have security implications), then you have just proven the case that many of the kernel developers are trying to make, that tagging some patches as security related will cause people to ignore the others and have less security than updating to a newer version with all of the fixes