A setuid program loading unprivileged modules usually isn't a problem if that program drops its privs before doing so.
Usually.
This whole mess goes to show that security is a hard-to-solve problem.
Taking privs off pulseaudio (Linux has been capable of doing better than all-out-setuid for _how_ long, exactly ?!?) may or may not be the solution in this case, I haven't checked.