You may have to worry. When you give root access to an app you really want to trust it won't abuse it. While "good" root apps are meant to only grab root privileges for the minimum amount of work possible it's not enforced. Every app running with root privileges presents another potential attack vector onto your device which could be exploited.
Personally though I'm not overly paranoid, although I only tend to grant root access to apps install via F-Droid (and hence source code available FLOSS). Of course the availability of the source code doesn't guarantee anyone has actually audited it for badness.