If you're developing a code-base, yes, you don't need anything but the source code.
OTOH, if you're maintaining a code-base, adding patches temporarily (or not) while tracking an upstream, then it's more convenient to work with a system that allows you to easily distinguish between each change, relative to the upstream. E.g. a pristine tarball + patches, or a git tree.
I've worked on both developing an upstream code-base and maintaining a supported version of that same code-base, at a big distributor/vendor, and we used both methods, as appropriate. For the supported, released binaries - they were built from pristine+patches, and that's what got released as the source (just as RedHat used to).