When compiling a source file with -flto, GCC outputs *two* things; the traditional object file format, run all the way through the target assembler, and (in a single ELF section in the .o) the serialized representation of the GIMPLE tree that gave rise to it (and associated stuff). IIRC, both these outputs come from the same run (so parsing is only done once), but still this is more work and more data to write out than if we could *rely* on -flto being used at link time, because we could stop at the GIMPLE stage and not write out all the native code.
When linking with -flto, only the GIMPLE form is used and the native code in the .o files (and .a files if gold(1) is in use) is thrown away; when linking without it, only the non-GIMPLE form is used, and the GIMPLE in the .o files is thrown away.
(IIRC, of course. I haven't been paying enough attention to GCC development for the last year or so for this to be more authoritative than the ramblings of any passing madman. I should really have waited for jwakely to answer more authoritatively...)