What you were going to say would have been true. Writing GIMPLE doesn't cost much time. The problem is that after writing GIMPLE, the code is pushed through the entire compiler pipeline to write the rest of the assembler output too.
So the GIMPLE goes through the compiler pipeline twice: during compilation to an object file, and during link time optimizations. That is where the extra cost comes from.
We have our smartest people working on a solution for this... ;-)