What you should do then (if you can) is to treat the GPL2 code like a LGPL2 library - keep it in its own section of the source tree, and leave it at GPL2+.
If you make major changes you can then decide whether to move it into the main GPL3 tree or not.