Yes, you are overly naive...

well, the actual page table manipulation would not be that expensive, with some tradeoffs you can reduce it to changing a few top-level page table entries and a single TLB flush, which would be a few hundred cycles or so.

however there's more cost to this: TLB repopulation which would inevitably occur after returning to userland. that is the real expense as we're talking about up to hundreds of TLB entries on modern CPU cores, each potentially missing in the data cache and incurring hundreds of cycles.