It's not the address space of the debugger, it's the process being debugged, right? So you wouldn't actually have any code running while the processor has a mapping in the TLB at address zero, because fraid is stopped and gdb is probing it indirectly. So you wouldn't care if any process which has a mapping at address zero has lousy performance on return from interrupt or system call.