There's no mode bit in the hardware. The 32-bit ISA is just a subset of the 64-bit. Code that restricts itself to the 32-bit ISA can access only the low 32 bits of the integer registers.
The conflict is between the two ABIs. An object file compiled to use the 32-bit ABI will not inter-operate with one for the 64-bit ABI because, for example, it passes 64-bit function arguments in two registers instead of one.