in 32-bit mode, areas of memory can be marked with a page flag that says they contain 16-bit code. In 64-bit mode, no such luck. You need a full context switch to 32-bit mode to run 16-bit code. 26-bit code can run under a 64-bit OS, but only running a process native in 32-bit mode, since 32-bit protected mode supports 16-bit instructions.
Entering full-blown virtual 8086 mode is impossible, however.