The dialog doesn't need to be system-modal. It just needs to ensure exclusive access to the keyboard as long as it has the focus so that nothing else can observe your password.
Note that you're still vulnerable to impersonation attacks, since any other program can pretend to be the system dialog and capture your password that way. A reserved, unblockable shortcut key helps, but doesn't entirely eliminate the issue. To block this sort of attack you need some form of secure channel (like a dedicated LED) with which to indicate that the system dialog has successfully reserved the keyboard focus.