Howto fix the problem:

The LUKS password is relatively simple (plain english phrase), but it
should be enough to protect it if I drop it somewhere.

When you plug it into a Linux Gnome desktop automatically prompts you for
the password and opens up the folder for you. So it is very convenient.

I can't recommend this approach to normal folks because it only works if
you plug it in rarely. If you leave it plugged in all the time then it is
no better then having it in a folder.

But what is better (in terms of security) would be to print out keys into
ascii armor format or write down passwords to a hard copy. That way they
are impossible to hack! A person would have to physically break into my
house and search through my drawers and filing tower to find it.