I don't need a photo manager

I've got plenty of dumb image viewers and plenty of image/photo editors. I don't need simple editing because I can (and prefer to) use the GIMP. The tool I'm missing is something that gives me a good way to organize images.

Shotwell, and a lot of other photo organizers, seem to be oriented around the idea that you will take photographs and then do things to them, categorize them by who is in them, when they were taken, and so on. I don't need photo-like features; I don't care when, I don't care who, I don't care where. I simply need tagging at a massive scale. I have perhaps 20 images of photographs that I have taken. I have something like 30,000 images that are not photographs. When it comes to organization I simply don't need any for my photographs.

My images typically have multiple properties, such as what's depicted in them, or what I used to create them, or what project they're for. Some of these properties overlap. Today I use a directory tree and symlinks to attempt to cross-reference things correctly, but it's a nightmare to manage. Ideally I'd like to be able to add multiple tags to these images and then filter by tag, rather than attempt to put them in multiple directories and then list multiple directories and remove duplicates 'somehow'.

Adding tags needs to be easy. No more than one click (or keyboard shortcut) to invoke "add tag", auto-completion of existing tags, and automatic creation of "new" unknown tags when I hit Enter. The workflow should be: View image, CTRL+T (or whatever), type one or more tags, hit Enter. One or more tags have been added to the image and saved to disk.

Tags should be stored in the image. I don't care how, but I want those tags to stay with the image as I move it between computers. I ultimately want to be able to build my own command line filtering/sorting tools. My ideal image organizer would probably maintain a tag index for fast searching, but the tags themselves should be stored in the image. My images are in multiple formats (gif, jpeg, png and xcf) so this is quite a challenge.

I need speed and reliability. I've seen some image organizers crash when handed 10k images, or take days to process them. I've tried a lot of image managers and this is a point where most of them fail.

Am I the only one who isn't a photographer and still has a need to organize images?