So the name of the algorithm in passed as part of the file name, the content is written to the file descriptor. The hash is read from that same filedescriptor. The hash state is stored attached to the 'struct file'. See "Transaction based IO" in fs/libfs.c. It would need to be extended to work with writing a large file, but the concept is sound.
For encrypting.. using the same 'fd' for both read and write is problematic in a way that it isn't (so much) for the above. The original (Unix 6) pipe syscall returned only one fd which you could read from and write to. One problem with that was that it can be awkward to detect when the 'write' end has been closed (so the read end should get EOF), as there is no distinction between the two. If you happen to have two processes with the 'read' end open you never see EOF.
If we can either ignore that or work around it, then
is a promising file name to write to/ read from, except that there is a risk that the key would get stuck in the dcache and appear in /proc/$N/fd/$FD. I'm sure that is solvable though. The key would be HEX or BASE64 encoded of course.
The need to multiplex cyphertext and MAC is certainly a complication. I suspect there was a reason Herbert suggested 2 sockets rather than a simple multiplexing scheme. Without knowing that reason it is pointless trying to refine the design.
If it was to be done with sockets, it would seem to make much more sense to use 'socketpair(AF_ALG, SOCK_STREAM, ....)' rather than the sockets + accept model. Then you have distinct 'read' and 'write' ends. I would also use MSG_OOB to send the MAC beside the cyphertext rather than having two separate streams (not that I am a big fan of MSG_OOB, but it does seem to be a shoe that fits).