It sounds as though the problem happens when writing back dirty pages to a complicated storage device such as network storage, RAID and so on. The simple case of an ext4 filesystem on local disk does not appear to cause a problem. Why not define a safe set of storage devices which are guaranteed not to need lots of stack to do writeback, and require all writes to other devices to be done synchronously? If that causes a performance problem, the device could have its own writeback layer which has a known maximum memory usage, and will not accept new pages for writeback unless it can guarantee it will have the memory to flush them later.
I know that writeback disk I/O makes a massive performance difference for your desktop PC with a spinning platter hard disk. But is that still the case for other types of device?