If the problem is that the application might die without BTRFS_IOC_TRANS_END, can't we just come up with an alternate mechanism that doesn't rely on doing things in pairs?
For example, open a special file called "/btrfs-transaction".
As long as it's open by your process, subsequent operations from that process are part of a transaction. Those operations can still fail and return errors. But even if your app dies rudely, the file descriptor will get closed, and the transaction ends.