Remove b_end_io from struct buffer_head

There are three benefits to this patchset.  First, it removes an
indirect function call from the completion path.  Instead of setting
bio->bi_end_io to end_bio_bh_io_sync() which then calls bh->b_end_io(),
we set bio->bi_end_io to the appropriate completion handler, replacing
two indirect function calls with one.

Second, there is a slight security advantage to this.  It is one fewer
function pointer in the middle of a writable data structure that can
be corrupted.  Third, it shrinks struct buffer_head from 104 bytes to 96
bytes, allowing for appropriximately 7% reduction in the amount of memory
used by buffer_heads (or, alternatively, allows 7% more buffer_heads to
be cached in the same amount of memory).

I've run ext4 through its paces, and everything seems OK.  I've only
compiled ocfs2/gfs2/nilfs/md-bitmap.  Hopefully the maintainers can give
this series a try.  I'm sending the entire series to linux-fsdevel
and cc'ing the fs-specific mailing lists for the fs-specific patches.

I have some unresolved thoughts while working on this:
 - Should gfs2_aspace_write_folio() be passing its wbc into __bh_submit()?
 - Should we allow callers of bh_submit() to pass in NULL instead of
   bh_end_read / bh_end_write and infer which one to use from the opf
   being REQ_OP_READ or REQ_OP_WRITE?
 - Should some of these callers of bh_submit() be calling higher level
   functions in buffer.c insteaad?  eg nilfs_mdt_submit_block() could
   call __bh_read().

Matthew Wilcox (Oracle) (34):
  buffer: Remove forward declaration of submit_bh_wbc()
  buffer: Add bh_submit()
  buffer: Remove mark_buffer_async_write_endio()
  buffer: Add bh_end_read(), bh_end_write() and bh_end_async_write()
  buffer: Convert write_dirty_buffer to bh_submit()
  buffer: Convert __bread_slow to bh_submit()
  buffer: Convert __sync_dirty_buffer to bh_submit()
  buffer: Convert __bh_read to bh_submit()
  buffer: Convert __bh_read_batch to bh_submit()
  buffer: Convert block_read_full_folio to bh_submit()
  buffer: Convert __block_write_full_folio to __bh_submit()
  ext4; Convert __ext4_read_bh() to bh_submit()
  ext4: Convert ext4_fc_submit_bh() to bh_submit()
  ext4: Convert write_mmp_block_thawed() to bh_submit()
  ext4: Convert ext4_commit_super() to bh_submit()
  jbd2: Convert journal commit to bh_submit()
  jbd2: Convert jbd2_write_superblock() to bh_submit()
  ocfs2: Convert ocfs2_write_block to bh_submit()
  ocfs2: Convert ocfs2_read_block to bh_submit()
  ocfs2: Convert ocfs2_read_blocks to bh_submit()
  ocfs2: Convert ocfs2_write_super_or_backup to bh_submit()
  gfs2: Convert gfs2_metapath_ra to bh_submit()
  gfs2: Convert gfs2_dir_readahead to bh_submit()
  gfs2: Remove use of b_end_io in gfs2_meta_read_endio()
  gfs2: Convert gfs2_aspace_write_folio to bh_submit()
  buffer: Remove mark_buffer_async_write()
  nilfs2: Convert nilfs_btnode_submit_block to bh_submit()
  nilfs2: Convert nilfs_gccache_submit_read_data to bh_submit()
  nilfs2: Convert nilfs_mdt_submit_block to bh_submit()
  md-bitmap: Convert read_file_page and write_file_page to bh_submit()
  buffer: Remove submit_bh()
  buffer: Remove b_end_io
  buffer: Remove __end_buffer_read_notouch()
  buffer: Remove end_buffer_write_sync()

 Documentation/filesystems/locking.rst |  14 -
 Documentation/trace/ftrace.rst        |   4 +-
 drivers/md/md-bitmap.c                |  27 +-
 drivers/md/raid5.h                    |   6 +-
 fs/buffer.c                           | 372 ++++++++++++++------------
 fs/ext4/ext4.h                        |  10 +-
 fs/ext4/fast_commit.c                 |   8 +-
 fs/ext4/ialloc.c                      |   5 +-
 fs/ext4/mmp.c                         |   4 +-
 fs/ext4/super.c                       |  16 +-
 fs/gfs2/bmap.c                        |   7 +-
 fs/gfs2/dir.c                         |   6 +-
 fs/gfs2/meta_io.c                     |  11 +-
 fs/jbd2/commit.c                      |  13 +-
 fs/jbd2/journal.c                     |   3 +-
 fs/nilfs2/btnode.c                    |   3 +-
 fs/nilfs2/gcinode.c                   |   3 +-
 fs/nilfs2/mdt.c                       |   3 +-
 fs/ocfs2/buffer_head_io.c             |  20 +-
 include/linux/buffer_head.h           |  16 +-
 mm/vmscan.c                           |   2 +-
 21 files changed, 280 insertions(+), 273 deletions(-)

-- 
2.47.3