Page lock stack trace instrumentation

The following patches implement a mechanism for capturing the stack
backtrace of processes that lock and unlock pages.  This
instrumentation was used to track down an application hang that
occurred when a ^C was used to kill a multi-threaded application
(often referred to euphemistically as a "well-known commercial OLTP
database") running on top of NFS.

The patches are somewhat specific to the particular problem we
were chasing, so first, a little background:

Using sysRq-T, our hang was tracked down to a process waiting in
invalidate_inode_pages2_range() for a page to be unlocked.  The
process was holding the i_mutex on a particular file, and a set of
other processes were waiting for the same i_mutex.

The process waiting for the page was never awoken in this case, though
the page flags for that page showed that it was in fact not locked.
Since this process never awoke, it never released the i_mutex, thus
all of the other processes were waiting forever for the mutex.

This problem appeared only in SMP configurations, and only after
Matthew Wilcox's work to use lock_page_killable() to eliminate the
need for the "intr" and "nointr" NFS mount options.

To pursue this further, I created some instrumentation, suggested by
Chris Mason, to identify the processes and functions that had last
touched the problematic page.  A queue of pages waiting in
invalidate_inode_pages2_range() quickly identifies which page or pages
we are stuck on.

The first patch is what our test team was using when Chris discovered
the problem.  It evolved over several months of testing.

The second patch is some clean-up that I had planned for the next
iteration of this patch, before the fix was discovered.  It was never
tested, and is included for completeness.

These patches are against 2.6.27 stable, and are posted here for the
record, at Chris's request.

---

Chuck Lever (2):
      PAGECACHE: Page lock tracing clean up
      PAGECACHE: Record stack backtrace in lock_page()


 drivers/char/sysrq.c     |    4 +
 include/linux/mm_types.h |   21 +++++++
 include/linux/pagemap.h  |   21 +++++++
 include/linux/wait.h     |    2 +
 kernel/sched.c           |   24 ++++++++
 kernel/wait.c            |    9 +++
 mm/filemap.c             |  130 ++++++++++++++++++++++++++++++++++++++++++++--
 mm/hugetlb.c             |    1 
 mm/page_alloc.c          |    4 +
 mm/slub.c                |    5 ++
 mm/truncate.c            |   64 ++++++++++++++++++++++-
 11 files changed, 277 insertions(+), 8 deletions(-)

-- 
Chuck Lever <chuck.lever@oracle.com>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>