tracepoint: Avoid double static_branch evaluation at guarded call sites
| From: | "Vineeth Pillai (Google)" <vineeth-AT-bitbyteword.org> | |
| To: | Steven Rostedt <rostedt-AT-goodmis.org>, Peter Zijlstra <peterz-AT-infradead.org>, Dmitry Ilvokhin <d-AT-ilvokhin.com> | |
| Subject: | [PATCH v2 00/19] tracepoint: Avoid double static_branch evaluation at guarded call sites | |
| Date: | Mon, 23 Mar 2026 12:00:19 -0400 | |
| Message-ID: | <20260323160052.17528-1-vineeth@bitbyteword.org> | |
| Cc: | "Vineeth Pillai (Google)" <vineeth-AT-bitbyteword.org>, Masami Hiramatsu <mhiramat-AT-kernel.org>, Mathieu Desnoyers <mathieu.desnoyers-AT-efficios.com>, Ingo Molnar <mingo-AT-redhat.com>, Jens Axboe <axboe-AT-kernel.dk>, io-uring-AT-vger.kernel.org, "David S. Miller" <davem-AT-davemloft.net>, Eric Dumazet <edumazet-AT-google.com>, Jakub Kicinski <kuba-AT-kernel.org>, Paolo Abeni <pabeni-AT-redhat.com>, Alexei Starovoitov <ast-AT-kernel.org>, Daniel Borkmann <daniel-AT-iogearbox.net>, Marcelo Ricardo Leitner <marcelo.leitner-AT-gmail.com>, Xin Long <lucien.xin-AT-gmail.com>, Jon Maloy <jmaloy-AT-redhat.com>, Aaron Conole <aconole-AT-redhat.com>, Eelco Chaudron <echaudro-AT-redhat.com>, Ilya Maximets <i.maximets-AT-ovn.org>, netdev-AT-vger.kernel.org, bpf-AT-vger.kernel.org, linux-sctp-AT-vger.kernel.org, tipc-discussion-AT-lists.sourceforge.net, dev-AT-openvswitch.org, Jiri Pirko <jiri-AT-resnulli.us>, Oded Gabbay <ogabbay-AT-kernel.org>, Koby Elbaz <koby.elbaz-AT-intel.com>, dri-devel-AT-lists.freedesktop.org, "Rafael J. Wysocki" <rafael-AT-kernel.org>, Viresh Kumar <viresh.kumar-AT-linaro.org>, "Gautham R. Shenoy" <gautham.shenoy-AT-amd.com>, Huang Rui <ray.huang-AT-amd.com>, Mario Limonciello <mario.limonciello-AT-amd.com>, Len Brown <lenb-AT-kernel.org>, Srinivas Pandruvada <srinivas.pandruvada-AT-linux.intel.com>, linux-pm-AT-vger.kernel.org, MyungJoo Ham <myungjoo.ham-AT-samsung.com>, Kyungmin Park <kyungmin.park-AT-samsung.com>, Chanwoo Choi <cw00.choi-AT-samsung.com>, Christian König <christian.koenig-AT-amd.com>, Sumit Semwal <sumit.semwal-AT-linaro.org>, linaro-mm-sig-AT-lists.linaro.org, Eddie James <eajames-AT-linux.ibm.com>, Andrew Jeffery <andrew-AT-codeconstruct.com.au>, Joel Stanley <joel-AT-jms.id.au>, linux-fsi-AT-lists.ozlabs.org, David Airlie <airlied-AT-gmail.com>, Simona Vetter <simona-AT-ffwll.ch>, Alex Deucher <alexander.deucher-AT-amd.com>, Danilo Krummrich <dakr-AT-kernel.org>, Matthew Brost <matthew.brost-AT-intel.com>, Philipp Stanner <phasta-AT-kernel.org>, Harry Wentland <harry.wentland-AT-amd.com>, Leo Li <sunpeng.li-AT-amd.com>, amd-gfx-AT-lists.freedesktop.org, Jiri Kosina <jikos-AT-kernel.org>, Benjamin Tissoires <bentiss-AT-kernel.org>, linux-input-AT-vger.kernel.org, Wolfram Sang <wsa+renesas-AT-sang-engineering.com>, linux-i2c-AT-vger.kernel.org, Mark Brown <broonie-AT-kernel.org>, Michael Hennerich <michael.hennerich-AT-analog.com>, Nuno Sá <nuno.sa-AT-analog.com>, linux-spi-AT-vger.kernel.org, "James E.J. Bottomley" <James.Bottomley-AT-HansenPartnership.com>, "Martin K. Petersen" <martin.petersen-AT-oracle.com>, linux-scsi-AT-vger.kernel.org, Chris Mason <clm-AT-fb.com>, David Sterba <dsterba-AT-suse.com>, linux-btrfs-AT-vger.kernel.org, Thomas Gleixner <tglx-AT-linutronix.de>, Andrew Morton <akpm-AT-linux-foundation.org>, SeongJae Park <sj-AT-kernel.org>, linux-mm-AT-kvack.org, Borislav Petkov <bp-AT-alien8.de>, Dave Hansen <dave.hansen-AT-linux.intel.com>, x86-AT-kernel.org, linux-trace-kernel-AT-vger.kernel.org, linux-kernel-AT-vger.kernel.org | |
| Archive-link: | Article |
When a caller already guards a tracepoint with an explicit enabled check:
if (trace_foo_enabled() && cond)
trace_foo(args);
trace_foo() internally re-evaluates the static_branch_unlikely() key.
Since static branches are patched binary instructions the compiler cannot
fold the two evaluations, so every such site pays the cost twice.
This series introduces trace_call__##name() as a companion to
trace_##name(). It calls __do_trace_##name() directly, bypassing the
redundant static-branch re-check, while preserving all other correctness
properties of the normal path (RCU-watching assertion, might_fault() for
syscall tracepoints). The internal __do_trace_##name() symbol is not
leaked to call sites; trace_call__##name() is the only new public API.
if (trace_foo_enabled() && cond)
trace_call__foo(args); /* calls __do_trace_foo() directly */
The first patch adds the three-location change to
include/linux/tracepoint.h (__DECLARE_TRACE, __DECLARE_TRACE_SYSCALL,
and the !TRACEPOINTS_ENABLED stub). The remaining 18 patches
mechanically convert all guarded call sites found in the tree:
kernel/, io_uring/, net/, accel/habanalabs, cpufreq/, devfreq/,
dma-buf/, fsi/, drm/, HID, i2c/, spi/, scsi/ufs/, btrfs/,
net/devlink/, kernel/time/, kernel/trace/, mm/damon/, and arch/x86/.
This series is motivated by Peter Zijlstra's observation in the discussion
around Dmitry Ilvokhin's locking tracepoint instrumentation series, where
he noted that compilers cannot optimize static branches and that guarded
call sites end up evaluating the static branch twice for no reason, and
by Steven Rostedt's suggestion to add a proper API instead of exposing
internal implementation details like __do_trace_##name() directly to
call sites:
https://lore.kernel.org/linux-trace-kernel/8298e098d3418c...
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Changes in v2:
- Renamed trace_invoke_##name() to trace_call__##name() (double
underscore) per review comments.
- Added 4 new patches covering sites missed in v1, found using
coccinelle to scan the tree (Keith Busch):
* net/devlink: guarded tracepoint_enabled() block in trap.c
* kernel/time: early-return guard in tick-sched.c (tick_stop)
* kernel/trace: early-return guard in trace_benchmark.c
* mm/damon: early-return guard in core.c
* arch/x86: do_trace_*() wrapper functions in lib/msr.c, which
are called exclusively from tracepoint_enabled()-guarded sites
in asm/msr.h
v1: https://lore.kernel.org/linux-trace-kernel/abSqrJ1J59RQC4...
Vineeth Pillai (Google) (19):
tracepoint: Add trace_call__##name() API
kernel: Use trace_call__##name() at guarded tracepoint call sites
io_uring: Use trace_call__##name() at guarded tracepoint call sites
net: Use trace_call__##name() at guarded tracepoint call sites
accel/habanalabs: Use trace_call__##name() at guarded tracepoint call
sites
cpufreq: Use trace_call__##name() at guarded tracepoint call sites
devfreq: Use trace_call__##name() at guarded tracepoint call sites
dma-buf: Use trace_call__##name() at guarded tracepoint call sites
fsi: Use trace_call__##name() at guarded tracepoint call sites
drm: Use trace_call__##name() at guarded tracepoint call sites
HID: Use trace_call__##name() at guarded tracepoint call sites
i2c: Use trace_call__##name() at guarded tracepoint call sites
spi: Use trace_call__##name() at guarded tracepoint call sites
scsi: ufs: Use trace_call__##name() at guarded tracepoint call sites
btrfs: Use trace_call__##name() at guarded tracepoint call sites
net: devlink: Use trace_call__##name() at guarded tracepoint call
sites
kernel: time, trace: Use trace_call__##name() at guarded tracepoint
call sites
mm: damon: Use trace_call__##name() at guarded tracepoint call sites
x86: msr: Use trace_call__##name() at guarded tracepoint call sites
arch/x86/lib/msr.c | 6 +++---
drivers/accel/habanalabs/common/device.c | 12 ++++++------
drivers/accel/habanalabs/common/mmu/mmu.c | 3 ++-
drivers/accel/habanalabs/common/pci/pci.c | 4 ++--
drivers/cpufreq/amd-pstate.c | 10 +++++-----
drivers/cpufreq/cpufreq.c | 2 +-
drivers/cpufreq/intel_pstate.c | 2 +-
drivers/devfreq/devfreq.c | 2 +-
drivers/dma-buf/dma-fence.c | 4 ++--
drivers/fsi/fsi-master-aspeed.c | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++--
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 2 +-
drivers/gpu/drm/scheduler/sched_entity.c | 4 ++--
drivers/hid/intel-ish-hid/ipc/pci-ish.c | 2 +-
drivers/i2c/i2c-core-slave.c | 2 +-
drivers/spi/spi-axi-spi-engine.c | 4 ++--
drivers/ufs/core/ufshcd.c | 12 ++++++------
fs/btrfs/extent_map.c | 4 ++--
fs/btrfs/raid56.c | 4 ++--
include/linux/tracepoint.h | 11 +++++++++++
io_uring/io_uring.h | 2 +-
kernel/irq_work.c | 2 +-
kernel/sched/ext.c | 2 +-
kernel/smp.c | 2 +-
kernel/time/tick-sched.c | 12 ++++++------
kernel/trace/trace_benchmark.c | 2 +-
mm/damon/core.c | 2 +-
net/core/dev.c | 2 +-
net/core/xdp.c | 2 +-
net/devlink/trap.c | 2 +-
net/openvswitch/actions.c | 2 +-
net/openvswitch/datapath.c | 2 +-
net/sctp/outqueue.c | 2 +-
net/tipc/node.c | 2 +-
35 files changed, 74 insertions(+), 62 deletions(-)
--
2.53.0