What's Changed

• argument capture support for ARM64 added;
• specially handle BPF helpers tracing, as normally they would be detected as non-failing (as they return u64); now retsnoop has hard-coded table with all BPF helpers and overrides return result according to UAPI semantics: pointer, signed integer, void, etc;
• work around objpool -ENOMEM kernel bug that can prevent multi-kprobe attachment; now retsnoop will automatically reduce attachment batch size until it succeeds, finding reasonable tradeoff between very slow singular kprobe mode and full multi-kprobe mode;
• added warning when entry (-e) function is detected as non-failing by retsnoop, which otherwise would lead to user confusion (unless --successes/-S mode is specified);
• detect bogus /proc/kallsyms addresses (most probably due to kernel.kptr_restrict=2 sysctl setting), and propose steps to resolve this;
• extended denylist of dangerous functions which shouldn't be traced, increasing safety.

Full Changelog: v0.10.1...v0.11

Bug fixes

• handle crash that might happen when function argument capture is enabled and some data is dropped (due to too high rate of function calls, overflowing ringbuf);
• fix handling of thread/process name filter (-n/-N);
• fix problem with BPF verification failure on some kernel versions;
• sidecar updates, removing deprecated addr2line API usage and updating to clap 4 for command line args parsing.

What's Changed

• logic: fix handling of partial fnargs/ctxargs data by @anakryiko in #76
• retsnoop: avoid checking if (sess->defunct) twice by @anakryiko in #81
• sidecar: Remove usage of deprecated APIs by @d-e-s-o in #80
• retsnoop: fix handling -n (--comm) and -N (--no-comm) filtering by @mannkafai in #82
• sidecar: migrate from clap 2 to clap 4 by @theihor in #83
• Re-generate vmlinux.h by @anakryiko in #84
• vmlinux.h: regenerate vmlinux.h again, with feeling by @anakryiko in #85

New Contributors

• @d-e-s-o made their first contribution in #80
• @mannkafai made their first contribution in #82
• @theihor made their first contribution in #83

Full Changelog: v0.10...v0.10.1

Contributors

• @anakryiko
• @mannkafai
• @theihor
• @d-e-s-o

Major changes

This is a big release with some new major features added (though we still stay within minor version update, as there might be still some minor breaking changes). Most notable changes:

• Function arguments capture (-A argument). Retsnoop now can capture all input arguments for all traced functions and print them in human-readable form. See README for more details.
• Injected probes (-J). In addition to traced functions specified with -e and -a flags, it's now possible to also specify a single-point injected probes (kprobes, kretprobes, tracepoints, and raw tracepoints). Note that for kprobe, it's possible to specify extra offset (e.g., -J kprobe:bprm_execve+12), which allows to trace inlined functions and internals of functions (normally retsnoop only traces function entry and exit). See README for more details.
• Retsnoop can also capture extra context for injected probes, just use -A and -J together. For kprobes and kretprobes registers state is captured, for tracepoints and raw tracepoints their actual arguments are captured. See README for more details.
• (Breaking change!) It's now possible to enable only function call trace mode (-T) separately from default call stack mode. The latter now is controlled with -E flag. The important distinction and a breaking change is that with function call trace mode --success-stacks/-S option is implied, which makes most sense for function call tracing. When retsnoop -E is specified, even with -T, the original behavior of tracing and emitting only erroring call stacks (i.e., those that end up returning error from entry functions specified with -e arguments). So, in short:
  • retsnoop -T emits all function call trace, both successful and erroring;
  • retsnoop -E (or just retsnoop, as -E is the default mode) emits only erroring call stacks (no function call traces);
  • retsnoop -E -S will emit call stacks only (no function call traces), but both erroring and successful ones;
  • retsnoop -E -T will emit both call stacks and function call traces, but only erroring ones;
  • retsnoop -E -T -S will do both call stacks and function call trace for both successful and erroring cases.
• Added kernel module BTF support, improving tracing functions defined in kernel modules.
• Added advanced configuration options, specified with -C flag. See retsnoop --config-help for list of supported options and more details.
• Significant rework of --help output.
• Many smaller bug fixes and usability improvements.

What's Changed

• Fix typos in README by @Antiz96 in #57
• Makefile: use LDFLAGS for linking by @martinetd in #58
• Makefile: Update variables for package builds with external artifacts by @martinetd in #56
• Makefile: Do not rebuild the sidecar if not the default path by @martinetd in #59
• Support module BTF and lots of log improvements by @anakryiko in #60
• Lbr improvements by @anakryiko in #61
• Retsnoop session revamp by @anakryiko in #62
• Retsnoop function args capture support by @anakryiko in #63
• Retsnoop config and function args capture polish by @anakryiko in #64
• Bump gimli stack by @michel-slm in #66
• Retsnoop vararg support in printf-like functions by @anakryiko in #68
• retsnoop: make func call trace and call stack modes independent by @anakryiko in #69
• Retsnoop injection probes and other improvements by @anakryiko in #70
• retsnoop: handle idle threads properly by @anakryiko in #71
• Retsnoop improvements for LBR, stitched stacks, and interim stacks by @anakryiko in #72
• Retsnoop ARM64 improvements by @anakryiko in #75

New Contributors

• @Antiz96 made their first contribution in #57

Full Changelog: v0.9.8...v0.10

Contributors

• @michel-slm
• @anakryiko
• @martinetd
• @Antiz96

What's Changed

• A few small fixes and clean ups by @anakryiko in #53
• bpftool: Pass additional compile flags as EXTRA_CFLAGS, not CFLAGS by @qmonnet in #52
• sidecar: Update addr2line dependency to 0.21 by @danielocfb in #55

New Contributors

• @danielocfb made their first contribution in #55

Full Changelog: v0.9.7...v0.9.8

Contributors

• @anakryiko
• @qmonnet
• @danielocfb

What's Changed

• Add release workflow for shipping binaries and combined sources by @qmonnet in #42 and #44
• retsnoop: Fix sign extension failure logic by @anakryiko in #50
• retsnoop: Remove a limit of 4096 attachable functions by @anakryiko in #48
• retsnoop: Remove hard-coded maximum of 256 CPUs supported by @anakryiko in #49
• retsnoop: Use bpf_probe_read_kernel() instead of bpf_probe_read() by @iii-i in #46
• retsnoop: Pass the real envp to the sidecar by @erthalion in #47

New Contributors

• @qmonnet made their first contribution in #42
• @iii-i made their first contribution in #46
• @erthalion made their first contribution in #47

Full Changelog: v0.9.6...v0.9.7

Contributors

• @anakryiko
• @erthalion
• @qmonnet
• @iii-i

What's Changed

• Fix calibration unreliability on some new kernels by @anakryiko in #41

Full Changelog: v0.9.5...v0.9.6

Contributors

• @anakryiko

What's Changed

Massive improvements in how retsnoop determines whether kprobes are attachable:

• add --debug multi-kprobe mode to bisect failing multi-kprobe attachment; it quickly narrows down and logs which kprobes were attempted but failed to be attached;
• skip attaching to kernel functions that have non-unique name and some of instances are not traceable;
• resolve internal mix up of function and data ksyms;
• internal fixes to consistently take into account kernel module to which ksym/kprobe belongs to.

Overall, these fixes and improvements make retsnoop's mass-attach behavior more reliable.

Full Changelog: v0.9.4...v0.9.5

Bug fixes

• fix IP (instruction pointer) fetching on non-x86_64 architectures on older kernels;
• handle io_uring source code files better, after Linux code reorganization;
• automatically pick debugfs (/sys/kernel/debug/tracing) or tracefs (/sys/kernel/tracing), whichever is available;
• handle very old kernels that don't support BPF global data more gracefully.

What's Changed

retsnoop now supports DWARF-based symbolization (i.e.,
source code file/line info and inline functions) on
KASLR-enabled Linux kernels.

What's Changed

• -F (fentry/fexit) mode now supports tracing void-returning functions;
• few fixes for -F (fentry/fexit) mode interacting weirdly with source code globs;
• retsnoop now can be compiled across x86_64, i686, aarch64, ppc64le, s390x, riscv64 architectures by using per-architecture pre-generated minimal vmlinux.h (see gen-vmlinux-headers.sh script);
• retsnoop now builds bootstrap (lightweight) version of bpftool from submodule, which allows it to be compilable on multiple-architectures. Previously retsnoop's Makefile relied on checked in pre-built x86_64 bpftool binary;
• massive revamp of README.md;
• usage text fixes and improvements.

Full Changelog: v0.9.1...v0.9.2