pwru is an eBPF-based tool for tracing network packets in
the Linux kernel with advanced filtering capabilities. It allows fine-grained
introspection of kernel state to facilitate debugging network connectivity issues.
The following example shows where the packets of a curl request are dropped
after installing an IP tables rule:
pwru requires >= 5.3 kernel to run. For --output-skb >= 5.9 kernel is required. For --backend=kprobe-multi >= 5.18 kernel is required.
pwru optionally requires debugfs. It has to be mounted in /sys/kernel/debug. In case the folder is empty, it can be mounted with:
mount -t debugfs none /sys/kernel/debug
The following kernel configuration is required.
| Option | Backend | Note |
|---|---|---|
| CONFIG_DEBUG_INFO_BTF=y | both | available since >= 5.3 |
| CONFIG_KPROBES=y | both | |
| CONFIG_PERF_EVENTS=y | both | |
| CONFIG_BPF=y | both | |
| CONFIG_BPF_SYSCALL=y | both | |
| CONFIG_FUNCTION_TRACER=y | kprobe-multi | /sys/kernel/debug/tracing/available_filter_functions |
| CONFIG_FPROBE=y | kprobe-multi | available since >= 5.18 |
You can use zgrep $OPTION /proc/config.gz to validate whether option is enabled.
You can download the statically linked executable for x86_64 and arm64 from the release page.
$ ./pwru --help
Usage: ./pwru [options] [pcap-filter]
Available pcap-filter: see "man 7 pcap-filter"
Available options:
--all-kmods attach to all available kernel modules
--backend string Tracing backend('kprobe', 'kprobe-multi'). Will auto-detect if not specified.
--filter-func string filter kernel functions to be probed by name (exact match, supports RE2 regular expression)
--filter-ifname string filter skb ifname in --filter-netns (if not specified, use current netns)
--filter-kprobe-batch uint batch size for kprobe attaching/detaching (default 10)
--filter-mark mark[/mask] filter skb mark (format: mark[/mask], e.g., 0xa00/0xf00) (default 0x0)
--filter-netns string filter netns ("/proc/<pid>/ns/net", "inode:<inode>")
--filter-non-skb-funcs strings filter non-skb kernel functions to be probed (--filter-track-skb-by-stackid will be enabled)
--filter-trace-tc trace TC bpf progs
--filter-trace-xdp trace XDP bpf progs
--filter-track-bpf-helpers trace BPF helper functions
--filter-track-skb trace a packet even if it does not match given filters (e.g., after NAT or tunnel decapsulation)
--filter-track-skb-by-stackid trace a packet even after it is kfreed (e.g., traffic going through bridge)
--filter-tunnel-pcap-l2 string pcap expression for vxlan/geneve tunnel (l2)
--filter-tunnel-pcap-l3 string pcap expression for vxlan/geneve tunnel (l3)
-h, --help display this message and exit
--kernel-btf string specify kernel BTF file
--kmods strings list of kernel modules names to attach to
--output-caller print caller function name
--output-file string write traces to file (rotates at 100MB by default)
--output-file-max-size int max size in MB per file before rotation (default 100)
--output-file-max-age int max age in days to keep rotated files (default 0, no limit)
--output-file-max-backups int max number of rotated files to keep (default 0, no limit)
--output-file-compress compress rotated files with gzip (default false)
--output-json output traces in JSON format
--output-limit-lines uint exit the program after the number of events has been received/printed
--output-meta print skb metadata (default true)
--output-skb print skb
--output-skb-cb print skb->cb
--output-skb-metadata strings print skb metadata (e.g., "skb->mark", "skb->hash"), 4 at most
--output-skb-shared-info print skb shared info
--output-stack print stack
--output-tcp-flags print TCP flags
--output-tunnel print encapsulated tunnel header data
--output-tuple print L4 tuple (default true)
--output-xdp-metadata strings print xdp metadata (e.g., "xdp->rxq->queue_index"), 4 at most
--set-percpu-buf uint32 set the size of buffers to print skb data (used by --output-skb and --output-skb-shared-info) (default 4096)
--timestamp string print timestamp per skb ("current", "relative", "absolute", "none") (default "none")
--version show pwru version and exit
The --filter-func switch does an exact match on function names i.e.
--filter-func=foo only matches foo(); for a wildcarded match, try
--filter-func=".*foo.*" instead.
Docker images for pwru are published at https://hub.docker.com/r/cilium/pwru.
An example how to run pwru with Docker:
docker run --privileged --rm -t --pid=host -v /sys/kernel/debug/:/sys/kernel/debug/ cilium/pwru pwru --output-tuple 'host 1.1.1.1'
The following example shows how to run pwru on a given node:
#!/usr/bin/env bash
NODE=kind-control-plane
PWRU_ARGS="--output-tuple 'host 1.1.1.1'"
trap " kubectl delete --wait=false pod pwru " EXIT
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: pwru
spec:
nodeSelector:
kubernetes.io/hostname: ${NODE}
containers:
- image: docker.io/cilium/pwru:latest
name: pwru
volumeMounts:
- mountPath: /sys/kernel/debug
name: sys-kernel-debug
securityContext:
privileged: true
command: ["/bin/sh"]
args: ["-c", "pwru ${PWRU_ARGS}"]
volumes:
- name: sys-kernel-debug
hostPath:
path: /sys/kernel/debug
type: DirectoryOrCreate
hostNetwork: true
hostPID: true
EOF
kubectl wait pod pwru --for condition=Ready --timeout=90s
kubectl logs -f pwru
Also see contrib/scripts/pwru-on-k8s-node for a reusable version of this script.
pwru is available in some distro packages. GH#89 tracks any packaging work and progress.
See docs/vagrant.md
- Go >= 1.16
- LLVM/clang >= 12
- Bison
- Lex/Flex >= 2.5.31
make
Alternatively, you can build in the Docker container:
make release
Ensure that all commits have Developer Certificate of Origin by adding a Signed-off-by line to your commit messages.
Join the #pwru Slack channel to chat with
developers, maintainers, and other users. This is a good first stop to ask
questions and share your experiences.
The detective gopher is based on the Go gopher designed by Renee French.