Ubuntu Wiki

BCC: a powerful front end to extended Berkeley Packet Filters

The BPF Compiler Collection (BCC) is a toolkit for building kernel tracing tools that leverage the functionality provided by the Linux extended Berkeley Packet Filters (BPF).

BCC allows one to write BPF programs with front-ends in Python or Lua with kernel instrumentation written in C. The instrumentation code is built into sandboxed eBPF byte code and is executed in the kernel.

The BCC github project README file provides an excellent overview and description of BCC and the various available BCC tools. Building BCC from scratch can be a bit time consuming, however, the good news is that the BCC tools are now available as a snap and so BCC can be quickly and easily installed just using:

sudo snap install --devmode bcc

There are currently over 50 BCC tools in the snap:

bcc.argdist bcc.cpuunclaimed bcc.gethostlatency bcc.slabratetop bcc.tcpretrans
bcc.bashreadline bcc.dcsnoop bcc.hardirqs bcc.softirqs bcc.tcptop
bcc.biolatency bcc.dcstat bcc.killsnoop bcc.sslsniff bcc.trace
bcc.biosnoop bcc.execsnoop bcc.mdflush bcc.stackcount bcc.ttysnoop
bcc.biotop bcc.ext4dist bcc.memleak bcc.stacksnoop bcc.vfscount
bcc.bitesize bcc.ext4slower bcc.offcputime bcc.statsnoop bcc.vfsstat
bcc.btrfsdist bcc.filelife bcc.offwaketime bcc.syncsnoop bcc.wakeuptime
bcc.btrfsslower bcc.fileslower bcc.oomkill bcc.tcpaccept bcc.xfsdist
bcc.cachestat bcc.filetop bcc.opensnoop bcc.tcpconnect bcc.xfsslower
bcc.cachetop bcc.funccount bcc.pidpersec bcc.tcpconnlat bcc.zfsdist
bcc.cpudist bcc.funclatency bcc.runqlat bcc.tcplife bcc.zfsslower

Let's have a quick look at a few:

cachetop

cachetop allows one to view the top page cache hit/miss statistics. To run this use:

sudo bcc.cachetop

23:11:26 Buffers MB: 123 / Cached MB: 1025 / Sort: HITS / Order: ascending
PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT%
 10865 _apt gpgv 2 0 0 100.0% 0.0%
 373 root jbd2/vda1-8 3 1 1 50.0% 25.0%
 10856 root python 4 0 0 100.0% 0.0%
 2190 messageb dbus-daemon 6 0 0 100.0% 0.0%
 421 root systemd-journal 6 0 0 100.0% 0.0%
 2261 syslog rs:main Q:Reg 7 0 2 71.4% 0.0%
 10857 root sudo 13 0 0 100.0% 0.0%
 10857 king sudo 14 0 0 100.0% 0.0%
 10862 _apt http 34 0 0 100.0% 0.0%
 10861 root http 48 0 0 100.0% 0.0%
 10862 root http 48 0 0 100.0% 0.0%
 10863 root http 48 0 0 100.0% 0.0%
 10865 root gpgv 48 0 0 100.0% 0.0%
 11092 root store 48 0 0 100.0% 0.0%
 10975 _apt apt-key 150 0 0 100.0% 0.0%
 10977 _apt apt-key 150 0 0 100.0% 0.0%
 10978 _apt apt-key 150 0 0 100.0% 0.0%
 10987 _apt apt-key 150 0 0 100.0% 0.0%
 10989 _apt apt-key 150 0 0 100.0% 0.0%
 10990 _apt apt-key 150 0 0 100.0% 0.0%
 10997 _apt apt-key 150 0 0 100.0% 0.0%

funccount

The funccount tool allows one to count the number of times specific functions get called. For example, to see how many kernel functions with the name starting with "do_" get called per second one can use:

sudo bcc.funccount "do_*" -i 1
FUNC COUNT
do_task_dead 1
do_futex 1
do_exit 1
do_wp_page 1
do_writepages 1
do_truncate 1
do_nanosleep 2
do_flush_tlb_all 3
do_get_write_access 3
do_output_char 3
do_select 5
do_sys_poll 6
do_sys_ftruncate.constprop.15 10
do_timerfd_settime 15
do_IRQ 32
do_check 146
do_timer 150
do_mmap 201
do_dentry_open 332
do_filp_open 522
do_sys_open 636
do_vfs_ioctl 803
do_kernel_range_flush 1004
do_syscall_64 4236

To see how to use all the options in this tool, use the -h option:

sudo bcc.funccount -h

The funccount tool can be useful to check on kernel activity by checking on hits on specific function names.

slabratetop

The slabratetop tool is useful to see the active kernel SLAB/SLUB memory allocation rates:

sudo bcc.slabratetop
14:22:01 loadavg: 2.10 2.47 1.63 3/183 9955

CACHE ALLOCS BYTES
TCP 4986 9892224
sock_inode_cache 4986 3191040
kmalloc-256 7549 1932544
dentry 4999 959808
request_sock_TCP 2493 817704
tw_sock_TCP 2493 678096
kmalloc-4096 74 303104
kmalloc-64 2493 159552
vm_area_struct 120 24000
anon_vma_chain 126 8064
shmem_inode_cache 11 7656
anon_vma 65 5200
mm_struct 2 3968
cred_jar 12 2304
sighand_cache 1 2112
signal_cache 1 1088
files_cache 1 704
proc_inode_cache 1 648
inode_cache 1 592
buffer_head 2 208

opensnoop

If you want to see which process is opening specific files, one can snoop on open system calls use the opensnoop tool:

sudo bcc.opensnoop -T
PID COMM FD ERR PATH
2433 irqbalance 3 0 /proc/interrupts
2433 irqbalance 3 0 /proc/stat
2433 irqbalance 3 0 /proc/interrupts
2433 irqbalance 3 0 /proc/stat
2433 irqbalance 3 0 /proc/interrupts
2433 irqbalance 3 0 /proc/stat
2433 irqbalance 3 0 /proc/interrupts
2433 irqbalance 3 0 /proc/stat
1 systemd 15 0 /proc/2223/cgroup
1 systemd 15 0 /proc/562/cgroup
1 systemd 15 0 /proc/2290/cgroup
2433 irqbalance 3 0 /proc/interrupts
2433 irqbalance 3 0 /proc/stat

Hopefully this will give you a taste of some of the useful tools that are available in BCC. The BCC snap is regularly updated to be in-sync with the bcc project, so it is worth regularly checking for new or updated bcc tools.