BCC – Dynamic Linux Tracing Tools for Network & Monitoring

BCC (BPF Compiler Collection) is a powerful set of appropriate tools and example files for creating resourceful kernel tracing and manipulation programs.

It utilizes extended BPF (Berkeley Packet Filters), initially known as eBPF which was one of the new features in Linux 3.15.

Practically, most of the components used by BCC require Linux 4.1 or above, and its noteworthy features include:

• Requires no 3rd party kernel module, since all the tools work based on BPF which is built into the kernel, and BCC uses features added in Linux 4.x series.
• Enables observation of software execution.
• Comprises of several performance analysis tools with example files and man pages.

Best suited for advanced Linux users, BCC makes it easy to write BPF programs using kernel instrumentation in C, and front-ends in Python and lua.

Additionally, it supports multiple tasks such as performance analysis, monitoring, network traffic control plus lots more.

How To Install BCC in Linux Systems

Remember that BCC uses features added in Linux kernel version 4.1 or above, and as a requirement, the kernel should have been compiled with the flags set below:

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# [optional, for tc filters]
CONFIG_NET_CLS_BPF=m
# [optional, for tc actions]
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
# [for Linux kernel versions 4.1 through 4.6]
CONFIG_HAVE_BPF_JIT=y
# [for Linux kernel versions 4.7 and later]
CONFIG_HAVE_EBPF_JIT=y
# [optional, for kprobes]
CONFIG_BPF_EVENTS=y
# Need kernel headers through /sys/kernel/kheaders.tar.xz
CONFIG_IKHEADERS=y

You can check all the specified kernel configuration flags with a single grep command and provide all the flag names as patterns.

grep -E 'CONFIG_BPF=y|CONFIG_BPF_SYSCALL=y|CONFIG_NET_CLS_BPF=m|CONFIG_NET_ACT_BPF=m|CONFIG_BPF_JIT=y|CONFIG_HAVE_BPF_JIT=y|CONFIG_HAVE_EBPF_JIT=y|CONFIG_BPF_EVENTS=y|CONFIG_IKHEADERS=y' /boot/config-$(uname -r)

After verifying kernel flags, it’s time to install BCC tools in Linux systems.

Install BCC Tools on Debian

You can find bcc and its tools in the regular Debian main repository. They come from the source package bpfcc and are named bpfcc-tools, python3-bpfcc, libbpfcc, and libbpfcc-dev.

echo deb http://cloudfront.debian.net/debian sid main >> /etc/apt/sources.list
sudo apt-get install -y bpfcc-tools libbpfcc libbpfcc-dev linux-headers-$(uname -r)

Install BCC Tools on Ubuntu

You can find different versions of bcc in the regular Ubuntu Universe repository.

sudo apt-get install bpfcc-tools linux-headers-$(uname -r)

Install BCC Tools on Fedora

As of Fedora 40, bcc binaries are available in the standard repository.

sudo dnf install bcc

Install BCC Tools on RHEL

bcc is already included in the official yum repository as bcc-tools.

sudo yum install bcc-tools

Install BCC Tools on Arch Linux

bcc is available in the standard Arch repository and can be installed with the pacman command.

pacman -S bcc bcc-tools python-bcc

Install BCC Tools on openSUSE

For openSUSE Leap 42.2 (and later) and Tumbleweed, bcc is already included in the official repo.

sudo zypper ref
sudo zypper in bcc-tools bcc-examples

How To Use BCC Tools in Linux Systems

All the BCC tools are installed under /usr/share/bcc/tools directory. However, you can alternatively run them from the BCC Github repository under /tools where they end with an .py extension.

ls /usr/share/bcc/tools

We shall cover a few examples of – monitoring general Linux system performance and networking.

Tracing open() Syscalls in Linux

Let’s start by tracing all open() syscalls using opensnoop tool. This enables us to tell how various applications work by identifying their data files, config files, and many more:

cd /usr/share/bcc/tools 
sudo ./opensnoop

Analyzing Disk I/O Latency in Linux

This example shows a summarized distribution of disk I/O latency using biolatency tool. After executing the command, wait for a few minutes and hit Ctrl-C to end it and view the output.

sudo ./biolatency

Tracing New Processes in Linux

In this section, we shall move to tracing new processes in execution using execsnoop tool. Each time a process is forked by fork() and exec() syscalls, it is shown in the output. However, not all processes are captured.

sudo ./execsnoop

Trace Slower Ext4 File System Operations

Using ext4slower to trace the ext4 file system common operations that are slower than 10ms, to help us identify independently slow disk I/O via the file system.

It only outputs those operations that exceed a threshold:

sudo ./ext4slower

Exploring Disk I/O Activity with biosnoop

Next off, let’s dive into printing a line per disk I/O each second, with details such as process ID, sector, bytes, and latency among others using biosnoop tool:

sudo ./biosnoop

Analyzing System Cache Performance

Thereafter, we proceed to use cachestat tool to display one line of summarized statistics from the system cache every second. This enables for system tuning operations by pointing out low cache hit ratio and high rate of misses:

sudo ./cachestat

Monitoring TCP Connections in Linux

Monitoring TCP connections every second using tcpconnect tool. Its output includes source and destination address, and port number. This tool is useful for tracing unexpected TCP connections, thereby helping us to identify inefficiencies in application configurations or an attacker.

sudo ./tcpconnect

All the tools above can also be used with various options, to enable the help page for a given tool, make use of the -h option, for example:

sudo ./tcpconnect -h

Trace Failed exec()s Syscalls

To trace failed exec()s syscalls, employ the -x option with opensnoop as below:

sudo ./opensnoop -x

Trace Particular Process Functions

The last example below demonstrates how to execute a custom trace operation. We are tracing a particular process using its PID.

First, determine the process ID:

pidof firefox

Later on, run the custom trace command. The command below -p specifies the process ID, do_sys_open() is a kernel function that is traced dynamically including its second argument as a string.

sudo ./trace -p 4095 'do_sys_open "%s", arg2'

Summary

BCC is a powerful and easy-to-use toolkit for various system administration tasks such as tracing system performance monitoring, tracing block device I/O, TCP functions, file system operations, syscalls, Node.js probes, plus lots more. Importantly, it ships with several example files and man pages for the tools to guide you, making it user friendly and reliable.

Last but not least, you can get back to us by sharing your thoughts about the subject, asking questions, making useful suggestions, or any constructive feedback via the comment section below.