Introduction
Software Infrastructure Observability allows Data Ops. Engineers, SREs and other stakeholders in the ecosystem to easily identify components that need to be tuned proactively for optimal performance. Faster transactions accelerate our customer technology infrastructure. BPF Compiler Collection (BCC) is a collection of tools for Linux Kernel tracing which makes use of extended BPF (Berkeley Packet Filters) with kernel instrumentation in C (and includes a C wrapper around LLVM), and front-ends in Python and lua. Technically, eBPF does to the kernel what JavaScript does to websites: it allows all sorts of new applications to be created. BPF can be used for ultra low-level observability of Linux and all the related software infrastructure. BPF is capable of running a new type of user-defined and kernel-mode applications.
Configuring Linux Kernel: Pre-requisites
We strongly recommend a Linux kernel version 4.1 or newer for optimal operations/performance of BCC. Please also get kernel compiled with the following flags set:
CONFIG_BPF=y CONFIG_BPF_SYSCALL=y # [optional, for tc filters] CONFIG_NET_CLS_BPF=m # [optional, for tc actions] CONFIG_NET_ACT_BPF=m CONFIG_BPF_JIT=y # [for Linux kernel versions 4.1 through 4.6] CONFIG_HAVE_BPF_JIT=y # [for Linux kernel versions 4.7 and later] CONFIG_HAVE_EBPF_JIT=y # [optional, for kprobes] CONFIG_BPF_EVENTS=y # Need kernel headers through /sys/kernel/kheaders.tar.xz CONFIG_IKHEADERS=y
Optional kernel flags needed for running bcc networking examples on vanilla kernel:
CONFIG_NET_SCH_SFQ=m CONFIG_NET_ACT_POLICE=m CONFIG_NET_ACT_GACT=m CONFIG_DUMMY=m CONFIG_VXLAN=m
Note: Kernel compile flags can usually be checked by looking at /proc/config.gz or /boot/config-<kernel-version>.
iovisor packages (Upstream Stable and Signed Packages)
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/iovisor.list sudo apt-get update sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)
Note: (replace xenial with artful or bionic as appropriate). Tools will be installed under /usr/share/bcc/tools.
Upstream Nightly Packages
echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list sudo apt-get update sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)
Note: (replace xenial with artful or bionic as appropriate)
BCC installation from source on Debian
Repositories
/etc/apt/sources.list should include the non-free repository and look something like this:
deb http://deb.debian.org/debian sid main contrib non-free deb-src http://deb.debian.org/debian sid main contrib non-free
Install Build Dependencies
# Before you begin apt-get update # According to https://packages.debian.org/source/sid/bpfcc, # BCC build dependencies: sudo apt-get install arping bison clang-format cmake dh-python \ dpkg-dev pkg-kde-tools ethtool flex inetutils-ping iperf \ libbpf-dev libclang-dev libclang-cpp-dev libedit-dev libelf-dev \ libfl-dev libzip-dev linux-libc-dev llvm-dev libluajit-5.1-dev \ luajit python3-netaddr python3-pyroute2 python3-distutils python3
Install and compile BCC
git clone https://github.com/iovisor/bcc.git mkdir bcc/build; cd bcc/build cmake .. make sudo make install
Install BCC from source on Ubuntu
Prerequisite
To build the BCC from source, you need following:
- LLVM 3.7.1 or newer, compiled with BPF support (default=on)
- Clang, built from the same tree as LLVM
- cmake (>=3.1), gcc (>=4.7), flex, bison
- LuaJIT, if you want Lua support
Install build dependencies
# Trusty (14.04 LTS) and older VER=trusty echo "deb http://llvm.org/apt/$VER/ llvm-toolchain-$VER-3.7 main deb-src http://llvm.org/apt/$VER/ llvm-toolchain-$VER-3.7 main" | \ sudo tee /etc/apt/sources.list.d/llvm.list wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key | sudo apt-key add - sudo apt-get update # For Bionic (18.04 LTS) sudo apt-get -y install bison build-essential cmake flex git libedit-dev \ libllvm6.0 llvm-6.0-dev libclang-6.0-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils # For Eoan (19.10) or Focal (20.04.1 LTS) sudo apt install -y bison build-essential cmake flex git libedit-dev \ libllvm7 llvm-7-dev libclang-7-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils # For Hirsute (21.04) or Impish (21.10) sudo apt install -y bison build-essential cmake flex git libedit-dev libllvm11 llvm-11-dev libclang-11-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils # For other versions sudo apt-get -y install bison build-essential cmake flex git libedit-dev \ libllvm3.7 llvm-3.7-dev libclang-3.7-dev python zlib1g-dev libelf-dev python3-distutils # For Lua support sudo apt-get -y install luajit luajit-5.1-dev
Install and compile BCC
# git clone https://github.com/iovisor/bcc.git # mkdir bcc/build; cd bcc/build # cmake .. # make # sudo make install # cmake -DPYTHON_CMD=python3 .. # build python3 binding # pushd src/python/ # make # sudo make install # popd
BCC Tools we use for Performance Troubleshooting
| BCC Performance Matrix Tool | Description | Beneficiary |
|---|---|---|
| execsnoop | execsnoop traces process execution patterns, Reporting PID, PPID, and other argument details if possible. | CPU usage performance matrix |
| runqlat | To measures the time a task spends waiting on a run queue (or equivalent scheduler data structure) for a turn on-CPU, and shows this time as a histogram. | CPU usage performance matrix |
| runqlen | Aggregate and report scheduler queue length as a histogram. Like, The processes which are bound to CPUs causing queueing, or interrupt mappings causing the same. | CPU usage performance matrix |
| opensnoop | showing which processes are attempting to open which files | CPU usage performance matrix |
| memleak | memleak traces and matches memory allocation and deallocation requests, and collects call stacks for each allocation. | Memory/RAM usage performance matrix |
| bitesize | Show I/O distribution for requested block sizes, by process name. | Disk Ops performance matrix |
| biotop | This traces block device I/O (disk I/O) and prints a per-process summary every interval (by default, 1 second). | Disk Ops performance matrix |
| biosnoop | This tool traces block device I/O (disk I/O) and prints a one-line summary for each I/O showing various details. | Disk Ops performance matrix |
| biolatency | biolatency traces block device I/O (disk I/O), and records the distribution of I/O latency (time). | Disk Ops performance matrix |
ChistaDATA – Your ClickHouse Partner
A partial list of our customers
- Applied Materials – ClickHouse Consultative Support
- Orange Communications – ClickHouse Consultative Support
- Garmin – ClickHouse Consulting and Enterprise-Class Support
- ClassPlus – ClickHouse Enterprise-Class Support
- Morgan Stanley – ClickHouse Enterprise-Class Support
- Blue Dart – ClickHouse Consulting / Professional Services and Enterprise-Class Consultative Support
- Carlsberg – ClickHouse Enterprise-Class Support
- PRADA – ClickHouse Consulting and Managed Database Services
- Netflix – ClickHouse Enterprise-Class Support
- MPL – ClickHouse Enterprise-Class Support
- Burberry – ClickHouse Enterprise-Class Support
- Edward Jones – ClickHouse Consulting and Enterprise-Class Support
- Cambridge Investment Research – ClickHouse Consulting and Enterprise-Class Support
- National Geographic – ClickHouse Consulting and Enterprise-Class Support
- American Express Travel – ClickHouse Consulting and Enterprise-Class Support
- Sony – ClickHouse Consultative Support and Managed Services
- Nintendo – ClickHouse Consultative Support and Managed Services
- Unilever – ClickHouse Consultative Support
- VISA – ClickHouse Consultative Support and Database Architect Services for Big Data Analytics
Conclusion
In this guide we have walked through the detailed steps for the installation of the BPF Compiler Collection (BCC) on Ubuntu, and how we might use it to troubleshoot ClickHouse performance.
To read more on eBPF and ClickHouse, do consider reading the following articles
