Installing BCC on Ubuntu for Troubleshooting Full-Stack Performance

Installing BCC for Troubleshooting Full-Stack Performance


Software Infrastructure Observability allows Data Ops. Engineers, SREs and other stakeholders in the ecosystem to easily identify components that need to be tuned proactively for optimal performance. Faster transactions accelerate our customer technology infrastructure. BPF Compiler Collection (BCC) is a collection of tools for Linux Kernel tracing which makes use of extended BPF (Berkeley Packet Filters) with kernel instrumentation in C (and includes a C wrapper around LLVM), and front-ends in Python and lua. Technically, eBPF does to the kernel what JavaScript does to websites: it allows all sorts of new applications to be created. BPF can be used for ultra low-level observability of Linux and all the related software infrastructure. BPF is capable of running a new type of user-defined and kernel-mode applications.

Configuring Linux Kernel: Pre-requisites

We strongly recommend a Linux kernel version 4.1 or newer for optimal operations/performance of BCC. Please also get kernel compiled with the following flags set:

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# [optional, for tc filters]
CONFIG_NET_CLS_BPF=m
# [optional, for tc actions]
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
# [for Linux kernel versions 4.1 through 4.6]
CONFIG_HAVE_BPF_JIT=y
# [for Linux kernel versions 4.7 and later]
CONFIG_HAVE_EBPF_JIT=y
# [optional, for kprobes]
CONFIG_BPF_EVENTS=y
# Need kernel headers through /sys/kernel/kheaders.tar.xz
CONFIG_IKHEADERS=y

Optional kernel flags needed for running bcc networking examples on vanilla kernel:

CONFIG_NET_SCH_SFQ=m
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_DUMMY=m
CONFIG_VXLAN=m

Note: Kernel compile flags can usually be checked by looking at /proc/config.gz or /boot/config-<kernel-version>.

iovisor packages (Upstream Stable and Signed Packages)

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)

Note: (replace xenial with artful or bionic as appropriate). Tools will be installed under /usr/share/bcc/tools.

Upstream Nightly Packages

echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)

Note: (replace xenial with artful or bionic as appropriate)

BCC installation from source on Debian

Repositories

/etc/apt/sources.list should include the non-free repository and look something like this:

deb http://deb.debian.org/debian sid main contrib non-free
deb-src http://deb.debian.org/debian sid main contrib non-free

Install Build Dependencies

# Before you begin
apt-get update
# According to https://packages.debian.org/source/sid/bpfcc,
# BCC build dependencies:
sudo apt-get install arping bison clang-format cmake dh-python \
  dpkg-dev pkg-kde-tools ethtool flex inetutils-ping iperf \
  libbpf-dev libclang-dev libclang-cpp-dev libedit-dev libelf-dev \
  libfl-dev libzip-dev linux-libc-dev llvm-dev libluajit-5.1-dev \
  luajit python3-netaddr python3-pyroute2 python3-distutils python3

Install and compile BCC

git clone https://github.com/iovisor/bcc.git
mkdir bcc/build; cd bcc/build
cmake ..
make
sudo make install

Install BCC from source on Ubuntu

Prerequisite

To build the BCC from source, you need following:

  • LLVM 3.7.1 or newer, compiled with BPF support (default=on)
  • Clang, built from the same tree as LLVM
  • cmake (>=3.1), gcc (>=4.7), flex, bison
  • LuaJIT, if you want Lua support

Install build dependencies

# Trusty (14.04 LTS) and older
VER=trusty
echo "deb http://llvm.org/apt/$VER/ llvm-toolchain-$VER-3.7 main
deb-src http://llvm.org/apt/$VER/ llvm-toolchain-$VER-3.7 main" | \
  sudo tee /etc/apt/sources.list.d/llvm.list
wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key | sudo apt-key add -
sudo apt-get update

# For Bionic (18.04 LTS)
sudo apt-get -y install bison build-essential cmake flex git libedit-dev \
  libllvm6.0 llvm-6.0-dev libclang-6.0-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils

# For Eoan (19.10) or Focal (20.04.1 LTS)
sudo apt install -y bison build-essential cmake flex git libedit-dev \
  libllvm7 llvm-7-dev libclang-7-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils
  
# For Hirsute (21.04)  or Impish (21.10)
sudo apt install -y bison build-essential cmake flex git libedit-dev   libllvm11 llvm-11-dev libclang-11-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils

# For other versions
sudo apt-get -y install bison build-essential cmake flex git libedit-dev \
  libllvm3.7 llvm-3.7-dev libclang-3.7-dev python zlib1g-dev libelf-dev python3-distutils

# For Lua support
sudo apt-get -y install luajit luajit-5.1-dev

Install and compile BCC

# git clone https://github.com/iovisor/bcc.git
# mkdir bcc/build; cd bcc/build
# cmake ..
# make
# sudo make install
# cmake -DPYTHON_CMD=python3 .. # build python3 binding
# pushd src/python/
# make
# sudo make install
# popd

☛ BCC Tools we use for Performance Troubleshooting

 

BCC Performance Matrix ToolDescriptionBeneficiary
execsnoop execsnoop traces process execution patterns, Reporting PID, PPID, and other argument details if possible.CPU usage performance matrix
runqlatTo measures the time a task spends waiting on a run queue (or equivalent scheduler data structure) for a turn on-CPU, and shows this time as a histogram.CPU usage performance matrix
runqlenAggregate and report scheduler queue length as a histogram. Like, The processes which are bound to CPUs causing queueing, or interrupt mappings causing the same.CPU usage performance matrix
opensnoop showing which processes are attempting to open which filesCPU usage performance matrix
memleakmemleak traces and matches memory allocation and deallocation requests, and collects call stacks for each allocation.Memory/RAM usage performance matrix
bitesizeShow I/O distribution for requested block sizes, by process name.Disk Ops performance matrix
biotopThis traces block device I/O (disk I/O) and prints a per-process summary every interval (by default, 1 second).Disk Ops performance matrix
biosnoopThis tool traces block device I/O (disk I/O) and prints a one-line summary for each I/O showing various details.Disk Ops performance matrix
biolatencybiolatency traces block device I/O (disk I/O), and records the distribution of I/O latency (time).Disk Ops performance matrix

☛ A partial list of customers – What do we do for them?

  • Applied Materials – ClickHouse Consultative Support
  • Orange Communications – ClickHouse Consultative Support
  • Garmin – ClickHouse Consulting and Enterprise-Class Support
  • ClassPlus – ClickHouse Enterprise-Class Support
  • Morgan Stanley – ClickHouse Enterprise-Class Support
  • Blue Dart – ClickHouse Consulting / Professional Services and Enterprise-Class Consultative Support
  • Carlsberg – ClickHouse Enterprise-Class Support
  • PRADA – ClickHouse Consulting and Managed Database Services
  • Netflix – ClickHouse Enterprise-Class Support
  • MPL – ClickHouse Enterprise-Class Support
  • Burberry – ClickHouse Enterprise-Class Support
  • Edward Jones – ClickHouse Consulting and Enterprise-Class Support
  • Cambridge Investment Research – ClickHouse Consulting and Enterprise-Class Support
  • National Geographic – ClickHouse Consulting and Enterprise-Class Support
  • American Express Travel – ClickHouse Consulting and Enterprise-Class Support
  • Sony – ClickHouse Consultative Support and Managed Services
  • Nintendo – ClickHouse Consultative Support and Managed Services
  • Unilever – ClickHouse Consultative Support
  • VISA – ClickHouse Consultative Support and Database Architect Services for Big Data Analytics

About ChistaDATA Inc. 10 Articles
We are an full-stack ClickHouse infrastructure operations Consulting, Support and Managed Services provider with core expertise in performance, scalability and data SRE. Based out of California, Our consulting and support engineering team operates out of San Francisco, Vancouver, London, Germany, Russia, Ukraine, Australia, Singapore and India to deliver 24*7 enterprise-class consultative support and managed services. We operate very closely with some of the largest and planet-scale internet properties like PayPal, Garmin, Honda cars IoT project, Viacom, National Geographic, Nike, Morgan Stanley, American Express Travel, VISA, Netflix, PRADA, Blue Dart, Carlsberg, Sony, Unilever etc