argdist - Trace a function and display a histogram or frequency count of its
parameter values. Uses Linux eBPF/bcc.
argdist [-h] [-p PID] [-z STRING_SIZE] [-i INTERVAL] [-n COUNT] [-v] [-T TOP]
[-H specifier] [-C specifier] [-I header]
argdist attaches to function entry and exit points, collects specified parameter
values, and stores them in a histogram or a frequency collection that counts
the number of times a parameter value occurred. It can also filter parameter
values and instrument multiple entry points at once.
Since this uses BPF, only the root user can use this tool.
CONFIG_BPF and bcc.
- Print usage message.
- -p PID
- Trace only functions in the process PID.
- -z STRING_SIZE
- When collecting string arguments (of type char*), collect
up to STRING_SIZE characters. Longer strings will be truncated.
- -i INTERVAL
- Print the collected data every INTERVAL seconds. The
default is 1 second.
- -n NUMBER
- Print the collected data COUNT times and then exit.
- Display the generated BPF program, for debugging
- -T TOP
- When collecting frequency counts, display only the top TOP
- -H specifiers, -C specifiers
- One or more probe specifications that instruct argdist
which functions to probe, which parameters to collect, how to aggregate
them, and whether to perform any filtering. See SPECIFIER SYNTAX
- -I header
- One or more header files that should be included in the BPF
program. This enables the use of structure definitions, enumerations, and
constants that are available in these headers. You should provide the same
path you would include in the BPF program, e.g. 'linux/blkdev.h' or
'linux/time.h'. Note: in many cases, argdist will deduce the necessary
header files automatically.
The general specifier syntax is as follows:
- Probe type - "p" for function entry,
"r" for function return, "t" for kernel tracepoint,
"u" for USDT probe; -H for histogram collection, -C for
frequency count. Indicates where to place the probe and whether the probe
should collect frequency count information, or aggregate the collected
values into a histogram. Counting probes will collect the number of times
every parameter value was observed, whereas histogram probes will collect
the parameter values into a histogram. Only integral types can be used
with histogram probes; there is no such limitation for counting
- Library containing the probe. Specify the full path to the
.so or executable file where the function to probe resides. Alternatively,
you can specify just the lib name: for example, "c" refers to
libc. If no library name is specified, the kernel is assumed.
- The category of the kernel tracepoint. For example: net,
- The function to probe, and its signature. The function name
must match exactly for the probe to be placed. The signature, on the other
hand, is only required if you plan to collect parameter values based on
that signature. For example, if you only want to collect the first
parameter, you don't have to specify the rest of the parameters in the
signature. When capturing kernel tracepoints, this should be the name of
the event, e.g. net_dev_start_xmit. The signature for kernel tracepoints
should be empty. When capturing USDT probes, this should be the name of
the probe, e.g. reloc_complete. The signature for USDT probes should be
- The type(s) of the expression(s) to capture. This is the
type of the keys in the histogram or raw event collection that are
collected by the probes.
- The expression(s) to capture. These are the values that are
assigned to the histogram or raw event collection. You may use the
parameters directly, or valid C expressions that involve the parameters,
such as "size % 10". Tracepoints may access a special structure
called "args" that is formatted according to the tracepoint
format (which you can obtain using tplist). For example, the
block:block_rq_complete tracepoint can access args->nr_sector. USDT
probes may access the arguments defined by the tracing program in the
special arg1, arg2, ... variables. To obtain their types, use the tplist
tool. Return probes can use the argument values received by the function
when it was entered, through the $entry(paramname) special variable.
Return probes can also access the function's return value in $retval, and
the function's execution time in nanoseconds in $latency. Note that adding
the $latency or $entry(paramname) variables to the expression will
introduce an additional probe at the function's entry to collect this
data, and therefore introduce additional overhead.
- The filter applied to the captured data. Only parameter
values that pass the filter will be collected. This is any valid C
expression that refers to the parameter values, such as "fd == 1
&& length > 16". The $entry, $retval, and $latency
variables can be used here as well, in return probes. The filter
expression may also use the STRCMP pseudo-function to compare a predefined
string to a string argument. For example: STRCMP("test.txt",
file). The order of arguments is important: the first argument MUST be a
quoted literal string, and the second argument can be a runtime
- The label that will be displayed when printing the probed
values. By default, this is the probe specifier.
- Print a histogram of allocation sizes passed to
- # argdist -H 'p::__kmalloc(u64 size):u64:size'
- Print a count of how many times process 1005 called malloc
with an allocation size of 16 bytes:
- # argdist -p 1005 -C 'p:c:malloc(size_t
- Snoop on all strings returned by gets():
- # argdist -C 'r:c:gets():char*:$retval'
- Print a histogram of read sizes that were longer than
- # argdist -H 'r::__vfs_read(void *file, void *buf,
size_t count):size_t:$entry(count):$latency > 1000000'
- Print frequency counts of how many times writes were issued
to a particular file descriptor number, in process 1005:
- # argdist -p 1005 -C 'p:c:write(int fd):int:fd'
- Print a histogram of error codes returned by read() in
- # argdist -p 1005 -H 'r:c:read()'
- Print a histogram of buffer sizes passed to write() across
all processes, where the file descriptor was 1 (STDOUT):
- # argdist -H 'p:c:write(int fd, const void *buf, size_t
- Count fork() calls in libc across all processes, grouped by
- # argdist -C 'p:c:fork():int:$PID;fork per
- Print histogram of number of sectors in completing block
- # argdist -H
- Aggregate interrupts by interrupt request (IRQ):
- # argdist -C
- Print the functions used as thread entry points and how
common they are:
- # argdist -C 'u:pthread:pthread_start():u64:arg2' -p
- Print histograms of sleep() and nanosleep() parameter
- # argdist -H 'p:c:sleep(u32 seconds):u32:seconds' -H
'p:c:nanosleep(struct timespec *req):long:req->tv_nsec'
- Spy on writes to STDOUT performed by process 2780, up to a
string size of 120 characters:
- # argdist -p 2780 -z 120 -C 'p:c:write(int fd, char*
buf, size_t len):char*:buf:fd==1'
- Group files being read from and the read sizes from
- # argdist -C 'p::__vfs_read(struct file *file, void
This is from bcc.
Also look in the bcc distribution for a companion _examples.txt file containing
example usage, output, and commentary for this tool.
Unstable - in development.