funclatency - Time functions and print latency as a histogram.
funclatency [-h] [-p PID] [-i INTERVAL] [-T] [-u] [-m] [-F] [-r] [-v]
This tool traces function calls and times their duration (latency), and shows
the latency distribution as a histogram. The time is measured from when the
function is called to when it returns, and is inclusive of both on-CPU time
and time spent blocked.
This tool uses in-kernel eBPF maps for storing timestamps and the histogram, for
Currently nested or recursive functions are not supported properly, and
timestamps will be overwritten, creating dubious output. Try to match single
functions, or groups of functions that run at the same stack layer, and don't
ultimately call each other.
WARNING: This uses dynamic tracing of (what can be many) functions, an activity
that has had issues on some kernel versions (risk of panics or freezes). Test,
and know what you are doing, before use.
Since this uses BPF, only the root user can use this tool.
CONFIG_BPF and bcc.
pattern Function name or search pattern. Supports "*" wildcards. See
EXAMPLES. You can also use -r for regular expressions. -h Print usage message.
- -p PID
- Trace this process ID only.
- -i INTERVAL
- Print output every interval seconds.
- Include timestamps on output.
- Output histogram in microseconds.
- Output histogram in milliseconds.
- Print a separate histogram per function matched.
- Use regular expressions for the search pattern.
- Print the BPF program (for debugging purposes).
- Time the do_sys_open() kernel function, and print the
distribution as a histogram:
- # funclatency do_sys_open
- Time the read() function in libc across all processes on
- # funclatency c:read
- Time vfs_read(), and print the histogram in units of
- # funclatency -u vfs_read
- Time do_nanosleep(), and print the histogram in units of
- # funclatency -m do_nanosleep
- Time vfs_read(), and print output every 5 seconds, with
- # funclatency -mTi 5 vfs_read
- Time vfs_read() for process ID 181 only:
- # funclatency -p 181 vfs_read:
- Time both vfs_fstat() and vfs_fstatat() calls, by use of a
- # funclatency 'vfs_fstat*'
- Time both vfs_fstat* calls, and print a separate histogram
- # funclatency -F 'vfs_fstat*'
- Nanosecond range
- Microsecond range
- Millisecond range
- How many calls fell into this range
- An ASCII bar chart to visualize the distribution (count
This traces kernel functions and maintains in-kernel timestamps and a histogram,
which are asynchronously copied to user-space. While this method is very
efficient, the rate of kernel functions can also be very high (>1M/sec), at
which point the overhead is expected to be measurable. Measure in a test
environment and understand overheads before use. You can also use funccount to
measure the rate of kernel functions over a short duration, to set some
expectations before use.
This is from bcc.
Also look in the bcc distribution for a companion _examples.txt file containing
example usage, output, and commentary for this tool.
Unstable - in development.
Brendan Gregg, Sasha Goldshtein