offwaketime - Summarize blocked time by off-CPU stack + waker stack. Uses Linux
offwaketime [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-f]
[--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M
This program shows kernel stack traces and task names that were blocked and
"off-CPU", along with the stack traces and task names for the
threads that woke them, and the total elapsed time from when they blocked to
when they were woken up. This combines the summaries from both the offcputime
and wakeuptime tools. The time measurement will be very similar to off-CPU
time, however, off-CPU time may include a little extra time spent waiting on a
run queue to be scheduled. The combined stacks, task names, and total time is
summarized in kernel context for efficiency, using an eBPF map.
The output summary will further help you identify reasons why threads were
blocking, and quantify the time from when they were blocked to woken up. This
spans all types of blocking activity: disk I/O, network I/O, locks, page
faults, swapping, sleeping, involuntary context switches, etc.
This is complementary to CPU profiling (e.g., CPU flame graphs) which shows the
time spent on-CPU. This shows the time spent blocked off-CPU, and the output,
especially the -f format, can be used to generate an "off-wake time flame
CONFIG_BPF and bcc.
- Print usage message.
- Print output in folded stack format.
- -p PID
- Trace this process ID only (filtered in-kernel).
- -t TID
- Trace this thread ID only (filtered in-kernel).
- Only trace user threads (no kernel threads).
- Only trace kernel threads (no user threads).
- Show stacks from user space only (no kernel space
- Show stacks from kernel space only (no user space
- --stack-storage-size STACK_STORAGE_SIZE
- Change the number of unique stack traces that can be stored
- Duration to trace, in seconds.
- -m MIN_BLOCK_TIME
- The amount of time in microseconds over which we store
traces (default 1)
- -M MAX_BLOCK_TIME
- The amount of time in microseconds under which we store
traces (default U64_MAX)
- Trace all thread blocking events, and summarize (in-kernel)
by user and kernel off-CPU stack trace, waker stack traces, task names, and
total blocked time:
- # offwaketime
- Trace for 5 seconds only:
- # offwaketime 5
- Trace for 5 seconds, and emit output in folded stack format
(suitable for flame graphs), user-mode threads only:
- # offwaketime -fu 5
- Trace PID 185 only:
- # offwaketime -p 185
This summarizes unique stack trace pairs in-kernel for efficiency, allowing it
to trace a higher rate of events than methods that post-process in user space.
The stack trace and time data is only copied to user space once, when the
output is printed. While these techniques greatly lower overhead, scheduler
events are still a high frequency event, as they can exceed 1 million events
per second, and so caution should still be used. Test before production use.
If the overhead is still a problem, take a look at the min block option. If your
aim is to chase down longer blocking events, then this could be increased to
filter shorter blocking events, further lowering overhead.
This is from bcc.
Also look in the bcc distribution for a companion _examples.txt file containing
example usage, output, and commentary for this tool.
Unstable - in development.