Man pages sections > man8 > uprobe-perf

uprobe - trace a given uprobe definition. User-level dynamic tracing. Uses Linux

uprobe(8) System Manager's Manual uprobe(8)


uprobe - trace a given uprobe definition. User-level dynamic tracing. Uses Linux ftrace. EXPERIMENTAL.


uprobe [-FhHsv] [-d secs] [-p PID] [-L TID] {-l target | uprobe_definition [filter]}


This will create, trace, then destroy a given uprobe definition. See Documentation/trace/uprobetracer.txt in the Linux kernel source for the syntax of a uprobe definition, and "uprobe -h" for examples. With this tool, the probe alias is optional (it will default to something meaningful).
WARNING: This uses dynamic tracing of user-level functions, using some relatively new kernel code. I have seen this cause target processes to fail, either entering endless spin loops or crashing on illegal instructions. I believe newer kernels (post 4.0) are relatively safer, but use caution. Test in a lab environment, and know what you are doing, before use. Also consider other (more developed) user-level tracers (perf_events, LTTng, etc.).
Use extreme caution with the raw address mode: eg, "p:libc:0xbf130". uprobe does not check for instruction alignment, so tracing the wrong address (eg, mid-way through a multi-byte instruction) will corrupt the target's memory. Other tracers (eg, perf_events with debuginfo) check alignment.
Also beware of widespread tracing that interferes with the operation of the system, eg, tracing libc:malloc, which by-default will trace _all_ processes.
I wrote this because I kept testing different custom uprobes at the command line, and wanted a way to automate the steps. For generic user-level tracing, use perf_events directly.
Since this uses ftrace, only the root user can use this tool.


REQUIREMENTS: FTRACE and UPROBE CONFIG, which you may already have on recent kernel versions, file(1), ldconfig(8), objdump(1), and some version of awk. Also, currently only executes on Linux 4.0+ (see WARNING) unless -F is used.


Force. Trace despite kernel version warnings. Use on older kernels may expose you to (since fixed) bugs, which can lock up or crash target processes, which could also lock up the entire system. Test in a lab environment before use, and consider other more developed user-level tracers (perf_events, LTTng, etc.).
-d seconds
Set the duration of tracing, in seconds. Trace output will be buffered and printed at the end. This also reduces overheads by buffering in-kernel, instead of printing events as they occur.
The ftrace buffer has a fixed size per-CPU (see /sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing, try increasing that size.
Print usage message.
Print column headers.
Print user-level stack traces after each event. These are currently printed in hex, and need post-processing to see user-level symbols (eg, addr2line; I should automate that).
Show the uprobe format file only (do not trace), identifying possible variables for use in a custom filter.
-p PID
Only trace user-level functions when this process ID is on-CPU.
Only trace user-level functions when this thread ID is on-CPU.
A full uprobe definition, as documented by Documentation/trace/uprobetracer.txt in the Linux kernel source. Note that the probe alias name is optional with uprobe(8), and if not specified, it will default to something meaningful. See the EXAMPLES section.
An ftrace filter definition.


These examples may need modification to match your target software function names and platform's register usage. If using platform specific registers becomes too painful in practice, consider a debuginfo-based tracer, which can trace variables names instead (eg, perf_events).
trace readline() calls in all running "bash" executables:
# uprobe p:bash:readline
trace readline() with explicit executable path:
# uprobe p:/bin/bash:readline
trace the return of readline() with return value as a string:
# uprobe 'r:bash:readline +0($retval):string'
trace sleep() calls in all running libc shared libraries:
# uprobe p:libc:sleep
trace sleep() with register %di (x86):
# uprobe 'p:libc:sleep %di'
trace this address (use caution: must be instruction aligned):
# uprobe p:libc:0xbf130
trace gettimeofday() for PID 1182 only:
# uprobe -p 1182 p:libc:gettimeofday
trace the return of fopen() only when it returns NULL:
# uprobe 'r:libc:fopen file=$retval' 'file == 0'


The output format depends on the kernel version, and headings can be printed using -H. The format is the same as the ftrace function trace format, described in the kernel source under Documentation/trace/ftrace.txt.
Typical fields are:
The process name (which could include dashes), a dash, and the process ID.
The CPU ID, in brackets.
Kernel state flags. For example, on Linux 3.16 these are for irqs-off, need-resched, hardirq/softirq, and preempt-depth.
Time of event, in seconds.
User-level function name.


This can generate a lot of trace data quickly, depending on the frequency of the traced events. Such data will cause performance overheads. This also works without buffering by default, printing function events as they happen (uses trace_pipe), context switching and consuming CPU to do so. If needed, you can try the "-d secs" option, which buffers events instead, reducing overhead. If you think the buffer option is losing events, try increasing the buffer size (buffer_size_kb).
If you find a use for uprobe(8) where the overhead is prohibitive, consider the same enabling using perf_events where overhead should be reduced.


This is from the perf-tools collection:
Also look under the examples directory for a text file containing example usage, output, and commentary for this tool.




Unstable - in development.


Brendan Gregg


2014-07-20 USER COMMANDS