crash - Analyze Linux crash dump data or a live system
]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile
]... [NAMELIST] (live system form)
is a tool for interactively analyzing the state of the Linux system
while it is running, or after a kernel crash has occurred and a core dump has
been created by the netdump, diskdump, LKCD,
facilities. It is loosely based
on the SVR4 UNIX crash command, but has been significantly enhanced by
completely merging it with the gdb(1)
debugger. The marriage of the two
effectively combines the kernel-specific nature of the traditional UNIX crash
utility with the source code level debugging capabilities of gdb(1).
In the dumpfile form,
both a NAMELIST and a MEMORY-IMAGE argument must be
entered. In the live system form,
the NAMELIST argument must be entered
if the kernel's vmlinux
file is not located in a known location, such
as the /usr/lib/debug/lib/modules/<kernel-version>
utility has also been extended to support the analysis of
dumpfiles generated by a crash of the Xen hypervisor. In that case, the
NAMELIST argument must be that of the xen-syms
binary. Live system
analysis is not supported for the Xen hypervisor.
utility command set consists of common kernel core analysis
tools such as kernel stack back traces of all processes, source code
disassembly, formatted kernel structure and variable displays, virtual memory
data, dumps of linked-lists, etc., along with several commands that delve
deeper into specific kernel subsystems. Appropriate gdb
also be entered, which in turn are passed on to the gdb
execution. If desired, commands may be placed in either a
file and/or in a .crashrc
file in the current
directory. During initialization, the commands in $HOME/.crashrc
executed first, followed by those in the ./.crashrc
utility is designed to be independent of Linux version
dependencies. When new kernel source code impacts the correct functionality of
and its command set, the utility will be updated to recognize new
kernel code changes, while maintaining backwards compatibility with earlier
- This is a pathname to an uncompressed kernel image (a
vmlinux file), or a Xen hypervisor image (a xen-syms file)
which has been compiled with the "-g" option. If using the
dumpfile form, a vmlinux file may be compressed in either
gzip or bzip2 formats.
- A kernel core dump file created by the netdump,
diskdump, LKCD kdump, xendump or
If a MEMORY-IMAGE argument is not entered, the session will be invoked on
the live system, which typically requires root privileges because of the
device file used to access system RAM. By default, /dev/crash will
be used if it exists. If it does not exist, then /dev/mem will be
used; but if the kernel has been configured with
CONFIG_STRICT_DEVMEM, then /proc/kcore will be used. It is
permissible to explicitly enter /dev/crash, /dev/mem or
An @ADDRESS value must be appended to the MEMORY-IMAGE if the dumpfile is a
raw RAM dumpfile that has no header information describing the file
contents. Multiple MEMORY-IMAGE@ADDRESS ordered pairs may be entered, with
each dumpfile containing a contiguous block of RAM, where the ADDRESS
value is the physical start address of the block expressed in hexadecimal.
The physical address value(s) will be used to create a temporary ELF
header in /var/tmp, which will only exist during the crash session. If a
raw RAM dumpile represents a live memory source, such as that specified by
the QEMU mem-path argument of a memory-backend-file object, then
"live:" must be prepended to the MEMORY-IMAGE name.
- If the NAMELIST file is not the same kernel that is running
(live system form), or the kernel that was running when the system crashed
(dumpfile form), then the System.map file of the original kernel
should be entered on the command line.
Without an option argument, display a
crash usage help message. If the option argument is a
crash command name, the help page for that command is displayed. If it
is the string "input", a page describing the various crash
command line input options is displayed. If it is the string
"output", a page describing command line output options is
displayed. If it is the string "all", then all of the possible help
messages are displayed. After the help message is displayed, crash
- Silently proceed directly to the "crash>"
prompt without displaying any version, GPL, or crash initialization
data during startup, and by default, runtime command output is not passed
to any scrolling command.
- -i file
- Execute the command(s) contained in file prior to
displaying the "crash>" prompt for interactive user
- -d num
- Set the internal debug level. The higher the number, the
more debugging data will be printed when crash initializes and
- Use /boot/System.map as the
- -e vi | emacs
- Set the readline(3) command line editing mode to
"vi" or "emacs". The default editing mode is
- Force the usage of a compressed vmlinux file if its
original name does not start with "vmlinux".
- Indicate that the NAMELIST file is an LKCD
"Kerntypes" debuginfo file.
- -g [namelist]
- Determine if a vmlinux or xen-syms namelist
file contains debugging data.
- Display the system-crash timestamp and exit.
- Attempt to lock all of its virtual address space into
memory by calling mlockall(MCL_CURRENT|MCL_FUTURE) during initialization.
If the system call fails, an error message will be displayed, but the
- -c tty-device
- Open the tty-device as the console used for debug
- -p page-size
- If a processor's page size cannot be determined by the
dumpfile, and the processor default cannot be used, use
- -o filename
- Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM
dumpfiles, specifies a filename of a new ELF vmcore that will be created
and used as the dumpfile. It will be saved to allow future use as a
standalone vmcore, replacing the original raw RAM dumpfile.
Pass an option and value pair to
machine-dependent code. These architecture-specific option/pairs should only
be required in very rare circumstances:
vm=orig (pre-2.6.11 virtual memory address ranges)
vm=2.6.11 (2.6.11 and later virtual memory address ranges)
vm=xen (Xen kernel virtual memory address ranges)
vm=xen-rhel4 (RHEL4 Xen kernel virtual address ranges)
vm=2.6.14 (4-level page tables)
vm=4l (4-level page tables)
- Automatically load extension modules from a particular
directory. If a directory is specified in the CRASH_EXTENSIONS
shell environment variable, then that directory will be used. Otherwise
/usr/lib64/crash/extensions (64-bit architectures) or
/usr/lib/crash/extensions (32-bit architectures) will be used; if
they do not exist, then the ./extensions directory will be
- Track only the active task on each cpu.
- Display the crash binary's build date, the user ID of the
builder, the hostname of the machine where the build was done, the target
architecture, the version number, and the compiler version.
- --memory_module modname
- Use the modname as an alternative kernel module to
the crash.ko module that creates the /dev/crash device.
- --memory_device device
- Use device as an alternative device to the
/dev/crash, /dev/mem or /proc/kcore devices.
- --log dumpfile
- Dump the contents of the kernel log buffer. A kernel
namelist argument is not necessary, but the dumpfile must contain the
VMCOREINFO data taken from the original /proc/vmcore ELF header.
- Do not use kallsyms-generated symbol information contained
within kernel module object files.
- Do not access or display any kernel module related
- Do not attempt to read configuration data that was built
into kernels configured with CONFIG_IKCONFIG.
- Do not verify the validity of all structure member offsets
and structure sizes that it uses.
- Do not initialize the kernel's slab cache infrastructure,
and commands that use kmem_cache-related data will not work.
- Do not use the registers from the ELF NT_PRSTATUS notes
saved in a compressed kdump header for backtraces.
- Delay the initialization of the kernel's slab cache
infrastructure until it is required by a run-time command.
- Pass this flag to the embedded gdb module, which
will override its two-stage strategy that it uses for reading symbol
tables from the NAMELIST.
- Specify that the system being analyzed is an SMP
Display the version of the crash
utility, the version of the embedded gdb module, GPL information, and
- --cpus number
- Specify the number of cpus in the SMP system being
- --osrelease dumpfile
- Display the OSRELEASE vmcoreinfo string from a kdump
- Force the session to be that of a Xen hypervisor.
- --p2m_mfn pfn
- When a Xen Hypervisor or its dom0 kernel crashes, the
dumpfile is typically analyzed with either the Xen hypervisor or the dom0
kernel. It is also possible to analyze any of the guest domU kernels if
the pfn_to_mfn_list_list pfn value of the guest kernel is passed on
the command line along with its NAMELIST and the dumpfile.
- --xen_phys_start physical-address
- Supply the base physical address of the Xen hypervisor's
text and static data for older xendump dumpfiles that did not pass that
information in the dumpfile header.
- If the makedumpfile(8) facility has filtered a compressed
kdump dumpfile to exclude various types of non-essential pages, or has
marked a compressed or ELF kdump dumpfile as incomplete due to an ENOSPC
or other error during its creation, any attempt to read missing pages will
fail. With this flag, reads from any of those pages will return
- Do not attempt to find the task that was running when the
kernel crashed. Set the initial context to that of the "swapper"
task on cpu 0.
- Use /bin/more as the command output scroller,
overriding the default of /usr/bin/less and any settings in either
./.crashrc or $HOME/.crashrc.
- Use /usr/bin/less as the command output scroller,
overriding any settings in either ./.crashrc or
- Set the default command output radix to 16, overriding the
default radix of 10, and any radix settings in either ./.crashrc or
- Set the default command output radix to 10, overriding any
radix settings in either ./.crashrc or $HOME/.crashrc. This is
the default radix setting.
- Use the output paging command defined in the
CRASHPAGER shell environment variable, overriding any settings in
either ./.crashrc or $HOME/.crashrc.
- Do not pass run-time command output to any scrolling
- Do not strip cloned kernel text symbol names.
- Do not execute the commands in either $HOME/.crashrc
- --mod directory
- When loading the debuginfo data of kernel modules with the
mod -S command, search for their object files in directory
instead of in the standard location.
- --kaslr offset|auto
- If an x86_64 kernel was configured with
CONFIG_RANDOMIZE_BASE, the offset value is equal to the difference
between the symbol values compiled into the vmlinux file and their
relocated KASLR values. If set to auto, the KASLR offset value will be
- --reloc size
- When analyzing live x86 kernels that were configured with a
CONFIG_PHYSICAL_START value that is larger than its
CONFIG_PHYSICAL_ALIGN value, then it will be necessary to enter a
relocation size equal to the difference between the two values.
- --hash count
- Set the number of internal hash queue heads used for list
gathering and verification. The default count is 32768.
- Bring up a session that is restricted to the log, dis,
rd, sym, eval, set and exit commands. This option may provide a
way to extract some minimal/quick information from a corrupted or
truncated dumpfile, or in situations where one of the several kernel
subsystem initialization routines would abort the crash
- --kvmhost [32|64]
- When examining an x86 KVM guest dumpfile, this option
specifies that the KVM host that created the dumpfile was an x86 (32-bit)
or an x86_64 (64-bit) machine, overriding the automatically determined
- --kvmio <size>
- override the automatically-calculated KVM guest I/O hole
- --offline [show|hide]
- Show or hide command output that is related to offline
cpus. The default setting is show.
command generally falls into one of the following categories:
- Symbolic display
- Displays of kernel text/data, which take full advantage of
the power of gdb to format and display data structures
- System state
- The majority of crash commands consist of a set of
"kernel-aware" commands, which delve into various kernel
subsystems on a system-wide or per-task basis.
- Utility functions
- A set of useful helper commands serving various purposes,
some simple, others quite powerful.
- Session control
- Commands that control the crash session itself.
The following alphabetical list consists of a very simple overview of each
command. However, since individual commands often have several
options resulting in significantly different output, it is suggested that the
full description of each command be viewed by executing
crash -h <command>,
or during a
session by simply entering help command.
- "pointer to" is shorthand for either the
struct or union commands. It displays the contents of a
kernel structure or union.
- creates a single-word alias for a command.
- displays an ascii chart or translates a numeric value into
its ascii components.
- displays a task's kernel-stack backtrace. If it is given
the -a option, it displays the stack traces of the active tasks on
all CPUs. It is often used with the foreach command to display the
backtraces of all tasks with one command.
- translates a byte value (physical offset) to its page
- displays data concerning the character and block device
assignments, I/O port usage, I/O memory usage, and PCI device data.
- disassembles memory, either entire kernel functions, from a
location for a specified number of instructions, or from the start of a
function up to a specified memory location.
- evaluates an expression or numeric type and displays the
result in hexadecimal, decimal, octal and binary.
- causes crash to exit.
- dynamically loads or unloads crash shared object
- displays information about open files in a context.
- repeats a specified command for the specified (or all)
tasks in the system.
- displays the tasks using the specified file or socket.
- passes its argument to the embedded gdb module. It
is useful for executing gdb commands that have the same name as
- alone displays the command menu; if followed by a command
name, a full description of a command, its options, and examples are
displayed. Its output is far more complete and useful than this man
- displays data about the System V IPC facilities.
- displays data concerning interrupt request numbers and
bottom-half interrupt handling.
- displays information about the use of kernel memory.
- displays the contents of a linked list.
- displays the kernel log_buf contents in chronological
- displays data specific to the machine type.
- displays information about the currently installed kernel
modules, or adds or deletes symbolic or debugging information about
specified kernel modules.
- displays information about the currently-mounted
- display various network related data.
- passes its arguments to the gdb "print"
command for evaluation and display.
- displays process status for specified, or all, processes in
- translates the hexadecimal contents of a PTE into its
physical page address and page bit settings.
- translates a page frame number to its byte value.
- translates a hexadecimal physical address into a kernel
- is an alias for the "exit" command.
- displays the contents of memory, with the output formatted
in several different manners.
- repeats a command indefinitely, optionally delaying a given
number of seconds between each command execution.
- displays the tasks on the run queue.
- searches a range of user or kernel memory space for given
- either sets a new context, or gets the current context for
- displays signal-handling data of one or more tasks.
- displays either a structure definition or the contents of a
kernel structure at a specified address.
- displays information about each configured swap
- translates a symbol to its virtual address, or a static
kernel virtual address to its symbol -- or to a symbol-plus-offset value,
- displays system-specific data.
- displays the contents of a task_struct.
- displays the contents of a red-black tree or a radix
- displays the timer queue entries, both old- and new-style,
in chronological order.
- is similar to the struct command, except that it
works on kernel unions.
- displays basic virtual memory information of a
- translates a user or kernel virtual address to its physical
- walks the wait queue list displaying the tasks which are
blocked on the specified wait queue.
- displays the definition of structures, unions, typedefs or
- modifies the contents of memory on a live system. It can
only be used if /dev/mem is the device file being used to access
system RAM, and should obviously be used with great care.
is invoked with a Xen hypervisor binary as the NAMELIST, the
command set is slightly modified. The *, alias, ascii, bt, dis, eval, exit,
extend, gdb, help, list, log, p, pte, rd, repeat, search, set,
struct, sym, sys, union, whatis, wr
commands are the
same as above. The following commands are specific to the Xen hypervisor:
- displays the contents of the domain structure for selected,
or all, domains.
- displays domain status for selected, or all, domains.
- displays Xen dump information for selected, or all,
- displays physical cpu information for selected, or all,
- displays vcpu status for selected, or all, vcpus.
- Initialization commands. The file can be located in the
user's HOME directory and/or the current directory. Commands found
in the .crashrc file in the HOME directory are executed
before those in the current directory's .crashrc file.
- Command input is read using readline(3). If
EDITOR is set to emacs or vi then suitable
keybindings are used. If EDITOR is not set, then vi is used.
This can be overridden by set vi or set emacs commands
located in a .crashrc file, or by entering -e emacs on the
crash command line.
- If CRASHPAGER is set, its value is used as the name
of the program to which command output will be sent. If not, then command
output is sent to /usr/bin/less -E -X by default.
- Specifies an alternative directory tree to search for
kernel module object files.
- Specifies a directory containing extension modules that
will be loaded automatically if the -x command line option is
does not work, look for a newer version: kernel evolution
frequently makes crash
The command set scroll off
will cause output to be sent directly to the
terminal rather than through a paging program. This is useful, for example, if
you are running crash
in a window of emacs
Dave Anderson <email@example.com> wrote crash.
- Jay Fenlason <firstname.lastname@example.org> and Dave Anderson
<email@example.com> wrote this man page.
command within crash
provides more complete and accurate
documentation than this man page.
- the home page of the crash