criu - checkpoint/restore in userspace
is a tool for checkpointing and restoring running applications. It
does this by saving their state as a collection of files (see the dump
command) and creating equivalent processes from those files (see the
command). The restore operation can be performed at a later
time, on a different system, or both.
Most of the true / false long options (the ones without arguments) can be
prefixed with --no-
to negate the option (example:
Common options are applicable to any command
Increase verbosity up from the default level.
Multiple v can be used, each increasing verbosity by one level. Using
long option without argument increases verbosity by one level.
Set verbosity level to num
. The higher
the level, the more output is produced.
The following levels are available:
•-v0 no output;
•-v1 only errors;
•-v2 above plus warnings (this
is the default level);
•-v3 above plus information
messages and timestamps;
•-v4 above plus lots of
Write root task, service or page-server pid
into a file.
, --log-file file
Write logging messages to file.
Write separate logging files per each
During dump as well as during restore
criu collects information like the time required to dump or restore the
process or the number of pages dumped or restored. This information is always
written to the files stats-dump and stats-restore and can be
easily displayed using crit. The option --display-stats
additionally prints out this information on the console at the end of a dump
or a restore.
, --images-dir path
Use path as a base directory where to
look for sets of image files.
Use path as a parent directory where to
look for sets of image files. This option makes sense in case of incremental
, --work-dir dir
Use directory dir for putting logs,
pidfiles and statistics. If not specified, path from -D option
Close file descriptor fd before
performing any actions.
, --libdir path
Path to plugins directory.
Add an external action script to be executed
at certain stages. The environment variable CRTOOLS_SCRIPT_ACTION
available to the script to find out which action is being executed, and its
value can be one of the following:
run prior to beginning a dump
run upon dump completion
run prior to beginning a restore
run when all processes and resources are
restored but tasks are stopped waiting for final kick to run. Must not
run upon restore completion
run to lock network in a target network
run to unlock network in a target network
run once root task just been created with
required namespaces. Note it is an early stage of restore, when nothing is
restored yet except for namespaces themselves
Print program version and exit.
Print some help and exit.
Performs the pre-dump procedure, during which criu
creates a snapshot of
memory changes since the previous pre-dump
. Note that during this
also creates the fsnotify cache which speeds up the restore
requires at least -t
option (see dump
below). In addition, page-server
options may be specified.
Turn on memory changes tracker in the kernel.
If the option is not passed the memory tracker get turned on implicitly.
Performs a checkpoint procedure.
, --tree pid
Checkpoint the whole process tree starting
Leave tasks in running state after checkpoint,
instead of killing. This option is pretty dangerous and should be used only if
you understand what you are doing.
Note if task is about to run after been checkpointed, it can modify TCP
connections, delete files and do other dangerous actions. Therefore,
can not guarantee that the next restore
succeed. Most likely if this option is used, at least the file system snapshot
must be made with the help of post-dump
In other words, do not use it unless really needed.
Leave tasks in stopped state after checkpoint,
instead of killing.
Dump an instance of an external resource. The
generic syntax is type of resource, followed by resource id
(enclosed in literal square brackets), and optional value (prepended by
a literal semicolon). The following resource types are currently supported:
mnt, dev, file, tty, unix. Syntax depends
on type. Note to restore external resources, either --external or
--inherit-fd is used, depending on resource type.
Dump an external bind mount referenced by
mountpoint, saving it to image under the identifier name.
Dump all external bind mounts, autodetecting
those. Optional flags can contain m to also dump external master
mounts, s to also dump external shared mounts (default behavior is to
abort dumping if such mounts are found). If flags are not provided,
semicolon is optional.
Allow to dump a mount namespace having a real
block device mounted. A block device is identified by its major and
minor numbers, and criu saves its information to image under the
Dump an external file, i.e. an opened file
that is can not be resolved from the current mount namespace, which can not be
dumped without using this option. The file is identified by mnt_id (a
field obtained from /proc/pid/fdinfo/N) and
inode (as returned by stat(2)).
Dump an external TTY, identified by
st_rdev and st_dev fields returned by stat(2).
Tell criu that one end of a pair of
UNIX sockets (created by socketpair(2)) with id is OK to be
Use cgroup freezer to collect processes.
Collect cgroups into the image thus they gonna
be restored then. Without this option, criu will not save cgroups
configuration associated with a task.
Specify controllers and their properties to be
saved into the image file. criu
predefines specifications for common
controllers, but since the kernel can add new controllers and modify their
properties, there should be a way to specify ones matched the kernel.
argument describes the controller and properties specification in a
simplified YAML form:
- "strategy": "merge"
- "properties": ["a", "b"]
- "strategy": "replace"
- "properties": ["c", "d"]
are controllers names, and a
are their properties.
Note the format: double quotes, spaces and new lines are required. The
specifies what to do if a controller specified already exists
as a built-in one: criu
can either merge
For example, the command line for the above example should look like this:
--cgroup-props "\"c1\":\n - \"strategy\": \"merge\"\n - \"properties\": [\"a\", \"b\"]\n \"c2\":\n - \"strategy\": \"replace\"\n - \"properties\": [\"c\", \"d\"]"
Same as --cgroup-props, except the
specification is read from the file.
Dump a controller with name only,
skipping anything else that was discovered automatically (usually via
/proc). This option is useful when one needs criu to skip some
When combined with --cgroup-props,
makes criu substitute a predefined controller property with the new one
shipped. If the option is not used, the predefined properties are merged with
the provided ones.
Checkpoint established TCP connections.
This option skips in-flight TCP connections.
If any TCP connections that are not yet completely established are found,
criu ignores these connections, rather than errors out. The TCP stack
on the client side is expected to handle the re-connect gracefully.
Restore connected TCP sockets in closed
Use any path to a device file if the original
one is inaccessible.
Send pages to a page server (see the
Force resolving names for inotify and fsnotify
Deduplicate "old" data in pages
images of previous dump. This option implies incremental dump
mode (see the pre-dump command).
Dump file locks. It is necessary to make sure
that all file lock users are taken into dump, so it is only safe to use this
for enclosed containers where locks are not held by any processes outside of
dumped process tree.
Allows to link unlinked files back, if
possible (modifies filesystem during restore).
Set the maximum size of deleted file to be
carried inside image. By default, up to 1M file is allowed. Using this option
allows to not put big deleted files inside images. Argument size may be
postfixed with a K, M or G, which stands for kilo-, mega,
and gigabytes, accordingly.
Allow one to dump shell jobs. This implies the
restored task will inherit session and process group ID from the criu
itself. This option also allows to migrate a single external tty connection,
to migrate applications like top. If used with dump command, it
must be specified with restore as well.
Specify CPU capabilities to write to an image
file. The argument is a comma-separated list of none, fpu,
cpu, ins, all. If the argument is omitted or set to
none, capabilities will not be written, which is the default
Change the root for the controller that will
be dumped. By default, criu simply dumps everything below where any of
the tasks live. However, if a container moves all of its tasks into a cgroup
directory below the container engine’s default directory for tasks,
permissions will not be preserved on the upper directories with no tasks in
them, which may cause problems.
Perform the dump procedure without writing
memory pages into the image files and prepare to service page requests over
the network. When dump runs in this mode it presumes that
lazy-pages daemon will connect to it and fetch memory pages to lazily
inject them into the restored process address space. This option is intended
for post-copy (lazy) migration and should be used in conjunction with
restore with appropriate options.
Restores previously checkpointed processes.
Inherit a file descriptor. This option lets
use an already opened file descriptor N
for restoring a
file identified by resource
. This option can be used to restore an
external resource dumped with the help of --external file
, and unix
argument can be one of the following:
Note that square brackets used in this option arguments are literals and usually
need to be escaped from shell.
Detach criu itself once restore is
Leave tasks in stopped state after restore
(rather than resuming their execution).
Restore root task as a sibling (makes sense
only with --restore-detached).
, --root path
Change the root filesystem to path
(when run in a mount namespace).
Restore an instance of an external resource.
The generic syntax is type of resource, followed by resource id
(enclosed in literal square brackets), and optional value (prepended by
a literal semicolon). The following resource types are currently supported:
mnt, dev, veth, macvlan. Syntax depends on type.
Note to restore external resources dealing with opened file descriptors (such
as dumped with the help of --external file, tty, and
unix options), option --inherit-fd should be used.
Restore an external bind mount referenced in
the image by name, bind-mounting it from the host mountpoint to
a proper mount point.
Restore all external bind mounts (dumped with
the help of --external mnt auto-detection).
Restore an external mount device, identified
in the image by name, using the existing block device
Set the outer VETH device name (corresponding
to inner_dev being restored) to outer_dev. If optional
@bridge is specified, outer_dev is added to that bridge.
If the option is not used, outer_dev will be autogenerated by the
When restoring an image that have a MacVLAN
device in it, this option must be used to specify to which outer_dev
(an existing network device in CRIU namespace) the restored inner_dev
should be bound to.
Restore cgroups configuration associated with
a task from the image. Controllers are always restored in an optimistic way
— if already present in system, criu reuses it, otherwise it
will be created.
may be one of the following:
Do not restore cgroup properties but require
cgroup to pre-exist at the moment of restore procedure.
Restore cgroup properties and require cgroup
Restore cgroup properties if only cgroup has
been created by criu, otherwise do not restore properties. This is the
default if mode is unspecified.
Always restore all cgroups and their
Restore all cgroups and their properties from
the scratch, requiring them to not present in the system.
Change the root cgroup the controller will be
installed into. No controller means that root is the default for all
controllers not specified.
Restore previously dumped established TCP
connections. This implies that the network has been locked between dump
and restore phases so other side of a connection simply notice a kind
Correspondence between outside and inside
names of veth devices.
Restore file locks from the image.
As soon as a page is restored it get punched
out from image.
Restore shell jobs, in other words inherit
session and process group ID from the criu itself.
Specify CPU capabilities to be present on the
CPU the process is restoring. To inverse a capability, prefix it with
^. This option implies that --cpu-cap has been passed on
dump as well, except fpu option case. The cap argument
can be the following (or a set of comma-separated values):
Require all capabilities. This is
default mode if --cpu-cap is passed without arguments. Most safe
Require the CPU to have all capabilities in
image to match runtime CPU.
Require the CPU to have compatible FPU. For
example the process might be dumped with xsave capability but attempted to
restore without it present on target CPU. In such case we refuse to proceed.
This is default mode if --cpu-cap is not present in command
line. Note this argument might be passed even if on the dump no
--cpu-cap have been specified because FPU frames are always encoded
Require CPU compatibility on instructions
Ignore capabilities. Most dangerous mode. The
behaviour is implementation dependent. Try to not use it until really
For example, this option can be used in case --cpu-cap=cpu
, and images are migrated to a less capable CPU and are to
be restored. By default, criu
shows an error that CPU capabilities are
not adequate, but this can be suppressed by using --cpu-cap=none
Silently skip restoring sysctls that are not
available. This allows to restore on an older kernel, or a kernel configured
without some options.
Restore the processes without filling out the
entire memory contents. When this option is used, restore sets up the
infrastructure required to fill memory pages either on demand when the process
accesses them or in the background without stopping the restored process. This
option requires running lazy-pages daemon.
Checks whether the kernel supports the features needed by criu
and restore a process tree.
There are three categories of kernel support, as described below. criu
always checks Category 1 features unless --feature
specified which only checks a specified feature.
Absolutely required. These are features like
support for /proc/PID/map_files, NETLINK_SOCK_DIAG socket
monitoring, /proc/sys/kernel/ns_last_pid etc.
Required only for specific cases. These are
features like AIO remap, /dev/net/tun and others that are only required
if a process being dumped or restored is using those.
Experimental. These are features like
task-diag that are used for experimental purposes (mostly during
If there are no errors or warnings, criu
prints "Looks good."
and its exit code is 0.
A missing Category 1 feature causes criu
to print "Does not look
good." and its exit code is non-zero.
Missing Category 2 and 3 features cause criu
to print "Looks good
but ..." and its exit code is be non-zero.
Without any options, criu check
checks Category 1 features. This behavior
can be changed by using the following options:
Check kernel support for Category 2
Check kernel support for Category 3
Check kernel support for Category 1, 2, and 3
Check a specific feature. If name is
list, a list of valid kernel feature names that can be checked will be
in page server mode.
Runs page server as a daemon (background
Write \\0 to the FD and close it once
page-server is ready to handle requests. The status-fd allows to not daemonize
a process and get its exit code at the end. It isn’t supposed to use
--daemon and --status-fd together.
Page server IP address.
Page server port number.
Serve local memory dump to a remote
lazy-pages daemon. In this mode the page-server reads local
memory dump and allows the remote lazy-pages deamon to request memory
pages in random order.
in lazy-pages daemon mode.
daemon is responsible for managing user-level demand
paging for the restored processes. It gets information required to fill the
process memory pages from the restore
and from the checkpont directory.
When a restored process access certain memory page for the first time, the
daemon injects its contents into the process address space.
The memory pages that are not yet requested by the restored processes are
injected in the background.
Executes a system call inside a destination task's context. This functionality
is deprecated; please use Compel
in RPC daemon mode, where criu
is listening for RPC
commands over socket to perform. This is convenient for a case where daemon
itself is running in a privileged (superuser) mode but clients are not.
Starts pagemap data deduplication procedure, where criu
scans over all
pagemap files and tries to minimize the number of pagemap entries by obtaining
the references from a parent pagemap image.
Fetches current CPU features and write them into an image file.
Fetches current CPU features (i.e. CPU the criu
is running on) and test
if they are compatible with the ones present in an image file.
To checkpoint a program with pid of 1234
and write all image files into
criu dump -D checkpoint -t 1234
To restore this program detaching criu itself:
criu restore -d -D checkpoint
The CRIU team.
Copyright (C) 2011-2016, Parallels Holdings, Inc.