Synchronize directories in parallel using fpart
tool synchronizes directories in
parallel using fpart(1)
. It computes subsets of
jobs to synchronize them to
Synchronization jobs can be executed either locally or remotely (using SSH
workers, see option -w
) and are executed
on-the-fly while filesystem crawling goes on. This makes
a good tool for migrating large
- Print help
- Verbose mode. Can be be specified several times to increase
- Start jobs concurrent sync
jobs (either locally or remotely, see below). Default:
- Transfer at most files
files per sync job. Default: 2000
- Transfer at most size
bytes per sync job.
Default: 4294967296 (4 GB)
- Use remote SSH wrks to
synchronize files. Synchronization jobs are executed locally when this
option is not set. wrks is a
space-separated list of login@machine connection strings and can be
specified several times. You must be allowed to connect to those machines
using a SSH key to avoid user interaction.
- Set fpsync shared directory to
shdir. This option is mandatory when
using SSH workers and set by default to
tmpdir when running locally. The
specified directory must be an absolute path ; it will be used to handle
communications with SSH hosts (sharing partitions and log files) and, as a
consequence, must be made available to all participating hosts (e.g.
through a r/w NFS mount), including the master one running
- Set fpsync temporary directory
to tmpdir. This directory remains local
and does not need to be shared amongst SSH workers when using the
-w option. Default:
- Resume job jobname and
restart synchronizing remaining partitions from a previous run.
jobname can be obtained using verbose
mode (see option -v). Note that filesystem
crawling is skipped when resuming a previous run. As a consequence,
options -f, -s,
and dst_dir/ are ignored.
- Override default rsync(1)
options with rsyncopts. Use this option
with care as certain options are incompatible with a parallel usage (e.g.
- Override default fpart(1)
options with fpartopts.
Default: -x .zfs -x .snapshot* -x .ckpt
- Sudo mode. Use sudo(8) for
filesystem crawling and synchronizations.
- Source directory. It must be absolute and available on all
participating hosts (including the master one, running
- Destination directory. It must be absolute and available on
all participating workers.
run generates a unique
, which is displayed in verbose mode
(see option -v
) and within log files. You can use
to resume a previous run (see
will then restart synchronizing data from the parts that were being
synchonized at the time it stopped.
This unique feature gives the administrator the ability to stop
and restart it later, without having to
restart the whole filesystem crawling and synchronization process. Note that
resuming is only possible when filesystem crawling step has finished.
During synchronization, you can press CTRL-C to interrupt the process. The first
CTRL-C prevents new synchronizations from being submitted and the process will
wait for current synchronizations to be finished before exiting. If you press
CTRL-C again, current synchronizations will be killed and
will exit immediately.
On certain systems, CTRL-T can be pressed to get the status of current and
remaining parts to be synchronized. This can also be achieved by sending a
SIGINFO to the fpsync
Whether you use verbose mode or not, everything is logged within
Here are some examples:
-n 4 /usr/src/ /var/src/
Synchronizes /usr/src/ to
/var/src/ using 4 local jobs.
-n 2 -w login@machine1 -w login@machine2 -d /mnt/fpsync /mnt/src/
Synchronizes /mnt/src/ to
/mnt/dst/ using 2 concurrent jobs executed
remotely on 2 SSH workers (machine1 and machine2). The shared directory is
set to /mnt/fpsync and mounted on the machine
running fpsync, as well as on machine1 and
machine2. The source directory (/mnt/src/) is
also available on those 3 machines, while the destination directory
(/mnt/dst/) is mounted on SSH workers only
(machine1 and machine2).
makes several options not
usable, such as --delete
. If your source
directory is live while fpsync
is running, you
will have to delete extra files from destination directory. This is usually
done by using a final -offline- rsync(1)
that will use this option.
enqueues synchronization jobs on disk,
within the tmpdir/queue
directory. Be careful to
host this queue on a filesystem that can handle fine-grained mtime timestamps
(i.e. with a sub-second precision) if you want the queue to be processed in
order when fpart(1)
generates several jobs per
second. On FreeBSD, VFS(9)
can be tuned using the 'vfs.timestamp_precision' sysctl. See
Fpsync has been written by Ganaël
and is available under the BSD license on
No bug known (yet).