buffer - very fast reblocking program
[ -S size
] [-b blocks
] [-s size
] [ -m size
] [ -p percentage
] [-u microseconds
] [ -Z
] [-i filename
] [-o filename
- -i filename
- Use the given file as the input file. The default is
- -o filename
- Use the given file as the output file. The default is
- -S size
- After every chunk of this size has been written, print out
how much has been written so far. Also prints the total throughput. By
default this is not set.
- -s size
- Size in bytes of each block. The default blocksize is 10k
to match the normal output of the tar(1) program.
- -z size
- Combines the -S and -s flags.
- -b blocks
- Number of blocks to allocate to shared memory circular
buffer. Defaults to the number required to fill up the shared memory
- -m size
- Maximum size of the shared memory chunk to allocate for the
circular queue. Defaults to one megabyte.
- -p percentage
- Only start a write when the given percentage of the
internal queue is full. A percentage around 75 often proves best. Defaults
- -u microseconds
- After every write pause for this many microseconds.
Defaults to zero. (Surprisingly a small sleep, 100 usecs, after each write
can greatly enhance throughput on some drives.)
- Force each block written to be padded out to the blocksize.
This is needed by some tape and cartridge drives. Defaults to unpadded.
This only affects the last block written.
- On exiting print to stderr a brief message showing the
total number of bytes written.
- If reading/writing directly to a character device (like a
tape drive) then after each gigabyte perform an lseek to the start of the
file. Use this flag with extreme care. It can only be used on devices
where an lseek does not rewind the tape but does reset the kernels
position flags. It is used to allow more than 2 gigabytes to be
- Print debug information to stderr during operation.
Sizes are a number with an optional trailing character. A 'b' multiplies the
size by 512, a 'k' by 1024 and an 'm' by a meg.
reads from standard input reblocking to the given blocksize and
writes each block to standard output.
is a pair of processes communicating via a large
circular queue held in shared memory. The reader process only has to block
when the queue is full and the writer process when the queue is empty.
is designed to try and keep the writer side continuously busy so
that it can stream when writing to tape drives. When used to write tapes with
an intervening network buffer
can result in a considerable increase in
The default settings for buffer
are normally good enough. If you are a
heavy tape user then it is worth your while trying out various different
combinations of options. In particular running a buffer
at both ends of
the pipe can provide a substantial increase (see last example below).
$ buffer < /etc/termcap > /dev/rst8
$ tar cf - . | rsh somehost 'buffer > /dev/rst8'
$ dump fu - | rsh somehost 'buffer -s 16k > /dev/nrst8'
$ tar cf - . | buffer |
rsh somehost 'buffer -S 500K -p 75 > /dev/rst0'
There are 2 limits in buffer
affecting the maximum total buffer size: The
maximum number of blocks (2048) and the maximum size of a block (512kB). This
results in a maximum total buffer size of 1GB.
Note that there is also a kernel limit for the maximum size of a shared memory
segment (used by buffer
internally) which is typically much lower than
the limits in buffer
. So if buffer
complains about not being
able to create a shared memory segment, this limit might have to be raised
using the command sysctl kernel.shmmax=XXX
(see also sysctl(1)
Internally, for printing purposes, buffer counts in terms of the number of
kilobytes output. If the blocksize you use is not a whole number of kilobytes
then the numbers printed will be inaccurate.
Thanks to Kevin Twidle <firstname.lastname@example.org> for a lot of early suggestions
and patches to make it work with non-tar/dump tapes to exabyte drives.
Thanks to Andi Karrer <email@example.com>, Rumi Zahir
<firstname.lastname@example.org> and Christoph Wicki <email@example.com> for
patches to make buffer work when trying to write single tape files of greater
than 2 gigabytes.
Copyright (C) 1990, 1991 by Lee McLoughlin.
dd(1), tar(1), rsh(1)