nnmaster - nn database manager
[ "lock message
] [ groups
] [ groups
is the daemon which is responsible for building and maintaining
the database used by the nn
(1) news reader.
is started when the system enters multi-user mode, and
runs until system shutdown. To facilitate this, you should place the following
call in /etc/rc (or similar) to invoke the nnmaster
$master/nnmaster -l -r -C
where $master is the MASTER_DIRECTORY defined during configuration of nn
is started as specified above, it will first unlock the
database in case it was locked (- l
), perform a thorough consistency
check on the database (- C
Then, every 10 minutes (- r
), it will look at the time-stamp of the news
active file to see whether new articles have arrived on the system (or whether
articles have been expired). (See - U
If the active file has been modified, nnmaster
will collect the header
information from the new articles and enter them into the database (or remove
the headers of the expired articles from the database).
If it detects that some articles have been expired, it will automatically remove
the header information of the expired articles from the database.
will collect all available news groups defined in the
news active file. The set of collected groups can be controlled via the
argument line. Groups can be either included or excluded:
A group name, e.g. comp, will cause the group and all its subgroups to be
collected. Individual groups, e.g. news.software.nn, can also be specified
A group name preceded by an exclamation mark, e.g. !talk.politics, will cause
the group and all its subgroups to be ignored.
An empty argument, i.e. "", will cause all groups that are not ignored
to be collected. For example, to collect everything but rec and misc, use the
-r !rec !misc ""
If the empty argument had been omitted, nothing would be collected, since the
presence of any groups
arguments causes nnmaster
to ignore all
groups which are not explicitly mentioned.
Example 1: The following commands can be executed by cron to collect different
sets of groups at different intervals or under different conditions:
-B -O14 rec misc sci -LBO -u
!rec !misc !sci "" -u
Example 2: The group arguments are used in the given sequence, e.g. to leave out
comp.sys, but still collect comp.sys.ibm.pc, use the command:
-r comp.sys.ibm.pc !comp.sys ""
The use of the - u
option in the first example is essential, since each
of the commands will update the active file time stamp which will prevent the
other command from detecting new articles that have arrived.
Using this method to keep specific groups out of the database must be used with
great caution; just a single invocation of nnmaster
arguments will collect all the otherwise ignored groups!
The following options control how nnmaster
performs the collection of new
- -r [ min ]
Daemon mode. The nnmaster will put itself in the background (unless -
f is also specified), and will checks for arrival of new articles
and expired articles every min minutes (and update the database
accordingly). If min is omitted, the default is to check every 10
Without the -r option, the nnmaster will just perform a single
collection of new articles (if any) and then exit. This can be used to
have the nnmaster started by cron(8) at regular intervals
instead of having it as a daemon which sleeps between checking for new
articles. Since the nnmaster is a bit expensive to start up (it has
to read a few files), it is up to you to decide which mode is best on your
system. (I have also heard that it works to call nnmaster without
-r from inews(1). I cannot recommend this unless you receive
batched news; invoking nnmaster for every received article sounds
too expensive to me.)
- -h [ sec ]
- Hold collection of new articles until next scan if new
articles have arrived withing the last sec  seconds. This is
useful to prevent nnmaster from competing for disk i/o with an
rnews process which is unbatching incoming news, or a running
expire process. It will have the side effect of limiting the number
of C: entries in the log, since collection of incoming batches will
typically be done in larger thunks.
- Run nnmaster in foreground in daemon mode (see
-r). Useful if nnmaster is invoked from inittab. (Notice
that if you use a respawn entry in inittab, you will not be able to
stop nnmaster using the -k option, since init will
immediately start another master.)
- Perform a consistency check on the database on start-up,
and rebuild corrupted database files. This operation can be quite
time-consuming since it reads through all the database files.
- Normally, articles without a proper news header (no
Newsgroups: line) are ignored. Specifying the - b option causes
these `bad' articles to be included in the database (normally with no
sender or subject).
- Remove `bad' articles. Sometimes, articles without a header
ends up in the news spool directory. These articles have no article id,
and therefore, they will never be expired by expire(8). This option
will allow the nnmaster to silently remove these articles (a `B'
entry will occur in the log file).
- -O days
- Ignore articles which are older than the given number of
days. This may help keep old 'stray' articles out of the database.
If the - B options is also specified, the old articles will be
removed from the news spool directories. Old ignored or removed articles
will be reported with an `O' entry in the log file. This option can be
disable for individual groups by the O flag in the GROUPS file (see
- -R N
- Specifies how the auto-recollect operation is
performed on the groups having this option set in the GROUPS file (see
below). Four methods are available (default is method 1):
1: Run expire on the group when new articles arrive.
2: Run expire on the group on every scan.
3: Recollect all articles when new articles arrive.
4: Recollect all articles on every scan.
- -M mode
- Normally, nnmaster will send a message via mail to
the news administrator (OWNER) when an error or potential problems
(primarily nntp related) occur. This can be restricted to only fatal
errors ( nnmaster terminated) if mode is 1, and disabled
completely if mode is 0.
- Normally, nnmaster will print a message on the
system console or in the syslog if a fatal error happens. This option will
prevent this, so only a type 'E' entry is written to the Log file.
Since articles does not stay forever in the news system, the database must be
cleaned up regularly to remove the information stored for expired articles.
Expiration of the database is normally scheduled
(1M) command executed by cron at a suitable time when expire on
the news articles has completed. The following command will send a message to
and cause it to initiate expire on all news groups:
Selective expiration of individual groups can be done from nnadmin
(interactive mode). It can also be done by invoking nnmaster
option. For example, the following command will run expire on all
groups except the `rec' groups:
nnmaster -F -k !rec ""
The - k
option is required to terminate the currently running master
since two daemons cannot be running at the same time. Thus to run expire (on
all groups) in this way from cron, the following commands must be used:
nnmaster -Fk "" ; nnmaster -r ...
It is also possible to have nnmaster
detect expiration automatically (see
). This requires that the min
field in the active file is
updated by the news expire (this is not the default case when Cnews is used).
However, this is not always a safe indication since the first article may not
have been expired, while a lot of other articles have been deleted.
There are several strategies available in the nnmaster
to do this
clean-up, each having their strengths and weaknesses.
(default): Rebuilds the database from the existing database
information by comparing the current database contents with the contents of
the news group directories, eliminating entries whose file no longer exists.
This method is guaranteed to eliminate all expired articles from the database,
and it is reasonably fast because it only has to read the directories, not
each article file.
If news is accessed remotely via nntp, the list of existing articles cannot
efficiently be fetched by reading a local directory. Instead an XHDR request
is sent to the nntp server to get a list of articles.
: Eliminates only the expired articles before the first article
in the group. This is very fast since only the active file and the database
itself is accessed, but it will obviously leave some expired articles in the
database. This method requires that the min
field in the active file is
updated by expire
: Discard the current database information and recollects all
articles. This is obviously very time consuming, and it is therefore not
recommended, especially not with nntp.
The options related to database expiration are:
- -E N
- Select expire method N. (If N is omitted, the
default method is used).
- -e [N]
- Automatically run expire in the database on groups where
the min number in the active file has increased by N (1 if
omitted) articles. This is disabled by default (since the min field
is often unreliable).
- Run expire once and exit. If a list of groups is
specified on the command line, the matched groups (see the rules above)
will be marked for expiration. If no groups are specified, only the groups
already scheduled for expire will be expired. Consequently, to expire all
groups, a blank argument "" (matching all groups) must be
The database can be locked centrally, which will normally disallow all access to
the database, and even block nnmaster
from being (accidentally)
started. When a lock is set on the database, all currently running clients
will terminate the next time they try to access the database. Setting a lock
on the database can thus also be used to force running clients to terminate.
The following options set and clear locks on the database:
- -l message
- Locks the database with the given message. The
message will be displayed every time a program tries to access the
- Unlock the database if it was locked.
- Ignore a possible lock and continue. This can be used to
have nnmaster operate on a database which is blocked from normal
Since only one nnmaster
can operate on the database at any one time, a
daemon must be stopped before a lock can be set on the
database. If neither -f
is specified with the -l
option (in both forms), nnmaster
will terminate after setting or
clearing the lock.
The following options are used to initialize and update the central database
- -I [limit]
- Initialize database. This option will erase an existing
database, and create an empty database containing entries for the
currently known groups. nnmaster will offer you to use an existing
GROUPS file when initializing the database.
The optional limit can be used to put a limit on the number of
articles that will be collected in each group during the first collection
of articles following the database initialization. This is useful on
systems where the 'min' field in the active file is unreliable or not
maintained (Cnews doesn't) to limit the work done to do the initial
collection of news after the initialization of the database. If news is
accessed remotely from an NNTP server, this is even more important! If
limit is omitted, or is zero, nnmaster will trust the min
field and collect all articles in the range min..last.
- Reread the GROUPS file. This option is used to instruct
nnmaster to parse the GROUPS file after it has been edited. See the
section on the GROUPS file below.
The following options controls various details of the nnmaster
- -D [ N ]
- Run nnmaster in "debug mode". If N
is omitted, or equals 1 or 3, this will produce a compact but still very
informative trace of the collection or expire process directly on the
terminal. This is most useful when doing the first collection of articles
after initializing the database with -I. If N is 2 or 3, a trace of
the NNTP traffic is written to a file nnmaster.log in the TMP directory.
This option disables - r.
- Identifies the host which nnmaster is running on as
the nntp-server for its clients, i.e. that it can access the news spool
directory locally without using NNTP. Normally, nnmaster will
detect this by itself by comparing the host name to the contents of the
nntp_server file, so this option should really be superfluous.
- -y retries
In some networked environment, opening an article (shared from another
machine via NFS) may fail for no obvious reason. Using this option, it is
possible to cause nnmaster to perform retries attempts to
open an article before marking the article as non-existing in the
- -L types
- Exclude the specified entry types from the log file. This
is normally used to exclude the 'C'ollecting and e'X'pire entries (-
- Trace the collection process. This will place a lot of
information into the log file (T: entries).
- Normally, nnmaster will just compare the time-stamp
on the active file with a time-stamp saved in the database to see if new
articles have arrived. The -u option forces the nnmaster to
read the active file on start-up to see if new articles have
- Some SVR4 systems (and maybe SunOS) have a useful
"feature". Writing files with mmap() may not update the
last-changed timestamp on the file. Since INN uses mmap() for writing the
active file, this becomes a problem for nnmaster. The -U option
causes nnmaster to unconditionally read the active file each time
the repeat delay (-r) time expires.
- Print the release and version identification for
nnmaster, and exit.
- Wakeup the real nnmaster. Send a signal to the
nnmaster daemon to have it check for new articles immediately.
- Kill the already running nnmaster daemon before
proceeding with the operation specified by the other options (or terminate
if no other operation is requested).
The primary purpose of the GROUPS file is to store the names of the news groups
represented in the database. Each line in the file corresponds to an entry in
the (binary) MASTER file, and the sequence of the lines in the GROUPS file
must never be changed unless the database is reinitialized afterwards.
However, the contents of the lines in the GROUPS file can be edited to control
how the nnmaster
should handle each individual group.
The format of each line is:
news.group.name [ timestamp ] [ flags ]
The news group name is the name of the group, and must not be changed in any
way. If the group is no longer in the news active file, and consequently the
group does no longer exist, group name can be replaced by a `@' character
which will instruct nnmaster
to ignore this entry without having to
rebuild the entire database.
The optional time stamp indicates when the line was added to the GROUPS file and
is used by nn
to detect new groups. When the GROUPS file is built
initially from the active file, the time stamps are omitted which simply means
that they are "old".
One or more of the following flags can be added to the GROUPS line to control
's handling of the group:
- Causes nnmaster to treat all articles in the group
as digests, even when they don't initially look like digests. Articles
which are found not to be digests after all, are still not digested.
- Instructs nnmaster to never digest any articles in
- Disables the -O option for this group, i.e. all
existing articles will be included in the database (and they will not be
removed if the - B option is specified). This flag should be set on
groups which you never expire, or have a very long expire time!
- Causes nnmaster to recollect all available
articles in the group whenever a new article is received. This is said to
be useful is some high-traffic clarinet groups with many cancelled
- Instructs nnmaster to append all new articles
to the specified file. This makes it possible to keep specific groups out
of the way of expire. The archive file can be access directly from the
nn client using the goto-group command. The file name must
be a full path name to a file in an existing, writeable directory.
- Instructs nnmaster to completely ignore this group -
this is equivalent to setting the group name to `@' as described
- ! or X
- Causes nnmaster to ignore the group and not collect
the group's articles in the database.
Comments (starting with `#' and continuing to the end of line) and empty lines
are allow in the GROUPS file, but it is strongly recommended to keep the
changes to the GROUPS file as small as possible, because of the implicit
correspondence with the binary MASTER file.
It is not recommended to edit the GROUPS file while nnmaster
because it may add new groups to the file. After editing the GROUPS file, the
must be run before restarting the nnmaster
to parse and verify the new
can access the news articles from a local news spool
directory as well as from an NNTP server. When compiled with NNTP enabled,
will compare the name of the NNTP server and the name of the
local host; if they are identical, nnmaster
will bypass NNTP and access
the articles directly.
When it has to access the news articles via NNTP, it cannot time-stamp the
active file, so instead it transfers the entire active file from the NNTP
server and compares it with a local copy of the last active file fetched from
the NNTP server. This is not very expensive in terms of cpu-cycles, disk-load,
or net-bandwidth, but to stay on friendly terms with the NNTP server
administrator, you should probably not use shorter update intervals than the
standard 10 minutes.
Setting a much higher update interval than the standard 10 minutes is not really
recommended either, since an update normally implies fetching a burst of news
articles from the NNTP server, so setting the interval too long may imply that
the load on the NNTP server will be more un-even.
In expire method 1, the use of XHDR just to get a list of existing articles in a
group is definitely a waste of resources on the nntp server (but still lower
than using method 3). Before using the XHDR request, nnmaster
a non-standard "LISTGROUP" request; if the nntp server supports this
request, it should return an OK_HEAD status followed by an (unordered) list of
article numbers (one per line) terminated by a `.' line. The nntp servers
supporting this request will be much less loaded during expire.
The - O
option does not work with NNTP. The -B
option will only
work with NNTP if the nnmaster
is running on the NNTP server.
The $db, $master, and $news names used below are synonyms for the DB_DIRECTORY,
MASTER_DIRECTORY, and NEWS_LIB_DIRECTORY defined during configuration.
$db/MASTER Database master index
$db/GROUPS News group names and flags in MASTER file order
.[dx] Database files for group number nnn
.../.nn[dx] Database files if located in the group directories
$master/GATE Message channel from nnadmin
$master/MPID The process id of the nnmaster
$Log The log file (the location is configuration dependent)
$news/active Existing articles and groups
/usr/lib/nntp_server Contains the name of the NNTP server.
The MASTER file contains a record for each news group, occurring in the same
sequence as the group names in the GROUPS file. The sequence also defines the
group numbers used to identify the files in the database's DATA directory.
The GATE file will be created by nnadmin
when needed, and removed by
when it has read it. Therefore, to send a message to the
requires that you are allowed to write in the $master
The contents of the Log file are described in the nnadmin
nn(1), nncheck(1), nngrep(1), nntidy(1)
nnadmin(1M), nnspew(8), nnusage(1M)
Kim F. Storm, Texas Instruments A/S, Denmark