AWFFull - A Webalizer Fork, Full o' features
AWFFull is a web server log analysis program based on The Webalizer. AWFFull
produces usage statistics in HTML format for viewing with a browser. The
results are presented in both columnar and graphical format, which facilitates
interpretation. Yearly, monthly, daily and hourly usage statistics are
presented, along with the ability to display usage by site, URL, referrer,
user agent (browser), user name, search strings, entry/exit pages, and country
(some information may not be available if not present in the log file being
AWFFull supports the following log formats shown in the following variable list:
- (common log format) log files
- log formats as defined by NCSA and others, and variations
of these which it attempts to handle intelligently
- wu-ftpd formatted log files allowing analysis of ftp
servers, and squid proxy logs.
Logs may also be compressed, via gzip
. If a compressed log file is
detected, it will be automatically uncompressed while it is read. Compressed
logs must have the standard gzip
extension of .gz
This documentation applies to AWFFull Version 3.8.2
AWFFull is based on The Webalizer code and has a number of large and small
changes. These include:
- Beyond the raw statistics: Making use of published formulae
to provide additional insights into site usage
- GeoIP IP Address look-ups for more accurate country
- Resizable graphs
- Integration with GNU gettext allowing for ease of
translations. Currently 32 languages are supported.
- Display more than 12 months of the site history on the
- Additional page count tracking and sort by same.
- Some minor visual tweaks, including Geolizer's use of Kb,
Mb etc for Volumes
- Additional Pie Charts for URL counts, Entry and Exit Pages,
- Horizontal lines on graphs that are more sensible and
easier to read
- User Agent and Referral tracking is now calculated via
PAGES not HITS
- GNU style long command line options are now supported (eg
- Can choose what is a page by excluding ‘what
isn't’ vs the original ‘what is’ method
- Requests to the site being analysed are displayed with the
matching referring URL
- A Table of 404 Errors, and the referring URL can be
- An external CSS file can be used with the generated
- Manual performance optimisation of the config file is now
easier with a post analysis summary output
- Specified IP Addresses can be assigned to a given
- Additional Dump options for detailed analysis with other
- Lotus Domino v6 logs are now detected and processed
Additional changes and improvements are planned and undergoing implementation.
See the TODO file for details.
With version 3.8.1 of AWFFull, several new measured results have been added to
the detailed report monthly page.
- Single Access
- Single Access Pages - the only page seen within a given
- How useful a given entry page is to draw Visitors deeper
into your site
- The Ratio of Page Entries to Page Exits
These metrics can help towards improving insight in the usage of the processed
web site. And hence allow the site owner to make positive change to make the
site more useful to site visitors. All three metrics appear in the
‘Entry Pages’ Report. ‘Popularity’ is also on the
‘Exit Pages’ Report.
More completely: ‘Single Access Pages’. This is a report on the
number of times that a given page was the only page viewed within a Visit. Or
in English, Someone came to your website. They only viewed one page. The
number is the cumulative count of people who did this for that particular
page. Why is this useful? Identifying those entry pages that don't draw
visitors deeper into your site. Or seeing entry pages that shouldn't be entry
pages. It's also a reality check against the next two values which are
calculated from this number. The number generated should be a subset of the
‘Entry Page Views’ and/or ‘Exit Page Views’
metric. If it isn't? Let me know, we have a bug. :-)
Is calculated as 1 - (Single Access / Entry Page Views) expressed as a
percentage. In essence Stickiness describes how useful a given entry page is
to draw Visitors deeper into your site. The stickier the page, the more folk
are caught by it. :-) The closer to 100% the better. Generally. Certain pages
within YOUR website may not make sense to have a high stickiness or even >
5%. This measurement is a clue to understanding how your site is used, it is
not a rule. How is this useful? How and where are people entering your web
site. Does that make sense? Should it be here or there? What can you change to
fix this and hence improve their use of your website.
Popularity is the Ratio of Page Entries to Page Exits. o If it equals 1.0? Then
the number of visitors to your site who started with that page, equals the
number who left at that page. o If greater then 1.0, then more people entered
here then left. o If less then 0? More people left from here then entered. I
personally find this metric one of the more useful "At a Glance: How are
Pages Performing" metrics. One of the difficulties with using this
particular metric is that certain numbers will NOT make sense for YOUR site.
In that a natural exit page would expect to have a very low Popularity. It's
an exit page, not an entry page. So if an exit page has a high popularity,
then you have a real problem. Likewise, a low Popularity for an entry page is
unlikely to be a Good Thing(tm).
"Where & Why?" All three of these metrics are covered very nicely
in Hack #58 from "Web Site Measurement Hacks" . Which is where,
credit where credit due, the inspiration to add these metrics came from.
AWFFull is designed to be run from a Unix command line prompt or as a crond(8)
job. There is no need to run with super-user privleges, and indeed, is
preferable NOT to.
Once executed, the general flow of the program is:
A default configuration file is scanned for, /etc/awffull/awffull.conf
and, if found, is used.
Any command line arguments given to the program are parsed. This may include the
specification of one or more configuration files, which are processed at the
time it is encountered.
It can be useful to have multiple config files. A master used for multiple
sites, and individualised config files. Do be aware that last option set wins.
So last config file, or if after a config file, command line options. Useful
if you desire to send the output to an alternate directory.
If a log file was specified, it is opened and made ready for processing. If no
log file was given, STDIN is used for input. If the log filename '-' is
specified, STDIN will be forced.
If an output directory was specified, AWFFull changes to that directory in
preparation for generating output. If no output directory was given, the
current directory is used.
If no hostname was given, the program attempts to get the hostname using a
uname(2) system call. If that fails, localhost is used.
A history file is searched for in the current directory (output directory) and
read if found. This file keeps totals for previous months, which is used in
the main index.html HTML document. Note: The file location can now be
specified with the HistoryName configuration option.
If incremental processing was specified, a data file is searched for and loaded
if found, containing the 'internal state' data of the program at the end of a
previous run. Note: The file location can now be specified with the
IncrementalName configuration option.
Main processing begins on the log file. If the log spans multiple months, a
separate HTML document is created for each month. After main processing, the
main index.html page is created, which has totals by month and links to each
months HTML document.
A new history file is saved to disk, which includes totals generated by AWFFull
during the current run.
If incremental processing was specified, a data file is written that contains
the 'internal state' data at the end of this run.
Version 1.2x of The Webalizer added incremental run capability. Simply put, this
allows processing large log files by breaking them up into smaller pieces, and
processing these pieces instead. What this means in real terms is that you can
now rotate your log files as often as you want, and still be able to produce
monthly usage statistics without the loss of any detail. Basically, AWFFull
saves and restores all internal data in a file named awffull.current
This allows the program to 'start where it left off' so to speak, and allows
the preservation of detail from one run to the next. The data file is placed
in the current output directory, and is a plain ASCII text file that can be
viewed with any standard text editor. It's location and name may be changed
using the IncrementalName configuration keyword.
Some special precautions need to be taken when using the incremental run
capability of AWFFull. Configuration options should not be changed between
runs, as that could cause corruption of the internal data stored. For example,
changing the MangleAgents level will cause different representations of user
agents to be stored, producing invalid results in the user agents section of
the report. If you need to change configuration options, do it at the end of
the month after normal processing of the previous month and before processing
the current month. You may also want to delete the awffull.current file as
AWFFull also attempts to prevent data duplication by keeping track of the
timestamp of the last record processed. This timestamp is then compared to
current records being processed, and any records that were logged previous to
that timestamp are ignored. This, in theory, should allow you to re-process
logs that have already been processed, or process logs that contain a mix of
processed/not yet processed records, and not produce duplication of
The only time this may break is if you have duplicate timestamps in two separate
log files. Any records in the second log file that do have the same timestamp
as the last record in the previous log file processed, will be discarded as if
they had already been processed. There are lots of ways to prevent this
however, for example, stopping the web server before rotating logs will
prevent this situation, or using a tool such as cronolog
(⟨http://cronolog.org/⟩). This setup also necessitates that you
always process logs in chronological order, otherwise data loss will occur as
a result of the timestamp compare.
AWFFull no longer supports DNS lookups. Please use an external program such as
DNShistory or DNSTran instead.
With version 3.7.1 of AWFFull, GeoIP capability can be used for improved country
AWFFull supports many different configuration options that will alter the way
the program behaves and generates output. Most of these can be specified on
the command line, while some can only be specified in a configuration file.
The command line options are listed below, with references to the
corresponding configuration file keywords. See also awffull.conf(5).
- -h, --help
- Display all available command line options and exit
- -V, --version
- Display program version and exit program
- -v --verbose
- Verbosity Display debugging information for errors
and warnings. Multiple v's will increase the amount of information
- Display optimisation useful information pertaining to the
number of matches against various Group, Hide and Ignore options.
- -i --ignore_history
- IgnoreHist Ignore history. USE WITH CAUTION. This
will cause AWFFull to ignore any previous monthly history file only.
Incremental data (if present) is still processed.
- -p --preserve_state
- Incremental Preserve internal data between
- -T --timing
- TimeMe. Force display of timing information at end
- -c --config=FILE
- Use configuration file FILE
- -n NAME
- HostName. Use the hostname NAME.
- -o --output=DIR
- OutputDir. Use output directory DIR.
- -t NAME
- ReportTitle. Use NAME for report title.
- F --logtype=TYPE
- LogType. Specify log type to be processed. Value can
be one of: auto, clf, combined, domino, ftp or squid format. If not
specified, will default to auto format. FTP logs must be in standard
wu-ftpd xferlog format. A value of ‘auto’ states that the
log format automatically ascertained.
- -f --fold
- FoldSeqErr. Fold out of sequence log records back
into analysis, by treating as if they were the same date/time as the last
good record. Normally, out of sequence log records are simply
- CountryGraph. Suppress country graph.
- HourlyGraph. Suppress hourly graph.
- -x NAME
- HTMLExtension. Defines the HTML file extension to
use on the created report files. If not specified, defaults to html. Do
not include the leading period.
- HourlyStats. Suppress hourly statistics.
- GraphLegend. Suppress color coded graph
- -l NUM
- GraphLines. Use background lines. The number of
lines and where to place are automatically calculated. For backwards
compatibility, any number > 0 enables. Use zero ('0') to disable the
- -P NAME
- PageType. Specify file extensions that are
considered pages. Sometimes referred to as pageviews.
- -m NUM
- VisitTimeout. Specify the Visit timeout period.
Specified in number of seconds. Default is 1800 seconds (30 minutes).
Sometimes referred to as sessions.
- -I NAME
- IndexAlias. Use the filename name as an additional
alias for index.
- -M NUM
- MangleAgents. Mangle user agent names according to
the mangle level specified by num.
Mangle levels are:
5 - Browser name and major version
4 - Browser name, major and minor version
3 - Browser name, major version, minor version to two decimal places
2 - Browser name, major and minor versions and sub-version
1 - Browser name, version and machine type if possible
0 - All information (left unchanged).
- -g NUM
- GroupDomains. Automatically group sites by domain. The
grouping level specified by num can be thought of as 'the number of dots'
to display in the grouping. The default value of 0 disables any domain
- -a NAME
- HideAgent. Hide user agents matching name.
- -r NAME
- HideReferrer. Hide referrer matching name.
- -s NAME
- HideSite. Hide site matching name.
- -X NAME
- HideAllSites. Hide all individual sites (only
- -u NAME
- HideURL. Hide URL matching name.
Table size options
- -A --top_agents=NUM
- TopAgents. Display the top num user agents
- -R --top_refers=NUM
- TopReferrers. Display the top num referrers
- -S --top_sites=NUM
- TopSites. Display the top num sites table.
- -U --top_urls=NUM
- TopURLs. Display the top num URL's table.
- -C --top_countries=NUM
- TopCountries. Display the top num countries
- -e --top_entry=NUM
- TopEntry. Display the top num entry pages
- -E --top_exit=NUM
- TopExit. Display the top num exit pages table.
- Enables the use of the Maxmind GeoIP capability for more
accurate detection of countries.
NOTE! Do not enable GeoIP if you analyse files that have had the
IP Address translated to a Fully Qualified Host Name. Use either
raw IP Addresses and GeoIP, or Names and disable GeoIP. ie. Don't
use GeoIP AND DNShistory.
- Display the various Group/Hide etc Match Counts. This
option is ideal for optimisation of the awffull.conf file. Just be careful
with optimising Agents in particular, as the order is typically
See the awffull.conf(5) man page for complete details of all configuration
Configuration files are standard ASCII(7) text files that may be created or
edited using any standard editor.
Blank lines and lines that begin with a pound sign ('#') are ignored.
Any other lines are considered to be configuration lines, and have the form
‘Keyword Value’, where the ‘Keyword’ is one of the
currently available configuration keywords (see awffull.conf(5)), and
‘Value’ is the value to assign to that particular option.
Any text found after the keyword up to the end of the line is considered the
keyword's value, so you should not include anything after the actual value on
the line that is not actually part of the value being assigned. The file
provided with the distribution contains lots of useful
documentation and examples as well.
Certain "Keywords" will accept a 2nd value. In those situations, the
first value may be enclosed in double quotes (") to allow for whitespace.
None currently known. YMMV....
Report bugs to ⟨https://bugs.launchpad.net/awffull⟩, or use the
email discussion list: <email@example.com>
In case it is not obvious: AWFFull is a play/pun on the word
‘awful’, and is pronounced the same way. Yes it was deliberate.
 Web Site Measurement Hacks. Eric T. Peterson (and others). O'Reilly. ISBN