Critical Support

Even the best sys admins can feel overwhelmed with the logfiles generated by a single production server. The problem is especially severe if you are running multiple applications on one machine. Meet logwatch, a "customizable log analysis system that parses through your system's logs and creates a report analyzing areas that you specify" [1]. In this article, I look at this very popular tool, which I suspect many of you already use, and explore some of the lesser known options.

Lifesaver

A server that writes to many logs presents a couple of conspicuous challenges. The first challenge is physical limitation: Do you have enough spare space on your /var/log disk partition? If a server performs several roles, you might have firewall logs, user login logs, and HTTP logs – all in addition to the multitude of mandatory system logs. Before you know it, you might have dozens of applications vying for access to your overburdened /var/log directory.

The second issue is how to navigate the seemingly infinite screeds of recorded data successfully. It's an understatement to say the potentially critical information is surrounded by 10 times its volume of relatively useless noise. Faced with such a volume of data, it is imperative that you be able to zero in on the information you really need.

Timing is especially important when you are presented with a business-critical server issue, such as an attack. You need to know in advance the variety of logging formats employed by different packages, certain idiosyncrasies attributed to quirky applications, and whether you need to enable extra detail with debugging mode. All of these facets are essential if you want to pinpoint that elusive empirical evidence required to identify how an attacker has gained access to one or more of your servers. Empowered with that critical information, you can plug the hole through which the attackers gained access, and you can then use the log evidence to report the culprits to the authorities.

Basics

Straight out of the box, the excellent logwatch can sift through your seemingly endless logfiles and dig out the information you request, dutifully generating a well-considered, nicely formatted report, having sorted the wheat from the chaff.

In this article, I'll be using a Debian-based server for my examples, and I simply run the following command to install the package from the repositories:

# apt-get install logwatch
The following NEW packages will be installed:
libdate-manip-perl libyaml-syck-perl logwatch

As a result, about 12.5MB of new software is installed.

Logwatch gets configuration details several ways:

• from script or command-line arguments,
• /etc/logwatch/conf/ (for your system)
• /usr/share/logwatch/dist.conf/ (for your operating system)
• /usr/share/logwatch/default.conf/ (general defaults)

You can customize logwatch by editing the files in /etc/logwatch/conf/. You might want to create a file with only the modified variables (and include newly added scripts, if needed), or you might copy a config file from /usr/share/logwatch/default.conf to its namesake location in /etc/logwatch/conf and change what is needed in there. The comprehensive Debian HOWTO can be found at:

/usr/share/doc/logwatch/HOWTO-Customize-LogWatch.gz

(see the Documentation section).

Precedence of settings is from top (command line) to bottom (general defaults) in the list. If you see a config option with the format --<option>, you can assume it is used at run time; otherwise, you can assume the options are in the config files.

So powerful is logwatch that you might want to make just one change to the standard installation so that the root user does not receive a lot of email. To make this change, go to /usr/share/logwatch/default.conf/logwatch.conf. This config file presents a number of useful, configurable options that help, along with some other files I explore later, to control logwatch. In the MailTo = root line, you can replace root with a full email address. You might also want to change the Real Name option to something like Logwatch For Mail Server #1. A little farther down logwatch.conf is the mailer = sendmail -t entry, which I discuss in the "Mailer" box.

Reporting

If you have a Syslog server that pulls logs from multiple devices (e.g., servers, routers, switches) to a centralized locations, you might be pleasantly surprised to discover that the flexible logwatch can cope with the complexities presented when scanning logfiles from multiple hosts. For example, you can use --multiemail at run time to set up an email address for each host processed.

If the mail server admin doesn't need or want to know about a router's logwatch report, you can split log reporting recipients very simply using this option. If you want the reporting for many different hosts (i.e., networked devices) to be in one email report, then the following change in your config file will split up the entries by hostname:

SplitHosts = yes

In this case, do not limit hosts with HostLimit = no; otherwise, only the server's logs will be processed. Don't forget, of course, to keep email recipients to just you by selecting MultiEmail = no inside the config file (rather than the use --multiemail at run time as mentioned previously). If you want to send a report to multiple people, it's just a case of separating email addresses with a space in the config file.

You can also change where logwatch drops its temporary files, which might be important on servers with a number of applications installed, so that logwatch is therefore dealing with many different log files. You can change where temporary files are dropped by altering the TmpDir = /var/cache/logwatch variable. Also, if you are used to logging to a nonstandard directory (e.g., not /var/log), change the LogDir or add a new one in logwatch.conf. If you want to include archived logfiles (e.g., messages.1.gz or syslog.6.gz) in addition to the logging directory, you use the following option (which is No by default):

Archives = Yes

By default you are set up to check a date range of only yesterday; however, this is configurable to today or all:

Range = All

I'd suggest experimenting until you're happy with the level of output.

Incidentally, if you want, you can also output the report to the console with --output stdout at the command line or Output = stdout in logwatch.conf. This might be useful if you need the salient information quickly and you're in a closed environment where SMTP is not available.

Execution

The file /etc/cron.daily/00logwatch (Figure 1) shows how to adjust the email recipient on Debian-based servers. However, the sendmail method explored earlier is more universally accepted. The cron implementation on other Unix-like flavors varies dramatically, and I've seen unwieldy logwatch scripts run straight from cron. Because logwatch is triggered as a cron job, however, that means you can also run it manually, and because it's a complex shell script, you can change much of its internal functionality.

For now, however, assume the scripts are powerful enough and extensible enough that you don't need to make many changes; instead, you can alter its configuration. A user with sufficient privileges to read the logfile directory is allowed to execute the cron entry. (However, many Unix-like systems consider system logs to be privileged, so to process all of the logs with logwatch, you might need be root.)

The following command runs logwatch manually on Debian and sends a detailed report by email to recipients pulled from config files:

# /usr/sbin/logwatch --output mail

A large number of logfiles could take a few minutes to complete, but on servers whose performance is tuned correctly, it can be pretty much instant for at least two of the three date ranges (yesterday or today, with all being the slowest).

The logwatch report in Figure 2 shows information relating to how the report is configured, when it was run, and how logwatch deals with pluggable authentication modules (PAM) or user logins by default. This section is included at the top of the report because it is key to checking up on the security of your server and makes you aware of which users have managed to elevate their privileges to root with both su and sudo. If root became another user (e.g., for testing), you will also see that information here, if it was user initiated. Anything unexpected in this section will almost certainly warrant further investigation.

The file that cron calls, which is found in /usr/share/ … scripts/logwatch.pl, holds a staggering 1,570 lines of code. If you scroll down a little, you can see where HTML can be added to both the header and the footer of the emailed reports. The logwatch script also mentions a debug option a few times and has a section for processing each logfile. A lot of attention is paid to cleaning up old logwatch temporary directories, and one section checks permissions:

print "File $ThisFile is not readable. Check permissions.";
if ($> != 0) {
  print " You are not running as superuser.";
}

Finally, a plethora of sensible, neat-and-tidy formatting options provide the clarity required.

When specifying which applications you want to include in your morning reports, you can specify a run-time --detail level that allows you to clarify how much noise you want passed down via the reports. The default level of detail is well considered, but if you are otherwise inclined, the options are a positive integer or high, med, low (i.e., 10, 5, 0).

Services

To specify which services should generate a report, you use an /etc/services-type format; logwatch looks through any config files present in /usr/share/ … scripts/services. Along with the default services (ntp, pam, tftp, etc.), you can ask that logwatch run through all the services found on the server (and any logfile-groups, which I'll look at shortly) that have filters:

Service = All

Sometimes saving reports to disk can be more useful than email, because if you have a struggling server exhibiting symptoms of an ailment over a few days, you might be able to diagnose it simply by turning up the debug level for a short period of time and running a command-line diff over the salient sections of the saved reports. To save your files to disk, just add --save filename as a parameter at launch. Incidentally, you might want to try that debug option mentioned in the logwatch.pl script by enabling a debug setting from 0 to 100 at run time with, for example, --debug 25.

The logwatch.conf file lets you select and ignore specific services. To ignore a service, prepend its name with a minus:

Service = "-ftp"
Service = "-bgpd"
Service = "-http"

The syntax of the Service variable takes a few different forms. For example, if you want only to receive information about FTP daemon messages in /var/log/messages and /var/log/xferlog, use the entries:

Service = "ftpd-messages"
Service = "ftpd-xferlog"

The comments in the logwatch.conf file suggest using the options

Service = pam_pwdb
Service = pam

if you only want PAM messages. Another option is to switch off DNS lookups with --numeric so that only IP addresses are displayed and not DNS names.

If you would like to see a number options implemented at the command line, you can enter something like the following command, as shown in the logwatch man pages:

logwatch --service pam_pwdb --range yesterday --detail high--print

This command faithfully checks any user login data that was logged for the previous day and then prints out a high level of detail.

In Figure 3 you can see an example of logwatch showing all upgrades and new packages installed. If you archive your email over a long period of time, you can always search for this information at a later date.

Documentation

Before using logwatch and creating your own filters, you should explore the excellent documentation and a few examples. As touched upon earlier, the logwatch documentation provided in the /usr/share/doc/logwatch/HOWTO-Customize-LogWatch.gz file [2] demonstrates in detail how to customize configuration files and create new service filters. If for some reason your less viewer doesn't uncompress this file, you can try to read it using zless:

# zless /usr/share/doc/logwatch/HOWTO-Customize-LogWatch.gz

The HOWTO file explains that the bundled dist.conf directory contains very specific files relating to your distribution's build requirements. The default.conf directory, however, holds the majority of pertinent configuration files, and under the services directory therein is a specific noise setting for each application.

The ignore.conf file suppresses output matching any Perl-style regular expressions (regex) you add to the file, and it even has a standalone logfile feature that processes /var/log/messages in the event that you have only one logfile to check:

LogFile = messages

This command will run through any filters associated with that logfile, but setting Service = All will run through all your logs, which is probably more pragmatic in most scenarios.

Configuration

In these next two sections, I'll use the Postfix mail server as an example. Because of the vast possibilities available in customizing logwatch, I have only scanned through the configuration options briefly; however, the HOWTO file offers plenty of additional advice should you need it.

If you look at the config file, /usr/share/logwatch/default.conf/services/postfix.conf, you will see that it takes 327 lines of comments and code just to parse the logs for this one application.

Filters

Now you're ready to create your own logwatch filters. The very structure of logwatch makes it what you might call pluggable or modular. By that, I mean that you can keep adding application filters to your heart's content. By writing your own customized versions of filters, the intelligent logwatch reports according to your preferences, with the most suitable level of detail for your needs.

I will now take a look at the Postfix filter example. Even if you don't delve into the /usr/share/logwatch/scripts/services/postfix filter file too deeply, you can see the following pattern-matching rules in place to capture SSL certificate logging errors and events (Listing 1).

push @ignore_list, qr/^Peer verification:/;
push @ignore_list, qr/^Server certificate could not be verified/;
push @ignore_list, qr/^cannot load .SA certificate and key data/;
push @ignore_list, qr/^tlsmgr_cache_run_event/;
push @ignore_list, qr/^SSL_accept/;
push @ignore_list, qr/^SSL_connect:/;
push @ignore_list, qr/^connection (?:closed|established)/;

These lines are much easier to understand than other parts of the script, and I'm pointing this out so you can add more lines to these rules or modify them slightly if you ever have the need.

To get started creating your own filters, I encourage you to look at config and script files held within /usr/share/logwatch. You can then remove anything from the example filters already there that you don't need. On that note, if you make any small mistakes, don't worry, because logwatch cleverly ignores and removes any duplicate values, so you're safe with a typo or two.

To begin, you create a logfile group, which is essentially a list of logfiles, and then you need to create a service filter. You can then save these in /etc/logwatch/conf/services/ with the filename format chris_new_service.conf. Creating a logfile group requires only one config line, as in the following:

LogFile = /var/log/messages
LogFile = /var/log/syslog
LogFile = /var/log/authd.log

The caveat here is that LogFile = messages refers to the logfile group named messages and not the filename, so if you don't keep system syslog messages inside the file /var/log/messages or if you've changed your logging path in logwatch, you should declare a full path.

To allow details to pass between your config files, you can use:

*SharedScriptName = Arguments

These pre-written shared scripts reside in /usr/share/logwatch/scripts/shared. If you look in that directory, you will see a number of existing scripts (Listing 2), and you can provide variables accessible to your filter:

applybinddate* applystddate*
eventlogonlyservice* hosthash*
onlycontains* remove* applyeurodate*
applytaidate* eventlogremoveservice*
hostlist* onlyhost* removeheaders*
applyhttpdate* applyusdate* expandrepeats*

$EnvironmentVariable = Sula

In your config files; you can incorporate Bash scripts as well as Perl scripts; that is, you can trigger (from within the scripts/ directory) any script capable of pattern-matching or text manipulation.

The HOWTO docs provide an insightful filter example (Listing 3) with a perfect amount of detail. You should try tweaking this example as an exercise. If in doubt, reference the filters already in use by other applications for inspiration.

01 #!/bin/bash
02 # This is as nice script that will show you the lines you will
03 # be processing and reporting on.  It will first display the
04 # standard environment variables and then it takes STDIN and
05 # dump it right back out to STDOUT.
06
07 # These are the standard environment variables.  You can define
08 # more in your service config file (see above).
09 echo "Date Range: $LOGWATCH_DATE_RANGE"
10 echo "Detail Level: $LOGWATCH_DETAIL_LEVEL"
11 echo "Temp Dir: $LOGWATCH_TEMP_DIR"
12 echo "Debug Level: $LOGWATCH_DEBUG"
13
14 # Now take STDIN and dump it to STDOUT
15 cat

Once you have written your filter, save the file in the directory /etc/logwatch/scripts/services/ and call it something like name_filter. Naming is important because you then create your config file as name_filter.conf, which would live in the directory /usr/share/logwatch/dist.conf/logfiles/. Its contents might then be LogFile = name.log.1. At first, I was a little lost with file paths, especially because I first used this on a non-Debian system, so if you get stuck, just check the "HOWTO" again for the specifics for your operating system. The file that helped me the most when working on Debian was /usr/share/doc/logwatch/README.Debian.

A word of advice about how logwatch was ported to Debian from Red Hat suggests the easiest way to create your own filters is as follows: "… copy the filterscript from /usr/share/logwatch/scripts to /etc/logwatch/scripts, preserving the hierarchy. Then modify it. The disadvantage is that you will not be able to track upstream changes" [3].

Incidentally, any custom filters you create will apparently be welcomed at the logwatch SourceForge site [4] if there is likely to be any interest in other people in using them.

Conclusions

Despite the complexity of structure and Perl scripts, there's little debate that logwatch is exceptionally well constructed. Even many enterprise applications don't provide such a comprehensive out-of-the-box experience coupled with such varied, extensible, customization options. With some tinkering, even the porting of packages between different operating systems shouldn't be too difficult.

In this article, I hoped to provide you with enough information to get started using logwatch. That said, you might want to keep an eye out for a time where writing your own custom filter becomes a useful addition to your logging arsenal.