The Postman Cometh

The GNU mailing list manager Mailman incorporates archiving, bounce processing, content filtering, digest delivery, and spam filters, among other features. The project written in Python is a professional software tool, and the fan base appreciates its solid operating principles. Version 1.0 was released in mid-1999 [1], followed by 2.0 in 2000. The latest stable branch 2.1 appeared about 12 years ago, Mailman 3.0 is now an almost completely refurbished major version.

New Design

A lot has happened technologically speaking. According to main developer Barry Warsaw [2], the step to version 3.0 was necessary to iron out problematic design decisions from the previous major releases. Since its inception, the software has not just run on Python, it has contributed in some areas to the development of Python. Warsaw's code, for example, contributed to Unicode handling in Python 3 [3]. The core of Mailman, which handles the message processing, now requires Python 3.

The web interface, which is still rather rudimentary in version 2.x, has been replaced by a modern component, and the developers have also created an attractive solution with many useful features for archiving mailing lists. More specifically, the Mailman 3 suite consists of the following components [4]:

• Mailman Core: The core component for message processing with a uniform user model, moderation, interfaces for mailing list archiving, and a REST interface for connecting additional components.
• Postorius: A web interface based on Django for users and list administrators.
• HyperKitty: An archiving component also based on Django for mailing lists, with extensive possibilities for managing historical messages and thread views, an integrated archive search, day and voting functions, and an integrated response option via web interface.
• Mailman Bundler: A collection of scripts for setting up the entire suite easily in virtual Python environments.

Both the web interface for administration and the archive component are optional. Mailman Core can also be integrated directly into other applications via the REST interface or provided with its own web interface. A Python interface is available for the archive component.

Besides HyperKitty (Figure 1), links to MHonArc are included for the publicly accessible Mail-Archive.com, as well as a simple prototype plugin for connecting your own systems. The use of multiple archives at the same time is also possible and can be enabled individually by mailing list.

Relational Database

In previous versions, Mailman used the Python pickle module [5] to serialize and de-serialize Python objects for data storage, which made working with the data they contained more difficult and access from non-Python applications almost impossible.

The major innovation in Mailman 3 and the associated components of the suite is to store data in relational databases such as SQLite 3 (as a simple embedded version), PostgreSQL, and MySQL. However, Mailman Core still has limitations with MySQL.

With Mailman 3, admins can now operate mailing lists under several domains without problem. No domain names were considered when designing version 2, which meant you had to use aliases or each domain had to operate separate Mailman installations. User accounts can now also condense memberships into several mailing lists.

Barry Warsaw emphasized that Mailman no longer sends monthly notifications with passwords in plain text, because version 3 with Mailman Core now only saves password hashes.

Currently, Mailman Core officially supports Postfix and Exim mail servers, with example configurations only for Postfix; Mailman Bundler also expects a Postfix installation. The test installation using Mailman Bundler is also possible without Postfix – or theoretically without a mailer at all. When creating a mailing list, it accesses the postmap tool by default; otherwise, unsightly error messages appear.

Python Mixed

Many server environments still use Python 2, and Python 3 is only available with additionally installed software collections or, as of recently, via the EPEL testing repository [6] with a current RHEL or CentOS 7. Debian 8, for example, has parallel installable packages. For some administrators, this might be the first software package that Python 3 (specifically, Python 3.4) requires. However, only Mailman Core makes this a requirement; Postorius and HyperKitty, on the other hand, currently still need to be run using Python version 2.7 explicitly.

To operate this mixed environment, Mailman Bundler creates virtual environments for each version branch, so required libraries run in parallel, with the flexibility to choose the version. Otherwise, parallel use of the two versions could cause problems, even with distributions that offer all the necessary libraries as normal packages.

Virtualenv implements the virtual environments, which are not virtual machines but merely directory trees containing a few scripts and binaries. Activating such an environment requires environment variables. This translates to practical advantages for installing a development environment and enables an installation on a wide range of distributions. However, packages installed in this way are not installed via the usual package manager and need special attention for updates.

Installation with Bundler

The following procedure is the one I used to install Mailman for this article. However, since I began testing Mailman 3, the project has been brought together on GitLab [7] and includes all the components mentioned, as well as a Mailman suite. The installation process may differ somewhat now, and you should check the most recent documentation.

Mailman Bundler installs most required components. The following are required to start:

• Packages for Python 2.7 (package name is usually python) and Python 3.4 (usually python3 or python34).
• Virtualenv (package name python-virtualenv).
• Pip installer (python-pip).
• GCC for some packages.
• Node.js LESS CSS compiler (node-less).
• The Postfix mail server.

An error occurred during test installations of the Mailman Bundler package 3.0.0 [8] used for this article, but it has already been fixed, so you should be sure to load a current development version.

Installing the test environment is accomplished easily in a subdirectory. Alternatively, you can create a separate user on the system to operate Mailman and simply put the Mailman environment in its home directory.

First, I loaded the current version of Mailman Bundler from the repository, switched to that directory, created a virtual Python environment (for Python 2.7), and activated it:

bzr branch lp:mailman-bundler
cd mailman-bundler
virtualenv venv
source venv/bin/activate

Second, I used the Pip installer to install the Buildout tool.

The installer performs the installation of the components based on the specifications in buildout.cfg:

pip install zc.buildout
buildout

If the installation terminates (e.g., because of a missing library), it is possible to continue until it is complete by restarting buildout.

Buildout was not able to install the python-social-auth component in my tests. Although this problem has been solved [9], at the time of publication it had not yet been added to the library. Simply install the package manually and then continue with the installation:

pip install python-social-auth

Manual installation can also be repeated for other components, as required.

Buildout creates a virtual environment for Python 3.4 during the installation process, which can be enabled as required with:

source venv-3.4/bin/activate

The components' start scripts ensure use of the respective environment.

Initialize the database for the Django-based web components and create a superuser:

./bin/mailman-post-update
./bin/mailman-web-django-admin   createsuperuser

Make sure to configure a superuser email address; otherwise, at least in the tested version, you'll get mysterious error messages when using the web interface.

To operate the services in the test environment, start:

./bin/mailman start
./bin/mailman-web-django-admin runserver &

New services that can only be accessed from the local machine via the IP address 127.0.0.1 are running on three TCP ports. Ports 8001 and 8024 appeared in a netstat listing in connection with a Python 3.4 process. This is Mailman Core, which connects the REST API on port 8001 and the local mail transfer protocol (LMTP) connection for the mail server's incoming mail on port 8024.

The Django web interface is accessible on port 8000 and presents the interface to the user. You can supply this port externally through a firewall or proxy rule or use it locally first in the environment only intended for tests.

Email Connection to Postfix

An example configuration is available as an aid for connecting to Postfix. Mailman Bundler adapts the paths contained therein to the local environment. Unfortunately, it is not possible to download further settings via include files in Postfix's main configuration file.

Therefore, you need to add them to the existing configuration, usually in /etc/postfix/main.cf. Three lines with references to files that Mailman Core manages are included. They are automatically maintained when creating a new mailing list. Mailman automatically accesses the postmap command after the update, because the files are hashed instead of in plain text.

Getting Started

You can log on to the front end via a web browser. The two standalone web applications Postorius and HyperKitty have a uniform look and are rudimentarily cross-linked by Mailman Bundler during the installation. Fundamentally, however, both systems can be operated separately and independently of Mailman Core.

To add a domain to the system, you use the management interface Postorius (Figure 2) then add lists to the domain in another step. If any error messages appear after creating a list, you are often missing a postmap or running with restricted file permissions.

List Configuration

The developers' aim was to have as many features of version 2 in Mailman 3 as possible, including many of the core functions. Their years of experience in dealing with mailing lists has been beneficial: The user has extensive list configuration options.Mailman can accept, reject, or modify messages from mailing list (non-)subscribers.

Other list options include whether to add a prefix to the Subject line, whether to set a Reply To, or whether to hide the original sender of the message. The structure of the messages can be cleaned up (only accept the first part of multipart/alternative) or converted to a text-only version. For archiving, the admin individually defines several archiver plugins per list from multiple preconfigured plugins, which should receive the new messages via the list.

After enabling the HyperKitty archiver and adding a few subscribers to the mailing list (Figure 3), the basic configuration is ready to send a first message to the list.

HyperKitty

If everything works, Postfix accepts the message, finds the path to Mailman via the autoconfigured transports, and forwards the message via LMTP on port 8024. If configured accordingly, the message is distributed to subscribers and also forwarded to HyperKitty.

The configured mailing list has now been published on the web interface start screen. Clicking here leads to the Summary page, which provides a first impression of the available features. Anyone who has worked with the Mailman 2 archive will be in for a surprise at this point: HyperKitty is visually and functionally a highlight in the Mailman 3 suite.

The latest discussions (Figure 4), statistics on message frequency and participants, popularity ratings for discussions, and personal lists of discussions marked as favorites or discussions in which the user has participated are shown in the overview.

The discussion itself shows the message contents; the user can respond directly on the web interface, mark messages with thumbs up or thumbs down, place a favorite tag, and add tags to the discussion. In addition to search capabilities on tags, you also get an integrated full-text search.

Selecting an Environment

The environment installed using Mailman Bundler starts in a mode intended for development and testing purposes by default. It is possible to switch to production mode in the file buildout.cfg using deployment. This causes a switch in the Mailman web components (Postorius and HyperKitty) from the settings file mailman_web/development.py to mailman_web/production.py. In the production file, you must set the SECRET _KEY setting to a value of your choice. This key is used for communication between the components of the Mailman suite. Any changes you make can be implemented by accessing buildout again.

By changing the configuration file, the default configuration then suggests switching from SQLite 3 to PostgreSQL. MySQL is also available for Django-based web components. In Mailman Core, this does not officially work, and tests showed that adjustments are indeed still needed.The PostgreSQL libraries' headers and a Python module are still required to operate using PostgreSQL. Administrators typically install the PostgreSQL libraries and the server with packages directly from the respective distribution. The Pip installer for the virtual environment is used again for the Python package:

sudo dnf install -y postgresql-devel
pip install psycopg2

The database can be initialized or updated via Django for the web components using syncdb. You can also switch from the SQLite 3 database to a MySQL database in the test installation

bin/django-admin dumpdata >dump.json

or adjust DB settings in the configuration and create the database:

bin/django-admin syncdb
bin/django-admin loaddata dump.json

SQLite 3 should be an easy option for a small installation. The Mailman developers would like to get more feedback about the configuration in practice. They say that locking problems, which should not occur with a huge database engine, have been observed. Accordingly, the official recommendation for production operation is to use PostgreSQL as a database.

The integrated web server previously used in the test environment is also not intended for production use. A web server such as Apache or Nginx linked to the Python components via the web server gateway interface (WSGI) or operating the standalone web server Gunicorn linked to the outside world via proxy functionality is recommended in practice. Apache with WSGI requires the mod_wsgi module and Nginx the uwsgi module.

No Package Service

Although Mailman 3 has now been available since May 2015, there are still no easily installable packages for the major distributions, although everything has been collected together at GitLab, as previously mentioned. The following packages must be installed from the repository along with all associated dependencies:

• mailman3,
• mailman3-hyperkitty (an adapter between Mailman 3 and HyperKitty),
• hyperkitty,
• postorius,

Additionally, the packages python-whoosh and python-BeautifulSoup – which should have been installed automatically as dependencies – must be present. Apache (httpd) is used as a web server, and the mod_wsgi module is for Python adaptations.

The packages install the startup scripts for the services, as well as configurations for logrotate and Postorius and HyperKitty in /etc/httpd/conf.d/. There you will find the configuration for the WSGI connection:

WSGIScriptAlias /hyperkitty   /etc/hyperkitty/sites/default/wsgi.py
WSGIDaemonProcess hyperkitty   threads=25 python-path=  /etc/hyperkitty/sites/default

When working with the pre-made packages, you only have to correct a few minor errors in postorius.conf and hyperkitty.conf. Apache httpd version 2.4.x is used in CentOS 7, so instead of the information contained for Apache 2.2,

Order deny,allow
Allow from all

the administrator needs this 2.4 counterpart:

Require all granted

When installing Postorius and HyperKitty in parallel on the same server, the aliases for /static (and for the favicon.ico) have overlapping configurations. Simply use individual aliases here for the required directories below /static, as shown in Listing 1. The admin and browserid directories from the Postorius package are also used automatically via the eponymous HyperKitty directories.

# for hyperkitty
Alias /static/admin               /var/lib/hyperkitty/sites/default/static/admin
Alias /static/browserid           /var/lib/hyperkitty/sites/default/static/browserid
Alias /static/CACHE               /var/lib/hyperkitty/sites/default/static/CACHE
Alias /static/django_extensions   /var/lib/hyperkitty/sites/default/static/django_extensions
Alias /static/hyperkitty          /var/lib/hyperkitty/sites/default/static/hyperkitty
Alias /static/paintstore          /var/lib/hyperkitty/sites/default/static/paintstore
Alias /static/rest_framework      /var/lib/hyperkitty/sites/default/static/rest_framework
# for postorius
Alias /static/postorius           /var/lib/postorius/sites/default/static/postorius

Another problem with HyperKitty emerged in later tests. At the start, it was possible neither to read its settings nor to create the database or cache files. Unfortunately, in the worst case, I even received a nondescript Internal Server Error, because Mailman 3 does not yet intercept errors cleanly everywhere.

Missing Apache user rights to the corresponding folders from the packages cause this error. A simple workaround is to close those directories to the Apache user,

chown -R apache:apache   /var/lib/hyperkitty/sites/default
chown -R apache:apache   /var/lib/postorius/sites/default

then perform standard tasks such as installing Postfix, supplementing the Postfix configuration (main.cf), installing PostgreSQL (if desired), launching the services, and releasing the ports in the firewall.

Conclusions

The obstacles that still remain when installing Mailman 3 spoil the fun in the new version. The fact that there are still (almost) no packages for major distributions also makes the first few steps more difficult. It is difficult to get rid of the uncomfortable feeling of working with very fresh software despite the "stable" addition.

If the installation is mastered, the basic configuration sorted, and the first message successfully sent via lists, a look at the HyperKitty web interface will compensate for a few hardships. In fact, the Mailman developers have already written a lot of error corrections or urgently required functions on the list for version 3.1. Additional features, such as connecting Mailman Core via NNTP, which Mailman 2 could do, are firmly on the wish list but have not yet been ported to the new architecture.

Those who can boast experience with Mailman 2 will soon find that Mailman 3 is a suite designed from scratch. If you consider the current version to be a technical preview of a new Mailman era, clear strengths and a lot of potential for the program are apparent. The seven years of work have paid off. The individual sub-components and their interfaces and the clear segregation of duties in Mailman 3 make complete sense from a design standpoint. Modularity will also allow custom integrations in the future.