Mailman¶

License: GPLv3

This guide is tested with Mailman 3.3.10 on Uberspace 8.0.93. We can't guarantee it to work with newer versions.

Mailman 3 is a widely used mailing list manager. It consists of the mail-processing core, the Postorius web interface for administration and self-service subscription, and the HyperKitty web archiver.

Due to the architecture of our mail system, this guide receives list mail from a catch-all mailbox over IMAP and sends outgoing mail through the local sendmail. Everything runs from a Python virtual environment as systemd user services.

Note

For this guide you should be familiar with the basic concepts of:

Warning

Add credit to your account before testing or running a mailing list. Trial accounts have a mail quota that can interrupt list delivery and web-account verification, returning 451 ... You are in your trial period. Mailman queues list mail for retry when this happens.

We use a dedicated subdomain for the lists throughout this guide: lists.isabell.uber.space. Replace it with your own domain everywhere. Also replace /home/isabell with your home directory and moondust.uberspace.de with your hostname in the configuration files.

Prerequisites¶

Mail domain and catch-all¶

Add a mail domain for your lists and a catch-all mailbox that receives mail for every list address (mylist@…, mylist-join@…, mylist-bounces@…, …). Note down the password — you will need it later.

[isabell@moondust ~]$ uberspace mail domain add lists.isabell.uber.space
OK: Added maildomain 'lists.isabell.uber.space' to your Asteroid
[isabell@moondust ~]$ uberspace mail address add catchall@lists.isabell.uber.space --catchall --password-random

Web domain¶

Add the same subdomain as a web domain so Postorius and HyperKitty are reachable:

[isabell@moondust ~]$ uberspace web domain add lists.isabell.uber.space

Python¶

This guide uses Python 3.12, which Uberspace provides as python3.12. Keep Mailman and its dependencies in a virtual environment, separate from the system Python.

Installation¶

Create a working directory, a virtual environment with Python 3.12, and install Mailman core, the web components, and the SCSS compiler:

[isabell@moondust ~]$ mkdir -p ~/bin ~/mailman/bin ~/mailman/web/logs ~/mailman/var ~/mailman/secrets
[isabell@moondust ~]$ chmod 700 ~/mailman/secrets
[isabell@moondust ~]$ uv venv --python /usr/bin/python3.12 ~/mailman/venv
[isabell@moondust ~]$ uv pip install --python ~/mailman/venv/bin/python mailman mailman-web mailman-hyperkitty gunicorn libsass
[…]

Link the mailman and mailman-web commands into ~/bin, which is on your PATH, so you can run them directly:

[isabell@moondust ~]$ ln -s ~/mailman/venv/bin/mailman ~/bin/mailman
[isabell@moondust ~]$ ln -s ~/mailman/venv/bin/mailman-web ~/bin/mailman-web

Configuration¶

Secrets¶

Generate three random secrets and keep them handy — you will paste them into the config files below:

[isabell@moondust ~]$ echo "REST password: $(openssl rand -hex 16)"
[isabell@moondust ~]$ echo "Archiver key:  $(openssl rand -hex 20)"
[isabell@moondust ~]$ echo "Django key:    $(openssl rand -hex 32)"

Store the catch-all mailbox password (from the prerequisites) in ~/mailman/secrets/imap_password, which the IMAP bridge reads. The other secrets go in the configuration files below:

[isabell@moondust ~]$ nano ~/mailman/secrets/imap_password
[isabell@moondust ~]$ chmod 600 ~/mailman/secrets/imap_password

Mailman core¶

NullMTA disables the MTA integration (we handle mail ourselves), and outgoing mail is sent to the local relay you set up further below. Create the core configuration ~/mailman/mailman.cfg, copy and paste the contents of the code box:

[mailman]
site_owner: sysmail@isabell.uber.space
layout: here

[paths.here]
var_dir: /home/isabell/mailman/var

[database]
class: mailman.database.sqlite.SQLiteDatabase
url: sqlite:////home/isabell/mailman/var/data/mailman.db

[webservice]
hostname: 127.0.0.1
port: 8001
admin_user: restadmin
admin_pass: <REST password>

[mta]
incoming: mailman.mta.null.NullMTA
outgoing: mailman.mta.deliver.deliver
lmtp_host: 127.0.0.1
lmtp_port: 8024
smtp_host: 127.0.0.1
smtp_port: 8025
smtp_secure_mode: smtp

[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /home/isabell/mailman/hyperkitty.cfg

Link this file to ~/.mailman.cfg so the mailman command finds it automatically:

[isabell@moondust ~]$ ln -s ~/mailman/mailman.cfg ~/.mailman.cfg

Create the archiver configuration ~/mailman/hyperkitty.cfg, copy and paste the contents of the code box; the api_key must match the archiver key you generated:

[general]
base_url: http://127.0.0.1:8967/hyperkitty
api_key: <Archiver key>

Mail bridges¶

Because our mail system does not deliver mail to programs, two small helpers connect Mailman to it.

The outgoing relay hands Mailman's mail to the local sendmail. Create the script ~/mailman/bin/smtp2sendmail.py, copy and paste the contents of the code box:

smtp2sendmail.py

#!/usr/bin/env python3
"""Relay mail from Mailman to the local sendmail (Uberspace's msmtp)."""
import subprocess
import time

from aiosmtpd.controller import Controller

SENDMAIL = "/usr/sbin/sendmail"


class Handler:
    async def handle_DATA(self, server, session, envelope):
        args = [SENDMAIL, "-oi", "-f", envelope.mail_from or "", "--", *envelope.rcpt_tos]
        proc = subprocess.run(args, input=envelope.content)
        if proc.returncode != 0:
            return "451 local sendmail failed"
        return "250 Message accepted for delivery"


if __name__ == "__main__":
    Controller(Handler(), hostname="127.0.0.1", port=8025).start()
    while True:
        time.sleep(3600)

The incoming bridge collects list mail from the catch-all mailbox every 30 seconds and injects it into Mailman's LMTP listener. Create the script ~/mailman/bin/imap2lmtp.py and copy and paste the contents of the code box, adjusting IMAP_USER and LIST_DOMAIN to your subdomain:

The bridge reads the original recipient from the local Postfix delivery header, so Bcc posts and bounce addresses work too. Send to one list address per message. If the delivery header has no unambiguous recipient, the bridge keeps the message in the catch-all mailbox and logs a warning for manual inspection.

imap2lmtp.py

#!/usr/bin/env python3
"""Deliver list mail from the catch-all mailbox into Mailman via LMTP."""
import email
import imaplib
import logging
import re
import smtplib
import ssl
import time
from email.utils import parseaddr

IMAP_HOST = "moondust.uberspace.de"
IMAP_USER = "catchall@lists.isabell.uber.space"
IMAP_PASSWORD = open("/home/isabell/mailman/secrets/imap_password").read().strip()
LIST_DOMAIN = "lists.isabell.uber.space"
LMTP_HOST, LMTP_PORT = "127.0.0.1", 8024


def list_recipients(message):
    # Dovecot adds the first Received header; the local Postfix adds the second.
    # Only trust that local hop, never To/Cc or sender-supplied trace headers.
    received = message.get_all("Received", [])
    if len(received) < 2:
        return []
    match = re.search(
        r"\bby\s+" + re.escape(IMAP_HOST) + r"\s+\(Postfix\).*?\bfor\s+<([^<>]+)>;",
        received[1], re.IGNORECASE | re.DOTALL,
    )
    if match:
        address = match[1].lower()
        if address.endswith("@" + LIST_DOMAIN) and address != IMAP_USER.lower():
            return [address]
    return []


def deliver(raw, recipients, sender):
    lmtp = smtplib.LMTP(LMTP_HOST, LMTP_PORT, timeout=30)
    try:
        lmtp.sendmail(sender, recipients, raw)
    finally:
        try:
            lmtp.quit()
        except smtplib.SMTPException:
            pass


def process(imap):
    _typ, data = imap.search(None, "UNSEEN")
    for num in data[0].split():
        _typ, fetched = imap.fetch(num, "(BODY.PEEK[])")
        raw = fetched[0][1]
        message = email.message_from_bytes(raw)
        recipients = list_recipients(message)
        sender = parseaddr(message.get("Return-Path") or message.get("From") or "")[1]
        if not recipients:
            logging.warning("Cannot determine envelope recipient; keeping message %s", num)
            continue
        try:
            deliver(raw, recipients, sender)
        except smtplib.SMTPRecipientsRefused as exc:
            if any(code < 500 for code, _ in exc.recipients.values()):
                continue  # temporary refusal -> retry later
        except smtplib.SMTPException:
            continue  # temporary failure -> keep and retry later
        imap.store(num, "+FLAGS", "(\\Deleted)")
    imap.expunge()


def main():
    while True:
        try:
            with imaplib.IMAP4_SSL(
                IMAP_HOST, 993, ssl_context=ssl.create_default_context(), timeout=30
            ) as imap:
                imap.login(IMAP_USER, IMAP_PASSWORD)
                imap.select("INBOX")
                process(imap)
        except (OSError, imaplib.IMAP4.error, smtplib.SMTPException):
            logging.exception("Mail delivery failed; retrying in 30 seconds")
        time.sleep(30)


if __name__ == "__main__":
    main()

Web interface¶

Create the Django configuration for Postorius and HyperKitty at ~/mailman/web/settings.py. Static files must live under your document root, because your home directory is not readable by the web server. Copy and paste the contents of the code box, filling in the secrets and your domain:

import os

from mailman_web.settings.base import *          # noqa
from mailman_web.settings.mailman import *        # noqa

SECRET_KEY = "<Django key>"
DEBUG = False
ALLOWED_HOSTS = ["lists.isabell.uber.space", "localhost", "127.0.0.1"]
CSRF_TRUSTED_ORIGINS = ["https://lists.isabell.uber.space"]
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")

DATABASES = {"default": {"ENGINE": "django.db.backends.sqlite3",
                         "NAME": "/home/isabell/mailman/web/mailman-web.db"}}

MAILMAN_REST_API_URL = "http://127.0.0.1:8001"
MAILMAN_REST_API_USER = "restadmin"
MAILMAN_REST_API_PASS = "<REST password>"
MAILMAN_ARCHIVER_KEY = "<Archiver key>"
MAILMAN_ARCHIVER_FROM = ("127.0.0.1", "::1")

STATIC_ROOT = "/var/www/virtual/" + os.environ["USER"] + "/html/static"
STATIC_URL = "/static/"
COMPRESS_OFFLINE = True
COMPRESS_PRECOMPILERS = (
    ("text/x-scss", "/home/isabell/mailman/venv/bin/pysassc {infile} {outfile}"),
    ("text/x-sass", "/home/isabell/mailman/venv/bin/pysassc {infile} {outfile}"),
)

HAYSTACK_CONNECTIONS = {"default": {"ENGINE": "haystack.backends.whoosh_backend.WhooshEngine",
                                    "PATH": "/home/isabell/mailman/web/fulltext_index"}}

EMAIL_HOST = "127.0.0.1"
EMAIL_PORT = 8025
DEFAULT_FROM_EMAIL = "lists@isabell.uber.space"
SERVER_EMAIL = "lists@isabell.uber.space"
POSTORIUS_TEMPLATE_BASE_URL = "http://127.0.0.1:8967"
SITE_ID = 1

LOGGING = {"version": 1, "disable_existing_loggers": False,
           "handlers": {"file": {"class": "logging.FileHandler",
                                 "filename": "/home/isabell/mailman/web/logs/web.log"}},
           "root": {"handlers": ["file"], "level": "INFO"}}

Protect the configuration files containing secrets:

[isabell@moondust ~]$ chmod 600 ~/mailman/mailman.cfg ~/mailman/hyperkitty.cfg ~/mailman/web/settings.py

Initialise the database, set the site domain, create an administrator, and build the static files:

[isabell@moondust ~]$ export MAILMAN_WEB_CONFIG=$HOME/mailman/web/settings.py
[isabell@moondust ~]$ mailman-web migrate
[isabell@moondust ~]$ mailman-web shell -c "from django.contrib.sites.models import Site; s=Site.objects.get(pk=1); s.domain='lists.isabell.uber.space'; s.name='My Lists'; s.save()"
[isabell@moondust ~]$ mailman-web createsuperuser
[isabell@moondust ~]$ mailman-web collectstatic --noinput
[isabell@moondust ~]$ mailman-web compress --force

Symlink the log for easy access:

[isabell@moondust ~]$ ln -s ~/mailman/web/logs/web.log ~/logs/mailman-web.log

Services¶

Register the five components as systemd user services:

[isabell@moondust ~]$ uberspace service add mailman-core "$HOME/mailman/venv/bin/master --force" -e MAILMAN_CONFIG_FILE=$HOME/mailman/mailman.cfg -w $HOME/mailman
[isabell@moondust ~]$ uberspace service add mailman-relay "$HOME/mailman/venv/bin/python $HOME/mailman/bin/smtp2sendmail.py" -w $HOME/mailman
[isabell@moondust ~]$ uberspace service add mailman-imap "$HOME/mailman/venv/bin/python $HOME/mailman/bin/imap2lmtp.py" -w $HOME/mailman
[isabell@moondust ~]$ uberspace service add mailman-web "$HOME/mailman/venv/bin/gunicorn --bind 0.0.0.0:8967 --workers 2 mailman_web.wsgi:application" -e MAILMAN_WEB_CONFIG=$HOME/mailman/web/settings.py -w $HOME/mailman/web
[isabell@moondust ~]$ uberspace service add mailman-qcluster "$HOME/mailman/venv/bin/mailman-web qcluster" -e MAILMAN_WEB_CONFIG=$HOME/mailman/web/settings.py -w $HOME/mailman/web

Web backends¶

Send dynamic requests to the web interface and serve the static files directly from the document root:

[isabell@moondust ~]$ uberspace web backend add lists.isabell.uber.space PORT 8967
[isabell@moondust ~]$ uberspace web backend add lists.isabell.uber.space/static/ STATIC

Your Mailman is now available at https://lists.isabell.uber.space/.

Create your first list¶

Create a list on the command line, then manage it in Postorius:

[isabell@moondust ~]$ mailman create announce@lists.isabell.uber.space
Created mailing list: announce@lists.isabell.uber.space

Log in to https://lists.isabell.uber.space/ with the administrator account you created and configure the list, subscribers, and archiving from there.

Cronjobs¶

Mailman sends digests and moderator reminders from cron. Mailman Web also needs scheduled jobs for archive maintenance and search indexing. Open your crontab:

[isabell@moondust ~]$ crontab -e

Add the following jobs. Set MAILMAN_WEB_CONFIG for the web commands and use -C for the core commands, because cron runs without your shell environment:

MAILMAN_WEB_CONFIG=/home/isabell/mailman/web/settings.py

@daily $HOME/mailman/venv/bin/mailman -C $HOME/mailman/mailman.cfg digests --periodic
@hourly $HOME/mailman/venv/bin/mailman -C $HOME/mailman/mailman.cfg notify

* * * * * $HOME/mailman/venv/bin/mailman-web runjobs minutely
*/15 * * * * $HOME/mailman/venv/bin/mailman-web runjobs quarter_hourly
@hourly $HOME/mailman/venv/bin/mailman-web runjobs hourly
@daily $HOME/mailman/venv/bin/mailman-web runjobs daily
@weekly $HOME/mailman/venv/bin/mailman-web runjobs weekly
@monthly $HOME/mailman/venv/bin/mailman-web runjobs monthly
@yearly $HOME/mailman/venv/bin/mailman-web runjobs yearly

Debugging¶

Check the services and their logs:

[isabell@moondust ~]$ systemctl --user status mailman-core mailman-web
[isabell@moondust ~]$ journalctl --user -u mailman-core -f
[isabell@moondust ~]$ tail -f ~/mailman/var/logs/mailman.log

Note

Uberspace filters outgoing mail for spam. Well-formed messages sent by Mailman pass, but hand-crafted test mails may be rejected with 554 Spam message rejected. Test by posting a real message through a subscribed address.

Updates¶

Note

Watch the Mailman announcements and the PyPI release feeds to stay informed about new versions.

Stop the services and back up ~/mailman, which contains your configuration and databases. Then upgrade the packages, apply migrations, rebuild the static files, and restart the services:

[isabell@moondust ~]$ systemctl --user stop mailman-core mailman-web mailman-qcluster mailman-relay mailman-imap
[isabell@moondust ~]$ export MAILMAN_WEB_CONFIG=$HOME/mailman/web/settings.py
[isabell@moondust ~]$ uv pip install --python ~/mailman/venv/bin/python --upgrade mailman mailman-web mailman-hyperkitty
[isabell@moondust ~]$ mailman-web migrate
[isabell@moondust ~]$ mailman-web collectstatic --noinput
[isabell@moondust ~]$ mailman-web compress --force
[isabell@moondust ~]$ systemctl --user restart mailman-core mailman-web mailman-qcluster mailman-relay mailman-imap

Mailman¶

Prerequisites¶

Mail domain and catch-all¶

Web domain¶

Python¶

Installation¶

Configuration¶

Secrets¶

Mailman core¶

Mail bridges¶

Web interface¶

Services¶

Web backends¶

Create your first list¶

Cronjobs¶

Debugging¶

Updates¶

Further Reading¶