Mailman¶

License: GPLv3

This guide is tested with Mailman 3.3.10 on Uberspace 8.0.76. We can't guarantee it to work with newer versions.

Mailman 3 is a widely used mailing list manager. It consists of the mail-processing core, the Postorius web interface for administration and self-service subscription, and the HyperKitty web archiver.

Due to the architecture of our mail system, this guide receives list mail from a catch-all mailbox over IMAP and sends outgoing mail through the local sendmail. Everything runs from a private Python installation as systemd user services.

Note

For this guide you should be familiar with the basic concepts of:

Warning

A mailing list needs to send mail to the outside world. Trial accounts are limited here, so add credit to your account before running a list in production.

We use a dedicated subdomain for the lists throughout this guide: lists.isabell.uber.space. Replace it with your own domain everywhere.

Prerequisites¶

Mail domain and catch-all¶

Add a mail domain for your lists and a catch-all mailbox that receives mail for every list address (mylist@…, mylist-join@…, mylist-bounces@…, …). Note down the password — you will need it later.

[isabell@moondust ~]$ uberspace mail domain add lists.isabell.uber.space
OK: Added maildomain 'lists.isabell.uber.space' to your Asteroid
[isabell@moondust ~]$ uberspace mail address add catchall@lists.isabell.uber.space --catchall --password-random

Web domain¶

Add the same subdomain as a web domain so Postorius and HyperKitty are reachable:

[isabell@moondust ~]$ uberspace web domain add lists.isabell.uber.space

Python¶

Mailman needs Python 3.12; Uberspace ships a newer Python that Mailman does not support yet. Install a private Python 3.12 with uv:

[isabell@moondust ~]$ uv python install 3.12
Installed Python 3.12.13 in 2.00s

Installation¶

Create a working directory, a virtual environment with Python 3.12, and install Mailman core, the web components, and the SCSS compiler:

[isabell@moondust ~]$ mkdir -p ~/bin ~/mailman/bin ~/mailman/web/logs ~/mailman/var ~/mailman/secrets
[isabell@moondust ~]$ chmod 700 ~/mailman/secrets
[isabell@moondust ~]$ uv venv --python 3.12 --seed ~/mailman/venv
[isabell@moondust ~]$ ~/mailman/venv/bin/pip install --upgrade pip
[isabell@moondust ~]$ ~/mailman/venv/bin/pip install mailman mailman-web mailman-hyperkitty gunicorn libsass
[…]
Successfully installed mailman-3.3.10 mailman-web-0.0.9 postorius-1.3.13 hyperkitty-1.3.12 […]

Link the mailman and mailman-web commands into ~/bin, which is on your PATH, so you can run them directly:

[isabell@moondust ~]$ ln -s ~/mailman/venv/bin/mailman ~/bin/mailman
[isabell@moondust ~]$ ln -s ~/mailman/venv/bin/mailman-web ~/bin/mailman-web

Configuration¶

Secrets¶

Generate three random secrets and keep them handy — you will paste them into the config files below:

[isabell@moondust ~]$ echo "REST password: $(openssl rand -hex 16)"
[isabell@moondust ~]$ echo "Archiver key:  $(openssl rand -hex 20)"
[isabell@moondust ~]$ echo "Django key:    $(openssl rand -hex 32)"

We keep all secrets together in ~/mailman/secrets, so store the catch-all mailbox password (from the prerequisites) there in a file the IMAP bridge can read:

[isabell@moondust ~]$ nano ~/mailman/secrets/imap_password
[isabell@moondust ~]$ chmod 600 ~/mailman/secrets/imap_password

Mailman core¶

NullMTA disables the MTA integration (we handle mail ourselves), and outgoing mail is sent to the local relay you set up further below. Create the core configuration ~/mailman/mailman.cfg, copy and paste the contents of the code box:

[mailman]
site_owner: sysmail@isabell.uber.space
layout: here

[paths.here]
var_dir: /home/isabell/mailman/var

[database]
class: mailman.database.sqlite.SQLiteDatabase
url: sqlite:////home/isabell/mailman/var/data/mailman.db

[webservice]
hostname: 127.0.0.1
port: 8001
admin_user: restadmin
admin_pass: <REST password>

[mta]
incoming: mailman.mta.null.NullMTA
outgoing: mailman.mta.deliver.deliver
lmtp_host: 127.0.0.1
lmtp_port: 8024
smtp_host: 127.0.0.1
smtp_port: 8025
smtp_secure_mode: smtp

[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /home/isabell/mailman/hyperkitty.cfg

Link this file to ~/.mailman.cfg so the mailman command finds it automatically:

[isabell@moondust ~]$ ln -s ~/mailman/mailman.cfg ~/.mailman.cfg

Create the archiver configuration ~/mailman/hyperkitty.cfg, copy and paste the contents of the code box; the api_key must match the archiver key you generated:

[general]
base_url: http://127.0.0.1:8967/hyperkitty
api_key: <Archiver key>

Mail bridges¶

Because our mail system does not deliver mail to programs, two small helpers connect Mailman to it.

The outgoing relay hands Mailman's mail to the local sendmail. Create the script ~/mailman/bin/smtp2sendmail.py, copy and paste the contents of the code box:

smtp2sendmail.py

#!/usr/bin/env python3
"""Relay mail from Mailman to the local sendmail (Uberspace's msmtp)."""
import subprocess
import time

from aiosmtpd.controller import Controller

SENDMAIL = "/usr/sbin/sendmail"


class Handler:
    async def handle_DATA(self, server, session, envelope):
        args = [SENDMAIL, "-oi", "-f", envelope.mail_from or "", "--", *envelope.rcpt_tos]
        proc = subprocess.run(args, input=envelope.content)
        if proc.returncode != 0:
            return "451 local sendmail failed"
        return "250 Message accepted for delivery"


if __name__ == "__main__":
    Controller(Handler(), hostname="127.0.0.1", port=8025).start()
    while True:
        time.sleep(3600)

The incoming bridge collects list mail from the catch-all mailbox with IMAP IDLE and injects it into Mailman's LMTP listener. Create the script ~/mailman/bin/imap2lmtp.py and copy and paste the contents of the code box, adjusting IMAP_USER and LIST_DOMAIN to your subdomain:

imap2lmtp.py

#!/usr/bin/env python3
"""Deliver list mail from the catch-all mailbox into Mailman via LMTP."""
import email
import imaplib
import smtplib
import ssl
import time
from email.utils import getaddresses, parseaddr

IMAP_HOST = "moondust.uberspace.de"
IMAP_USER = "catchall@lists.isabell.uber.space"
IMAP_PASSWORD = open("/home/isabell/mailman/secrets/imap_password").read().strip()
LIST_DOMAIN = "lists.isabell.uber.space"
LMTP_HOST, LMTP_PORT = "127.0.0.1", 8024


def list_recipients(message):
    found = set()
    for header in ("To", "Cc", "X-Original-To", "Delivered-To"):
        for _name, addr in getaddresses(message.get_all(header, [])):
            addr = addr.lower().strip()
            if addr.endswith("@" + LIST_DOMAIN) and not addr.startswith("catchall@"):
                found.add(addr)
    return sorted(found)


def deliver(raw, recipients, sender):
    lmtp = smtplib.LMTP(LMTP_HOST, LMTP_PORT)
    try:
        lmtp.sendmail(sender or ("nobody@" + LIST_DOMAIN), recipients, raw)
    finally:
        try:
            lmtp.quit()
        except smtplib.SMTPException:
            pass


def process(imap):
    _typ, data = imap.search(None, "UNSEEN")
    for num in data[0].split():
        _typ, fetched = imap.fetch(num, "(RFC822)")
        raw = fetched[0][1]
        message = email.message_from_bytes(raw)
        recipients = list_recipients(message)
        sender = parseaddr(message.get("Return-Path") or message.get("From") or "")[1]
        if recipients:
            try:
                deliver(raw, recipients, sender)
            except smtplib.SMTPRecipientsRefused:
                pass  # no matching list -> drop
            except smtplib.SMTPException:
                continue  # temporary failure -> keep and retry later
        imap.store(num, "+FLAGS", "(\\Deleted)")
    imap.expunge()


def idle(imap, timeout=1500):
    tag = imap._new_tag()
    imap.send(b"%s IDLE\r\n" % tag)
    if not imap.readline().startswith(b"+"):
        return
    imap.sock.settimeout(timeout)
    try:
        while True:
            line = imap.readline()
            if not line or b"EXISTS" in line or b"RECENT" in line:
                break
    except (ssl.SSLError, OSError):
        pass
    finally:
        imap.send(b"DONE\r\n")
        imap.sock.settimeout(None)
        while True:
            line = imap.readline()
            if line.startswith(tag) or not line:
                break


def main():
    while True:
        try:
            imap = imaplib.IMAP4_SSL(IMAP_HOST, 993, ssl_context=ssl.create_default_context())
            imap.login(IMAP_USER, IMAP_PASSWORD)
            imap.select("INBOX")
            process(imap)
            while True:
                idle(imap)
                imap.select("INBOX")
                process(imap)
        except Exception:
            time.sleep(30)  # reconnect on any error


if __name__ == "__main__":
    main()

Web interface¶

Create the Django configuration for Postorius and HyperKitty at ~/mailman/web/settings.py. Static files must live under your document root, because your home directory is not readable by the web server. Copy and paste the contents of the code box, filling in the secrets and your domain:

import os

from mailman_web.settings.base import *          # noqa
from mailman_web.settings.mailman import *        # noqa

SECRET_KEY = "<Django key>"
DEBUG = False
ALLOWED_HOSTS = ["lists.isabell.uber.space", "localhost", "127.0.0.1"]
CSRF_TRUSTED_ORIGINS = ["https://lists.isabell.uber.space"]
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")

DATABASES = {"default": {"ENGINE": "django.db.backends.sqlite3",
                         "NAME": "/home/isabell/mailman/web/mailman-web.db"}}

MAILMAN_REST_API_URL = "http://127.0.0.1:8001"
MAILMAN_REST_API_USER = "restadmin"
MAILMAN_REST_API_PASS = "<REST password>"
MAILMAN_ARCHIVER_KEY = "<Archiver key>"
MAILMAN_ARCHIVER_FROM = ("127.0.0.1", "::1")

STATIC_ROOT = "/var/www/virtual/" + os.environ["USER"] + "/html/static"
STATIC_URL = "/static/"
COMPRESS_OFFLINE = True
COMPRESS_PRECOMPILERS = (
    ("text/x-scss", "/home/isabell/mailman/venv/bin/pysassc {infile} {outfile}"),
    ("text/x-sass", "/home/isabell/mailman/venv/bin/pysassc {infile} {outfile}"),
)

HAYSTACK_CONNECTIONS = {"default": {"ENGINE": "haystack.backends.whoosh_backend.WhooshEngine",
                                    "PATH": "/home/isabell/mailman/web/fulltext_index"}}

EMAIL_HOST = "127.0.0.1"
EMAIL_PORT = 8025
DEFAULT_FROM_EMAIL = "lists@isabell.uber.space"
SERVER_EMAIL = "lists@isabell.uber.space"
POSTORIUS_TEMPLATE_BASE_URL = "http://127.0.0.1:8967"
SITE_ID = 1

LOGGING = {"version": 1, "disable_existing_loggers": False,
           "handlers": {"file": {"class": "logging.FileHandler",
                                 "filename": "/home/isabell/mailman/web/logs/web.log"}},
           "root": {"handlers": ["file"], "level": "INFO"}}

Initialise the database, set the site domain, create an administrator, and build the static files:

[isabell@moondust ~]$ export MAILMAN_WEB_CONFIG=$HOME/mailman/web/settings.py
[isabell@moondust ~]$ mailman-web migrate
[isabell@moondust ~]$ mailman-web shell -c "from django.contrib.sites.models import Site; s=Site.objects.get(pk=1); s.domain='lists.isabell.uber.space'; s.name='My Lists'; s.save()"
[isabell@moondust ~]$ mailman-web createsuperuser
[isabell@moondust ~]$ mailman-web collectstatic --noinput
[isabell@moondust ~]$ mailman-web compress --force

Symlink the log for easy access:

[isabell@moondust ~]$ ln -s ~/mailman/web/logs/web.log ~/logs/mailman-web.log

Services¶

Register the five components as systemd user services:

[isabell@moondust ~]$ uberspace service add mailman-core "$HOME/mailman/venv/bin/master --force" -e MAILMAN_CONFIG_FILE=$HOME/mailman/mailman.cfg -w $HOME/mailman
[isabell@moondust ~]$ uberspace service add mailman-relay "$HOME/mailman/venv/bin/python $HOME/mailman/bin/smtp2sendmail.py" -w $HOME/mailman
[isabell@moondust ~]$ uberspace service add mailman-imap "$HOME/mailman/venv/bin/python $HOME/mailman/bin/imap2lmtp.py" -w $HOME/mailman
[isabell@moondust ~]$ uberspace service add mailman-web "$HOME/mailman/venv/bin/gunicorn --bind 0.0.0.0:8967 --workers 2 mailman_web.wsgi:application" -e MAILMAN_WEB_CONFIG=$HOME/mailman/web/settings.py -w $HOME/mailman/web
[isabell@moondust ~]$ uberspace service add mailman-qcluster "$HOME/mailman/venv/bin/mailman-web qcluster" -e MAILMAN_WEB_CONFIG=$HOME/mailman/web/settings.py -w $HOME/mailman/web

Web backends¶

Send dynamic requests to the web interface and serve the static files directly from the document root:

[isabell@moondust ~]$ uberspace web backend add lists.isabell.uber.space PORT 8967
[isabell@moondust ~]$ uberspace web backend add lists.isabell.uber.space/static/ STATIC

Your Mailman is now available at https://lists.isabell.uber.space/.

Create your first list¶

Create a list on the command line, then manage it in Postorius:

[isabell@moondust ~]$ mailman create announce@lists.isabell.uber.space
Created mailing list: announce@lists.isabell.uber.space

Log in to https://lists.isabell.uber.space/ with the administrator account you created and configure the list, subscribers, and archiving from there.

Cronjobs¶

Mailman sends digests and moderator reminders from cron. Open your crontab:

[isabell@moondust ~]$ crontab -e

Add the following jobs. The -C option points at your configuration, because cron runs without your shell environment:

@daily $HOME/mailman/venv/bin/mailman -C $HOME/mailman/mailman.cfg digests --periodic
@hourly $HOME/mailman/venv/bin/mailman -C $HOME/mailman/mailman.cfg notify

Debugging¶

Check the services and their logs:

[isabell@moondust ~]$ systemctl --user status mailman-core mailman-web
[isabell@moondust ~]$ journalctl --user -u mailman-core -f
[isabell@moondust ~]$ tail -f ~/mailman/var/logs/mailman.log

Note

Uberspace filters outgoing mail for spam. Well-formed messages sent by Mailman pass, but hand-crafted test mails may be rejected with 554 Spam message rejected. Test by posting a real message through a subscribed address.

Updates¶

Note

Watch the Mailman announcements and the PyPI release feeds to stay informed about new versions.

Upgrade the packages, apply migrations, rebuild the static files, and restart the services:

[isabell@moondust ~]$ export MAILMAN_WEB_CONFIG=$HOME/mailman/web/settings.py
[isabell@moondust ~]$ ~/mailman/venv/bin/pip install --upgrade mailman mailman-web mailman-hyperkitty
[isabell@moondust ~]$ mailman-web migrate
[isabell@moondust ~]$ mailman-web collectstatic --noinput
[isabell@moondust ~]$ mailman-web compress --force
[isabell@moondust ~]$ systemctl --user restart mailman-core mailman-web mailman-qcluster mailman-relay mailman-imap

Mailman¶

Prerequisites¶

Mail domain and catch-all¶

Web domain¶

Python¶

Installation¶

Configuration¶

Secrets¶

Mailman core¶

Mail bridges¶

Web interface¶

Services¶

Web backends¶

Create your first list¶

Cronjobs¶

Debugging¶

Updates¶

Further Reading¶