left4me: wire LOG_LISTENER_{ADDR,BIND} into web.env

Derives LOG_LISTENER_ADDR from the node's external IPv4 (stripping the
CIDR suffix). The destination MUST be non-loopback — Source silently
drops logaddress UDP destinations in 127.0.0.0/8 (registration
succeeds but no sendto ever happens). LOG_LISTENER_BIND is pinned
to 0.0.0.0:28000 so the listener accepts on every interface,
including the lo-routed packets the kernel uses for same-host
destinations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.envrc

@@ -2,6 +2,6 @@
 
 PATH_add bin
 
-source_env ~/.local/share/direnv/pyenv
+layout uv
-source_env ~/.local/share/direnv/venv
+
 source_env ~/.local/share/direnv/bundlewrap

.gitignore

@@ -2,3 +2,8 @@
 .venv
 .cache
 *.pyc
+.bw_debug_history
+# CocoIndex Code (ccc)
+/.cocoindex_code/
+# bundlewrap git_deploy local-mirror map (operator-specific paths)
+git_deploy_repos

AGENTS.md

@@ -0,0 +1,108 @@
+# ckn-bw — agent & contributor guide
+
+## What this repo is
+
+A [BundleWrap](https://bundlewrap.org/) configuration-management repo
+for ~22 personal/family-infra nodes. Nodes, groups, and bundles are
+defined in plain Python; `bw apply` deploys the resulting state to
+real machines.
+
+Note: the root `README.md` is the maintainer's personal scratchpad,
+not project documentation. Onboarding lives **here**, in `AGENTS.md`.
+
+## Quickstart for agents
+
+Five rules; follow these and you won't break things:
+
+1. **Read-only by default.** Never run `bw apply`, `bw run`, or
+   `bw lock` without explicit user request — even with `-i`. Stick
+   to `bw test`, `bw nodes`, `bw groups`, `bw items`,
+   `bw metadata`, `bw hash`, `bw verify`, `bw debug`. See
+   [`docs/agents/commands.md`](docs/agents/commands.md) and the
+   fork's [safety envelope](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md).
+2. **Never echo decrypted secrets.** Don't print, paste, or log the
+   value behind a `!password_for:`, `!decrypt:`, or
+   `!32_random_bytes_as_base64_for:` magic string — not even from
+   `bw debug` exploration. See
+   [`conventions.md#secrets`](docs/agents/conventions.md#secrets).
+3. **Don't touch the do-not-modify list.** `.secrets.cfg*`, `.venv`,
+   `.cache`, `.bw_debug_history`, root `README.md`. Treat
+   `hooks/` and `items/` (custom item types) with extra care: a
+   broken hook or item type breaks every `bw` command repo-wide.
+4. **Use the fork.** Bundlewrap is pinned to the `main` branch of
+   [`github.com/CroneKorkN/bundlewrap`](https://github.com/CroneKorkN/bundlewrap)
+   via `[tool.uv.sources]` in `pyproject.toml`; `uv sync` (run by
+   direnv on entry) installs it. Behavior tracks the fork's `main`;
+   the fork's
+   [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md)
+   is the canonical bundlewrap-language reference. See
+   [`conventions.md#bundlewrap-version`](docs/agents/conventions.md#bundlewrap-version).
+5. **Prefer adding helpers to `libs/`** over duplicating logic across
+   bundles. Repo-wide helpers go in
+   [`libs/`](libs/AGENTS.md), reachable as `repo.libs.<x>`.
+
+## Layout
+
+| Dir | What's there |
+|---|---|
+| [`bundles/`](bundles/AGENTS.md)              | 103 bundles. One subdir per bundle (`items.py`, `metadata.py`, `files/`). |
+| [`nodes/`](nodes/AGENTS.md)                  | One file per node (~22). `eval()`-loaded; demagified through `repo.vault`. |
+| [`groups/`](groups/AGENTS.md)                | Group definitions, organized by axis (`applications/`, `locations/`, `machine/`, `os/`). |
+| [`libs/`](libs/AGENTS.md)                    | Shared Python helpers reachable as `repo.libs.<modulename>`. |
+| [`hooks/`](hooks/AGENTS.md)                  | bw lifecycle hooks (`apply_start`, `test`, `node_apply_start`, …). |
+| [`data/`](data/AGENTS.md)                    | Out-of-bundle data assets (apt keys, grafana dashboards, …). |
+| [`items/`](items/AGENTS.md)                  | Custom item types (currently `download:`). |
+| [`bin/`](bin/AGENTS.md)                      | Operator scripts; not invoked by bundlewrap. |
+| [`docs/agents/`](docs/agents/conventions.md) | Repo conventions and command deltas. |
+
+## How nodes, groups, and bundles fit together
+
+- A **node** (`nodes/<location>.<role>.py`) declares the groups it
+  belongs to and any node-local bundles + metadata overrides.
+- A **group** (`groups/<axis>/<x>.py`) attaches bundles and shared
+  metadata to its members. Groups inherit via `supergroups`.
+- A **bundle** (`bundles/<x>/`) is one chunk of configuration:
+  `items.py` produces the items (files, services, packages),
+  `metadata.py` declares `defaults` and `@metadata_reactor` functions
+  that derive metadata from other metadata.
+- The repo-root loaders (`nodes.py`, `groups.py`) walk these dirs and
+  `eval()` each file. `nodes.py` additionally **demagifies** the
+  result, resolving `!password_for:` etc. through `repo.vault`. See
+  [`conventions.md#eval-loaded-node-and-group-files`](docs/agents/conventions.md#eval-loaded-node-and-group-files)
+  for the constraints this places on editors.
+- Metadata merges along: `all → location → os → machine →
+  applications → node`.
+
+## Conventions you must know
+
+| Topic | Where |
+|---|---|
+| Bundlewrap-language reference (item types, dep keywords, reactors) | Fork's [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) — read first if new to bundlewrap |
+| Vault / demagify magic strings                                     | [`conventions.md#secrets`](docs/agents/conventions.md#secrets) |
+| Bundlewrap install (uv-pinned to the fork's `main`)                | [`conventions.md#bundlewrap-version`](docs/agents/conventions.md#bundlewrap-version) |
+| Group inheritance order, naming patterns                           | [`conventions.md#group-inheritance-order`](docs/agents/conventions.md#group-inheritance-order), [`#naming-conventions`](docs/agents/conventions.md#naming-conventions) |
+| Repo-specific bw command deltas (apt keys, suspended nodes, vault echo) | [`commands.md`](docs/agents/commands.md) |
+| Lib helpers                                                        | top-of-file docstrings in `libs/*.py` (`head -1 libs/*.py`) |
+| Suspension idioms (`*.py_`, `_old/`, "for now")                    | [`conventions.md#suspension-and-soft-delete-idioms`](docs/agents/conventions.md#suspension-and-soft-delete-idioms) |
+
+## Where to look for examples
+
+When writing a new bundle, copy patterns from one that already does
+the thing you need:
+
+| Pattern                                  | Look at |
+|---|---|
+| Vault calls inside metadata reactors     | `bundles/dm-crypt/metadata.py` (compact, focused) |
+| Mako-templated files                     | `bundles/bind/items.py` (DNS zonefile rendering) |
+| Cross-bundle reactor writing             | `bundles/nextcloud/metadata.py` (writes into `apt.packages`, `archive.paths`) |
+| Custom `download:` items                 | `bundles/minecraft/items.py` |
+| Node file (single-purpose)               | `nodes/home.server.py` |
+| Group with `supergroups` chain           | `groups/os/debian-13.py` |
+
+## Where this doc lives
+
+- This file: `AGENTS.md` at the repo root.
+- `CLAUDE.md` is a symlink to this file — both names point to the same
+  content so different tools can find it.
+- The personal TODO scratchpad (`README.md`) is **separate** and not
+  project documentation.

CLAUDE.md

@@ -0,0 +1 @@
+AGENTS.md

README.md

@@ -13,10 +13,6 @@ Raspberry pi as soundcard
 - OTG g_audio
 - https://audiosciencereview.com/forum/index.php?threads/raspberry-pi-as-usb-to-i2s-adapter.8567/post-215824
 
-# install bw fork
-
-pip3 install --editable git+file:///Users/mwiegand/Projekte/bundlewrap-fork@main#egg=bundlewrap
-
 # monitor timers
 
 ```sh
@@ -37,3 +33,12 @@ fi
 telegraf: execd for daemons
 
 TEST
+
+# git signing
+
+git config --global gpg.format ssh
+git config --global commit.gpgsign true
+
+git config user.name CroneKorkN
+git config user.email i@ckn.li
+git config user.signingkey "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILMVroYmswD4tLk6iH+2tvQiyaMe42yfONDsPDIdFv6I"

bin/AGENTS.md

@@ -0,0 +1,62 @@
+# bin/
+
+## What's here
+
+Operator scripts — invoked manually by the maintainer, **not** by
+bundlewrap itself. Each is a standalone Python (or shell) script that
+opens the repo via `Repository(dirname(dirname(realpath(__file__))))`.
+
+Discovery is by `ls bin/` plus the `# purpose:` header line at the top
+of each script:
+
+```sh
+head -2 bin/*
+```
+
+## Conventions
+
+- **`# purpose:` header.** Every script under `bin/` starts with
+  `#!/usr/bin/env python3` (or appropriate shebang), then a
+  `# purpose: <one-line description>` comment. Baseline enforced by
+  `grep -L '^# purpose' bin/*`.
+- **Self-contained.** A script must work when run from anywhere — it
+  resolves the repo via the script's own path, not `cwd`.
+- **Read-only by default.** Most operator scripts query/print state
+  (`passwords-for`, `wireguard-client-config`). Mutating scripts
+  (`upgrade_and_restart_all`, `mikrotik-firmware-updater`,
+  `sync_1password`) are the exception, not the rule, and prompt for
+  confirmation.
+
+## How to add a script
+
+1. Start from [`bin/script_template`](script_template) — it carries
+   the canonical shebang + `# purpose:` header + `Repository(...)`
+   bootstrap.
+2. Add the `# purpose:` line; lowercase, terse, include a `usage:`
+   example if the script takes arguments.
+3. `chmod +x bin/<name>`.
+4. The script can reach helpers via `bw.libs.<x>` exactly like a
+   bundle does.
+
+## Pitfalls
+
+- **`bin/` is not on `$PATH` by default.** Invoke as `bin/<name>` from
+  the repo root, or via `direnv` if `.envrc` exposes it.
+- **Mutating scripts can hit Tier-3 territory** (per the fork's
+  safety envelope). Don't run `upgrade_and_restart_all`,
+  `mikrotik-firmware-updater`, or anything that does `node.run(...)`
+  without explicit user instruction. See the fork's
+  [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md)
+  for the three-tier model.
+- **Vault echo.** Scripts like `passwords-for` print decrypted values
+  by design; that's allowed for the human at the terminal but *not*
+  for the agent — never paste output into chat, ticket, or PR
+  description.
+
+## See also
+
+- [`script_template`](script_template) — canonical starter.
+- [`docs/agents/conventions.md`](../docs/agents/conventions.md) —
+  vault rules.
+- [`docs/agents/commands.md`](../docs/agents/commands.md) — read-only
+  bw-command guidance.

bin/mikrotik-firmware-updater

@@ -0,0 +1,149 @@
+#!/usr/bin/env python3
+# purpose: upgrade RouterOS and routerboard firmware on `bundle:routeros` (or any selector) — usage: mikrotik-firmware-updater [<selector>...] [--yes].
+
+from argparse import ArgumentParser
+from time import sleep
+
+from bundlewrap.exceptions import RemoteException
+from bundlewrap.utils.cmdline import get_target_nodes
+from bundlewrap.utils.ui import io
+from bundlewrap.repo import Repository
+from os.path import realpath, dirname
+
+
+# parse args
+parser = ArgumentParser()
+parser.add_argument("targets", nargs="*", default=['bundle:routeros'], help="bw nodes selector")
+parser.add_argument("--yes", action="store_true", default=False, help="skip confirmation prompts")
+args = parser.parse_args()
+
+
+def wait_up(node):
+    sleep(5)
+    while True:
+        try:
+            node.run_routeros('/system/resource/print')
+        except RemoteException:
+            sleep(2)
+            continue
+        else:
+            io.debug(f"{node.name}: is up")
+            sleep(10)
+            return
+
+
+def upgrade_switch_os(node):
+    # get versions for comparison
+    with io.job(f"{node.name}: checking OS version"):
+        response = node.run_routeros('/system/package/update/check-for-updates').raw[-1]
+        installed_os = bw.libs.version.Version(response['installed-version'])
+        latest_os = bw.libs.version.Version(response['latest-version'])
+        io.debug(f"{node.name}: installed: {installed_os} >= latest: {latest_os}")
+
+    # compare versions
+    if installed_os >= latest_os:
+        # os is up to date
+        io.stdout(f"{node.name}: os up to date ({installed_os})")
+    else:
+        # confirm os upgrade
+        if not args.yes and not io.ask(
+            f"{node.name}: upgrade os from {installed_os} to {latest_os}?", default=True
+        ):
+            io.stdout(f"{node.name}: skipped by user")
+            return
+
+        # download os
+        with io.job(f"{node.name}: downloading OS"):
+            response = node.run_routeros('/system/package/update/download').raw[-1]
+            io.debug(f"{node.name}: OS upgrade download response: {response['status']}")
+
+        # install and wait for reboot
+        with io.job(f"{node.name}: upgrading OS"):
+            try:
+                response = node.run_routeros('/system/package/update/install').raw[-1]
+            except RemoteException:
+                pass
+            wait_up(node)
+
+        # verify new os version
+        with io.job(f"{node.name}: checking new OS version"):
+            new_os = bw.libs.version.Version(node.run_routeros('/system/package/update/check-for-updates').raw[-1]['installed-version'])
+            if new_os == latest_os:
+                io.stdout(f"{node.name}: OS successfully upgraded from {installed_os} to {new_os}")
+            else:
+                raise Exception(f"{node.name}: OS upgrade failed, expected {latest_os}, got {new_os}")
+
+
+def upgrade_switch_firmware(node):
+    # get versions for comparison
+    with io.job(f"{node.name}: checking Firmware version"):
+        response = node.run_routeros('/system/routerboard/print').raw[-1]
+        current_firmware = bw.libs.version.Version(response['current-firmware'])
+        upgrade_firmware = bw.libs.version.Version(response['upgrade-firmware'])
+        io.debug(f"{node.name}: firmware installed: {current_firmware}, upgrade: {upgrade_firmware}")
+
+    # compare versions
+    if current_firmware >= upgrade_firmware:
+        # firmware is up to date
+        io.stdout(f"{node.name}: firmware is up to date ({current_firmware})")
+    else:
+        # confirm firmware upgrade
+        if not args.yes and not io.ask(
+            f"{node.name}: upgrade firmware from {current_firmware} to {upgrade_firmware}?", default=True
+        ):
+            io.stdout(f"{node.name}: skipped by user")
+            return
+
+        # upgrade firmware
+        with io.job(f"{node.name}: upgrading Firmware"):
+            node.run_routeros('/system/routerboard/upgrade')
+
+        # reboot and wait
+        with io.job(f"{node.name}: rebooting"):
+            try:
+                node.run_routeros('/system/reboot')
+            except RemoteException:
+                pass
+            wait_up(node)
+
+        # verify firmware version
+        new_firmware = bw.libs.version.Version(node.run_routeros('/system/routerboard/print').raw[-1]['current-firmware'])
+        if new_firmware == upgrade_firmware:
+            io.stdout(f"{node.name}: firmware successfully upgraded from {current_firmware} to {new_firmware}")
+        else:
+            raise Exception(f"firmware upgrade failed, expected {upgrade_firmware}, got {new_firmware}")
+
+
+def upgrade_switch(node):
+    with io.job(f"{node.name}: checking"):
+        # check if routeros
+        if node.os != 'routeros':
+            io.progress_advance(2)
+            io.stdout(f"{node.name}: skipped, unsupported os {node.os}")
+            return
+
+        # check switch reachability
+        try:
+            node.run_routeros('/system/resource/print')
+        except RemoteException as error:
+            io.progress_advance(2)
+            io.stdout(f"{node.name}: skipped, error {error}")
+            return
+
+    upgrade_switch_os(node)
+    io.progress_advance(1)
+
+    upgrade_switch_firmware(node)
+    io.progress_advance(1)
+
+
+with io:
+    bw = Repository(dirname(dirname(realpath(__file__))))
+
+    nodes = get_target_nodes(bw, args.targets)
+
+    io.progress_set_total(len(nodes) * 2)
+    io.stdout(f"upgrading {len(nodes)} switches: {', '.join([node.name for node in sorted(nodes)])}")
+
+    for node in sorted(nodes):
+        upgrade_switch(node)

bin/passwords-for

@@ -0,0 +1,23 @@
+#!/usr/bin/env python3
+# purpose: print node.password and selected metadata-key passwords for one node — usage: passwords-for <node>.
+
+from bundlewrap.repo import Repository
+from os.path import realpath, dirname
+
+import argparse
+
+parser = argparse.ArgumentParser()
+parser.add_argument('node', help='Node to generate passwords for')
+args = parser.parse_args()
+
+bw = Repository(dirname(dirname(realpath(__file__))))
+node = bw.get_node(args.node)
+
+if node.password:
+    print(f"password: {node.password}")
+
+for metadata_key in sorted([
+    'users/root/password',
+]):
+    if value := node.metadata.get(metadata_key, None):
+        print(f"{metadata_key}: {value}")

bin/rcon

@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# purpose: send an RCON command to a left4dead2 server defined in node metadata — usage: rcon (list) | rcon <server> <command>.
 
 from sys import argv
 from os.path import realpath, dirname

bin/script_template

@@ -1,6 +1,7 @@
 #!/usr/bin/env python3
+# purpose: starter template for new operator scripts under bin/.
 
 from bundlewrap.repo import Repository
 from os.path import realpath, dirname
 
-repo = Repository(dirname(dirname(realpath(__file__))))
+bw = Repository(dirname(dirname(realpath(__file__))))

bin/sync_1password

@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+# purpose: upsert one 1Password login per `bundle:routeros` node, keyed on the bw node id.
+
+from bundlewrap.repo import Repository
+from os.path import realpath, dirname
+
+import json
+import os
+import subprocess
+from dataclasses import dataclass
+from typing import Optional, List
+
+bw = Repository(dirname(dirname(realpath(__file__))))
+
+VAULT=bw.vault.decrypt('encrypt$gAAAAABpLgX_xxb5NmNCl3cgHM0JL65GT6PHVXO5gwly7IkmWoEgkCDSuAcSAkNFB8Tb4RdnTdpzVQEUL1XppTKVto_O7_b11GjATiyQYiSfiQ8KZkTKLvk=').value
+BW_TAG = "bw"
+BUNDLEWRAP_FIELD_LABEL = "bundlewrap node id"
+
+
+@dataclass
+class OpResult:
+    stdout: str
+    stderr: str
+    returncode: int
+
+
+def main():
+    for node in bw.nodes_in_group('routeros'):
+        upsert_node_item(
+            node_name=node.name,
+            node_uuid=node.metadata.get('id'),
+            username=node.username,
+            password=node.password,
+            url=f'http://{node.hostname}',
+        )
+
+
+def run_op(args):
+    proc = subprocess.run(
+        ["op", "--vault", VAULT] + args,
+        env=os.environ.copy(),
+        capture_output=True,
+        text=True,
+    )
+
+    if proc.returncode != 0:
+        raise RuntimeError(
+            f"op {' '.join(args)} failed with code {proc.returncode}:\n"
+            f"STDOUT:\n{proc.stdout}\n\nSTDERR:\n{proc.stderr}"
+        )
+
+    return OpResult(stdout=proc.stdout, stderr=proc.stderr, returncode=proc.returncode)
+
+
+def op_item_list_bw():
+    out = run_op([
+        "item", "list",
+        "--tags", BW_TAG,
+        "--format", "json",
+    ])
+    stdout = out.stdout.strip()
+    return json.loads(stdout) if stdout else []
+
+
+def op_item_get(item_id):
+    args = ["item", "get", item_id, "--format", "json"]
+    return json.loads(run_op(args).stdout)
+
+
+def op_item_create(title, node_uuid, username, password, url):
+    print(f"creating {title}")
+    return json.loads(run_op([
+        "item", "create",
+        "--category", "LOGIN",
+        "--title", title,
+        "--tags", BW_TAG,
+        "--url", url,
+        "--format", "json",
+        f"username={username}",
+        f"password={password}",
+        f"{BUNDLEWRAP_FIELD_LABEL}[text]={node_uuid}",
+    ]).stdout)
+
+
+def op_item_edit(item_id, title, username, password, url):
+    print(f"updating {title}")
+    return json.loads(run_op([
+        "item", "edit",
+        item_id,
+        "--title", title,
+        "--url", url,
+        "--format", "json",
+        f"username={username}",
+        f"password={password}",
+    ]).stdout)
+
+
+def find_node_item_id(node_uuid):
+    for summary in op_item_list_bw():
+        item_id = summary.get("id")
+        if not item_id:
+            continue
+
+        item = op_item_get(item_id)
+        for field in item.get("fields") or []:
+            label = field.get("label")
+            value = field.get("value")
+            if label == BUNDLEWRAP_FIELD_LABEL and value == node_uuid:
+                return item_id
+    return None
+
+
+def upsert_node_item(node_name, node_uuid, username, password, url):
+    if item_id := find_node_item_id(node_uuid):
+        return op_item_edit(
+            item_id=item_id,
+            title=node_name,
+            username=username,
+            password=password,
+            url=url,
+        )
+    else:
+        return op_item_create(
+            title=node_name,
+            node_uuid=node_uuid,
+            username=username,
+            password=password,
+            url=url,
+        )
+
+
+if __name__ == "__main__":
+    main()

bin/timestamp_icloud_photos_for_nextcloud

@@ -0,0 +1,217 @@
+#!/usr/bin/env python3
+# purpose: add missing EXIF/QuickTime timestamps to photos in a directory using mdls + exiftool — usage: timestamp_icloud_photos_for_nextcloud -d <dir>.
+
+from subprocess import check_output, CalledProcessError
+from datetime import datetime, timedelta
+from pathlib import Path
+import json
+from argparse import ArgumentParser
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from os import cpu_count
+from time import sleep
+
+EXT_GROUPS = {
+    "quicktime": {".mp4", ".mov", ".heic", ".cr3"},
+    "exif": {".jpg", ".jpeg", ".cr2"},
+}
+DATETIME_KEYS = [
+    ("Composite", "SubSecDateTimeOriginal"),
+    ("Composite", "SubSecCreateDate"),
+    ('ExifIFD', 'DateTimeOriginal'),
+    ('ExifIFD', 'CreateDate'),
+    ('XMP-xmp', 'CreateDate'),
+    ('Keys', 'CreationDate'),
+    ('QuickTime', 'CreateDate'),
+    ('XMP-photoshop', 'DateCreated'),
+]
+
+def run(command):
+    return check_output(command, text=True).strip()
+
+
+def mdls_timestamp(file):
+    for i in range(5): # retry a few times in case of transient mdls failures
+        try:
+            output = run(('mdls', '-raw', '-name', 'kMDItemContentCreationDate', file))
+        except CalledProcessError as e:
+            print(f"{file}: Error running mdls (attempt {i+1}/5): {e}")
+            continue
+
+        try:
+            return datetime.strptime(output, "%Y-%m-%d %H:%M:%S %z")
+        except ValueError as e:
+            print(f"{file}: Error parsing mdls output (attempt {i+1}/5): {e}")
+            continue
+
+        sleep(1)
+
+    raise RuntimeError(f"Failed to get mdls timestamp for {file} after 5 attempts")
+
+
+def exiftool_data(file):
+    try:
+        output = run((
+            'exiftool',
+            '-j', # json
+            '-a', # unknown tags
+            '-u', # unknown values
+            '-g1', # group by category
+            '-time:all', # all time tags
+            '-api', 'QuickTimeUTC=1', # use UTC for QuickTime timestamps
+            '-d', '%Y-%m-%dT%H:%M:%S%z',
+            file,
+        ))
+    except CalledProcessError as e:
+        print(f"Error running exiftool: {e}")
+        return None
+    else:
+        return json.loads(output)[0]
+
+def exiftool_timestamp(file):
+    data = exiftool_data(file)
+    for category, key in DATETIME_KEYS:
+        try:
+            value = data[category][key]
+            return category, key, datetime.strptime(value, '%Y-%m-%dT%H:%M:%S%z')
+        except (TypeError, KeyError, ValueError) as e:
+            continue
+    print(f"⚠️ {file}: No timestamp found in exiftool: " + json.dumps(data, indent=2))
+    return None, None, None
+
+
+def photo_has_embedded_timestamp(file):
+    mdls_ts = mdls_timestamp(file)
+    category, key, exiftool_ts = exiftool_timestamp(file)
+
+    if not exiftool_ts:
+        print(f"⚠️ {file}: No timestamp found in exiftool")
+        return False
+
+    # normalize timezone for comparison
+    exiftool_ts = exiftool_ts.astimezone(mdls_ts.tzinfo)
+    delta = abs(mdls_ts - exiftool_ts)
+
+    if delta < timedelta(hours=1): # allow for small differences
+        print(f"✅ {file}: {mdls_ts.isoformat()} (#{category}:{key})")
+        return True
+    else:
+        print(f"⚠️ {file}: {mdls_ts.isoformat()} != {exiftool_ts} (Δ={delta})")
+        return False
+
+
+def photos_without_embedded_timestamps(directory):
+    executor = ThreadPoolExecutor(max_workers=cpu_count()//2)
+    try:
+        futures = {
+            executor.submit(photo_has_embedded_timestamp, file): file
+                for file in directory.iterdir()
+                if file.is_file()
+                if file.suffix.lower() not in {".aae"}
+                if not file.name.startswith('.')
+        }
+
+        for future in as_completed(futures):
+            file = futures[future]
+            has_ts = future.result()   # raises immediately on first failed future
+
+            if has_ts:
+                file.rename(file.parent / 'ok' / file.name)
+            else:
+                yield file
+
+    except Exception:
+        executor.shutdown(wait=False, cancel_futures=True)
+        raise
+    else:
+        executor.shutdown(wait=True)
+
+
+def exiftool_write(file, assignments):
+    print(f"🔵 {file}: Writing -- {assignments}")
+    return run((
+        "exiftool", "-overwrite_original",
+        "-api", "QuickTimeUTC=1",
+        *[
+            f"-{group}:{tag}={value}"
+            for group, tag, value in assignments
+        ],
+        str(file),
+    ))
+
+
+def add_missing_timestamp(file):
+    data = exiftool_data(file)
+    mdls_ts = mdls_timestamp(file)
+
+    offset = mdls_ts.strftime("%z")
+    offset = f"{offset[:3]}:{offset[3:]}" if len(offset) == 5 else offset
+
+    exif_ts = mdls_ts.strftime("%Y:%m:%d %H:%M:%S")
+    qt_ts = mdls_ts.strftime("%Y:%m:%d %H:%M:%S")
+    qt_ts_tz = f"{qt_ts}{offset}"
+    ext = file.suffix.lower()
+
+    try:
+        if ext in {".heic"}:
+            exiftool_write(file, [
+                ("ExifIFD", "DateTimeOriginal", qt_ts),
+                ("ExifIFD", "CreateDate", qt_ts),
+                ("ExifIFD", "OffsetTime", offset),
+                ("ExifIFD", "OffsetTimeOriginal", offset),
+                ("ExifIFD", "OffsetTimeDigitized", offset),
+                ("QuickTime", "CreateDate", qt_ts_tz),
+                ("Keys", "CreationDate", qt_ts_tz),
+                ("XMP-xmp", "CreateDate", qt_ts_tz),
+            ])
+        elif "QuickTime" in data or ext in {".mp4", ".mov", ".heic", ".cr3"}:
+            exiftool_write(file, [
+                ("QuickTime", "CreateDate", qt_ts_tz),
+                ("Keys", "CreationDate", qt_ts_tz),
+            ])
+        elif "ExifIFD" in data or ext in {".jpg", ".jpeg", ".cr2", ".webp"}:
+            exiftool_write(file, [
+                ("ExifIFD", "DateTimeOriginal", exif_ts),
+                ("ExifIFD", "CreateDate", exif_ts),
+                ("IFD0", "ModifyDate", exif_ts),
+                ("ExifIFD", "OffsetTime", offset),
+                ("ExifIFD", "OffsetTimeOriginal", offset),
+                ("ExifIFD", "OffsetTimeDigitized", offset),
+            ])
+        elif ext in {".png", ".gif", ".avif"}:
+            exiftool_write(file, [
+                ("XMP-xmp", "CreateDate", qt_ts_tz),
+                ("XMP-photoshop", "DateCreated", exif_ts),
+            ])
+        else:
+            print(f"❌ {file}: unsupported type, skipped")
+            return
+
+        if photo_has_embedded_timestamp(file):
+            print(f"✅ {file}: Timestamp successfully added: {mdls_ts.isoformat()}")
+            file.rename(file.parent / 'processed' / file.name)
+            return
+        else:
+            category, key, exiftool_ts = exiftool_timestamp(file)
+            print(f"❌ {file}: Timestamp still wrong/missing after write '{category}:{key}:{exiftool_ts}': #{json.dumps(data, indent=4)}")
+            return
+    except CalledProcessError as e:
+        print(f"❌ {file}: Failed to write timestamp: {e}")
+        return
+
+
+if __name__ == "__main__":
+    parser = ArgumentParser(description="Print timestamps of photos in the current directory.")
+    parser.add_argument("-d", "--directory", help="Directory to scan for photos")
+    args = parser.parse_args()
+
+    directory = Path(args.directory)
+    (directory/'ok').mkdir(exist_ok=True)
+    (directory/'processed').mkdir(exist_ok=True)
+
+    _photos_without_embedded_timestamps = list(photos_without_embedded_timestamps(directory))
+    print(f"{len(_photos_without_embedded_timestamps)} photos without embedded timestamps found.")
+    print("Press Enter to add missing timestamps...")
+    input()
+
+    for file in _photos_without_embedded_timestamps:
+        add_missing_timestamp(file)

bin/upgrade_and_restart_all

@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# purpose: apt-update and full-upgrade every non-dummy debian node, then reboot in WireGuard-aware order.
 
 from bundlewrap.repo import Repository
 from os.path import realpath, dirname
@@ -23,7 +24,7 @@ for node in nodes:
     print(node.run('DEBIAN_FRONTEND=noninteractive apt update').stdout.decode())
     print(node.run('DEBIAN_FRONTEND=noninteractive apt list --upgradable').stdout.decode())
     if int(node.run('DEBIAN_FRONTEND=noninteractive apt list --upgradable 2> /dev/null | grep upgradable | wc -l').stdout.decode()):
-        print(node.run('DEBIAN_FRONTEND=noninteractive apt -y dist-upgrade').stdout.decode())
+        print(node.run('DEBIAN_FRONTEND=noninteractive apt -qy full-upgrade').stdout.decode())
 
 # REBOOT IN ORDER
 

bin/wake

@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# purpose: wake one node via WoL by name — usage: wake <node>.
 
 from bundlewrap.repo import Repository
 from os.path import realpath, dirname

bin/wireguard-client-config

@@ -1,23 +1,25 @@
 #!/usr/bin/env python3
+# purpose: print or QR-render a WireGuard client config from htz.mails metadata — usage: wireguard-client-config <client>.
 
 from bundlewrap.repo import Repository
 from os.path import realpath, dirname
 from sys import argv
 from ipaddress import ip_network, ip_interface
+import argparse
 
-if len(argv) != 3:
-    print(f'usage: {argv[0]} <node> <client>')
-    exit(1)
 
+# get info from repo
 repo = Repository(dirname(dirname(realpath(__file__))))
-server_node = repo.get_node(argv[1])
+server_node = repo.get_node('htz.mails')
+available_clients = server_node.metadata.get('wireguard/clients').keys()
 
-if argv[2] not in server_node.metadata.get('wireguard/clients'):
+# parse args
-    print(f'client {argv[2]} not found in: {server_node.metadata.get("wireguard/clients").keys()}')
+parser = argparse.ArgumentParser(description='Generate WireGuard client configuration.')
-    exit(1)
+parser.add_argument('client', choices=available_clients, help='The client name to generate the configuration for.')
-
+args = parser.parse_args()
-data = server_node.metadata.get(f'wireguard/clients/{argv[2]}')
 
+# get cert
+data = server_node.metadata.get(f'wireguard/clients/{args.client}')
 vpn_network = ip_interface(server_node.metadata.get('wireguard/my_ip')).network
 allowed_ips = [
     vpn_network,
@@ -43,10 +45,15 @@ Endpoint = {ip_interface(server_node.metadata.get('network/external/ipv4')).ip}:
 PersistentKeepalive = 10
 '''
 
-print('>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>')
+answer = input("print config or qrcode? [Cq]: ").strip().upper()
-print(conf)
+match answer:
-print('<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<')
+    case '' | 'C':
-
+        print('>>>>>>>>>>>>>>>')
-if input("print qrcode? [Yn]: ").upper() in ['', 'Y']:
+        print(conf)
-    import pyqrcode
+        print('<<<<<<<<<<<<<<<')
-    print(pyqrcode.create(conf).terminal(quiet_zone=1))
+    case 'Q':
+        import pyqrcode
+        print(pyqrcode.create(conf).terminal(quiet_zone=1))
+    case _:
+        print(f'Invalid option "{answer}".')
+        exit(1)

bundles/AGENTS.md

@@ -0,0 +1,204 @@
+# bundles/
+
+## Before you start
+
+Read [`docs/agents/conventions.md`](../docs/agents/conventions.md) first
+— it covers vault calls, demagify, the `repo.libs.<x>` helpers, and the
+files agents must not modify. Skipping it leads to subtly broken bundles
+(vault calls in the wrong place, dict-in-set `TypeError` because of
+unhashable nesting, etc.).
+
+For bundlewrap-language reference (item types, dep keywords,
+`metadata_reactor`, `defaults`, item-file template syntax) see the fork's
+[`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md)
+and its [`docs/content/guide/item_file_templates.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/docs/content/guide/item_file_templates.md).
+
+## What's here
+
+103 bundles. Each is a directory `bundles/<name>/` containing some of:
+
+```
+bundles/<name>/
+├── items.py        # the items this bundle creates (files, services, packages, …)
+├── metadata.py     # `defaults` + `@metadata_reactor` functions
+├── files/          # static or templated file payloads referenced from items.py
+└── README.md       # one doc per bundle, for humans and agents (see "Per-bundle README" below)
+```
+
+## Conventions
+
+- **Bundle names** are lowercase, hyphen-separated: `backup-server`,
+  `bind-acme`, `dm-crypt`. No underscores in new bundle names — see
+  [`conventions.md#naming-conventions`](../docs/agents/conventions.md#naming-conventions).
+- **`items.py`** is plain Python; it produces `files = {...}`,
+  `pkg_apt = {...}`, `svc_systemd = {...}`, etc. dicts at module scope.
+  Cross-item dependencies use `needs` / `triggers` / `triggered_by` —
+  see the fork's `AGENTS.md` for the full keyword cheat sheet.
+- **`metadata.py`** uses `defaults = {...}` for static seed values and
+  `@metadata_reactor.provides(...)` for derived values. Reactors are
+  pure functions of `(metadata,)` — no side effects, no I/O.
+- **Helpers go in [`libs/`](../libs/AGENTS.md)** when they're useful to
+  more than one bundle. Don't duplicate logic across bundles.
+- **Custom item types** (e.g. `download:`) live in
+  [`items/`](../items/AGENTS.md), not per-bundle.
+- **Bundles own application-wide knowledge; nodes carry only the few
+  per-host knobs the bundle actually needs.** When designing a bundle,
+  identify the per-node knobs (e.g. domain, uplink interface, a
+  vault-id suffix) and put everything else in `defaults`, or in a
+  reactor that derives from those knobs. Per-node random secrets
+  belong in `defaults` via `repo.vault.random_bytes_as_base64_for(...)`
+  keyed on the node — not in the node file. See
+  `bundles/left4me/metadata.py:10` (`secret_key` derived in defaults)
+  and `bundles/postgresql/metadata.py:4` (vault-derived `password_for`
+  at module scope).
+
+## How to add a new bundle
+
+1. `mkdir bundles/<name>/` (lowercase, hyphenated).
+2. Write `items.py` and (if anything is configurable) `metadata.py`.
+   Use `repo.libs.hashable.hashable(...)` when you need to nest a dict
+   or set inside a metadata set; raw dicts/sets aren't hashable.
+3. Drop static payloads into `bundles/<name>/files/`. For Mako-templated
+   files, declare `'content_type': 'mako'` on the `file:` item — see
+   the fork's
+   [item-file-templates guide](https://github.com/CroneKorkN/bundlewrap/blob/main/docs/content/guide/item_file_templates.md).
+4. **Wire to nodes.** Either add an entry to the relevant
+   [`groups/<axis>/<x>.py`](../groups/AGENTS.md) (preferred for shared
+   bundles) or to the node's `bundles` list directly
+   ([`nodes/AGENTS.md`](../nodes/AGENTS.md)).
+5. **Verify, in this order:**
+   - `bw test` — repo-wide parse + cross-cutting hooks. Loads every
+     bundle, but reactors don't fire for nodes that haven't opted into
+     the bundle yet — bugs in new reactors stay hidden here.
+   - **Attach the bundle to a node** (via the node's `bundles` list, or
+     a group it belongs to). Until you do, the next steps don't actually
+     exercise the bundle.
+   - `bw test <node>` — exercises every reactor and item-graph edge for
+     that node. This is where most new-bundle bugs surface.
+   - `bw items <node> --blame` — confirm items materialise with the
+     right paths, authored by the expected bundle.
+   - `bw metadata <node> -k <a/b>` — spot-check derived metadata.
+   - `bw hash <node>` — preview vs current host state.
+
+   See [`docs/agents/commands.md#bundle-validation-workflow`](../docs/agents/commands.md#bundle-validation-workflow)
+   for the rationale.
+6. Add a `bundles/<name>/README.md`. See "Per-bundle README" below
+   for what to cover.
+
+## How to remove a bundle
+
+1. `git grep '<name>'` in `nodes/`, `groups/`, and other `bundles/` to
+   find references.
+2. Remove those references.
+3. `rm -rf bundles/<name>/`.
+4. `bw test` and `bw nodes` to confirm clean.
+
+## Pitfalls
+
+- **`metadata.py` is evaluated at load time** for *every* node, every
+  invocation of `bw`. Heavy work or I/O slows the whole repo. Keep
+  reactors pure and fast; pre-compute in `libs/` if you must.
+- **Static files vs templates.** `bundles/<x>/files/<f>` is static
+  unless the matching `file:` item declares `content_type='mako'`
+  (or a templating extension triggers it). To check, read the matching
+  `file:` entry in `items.py`.
+- **`file:` `source` defaults to the destination basename.** For a
+  destination of `/etc/foo/bar.conf` with no `source` key, bw looks
+  for `bundles/<bundle>/files/bar.conf`. Only declare `source`
+  explicitly when the basename you want differs (e.g. shipping a Mako
+  template named `bar.conf.mako` to a destination of
+  `/etc/foo/bar.conf`).
+- **Reactors writing across namespaces.** Some bundles' reactors write
+  into other bundles' metadata namespaces (e.g. `nextcloud` writes
+  into `apt.packages`, `archive.paths`). When you change such a bundle,
+  every consumer's metadata changes too. The bundle's `README.md`
+  often calls these out — but the authoritative source is `metadata.py`
+  itself; grep `'<other-bundle>':` in the reactors when in doubt.
+- **`bw hash` doesn't accept selectors.** Use `bw hash <node>` per
+  literal name; see the fork's runbook.
+- **Reactors must read metadata.** If a reactor body returns a static
+  dict without calling `metadata.get(...)`, bw raises
+  `ValueError: <reactor> on <node> did not request any metadata, you
+  might want to use defaults instead` once a node consumes the bundle.
+  Fix: fold the contribution into `defaults`. The rule applies even
+  when the reactor writes into another bundle's namespace — a static
+  contribution to e.g. `nftables/output` belongs in `defaults`, where
+  bw merges it with other bundles' contributions.
+- **`triggers` ↔ `triggered: True` invariant.** Any item listed in
+  another's `triggers` list must declare `triggered: True`. bw
+  enforces this at `bw test` time: *"…triggered by …, but missing
+  'triggered' attribute"*. Corollary: an action can't be both in an
+  upstream `triggers` list AND self-healing every apply — pick one.
+- **Triggered actions don't recover from partial failure.** When an
+  upstream item's apply succeeds but its triggered downstream action
+  fails, subsequent applies can't recover via the trigger chain —
+  upstream is "already in desired state" and never re-triggers. For
+  actions that must self-heal (pip installs, chowns, migrations),
+  drop `triggered: True` and gate the command with `unless: <fast-check>`.
+  `unless` is a shell command on the target host whose exit status
+  decides whether the main command runs (exit 0 = skip); it's checked
+  at fire time, after `triggered:` filtering.
+
+## Per-bundle README
+
+Each bundle has (or should have) a `README.md`. One doc per bundle,
+written for humans and agents both. There's no fixed structure —
+match the bundle's actual surface, write what helps a future reader
+(or future you) avoid trial-and-error.
+
+The existing READMEs vary in quality and shape. For orientation,
+look at the bigger ones, not the two-line ones:
+
+- [`bundles/flask/README.md`](flask/README.md) — title + one-sentence
+  purpose, a metadata example as a Python dict, then the contract
+  the consuming git repo has to satisfy + a logging pitfall. The
+  closest thing to a "balanced doc" in tree.
+- [`bundles/dm-crypt/README.md`](dm-crypt/README.md) — same shape,
+  shorter: purpose + metadata example + one sentence on effect.
+- [`bundles/apt/README.md`](apt/README.md) — relevant upstream URLs
+  at the top, then a Python metadata example with rich inline
+  comments (type / optionality / where keys come from).
+- [`bundles/nextcloud/README.md`](nextcloud/README.md) — operational
+  scratchpad: iPhone-import recipe, preview-generator commands,
+  reset queries. Captures muscle-memory the maintainer would
+  otherwise re-learn each time.
+
+Useful things to include, when relevant:
+
+- A sentence or two on what the bundle does and when you'd attach it.
+- A metadata example as a Python dict literal, with `#` comments
+  on each key (type, required vs default, units, where it comes
+  from). This is the cleanest way to communicate the schema and
+  matches how `metadata.py` actually looks.
+- Anything non-obvious about wiring it up — required keys without
+  defaults, group-membership expectations, manual one-time steps.
+- Cross-namespace metadata writes, when this bundle's reactors
+  populate another bundle's namespace. Easy to miss, cheap to flag.
+- Gotchas, debug recipes, failure modes you've actually hit.
+
+What to skip:
+
+- An exhaustive item list — `items.py` is shorter and more accurate.
+- Anything that would just rot — version numbers, "TODO" lists,
+  change notes. Use git history.
+
+If a single paragraph is enough to say what's worth saying, write a
+single paragraph. Verbosity isn't a goal.
+
+Convention going forward is leave-as-you-go: any time you materially
+edit a bundle, top up its README (or write one if it's missing).
+Don't burn a session bulk-reformatting the existing ones — uneven
+quality is part of what we accept in exchange for not blocking other
+work.
+
+## See also
+
+- [`docs/agents/conventions.md`](../docs/agents/conventions.md) — repo
+  idioms (vault, demagify, naming, do-not-touch list).
+- [`docs/agents/commands.md`](../docs/agents/commands.md) — repo-specific
+  command deltas.
+- [`items/AGENTS.md`](../items/AGENTS.md) — custom item types
+  (`download:`); when to write a new one vs use `file:`.
+- [`libs/AGENTS.md`](../libs/AGENTS.md) — shared helpers.
+- Fork's [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md)
+  — bundlewrap-language reference + safety envelope.

bundles/apcupsd/metadata.py

@@ -13,16 +13,14 @@ defaults = {
         },
     },
     'telegraf': {
-        'config': {
+        'inputs': {
-            'inputs': {
+            'exec': {
-                'exec': {
+                'apcupsd': {
-                    repo.libs.hashable.hashable({
+                    'commands': ["sudo /usr/local/share/telegraf/apcupsd"],
-                        'commands': ["sudo /usr/local/share/telegraf/apcupsd"],
+                    'name_override': "apcupsd",
-                        'name_override': "apcupsd",
+                    'data_format': "influx",
-                        'data_format': "influx",
+                    'interval': '30s',
-                        'interval': '30s',
+                    'flush_interval': '30s',
-                        'flush_interval': '30s',
-                    }),
                 },
             },
         },

bundles/apt/README.md

@@ -13,6 +13,9 @@
                     'deb',
                     'deb-src',
                 },
+                'options': { # optional
+                    'aarch': 'amd64',
+                },
                 'urls': {
                     'https://deb.debian.org/debian',
                 },

bundles/apt/items.py

@@ -62,6 +62,7 @@ files = {
     '/usr/lib/nagios/plugins/check_apt_upgradable': {
         'mode': '0755',
     },
+    # /etc/kernel/postinst.d/apt-auto-removal
 }
 
 actions = {

bundles/apt/metadata.py

@@ -4,6 +4,8 @@ defaults = {
             'apt-listchanges': {
                 'installed': False,
             },
+            'ca-certificates': {},
+            'unattended-upgrades': {},
         },
         'config': {
             'DPkg': {
@@ -21,6 +23,10 @@ defaults = {
                 },
             },
             'APT': {
+                'Periodic': {
+                    'Update-Package-Lists': '1',
+                    'Unattended-Upgrade': '1',
+                },
                 'NeverAutoRemove': {
                     '^firmware-linux.*',
                     '^linux-firmware$',
@@ -48,6 +54,11 @@ defaults = {
                     'Error-Mode': 'any',
                 },
             },
+            'Unattended-Upgrade': {
+                'Origins-Pattern': {
+                    "origin=*",
+                },
+            },
         },
         'sources': {},
     },
@@ -106,33 +117,6 @@ def signed_by(metadata):
     }
 
 
-@metadata_reactor.provides(
-    'apt/config',
-    'apt/packages',
-)
-def unattended_upgrades(metadata):
-    return {
-        'apt': {
-            'config': {
-                'APT': {
-                    'Periodic': {
-                        'Update-Package-Lists': '1',
-                        'Unattended-Upgrade': '1',
-                    },
-                },
-                'Unattended-Upgrade': {
-                    'Origins-Pattern': {
-                        "origin=*",
-                    },
-                },
-            },
-            'packages': {
-                'unattended-upgrades': {},
-            },
-        },
-    }
-
-
 # @metadata_reactor.provides(
 #     'apt/config',
 #     'apt/list_changes',

bundles/backup/files/backup_all

@@ -1,13 +1,31 @@
 #!/bin/bash
 
-set -exu
+set -u
 
 # FIXME: inelegant
 % if wol_command:
 ${wol_command}
 % endif
 
+exit=0
+failed_paths=""
+
 for path in $(jq -r '.paths | .[]' < /etc/backup/config.json)
 do
+  echo backing up $path
   /opt/backup/backup_path "$path"
+  # set exit to 1 if any backup fails
+  if [ $? -ne 0 ]
+  then
+    echo ERROR: backing up $path failed >&2
+    exit=5
+    failed_paths="$failed_paths $path"
+  fi
 done
+
+if [ $exit -ne 0 ]
+then
+  echo "ERROR: failed to backup paths: $failed_paths" >&2
+fi
+
+exit $exit

bundles/backup/files/backup_path_via_zfs

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-set -exu
+set -eu
 
 path=$1
 uuid=$(jq -r .client_uuid < /etc/backup/config.json)

bundles/bind-acme/metadata.py

@@ -33,6 +33,7 @@ def acme_zone(metadata):
             str(ip_interface(other_node.metadata.get('network/internal/ipv4')).ip)
                 for other_node in repo.nodes
                 if other_node.metadata.get('letsencrypt/domains', {})
+                and other_node.metadata.get('network/internal/ipv4', None)
         },
         *{
             str(ip_interface(other_node.metadata.get('wireguard/my_ip')).ip)

bundles/bind/README.md

@@ -0,0 +1,30 @@
+# bind
+
+Authoritative DNS — primary plus optional `bind/master_node` slaves.
+
+## Applying changes needs both nodes
+
+The slave's bw-managed zone files are rendered from the master's
+metadata at slave-apply time (see `bundles/bind/items.py:100`). When
+you change a record on the master (adding a `letsencrypt/domains`
+entry, a new vhost, etc.), the change is only published once you
+apply BOTH:
+
+```sh
+bw apply htz.mails        # primary (where the source records live)
+bw apply ovh.secondary    # secondary (renders its own zone files)
+```
+
+Until both have been applied, `bw verify ovh.secondary` will show
+stale zones and consumers that hit the secondary (Let's Encrypt's
+secondary-region validators in particular) will see NXDOMAIN. Even
+though the slave's named.conf.local declares `type slave;`, don't
+rely on bind's own AXFR catching up — the bw-rendered file on disk
+is what `bw verify` measures.
+
+## See also
+
+- `bundles/bind-acme/` — the in-house ACME-update receiver.
+- `bundles/letsencrypt/README.md` — DNS-01 prerequisites and the
+  negative-cache penalty (the most common operational consequence
+  of forgetting to apply the secondary).

bundles/bind/files/db.empty

@@ -0,0 +1,8 @@
+$TTL    86400
+@       IN      SOA     localhost. root.localhost. (
+                              1         ; Serial
+                              604800    ; Refresh
+                              86400     ; Retry
+                              2419200   ; Expire
+                              86400 )   ; Negative Cache TTL
+        IN      NS      localhost.

bundles/bind/files/named.conf.local

@@ -29,6 +29,7 @@ view "${view_name}" {
 
   % if view_conf['is_internal']:
   recursion yes;
+  include "/etc/bind/zones.rfc1918";
   % else:
   recursion no;
   rate-limit {
@@ -62,9 +63,6 @@ view "${view_name}" {
     file "/var/lib/bind/${view_name}/${zone_name}";
   };
   % endfor
-
-  include "/etc/bind/named.conf.default-zones";
-  include "/etc/bind/zones.rfc1918";
 };
 
 % endfor

bundles/bind/files/named.conf.options

@@ -10,7 +10,7 @@ options {
 
 % if type == 'master':
   notify yes;
-  also-notify { ${' '.join([f'{ip};' for ip in slave_ips])} };
+  also-notify { ${' '.join(sorted(f'{ip};' for ip in slave_ips))} };
-  allow-transfer { ${' '.join([f'{ip};' for ip in slave_ips])} };
+  allow-transfer { ${' '.join(sorted(f'{ip};' for ip in slave_ips))} };
 % endif
 };

bundles/bind/files/zones.rfc1918

@@ -0,0 +1,19 @@
+zone "10.in-addr.arpa"       { type master; file "/etc/bind/db.empty"; };
+zone "16.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "17.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "18.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "19.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "20.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "21.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "22.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "23.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "24.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "25.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "26.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "27.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "28.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "29.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "30.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "31.172.in-addr.arpa"   { type master; file "/etc/bind/db.empty"; };
+zone "168.192.in-addr.arpa"  { type master; file "/etc/bind/db.empty"; };
+zone "254.169.in-addr.arpa"  { type master; file "/etc/bind/db.empty"; };

bundles/bind/items.py

@@ -142,3 +142,21 @@ actions['named-checkconf'] = {
         'svc_systemd:bind9:reload',
     ]
 }
+
+# beantwortet Anfragen nach privaten IP-Adressen mit NXDOMAIN, statt sie ins Internet weiterzuleiten
+files['/etc/bind/zones.rfc1918'] = {
+    'needed_by': [
+        'svc_systemd:bind9',
+    ],
+    'triggers': [
+        'svc_systemd:bind9:reload',
+    ],
+}
+files['/etc/bind/db.empty'] = {
+    'needed_by': [
+        'svc_systemd:bind9',
+    ],
+    'triggers': [
+        'svc_systemd:bind9:reload',
+    ],
+}

bundles/bind/metadata.py

@@ -3,6 +3,7 @@ from json import dumps
 h = repo.libs.hashable.hashable
 repo.libs.bind.repo = repo
 
+
 defaults = {
     'apt': {
         'packages': {
@@ -48,13 +49,13 @@ defaults = {
         },
     },
     'telegraf': {
-        'config': {
+        'inputs': {
-            'inputs': {
+            'bind': {
-                'bind': [{
+                'default': {
                     'urls': ['http://localhost:8053/xml/v3'],
                     'gather_memory_contexts': False,
                     'gather_views': True,
-                }],
+                },
             },
         },
     },
@@ -211,7 +212,7 @@ def generate_keys(metadata):
                             'token':repo.libs.hmac.hmac_sha512(
                                 key,
                                 str(repo.vault.random_bytes_as_base64_for(
-                                    f"{metadata.get('id')} bind key {key}",
+                                    f"{metadata.get('id')} bind key {key} 20250713",
                                     length=32,
                                 )),
                             )

bundles/bootshorn/files/process

@@ -0,0 +1,165 @@
+#!/usr/bin/env python3
+
+import os
+import datetime
+import numpy as np
+import matplotlib.pyplot as plt
+import soundfile as sf
+from scipy.fft import rfft, rfftfreq
+import shutil
+import traceback
+
+
+RECORDINGS_DIR = "recordings"
+PROCESSED_RECORDINGS_DIR = "recordings/processed"
+DETECTIONS_DIR = "events"
+
+DETECT_FREQUENCY = 211 # Hz
+DETECT_FREQUENCY_TOLERANCE = 2 # Hz
+ADJACENCY_FACTOR = 2 # area to look for the frequency (e.g. 2 means 100Hz to 400Hz for 200Hz detection)
+BLOCK_SECONDS = 3 # seconds (longer means more frequency resolution, but less time resolution)
+DETECTION_DISTANCE_SECONDS = 30 # seconds (minimum time between detections)
+BLOCK_OVERLAP_FACTOR = 0.9 # overlap between blocks (0.2 means 20% overlap)
+MIN_SIGNAL_QUALITY = 1000.0 # maximum noise level (relative DB) to consider a detection valid
+PLOT_PADDING_START_SECONDS = 2 # seconds (padding before and after the event in the plot)
+PLOT_PADDING_END_SECONDS = 3 # seconds (padding before and after the event in the plot)
+
+DETECTION_DISTANCE_BLOCKS = DETECTION_DISTANCE_SECONDS // BLOCK_SECONDS # number of blocks to skip after a detection
+DETECT_FREQUENCY_FROM = DETECT_FREQUENCY - DETECT_FREQUENCY_TOLERANCE # Hz
+DETECT_FREQUENCY_TO = DETECT_FREQUENCY + DETECT_FREQUENCY_TOLERANCE # Hz
+
+
+def process_recording(filename):
+    print('processing', filename)
+
+    # get ISO 8601 nanosecond recording date from filename
+    date_string_from_filename = os.path.splitext(filename)[0]
+    recording_date = datetime.datetime.strptime(date_string_from_filename, "%Y-%m-%d_%H-%M-%S.%f%z")
+
+    # get data and metadata from recording
+    path = os.path.join(RECORDINGS_DIR, filename)
+    soundfile = sf.SoundFile(path)
+    samplerate = soundfile.samplerate
+    samples_per_block = int(BLOCK_SECONDS * samplerate)
+    overlapping_samples = int(samples_per_block * BLOCK_OVERLAP_FACTOR)
+
+    sample_num = 0
+    current_event = None
+
+    while sample_num < len(soundfile):
+        soundfile.seek(sample_num)
+        block = soundfile.read(frames=samples_per_block, dtype='float32', always_2d=False)
+
+        if len(block) == 0:
+            break
+
+        # calculate FFT
+        labels = rfftfreq(len(block), d=1/samplerate)
+        complex_amplitudes = rfft(block)
+        amplitudes = np.abs(complex_amplitudes)
+
+        # get the frequency with the highest amplitude within the search range
+        search_amplitudes = amplitudes[(labels >= DETECT_FREQUENCY_FROM/ADJACENCY_FACTOR) & (labels <= DETECT_FREQUENCY_TO*ADJACENCY_FACTOR)]
+        search_labels = labels[(labels >= DETECT_FREQUENCY_FROM/ADJACENCY_FACTOR) & (labels <= DETECT_FREQUENCY_TO*ADJACENCY_FACTOR)]
+        max_amplitude = max(search_amplitudes)
+        max_amplitude_index = np.argmax(search_amplitudes)
+        max_freq = search_labels[max_amplitude_index]
+        max_freq_detected = DETECT_FREQUENCY_FROM <= max_freq <= DETECT_FREQUENCY_TO
+
+        # calculate signal quality
+        adjacent_amplitudes = amplitudes[(labels < DETECT_FREQUENCY_FROM) | (labels > DETECT_FREQUENCY_TO)]
+        signal_quality = max_amplitude/np.mean(adjacent_amplitudes)
+        good_signal_quality = signal_quality > MIN_SIGNAL_QUALITY
+
+        # conclude detection
+        if (
+            max_freq_detected and
+            good_signal_quality
+        ):
+            block_date = recording_date + datetime.timedelta(seconds=sample_num / samplerate)
+
+            # detecting an event
+            if not current_event:
+                current_event = {
+                    'start_at': block_date,
+                    'end_at': block_date,
+                    'start_sample': sample_num,
+                    'end_sample': sample_num + samples_per_block,
+                    'start_freq': max_freq,
+                    'end_freq': max_freq,
+                    'max_amplitude': max_amplitude,
+                }
+            else:
+                current_event.update({
+                    'end_at': block_date,
+                    'end_freq': max_freq,
+                    'end_sample': sample_num + samples_per_block,
+                    'max_amplitude': max(max_amplitude, current_event['max_amplitude']),
+                })
+            print(f'- {block_date.strftime('%Y-%m-%d %H:%M:%S')}: {max_amplitude:.1f}rDB @ {max_freq:.1f}Hz (signal {signal_quality:.3f}x)')
+        else:
+            # not detecting an event
+            if current_event:
+                duration = (current_event['end_at'] - current_event['start_at']).total_seconds()
+                current_event['duration'] = duration
+                print(f'🔊 {current_event['start_at'].strftime('%Y-%m-%d %H:%M:%S')} ({duration:.1f}s): {current_event['start_freq']:.1f}Hz->{current_event['end_freq']:.1f}Hz @{current_event['max_amplitude']:.0f}rDB')
+
+                # read full audio clip again for writing
+                write_event(current_event=current_event, soundfile=soundfile, samplerate=samplerate)
+
+                current_event = None
+                sample_num += DETECTION_DISTANCE_BLOCKS * samples_per_block
+
+        sample_num += samples_per_block - overlapping_samples
+
+    # move to PROCESSED_RECORDINGS_DIR
+
+    os.makedirs(PROCESSED_RECORDINGS_DIR, exist_ok=True)
+    shutil.move(os.path.join(RECORDINGS_DIR, filename), os.path.join(PROCESSED_RECORDINGS_DIR, filename))
+
+
+# write a spectrogram using the sound from start to end of the event
+def write_event(current_event, soundfile, samplerate):
+    # date and filename
+    event_date = current_event['start_at'] - datetime.timedelta(seconds=PLOT_PADDING_START_SECONDS)
+    filename_prefix = event_date.strftime('%Y-%m-%d_%H-%M-%S.%f%z')
+
+    # event clip
+    event_start_sample = current_event['start_sample'] - samplerate * PLOT_PADDING_START_SECONDS
+    event_end_sample = current_event['end_sample'] + samplerate * PLOT_PADDING_END_SECONDS
+    total_samples = event_end_sample - event_start_sample
+    soundfile.seek(event_start_sample)
+    event_clip = soundfile.read(frames=total_samples, dtype='float32', always_2d=False)
+
+    # write flac
+    flac_path = os.path.join(DETECTIONS_DIR, f"{filename_prefix}.flac")
+    sf.write(flac_path, event_clip, samplerate, format='FLAC')
+
+    # write spectrogram
+    plt.figure(figsize=(8, 6))
+    plt.specgram(event_clip, Fs=samplerate, NFFT=samplerate, noverlap=samplerate//2, cmap='inferno', vmin=-100, vmax=-10)
+    plt.title(f"Bootshorn @{event_date.strftime('%Y-%m-%d %H:%M:%S%z')}")
+    plt.xlabel(f"Time {current_event['duration']:.1f}s")
+    plt.ylabel(f"Frequency {current_event['start_freq']:.1f}Hz -> {current_event['end_freq']:.1f}Hz")
+    plt.colorbar(label="Intensity (rDB)")
+    plt.ylim(50, 1000)
+    plt.savefig(os.path.join(DETECTIONS_DIR, f"{filename_prefix}.png"))
+    plt.close()
+
+
+def main():
+    os.makedirs(RECORDINGS_DIR, exist_ok=True)
+    os.makedirs(PROCESSED_RECORDINGS_DIR, exist_ok=True)
+
+    for filename in sorted(os.listdir(RECORDINGS_DIR)):
+        if filename.endswith(".flac"):
+            try:
+                process_recording(filename)
+            except Exception as e:
+                print(f"Error processing {filename}: {e}")
+                # print stacktrace
+                traceback.print_exc()
+
+
+if __name__ == "__main__":
+    main()

bundles/bootshorn/files/record

@@ -0,0 +1,25 @@
+#!/bin/sh
+
+mkdir -p recordings
+
+while true
+do
+    # get date in ISO 8601 format with nanoseconds
+    PROGRAMM=$(test $(uname) = "Darwin" && echo "gdate" || echo "date")
+    DATE=$($PROGRAMM "+%Y-%m-%d_%H-%M-%S.%6N%z")
+
+    # record audio using ffmpeg
+    ffmpeg \
+        -y \
+        -f pulse \
+        -i "alsa_input.usb-HANMUS_USB_AUDIO_24BIT_2I2O_1612310-00.analog-stereo" \
+        -ac 1 \
+        -ar 96000 \
+        -sample_fmt s32 \
+        -t "3600" \
+        -c:a flac \
+        -compression_level 12 \
+        "recordings/current/$DATE.flac"
+
+    mv "recordings/current/$DATE.flac" "recordings/$DATE.flac"
+done

bundles/bootshorn/files/temperature

@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+
+import requests
+import urllib3
+import datetime
+import csv
+urllib3.disable_warnings()
+import os
+
+
+HUE_IP = "${hue_ip}"  # replace with your bridge IP
+HUE_APP_KEY = "${hue_app_key}" # local only
+HUE_DEVICE_ID = "31f58786-3242-4e88-b9ce-23f44ba27bbe"
+TEMPERATURE_LOG_DIR = "/opt/bootshorn/temperatures"
+
+response = requests.get(
+    f"https://{HUE_IP}/clip/v2/resource/temperature",
+    headers={"hue-application-key": HUE_APP_KEY},
+    verify=False,
+)
+response.raise_for_status()
+data = response.json()
+
+for item in data["data"]:
+    if item["id"] == HUE_DEVICE_ID:
+        temperature = item["temperature"]["temperature"]
+        temperature_date_string = item["temperature"]["temperature_report"]["changed"]
+        temperature_date = datetime.datetime.fromisoformat(temperature_date_string).astimezone(datetime.timezone.utc)
+        break
+
+print(f"@{temperature_date}: {temperature}°C")
+
+filename = temperature_date.strftime("%Y-%m-%d_00-00-00.000000%z") + ".log"
+logpath  = os.path.join(TEMPERATURE_LOG_DIR, filename)
+now_utc = datetime.datetime.now(datetime.timezone.utc)
+
+with open(logpath, "a+", newline="") as logfile:
+    writer = csv.writer(logfile)
+    writer.writerow([
+        now_utc.strftime('%Y-%m-%d_%H-%M-%S.%f%z'), # current UTC time
+        temperature_date.strftime('%Y-%m-%d_%H-%M-%S.%f%z'), # date of temperature reading
+        temperature,
+    ])

bundles/bootshorn/items.py

@@ -0,0 +1,61 @@
+# nano /etc/selinux/config
+# SELINUX=disabled
+# reboot
+
+directories = {
+    '/opt/bootshorn': {
+        'owner': 'ckn',
+        'group': 'ckn',
+    },
+    '/opt/bootshorn/temperatures': {
+        'owner': 'ckn',
+        'group': 'ckn',
+    },
+    '/opt/bootshorn/recordings': {
+        'owner': 'ckn',
+        'group': 'ckn',
+    },
+    '/opt/bootshorn/recordings/current': {
+        'owner': 'ckn',
+        'group': 'ckn',
+    },
+    '/opt/bootshorn/recordings/processed': {
+        'owner': 'ckn',
+        'group': 'ckn',
+    },
+    '/opt/bootshorn/events': {
+        'owner': 'ckn',
+        'group': 'ckn',
+    },
+}
+
+files = {
+    '/opt/bootshorn/record': {
+        'owner': 'ckn',
+        'group': 'ckn',
+        'mode': '755',
+    },
+    '/opt/bootshorn/temperature': {
+        'content_type': 'mako',
+        'context': {
+            'hue_ip': repo.get_node('home.hue').hostname,
+            'hue_app_key': repo.vault.decrypt('encrypt$gAAAAABoc2WxZCLbxl-Z4IrSC97CdOeFgBplr9Fp5ujpd0WCCCPNBUY_WquHN86z8hKLq5Y04dwq8TdJW0PMSOSgTFbGgdp_P1q0jOBLEKaW9IIT1YM88h-JYwLf9QGDV_5oEfvnBCtO'),
+        },
+        'owner': 'ckn',
+        'group': 'ckn',
+        'mode': '755',
+    },
+    '/opt/bootshorn/process': {
+        'owner': 'ckn',
+        'group': 'ckn',
+        'mode': '755',
+    },
+}
+
+svc_systemd = {
+    'bootshorn-record.service': {
+        'needs': {
+            'file:/opt/bootshorn/record',
+        },
+    },
+}

bundles/bootshorn/metadata.py

@@ -0,0 +1,44 @@
+defaults = {
+    'systemd': {
+        'units': {
+            'bootshorn-record.service': {
+                'Unit': {
+                    'Description': 'Bootshorn Recorder',
+                    'After': 'network.target',
+                },
+                'Service': {
+                    'User': 'ckn',
+                    'Group': 'ckn',
+                    'Type': 'simple',
+                    'WorkingDirectory': '/opt/bootshorn',
+                    'ExecStart': '/opt/bootshorn/record',
+                    'Restart': 'always',
+                    'RestartSec': 5,
+                    'Environment': {
+                        "XDG_RUNTIME_DIR": "/run/user/1000",
+                        "PULSE_SERVER": "unix:/run/user/1000/pulse/native",
+                    },
+                },
+            },
+        },
+    },
+    'systemd-timers': {
+        'bootshorn-temperature': {
+            'command': '/opt/bootshorn/temperature',
+            'when': '*:0/10',
+            'working_dir': '/opt/bootshorn',
+            'user': 'ckn',
+            'group': 'ckn',
+        },
+        # 'bootshorn-process': {
+        #     'command': '/opt/bootshorn/process',
+        #     'when': 'hourly',
+        #     'working_dir': '/opt/bootshorn',
+        #     'user': 'ckn',
+        #     'group': 'ckn',
+        #     'after': {
+        #         'bootshorn-process.service',
+        #     },
+        # },
+    },
+}

bundles/build-agent/metadata.py

@@ -27,7 +27,7 @@ def ssh_keys(metadata):
         'users': {
             'build-agent': {
                 'authorized_users': {
-                    f'build-server@{other_node.name}'
+                    f'build-server@{other_node.name}': {}
                         for other_node in repo.nodes
                         if other_node.has_bundle('build-server')
                         for architecture in other_node.metadata.get('build-server/architectures').values()

bundles/build-ci/metadata.py

@@ -14,7 +14,7 @@ def ssh_keys(metadata):
         'users': {
             'build-ci': {
                 'authorized_users': {
-                    f'build-server@{other_node.name}'
+                    f'build-server@{other_node.name}': {}
                         for other_node in repo.nodes
                         if other_node.has_bundle('build-server')
                 },

bundles/crystal/metadata.py

@@ -8,6 +8,7 @@ defaults = {
         'sources': {
             'crystal': {
                 # https://software.opensuse.org/download.html?project=devel%3Alanguages%3Acrystal&package=crystal
+                # curl -fsSL https://download.opensuse.org/repositories/devel:/languages:/crystal/Debian_Testing/Release.key
                 'urls': {
                     'http://download.opensuse.org/repositories/devel:/languages:/crystal/Debian_Testing/',
                 },

bundles/dovecot/files/dovecot-sql.conf

@@ -1,17 +0,0 @@
-connect = host=${host} dbname=${name} user=${user} password=${password}
-driver = pgsql
-default_pass_scheme = ARGON2ID
-
-user_query = SELECT '/var/vmail/%u' AS home, 'vmail' AS uid, 'vmail' AS gid
-
-iterate_query = SELECT CONCAT(users.name, '@', domains.name) AS user \
-  FROM users \
-  LEFT JOIN domains ON users.domain_id = domains.id \
-  WHERE redirect IS NULL
-
-password_query = SELECT CONCAT(users.name, '@', domains.name) AS user, password \
-  FROM users \
-  LEFT JOIN domains ON users.domain_id = domains.id \
-  WHERE redirect IS NULL \
-  AND users.name = SPLIT_PART('%u', '@', 1) \
-  AND domains.name = SPLIT_PART('%u', '@', 2)

bundles/dovecot/files/dovecot.conf

@@ -1,13 +1,17 @@
+dovecot_config_version = ${config_version}
+dovecot_storage_version = ${storage_version}
+
 protocols = imap lmtp sieve
 auth_mechanisms = plain login
-mail_privileged_group = mail
 ssl = required
-ssl_cert = </var/lib/dehydrated/certs/${node.metadata.get('mailserver/hostname')}/fullchain.pem
+ssl_server_cert_file = /var/lib/dehydrated/certs/${hostname}/fullchain.pem
-ssl_key = </var/lib/dehydrated/certs/${node.metadata.get('mailserver/hostname')}/privkey.pem
+ssl_server_key_file = /var/lib/dehydrated/certs/${hostname}/privkey.pem
-ssl_dh = </etc/dovecot/dhparam.pem
+ssl_server_dh_file = /etc/dovecot/dhparam.pem
 ssl_client_ca_dir = /etc/ssl/certs
-mail_location = maildir:${node.metadata.get('mailserver/maildir')}/%u:INDEX=${node.metadata.get('mailserver/maildir')}/index/%u
+mail_driver     = maildir
-mail_plugins = fts fts_xapian
+mail_path       = ${maildir}/%{user}
+mail_index_path = ${maildir}/index/%{user}
+mail_plugins = fts fts_flatcurve
 
 namespace inbox {
   inbox = yes
@@ -30,14 +34,46 @@ namespace inbox {
   }
 }
 
-passdb {
+# postgres passdb userdb
-  driver = sql
+
-  args = /etc/dovecot/dovecot-sql.conf
+sql_driver = pgsql
+
+pgsql main {
+  parameters {
+    host     = ${db_host}
+    dbname   = ${db_name}
+    user     = ${db_user}
+    password = ${db_password}
+  }
 }
-# use sql for userdb too, to enable iterate_query
+
-userdb {
+passdb sql {
-  driver = sql
+  passdb_default_password_scheme = ARGON2ID
-  args = /etc/dovecot/dovecot-sql.conf
+
+  query = SELECT \
+            CONCAT(users.name, '@', domains.name) AS "user", \
+            password \
+          FROM users \
+          LEFT JOIN domains ON users.domain_id = domains.id \
+          WHERE redirect IS NULL \
+            AND users.name   = SPLIT_PART('%{user}', '@', 1) \
+            AND domains.name = SPLIT_PART('%{user}', '@', 2)
+}
+
+mail_uid = vmail
+mail_gid = vmail
+
+userdb sql {
+  query = SELECT \
+            '/var/vmail/%{user}' AS home, \
+            'vmail'              AS uid, \
+            'vmail'              AS gid
+
+  iterate_query = SELECT \
+                    CONCAT(users.name, '@', domains.name) AS username \
+                  FROM users \
+                  LEFT JOIN domains ON users.domain_id = domains.id \
+                  WHERE redirect IS NULL
 }
 
 service auth {
@@ -67,10 +103,9 @@ service stats {
   }
 }
 service managesieve-login {
-  inet_listener sieve {
+  #inet_listener sieve {}
-  }
+  process_min_avail = 1
-  process_min_avail = 0
+  process_limit = 1
-  service_count = 1
   vsz_limit = 64 M
 }
 service managesieve {
@@ -78,31 +113,53 @@ service managesieve {
 }
 
 protocol imap {
-   mail_plugins = $mail_plugins imap_sieve
+  mail_plugins = fts fts_flatcurve imap_sieve
-   mail_max_userip_connections = 50
+  mail_max_userip_connections = 50
-   imap_idle_notify_interval = 29 mins
+  imap_idle_notify_interval = 29 mins
 }
 protocol lmtp {
-   mail_plugins = $mail_plugins sieve
+  mail_plugins = fts fts_flatcurve sieve
 }
-protocol sieve {
+
-  plugin {
+# Persönliches Skript (deine alte Datei /var/vmail/sieve/%u.sieve)
-    sieve = /var/vmail/sieve/%u.sieve
+sieve_script personal {
-    sieve_storage = /var/vmail/sieve/%u/
+  driver      = file
-  }
+  # Verzeichnis mit (evtl. mehreren) Sieve-Skripten des Users
+  path        = /var/vmail/sieve/%{user}/
+  # Aktives Skript (entspricht früher "sieve = /var/vmail/sieve/%u.sieve")
+  active_path = /var/vmail/sieve/%{user}.sieve
+}
+
+# Globales After-Skript (dein früheres "sieve_after = …")
+sieve_script after {
+  type   = after
+  driver = file
+  path   = /var/vmail/sieve/global/spam-to-folder.sieve
 }
 
 # fulltext search
-plugin {
+language en {
-    fts = xapian
-    fts_xapian = partial=3 full=20 verbose=0
-    fts_autoindex = yes
-    fts_enforced = yes
-    # Index attachements
-    fts_decoder = decode2text
 }
+language de {
+  default = yes
+}
+language_tokenizers =  generic email-address
+
+fts flatcurve {
+  substring_search = yes
+  # rotate_count     = 5000   # DB-Rotation nach X Mails
+  # rotate_time      = 5s     # oder zeitbasiert rotieren
+  # optimize_limit   = 10
+  # min_term_size    = 3
+}
+
+fts_autoindex = yes
+fts_decoder_driver = script
+fts_decoder_script_socket_path = decode2text
+
 service indexer-worker {
-    vsz_limit = ${indexer_ram}
+  process_limit = ${indexer_cores}
+  vsz_limit = ${indexer_ram}M
 }
 service decode2text {
     executable = script /usr/local/libexec/dovecot/decode2text.sh
@@ -112,24 +169,39 @@ service decode2text {
     }
 }
 
-# spam filter
+mailbox Junk {
-plugin {
+  sieve_script learn_spam {
-  sieve_plugins = sieve_imapsieve sieve_extprograms
+    driver = file
-  sieve_dir = /var/vmail/sieve/%u/
+    type   = before
-  sieve = /var/vmail/sieve/%u.sieve
+    cause  = copy
-  sieve_pipe_bin_dir = /var/vmail/sieve/bin
+    path   = /var/vmail/sieve/global/learn-spam.sieve
-  sieve_extensions = +vnd.dovecot.pipe
+  }
-
-  sieve_after = /var/vmail/sieve/global/spam-to-folder.sieve
-
-  # From elsewhere to Spam folder
-  imapsieve_mailbox1_name = Junk
-  imapsieve_mailbox1_causes = COPY
-  imapsieve_mailbox1_before = file:/var/vmail/sieve/global/learn-spam.sieve
-
-  # From Spam folder to elsewhere
-  imapsieve_mailbox2_name = *
-  imapsieve_mailbox2_from = Junk
-  imapsieve_mailbox2_causes = COPY
-  imapsieve_mailbox2_before = file:/var/vmail/sieve/global/learn-ham.sieve
 }
+
+imapsieve_from Junk {
+  sieve_script learn_ham {
+    driver = file
+    type   = before
+    cause  = copy
+    path   = /var/vmail/sieve/global/learn-ham.sieve
+  }
+}
+
+# Extprograms-Plugin einschalten
+sieve_plugins {
+  sieve_extprograms = yes
+}
+
+# Welche Sieve-Erweiterungen dürfen genutzt werden?
+# Empfehlung: nur global erlauben (nicht in User-Skripten):
+sieve_global_extensions {
+  vnd.dovecot.pipe = yes
+  # vnd.dovecot.filter = yes      # nur falls gebraucht
+  # vnd.dovecot.execute = yes     # nur falls gebraucht
+}
+
+# Verzeichnis mit deinen Skripten/Binaries für :pipe
+sieve_pipe_bin_dir = /var/vmail/sieve/bin
+# (optional, analog für :filter / :execute)
+# sieve_filter_bin_dir  = /var/vmail/sieve/filter
+# sieve_execute_bin_dir = /var/vmail/sieve/execute

bundles/dovecot/items.py

@@ -44,6 +44,16 @@ files = {
         'context': {
             'admin_email': node.metadata.get('mailserver/admin_email'),
             'indexer_ram': node.metadata.get('dovecot/indexer_ram'),
+            'config_version': node.metadata.get('dovecot/config_version'),
+            'storage_version': node.metadata.get('dovecot/storage_version'),
+            'maildir': node.metadata.get('mailserver/maildir'),
+            'hostname': node.metadata.get('mailserver/hostname'),
+            'db_host': node.metadata.get('mailserver/database/host'),
+            'db_name': node.metadata.get('mailserver/database/name'),
+            'db_user': node.metadata.get('mailserver/database/user'),
+            'db_password': node.metadata.get('mailserver/database/password'),
+            'indexer_cores': node.metadata.get('vm/cores'),
+            'indexer_ram': node.metadata.get('vm/ram')//2,
         },
         'needs': {
             'pkg_apt:'
@@ -52,29 +62,9 @@ files = {
             'svc_systemd:dovecot:restart',
         },
     },
-    '/etc/dovecot/dovecot-sql.conf': {
-        'content_type': 'mako',
-        'context': node.metadata.get('mailserver/database'),
-        'needs': {
-            'pkg_apt:'
-        },
-        'triggers': {
-            'svc_systemd:dovecot:restart',
-        },
-    },
     '/etc/dovecot/dhparam.pem': {
         'content_type': 'any',
     },
-    '/etc/dovecot/dovecot-sql.conf': {
-        'content_type': 'mako',
-        'context': node.metadata.get('mailserver/database'),
-        'needs': {
-            'pkg_apt:'
-        },
-        'triggers': {
-            'svc_systemd:dovecot:restart',
-        },
-    },
     '/var/vmail/sieve/global/spam-to-folder.sieve': {
         'owner': 'vmail',
         'group': 'vmail',
@@ -131,7 +121,6 @@ svc_systemd = {
             'action:letsencrypt_update_certificates',
             'action:dovecot_generate_dhparam',
             'file:/etc/dovecot/dovecot.conf',
-            'file:/etc/dovecot/dovecot-sql.conf',
         },
     },
 }

bundles/dovecot/metadata.py

@@ -8,7 +8,7 @@ defaults = {
             'dovecot-sieve':        {},
             'dovecot-managesieved': {},
             # fulltext search
-            'dovecot-fts-xapian':   {}, # buster-backports
+            'dovecot-flatcurve':    {}, # buster-backports
             'poppler-utils':        {}, # pdftotext
             'catdoc':               {}, # catdoc, catppt, xls2csv
         },

bundles/download-server/metadata.py

@@ -5,6 +5,11 @@ defaults = {
             'needs': {
                 'zfs_dataset:tank/downloads'
             },
+            'authorized_users': {
+                f'build-server@{other_node.name}': {}
+                    for other_node in repo.nodes
+                    if other_node.has_bundle('build-server')
+            },
         },
     },
     'zfs': {
@@ -14,21 +19,13 @@ defaults = {
             },
         },
     },
-}
+    'systemd-mount': {
-
+        '/var/lib/downloads_nginx': {
-
+            'source': '/var/lib/downloads',
-@metadata_reactor.provides(
+            'user': 'www-data',
-    'systemd-mount'
-)
-def mount_certs(metadata):
-    return  {
-        'systemd-mount': {
-            '/var/lib/downloads_nginx': {
-                'source': '/var/lib/downloads',
-                'user': 'www-data',
-            },
         },
-    }
+    },
+}
 
 
 @metadata_reactor.provides(
@@ -47,20 +44,3 @@ def nginx(metadata):
             },
         },
     }
-
-
-@metadata_reactor.provides(
-    'users/downloads/authorized_users',
-)
-def ssh_keys(metadata):
-    return {
-        'users': {
-            'downloads': {
-                'authorized_users': {
-                    f'build-server@{other_node.name}'
-                        for other_node in repo.nodes
-                        if other_node.has_bundle('build-server')
-                },
-            },
-        },
-    }

bundles/flask/metadata.py

@@ -43,11 +43,11 @@ def units(metadata):
                     'Service': {
                         'Environment': {
                             f'{k}={v}'
-                                for k, v in conf.get('env', {}).items()
+                                for k, v in metadata.get(f'flask/{name}/env', {}).items()
                         },
-                        'User': conf['user'],
+                        'User': metadata.get(f'flask/{name}/user'),
-                        'Group': conf['group'],
+                        'Group': metadata.get(f'flask/{name}/group'),
-                        'ExecStart': f"/opt/{name}/venv/bin/gunicorn -w {conf['workers']} -b 127.0.0.1:{conf['port']} --timeout {conf['timeout']} {conf['app_module']}:app"
+                        'ExecStart': f"/opt/{name}/venv/bin/gunicorn -w {metadata.get(f'flask/{name}/workers')} -b 127.0.0.1:{metadata.get(f'flask/{name}/port')} --timeout {metadata.get(f'flask/{name}/timeout')} {metadata.get(f'flask/{name}/app_module')}:app"
                     },
                     'Install': {
                         'WantedBy': {
@@ -55,7 +55,7 @@ def units(metadata):
                         }
                     },
                 }
-                    for name, conf in metadata.get('flask').items()
+                    for name in metadata.get('flask')
             }
         }
     }

bundles/freescout/README.md

@@ -11,3 +11,13 @@ Enter it again:
 freescout=#
 \q
 ```
+
+
+# problems
+
+# check if /opt/freescout/.env is resettet
+# ckeck `psql -h localhost -d freescout -U freescout -W`with pw from .env
+# chown -R www-data:www-data /opt/freescout
+# sudo su - www-data -c 'php /opt/freescout/artisan freescout:clear-cache' -s /bin/bash
+# javascript funny? `sudo su - www-data -c 'php /opt/freescout/artisan storage:link' -s /bin/bash`
+# benutzer bilder weg? aus dem backup holen: `/opt/freescout/.zfs/snapshot/zfs-auto-snap_hourly-2024-11-22-1700/storage/app/public/users` `./customers`

bundles/freescout/items.py

@@ -12,33 +12,33 @@ directories = {
 }
 
 actions = {
-    'clone_freescout': {
+    # 'clone_freescout': {
-        'command': run_as('www-data', 'git clone https://github.com/freescout-helpdesk/freescout.git /opt/freescout'),
+    #     'command': run_as('www-data', 'git clone https://github.com/freescout-helpdesk/freescout.git /opt/freescout'),
-        'unless': 'test -e /opt/freescout/.git',
+    #     'unless': 'test -e /opt/freescout/.git',
-        'needs': [
+    #     'needs': [
-            'pkg_apt:git',
+    #         'pkg_apt:git',
-            'directory:/opt/freescout',
+    #         'directory:/opt/freescout',
-        ],
+    #     ],
-    },
+    # },
-    'pull_freescout': {
+    # 'pull_freescout': {
-        'command': run_as('www-data', 'git -C /opt/freescout fetch origin dist && git -C /opt/freescout reset --hard origin/dist && git -C /opt/freescout clean -f'),
+    #     'command': run_as('www-data', 'git -C /opt/freescout fetch origin dist && git -C /opt/freescout reset --hard origin/dist && git -C /opt/freescout clean -f'),
-        'unless': run_as('www-data', 'git -C /opt/freescout fetch origin && git -C /opt/freescout status -uno | grep -q "Your branch is up to date"'),
+    #     'unless': run_as('www-data', 'git -C /opt/freescout fetch origin && git -C /opt/freescout status -uno | grep -q "Your branch is up to date"'),
-        'needs': [
+    #     'needs': [
-            'action:clone_freescout',
+    #         'action:clone_freescout',
-        ],
+    #     ],
-        'triggers': [
+    #     'triggers': [
-            'action:freescout_artisan_update',
+    #         'action:freescout_artisan_update',
-            f'svc_systemd:php{php_version}-fpm.service:restart',
+    #         f'svc_systemd:php{php_version}-fpm.service:restart',
-        ],
+    #     ],
-    },
+    # },
-    'freescout_artisan_update': {
+    # 'freescout_artisan_update': {
-        'command': run_as('www-data', 'php /opt/freescout/artisan freescout:after-app-update'),
+    #     'command': run_as('www-data', 'php /opt/freescout/artisan freescout:after-app-update'),
-        'triggered': True,
+    #     'triggered': True,
-        'needs': [
+    #     'needs': [
-            f'svc_systemd:php{php_version}-fpm.service:restart',
+    #         f'svc_systemd:php{php_version}-fpm.service:restart',
-            'action:pull_freescout',
+    #         'action:pull_freescout',
-        ],
+    #     ],
-    },
+    # },
 }
 
 # svc_systemd = {

bundles/gitea/files/app.ini

@@ -40,7 +40,7 @@ ENABLE_OPENID_SIGNUP = false
 [service]
 REGISTER_EMAIL_CONFIRM = true
 ENABLE_NOTIFY_MAIL = true
-DISABLE_REGISTRATION = false
+DISABLE_REGISTRATION = true
 ALLOW_ONLY_EXTERNAL_REGISTRATION = false
 ENABLE_CAPTCHA = false
 REQUIRE_SIGNIN_VIEW = false

bundles/gitea/items.py

@@ -49,7 +49,7 @@ files['/etc/gitea/app.ini'] = {
     ),
     'owner': 'git',
     'mode': '0600',
-    'context': node.metadata['gitea'],
+    'context': node.metadata.get('gitea'),
     'triggers': {
         'svc_systemd:gitea:restart',
     },

bundles/grafana/items.py

@@ -127,7 +127,7 @@ for dashboard_id, monitored_node in enumerate(monitored_nodes, start=1):
             panel['gridPos']['y'] = (row_id - 1) * panel['gridPos']['h']
 
             if 'display_name' in panel_config:
-                panel['fieldConfig']['defaults']['displayName'] = '${'+panel_config['display_name']+'}'
+                panel['fieldConfig']['defaults']['displayName'] = panel_config['display_name']
 
             if panel_config.get('stacked'):
                 panel['fieldConfig']['defaults']['custom']['stacking']['mode'] = 'normal'
@@ -158,13 +158,14 @@ for dashboard_id, monitored_node in enumerate(monitored_nodes, start=1):
                         host=monitored_node.name,
                         negative=query_config.get('negative', False),
                         boolean_to_int=query_config.get('boolean_to_int', False),
-                        minimum=query_config.get('minimum', None),
+                        over=query_config.get('over', None),
                         filters={
                             'host': monitored_node.name,
                             **query_config['filters'],
                         },
                         exists=query_config.get('exists', []),
                         function=query_config.get('function', None),
+                        multiply=query_config.get('multiply', None),
                     ).strip()
                 })
 
@@ -178,4 +179,3 @@ for dashboard_id, monitored_node in enumerate(monitored_nodes, start=1):
             'svc_systemd:grafana-server:restart',
         ]
     }
-

bundles/grafana/metadata.py

@@ -26,9 +26,15 @@ defaults = {
         'config': {
             'server': {
                 'http_port': 8300,
+                'http_addr': '127.0.0.1',
+                'enable_gzip': True,
             },
             'database': {
-                'url': f'postgres://grafana:{postgres_password}@localhost:5432/grafana',
+                'type': 'postgres',
+                'host': '127.0.0.1:5432',
+                'name': 'grafana',
+                'user': 'grafana',
+                'password': postgres_password,
             },
             'remote_cache': {
                 'type': 'redis',
@@ -133,11 +139,13 @@ def dns(metadata):
 
 
 @metadata_reactor.provides(
+    'nginx/has_websockets',
     'nginx/vhosts',
 )
 def nginx(metadata):
     return {
         'nginx': {
+            'has_websockets': True,
             'vhosts': {
                 metadata.get('grafana/hostname'): {
                     'content': 'grafana/vhost.conf',

bundles/hardware/items.py

@@ -2,7 +2,7 @@ files = {
     '/usr/local/share/telegraf/cpu_frequency': {
         'mode': '0755',
         'triggers': {
-            'svc_systemd:telegraf:restart',
+            'svc_systemd:telegraf.service:restart',
         },
     },
 }

bundles/hardware/metadata.py

@@ -14,25 +14,25 @@ defaults = {
         },
     },
     'telegraf': {
-        'config': {
+        'inputs': {
-            'inputs': {
+            'sensors': {
-                'sensors': {repo.libs.hashable.hashable({
+                'default': {
                     'timeout': '2s',
-                })},
-                'exec': {
-                    repo.libs.hashable.hashable({
-                        'commands': ["sudo /usr/local/share/telegraf/cpu_frequency"],
-                        'name_override': "cpu_frequency",
-                        'data_format': "influx",
-                    }),
-                    # repo.libs.hashable.hashable({
-                    #     'commands': ["/bin/bash -c 'expr $(cat /sys/class/thermal/thermal_zone0/temp) / 1000'"],
-                    #     'name_override': "cpu_temperature",
-                    #     'data_format': "value",
-                    #     'data_type': "integer",
-                    # }),
                 },
             },
+            'exec': {
+                'cpu_frequency': {
+                    'commands': ["sudo /usr/local/share/telegraf/cpu_frequency"],
+                    'name_override': "cpu_frequency",
+                    'data_format': "influx",
+                },
+                # repo.libs.hashable.hashable({
+                #     'commands': ["/bin/bash -c 'expr $(cat /sys/class/thermal/thermal_zone0/temp) / 1000'"],
+                #     'name_override': "cpu_temperature",
+                #     'data_format': "value",
+                #     'data_type': "integer",
+                # }),
+            },
         },
     },
 }

bundles/homeassistant-supervised/README.md

@@ -1,23 +0,0 @@
-https://github.com/home-assistant/supervised-installer?tab=readme-ov-file
-https://github.com/home-assistant/os-agent/tree/main?tab=readme-ov-file#using-home-assistant-supervised-on-debian
-https://docs.docker.com/engine/install/debian/
-
-
-
-
-
-https://www.home-assistant.io/installation/linux#install-home-assistant-supervised
-https://github.com/home-assistant/supervised-installer
-https://github.com/home-assistant/architecture/blob/master/adr/0014-home-assistant-supervised.md
-
-DATA_SHARE=/usr/share/hassio dpkg --force-confdef --force-confold -i homeassistant-supervised.deb
-
-neu debian
-ha installieren
-gucken ob geht
-dann bw drüberbügeln
-
-
-https://www.home-assistant.io/integrations/http/#ssl_certificate
-
-`wget "$(curl -L https://api.github.com/repos/home-assistant/supervised-installer/releases/latest | jq -r '.assets[0].browser_download_url')" -O homeassistant-supervised.deb && dpkg -i homeassistant-supervised.deb`

bundles/homeassistant-supervised/items.py

@@ -1,30 +0,0 @@
-from shlex import quote
-
-
-version = node.metadata.get('homeassistant/os_agent_version')
-
-directories = {
-    '/usr/share/hassio': {},
-}
-
-actions = {
-    'install_os_agent': {
-        'command': ' && '.join([
-            f'wget -O /tmp/os-agent.deb https://github.com/home-assistant/os-agent/releases/download/{quote(version)}/os-agent_{quote(version)}_linux_aarch64.deb',
-            'DEBIAN_FRONTEND=noninteractive dpkg -i /tmp/os-agent.deb',
-        ]),
-        'unless': f'test "$(apt -qq list os-agent | cut -d" " -f2)" = "{quote(version)}"',
-        'needs': {
-            'pkg_apt:',
-            'zfs_dataset:tank/homeassistant',
-        },
-    },
-    'install_homeassistant_supervised': {
-        'command': 'wget -O /tmp/homeassistant-supervised.deb https://github.com/home-assistant/supervised-installer/releases/latest/download/homeassistant-supervised.deb && apt install /tmp/homeassistant-supervised.deb',
-        'unless': 'apt -qq list homeassistant-supervised | grep -q "installed"',
-        'needs': {
-            'action:install_os_agent',
-        },
-    },
-}
-

bundles/homeassistant-supervised/metadata.py

@@ -1,65 +0,0 @@
-defaults = {
-    'apt': {
-        'packages': {
-            # homeassistant-supervised
-            'apparmor': {},
-            'bluez': {},
-            'cifs-utils': {},
-            'curl': {},
-            'dbus': {},
-            'jq': {},
-            'libglib2.0-bin': {},
-            'lsb-release': {},
-            'network-manager': {},
-            'nfs-common': {},
-            'systemd-journal-remote': {},
-            'systemd-resolved': {},
-            'udisks2': {},
-            'wget': {},
-            # docker
-            'docker-ce': {},
-            'docker-ce-cli': {},
-            'containerd.io': {},
-            'docker-buildx-plugin': {},
-            'docker-compose-plugin': {},
-       },
-        'sources': {
-            # docker: https://docs.docker.com/engine/install/debian/#install-using-the-repository
-            'docker': {
-                'urls': {
-                    'https://download.docker.com/linux/debian',
-                },
-                'suites': {
-                    '{codename}',
-                },
-                'components': {
-                    'stable',
-                },
-            },
-        },
-    },
-    'zfs': {
-        'datasets': {
-            'tank/homeassistant': {
-                'mountpoint': '/usr/share/hassio',
-                'needed_by': {
-                    'directory:/usr/share/hassio',
-                },
-            },
-        },
-    },
-}
-
-@metadata_reactor.provides(
-    'nginx/vhosts',
-)
-def nginx(metadata):
-    return {
-        'nginx': {
-            'vhosts': {
-                metadata.get('homeassistant/domain'): {
-                    'content': 'homeassistant/vhost.conf',
-                },
-            },
-        },
-    }

bundles/icingaweb2/metadata.py

@@ -179,6 +179,7 @@ def nginx(metadata):
                     'context': {
                         'php_version': metadata.get('php/version'),
                     },
+                    'check_path': '/icingaweb2/index.php',
                 },
             },
         },

bundles/ifupdown/items.py

@@ -0,0 +1,3 @@
+# svc_systemd = {
+#     'ifupdown.service': {},
+# }

bundles/influxdb2/metadata.py

@@ -39,6 +39,17 @@ defaults = {
     },
 }
 
+if node.has_bundle('zfs'):
+    defaults['zfs'] = {
+        'datasets': {
+            'tank/influxdb': {
+                'mountpoint': '/var/lib/influxdb',
+                'recordsize': '8192',
+                'atime': 'off',
+            },
+        },
+    }
+
 @metadata_reactor.provides(
     'influxdb/password',
     'influxdb/admin_token',
@@ -52,26 +63,6 @@ def admin_password(metadata):
     }
 
 
-@metadata_reactor.provides(
-    'zfs/datasets',
-)
-def zfs(metadata):
-    if not node.has_bundle('zfs'):
-        return {}
-
-    return {
-        'zfs': {
-            'datasets': {
-                'tank/influxdb': {
-                    'mountpoint': '/var/lib/influxdb',
-                    'recordsize': '8192',
-                    'atime': 'off',
-                },
-            },
-        },
-    }
-
-
 @metadata_reactor.provides(
     'dns',
 )

bundles/kea-dhcpd/items.py

@@ -15,6 +15,7 @@ svc_systemd = {
         'needs': [
             'pkg_apt:kea-dhcp4-server',
             'file:/etc/kea/kea-dhcp4.conf',
+            'svc_systemd:systemd-networkd.service:restart',
         ],
     },
 }

bundles/kea-dhcpd/metadata.py

@@ -52,13 +52,14 @@ def subnets(metadata):
             if 'mac' in network_conf
     )
 
-    for network_name, network_conf in metadata.get('network').items():
+    for id, (network_name, network_conf) in enumerate(sorted(metadata.get('network').items())):
         dhcp_server_config = network_conf.get('dhcp_server_config', None)
 
         if dhcp_server_config:
             _network = ip_network(dhcp_server_config['subnet'])
 
             subnet4.add(hashable({
+                'id': id + 1,
                 'subnet': dhcp_server_config['subnet'],
                 'pools': [
                     {
@@ -72,7 +73,7 @@ def subnets(metadata):
                     },
                     {
                         'name': 'domain-name-servers',
-                        'data': '10.0.10.2',
+                        'data': '10.0.0.1',
                     },
                 ],
                 'reservations': set(

bundles/left4dead2/README.md

@@ -1,58 +1,22 @@
-https://developer.valvesoftware.com/wiki/List_of_L4D2_Cvars
+https://github.com/SirPlease/L4D2-Competitive-Rework/blob/master/Dedicated%20Server%20Install%20Guide/README.md
 
-Dead Center 	c1m1_hotel
+```python
-Dead Center 	c1m2_streets
+    'tick60_maps': {
-Dead Center 	c1m3_mall
+        'port': 27030,
-Dead Center 	c1m4_atrium
+        # add command line arguments
-Dark Carnival 	c2m1_highway
+        'arguments': ['-tickrate 60'],
-Dark Carnival 	c2m2_fairgrounds
+        # stack overlays, first is uppermost
-Dark Carnival 	c2m3_coaster
+        'overlays': ['tickrate', 'standard'],
-Dark Carnival 	c2m4_barns
+        # server.cfg contents
-Dark Carnival 	c2m5_concert
+        'config': [
-Swamp Fever 	c3m1_plankcountry
+            # configs from overlays are accessible via server_${overlay}.cfg
-Swamp Fever 	c3m2_swamp
+            'exec server_tickrate.cfg',
-Swamp Fever 	c3m3_shantytown
+            # add more options
-Swamp Fever 	c3m4_plantation
+            'sv_minupdaterate 101',
-Hard Rain 	c4m1_milltown_a
+            'sv_maxupdaterate 101',
-Hard Rain 	c4m2_sugarmill_a
+            'sv_mincmdrate 101',
-Hard Rain 	c4m3_sugarmill_b
+            'sv_maxcmdrate 101',
-Hard Rain 	c4m4_milltown_b
+            'sv_consistency 0',
-Hard Rain 	c4m5_milltown_escape
+        ],
-The Parish 	c5m1_waterfront_sndscape
+    },
-The Parish 	c5m1_waterfront
+```
-The Parish 	c5m2_park
-The Parish 	c5m3_cemetery
-The Parish 	c5m4_quarter
-The Parish 	c5m5_bridge
-The Passing 	c6m1_riverbank
-The Passing 	c6m2_bedlam
-The Passing 	c6m3_port
-The Sacrifice 	c7m1_docks
-The Sacrifice 	c7m2_barge
-The Sacrifice 	c7m3_port
-No Mercy 	c8m1_apartment
-No Mercy 	c8m2_subway
-No Mercy 	c8m3_sewers
-No Mercy 	c8m4_interior
-No Mercy 	c8m5_rooftop
-Crash Course 	c9m1_alleys
-Crash Course 	c9m2_lots
-Death Toll 	c10m1_caves
-Death Toll 	c10m2_drainage
-Death Toll 	c10m3_ranchhouse
-Death Toll 	c10m4_mainstreet
-Death Toll 	c10m5_houseboat
-Dead Air 	c11m1_greenhouse
-Dead Air 	c11m2_offices
-Dead Air 	c11m3_garage
-Dead Air 	c11m4_terminal
-Dead Air 	c11m5_runway
-Blood Harvest 	c12m1_hilltop
-Blood Harvest 	c12m2_traintunnel
-Blood Harvest 	c12m3_bridge
-Blood Harvest 	c12m4_barn
-Blood Harvest 	c12m5_cornfield
-Cold Stream 	c13m1_alpinecreek
-Cold Stream 	c13m2_southpinestream
-Cold Stream 	c13m3_memorialbridge
-Cold Stream 	c13m4_cutthroatcreek

bundles/left4dead2/files/scripts/helpers

@@ -0,0 +1,13 @@
+#!/bin/bash
+set -xeuo pipefail
+
+function steam() {
+    # for systemd, so it can terminate the process (for other things sudo would have been enough)
+    setpriv --reuid=steam --regid=steam --init-groups "$@" <&0
+    export HOME=/opt/l4d2/steam
+}
+
+function workshop() {
+    steam mkdir -p "/opt/l4d2/overlays/${overlay}/left4dead2/addons"
+    steam /opt/l4d2/scripts/steam-workshop-download --out "/opt/l4d2/overlays/${overlay}/left4dead2/addons" "$@"
+}

bundles/left4dead2/files/scripts/overlays/competitive_rework

@@ -0,0 +1,10 @@
+#!/bin/bash
+set -xeuo pipefail
+source /opt/l4d2/scripts/helpers
+overlay=$(basename "$0")
+
+# https://github.com/SirPlease/L4D2-Competitive-Rework
+
+steam mkdir -p /opt/l4d2/overlays/$overlay/left4dead2
+test -d /opt/l4d2/overlays/$overlay/left4dead2/cfg/cfgogl || \
+    curl -L https://github.com/SirPlease/L4D2-Competitive-Rework/archive/refs/heads/master.tar.gz | steam tar -xz --strip-components=1 -C /opt/l4d2/overlays/$overlay/left4dead2

bundles/left4dead2/files/scripts/overlays/l4d2center_maps

@@ -0,0 +1,128 @@
+#!/bin/bash
+set -xeuo pipefail
+source /opt/l4d2/scripts/helpers
+overlay=$(basename "$0")
+
+steam mkdir -p /opt/l4d2/overlays/$overlay/left4dead2/addons
+cd /opt/l4d2/overlays/$overlay/left4dead2/addons
+
+# https://l4d2center.com/maps/servers/l4d2center_maps_sync.sh.txt ->
+
+# Exit immediately if a command exits with a non-zero status.
+set -e
+
+# Function to print error messages
+error_exit() {
+    echo "Error: $1" >&2
+    exit 1
+}
+
+# Check if the current directory ends with /left4dead2/addons
+current_dir=$(pwd)
+expected_dir="/left4dead2/addons"
+
+if [[ ! "$current_dir" == *"$expected_dir" ]]; then
+    error_exit "Script must be run from your L4D2 \"addons\" folder. Current directory: $current_dir"
+fi
+
+# Check for required commands
+for cmd in curl md5sum 7z; do
+    if ! command -v "$cmd" >/dev/null 2>&1; then
+        error_exit "Required command '$cmd' is not installed. Please install it and retry."
+    fi
+done
+
+# URL of the CSV file
+CSV_URL="https://l4d2center.com/maps/servers/index.csv"
+
+# Temporary file to store CSV
+TEMP_CSV=$(mktemp)
+
+# Ensure temporary file is removed on exit
+trap 'rm -f "$TEMP_CSV"' EXIT
+
+echo "Downloading CSV from $CSV_URL..."
+curl -sSL -o "$TEMP_CSV" "$CSV_URL" || error_exit "Failed to download CSV."
+
+declare -A map_md5
+declare -A map_links
+
+# Read CSV and populate associative arrays
+{
+    # Skip the first line (header)
+    IFS= read -r header
+
+    while IFS=';' read -r Name Size MD5 DownloadLink || [[ $Name ]]; do
+        # Trim whitespace
+        Name=$(echo "$Name" | xargs)
+        MD5=$(echo "$MD5" | xargs)
+        DownloadLink=$(echo "$DownloadLink" | xargs)
+
+        # Populate associative arrays
+        map_md5["$Name"]="$MD5"
+        map_links["$Name"]="$DownloadLink"
+    done
+} < "$TEMP_CSV"
+
+# Get list of expected VPK files
+expected_vpk=("${!map_md5[@]}")
+
+# Remove VPK files not in expected list or with mismatched MD5
+echo "Cleaning up existing VPK files..."
+for file in *.vpk; do
+    # Check if it's a regular file
+    if [[ -f "$file" ]]; then
+        if [[ -z "${map_md5["$file"]}" ]]; then
+            echo "Removing unexpected file: $file"
+            rm -f "$file"
+        else
+            # Calculate MD5
+            echo "Calculating MD5 for existing file: $file..."
+            current_md5=$(md5sum "$file" | awk '{print $1}')
+            expected_md5="${map_md5["$file"]}"
+
+            if [[ "$current_md5" != "$expected_md5" ]]; then
+                echo "MD5 mismatch for $file. Removing."
+                rm -f "$file"
+            fi
+        fi
+    fi
+done
+
+# Download and extract missing or updated VPK files
+echo "Processing required VPK files..."
+for vpk in "${expected_vpk[@]}"; do
+    if [[ ! -f "$vpk" ]]; then
+        echo "Downloading and extracting $vpk..."
+        download_url="${map_links["$vpk"]}"
+
+        if [[ -z "$download_url" ]]; then
+            echo "No download link found for $vpk. Skipping."
+            continue
+        fi
+
+        encoded_url=$(echo "$download_url" | sed 's/ /%20/g')
+
+        # Download the .7z file to a temporary location
+        TEMP_7Z=$(mktemp --suffix=.7z)
+        curl -# -L -o "$TEMP_7Z" "$encoded_url"
+
+        # Check if the download was successful
+        if [[ $? -ne 0 ]]; then
+            echo "Failed to download $download_url. Skipping."
+            rm -f "$TEMP_7Z"
+            continue
+        fi
+
+        # Extract the .7z file
+        7z x -y "$TEMP_7Z" || { echo "Failed to extract $TEMP_7Z. Skipping."; rm -f "$TEMP_7Z"; continue; }
+
+        # Remove the temporary .7z file
+        rm -f "$TEMP_7Z"
+
+    else
+        echo "$vpk is already up to date."
+    fi
+done
+
+echo "Synchronization complete."

bundles/left4dead2/files/scripts/overlays/standard

@@ -0,0 +1,12 @@
+#!/bin/bash
+set -xeuo pipefail
+source /opt/l4d2/scripts/helpers
+overlay=$(basename "$0")
+
+# Ions Vocalizer
+workshop -i 698857882
+
+# admin system
+workshop --item 2524204971
+steam mkdir -p "/opt/l4d2/overlays/${overlay}/left4dead2/ems/admin system"
+steam echo "STEAM_1:0:12376499" > "/opt/l4d2/overlays/${overlay}/left4dead2/ems/admin system/admins.txt"

bundles/left4dead2/files/scripts/overlays/tickrate

@@ -0,0 +1,25 @@
+#!/bin/bash
+set -xeuo pipefail
+source /opt/l4d2/scripts/helpers
+overlay=$(basename "$0")
+
+# server config
+# https://github.com/SirPlease/L4D2-Competitive-Rework/blob/7ecc3a32a5e2180d6607a40119ff2f3c072502a9/cfg/server.cfg#L58-L69
+# https://www.programmersought.com/article/513810199514/
+steam mkdir -p /opt/l4d2/overlays/$overlay/left4dead2/cfg
+steam cat <<'EOF' > /opt/l4d2/overlays/$overlay/left4dead2/cfg/server.cfg
+# https://github.com/SirPlease/L4D2-Competitive-Rework/blob/7ecc3a32a5e2180d6607a40119ff2f3c072502a9/cfg/server.cfg#L58-L69
+sv_minrate 100000
+sv_maxrate 100000
+nb_update_frequency 0.014
+net_splitpacket_maxrate 50000
+net_maxcleartime 0.0001
+fps_max 0
+EOF
+
+# install tickrate enabler
+steam mkdir -p "/opt/l4d2/overlays/${overlay}/left4dead2/addons"
+for file in tickrate_enabler.dll tickrate_enabler.so tickrate_enabler.vdf
+do
+    curl -L "https://github.com/SirPlease/L4D2-Competitive-Rework/raw/refs/heads/master/addons/${file}" -o "/opt/l4d2/overlays/${overlay}/left4dead2/addons/${file}"
+done

bundles/left4dead2/files/scripts/overlays/workshop_maps

@@ -0,0 +1,13 @@
+#!/bin/bash
+set -xeuo pipefail
+source /opt/l4d2/scripts/helpers
+overlay=$(basename "$0")
+
+# workshop --collection 121115793 # Back To School
+
+# workshop --item 2957035482 # hehe30-part1
+# workshop --item 2973628334 # hehe30-part2
+# workshop --item 3013844371 # hehe30-part3
+
+# workshop --item 3478461158 # 虚伪黎明(Dawn's Deception)
+# workshop --item 3478934394 # 虚伪黎明(Dawn's Deception)PART2

bundles/left4dead2/files/server.cfg

@@ -1,40 +1,13 @@
-hostname "CroneKorkN : ${name}"
+// defaults
-sv_contact "admin@sublimity.de"
+hostname ${server_name}
-
-
-sv_steamgroup "${','.join(steamgroups)}"
-
-rcon_password "${rcon_password}"
-
-
 motd_enabled 0
+rcon_password ${rcon_password}
+sv_steamgroup "38347879"
 
+mp_autoteambalance 0
+sv_forcepreload 1
 
-sv_cheats 1
+// server specific
-
+% for line in config:
-
+${line}
-sv_consistency 0
+% endfor
-
-
-sv_lan 0
-
-
-sv_allow_lobby_connect_only 0
-
-
-sv_gametypes "coop,realism,survival,versus,teamversus,scavenge,teamscavenge"
-
-
-sv_minrate 30000
-sv_maxrate 60000
-sv_mincmdrate 66
-sv_maxcmdrate 101
-
-
-sv_logsdir "logs-${name}"        //Folder in the game directory where server logs will be stored.
-log on            //Creates a logfile (on | off)
-sv_logecho 0            //default 0; Echo log information to the console.
-sv_logfile 1        //default 1; Log server information in the log file.
-sv_log_onefile 0    //default 0; Log server information to only one file.
-sv_logbans 1        //default 0;Log server bans in the server logs.
-sv_logflush 0        //default 0; Flush the log files to disk on each write (slow).

bundles/left4dead2/files/setup

@@ -0,0 +1,72 @@
+#!/bin/bash
+
+set -xeuo pipefail
+
+# -- DEFINE FUNCTIONS AND VARIABLES -- #
+
+function steam() {
+    # for systemd, so it can terminate the process (for other things sudo would have been enough)
+    setpriv --reuid=steam --regid=steam --init-groups "$@" <&0
+    export HOME=/opt/l4d2/steam
+}
+
+# -- PREPARE SYSTEM -- #
+
+getent passwd steam >/dev/null || useradd -M -d /opt/l4d2 -s /bin/bash steam
+mkdir -p /opt/l4d2 /tmp/dumps
+chown steam:steam /opt/l4d2 /tmp/dumps
+dpkg --add-architecture i386
+apt update
+DEBIAN_FRONTEND=noninteractive apt install -y libc6:i386 lib32z1
+
+# workshop downloader
+test -f /opt/l4d2/scripts/steam-workshop-download || \
+    steam wget -4 https://git.sublimity.de/cronekorkn/steam-workshop-downloader/raw/branch/master/steam-workshop-download -P /opt/l4d2/scripts
+steam chmod +x /opt/l4d2/scripts/steam-workshop-download
+
+# -- STEAM -- #
+
+steam mkdir -p /opt/l4d2/steam
+test -f /opt/l4d2/steam/steamcmd_linux.tar.gz || \
+    steam wget http://media.steampowered.com/installer/steamcmd_linux.tar.gz -P /opt/l4d2/steam
+test -f /opt/l4d2/steam/steamcmd.sh || \
+    steam tar -xvzf /opt/l4d2/steam/steamcmd_linux.tar.gz -C /opt/l4d2/steam
+
+# fix for: /opt/l4d2/.steam/sdk32/steamclient.so: cannot open shared object file: No such file or directory
+steam mkdir -p /opt/l4d2/steam/.steam # needs to be in steam users home dir
+readlink /opt/l4d2/steam/.steam/sdk32 | grep -q ^/opt/l4d2/steam/linux32$ || \
+    steam ln -sf /opt/l4d2/steam/linux32 /opt/l4d2/steam/.steam/sdk32
+readlink /opt/l4d2/steam/.steam/sdk64 | grep -q ^/opt/l4d2/steam/linux64$ || \
+    steam ln -sf /opt/l4d2/steam/linux64 /opt/l4d2/steam/.steam/sdk64
+
+# -- INSTALL -- #
+
+# erst die windows deps zu installieren scheint ein workaround für x64 zu sein?
+steam mkdir -p /opt/l4d2/installation
+steam /opt/l4d2/steam/steamcmd.sh \
+    +force_install_dir /opt/l4d2/installation \
+    +login anonymous \
+    +@sSteamCmdForcePlatformType windows \
+    +app_update 222860 validate \
+    +quit
+steam /opt/l4d2/steam/steamcmd.sh \
+    +force_install_dir /opt/l4d2/installation \
+    +login anonymous \
+    +@sSteamCmdForcePlatformType linux \
+    +app_update 222860 validate \
+    +quit
+
+# -- OVERLAYS -- #
+
+for overlay_path in /opt/l4d2/scripts/overlays/*; do
+    overlay=$(basename "$overlay_path")
+    steam mkdir -p /opt/l4d2/overlays/$overlay
+    bash -xeuo pipefail "$overlay_path"
+    test -f /opt/l4d2/overlays/$overlay/left4dead2/cfg/server.cfg && \
+        steam cp /opt/l4d2/overlays/$overlay/left4dead2/cfg/server.cfg /opt/l4d2/overlays/$overlay/left4dead2/cfg/server_$overlay.cfg
+done
+
+# -- SERVERS -- #
+
+#steam rm -rf /opt/l4d2/servers
+steam mkdir -p /opt/l4d2/servers

bundles/left4dead2/files/start

@@ -0,0 +1,75 @@
+#!/bin/bash
+
+set -xeuo pipefail
+
+name=""
+port=""
+configfile=""
+overlays=""
+arguments=""
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    -n|--name)
+      name="$2"; shift 2
+      ;;
+    -p|--port)
+      port="$2"; shift 2
+      ;;
+    -c|--config)
+      configfile="$2"; shift 2
+      ;;
+    -o|--overlay)
+      overlays="/opt/l4d2/overlays/$2:$overlays"; shift 2
+      ;;
+    --)
+      shift
+      arguments+="$@"
+      break
+      ;;
+    *)
+      echo "ERROR: unknown argument $1"; exit 1
+      ;;
+  esac
+done
+
+[[ -n "${name}" ]] || { echo "ERROR: -n/--name missing"; exit 1; }
+[[ -n "${port}" ]] || { echo "ERROR: -p/--port missing"; exit 1; }
+
+# -- HELPER FUNCTIONS -- #
+
+function steam() {
+    # für systemd, damit es den prozess beenden kann
+    setpriv --reuid=steam --regid=steam --init-groups "$@"
+    export HOME=/opt/l4d2/steam
+}
+
+# -- TIDY UP -- #
+
+mountpoint -q "/opt/l4d2/servers/$name/merged" && umount "/opt/l4d2/servers/$name/merged"
+steam rm -rf "/opt/l4d2/servers/$name"
+
+# -- CREATE DIRECTORIES -- #
+
+steam mkdir -p \
+  "/opt/l4d2/servers/$name" \
+  "/opt/l4d2/servers/$name/work" \
+  "/opt/l4d2/servers/$name/upper" \
+  "/opt/l4d2/servers/$name/merged"
+
+# -- MOUNT OVERLAYFS -- #
+
+mount -t overlay overlay \
+  -o "lowerdir=$overlays/opt/l4d2/installation,upperdir=/opt/l4d2/servers/$name/upper,workdir=/opt/l4d2/servers/$name/work" \
+  "/opt/l4d2/servers/$name/merged"
+
+# -- REPLACE SERVER.CFG -- #
+
+if [[ -n "$configfile" ]]; then
+  cp "$configfile" "/opt/l4d2/servers/$name/merged/left4dead2/cfg/server.cfg"
+  chown steam:steam "/opt/l4d2/servers/$name/merged/left4dead2/cfg/server.cfg"
+fi
+
+# -- RUN L4D2 -- #
+
+steam "/opt/l4d2/servers/$name/merged/srcds_run" -norestart -pidfile "/opt/l4d2/servers/$name/pid" -game left4dead2 -ip 0.0.0.0 -port "$port" +hostname "Crone_$name" +map c1m1_hotel $arguments

bundles/left4dead2/files/stop

@@ -0,0 +1,19 @@
+#!/bin/bash
+
+set -xeuo pipefail
+
+name=""
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    -n|--name)
+      name="$2"; shift 2
+      ;;
+    *)
+      echo "ERROR: unknown argument $1"; exit 1
+      ;;
+  esac
+done
+
+mountpoint -q "/opt/l4d2/servers/$name/merged" && umount "/opt/l4d2/servers/$name/merged"
+steam rm -rf "/opt/l4d2/servers/$name"

bundles/left4dead2/items.py

@@ -1,122 +1,105 @@
-assert node.has_bundle('steam') and node.has_bundle('steam-workshop-download')
+users = {
+    'steam': {
+        'home': '/opt/l4d2/steam',
+        'shell': '/bin/bash',
+    },
+}
 
 directories = {
-    '/opt/steam/left4dead2-servers': {
+    '/opt/l4d2': {
-        'owner': 'steam',
+        'owner': 'steam', 'group': 'steam',
-        'group': 'steam',
+    },
-        'mode': '0755',
+    '/opt/l4d2/steam': {
+        'owner': 'steam', 'group': 'steam',
+    },
+    '/opt/l4d2/configs': {
+        'owner': 'steam', 'group': 'steam',
         'purge': True,
     },
-    # Current zfs doesnt support zfs upperdir. The support was added in October 2022. Move upperdir - unused anyway -
+    '/opt/l4d2/scripts': {
-    # to another dir. Also move workdir alongside it, as it has to be on same fs.
+        'owner': 'steam', 'group': 'steam',
-    '/opt/steam-zfs-overlay-workarounds': {
+    },
-        'owner': 'steam',
+    '/opt/l4d2/scripts/overlays': {
-        'group': 'steam',
+        'owner': 'steam', 'group': 'steam',
-        'mode': '0755',
         'purge': True,
     },
 }
 
-# /opt/steam/steam/.steam/sdk32/steamclient.so: cannot open shared object file: No such file or directory
+files = {
-symlinks = {
+    '/opt/l4d2/setup': {
-    '/opt/steam/steam/.steam/sdk32': {
+        'mode': '755',
-        'target': '/opt/steam/steam/linux32',
+        'triggers': {
-        'owner': 'steam',
+            'svc_systemd:left4dead2-initialize.service:restart',
-        'group': 'steam',
+        },
-    }
+    },
+    '/opt/l4d2/start': {
+        'mode': '755',
+        'triggers': {
+            f'svc_systemd:left4dead2-{server_name}.service:restart'
+                for server_name in node.metadata.get('left4dead2/servers').keys()
+        },
+    },
+    '/opt/l4d2/stop': {
+        'mode': '755',
+        'triggers': {
+            f'svc_systemd:left4dead2-{server_name}.service:restart'
+                for server_name in node.metadata.get('left4dead2/servers').keys()
+        },
+    },
+    '/opt/l4d2/scripts/helpers': {
+        'source': 'scripts/helpers',
+        'mode': '755',
+        'triggers': {
+            'svc_systemd:left4dead2-initialize.service:restart',
+        },
+    },
 }
 
-#
+for overlay in node.metadata.get('left4dead2/overlays'):
-# SERVERS
+    files[f'/opt/l4d2/scripts/overlays/{overlay}'] = {
-#
+        'source': f'scripts/overlays/{overlay}',
-
+        'mode': '755',
-for name, config in node.metadata.get('left4dead2/servers').items():
+        'triggers': {
-
+            'svc_systemd:left4dead2-initialize.service:restart',
-    #overlay
+        },
-    directories[f'/opt/steam/left4dead2-servers/{name}'] = {
-        'owner': 'steam',
-        'group': 'steam',
-    }
-    directories[f'/opt/steam-zfs-overlay-workarounds/{name}/upper'] = {
-        'owner': 'steam',
-        'group': 'steam',
-    }
-    directories[f'/opt/steam-zfs-overlay-workarounds/{name}/workdir'] = {
-        'owner': 'steam',
-        'group': 'steam',
     }
 
-    # conf
+svc_systemd = {
-    files[f'/opt/steam/left4dead2-servers/{name}/left4dead2/cfg/server.cfg'] = {
+    'left4dead2-initialize.service': {
-        'content_type': 'mako',
+        'enabled': True,
+        'running': None,
+        'needs': {
+            'tag:left4dead2-packages',
+            'file:/opt/l4d2/setup',
+            'file:/usr/local/lib/systemd/system/left4dead2-initialize.service',
+        },
+    },
+}
+
+for server_name, config in node.metadata.get('left4dead2/servers').items():
+    files[f'/opt/l4d2/configs/{server_name}.cfg'] = {
         'source': 'server.cfg',
+        'content_type': 'mako',
         'context': {
-            'name': name,
+            'server_name': server_name,
-            'steamgroups': node.metadata.get('left4dead2/steamgroups'),
+            'rcon_password': repo.vault.decrypt('encrypt$gAAAAABpAdZhxwJ47I1AXotuZmBvyZP1ecVTt9IXFkLI28JiVS74LKs9QdgIBz-FC-iXtIHHh_GVGxxKQZprn4UrXZcvZ57kCKxfHBs3cE2JiGnbWE8_mfs=').value,
-            'rcon_password': config['rcon_password'],
+            'config': config.get('config', []),
         },
         'owner': 'steam',
-        'group': 'steam',
+        'mode': '644',
-        'triggers': [
+        'triggers': {
-            f'svc_systemd:left4dead2-{name}.service:restart',
+            f'svc_systemd:left4dead2-{server_name}.service:restart',
-        ],
+        },
     }
 
-    # service
+    svc_systemd[f'left4dead2-{server_name}.service'] = {
-    svc_systemd[f'left4dead2-{name}.service'] = {
+        'enabled': True,
-        'needs': [
+        'running': True,
-            f'file:/opt/steam/left4dead2-servers/{name}/left4dead2/cfg/server.cfg',
+        'tags': {
-            f'file:/usr/local/lib/systemd/system/left4dead2-{name}.service',
+            'left4dead2-servers',
-        ],
+        },
-    }
+        'needs': {
-
+            'svc_systemd:left4dead2-initialize.service',
-    #
+            f'file:/usr/local/lib/systemd/system/left4dead2-{server_name}.service',
-    # ADDONS
+        },
-    #
-
-    # base
-    files[f'/opt/steam/left4dead2-servers/{name}/left4dead2/addons/readme.txt'] = {
-        'content_type': 'any',
-        'owner': 'steam',
-        'group': 'steam',
-    }
-    directories[f'/opt/steam/left4dead2-servers/{name}/left4dead2/addons'] = {
-        'owner': 'steam',
-        'group': 'steam',
-        'purge': True,
-        'triggers': [
-            f'svc_systemd:left4dead2-{name}.service:restart',
-        ],
-    }
-    for id in [
-        *config.get('workshop', []),
-        *node.metadata.get('left4dead2/workshop'),
-    ]:
-        files[f'/opt/steam/left4dead2-servers/{name}/left4dead2/addons/{id}.vpk'] = {
-            'content_type': 'any',
-            'owner': 'steam',
-            'group': 'steam',
-            'triggers': [
-                f'svc_systemd:left4dead2-{name}.service:restart',
-            ],
-        }
-
-    # admin system
-
-    directories[f'/opt/steam/left4dead2-servers/{name}/left4dead2/ems/admin system'] = {
-        'owner': 'steam',
-        'group': 'steam',
-        'mode': '0755',
-        'triggers': [
-            f'svc_systemd:left4dead2-{name}.service:restart',
-        ],
-    }
-    files[f'/opt/steam/left4dead2-servers/{name}/left4dead2/ems/admin system/admins.txt'] = {
-        'owner': 'steam',
-        'group': 'steam',
-        'mode': '0755',
-        'content': '\n'.join(sorted(node.metadata.get('left4dead2/admins'))),
-        'triggers': [
-            f'svc_systemd:left4dead2-{name}.service:restart',
-        ],
     }

bundles/left4dead2/metadata.py

@@ -1,110 +1,112 @@
-assert node.has_bundle('steam')
+from re import match
+from os import path, listdir
 
-from shlex import quote
 
 defaults = {
-    'steam': {
+    'apt': {
-        'games': {
+        'packages': {
-            'left4dead2': 222860,
+            'libc6_i386': { # installs libc6:i386
+                'tags': {'left4dead2-packages'},
+            },
+            'lib32z1': {
+                'tags': {'left4dead2-packages'},
+            },
+            'unzip': {
+                'tags': {'left4dead2-packages'},
+            },
+            'p7zip-full': { # l4d2center_maps_sync.sh
+                'tags': {'left4dead2-packages'},
+            },
         },
     },
     'left4dead2': {
-        'servers': {},
+        'overlays': set(listdir(path.join(repo.path, 'bundles/left4dead2/files/scripts/overlays'))),
-        'admins': set(),
+        'servers': {
-        'workshop': set(),
+            # 'port': 27017,
+            # 'overlays': ['competitive_rework'],
+            # 'arguments': ['-tickrate 60'],
+            # 'config': [
+            #     'exec server_original.cfg',
+            #     'sm_forcematch zonemod',
+            # ],
+        },
+    },
+    'nftables': {
+        'input': {
+            'udp dport { 27005, 27020 } accept',
+        },
+    },
+    'systemd': {
+        'units': {
+            'left4dead2-initialize.service': {
+                'Unit': {
+                    'Description': 'initialize left4dead2',
+                    'After': 'network-online.target',
+                },
+                'Service': {
+                    'Type': 'oneshot',
+                    'RemainAfterExit': 'yes',
+                    'ExecStart': '/opt/l4d2/setup',
+                    'StandardOutput': 'journal',
+                    'StandardError': 'journal',
+                },
+                'Install': {
+                    'WantedBy': {'multi-user.target'},
+                },
+            },
+        },
     },
 }
 
 
 @metadata_reactor.provides(
-    'left4dead2/servers',
-)
-def rconn_password(metadata):
-    # only works from localhost!
-    return {
-        'left4dead2': {
-            'servers': {
-                server: {
-                    'rcon_password': repo.vault.password_for(f'{node.name} left4dead2 {server} rcon', length=24),
-                }
-                    for server in metadata.get('left4dead2/servers')
-            },
-        },
-    }
-
-
-@metadata_reactor.provides(
-    'steam-workshop-download',
     'systemd/units',
 )
 def server_units(metadata):
     units = {}
-    workshop = {}
 
     for name, config in metadata.get('left4dead2/servers').items():
-        # mount overlay
+        assert match(r'^[A-z0-9-_-]+$', name)
-        mountpoint = f'/opt/steam/left4dead2-servers/{name}'
+        assert 27000 <= config["port"] <= 27100
-        mount_unit_name = mountpoint[1:].replace('-', '\\x2d').replace('/', '-') + '.mount'
+        for overlay in config.get('overlays', []):
-        units[mount_unit_name] = {
+            assert overlay in metadata.get('left4dead2/overlays'), f"unknown overlay {overlay}, known: {metadata.get('left4dead2/overlays')}"
-            'Unit': {
-                'Description': f"Mount left4dead2 server {name} overlay",
-                'Conflicts': {'umount.target'},
-                'Before': {'umount.target'},
-            },
-            'Mount': {
-                'What': 'overlay',
-                'Where': mountpoint,
-                'Type': 'overlay',
-                'Options': ','.join([
-                    'auto',
-                    'lowerdir=/opt/steam/left4dead2',
-                    f'upperdir=/opt/steam-zfs-overlay-workarounds/{name}/upper',
-                    f'workdir=/opt/steam-zfs-overlay-workarounds/{name}/workdir',
-                ]),
-            },
-            'Install': {
-                'RequiredBy': {
-                    f'left4dead2-{name}.service',
-                },
-            },
-        }
 
-        # individual workshop
+        cmd = f'/opt/l4d2/start -n {name} -p {config["port"]}'
-        workshop_ids = config.get('workshop', set()) | metadata.get('left4dead2/workshop', set())
+
-        if  workshop_ids:
+        if 'config' in config:
-            workshop[f'left4dead2-{name}'] = {
+            cmd += f' -c /opt/l4d2/configs/{name}.cfg'
-                'ids': workshop_ids,
+
-                'path': f'/opt/steam/left4dead2-servers/{name}/left4dead2/addons',
+        for overlay in config.get('overlays', []):
-                'user': 'steam',
+            cmd += f' -o {overlay}'
-                'requires': {
+
-                    mount_unit_name,
+        if 'arguments' in config:
-                },
+            cmd += ' -- ' + ' '.join(config['arguments'])
-                'required_by': {
-                    f'left4dead2-{name}.service',
-                },
-            }
 
-        # left4dead2 server unit
         units[f'left4dead2-{name}.service'] = {
             'Unit': {
                 'Description': f'left4dead2 server {name}',
-                'After': {'steam-update.service'},
+                'After': {'left4dead2-initialize.service'},
-                'Requires': {'steam-update.service'},
+                'Requires': {'left4dead2-initialize.service'},
             },
             'Service': {
-                'User': 'steam',
+                'Type': 'simple',
-                'Group': 'steam',
+                'ExecStart': cmd,
-                'WorkingDirectory': f'/opt/steam/left4dead2-servers/{name}',
+                'ExecStopPost': f'/opt/l4d2/stop -n {name}',
-                'ExecStart': f'/opt/steam/left4dead2-servers/{name}/srcds_run -port {config["port"]} +exec server.cfg',
                 'Restart': 'on-failure',
+                'Nice': -10,
+                'CPUWeight': 200,
+                'IOSchedulingClass': 'best-effort',
+                'IOSchedulingPriority': 0,
             },
             'Install': {
                 'WantedBy': {'multi-user.target'},
             },
+            'triggers': {
+                f'svc_systemd:left4dead2-{name}.service:restart',
+            },
         }
 
     return {
-        'steam-workshop-download': workshop,
         'systemd': {
             'units': units,
         },
@@ -114,14 +116,13 @@ def server_units(metadata):
 @metadata_reactor.provides(
     'nftables/input',
 )
-def firewall(metadata):
+def nftables(metadata):
-    ports = set(str(server['port']) for server in metadata.get('left4dead2/servers').values())
+    ports = sorted(str(config["port"]) for config in metadata.get('left4dead2/servers').values())
 
     return {
         'nftables': {
             'input': {
-                f"tcp dport {{ {', '.join(sorted(ports))} }} accept",
+                f'ip protocol {{ tcp, udp }} th dport {{ {", ".join(ports)} }} accept'
-                f"udp dport {{ {', '.join(sorted(ports))} }} accept",
             },
         },
     }

bundles/left4me/README.md

@@ -0,0 +1,167 @@
+# left4me
+
+L4D2 game-server management platform: a Flask web UI on gunicorn that
+provisions per-instance srcds servers via templated systemd units, with
+kernel-overlayfs layering for shared installations + per-overlay maps,
+and uid-based DSCP/priority marking on the egress path so CAKE on the
+external interface prioritizes srcds UDP over bulk traffic.
+
+## Metadata
+
+```python
+'metadata': {
+    'left4me': {
+        'domain': 'whatever.tld',  # required — the only per-node knob
+        # Everything below is optional and has a sensible default in the
+        # bundle. Override per-node only if the default is wrong:
+        # 'git_url': 'git@git.sublimity.de:cronekorkn/left4me',
+        # 'git_branch': 'master',
+        # 'gunicorn_workers': 1,
+        # 'gunicorn_threads': 32,
+        # 'job_worker_threads': 4,
+        # 'port_range_start': 27015,
+        # 'port_range_end': 27115,
+        # secret_key is auto-derived per node
+        # (repo.vault.random_bytes_as_base64_for f'{node.name} left4me secret_key').
+    },
+},
+```
+
+The bundle's `derived_from_domain` reactor reads `left4me/domain` and
+emits the corresponding `nginx/vhosts`, `letsencrypt/domains`,
+`monitoring/services/left4me-web` (HTTPS health check), and the game-
+port `nftables/input` accept rules. Backup paths
+(`/var/lib/left4me`, `/etc/left4me`) are set-merged into `backup/paths`
+from defaults. None of these need to be declared per-node.
+
+## What this bundle does
+
+The bundle delivers to `ovh.left4me` a mix of:
+
+### Target-side symlinks into the left4me checkout
+
+After `git_deploy:/opt/left4me/src` (root-owned — left4me cannot rewrite
+its own deployment artifacts at runtime), ckn-bw creates symlinks from
+canonical on-host paths into the checkout:
+
+| On-host path | Source in checkout |
+|---|---|
+| `/etc/sudoers.d/left4me` | `deploy/files/etc/sudoers.d/left4me` |
+| `/etc/sysctl.d/99-left4me.conf` | `deploy/files/etc/sysctl.d/99-left4me.conf` |
+| `/etc/systemd/system/left4me-web.service.d/10-hardening.conf` | `deploy/files/etc/systemd/system/left4me-web.service.d/10-hardening.conf` |
+| `/etc/systemd/system/left4me-server@.service.d/10-hardening.conf` | `deploy/files/etc/systemd/system/left4me-server@.service.d/10-hardening.conf` |
+| `/usr/local/libexec/left4me/{left4me-overlay,left4me-systemctl,left4me-journalctl,left4me-script-sandbox}` | `deploy/scripts/libexec/*` |
+| `/usr/local/sbin/left4me` | `deploy/scripts/sbin/left4me` |
+
+The hardening drop-ins and sudoers are the application's own security
+knowledge — they live in the left4me repo and are version-controlled there.
+The privileged helpers are also application code. The symlink pattern
+lets bw manage placement without duplicating content.
+
+Design rationale:
+`left4me/docs/superpowers/specs/2026-05-15-deployment-responsibility-design.md`.
+
+### Reactor-emitted units (per-host shape)
+
+Via `systemd/units` metadata in `metadata.py` (consumed by `bundles/systemd/`):
+
+- `left4me-web.service` — gunicorn on `127.0.0.1:8000`; worker/thread
+  counts from `web.env.mako`. TLS terminates upstream.
+- `left4me-server@.service` — per-instance srcds template; `SocketBindAllow=`
+  ranges from metadata.
+- `l4d2-game.slice` / `l4d2-build.slice` — cgroup slices with per-host
+  `AllowedCPUs=` from `left4me/system_cpus`.
+- `system.slice.d/99-left4me-cpuset.conf` + `user.slice.d/99-left4me-cpuset.conf`
+  — host CPU-set drop-ins, same source.
+
+### bw `files{}` — templated env files
+
+- `host.env.mako` → `/etc/left4me/host.env`
+- `web.env.mako` → `/etc/left4me/web.env`
+- `sandbox-resolv.conf` → `/etc/left4me/sandbox-resolv.conf`
+
+### Action chains — deploy lifecycle
+
+- `git_deploy` → `uv_sync` (`uv sync --frozen` against the workspace's
+  committed `uv.lock`; hatchling PEP 660 editable, doesn't touch source)
+  → `alembic_upgrade` → `seed_overlays` + web restart.
+- One-shot bootstrap: `install_uv` downloads a pinned `uv` binary
+  (SHA256-verified) into `/usr/local/bin` because `uv` isn't in Trixie's
+  apt archive. `unless`-gated, so it's a no-op once the version pin is
+  installed; re-runs only when the constant is bumped.
+- Idempotent gates: `chmod-sudoers` (0440 root:root), `chmod-scripts` (0755 root:root).
+- Post-git-deploy reloads: `systemctl daemon-reload`, `sysctl --system`.
+- Post-apply self-test: `verify-hardening-dropins` (asserts the drop-ins are
+  loaded by the live units before declaring apply done).
+
+### System user
+
+`left4me` (uid/gid 980, home `/var/lib/left4me`, mode 0755) — the same uid
+hosts the web app, gameservers, and the script-overlay sandbox unit (which
+drops privileges via systemd-run with a fully hardened transient service).
+Runtime mutable state lives under `/var/lib/left4me/`; `/opt/left4me/`
+stays as a root-owned deploy-artifact root.
+
+### nftables / nginx / monitoring
+
+- Contributes uid-based DSCP/priority marks for srcds UDP egress to
+  `nftables/output` (via `defaults`).
+- `derived_from_domain` reactor emits the corresponding `nginx/vhosts`,
+  `letsencrypt/domains`, and `monitoring/services/left4me-web` (HTTPS
+  health check).
+
+## Gotchas
+
+- **Requires `bundles/nftables` and `bundles/systemd` on the node.** The
+  bundle asserts membership at `bw test` time. On Debian-13 these ride
+  in via the `debian-13` group, so attaching the bundle to a Debian-13
+  node is enough.
+- **`left4me-web.service` does not have `NoNewPrivileges=true`.** This is
+  intentional — workers `sudo` the privileged helpers; `NoNewPrivileges`
+  would block setuid escalation. Per-instance `server@.service` units
+  *do* have it.
+- **CAKE shaping is configured separately**, via
+  `network/<iface>/cake` on the node (consumed by `bundles/network/`),
+  not by this bundle.
+- **First-run admin user is manual.** After `bw apply`, ssh to the host and
+  bootstrap the admin via the `left4me` wrapper (it sources the env files,
+  drops to the `left4me` user, and runs the flask CLI):
+  `sudo left4me create-user <username> --admin` (prompts for password via
+  the flask CLI, or set `LEFT4ME_ADMIN_PASSWORD` first). The bundle
+  deliberately doesn't seed an admin to keep credentials out of the
+  metadata pipeline. The same `left4me` wrapper accepts any other flask
+  subcommand: `sudo left4me seed-script-overlays <dir>`,
+  `sudo left4me routes`, `sudo left4me shell`, etc.
+- **CPU isolation is managed by this bundle**, driven by one required
+  per-node knob: `left4me/system_cpus` — a set of int CPU ids that
+  pins `system.slice` / `user.slice` / `l4d2-build.slice`. The
+  complement (`set(range(vm/threads)) - system_cpus`) pins
+  `l4d2-game.slice`. On HT hosts, list both SMT siblings of every
+  physical core you want to reserve for system, otherwise games end
+  up sharing L1/L2 with system. Find pairings via
+  `/sys/devices/system/cpu/cpu<n>/topology/thread_siblings_list`. On
+  the prod node (`ovh.left4me`, 4 physical / 8 threads, pairings
+  (0,4) (1,5) (2,6) (3,7)) the node sets `'system_cpus': {0, 4}` to
+  reserve physical core 0 entirely. `l4d2-game.slice` and
+  `l4d2-build.slice` carry `AllowedCPUs=` inline on their unit
+  definitions; `system.slice` and `user.slice` get drop-ins registered
+  under `systemd/units` with the `'<parent>.d/<basename>.conf'` key
+  convention (same shape nginx and autologin use), landing at
+  `/usr/local/lib/systemd/system/<slice>.d/99-left4me-cpuset.conf`.
+  The reactor raises if `system_cpus` includes CPUs outside
+  `[0, vm/threads)` or leaves no cores for games.
+- **Kernel feature requirement:** kernel-overlayfs (`CONFIG_OVERLAY_FS`).
+  Standard on debian-13.
+- **Game ports** open by the web app on demand in the range 27015-27115
+  (UDP+TCP). Add corresponding accept rules to `nftables/input` per
+  node if the host's policy is default-drop on input.
+- **Pinned UIDs/GIDs (980/981).** Chosen for deterministic ownership
+  across rebuilds and backup restores. If you add another bundle that
+  pins UIDs in this repo, make sure it doesn't collide.
+
+## Slice support requires `bundles/systemd` ≥ commit cc1c6a5
+
+This bundle's `l4d2-game.slice` and `l4d2-build.slice` units rely on
+`bundles/systemd/items.py` accepting the `.slice` extension. Older
+revisions raised `Exception(f'unknown type slice')` at apply time.
+The repo-wide `bw test` will catch this if it regresses.

bundles/left4me/files/etc/left4me/host.env.mako

@@ -0,0 +1,6 @@
+# Managed by ckn-bw bundles/left4me. Local edits will be reverted.
+# Deployment units use fixed /var/lib/left4me paths; regenerate units if this changes.
+LEFT4ME_ROOT=/var/lib/left4me
+# l4d2host invokes steamcmd by absolute path — bypasses PATH lookup so the
+# script's `cd "$(dirname "$0")"` resolves next to the real install dir.
+LEFT4ME_STEAMCMD=/var/lib/left4me/steam/steamcmd.sh

bundles/left4me/files/etc/left4me/sandbox-resolv.conf

@@ -0,0 +1,6 @@
+# Sandbox-only resolver config — bind-mounted into script-overlay sandboxes
+# at /etc/resolv.conf. The host's resolver (often a private/LAN DNS server)
+# is unreachable from inside the sandbox because IPAddressDeny= blocks
+# egress to RFC1918 / loopback. Public resolvers keep DNS working.
+nameserver 1.1.1.1
+nameserver 8.8.8.8

bundles/left4me/files/etc/left4me/web.env.mako

@@ -0,0 +1,14 @@
+# Managed by ckn-bw bundles/left4me. Local edits will be reverted.
+DATABASE_URL=sqlite:////var/lib/left4me/left4me.db
+SECRET_KEY=${node.metadata.get('left4me/secret_key')}
+JOB_WORKER_THREADS=${node.metadata.get('left4me/job_worker_threads')}
+SESSION_COOKIE_SECURE=true
+LEFT4ME_PORT_RANGE_START=${node.metadata.get('left4me/port_range_start')}
+LEFT4ME_PORT_RANGE_END=${node.metadata.get('left4me/port_range_end')}
+STEAM_WEB_API_KEY=${node.metadata.get('left4me/steam_web_api_key')}
+# Log listener destination — MUST be non-loopback because Source silently
+# drops logaddress destinations in 127.0.0.0/8. Derived from the node's
+# external IPv4; kernel routes same-host traffic via lo internally but the
+# destination IP in the packet header must not literally be 127.x.
+LOG_LISTENER_ADDR=${node.metadata.get('network/external/ipv4').split('/')[0]}:28000
+LOG_LISTENER_BIND=0.0.0.0:28000

bundles/left4me/items.py

@@ -0,0 +1,439 @@
+# Items for the left4me bundle.
+# Systemd units come from metadata via bundles/systemd/ — there are no
+# .service or .slice files in this bundle's files/ tree. Cpuset drop-ins
+# for system.slice / user.slice are likewise emitted via systemd/units
+# in metadata.py (key: '<parent>.d/<basename>.conf').
+
+directories = {
+    '/opt/left4me': {
+        # Deploy-artifact root. Only /opt/left4me/src lives here; runtime
+        # state (.venv, steamcmd) lives under /var/lib/left4me/. Root-owned
+        # so left4me cannot drop new files alongside src/ (e.g. an attacker
+        # with web-compromise can't plant a 'scripts.d/' loaded by future
+        # deploy logic).
+        'owner': 'root',
+        'group': 'root',
+        'mode': '0755',
+    },
+    '/opt/left4me/src': {
+        # Source checkout. Root-owned because the production install model
+        # is non-editable: pip_install copies the source to a left4me-owned
+        # tempdir before building, so the source tree on disk is never
+        # mutated at runtime and left4me only needs read access (which
+        # world-readable bits provide). Keeps left4me from being able to
+        # rewrite its own future hardening drop-ins / unit files under
+        # /opt/left4me/src/deploy/ (target-side symlink model in the
+        # deployment-responsibility reshape).
+        'owner': 'root',
+        'group': 'root',
+    },
+    '/etc/left4me': {
+        'owner': 'root',
+        'group': 'root',
+        'mode': '0755',
+    },
+    '/var/lib/left4me': {
+        # left4me's home dir — useradd creates with 0700; loosen to 0755 so
+        # the systemd-imposed FS view for transient script-sandbox units
+        # (running as left4me with TemporaryFileSystem=/var/lib + selective
+        # binds) can traverse on its way to the overlay bind targets.
+        'owner': 'left4me',
+        'group': 'left4me',
+        'mode': '0755',
+    },
+    '/var/lib/left4me/installation':   {'owner': 'left4me', 'group': 'left4me'},
+    '/var/lib/left4me/overlays':       {'owner': 'left4me', 'group': 'left4me'},
+    '/var/lib/left4me/instances':      {'owner': 'left4me', 'group': 'left4me'},
+    '/var/lib/left4me/runtime':        {'owner': 'left4me', 'group': 'left4me'},
+    '/var/lib/left4me/workshop_cache': {'owner': 'left4me', 'group': 'left4me'},
+    '/var/lib/left4me/tmp':            {'owner': 'left4me', 'group': 'left4me'},
+    '/var/lib/left4me/steam':          {'owner': 'left4me', 'group': 'left4me'},
+    # Note: the venv (/var/lib/left4me/.venv) is created by the
+    # left4me_create_venv action; declaring it here too would race with
+    # `python -m venv` which expects to create the directory itself.
+    '/usr/local/libexec/left4me': {
+        'owner': 'root',
+        'group': 'root',
+        'mode': '0755',
+    },
+    '/etc/systemd/system/left4me-web.service.d': {
+        'owner': 'root', 'group': 'root', 'mode': '0755',
+    },
+    '/etc/systemd/system/left4me-server@.service.d': {
+        'owner': 'root', 'group': 'root', 'mode': '0755',
+    },
+}
+
+groups = {
+    'left4me': {'gid': 980},
+}
+
+users = {
+    'left4me': {
+        'uid': 980,
+        'gid': 980,
+        'home': '/var/lib/left4me',
+        'shell': '/usr/sbin/nologin',
+    },
+}
+# UID/GID pinned in the system-package range (100-999, per Debian
+# policy) so file ownership is deterministic across rebuilds and
+# backup restores. 980 is unused elsewhere in this repo.
+# (981 — formerly l4d2-sandbox — was collapsed into 980 on 2026-05-15;
+# see left4me/docs/superpowers/plans/2026-05-15-uid-collapse.md.)
+
+# Privileged helpers are delivered via target-side symlinks (see the
+# `symlinks` dict below) pointing into the left4me checkout at
+# `/opt/left4me/src/deploy/scripts/{libexec,sbin}/`. No verbatim copy
+# in this bundle's files/ tree. Sudoers (further below) lists the
+# specific paths that left4me may invoke as root NOPASSWD.
+
+files = {
+    '/etc/left4me/sandbox-resolv.conf': {
+        'source': 'etc/left4me/sandbox-resolv.conf',
+        'mode': '0644',
+        'owner': 'root',
+        'group': 'root',
+    },
+    '/etc/left4me/host.env': {
+        'source': 'etc/left4me/host.env.mako',
+        'content_type': 'mako',
+        'mode': '0640',
+        'owner': 'root',
+        # group=left4me so the alembic + seed-overlays actions (which run as
+        # `sudo -u left4me sh -c '. /etc/left4me/host.env'`) can source it.
+        # Same pattern as web.env below.
+        'group': 'left4me',
+        'needs': [
+            'group:left4me',
+        ],
+    },
+    '/etc/left4me/web.env': {
+        'source': 'etc/left4me/web.env.mako',
+        'content_type': 'mako',
+        'mode': '0640',
+        'owner': 'root',
+        'group': 'left4me',
+        'needs': [
+            'group:left4me',
+        ],
+    },
+}
+
+symlinks = {
+    '/etc/sysctl.d/99-left4me.conf': {
+        'target': '/opt/left4me/src/deploy/files/etc/sysctl.d/99-left4me.conf',
+        'owner': 'root',
+        'group': 'root',
+        'needs': [
+            'git_deploy:/opt/left4me/src',
+        ],
+        'triggers': [
+            'action:left4me_sysctl_reload',
+        ],
+    },
+    '/etc/systemd/system/left4me-web.service.d/10-hardening.conf': {
+        'target': '/opt/left4me/src/deploy/files/etc/systemd/system/left4me-web.service.d/10-hardening.conf',
+        'owner': 'root', 'group': 'root',
+        'needs': [
+            'directory:/etc/systemd/system/left4me-web.service.d',
+            'git_deploy:/opt/left4me/src',
+        ],
+        'triggers': [
+            'action:left4me_daemon_reload',
+        ],
+    },
+    '/etc/systemd/system/left4me-server@.service.d/10-hardening.conf': {
+        'target': '/opt/left4me/src/deploy/files/etc/systemd/system/left4me-server@.service.d/10-hardening.conf',
+        'owner': 'root', 'group': 'root',
+        'needs': [
+            'directory:/etc/systemd/system/left4me-server@.service.d',
+            'git_deploy:/opt/left4me/src',
+        ],
+        'triggers': [
+            'action:left4me_daemon_reload',
+        ],
+    },
+    '/etc/sudoers.d/left4me': {
+        'target': '/opt/left4me/src/deploy/files/etc/sudoers.d/left4me',
+        'owner': 'root', 'group': 'root',
+        'needs': [
+            'action:left4me_chmod_sudoers',
+            'git_deploy:/opt/left4me/src',
+        ],
+        # sudo follows symlinks; with the target file at root:root 0440
+        # in a root-owned source tree, sudo accepts it. No daemon-reload
+        # equivalent — sudo re-reads /etc/sudoers.d/ on each invocation.
+    },
+}
+
+# Helper script source paths (in left4me's checkout) → deployed-form paths.
+# Each gets a symlink item merged into the symlinks dict above.
+_LEFT4ME_LIBEXEC_SCRIPTS = (
+    'left4me-overlay',
+    'left4me-systemctl',
+    'left4me-journalctl',
+    'left4me-script-sandbox',
+)
+_LEFT4ME_SBIN_SCRIPTS = (
+    'left4me',
+)
+
+for _script in _LEFT4ME_LIBEXEC_SCRIPTS:
+    symlinks[f'/usr/local/libexec/left4me/{_script}'] = {
+        'target': f'/opt/left4me/src/deploy/scripts/libexec/{_script}',
+        'owner': 'root', 'group': 'root',
+        'needs': [
+            'directory:/usr/local/libexec/left4me',
+            'action:left4me_chmod_scripts',
+            'git_deploy:/opt/left4me/src',
+        ],
+    }
+
+for _script in _LEFT4ME_SBIN_SCRIPTS:
+    symlinks[f'/usr/local/sbin/{_script}'] = {
+        'target': f'/opt/left4me/src/deploy/scripts/sbin/{_script}',
+        'owner': 'root', 'group': 'root',
+        'needs': [
+            'action:left4me_chmod_scripts',
+            'git_deploy:/opt/left4me/src',
+        ],
+    }
+
+actions = {
+    'left4me_sysctl_reload': {
+        'command': 'sysctl --system >/dev/null',
+        'triggered': True,
+    },
+    'left4me_daemon_reload': {
+        'command': 'systemctl daemon-reload',
+        'triggered': True,
+        'cascade_skip': False,
+    },
+    'left4me_verify_hardening_dropins_loaded': {
+        # Post-apply self-test: confirm systemd actually picked up the
+        # hardening drop-ins we shipped via symlink. Catches the failure
+        # mode where the symlink lands but daemon-reload didn't take or
+        # someone manually unlinked the drop-in. For the gameserver template
+        # we query an imaginary instance — systemd resolves drop-in paths
+        # for `name@instance.service` against the template (`name@.service.d/`),
+        # so the instance need not exist or ever have run.
+        'command': (
+            'systemctl show left4me-server@verify.service -p DropInPaths --value '
+            '| tr " " "\\n" '
+            '| grep -qx /etc/systemd/system/left4me-server@.service.d/10-hardening.conf '
+            '&& '
+            'systemctl show left4me-web.service -p DropInPaths --value '
+            '| tr " " "\\n" '
+            '| grep -qx /etc/systemd/system/left4me-web.service.d/10-hardening.conf'
+        ),
+        'cascade_skip': False,
+        'needs': [
+            'action:left4me_daemon_reload',
+            'symlink:/etc/systemd/system/left4me-web.service.d/10-hardening.conf',
+            'symlink:/etc/systemd/system/left4me-server@.service.d/10-hardening.conf',
+        ],
+    },
+    'left4me_chmod_sudoers': {
+        # sudo refuses sudoers.d entries that aren't 0440 (or 0400) root:root.
+        # git_deploy extracts as root with the in-repo file mode; this action
+        # is belt-and-braces in case the repo mode drifts. Idempotent via
+        # the `unless` gate.
+        'command': 'chmod 0440 /opt/left4me/src/deploy/files/etc/sudoers.d/left4me',
+        'unless': 'test "$(stat -c %a /opt/left4me/src/deploy/files/etc/sudoers.d/left4me)" = "440"',
+        'cascade_skip': False,
+        'needs': [
+            'git_deploy:/opt/left4me/src',
+        ],
+    },
+    'left4me_dpkg_add_i386_arch': {
+        # steamcmd is 32-bit and pulls libc6:i386 + lib32z1 from the i386 arch.
+        # apt-get update is part of this action because newly-added foreign
+        # archs need a fresh package list before any :i386 package resolves.
+        'command': 'dpkg --add-architecture i386 && apt-get update',
+        'unless': 'dpkg --print-foreign-architectures | grep -qx i386',
+        'cascade_skip': False,
+    },
+    'left4me_install_steamcmd': {
+        # Steam's tarball is rolling with no published checksum, so we can't
+        # use download: (which requires a hash). Guard with a presence check
+        # on steamcmd.sh — steamcmd self-updates at runtime, so chasing the
+        # tarball version from bw isn't useful.
+        'command': (
+            'sudo -u left4me sh -c "'
+            'cd /var/lib/left4me/steam && '
+            'curl -fsSL https://media.steampowered.com/installer/steamcmd_linux.tar.gz | '
+            'tar -xz'
+            '"'
+        ),
+        'unless': 'test -x /var/lib/left4me/steam/steamcmd.sh',
+        'cascade_skip': False,
+        'needs': [
+            'directory:/var/lib/left4me/steam',
+            'pkg_apt:curl',
+            'pkg_apt:libc6_i386',  # bw pkg_apt convention: _ → :
+            'pkg_apt:lib32z1',
+            'user:left4me',
+        ],
+    },
+}
+
+# steamcmd is invoked by absolute path (LEFT4ME_STEAMCMD in host.env),
+# not via PATH lookup — see l4d2host/cli.py:install. We don't need to put
+# anything in /usr/local/bin for it.
+
+git_deploy = {
+    '/opt/left4me/src': {
+        'repo': node.metadata.get('left4me/git_url'),
+        'rev': node.metadata.get('left4me/git_branch'),
+        'triggers': [
+            # Re-sync the workspace whenever the checkout changes. uv reads
+            # the committed uv.lock at /opt/left4me/src and installs both
+            # workspace members (l4d2host, l4d2web) editable into
+            # /var/lib/left4me/.venv. Hatchling's PEP 660 editable install
+            # doesn't write to the source tree, so /opt/left4me/src stays
+            # root-owned and untouched. uv_sync cascades into
+            # alembic_upgrade → seed_overlays → web restart.
+            'action:left4me_uv_sync',
+            # alembic upgrade head is idempotent — keeping it as a direct
+            # trigger off git_deploy is belt-and-braces in case the
+            # uv_sync cascade is ever short-circuited.
+            'action:left4me_alembic_upgrade',
+            # Reload systemd unit definitions whenever the checkout changes;
+            # handles updates to hardening drop-in content without requiring
+            # a symlink change.
+            'action:left4me_daemon_reload',
+        ],
+    },
+}
+
+actions['left4me_chmod_scripts'] = {
+    # sudo invokes the helpers by absolute path under /usr/local/...;
+    # those resolve to the checkout via the symlinks above. The target
+    # files must be executable (mode 0755). git_deploy extracts with
+    # the in-repo file modes; this action is belt-and-braces in case
+    # any helper's repo mode regresses to 0644.
+    'command': (
+        'chmod 0755 '
+        '/opt/left4me/src/deploy/scripts/libexec/* '
+        '/opt/left4me/src/deploy/scripts/sbin/*'
+    ),
+    'unless': (
+        '! find /opt/left4me/src/deploy/scripts -type f \\! -perm 755 -print -quit 2>/dev/null | grep -q .'
+    ),
+    'cascade_skip': False,
+    'needs': [
+        'git_deploy:/opt/left4me/src',
+    ],
+}
+
+actions['left4me_install_uv'] = {
+    # uv is not in Debian Trixie's apt archive (only experimental/sid).
+    # Pin to a specific release; download the tarball + its SHA256
+    # sibling from astral-sh/uv releases, verify, install to
+    # /usr/local/bin. Idempotent via `unless` — only re-runs when the
+    # pinned version changes (bump the constant in two places below).
+    # Pattern matches left4me_install_steamcmd (curl+tar) elsewhere in
+    # this bundle. Bump cadence: as needed; both dev (brew uv) and
+    # prod should track the same minor.
+    'command': """set -e
+tmpdir=$(mktemp -d); trap "rm -rf $tmpdir" EXIT
+base=https://github.com/astral-sh/uv/releases/download/0.11.8
+tar=uv-x86_64-unknown-linux-gnu.tar.gz
+curl -fsSL -o $tmpdir/$tar        $base/$tar
+curl -fsSL -o $tmpdir/$tar.sha256 $base/$tar.sha256
+(cd $tmpdir && sha256sum -c $tar.sha256)
+tar -xzf $tmpdir/$tar -C $tmpdir --strip-components=1
+install -m 0755 $tmpdir/uv  /usr/local/bin/uv
+install -m 0755 $tmpdir/uvx /usr/local/bin/uvx
+""",
+    'unless': '/usr/local/bin/uv --version 2>/dev/null | grep -qx "uv 0.11.8"',
+    'cascade_skip': False,
+    'needs': [
+        'pkg_apt:curl',
+    ],
+    # No triggers — install_uv is a one-shot bootstrap. uv_sync needs
+    # it (via `needs`), so the dependency runs install_uv first on a
+    # clean host. After that, this action is a no-op on every apply
+    # unless the version pin changes.
+}
+
+actions['left4me_uv_sync'] = {
+    # The whole "install/refresh the workspace" deploy step, in one
+    # action. uv reads /opt/left4me/src/uv.lock + the workspace's
+    # pyproject.toml and installs both members (l4d2host, l4d2web)
+    # editable into /var/lib/left4me/.venv. Hatchling's PEP 660
+    # editable install drops a .pth pointing at the source tree — no
+    # writes to source, so the root-owned /opt/left4me/src stays clean.
+    #
+    # UV_PROJECT_ENVIRONMENT redirects uv's default venv path
+    # (<project>/.venv) to our writable runtime location. HOME is set
+    # explicitly so uv's cache lands in /var/lib/left4me/.cache/uv
+    # instead of the inherited sudo HOME (which can be unwritable for
+    # the left4me user). cd /var/lib/left4me ensures uv's project-config
+    # walk-up doesn't trip over an unreadable parent (e.g., /root or
+    # /home/ckn). --frozen requires uv.lock to be present and
+    # consistent with pyproject.toml — refuses to silently update the
+    # lockfile during deploy.
+    'command': (
+        'sudo -u left4me sh -c "'
+        'cd /var/lib/left4me && '
+        'env HOME=/var/lib/left4me '
+        'UV_PROJECT_ENVIRONMENT=/var/lib/left4me/.venv '
+        '/usr/local/bin/uv sync --frozen --project /opt/left4me/src'
+        '"'
+    ),
+    'triggered': True,
+    'cascade_skip': False,
+    'needs': [
+        'git_deploy:/opt/left4me/src',
+        'action:left4me_install_uv',
+        'directory:/var/lib/left4me',
+        'user:left4me',
+    ],
+    'triggers': [
+        'action:left4me_alembic_upgrade',
+    ],
+}
+
+actions['left4me_alembic_upgrade'] = {
+    # Mirrors deploy-test-server.sh:239-242. Runs as left4me with both env
+    # files sourced; JOB_WORKER_ENABLED=false so a stray worker doesn't race
+    # with the migration.
+    'command': (
+        'sudo -u left4me sh -c "'
+        'cd /opt/left4me/src/l4d2web && '
+        'set -a && . /etc/left4me/host.env && . /etc/left4me/web.env && set +a && '
+        'env JOB_WORKER_ENABLED=false '
+        '/var/lib/left4me/.venv/bin/alembic -c /opt/left4me/src/l4d2web/alembic.ini upgrade head'
+        '"'
+    ),
+    'triggered': True,
+    'cascade_skip': False,
+    'needs': [
+        'action:left4me_uv_sync',
+        'file:/etc/left4me/host.env',
+        'file:/etc/left4me/web.env',
+    ],
+    'triggers': [
+        'action:left4me_seed_overlays',
+        'svc_systemd:left4me-web.service:restart',
+    ],
+}
+
+actions['left4me_seed_overlays'] = {
+    # Idempotent: refreshes script bodies in place; existing overlay rows keep their ids.
+    'command': (
+        'sudo -u left4me sh -c "'
+        'set -a && . /etc/left4me/host.env && . /etc/left4me/web.env && set +a && '
+        'env JOB_WORKER_ENABLED=false '
+        '/var/lib/left4me/.venv/bin/flask --app l4d2web.app:create_app '
+        'seed-script-overlays /opt/left4me/src/examples/script-overlays'
+        '"'
+    ),
+    'triggered': True,
+    'cascade_skip': False,
+    'needs': [
+        'action:left4me_alembic_upgrade',
+    ],
+}

bundles/left4me/metadata.py

@@ -0,0 +1,305 @@
+assert node.has_bundle('nftables')
+assert node.has_bundle('systemd')
+assert node.has_bundle('systemd-timers')
+
+
+defaults = {
+    'left4me': {
+        # Application-wide defaults; node only overrides if it really needs to.
+        'git_url': 'https://git.sublimity.de/cronekorkn/left4me.git',
+        'git_branch': 'master',
+        'secret_key': repo.vault.random_bytes_as_base64_for(f'{node.name} left4me secret_key', length=32).value,
+        'gunicorn_workers': 1,
+        'gunicorn_threads': 32,
+        'job_worker_threads': 4,
+        # Steam Web API key for the live-state panel's GetPlayerSummaries
+        # lookups (persona names + avatars). Empty default — nodes override
+        # in their own metadata with the actual key. If left empty in prod,
+        # the live-state panel still works but falls back to RCON in-game
+        # names and placeholder avatars.
+        'steam_web_api_key': '',
+        # Whole 27000-block: covers Steam's defaults (27015 game, 27005
+        # client/RCON) plus headroom for ad-hoc ports without further
+        # nftables changes. Mirrored into LEFT4ME_PORT_RANGE_{START,END}
+        # by web.env.mako and into the nftables input rule by the
+        # nftables_input reactor below.
+        'port_range_start': 27000,
+        'port_range_end': 27999,
+    },
+    'apt': {
+        'packages': {
+            'p7zip-full': {},
+            'nftables': {},
+            'iproute2': {},
+            'curl': {},
+            'ca-certificates': {},
+            'python3': {},
+            'python3-dev': {},
+            # steamcmd is a 32-bit ELF; needs i386 multiarch + these libs.
+            # `_` → `:` is bundlewrap's pkg_apt convention for multiarch
+            # names (see pkg_apt.py:48).
+            'libc6_i386': {  # installs libc6:i386
+                'needs': ['action:left4me_dpkg_add_i386_arch'],
+            },
+            'lib32z1': {
+                'needs': ['action:left4me_dpkg_add_i386_arch'],
+            },
+        },
+    },
+    'nftables': {
+        # Match deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft.
+        # Mark srcds UDP egress (uid left4me) with DSCP EF + skb priority 6
+        # so CAKE classifies it into the priority tin.
+        'output': {
+            'meta skuid "left4me" meta l4proto udp ip dscp set ef meta priority set 0006:0000',
+            'meta skuid "left4me" meta l4proto udp ip6 dscp set ef meta priority set 0006:0000',
+        },
+    },
+    'systemd': {
+        'services': {
+            'left4me-web.service': {
+                'enabled': True,
+                'running': True,
+                'needs': [
+                    'action:left4me_alembic_upgrade',
+                    'file:/etc/left4me/host.env',
+                    'file:/etc/left4me/web.env',
+                ],
+            },
+            # Note: left4me-server@.service is a TEMPLATE — instances are
+            # started on-demand by the web app via the left4me-systemctl
+            # helper. Don't enable/start it from here.
+            # The slices are installed (file present) but don't need
+            # enable/start — they're activated implicitly when a unit
+            # uses Slice=.
+        },
+    },
+    'backup': {
+        # Application-owned paths. Set-merged with backup group / node-level paths.
+        'paths': {
+            '/var/lib/left4me',
+            '/etc/left4me',
+        },
+    },
+    'systemd-timers': {
+        # Daily re-fetch of Steam Workshop metadata + .vpk downloads for any
+        # item whose author published an update. The CLI just inserts a
+        # `refresh_workshop_items` job; the web worker picks it up next.
+        # Idempotent — a re-fire while a refresh is already queued/running
+        # is a no-op (see l4d2web/cli.py:workshop_refresh).
+        'left4me-workshop-refresh': {
+            'command': '/var/lib/left4me/.venv/bin/flask --app l4d2web.app:create_app workshop-refresh',
+            'when': '*-*-* 04:00:00',
+            'persistent': True,
+            'user': 'left4me',
+            'working_dir': '/opt/left4me/src',
+            'environment_files': (
+                '/etc/left4me/host.env',
+                '/etc/left4me/web.env',
+            ),
+            'after': {
+                'network-online.target',
+                'left4me-web.service',
+            },
+        },
+    },
+}
+
+
+
+@metadata_reactor.provides(
+    'nginx/vhosts',
+)
+def nginx_vhosts(metadata):
+    # letsencrypt/domains and monitoring/services for the vhost are auto-
+    # populated by bundles/nginx/metadata.py. We just declare check_path:
+    # '/health' so the auto-check hits the Flask health endpoint, not '/'.
+    domain = metadata.get('left4me/domain')
+    return {
+        'nginx': {
+            'vhosts': {
+                domain: {
+                    'content': 'nginx/proxy_pass.conf',
+                    'context': {
+                        'target': 'http://127.0.0.1:8000',
+                    },
+                    'check_path': '/health',
+                },
+            },
+        },
+    }
+
+
+@metadata_reactor.provides(
+    'nftables/input',
+)
+def nftables_input(metadata):
+    port_start = metadata.get('left4me/port_range_start')
+    port_end = metadata.get('left4me/port_range_end')
+    return {
+        'nftables': {
+            'input': {
+                # Players connect via UDP. TCP on the same port range is RCON
+                # — only the local web app should reach it. Loopback bypasses
+                # the input chain (iifname lo accept in bundles/nftables/...),
+                # so 127.0.0.1 RCON works without an explicit TCP accept here.
+                # External TCP on these ports stays blocked by the default
+                # input-chain drop.
+                f'udp dport {port_start}-{port_end} accept',
+            },
+        },
+    }
+
+
+@metadata_reactor.provides(
+    'systemd/units',
+)
+def systemd_units(metadata):
+    workers = metadata.get('left4me/gunicorn_workers')
+    threads = metadata.get('left4me/gunicorn_threads')
+
+    # cgroup-v2 cpuset. `system_cpus` (set of int CPU ids, declared per
+    # node) pins system/user/build; the complement pins l4d2-game. On HT
+    # hosts, list both siblings of a physical core so games don't share
+    # L1/L2 with system work — pairings via
+    # /sys/devices/system/cpu/cpu<n>/topology/thread_siblings_list.
+    vm_threads = metadata.get('vm/threads', metadata.get('vm/cores'))
+    all_cpus = set(range(vm_threads))
+    system_cpus = metadata.get('left4me/system_cpus')
+    if not system_cpus <= all_cpus:
+        raise Exception(
+            f'left4me/system_cpus={sorted(system_cpus)} on {vm_threads}-thread host '
+            f'includes CPUs outside [0, {vm_threads})'
+        )
+    game_cpus = all_cpus - system_cpus
+    if not game_cpus:
+        raise Exception(
+            f'left4me/system_cpus={sorted(system_cpus)} on {vm_threads}-thread host '
+            f'leaves no cores for games'
+        )
+    system_cpus_string = ','.join(str(t) for t in sorted(system_cpus))
+    game_cpus_string = ','.join(str(t) for t in sorted(game_cpus))
+
+    # Drop-in for upstream system.slice / user.slice (units we don't own).
+    # Same '<parent>.d/<basename>.conf' convention as nginx and autologin.
+    cpuset_dropin = {'Slice': {'AllowedCPUs': system_cpus_string}}
+
+    return {
+        'systemd': {
+            'units': {
+                'left4me-web.service': {
+                    'Unit': {
+                        'Description': 'left4me web application',
+                        'After': 'network-online.target',
+                        'Wants': 'network-online.target',
+                    },
+                    'Service': {
+                        'Type': 'simple',
+                        'User': 'left4me',
+                        'Group': 'left4me',
+                        'WorkingDirectory': '/opt/left4me/src',
+                        'Environment': {
+                            'HOME=/var/lib/left4me',
+                            'PATH=/var/lib/left4me/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
+                        },
+                        'EnvironmentFile': (
+                            '/etc/left4me/host.env',
+                            '/etc/left4me/web.env',
+                        ),
+                        'ExecStart': (
+                            '/var/lib/left4me/.venv/bin/gunicorn '
+                            f'--workers {workers} --threads {threads} '
+                            "--bind 127.0.0.1:8000 'l4d2web.app:create_app()'"
+                        ),
+                        'Restart': 'on-failure',
+                        'RestartSec': '3',
+
+                        # Web app writes broadly under /var/lib/left4me. Kept inline
+                        # because it's web-specific (server@ uses BindPaths to bind
+                        # only its instance dir).
+                        'ReadWritePaths': '/var/lib/left4me',
+
+                        # Hardening profile delivered via
+                        # /etc/systemd/system/left4me-web.service.d/10-hardening.conf
+                        # (target-side symlink into left4me/deploy/files/, owned by left4me).
+                    },
+                    'Install': {
+                        'WantedBy': {'multi-user.target'},
+                    },
+                },
+                'left4me-server@.service': {
+                    'Unit': {
+                        'Description': 'left4me server instance %i',
+                        'After': 'network-online.target',
+                        'Wants': 'network-online.target',
+                        'StartLimitBurst': '5',
+                        'StartLimitIntervalSec': '60s',
+                    },
+                    'Service': {
+                        'Type': 'simple',
+                        'User': 'left4me',
+                        'Group': 'left4me',
+                        'EnvironmentFile': (
+                            '/etc/left4me/host.env',
+                            '/var/lib/left4me/instances/%i/instance.env',
+                        ),
+                        'WorkingDirectory': '-/var/lib/left4me/runtime/%i/merged/left4dead2',
+                        'ExecStartPre': '+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- /usr/local/libexec/left4me/left4me-overlay mount %i',
+                        # +ip 0.0.0.0 binds RCON (TCP) to all interfaces incl. loopback;
+                        # without this, Source auto-selects the primary IP and the web
+                        # app's 127.0.0.1 RCON connect gets ECONNREFUSED. External TCP
+                        # on the game port range is firewall-blocked in nftables_input.
+                        'ExecStart': '/var/lib/left4me/runtime/%i/merged/srcds_run -game left4dead2 +ip 0.0.0.0 +hostport ${L4D2_PORT} $L4D2_ARGS',
+                        'ExecStopPost': '+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- /usr/local/libexec/left4me/left4me-overlay umount %i',
+                        'Restart': 'on-failure',
+                        'RestartSec': '5',
+
+                        # Resource control (baseline from prior performance work).
+                        'Slice': 'l4d2-game.slice',
+                        'Nice': '-5',
+                        'IOSchedulingClass': 'best-effort',
+                        'IOSchedulingPriority': '4',
+                        'OOMScoreAdjust': '-200',
+                        'MemoryHigh': '1.5G',
+                        'MemoryMax': '2G',
+                        'TasksMax': '256',
+                        'LimitNOFILE': '65536',
+                        'KillSignal': 'SIGINT',
+                        'TimeoutStopSec': '15s',
+                        'LogRateLimitIntervalSec': '0',
+
+                        # Hardening profile delivered via
+                        # /etc/systemd/system/left4me-server@.service.d/10-hardening.conf
+                        # (target-side symlink into left4me/deploy/files/, owned by left4me).
+                    },
+                    'Install': {
+                        'WantedBy': {'multi-user.target'},
+                    },
+                },
+                'l4d2-game.slice': {
+                    'Unit': {
+                        'Description': 'left4me game-server slice',
+                        'Before': 'slices.target',
+                    },
+                    'Slice': {
+                        'CPUWeight': '1000',
+                        'IOWeight': '1000',
+                        'AllowedCPUs': game_cpus_string,
+                    },
+                },
+                'l4d2-build.slice': {
+                    'Unit': {
+                        'Description': 'left4me script-sandbox build slice',
+                        'Before': 'slices.target',
+                    },
+                    'Slice': {
+                        'CPUWeight': '10',
+                        'IOWeight': '10',
+                        'AllowedCPUs': system_cpus_string,
+                    },
+                },
+                'system.slice.d/99-left4me-cpuset.conf': cpuset_dropin,
+                'user.slice.d/99-left4me-cpuset.conf':   cpuset_dropin,
+            },
+        },
+    }

bundles/letsencrypt/README.md

@@ -1,9 +1,60 @@
-https://github.com/dehydrated-io/dehydrated/wiki/example-dns-01-nsupdate-script
+# letsencrypt
+
+Issues and renews Let's Encrypt certs via [dehydrated][upstream] with
+DNS-01 against the in-house bind-acme server.
+
+[upstream]: https://github.com/dehydrated-io/dehydrated/wiki/example-dns-01-nsupdate-script
+
+## First-apply behaviour
+
+Immediately after `bw apply <node>`, nginx serves a **self-signed
+cert** for each declared domain — generated by
+`/etc/dehydrated/letsencrypt-ensure-some-certificate` so nginx has
+something to start with. The real Let's Encrypt cert arrives at most
+24h later when the systemd timer fires
+(`/usr/bin/dehydrated --cron --accept-terms --challenge dns-01`). To
+shortcut the wait:
+
+```sh
+ssh <node> 'sudo /usr/bin/dehydrated --cron --accept-terms --challenge dns-01'
+ssh <node> 'sudo systemctl reload nginx'
+```
+
+## DNS-01 prerequisites
+
+`hook.sh` does `nsupdate` against the bind-acme server (referenced
+by `letsencrypt/acme_node`). For the challenge to succeed:
+
+1. The acme node must be in the same metadata graph (so
+   `bw metadata <node> -k letsencrypt/acme_node` resolves).
+2. **All NS servers** for the validated domain must serve the
+   `_acme-challenge.<domain>` CNAME — Let's Encrypt validates from
+   primary AND secondary geographic regions; both authoritative
+   servers must agree. If a secondary NS is also a bw-managed node,
+   `bw apply` it after adding the domain (see e.g. `ovh.secondary`).
+3. The bind-acme node's TSIG key must be reachable. `hook.sh` is
+   rendered with the bind-acme server's `network/internal/ipv4` —
+   for clients outside that LAN, the route must exist (typically via
+   wireguard `s2s` peer membership).
+
+## Negative-cache penalty
+
+If the first DNS-01 attempt fails (e.g. zone not yet applied to the
+secondary NS), Let's Encrypt's resolvers cache NXDOMAIN for the SOA's
+negative TTL (often 900s = 15 min). Subsequent attempts during that
+window also fail and refresh the cache. Combined with LE's rate limit
+of **5 failed authorisations per domain per hour**, recovery requires
+you to **stop retrying** for ~15 minutes after fixing the DNS, then
+make at most one attempt.
+
+## nsupdate sample
+
+For interactive testing of the bind-acme TSIG path:
 
 ```sh
 printf "server 127.0.0.1
 zone acme.resolver.name.
-update add _acme-challenge.ckn.li.acme.resolver.name. 600 IN TXT "hello"
+update add _acme-challenge.ckn.li.acme.resolver.name. 600 IN TXT \"hello\"
 send
-" | nsupdate -y hmac-sha512:acme:Y9BHl85l352BGZDXa/vg90hh2+5PYe4oJxpkq/oQvIODDkW8bAyQSFr0gKQQxjyIOyYlTjf0MGcdWFv46G/3Rg==
+" | nsupdate -y hmac-sha512:acme:XXXXXX
 ```

bundles/letsencrypt/files/hook.sh

@@ -31,6 +31,12 @@ deploy_cert() {
   % for domain, conf in sorted(domains.items()):
 <%   if not conf: continue %>\
     ${domain})
+      % if conf.get('scp', None):
+      scp "$KEYFILE" "${conf['scp']}/${conf.get('privkey_name', 'privkey.pem')}"
+      scp "$CERTFILE" "${conf['scp']}/${conf.get('cert_name', 'cert.pem')}"
+      scp "$FULLCHAINFILE" "${conf['scp']}/${conf.get('fullchain_name', 'fullchain.pem')}"
+      scp "$CHAINFILE" "${conf['scp']}/${conf.get('chain_name', 'chain.pem')}"
+      % endif
       % if conf.get('location', None):
       cat "$KEYFILE" > "${conf['location']}/${conf.get('privkey_name', 'privkey.pem')}"
       cat "$CERTFILE" > "${conf['location']}/${conf.get('cert_name', 'cert.pem')}"

bundles/letsencrypt/items.py

@@ -42,7 +42,7 @@ files = {
 }
 
 actions['letsencrypt_update_certificates'] = {
-    'command': 'dehydrated --cron --accept-terms --challenge dns-01',
+    'command': 'systemctl start letsencrypt.service',
     'triggered': True,
     'skip': delegated,
     'needs': {

bundles/letsencrypt/metadata.py

@@ -2,7 +2,7 @@ defaults = {
     'apt': {
         'packages': {
             'dehydrated': {},
-            'dnsutils': {},
+            'bind9-dnsutils': {},
         },
     },
     'letsencrypt': {

bundles/linux/items.py

@@ -12,9 +12,8 @@ def generate_sysctl_key_value_pairs_from_json(json_data, parents=[]):
 
 key_value_pairs = generate_sysctl_key_value_pairs_from_json(node.metadata.get('sysctl'))
 
-
 files= {
-    '/etc/sysctl.conf': {
+    '/etc/sysctl.d/managed.conf': {
         'content': '\n'.join(
             sorted(
                 f"{'.'.join(path)}={value}"
@@ -25,6 +24,9 @@ files= {
             'svc_systemd:systemd-sysctl.service:restart',
         ],
     },
+    '/etc/modules-load.d/managed.conf': {
+        'content': '\n'.join(sorted(node.metadata.get('modules-load'))),
+    }
 }
 
 svc_systemd = {

bundles/linux/metadata.py

@@ -1,3 +1,6 @@
 defaults = {
-    'sysctl': {},
+    'sysctl': {
+        'net.ipv4.icmp_ratelimit': '100',
+    },
+    'modules-load': set(),
 }

bundles/locale/metadata.py

@@ -7,12 +7,7 @@ defaults = {
     'locale': {
         'default': ('en_US.UTF-8', 'UTF-8'),
         'installed': {
-            ('de_AT.UTF-8', 'UTF-8'),
-            ('de_CH.UTF-8', 'UTF-8'),
             ('de_DE.UTF-8', 'UTF-8'),
-            ('de_LU.UTF-8', 'UTF-8'),
-            ('en_CA.UTF-8', 'UTF-8'),
-            ('en_GB.UTF-8', 'UTF-8'),
             ('en_US.UTF-8', 'UTF-8'),
         },
     },

bundles/macbook/files/bundlewrap

@@ -2,5 +2,5 @@
 
 cd "$OLDPWD"
 
-export BW_ITEM_WORKERS=$(expr "$(nproc)" '*' 12 '/' 10)
+export BW_ITEM_WORKERS=$(expr "$(sysctl -n hw.logicalcpu)" '*' 12 '/' 10)
 export BW_NODE_WORKERS=$(expr 320 '/' "$BW_ITEM_WORKERS")

bundles/macbook/files/gnu

@@ -2,7 +2,5 @@
 
 cd "$OLDPWD"
 
-GNU_PATH="$HOME/.local/gnu_bin"
+PATH_add "/opt/homebrew/opt/gnu-sed/libexec/gnubin"
-mkdir -p "$GNU_PATH"
+PATH_add "/opt/homebrew/opt/grep/libexec/gnubin"
-test -f "$GNU_PATH/sed" || ln -s "$(which gsed)" "$GNU_PATH/sed"
-PATH_add "$GNU_PATH"

bundles/macbook/files/pyenv

@@ -2,6 +2,8 @@
 
 cd "$OLDPWD"
 
+pyenv install --skip-existing
+
 if test -f .venv/bin/python && test "$(realpath .venv/bin/python)" != "$(realpath "$(pyenv which python)")"
 then
   echo "rebuilding venv für new python version"

bundles/mailman/README.md

@@ -0,0 +1,26 @@
+# Mailman
+
+- django admin udner /admin
+
+## Testmail
+
+`echo export REST_API_PASS=$(bw metadata mseibert.mailman -k mailman/api_password | jq -r .mailman.api_password)`
+```sh
+curl -s -o /dev/null \
+     -w "Status: %{http_code}\nTime: %{time_total}s\n" \
+     -u restadmin:$REST_API_PASS \
+     -H "Content-Type: application/json" \
+     -X POST http://localhost:8001/3.1/queues/in \
+     -d "{
+       \"list_id\": \"testlist-2.mailman.ckn.li\",
+       \"text\": \"From: i@ckn.li\nTo: testlist-2@mailman.ckn.li\nSubject: Curl Test $(date '+%Y-%m-%d %H:%M:%S')\n\nThis message was sent at $(date '+%Y-%m-%d %H:%M:%S').\"
+     }"
+```
+
+## Log locations
+
+`tail -f /var/log/mailman3/*.log`
+
+`journalctl -f | grep postfix/`
+
+`mailq | head -20`

bundles/mailman/files/mailman-hyperkitty.cfg

@@ -0,0 +1,22 @@
+# This is the mailman extension configuration file to enable HyperKitty as an
+# archiver. Remember to add the following lines in the mailman.cfg file:
+#
+# [archiver.hyperkitty]
+# class: mailman_hyperkitty.Archiver
+# enable: yes
+# configuration: /etc/mailman3/mailman-hyperkitty.cfg
+#
+
+[general]
+
+# This is your HyperKitty installation, preferably on the localhost. This
+# address will be used by Mailman to forward incoming emails to HyperKitty
+# for archiving. It does not need to be publicly available, in fact it's
+# better if it is not.
+# However, if your Mailman installation is accessed via HTTPS, the URL needs
+# to match your SSL certificate (e.g. https://lists.example.com/hyperkitty).
+base_url: http://${hostname}/mailman3/hyperkitty/
+
+# The shared api_key, must be identical except for quoting to the value of
+# MAILMAN_ARCHIVER_KEY in HyperKitty's settings.
+api_key: ${archiver_key}

bundles/mailman/files/mailman-web.py

@@ -0,0 +1,190 @@
+ACCOUNT_EMAIL_VERIFICATION='none'
+
+# This file is imported by the Mailman Suite. It is used to override
+# the default settings from /usr/share/mailman3-web/settings.py.
+
+# SECURITY WARNING: keep the secret key used in production secret!
+SECRET_KEY = '${secret_key}'
+
+ADMINS = (
+     ('Mailman Suite Admin', 'root@localhost'),
+)
+
+# Hosts/domain names that are valid for this site; required if DEBUG is False
+# See https://docs.djangoproject.com/en/1.8/ref/settings/#allowed-hosts
+# Set to '*' per default in the Deian package to allow all hostnames. Mailman3
+# is meant to run behind a webserver reverse proxy anyway.
+ALLOWED_HOSTS = [
+    '${hostname}',
+]
+
+# Mailman API credentials
+MAILMAN_REST_API_URL = 'http://localhost:8001'
+MAILMAN_REST_API_USER = 'restadmin'
+MAILMAN_REST_API_PASS = '${api_password}'
+MAILMAN_ARCHIVER_KEY = '${archiver_key}'
+MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1')
+
+# Application definition
+
+INSTALLED_APPS = (
+    'hyperkitty',
+    'postorius',
+    'django_mailman3',
+    # Uncomment the next line to enable the admin:
+    'django.contrib.admin',
+    # Uncomment the next line to enable admin documentation:
+    # 'django.contrib.admindocs',
+    'django.contrib.auth',
+    'django.contrib.contenttypes',
+    'django.contrib.sessions',
+    'django.contrib.sites',
+    'django.contrib.messages',
+    'django.contrib.staticfiles',
+    'rest_framework',
+    'django_gravatar',
+    'compressor',
+    'haystack',
+    'django_extensions',
+    'django_q',
+    'allauth',
+    'allauth.account',
+    'allauth.socialaccount',
+    'django_mailman3.lib.auth.fedora',
+    #'allauth.socialaccount.providers.openid',
+    #'allauth.socialaccount.providers.github',
+    #'allauth.socialaccount.providers.gitlab',
+    #'allauth.socialaccount.providers.google',
+    #'allauth.socialaccount.providers.facebook',
+    #'allauth.socialaccount.providers.twitter',
+    #'allauth.socialaccount.providers.stackexchange',
+)
+
+
+# Database
+# https://docs.djangoproject.com/en/1.8/ref/settings/#databases
+
+DATABASES = {
+    'default': {
+        # Use 'sqlite3', 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
+        #'ENGINE': 'django.db.backends.sqlite3',
+        'ENGINE': 'django.db.backends.postgresql_psycopg2',
+        #'ENGINE': 'django.db.backends.mysql',
+        # DB name or path to database file if using sqlite3.
+        #'NAME': '/var/lib/mailman3/web/mailman3web.db',
+        'NAME': 'mailman',
+        # The following settings are not used with sqlite3:
+        'USER': 'mailman',
+        'PASSWORD': '${db_password}',
+        # HOST: empty for localhost through domain sockets or '127.0.0.1' for
+        # localhost through TCP.
+        'HOST': '127.0.0.1',
+        # PORT: set to empty string for default.
+        'PORT': '5432',
+        # OPTIONS: Extra parameters to use when connecting to the database.
+        'OPTIONS': {
+            # Set sql_mode to 'STRICT_TRANS_TABLES' for MySQL. See
+            # https://docs.djangoproject.com/en/1.11/ref/
+            #     databases/#setting-sql-mode
+            #'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
+        },
+    }
+}
+
+
+# If you're behind a proxy, use the X-Forwarded-Host header
+# See https://docs.djangoproject.com/en/1.8/ref/settings/#use-x-forwarded-host
+USE_X_FORWARDED_HOST = True
+
+# And if your proxy does your SSL encoding for you, set SECURE_PROXY_SSL_HEADER
+# https://docs.djangoproject.com/en/1.8/ref/settings/#secure-proxy-ssl-header
+# SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
+# SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_SCHEME', 'https')
+
+# Other security settings
+# SECURE_SSL_REDIRECT = True
+# If you set SECURE_SSL_REDIRECT to True, make sure the SECURE_REDIRECT_EXEMPT
+# contains at least this line:
+# SECURE_REDIRECT_EXEMPT = [
+#     "archives/api/mailman/.*",  # Request from Mailman.
+#     ]
+# SESSION_COOKIE_SECURE = True
+# SECURE_CONTENT_TYPE_NOSNIFF = True
+# SECURE_BROWSER_XSS_FILTER = True
+# CSRF_COOKIE_SECURE = True
+# CSRF_COOKIE_HTTPONLY = True
+# X_FRAME_OPTIONS = 'DENY'
+
+
+# Internationalization
+# https://docs.djangoproject.com/en/1.8/topics/i18n/
+
+LANGUAGE_CODE = 'en-us'
+
+TIME_ZONE = 'UTC'
+
+USE_I18N = True
+USE_L10N = True
+USE_TZ = True
+
+
+# Set default domain for email addresses.
+EMAILNAME = 'localhost.local'
+
+# If you enable internal authentication, this is the address that the emails
+# will appear to be coming from. Make sure you set a valid domain name,
+# otherwise the emails may get rejected.
+# https://docs.djangoproject.com/en/1.8/ref/settings/#default-from-email
+# DEFAULT_FROM_EMAIL = "mailing-lists@you-domain.org"
+DEFAULT_FROM_EMAIL = 'postorius@{}'.format(EMAILNAME)
+
+# If you enable email reporting for error messages, this is where those emails
+# will appear to be coming from. Make sure you set a valid domain name,
+# otherwise the emails may get rejected.
+# https://docs.djangoproject.com/en/1.8/ref/settings/#std:setting-SERVER_EMAIL
+# SERVER_EMAIL = 'root@your-domain.org'
+SERVER_EMAIL = 'root@{}'.format(EMAILNAME)
+
+
+# Django Allauth
+ACCOUNT_DEFAULT_HTTP_PROTOCOL = "https"
+
+
+#
+# Social auth
+#
+SOCIALACCOUNT_PROVIDERS = {
+    #'openid': {
+    #    'SERVERS': [
+    #        dict(id='yahoo',
+    #             name='Yahoo',
+    #             openid_url='http://me.yahoo.com'),
+    #    ],
+    #},
+    #'google': {
+    #    'SCOPE': ['profile', 'email'],
+    #    'AUTH_PARAMS': {'access_type': 'online'},
+    #},
+    #'facebook': {
+    #   'METHOD': 'oauth2',
+    #   'SCOPE': ['email'],
+    #   'FIELDS': [
+    #       'email',
+    #       'name',
+    #       'first_name',
+    #       'last_name',
+    #       'locale',
+    #       'timezone',
+    #       ],
+    #   'VERSION': 'v2.4',
+    #},
+}
+
+# On a production setup, setting COMPRESS_OFFLINE to True will bring a
+# significant performance improvement, as CSS files will not need to be
+# recompiled on each requests. It means running an additional "compress"
+# management command after each code upgrade.
+# http://django-compressor.readthedocs.io/en/latest/usage/#offline-compression
+COMPRESS_OFFLINE = True
+
+POSTORIUS_TEMPLATE_BASE_URL = 'http://${hostname}/mailman3/'

bundles/mailman/files/mailman.cfg

@@ -0,0 +1,271 @@
+# Copyright (C) 2008-2017 by the Free Software Foundation, Inc.
+#
+# This file is part of GNU Mailman.
+#
+# GNU Mailman is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option)
+# any later version.
+#
+# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+# more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# GNU Mailman.  If not, see <http://www.gnu.org/licenses/>.
+
+# This file contains the Debian configuration for mailman.  It uses ini-style
+# formats under the lazr.config regime to define all system configuration
+# options.  See <https://launchpad.net/lazr.config> for details.
+
+
+[mailman]
+# This address is the "site owner" address.  Certain messages which must be
+# delivered to a human, but which can't be delivered to a list owner (e.g. a
+# bounce from a list owner), will be sent to this address.  It should point to
+# a human.
+site_owner: ${site_owner_email}
+
+# This is the local-part of an email address used in the From field whenever a
+# message comes from some entity to which there is no natural reply recipient.
+# Mailman will append '@' and the host name of the list involved.  This
+# address must not bounce and it must not point to a Mailman process.
+noreply_address: noreply
+
+# The default language for this server.
+default_language: de
+
+# Membership tests for posting purposes are usually performed by looking at a
+# set of headers, passing the test if any of their values match a member of
+# the list.  Headers are checked in the order given in this variable.  The
+# value From_ means to use the envelope sender.  Field names are case
+# insensitive.  This is a space separate list of headers.
+sender_headers: from from_ reply-to sender
+
+# Mail command processor will ignore mail command lines after designated max.
+email_commands_max_lines: 10
+
+# Default length of time a pending request is live before it is evicted from
+# the pending database.
+pending_request_life: 3d
+
+# How long should files be saved before they are evicted from the cache?
+cache_life: 7d
+
+# A callable to run with no arguments early in the initialization process.
+# This runs before database initialization.
+pre_hook:
+
+# A callable to run with no arguments late in the initialization process.
+# This runs after adapters are initialized.
+post_hook:
+
+# Which paths.* file system layout to use.
+# You should not change this variable.
+layout: debian
+
+# Can MIME filtered messages be preserved by list owners?
+filtered_messages_are_preservable: no
+
+# How should text/html parts be converted to text/plain when the mailing list
+# is set to convert HTML to plaintext?  This names a command to be called,
+# where the substitution variable $filename is filled in by Mailman, and
+# contains the path to the temporary file that the command should read from.
+# The command should print the converted text to stdout.
+html_to_plain_text_command: /usr/bin/lynx -dump $filename
+
+# Specify what characters are allowed in list names.  Characters outside of
+# the class [-_.+=!$*{}~0-9a-z] matched case insensitively are never allowed,
+# but this specifies a subset as the only allowable characters.  This must be
+# a valid character class regexp or the effect on list creation is
+# unpredictable.
+listname_chars: [-_.0-9a-z]
+
+
+[shell]
+# `mailman shell` (also `withlist`) gives you an interactive prompt that you
+# can use to interact with an initialized and configured Mailman system.  Use
+# --help for more information.  This section allows you to configure certain
+# aspects of this interactive shell.
+
+# Customize the interpreter prompt.
+prompt: >>>
+
+# Banner to show on startup.
+banner: Welcome to the GNU Mailman shell
+
+# Use IPython as the shell, which must be found on the system.  Valid values
+# are `no`, `yes`, and `debug` where the latter is equivalent to `yes` except
+# that any import errors will be displayed to stderr.
+use_ipython: no
+
+# Set this to allow for command line history if readline is available.  This
+# can be as simple as $var_dir/history.py to put the file in the var directory.
+history_file:
+
+
+[paths.debian]
+# Important directories for Mailman operation.  These are defined here so that
+# different layouts can be supported.   For example, a developer layout would
+# be different from a FHS layout.  Most paths are based off the var_dir, and
+# often just setting that will do the right thing for all the other paths.
+# You might also have to set spool_dir though.
+#
+# Substitutions are allowed, but must be of the form $var where 'var' names a
+# configuration variable in the paths.* section.  Substitutions are expanded
+# recursively until no more $-variables are present.  Beware of infinite
+# expansion loops!
+#
+# This is the root of the directory structure that Mailman will use to store
+# its run-time data.
+var_dir: /var/lib/mailman3
+# This is where the Mailman queue files directories will be created.
+queue_dir: $var_dir/queue
+# This is the directory containing the Mailman 'runner' and 'master' commands
+# if set to the string '$argv', it will be taken as the directory containing
+# the 'mailman' command.
+bin_dir: /usr/lib/mailman3/bin
+# All list-specific data.
+list_data_dir: $var_dir/lists
+# Directory where log files go.
+log_dir: /var/log/mailman3
+# Directory for system-wide locks.
+lock_dir: $var_dir/locks
+# Directory for system-wide data.
+data_dir: $var_dir/data
+# Cache files.
+cache_dir: $var_dir/cache
+# Directory for configuration files and such.
+etc_dir: /etc/mailman3
+# Directory containing Mailman plugins.
+ext_dir: $var_dir/ext
+# Directory where the default IMessageStore puts its messages.
+messages_dir: $var_dir/messages
+# Directory for archive backends to store their messages in.  Archivers should
+# create a subdirectory in here to store their files.
+archive_dir: $var_dir/archives
+# Root directory for site-specific template override files.
+template_dir: $var_dir/templates
+# There are also a number of paths to specific file locations that can be
+# defined.  For these, the directory containing the file must already exist,
+# or be one of the directories created by Mailman as per above.
+#
+# This is where PID file for the master runner is stored.
+pid_file: /run/mailman3/master.pid
+# Lock file.
+lock_file: $lock_dir/master.lck
+
+
+[database]
+# The class implementing the IDatabase.
+class: mailman.database.sqlite.SQLiteDatabase
+#class: mailman.database.mysql.MySQLDatabase
+#class: mailman.database.postgresql.PostgreSQLDatabase
+
+# Use this to set the Storm database engine URL.  You generally have one
+# primary database connection for all of Mailman.  List data and most rosters
+# will store their data in this database, although external rosters may access
+# other databases in their own way.  This string supports standard
+# 'configuration' substitutions.
+url: sqlite:///$DATA_DIR/mailman.db
+#url: mysql+pymysql://mailman3:mmpass@localhost/mailman3?charset=utf8&use_unicode=1
+#url: postgresql://mailman3:mmpass@localhost/mailman3
+
+debug: no
+
+
+[logging.debian]
+# This defines various log settings.  The options available are:
+#
+# - level     -- Overrides the default level; this may be any of the
+#                standard Python logging levels, case insensitive.
+# - format    -- Overrides the default format string
+# - datefmt   -- Overrides the default date format string
+# - path      -- Overrides the default logger path.  This may be a relative
+#                path name, in which case it is relative to Mailman's LOG_DIR,
+#                or it may be an absolute path name.  You cannot change the
+#                handler class that will be used.
+# - propagate -- Boolean specifying whether to propagate log message from this
+#                logger to the root "mailman" logger.  You cannot override
+#                settings for the root logger.
+#
+# In this section, you can define defaults for all loggers, which will be
+# prefixed by 'mailman.'.  Use subsections to override settings for specific
+# loggers.  The names of the available loggers are:
+#
+# - archiver        --  All archiver output
+# - bounce          --  All bounce processing logs go here
+# - config          --  Configuration issues
+# - database        --  Database logging (SQLAlchemy and Alembic)
+# - debug           --  Only used for development
+# - error           --  All exceptions go to this log
+# - fromusenet      --  Information related to the Usenet to Mailman gateway
+# - http            --  Internal wsgi-based web interface
+# - locks           --  Lock state changes
+# - mischief        --  Various types of hostile activity
+# - runner          --  Runner process start/stops
+# - smtp            --  Successful SMTP activity
+# - smtp-failure    --  Unsuccessful SMTP activity
+# - subscribe       --  Information about leaves/joins
+# - vette           --  Message vetting information
+format: %(asctime)s (%(process)d) %(message)s
+datefmt: %b %d %H:%M:%S %Y
+propagate: no
+level: info
+path: mailman.log
+
+[webservice]
+# The hostname at which admin web service resources are exposed.
+hostname: localhost
+
+# The port at which the admin web service resources are exposed.
+port: 8001
+
+# Whether or not requests to the web service are secured through SSL.
+use_https: no
+
+# Whether or not to show tracebacks in an HTTP response for a request that
+# raised an exception.
+show_tracebacks: yes
+
+# The API version number for the current (highest) API.
+api_version: 3.1
+
+# The administrative username.
+admin_user: restadmin
+
+# The administrative password.
+admin_pass: ${api_password}
+
+[mta]
+# The class defining the interface to the incoming mail transport agent.
+#incoming: mailman.mta.exim4.LMTP
+incoming: mailman.mta.postfix.LMTP
+
+# The callable implementing delivery to the outgoing mail transport agent.
+# This must accept three arguments, the mailing list, the message, and the
+# message metadata dictionary.
+outgoing: mailman.mta.deliver.deliver
+
+# How to connect to the outgoing MTA.  If smtp_user and smtp_pass is given,
+# then Mailman will attempt to log into the MTA when making a new connection.
+smtp_host: 127.0.0.1
+smtp_port: 25
+smtp_user:
+smtp_pass:
+
+# Where the LMTP server listens for connections.  Use 127.0.0.1 instead of
+# localhost for Postfix integration, because Postfix only consults DNS
+# (e.g. not /etc/hosts).
+lmtp_host: 127.0.0.1
+lmtp_port: 8024
+
+# Where can we find the mail server specific configuration file?  The path can
+# be either a file system path or a Python import path.  If the value starts
+# with python: then it is a Python import path, otherwise it is a file system
+# path.  File system paths must be absolute since no guarantees are made about
+# the current working directory.  Python paths should not include the trailing
+# .cfg, which the file must end with.
+#configuration: python:mailman.config.exim4
+configuration: python:mailman.config.postfix

bundles/mailman/files/postfix.cf

@@ -0,0 +1,53 @@
+# See /usr/share/postfix/main.cf.dist for a commented, more complete version
+
+# Debian specific:  Specifying a file name will cause the first
+# line of that file to be used as the name.  The Debian default
+# is /etc/mailname.
+#myorigin = /etc/mailname
+
+smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
+biff = no
+
+# appending .domain is the MUA's job.
+append_dot_mydomain = no
+
+# Uncomment the next line to generate "delayed mail" warnings
+#delay_warning_time = 4h
+
+readme_directory = no
+
+# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 3.6 on
+# fresh installs.
+compatibility_level = 3.6
+
+# TLS parameters
+smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
+smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
+smtpd_tls_security_level=may
+
+smtp_tls_CApath=/etc/ssl/certs
+smtp_tls_security_level=may
+smtp_tls_session_cache_database = <%text>btree:${data_directory}/smtp_scache</%text>
+
+smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
+myhostname = ${hostname}
+alias_maps = hash:/etc/aliases
+alias_database = hash:/etc/aliases
+mydestination = $myhostname, localhost, localhost.localdomain, ${hostname}
+relayhost =
+mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
+mailbox_size_limit = 0
+recipient_delimiter = +
+inet_interfaces = all
+#inet_protocols = all
+inet_protocols = ipv4
+
+unknown_local_recipient_reject_code = 550
+owner_request_special = no
+
+transport_maps =
+    hash:/var/lib/mailman3/data/postfix_lmtp
+local_recipient_maps =
+    hash:/var/lib/mailman3/data/postfix_lmtp
+relay_domains =
+    hash:/var/lib/mailman3/data/postfix_domains

bundles/mailman/files/uwsgi.ini

@@ -0,0 +1,50 @@
+[uwsgi]
+# Port on which uwsgi will be listening.
+uwsgi-socket = /run/mailman3-web/uwsgi.sock
+
+#Enable threading for python
+enable-threads = true
+
+# Move to the directory wher the django files are.
+chdir = /usr/share/mailman3-web
+
+# Use the wsgi file provided with the django project.
+wsgi-file = wsgi.py
+
+# Setup default number of processes and threads per process.
+master = true
+process = 2
+threads = 2
+
+# Drop privielges and don't run as root.
+uid = www-data
+gid = www-data
+
+plugins = python3
+
+# Setup the django_q related worker processes.
+attach-daemon = python3 manage.py qcluster
+
+# Setup hyperkitty's cron jobs.
+#unique-cron = -1 -1 -1 -1 -1 ./manage.py runjobs minutely
+#unique-cron = -15 -1 -1 -1 -1 ./manage.py runjobs quarter_hourly
+#unique-cron = 0 -1 -1 -1 -1 ./manage.py runjobs hourly
+#unique-cron = 0 0 -1 -1 -1 ./manage.py runjobs daily
+#unique-cron = 0 0 1 -1 -1 ./manage.py runjobs monthly
+#unique-cron = 0 0 -1 -1 0 ./manage.py runjobs weekly
+#unique-cron = 0 0 1 1 -1 ./manage.py runjobs yearly
+
+# Setup the request log.
+#req-logger = file:/var/log/mailman3/web/mailman-web.log
+
+# Log cron seperately.
+#logger = cron file:/var/log/mailman3/web/mailman-web-cron.log
+#log-route = cron uwsgi-cron
+
+# Log qcluster commands seperately.
+#logger = qcluster file:/var/log/mailman3/web/mailman-web-qcluster.log
+#log-route = qcluster uwsgi-daemons
+
+# Last log and it logs the rest of the stuff.
+#logger = file:/var/log/mailman3/web/mailman-web-error.log
+logto = /var/log/mailman3/web/mailman-web.log

bundles/mailman/items.py

@@ -0,0 +1,104 @@
+directories = {
+    '/var/lib/mailman3': {
+        'owner': 'list',
+        'group': 'list',
+        'needs': {
+            'zfs_dataset:tank/mailman',
+            'pkg_apt:mailman3-full',
+        },
+        'needed_by': {
+            'svc_systemd:mailman3.service',
+            'svc_systemd:mailman3-web.service',
+        },
+    },
+}
+
+files = {
+    '/etc/postfix/main.cf': {
+        'source': 'postfix.cf',
+        'content_type': 'mako',
+        'mode': '0644',
+        'context': {
+            'hostname': node.metadata.get('mailman/hostname'),
+        },
+        'needs': {
+            'pkg_apt:postfix',
+        },
+        'triggers': {
+            'svc_systemd:postfix.service:restart',
+        },
+    },
+    '/etc/mailman3/mailman.cfg': {
+        'content_type': 'mako',
+        'owner': 'root',
+        'group': 'list',
+        'mode': '0640',
+        'context': node.metadata.get('mailman'),
+        'needs': {
+            'pkg_apt:mailman3-full',
+        },
+        'triggers': {
+            'svc_systemd:mailman3.service:restart',
+            'svc_systemd:mailman3-web.service:restart',
+        },
+    },
+    '/etc/mailman3/mailman-web.py': {
+        'content_type': 'mako',
+        'owner': 'root',
+        'group': 'www-data',
+        'mode': '0640',
+        'context': node.metadata.get('mailman'),
+        'needs': {
+            'pkg_apt:mailman3-full',
+        },
+        'triggers': {
+            'svc_systemd:mailman3.service:restart',
+            'svc_systemd:mailman3-web.service:restart',
+        },
+    },
+    '/etc/mailman3/mailman-hyperkitty.cfg': {
+        'content_type': 'mako',
+        'owner': 'root',
+        'group': 'list',
+        'mode': '0640',
+        'context': node.metadata.get('mailman'),
+        'needs': {
+            'pkg_apt:mailman3-full',
+        },
+        'triggers': {
+            'svc_systemd:mailman3.service:restart',
+            'svc_systemd:mailman3-web.service:restart',
+        },
+    },
+    '/etc/mailman3/uwsgi.ini': {
+        'content_type': 'text',
+        'owner': 'root',
+        'group': 'root',
+        'mode': '0644',
+        'needs': {
+            'pkg_apt:mailman3-full',
+        },
+        'triggers': {
+            'svc_systemd:mailman3.service:restart',
+            'svc_systemd:mailman3-web.service:restart',
+        },
+    },
+}
+
+svc_systemd = {
+    'postfix.service': {
+        'needs': {
+            'pkg_apt:postfix',
+        },
+    },
+    'mailman3.service': {
+        'needs': {
+            'pkg_apt:mailman3-full',
+        },
+    },
+    'mailman3-web.service': {
+        'needs': {
+            'pkg_apt:mailman3-full',
+        },
+    },
+}

bundles/mailman/metadata.py

@@ -0,0 +1,149 @@
+import base64
+
+def derive_mailadmin_secret(metadata, salt):
+    node_id = metadata.get('id')
+    raw = base64.b64decode(
+        repo.vault.random_bytes_as_base64_for(f'{node_id}_{salt}', length=32).value
+    )
+    return base64.urlsafe_b64encode(raw).rstrip(b'=').decode('ascii')
+
+
+defaults = {
+    'apt': {
+        'packages': {
+            'mailman3-full': {
+                'needs': {
+                    'postgres_db:mailman',
+                    'postgres_role:mailman',
+                    'zfs_dataset:tank/mailman',
+                }
+            },
+            'postfix': {},
+            'python3-psycopg2': {
+                'needed_by': {
+                    'pkg_apt:mailman3-full',
+                },
+            },
+            'apache2': {
+                'installed': False,
+                'needs': {
+                    'pkg_apt:mailman3-full',
+                },
+            },
+        },
+    },
+    'zfs': {
+        'datasets': {
+            'tank/mailman': {
+                'mountpoint': '/var/lib/mailman3',
+            },
+        },
+    },
+}
+
+
+@metadata_reactor.provides(
+    'postgresql',
+    'mailman',
+)
+def postgresql(metadata):
+    node_id = metadata.get('id')
+    db_password = repo.vault.password_for(f'{node_id} database mailman')
+
+    return {
+        'postgresql': {
+            'databases': {
+                'mailman': {
+                    'owner': 'mailman',
+                },
+            },
+            'roles': {
+                'mailman': {
+                    'password': db_password,
+                },
+            },
+        },
+        'mailman': {
+            'db_password': db_password,
+        },
+    }
+
+
+@metadata_reactor.provides(
+    'nginx/vhosts',
+)
+def nginx(metadata):
+    return {
+        'nginx': {
+            'vhosts': {
+                metadata.get('mailman/hostname'): {
+                    'content': 'mailman/vhost.conf',
+                },
+            },
+        },
+    }
+
+
+@metadata_reactor.provides(
+    'mailman/secret_key',
+)
+def secret_key(metadata):
+    import base64
+
+    node_id = metadata.get('id')
+    raw = base64.b64decode(
+        repo.vault.random_bytes_as_base64_for(f'{node_id}_mailman_secret_key', length=32).value
+    )
+    secret_key = base64.urlsafe_b64encode(raw).rstrip(b'=').decode('ascii')
+
+    return {
+        'mailman': {
+            'secret_key': secret_key,
+        },
+    }
+
+
+@metadata_reactor.provides(
+    'mailman',
+)
+def secrets(metadata):
+    return {
+        'mailman': {
+            'web_secret': derive_mailadmin_secret(metadata, 'secret_key'),
+            'api_password': derive_mailadmin_secret(metadata, 'api_password'),
+            'archiver_key': derive_mailadmin_secret(metadata, 'archiver_key'),
+        },
+    }
+
+
+@metadata_reactor.provides(
+    'dns',
+)
+def dns(metadata):
+    report_email = metadata.get('mailman/dmarc_report_email')
+
+    return {
+        'dns': {
+            metadata.get('mailman/hostname'): {
+                'MX': [f"5 {metadata.get('mailman/hostname')}."],
+                'TXT': [
+                    'v=spf1 a mx -all',
+                    '; '.join(f'{k}={v}' for k, v in {
+                    # dmarc version
+                    'v': 'DMARC1',
+                    # reject on failure
+                    'p': 'reject',
+                    # standard reports
+                    'rua': f'mailto:{report_email}',
+                    # forensic reports
+                    'fo': 1,
+                    'ruf': f'mailto:{report_email}',
+                    # require alignment between the DKIM domain and the parent Header From domain
+                    'adkim': 's',
+                    # require alignment between the SPF domain (the sender) and the Header From domain
+                    'aspf': 's',
+                }.items())
+                ],
+            },
+        },
+    }

bundles/mailserver/metadata.py

@@ -32,10 +32,14 @@ defaults = {
             'tank/vmail': {
                 'mountpoint': '/var/vmail',
                 'compression': 'on',
+                'atime': 'off',
+                'recordsize': '16384',
             },
             'tank/vmail/index': {
                 'mountpoint': '/var/vmail/index',
                 'compression': 'on',
+                'atime': 'off',
+                'recordsize': '4096',
                 'com.sun:auto-snapshot': 'false',
                 'backup': False,
             },
@@ -78,6 +82,7 @@ def dns(metadata):
         'dns': dns,
     }
 
+
 @metadata_reactor.provides(
     'letsencrypt/domains',
 )

bundles/mariadb/files/override.conf

@@ -1,11 +0,0 @@
-% for section, options in sorted(conf.items()):
-[${section}]
-% for key, value in sorted(options.items()):
-% if value is None:
-${key}
-% else:
-${key} = ${value}
-% endif
-% endfor
-
-% endfor

bundles/mariadb/items.py

@@ -10,8 +10,6 @@ directories = {
         'group': 'mysql',
         'needs': [
             'zfs_dataset:tank/mariadb',
-        ],
-        'needed_by': [
             'pkg_apt:mariadb-server',
             'pkg_apt:mariadb-client',
         ],
@@ -20,10 +18,8 @@ directories = {
 
 files = {
     '/etc/mysql/conf.d/override.conf': {
-        'context': {
+        'content': repo.libs.ini.dumps(node.metadata.get('mariadb/conf')),
-            'conf': node.metadata.get('mariadb/conf'),
+        'content_type': 'text',
-        },
-        'content_type': 'mako',
     },
 }