spec(deploy-dir-rethink): plan + mark adjacent specs resolved

Adds the implementation plan that landed in the preceding commit
(2026-05-15-deploy-dir-rethink.md) under docs/superpowers/plans/, and
marks the two related specs:

- 2026-05-15-deploy-dir-rethink-design.md (the source handoff) gets a
  "Resolved by …" banner at the top with a one-paragraph summary of
  the decisions taken. Body preserved for archaeology.

- 2026-05-15-janitorial-cleanup.md gets a status banner noting that
  items 1, 3, 4, 5 are fully resolved by the deploy-dir-rethink plan
  and item 2 is partially resolved with a third option the original
  enumeration didn't list: only the truly-dead two static units
  (cake.service, nft-mark.service) deleted, the reactor-emitted set
  (server@, web, workshop-refresh.{service,timer}, slices) retained
  as curated examples. Resolved items left in place but flagged.

Remaining live janitorial items: 6 (bubblewrap doc drift), 7
(conditional on build-overlay-unit refactor), 8 (operational idmap
bind cleanup), 9 (Optimized Settings overlay verification), 10 (SM
1.13 calendar reminder).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

deploy/README.md

@@ -1,117 +1,126 @@
-# left4me Deployment
+# left4me deploy — reference exemplar
 
-> Production provisioning of left4me on `ovh.left4me` is driven by
+> **This directory is reference material, not the source of truth.**
-> [ckn-bw](https://git.sublimity.de/cronekorkn/ckn-bw)
+> The canonical deploy of `ovh.left4me` is driven by
-> (`bundles/left4me/`, attached via `groups/applications/left4me.py`).
+> [ckn-bw](https://git.sublimity.de/cronekorkn/ckn-bw)'s `bundles/left4me/`
-> Run `bw apply ovh.left4me` from the ckn-bw repo to deploy.
+> (attached via `groups/applications/left4me.py`); run `bw apply ovh.left4me`
+> from the ckn-bw repo to deploy.
 >
-> **What's canonical in this directory** (`deploy/files/`, `deploy/templates/`,
+> The privileged scripts the application installs live at the repo root
-> `deploy/tests/`): the actual file payload ckn-bw deploys. ckn-bw fetches
+> under [`scripts/libexec/`](../scripts/libexec/) and
-> the left4me repo via `git_deploy` to `/opt/left4me/src/` and `install`s
+> [`scripts/sbin/`](../scripts/sbin/) — application code, not deploy
-> the privileged scripts from `deploy/files/usr/local/{libexec,sbin}/`
+> artifacts. ckn-bw's `install_left4me_scripts` action reads them from
-> directly onto the target. Sudoers, sysctl, and env-template content
+> `/opt/left4me/src/scripts/{libexec,sbin}/` after `git_deploy` and
-> ships from `deploy/files/etc/` and `deploy/templates/etc/`. **Edit
+> installs them into the standard FHS targets on the host.
-> these files here; ckn-bw picks them up on the next apply.** No
-> duplicate copy of the file content lives in ckn-bw.
 >
-> **What's superseded**: the `deploy-test-server.sh` script — an older
+> What remains under `deploy/files/` and `deploy/templates/` is a set of
-> one-shot bash deploy that ckn-bw replaced. It's kept as a readable
+> readable **examples** — sudoers, sysctl, sandbox-resolv.conf, env
-> description of the install steps the bundle now performs declaratively.
+> templates, and a curated subset of the systemd units ckn-bw's reactor
-> Don't run it against an ovh.left4me node managed by ckn-bw; the two
+> emits at apply time. They exist so a fresh consumer (other than ckn-bw)
-> would fight over file ownership.
+> could read this tree and assemble an equivalent deployment. They are
->
+> **not** the bytes ckn-bw installs; ckn-bw carries its own copies of the
-> **What's obsolete** (kept for greppability, not currently used): CAKE
+> verbatim configs in `bundles/left4me/files/etc/`, and emits the live
-> traffic shaping (now in systemd-networkd via `network/<iface>/cake`
+> units from `bundles/left4me/metadata.py`'s `systemd_units` reactor.
-> metadata in ckn-bw), nft marking (now in the central `nftables/output`
-> set), and the systemd unit files under `files/usr/local/lib/systemd/system/`
-> (emitted by the bundle's `systemd_units` reactor instead of being shipped
-> as static files). The obsolete bits stay here intact so the original
-> choices and tradeoffs remain greppable.
 
-## What lives here (and what corresponds to it in ckn-bw)
+## What's here
 
-| Path here | Status under ckn-bw |
+| Path | Role |
 |---|---|
-| `deploy-test-server.sh` | replaced by `bw apply` |
+| `files/etc/sudoers.d/left4me` | Example sudoers grants. Lockdown test: `scripts/tests/test_sudoers_grants.py`. |
-| `files/etc/sudoers.d/left4me` | shipped verbatim by `bundles/left4me/files/etc/sudoers.d/left4me` (validated with `visudo -cf` via `test_with`) |
+| `files/etc/sysctl.d/99-left4me.conf` | Example sysctl perf baseline (UDP buffers, fq_codel + BBR). |
-| `files/etc/sysctl.d/99-left4me.conf` | shipped verbatim by the bundle |
+| `files/etc/left4me/sandbox-resolv.conf` | Example `/etc/resolv.conf` bound into the script-overlay sandbox. |
-| `files/etc/left4me/sandbox-resolv.conf` | shipped verbatim by the bundle |
+| `files/usr/local/lib/systemd/system/left4me-web.service` | Example of the web-app unit the reactor emits. |
-| `files/usr/local/libexec/left4me/{left4me-systemctl,journalctl,overlay,script-sandbox}` | installed onto the target by the `install_left4me_scripts` action in `bundles/left4me/items.py`, reading directly from `/opt/left4me/src/deploy/files/usr/local/libexec/left4me/` after `git_deploy`. The bundle does **not** carry a duplicate copy. |
+| `files/usr/local/lib/systemd/system/left4me-server@.service` | Example of the per-instance gameserver unit. |
-| `files/usr/local/sbin/left4me` | same install action; admin CLI wrapper (`sudo left4me <flask-subcommand>`) |
+| `files/usr/local/lib/systemd/system/left4me-workshop-refresh.{service,timer}` | Example of the daily workshop-refresh cron-equivalent. |
-| `files/usr/local/lib/systemd/system/left4me-web.service` | emitted by `systemd_units` reactor in `bundles/left4me/metadata.py` (intentional change: `--bind 0.0.0.0:8000` → `127.0.0.1:8000` because nginx now terminates TLS) |
+| `files/usr/local/lib/systemd/system/l4d2-{game,build}.slice` | Example slice definitions (CPU/IO weights). |
-| `files/usr/local/lib/systemd/system/left4me-server@.service` | emitted by the same reactor |
+| `templates/etc/left4me/host.env` | Example host-library env (deployment-fixed paths). |
-| `files/usr/local/lib/systemd/system/{l4d2-game,l4d2-build}.slice` | emitted by the same reactor |
+| `templates/etc/left4me/web.env.template` | Example web-app env. ckn-bw renders the real version via the matching Mako template in `bundles/left4me/files/etc/left4me/web.env.mako`. |
-| `files/usr/local/lib/systemd/system/left4me-cake.service` | **obsolete** — CAKE applied via systemd-networkd (`network/<iface>/cake` metadata in `bundles/network/`) |
+| `tests/test_example_units.py` | Locks down the example units & env templates above. |
-| `files/usr/local/libexec/left4me/left4me-apply-cake` | **obsolete** — same as above |
-| `files/etc/left4me/cake.env` | **obsolete** — bandwidth lives in node metadata under `network/external/cake/Bandwidth` |
-| `files/usr/local/lib/systemd/system/left4me-nft-mark.service` | **obsolete** — central `bundles/nftables/` consumes the rules from `bundles/left4me/`'s defaults |
-| `files/usr/local/lib/left4me/nft/left4me-mark.nft` | **obsolete** — same as above |
-| `templates/etc/left4me/host.env` | rendered as Mako by `bundles/left4me/files/etc/left4me/host.env.mako` |
-| `templates/etc/left4me/web.env.template` | rendered as Mako by `bundles/left4me/files/etc/left4me/web.env.mako` (intentional change: `SESSION_COOKIE_SECURE=false` → `true`, plus `LEFT4ME_PORT_RANGE_*` are now wired through) |
-| First-run admin bootstrap (`flask create-user … --admin` near the end of `deploy-test-server.sh`) | manual one-time step after `bw apply`; the bundle deliberately doesn't seed an admin to keep credentials out of the metadata pipeline |
-| CPU isolation drop-ins (`/etc/systemd/system/{system,user,l4d2-game,l4d2-build}.slice.d/99-left4me-cpuset.conf`) | **not managed by the bundle** — generated dynamically based on `nproc --all` in the script; that logic doesn't fit static bundle metadata, apply manually post-deploy if needed |
 
----
+The privileged scripts (`left4me-overlay`, `left4me-script-sandbox`,
+`left4me-systemctl`, `left4me-journalctl`, `sbin/left4me`) used to live
+under this tree at `files/usr/local/{libexec,sbin}/`; they moved to
+`scripts/{libexec,sbin}/` because they are application code, not deploy
+artifacts.
 
-## Original notes (still accurate as a description of the install steps)
+## Target layout
 
-This directory contains the production-like test deployment for a Linux server. It installs the repository into a fixed host layout, configures a dedicated runtime user, installs systemd units, and wires the web app to host operations through privileged helper commands.
+The deployment uses these on-host paths (FHS-aligned):
 
-## Target Layout
+- `/etc/left4me/host.env` — host library environment configuration.
+- `/etc/left4me/web.env` — web app environment configuration.
+- `/etc/left4me/sandbox-resolv.conf` — DNS resolv.conf bound into the
+  script-overlay sandbox.
+- `/etc/sudoers.d/left4me` — sudoers rules letting the `left4me` uid call
+  the privileged helpers non-interactively.
+- `/etc/sysctl.d/99-left4me.conf` — perf-baseline sysctls.
+- `/opt/left4me` — deployed repository contents (via ckn-bw `git_deploy`).
+- `/opt/left4me/.venv` — Python virtual environment for the web app.
+- `/var/lib/left4me/left4me.db` — SQLite database used by the web app.
+- `/var/lib/left4me/installation` — shared L4D2 installation.
+- `/var/lib/left4me/overlays` — overlay directories. Each overlay lives
+  at `${overlay_id}` under here.
+- `/var/lib/left4me/workshop_cache` — deduplicated cache of `.vpk` files
+  downloaded for workshop overlays. One file per Steam item, named
+  `{steam_id}.vpk`. Workshop overlays symlink into this tree.
+- `/var/lib/left4me/instances` — rendered instance specifications and
+  per-instance state.
+- `/var/lib/left4me/runtime` — per-instance runtime mount directories.
+- `/var/lib/left4me/tmp` — temporary files used by deployment/runtime
+  operations (incl. idmap staging binds).
+- `/usr/local/lib/systemd/system/` — global systemd unit files emitted
+  by ckn-bw's `systemd_units` reactor.
+- `/usr/local/libexec/left4me/` — privileged helper commands installed
+  from `scripts/libexec/`.
+- `/usr/local/sbin/left4me` — admin CLI wrapper installed from
+  `scripts/sbin/left4me`.
 
-The deployment uses these paths:
+## Runtime users
 
-- `/etc/left4me/host.env`: host library environment configuration.
+Two system users are involved:
-- `/etc/left4me/web.env`: web app environment configuration.
-- `/opt/left4me/.venv`: Python virtual environment for deployed commands.
-- `/opt/left4me`: deployed repository contents.
-- `/var/lib/left4me/left4me.db`: SQLite database used by the web app.
-- `/var/lib/left4me/installation`: shared L4D2 installation.
-- `/var/lib/left4me/overlays`: overlay directories. Each overlay lives at `${overlay_id}` under here.
-- `/var/lib/left4me/workshop_cache`: deduplicated cache of `.vpk` files downloaded for workshop overlays. One file per Steam item, named `{steam_id}.vpk`. Workshop overlays symlink into this tree.
-- `/var/lib/left4me/global_overlay_cache`: cache of non-Steam map archives and extracted `.vpk` files used by managed global map overlays.
-- `/var/lib/left4me/instances`: rendered instance specifications and per-instance state.
-- `/var/lib/left4me/runtime`: per-instance runtime mount directories.
-- `/var/lib/left4me/tmp`: temporary files used by deployment/runtime operations.
-- `/usr/local/lib/systemd/system`: global systemd unit files, including `left4me-server@.service`.
-- `/usr/local/libexec/left4me`: privileged helper commands, including `left4me-systemctl`, `left4me-journalctl`, and `left4me-overlay` (the latter mounts the per-instance kernel overlay in PID 1's mount namespace via `nsenter`).
-- `/etc/sudoers.d/left4me`: sudoers rules allowing the web/runtime commands to call the helpers non-interactively.
 
-Static units are generated for `/var/lib/left4me`. If `LEFT4ME_ROOT` changes, regenerate and reinstall the unit files instead of reusing the existing static units.
+- **`left4me`** (home `/var/lib/left4me`, shell `/usr/sbin/nologin`):
+  web app, host library, and gameserver runtime.
+- **`l4d2-sandbox`** (no home, shell `/usr/sbin/nologin`): unprivileged
+  uid the script-overlay sandbox drops into via `systemd-run`. The
+  `left4me-script-sandbox` helper sets up an idmapped bind from the
+  sandbox uid back to `left4me` on a staging path so overlay writes
+  land on disk as `left4me`-owned. The split is load-bearing: a
+  sandbox escape would otherwise see `web.env`, the SQLite DB, and
+  running gameservers.
 
-## Runtime User
+(Whether the gameserver runtime should be split off into a third uid is
+an open design question — see
+`docs/superpowers/specs/2026-05-15-user-uid-split-design.md`.)
 
-The deployment creates and runs host operations as the dedicated runtime user:
+## Deployment
 
-- Username: `left4me`
+Production deploy:
-- Home: `/var/lib/left4me`
-- Shell: `/usr/sbin/nologin`
 
-## Running A Test Deployment
+```sh
-
+# In the ckn-bw repo:
-Run the deployment from the repository root:
+bw apply ovh.left4me
-
-```bash
-deploy/deploy-test-server.sh deploy-user@example-host
 ```
 
-The SSH user must be able to run `sudo` on the target host. The deployment configures system packages, directories, environment files, helper scripts, sudoers rules, Python dependencies, and systemd units.
+Admin bootstrap is a manual one-time step after the first apply
+(ckn-bw deliberately doesn't seed an admin to keep credentials out of
+the metadata pipeline):
 
-## Admin Bootstrap
+```sh
-
+sudo -u left4me LEFT4ME_ADMIN_USERNAME=admin \
-Set the bootstrap credentials in the environment when creating the first admin user:
+  LEFT4ME_ADMIN_PASSWORD='change-me' \
-
+  /opt/left4me/.venv/bin/flask --app l4d2web.app:create_app \
-```bash
+  create-user "$LEFT4ME_ADMIN_USERNAME" --admin
-LEFT4ME_ADMIN_USERNAME=admin \
-LEFT4ME_ADMIN_PASSWORD='change-me' \
-flask create-user "$LEFT4ME_ADMIN_USERNAME" --admin
 ```
 
-Use a strong one-time password and rotate it after first login if needed.
+Rotate the bootstrap password after first login.
 
-## Overlay References
+## Overlay references
 
-Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`. With the default deployment root, they resolve under `/var/lib/left4me/overlays`. New overlays use `${overlay_id}` as their path; the digit-only form is the only one created by the web app.
+Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`.
+With the default deployment root, they resolve under
+`/var/lib/left4me/overlays`. New overlays use `${overlay_id}` as their
+path; the digit-only form is the only one created by the web app.
 
 Invalid references are rejected:
 
@@ -122,125 +131,117 @@ Invalid references are rejected:
 
 The web app currently supports two overlay surfaces:
 
-- `workshop` overlays (user-owned) — populated by downloading `.vpk` files from the public Steam Web API into `${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creating absolute symlinks under `${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`.
+- **`workshop` overlays** (user-owned) — populated by downloading
-- `script` overlays — populated by an arbitrary user-authored bash script that runs inside `bubblewrap` + `systemd-run --scope` as the unprivileged `l4d2-sandbox` UID, with the overlay directory bind-mounted RW at `/overlay`. Resource caps: 1h walltime, 4 GB RAM, 512 tasks, 200% CPU, 20 GB post-build disk cap.
+  `.vpk` files from the public Steam Web API into
+  `${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creating absolute
+  symlinks under
+  `${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`.
+- **`script` overlays** — populated by an arbitrary user-authored bash
+  script that runs inside `systemd-run` as the unprivileged
+  `l4d2-sandbox` UID, with the overlay directory bind-mounted RW at
+  `/overlay`. Resource caps: 1h walltime, 4 GB RAM, 512 tasks, 200% CPU,
+  20 GB post-build disk cap.
 
-Both the caches and the overlay directories are owned by the `left4me` runtime user; if the web service ever runs as a different uid, ensure it shares a group with the host process and that both trees are group-readable.
+Both caches and overlay directories are owned by `left4me`. If the web
+service ever runs as a different uid, ensure it shares a group with the
+host process and that both trees are group-readable.
 
-## Performance Tuning
+## Performance tuning
 
-The deployment ships a host-side perf baseline (slices, unit directives, sysctls). See `docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md` for design rationale.
+The deployment ships a host-side perf baseline (slices, unit directives,
+sysctls). See
+`docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md`
+for design rationale.
 
-The following knobs are documented escape hatches — they are **not** auto-applied. Apply only if you have measured a need and understand the failure modes.
+The knobs below are documented escape hatches — **not** auto-applied.
+Apply only after measuring a need and understanding the failure modes.
 
 ### Network shaping
 
-The deploy ships three things that affect player-experience network behaviour:
+Three pieces of the baseline affect player-experience network behaviour:
 
-1. **Per-flow marking.** `left4me-nft-mark.service` loads a small nftables
+1. **Per-flow marking.** ckn-bw's central `bundles/nftables/` consumes
-   table (`inet left4me_mark`) that marks every UDP packet from uid `left4me`
+   left4me's nftables defaults and marks every UDP packet from uid
-   with DSCP EF and `skb->priority` 6. srcds doesn't set these itself, so
+   `left4me` with DSCP EF and `skb->priority` 6. srcds doesn't set
-   without this rule its UDP is indistinguishable from any other flow.
+   these itself, so without this rule its UDP is indistinguishable
+   from any other flow.
 2. **Sysctl baseline.** `99-left4me.conf` sets `udp_rmem_min=16384`,
    `udp_wmem_min=16384`, `default_qdisc=fq_codel`, and
-   `tcp_congestion_control=bbr`. Reduces head-of-line blocking when bulk
+   `tcp_congestion_control=bbr`. Reduces head-of-line blocking when
-   TCP egress (backups, package fetches, web responses) coexists with
+   bulk TCP egress coexists with game UDP.
-   game UDP.
+3. **CAKE egress shaping.** Configured per-interface via systemd-networkd
-3. **CAKE egress shaping.** `left4me-cake.service` runs
+   metadata (`network/<iface>/cake` in ckn-bw's `bundles/network/`),
-   `tc qdisc replace dev <iface> root cake bandwidth Xmbit internet
+   which reapplies the CAKE qdisc across iface lifecycle events. Set
-   diffserv4 dual-dsthost` from `/etc/left4me/cake.env`. CAKE only shapes
+   the declared bandwidth to ≈95% of measured uplink — CAKE only shapes
-   if its declared bandwidth is **below** the real bottleneck, so set
+   if its declared bandwidth is *below* the real bottleneck. Idle links
-   `LEFT4ME_UPLINK_MBIT` to ≈95% of measured uplink:
+   with no competing egress see no visible CAKE effect; the win
-
+   materialises under bulk traffic that would otherwise bufferbloat the
-       sudoedit /etc/left4me/cake.env
+   link the players share.
-       # set LEFT4ME_UPLINK_MBIT=480 (or whatever ~95% of your uplink is)
-       sudo systemctl restart left4me-cake.service
-
-   `LEFT4ME_UPLINK_IFACE` is auto-detected from the IPv4 default route;
-   override only on hosts with multi-homed setups.
-
-   At idle 500 Mbit with no competing egress, CAKE shapes nothing — that's
-   expected, not a bug. The win materialises when bulk traffic on the
-   same uplink would otherwise bufferbloat the link the players share.
-
-**Production hosts running `systemd-networkd`** should NOT use the
-`left4me-cake.service` oneshot. Instead, configure the equivalent in the
-matching `.network` file, which systemd-networkd reapplies across iface
-lifecycle events:
-
-    # /etc/systemd/network/<your-uplink>.network
-    [CAKE]
-    Bandwidth=480M
-    OverheadKeyword=internet
-    PriorityQueueingPreset=diffserv4
-    EgressHostIsolation=yes
-
-The nftables marking from (1) is qdisc-installer-agnostic and ships
-unchanged on production.
-
-**Disabling network shaping.** To turn the whole feature off on a deployed
-host:
-
-    sudo systemctl stop left4me-cake.service left4me-nft-mark.service
-    sudo systemctl disable left4me-cake.service left4me-nft-mark.service
-
-The sysctl baseline (`99-left4me.conf`) and the BBR/fq_codel defaults stay
-applied; revert those by removing the file and running `sysctl --system`
-if needed.
 
 ### CPU governor
 
-The performance governor squeezes a few percent off jitter under bursty load. `schedutil` is acceptable for sustained UDP workloads.
+The performance governor squeezes a few percent off jitter under bursty
+load. `schedutil` is acceptable for sustained UDP workloads.
 
 ```sh
 sudo cpupower frequency-set -g performance
 ```
 
-Install via `sudo apt install linux-cpupower` if the binary isn't present.
+Install via `sudo apt install linux-cpupower` if the binary isn't
-
+present. Persist via your distro's CPU-frequency tooling (e.g.
-Persist via your distro's CPU-frequency tooling (e.g. `/etc/default/cpufrequtils`).
+`/etc/default/cpufrequtils`).
 
 ### CPU isolation (cores)
 
-The deploy script writes four `AllowedCPUs=` drop-ins so that, by default, only `l4d2-game.slice` is allowed to run on cores 1..N-1; `system.slice`, `user.slice`, and `l4d2-build.slice` are pinned to core 0. Game servers thus get the host minus core 0 exclusively, the build sandbox and the web app stay on core 0, and a logged-in admin running CPU-heavy work in their shell can't steal cycles from a live match.
+The deploy writes four `AllowedCPUs=` drop-ins so that by default only
+`l4d2-game.slice` is allowed to run on cores `1..N-1`; `system.slice`,
+`user.slice`, and `l4d2-build.slice` are pinned to core 0. Game servers
+get the host minus core 0 exclusively; the build sandbox and the web
+app stay on core 0; a logged-in admin running CPU-heavy work in their
+shell can't steal cycles from a live match. Single-core hosts skip the
+cpuset drop-ins entirely; the rest of the perf baseline (cgroup
+weights, sysctls, OOM scores) still applies.
 
-Override the split by setting either env var when running the deploy:
+Per-instance `CPUAffinity=` (next subsection) composes on top of this —
-
+the per-instance value must be a subset of `l4d2-game.slice`'s
-```sh
+`AllowedCPUs=`, which the kernel enforces.
-LEFT4ME_SYSTEM_CPUS="0,1" LEFT4ME_GAME_CPUS="2-7" deploy/deploy-test-server.sh deploy-user@host
-```
-
-On single-core hosts the deploy skips the cpuset drop-ins entirely and prints a warning to stderr; the rest of the perf baseline (cgroup weights, sysctls, OOM scores) still applies. To force isolation on a single-core host anyway (rarely useful), set either env var explicitly.
-
-Per-instance `CPUAffinity=` (next subsection) composes on top of this — the per-instance value must be a subset of `l4d2-game.slice`'s `AllowedCPUs=`, which the kernel enforces.
 
 ### Per-instance CPU affinity
 
-`srcds` is single-threaded per instance. On a multi-core host, pinning each instance to its own core can cut jitter under contention. Drop in `/etc/systemd/system/left4me-server@<name>.service.d/affinity.conf`:
+`srcds` is single-threaded per instance. On a multi-core host, pinning
+each instance to its own core can cut jitter under contention. Drop in
+`/etc/systemd/system/left4me-server@<name>.service.d/affinity.conf`:
 
 ```ini
 [Service]
 CPUAffinity=2
 ```
 
-This pins the instance to CPU 2 specifically; per-instance values would typically be 1, 2, 3, ... so each server has its own core.
+This pins the instance to CPU 2. A reasonable strategy on an N-core
-
+host: leave core 0 for the kernel + IRQs + system services, then pin
-A reasonable strategy on an N-core host: leave core 0 for the kernel + IRQs + system services, then pin one instance per remaining core.
+one instance per remaining core.
 
 ### NIC tuning
 
-Hardware-specific (install via `sudo apt install ethtool` if not present). On a host with a single primary interface (replace `eth0`):
+Hardware-specific (install via `sudo apt install ethtool` if not
+present). On a host with a single primary interface (replace `eth0`):
 
 ```sh
 sudo ethtool -G eth0 rx 4096 tx 4096
 sudo ethtool -K eth0 gro on lro off
 ```
 
-If you run a high instance count, also pin the NIC's interrupts off the cores that game servers occupy (see `/proc/interrupts` and `/proc/irq/<n>/smp_affinity`).
+If you run a high instance count, also pin the NIC's interrupts off
+the cores that game servers occupy (see `/proc/interrupts` and
+`/proc/irq/<n>/smp_affinity`).
 
 ### Real-time scheduling (advanced, opt-in)
 
-Source-engine servers do not need real-time scheduling, and a misbehaving `srcds` at any RT priority can starve kernel threads — even with the default `kernel.sched_rt_runtime_us=950000` throttling 5% of CPU back. Use only if you have a measured jitter problem that the baseline does not solve.
+Source-engine servers do not need real-time scheduling, and a
+misbehaving `srcds` at any RT priority can starve kernel threads — even
+with the default `kernel.sched_rt_runtime_us=950000` throttling 5% of
+CPU back. Use only if you have a measured jitter problem that the
+baseline does not solve.
 
 `/etc/systemd/system/left4me-server@.service.d/realtime.conf`:
 
@@ -252,13 +253,16 @@ LimitRTPRIO=10
 AmbientCapabilities=CAP_SYS_NICE
 ```
 
-The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the service runs as `User=left4me` with `NoNewPrivileges=true`; without it some kernels/systemd combinations refuse to apply the RT policy.
+The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the
+service runs as `User=left4me` with `NoNewPrivileges=true`; without it
+some kernels/systemd combinations refuse to apply the RT policy.
 
 ### Additional opt-in network knobs
 
 - **Ingress shaping via IFB.** Egress CAKE alone does not protect srcds
-  receive against ingress saturation (large workshop downloads, package
+  receive against ingress saturation (large workshop downloads,
-  fetches arriving at line rate). One-liner:
+  package fetches arriving at line rate). Worth flipping only when
+  measurement shows ingress hurting receive.
 
       sudo modprobe ifb && sudo ip link set ifb0 up
       sudo tc qdisc add dev <uplink> handle ffff: ingress
@@ -267,13 +271,11 @@ The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the service runs a
       sudo tc qdisc add dev ifb0 root cake bandwidth Xmbit ingress \
           diffserv4 dual-srchost
 
-  Worth flipping only when measurement shows ingress hurting receive.
+- **`net.core.busy_poll = 50` / `net.core.busy_read = 50`.** Reduces
-
+  UDP receive median latency by polling for incoming packets briefly
-- **`net.core.busy_poll = 50` / `net.core.busy_read = 50`.** Reduces UDP
+  at syscall boundaries. Cost: measurable CPU per syscall under load.
-  receive median latency by polling for incoming packets briefly at
+  Worth flipping if a host is dedicated to game serving and CPU
-  syscall boundaries. Cost: measurable CPU per syscall under load. Worth
+  headroom is plentiful.
-  flipping if a host is dedicated to game serving and CPU headroom is
-  plentiful.
 
 - **`ethtool -K <iface> gro off`.** Some Source-engine ops disable
   generic receive offload to avoid receive-side coalescing latency.
@@ -281,7 +283,8 @@ The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the service runs a
 
 ### Applying changes to running servers
 
-Unit-file changes do not apply to already-running services. After any change:
+Unit-file changes do not apply to already-running services. After any
+change:
 
 ```sh
 sudo systemctl daemon-reload

deploy/deploy-test-server.sh

@@ -1,354 +0,0 @@
-#!/bin/sh
-set -eu
-
-usage() {
-    printf 'Usage: %s <ssh-user@host>\n' "$0" >&2
-    exit 2
-}
-
-if [ "$#" -ne 1 ]; then
-    usage
-fi
-
-target=$1
-script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
-repo_root=$(CDPATH= cd -- "$script_dir/.." && pwd)
-tmp_dir=$(mktemp -d)
-archive="$tmp_dir/left4me.tar.gz"
-
-cleanup() {
-    rm -rf "$tmp_dir"
-}
-trap cleanup EXIT INT HUP TERM
-
-COPYFILE_DISABLE=1 tar -czf "$archive" \
-    --exclude .git \
-    --exclude .claude \
-    --exclude .venv \
-    --exclude __pycache__ \
-    --exclude .pytest_cache \
-    --exclude '*.egg-info' \
-    --exclude 'l4d2web.db*' \
-    --exclude '._*' \
-    -C "$repo_root" .
-
-remote_tmp=$(ssh "$target" 'mktemp -d')
-scp "$archive" "$target:$remote_tmp/left4me.tar.gz"
-
-admin_username_file=
-admin_password_file=
-if [ "${LEFT4ME_ADMIN_USERNAME+x}" = x ] && [ "${LEFT4ME_ADMIN_PASSWORD+x}" = x ]; then
-    admin_username_file="$tmp_dir/admin_username"
-    admin_password_file="$tmp_dir/admin_password"
-    umask 077
-    printf '%s' "$LEFT4ME_ADMIN_USERNAME" > "$admin_username_file"
-    printf '%s' "$LEFT4ME_ADMIN_PASSWORD" > "$admin_password_file"
-    scp "$admin_username_file" "$target:$remote_tmp/admin_username"
-    scp "$admin_password_file" "$target:$remote_tmp/admin_password"
-fi
-
-ssh "$target" sh -s -- "$remote_tmp" <<'REMOTE'
-set -eu
-
-remote_tmp=$1
-archive="$remote_tmp/left4me.tar.gz"
-repo_tmp="$remote_tmp/repo"
-
-if [ "$(id -u)" -eq 0 ]; then
-    sudo_cmd=
-else
-    sudo_cmd=sudo
-fi
-
-run_as_left4me() {
-    sudo -u left4me "$@"
-}
-
-run_left4me_with_env() {
-    run_as_left4me sh -c 'set -a; . /etc/left4me/host.env; . /etc/left4me/web.env; set +a; exec "$@"' sh "$@"
-}
-
-cleanup_remote() {
-    rm -rf "$remote_tmp"
-}
-trap cleanup_remote EXIT INT HUP TERM
-
-if ! id left4me >/dev/null 2>&1; then
-    $sudo_cmd useradd --system --home-dir /var/lib/left4me --create-home --shell /usr/sbin/nologin left4me
-fi
-
-# Sandbox uid for script-overlay builds. No home, no login shell — the bwrap
-# invocation uses --uid/--gid to drop to it.
-if ! id l4d2-sandbox >/dev/null 2>&1; then
-    $sudo_cmd useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox
-fi
-
-if command -v apt-get >/dev/null 2>&1; then
-    $sudo_cmd apt-get update
-    $sudo_cmd apt-get install -y python3 python3-venv python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip-full nftables iproute2
-elif command -v dnf >/dev/null 2>&1; then
-    $sudo_cmd dnf install -y python3 python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip p7zip-plugins nftables iproute
-else
-    printf 'Unsupported package manager: expected apt-get or dnf\n' >&2
-    exit 1
-fi
-
-$sudo_cmd mkdir -p \
-    /etc/left4me \
-    /opt/left4me \
-    /usr/local/lib/systemd/system \
-    /usr/local/libexec/left4me \
-    /usr/local/lib/left4me/nft \
-    /var/lib/left4me/installation \
-    /var/lib/left4me/overlays \
-    /var/lib/left4me/instances \
-    /var/lib/left4me/runtime \
-    /var/lib/left4me/workshop_cache \
-    /var/lib/left4me/tmp
-
-$sudo_cmd chown left4me:left4me \
-    /var/lib/left4me \
-    /var/lib/left4me/installation \
-    /var/lib/left4me/overlays \
-    /var/lib/left4me/instances \
-    /var/lib/left4me/runtime \
-    /var/lib/left4me/workshop_cache \
-    /var/lib/left4me/tmp
-
-# /var/lib/left4me is left4me's home dir (mode 0700 from useradd --create-home).
-# Allow other uids (notably l4d2-sandbox, used by script overlay builds) to
-# traverse — but not list — so the bwrap bind-mount can resolve the overlay
-# path under the dropped privilege.
-$sudo_cmd chmod 0711 /var/lib/left4me
-$sudo_cmd chown -R left4me:left4me /opt/left4me
-
-mkdir -p "$repo_tmp"
-tar -xzf "$archive" -C "$repo_tmp"
-
-if [ -d /opt/left4me/.venv ]; then
-    $sudo_cmd mv /opt/left4me/.venv "$remote_tmp/venv"
-fi
-$sudo_cmd find /opt/left4me -mindepth 1 -maxdepth 1 -exec rm -rf {} +
-$sudo_cmd cp -R "$repo_tmp"/. /opt/left4me/
-if [ -d "$remote_tmp/venv" ]; then
-    $sudo_cmd mv "$remote_tmp/venv" /opt/left4me/.venv
-fi
-$sudo_cmd chown -R left4me:left4me /opt/left4me
-
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-web.service /usr/local/lib/systemd/system/left4me-web.service
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-server@.service /usr/local/lib/systemd/system/left4me-server@.service
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/l4d2-game.slice /usr/local/lib/systemd/system/l4d2-game.slice
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/l4d2-build.slice /usr/local/lib/systemd/system/l4d2-build.slice
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service /usr/local/lib/systemd/system/left4me-nft-mark.service
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-cake.service /usr/local/lib/systemd/system/left4me-cake.service
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-workshop-refresh.service /usr/local/lib/systemd/system/left4me-workshop-refresh.service
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-workshop-refresh.timer /usr/local/lib/systemd/system/left4me-workshop-refresh.timer
-
-# CPU isolation via cgroup-v2 AllowedCPUs= drop-ins. Pin everything that
-# isn't a live game server to core 0; give game servers cores 1..N-1.
-# See docs/superpowers/specs/2026-05-09-l4d2-cpu-isolation-design.md.
-# `nproc --all` reports installed processors regardless of the calling
-# shell's CPU affinity. Plain `nproc` honors Cpus_allowed of the calling
-# process, so on a host that already has the cpuset drop-ins applied
-# (system.slice → AllowedCPUs=0), the SSH login lands in user.slice with
-# AllowedCPUs=0 and `nproc` would return 1 — making subsequent deploys
-# wrongly think they're on a single-core box and skip CPU isolation.
-NPROC=$(nproc --all)
-SYSTEM_CPUS=${LEFT4ME_SYSTEM_CPUS:-0}
-if [ "${LEFT4ME_GAME_CPUS+x}" = x ]; then
-    GAME_CPUS=$LEFT4ME_GAME_CPUS
-else
-    GAME_CPUS="1-$((NPROC - 1))"
-fi
-if [ "$NPROC" -lt 2 ] && [ "${LEFT4ME_SYSTEM_CPUS+x}${LEFT4ME_GAME_CPUS+x}" = "" ]; then
-    printf 'left4me deploy: skipping CPU isolation (nproc=%s); cpuset drop-ins not written.\n' "$NPROC" >&2
-else
-    for slice_drop_in in \
-        /etc/systemd/system/system.slice.d/99-left4me-cpuset.conf \
-        /etc/systemd/system/user.slice.d/99-left4me-cpuset.conf \
-        /etc/systemd/system/l4d2-build.slice.d/99-left4me-cpuset.conf; do
-        $sudo_cmd mkdir -p "$(dirname "$slice_drop_in")"
-        printf '[Slice]\nAllowedCPUs=%s\n' "$SYSTEM_CPUS" \
-            | $sudo_cmd install -m 0644 -o root -g root /dev/stdin "$slice_drop_in"
-    done
-    $sudo_cmd mkdir -p /etc/systemd/system/l4d2-game.slice.d
-    printf '[Slice]\nAllowedCPUs=%s\n' "$GAME_CPUS" \
-        | $sudo_cmd install -m 0644 -o root -g root /dev/stdin \
-          /etc/systemd/system/l4d2-game.slice.d/99-left4me-cpuset.conf
-fi
-
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-systemctl
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-journalctl
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-overlay
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-script-sandbox /usr/local/libexec/left4me/left4me-script-sandbox
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-apply-cake /usr/local/libexec/left4me/left4me-apply-cake
-$sudo_cmd cp /opt/left4me/deploy/files/usr/local/sbin/left4me /usr/local/sbin/left4me
-$sudo_cmd chmod 0755 /usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-script-sandbox /usr/local/libexec/left4me/left4me-apply-cake /usr/local/sbin/left4me
-$sudo_cmd cp /opt/left4me/deploy/files/etc/sudoers.d/left4me /etc/sudoers.d/left4me
-$sudo_cmd chmod 0440 /etc/sudoers.d/left4me
-$sudo_cmd visudo -cf /etc/sudoers.d/left4me
-
-$sudo_cmd cp /opt/left4me/deploy/templates/etc/left4me/host.env /etc/left4me/host.env
-$sudo_cmd chmod 0644 /etc/left4me/host.env
-
-# Sandbox-only resolver config; bind-mounted into the script sandbox's /etc/resolv.conf
-# so DNS still works when IPAddressDeny= blocks the host's (typically private-IP) resolver.
-$sudo_cmd install -m 0644 -o root -g root \
-    /opt/left4me/deploy/files/etc/left4me/sandbox-resolv.conf \
-    /etc/left4me/sandbox-resolv.conf
-
-# Network packet marking + shaping. See spec
-# docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md.
-$sudo_cmd install -m 0644 -o root -g root \
-    /opt/left4me/deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft \
-    /usr/local/lib/left4me/nft/left4me-mark.nft
-
-# Host perf-baseline sysctls. Apply with `sysctl --system` so values
-# take effect this deploy, not on next reboot.
-$sudo_cmd install -m 0644 -o root -g root \
-    /opt/left4me/deploy/files/etc/sysctl.d/99-left4me.conf \
-    /etc/sysctl.d/99-left4me.conf
-$sudo_cmd sysctl --system >/dev/null
-
-# CAKE config: ship the template only if the operator hasn't created one
-# (their LEFT4ME_UPLINK_MBIT value must survive re-deploys).
-if [ -e /etc/left4me/cake.env ]; then
-    : # operator file present; leave it intact
-else
-    $sudo_cmd install -m 0644 -o root -g root \
-        /opt/left4me/deploy/files/etc/left4me/cake.env \
-        /etc/left4me/cake.env
-fi
-
-# Stomp the file every deploy so newly added vars reach existing boxes.
-# SECRET_KEY is derived from /etc/machine-id so it stays stable across
-# redeploys (no session invalidation) without persisting state in /etc.
-secret_key=$(sha256sum < /etc/machine-id | awk '{print $1}')
-tmp_web_env="$remote_tmp/web.env"
-{
-    printf 'DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n'
-    printf 'SECRET_KEY=%s\n' "$secret_key"
-    printf 'JOB_WORKER_THREADS=4\n'
-    printf 'SESSION_COOKIE_SECURE=false\n'
-} > "$tmp_web_env"
-$sudo_cmd install -m 0640 -o root -g left4me "$tmp_web_env" /etc/left4me/web.env
-
-if [ ! -x /opt/left4me/.venv/bin/python ]; then
-    run_as_left4me python3 -m venv /opt/left4me/.venv
-fi
-run_as_left4me /opt/left4me/.venv/bin/python -m pip install --upgrade pip
-run_as_left4me /opt/left4me/.venv/bin/pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web
-
-run_as_left4me sh -c "cd /opt/left4me/l4d2web && set -a; . /etc/left4me/host.env; . /etc/left4me/web.env; set +a; env \
-    JOB_WORKER_ENABLED=false \
-    PYTHONPATH=/opt/left4me \
-    /opt/left4me/.venv/bin/alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head"
-
-# Tighten the application database to left4me:left4me 0640. The DB is
-# created by the web app at first start with the default 0644 umask, which
-# makes it world-readable on the host. The script-overlay sandbox runs as a
-# separate system uid (l4d2-sandbox) which is NOT in the left4me group —
-# 0640 blocks it via "other". The owner (left4me) keeps read+write so the
-# web service can update the DB.
-#
-# SQLite in WAL mode (the default in this app) maintains -wal and -shm
-# sidecar files; both must also be writable by the web service. If a previous
-# operator opened the DB as root (e.g. for ad-hoc inspection), the sidecars
-# may have ended up root-owned, which makes SQLite report "readonly database"
-# on the next write. Re-chown them defensively. Idempotent on rerun.
-for db_file in /var/lib/left4me/left4me.db /var/lib/left4me/left4me.db-wal /var/lib/left4me/left4me.db-shm; do
-    if [ -f "$db_file" ]; then
-        $sudo_cmd chown left4me:left4me "$db_file"
-        $sudo_cmd chmod 0640 "$db_file"
-    fi
-done
-
-if [ -f "$remote_tmp/admin_username" ] && [ -f "$remote_tmp/admin_password" ]; then
-    LEFT4ME_ADMIN_USERNAME=$(cat "$remote_tmp/admin_username")
-    LEFT4ME_ADMIN_PASSWORD=$(cat "$remote_tmp/admin_password")
-    if ! create_user_output=$(run_left4me_with_env env \
-        JOB_WORKER_ENABLED=false \
-        LEFT4ME_ADMIN_PASSWORD="$LEFT4ME_ADMIN_PASSWORD" \
-        /opt/left4me/.venv/bin/flask --app l4d2web.app:create_app create-user "$LEFT4ME_ADMIN_USERNAME" --admin 2>&1); then
-        case "$create_user_output" in
-            *'user already exists'*) printf '%s\n' "$create_user_output" ;;
-            *) printf '%s\n' "$create_user_output" >&2; exit 1 ;;
-        esac
-    else
-        printf '%s\n' "$create_user_output"
-    fi
-fi
-
-# One-shot migration: fuse-overlayfs running as the left4me user used
-# user.fuseoverlayfs.* xattrs for whiteouts and opaque-dir markers; kernel
-# overlayfs ignores those entirely, so a pre-existing upper/ from the fuse
-# era would resurrect "deleted" files. Wipe upper/ and work/ for every
-# instance once, gated by a sentinel file so reruns are no-ops.
-overlay_sentinel=/var/lib/left4me/.kernel-overlay-migrated
-if [ ! -e "$overlay_sentinel" ]; then
-    $sudo_cmd sh -c "systemctl stop 'left4me-server@*.service' 2>/dev/null || true"
-    $sudo_cmd systemctl stop left4me-web.service 2>/dev/null || true
-    $sudo_cmd sh -c "findmnt -t fuse.fuse-overlayfs -o TARGET --noheadings 2>/dev/null | xargs -r -n1 umount -l 2>/dev/null || true"
-    $sudo_cmd sh -c "findmnt -t overlay -o TARGET --noheadings 2>/dev/null | grep '/var/lib/left4me/runtime/' | xargs -r -n1 umount -l 2>/dev/null || true"
-    $sudo_cmd sh -c 'for d in /var/lib/left4me/runtime/*/; do [ -d "$d" ] || continue; rm -rf "$d/upper" "$d/work"; mkdir -p "$d/upper" "$d/work"; chown left4me:left4me "$d/upper" "$d/work"; done'
-    $sudo_cmd touch "$overlay_sentinel"
-    $sudo_cmd chown left4me:left4me "$overlay_sentinel"
-fi
-
-# One-shot migration: 0005_script_overlays drops the legacy
-# l4d2center_maps / cedapug_maps overlay rows but doesn't touch their
-# directories under /var/lib/left4me/overlays/{id}. Without cleanup, when
-# AUTOINCREMENT (or its absence after the 0002 batch_alter_table recreate)
-# re-issues an id matching one of those orphan dirs, the web app's
-# create_overlay_directory(exist_ok=False) fails with FileExistsError.
-# Sweep any overlay dir whose id has no matching DB row, plus the
-# now-unused global_overlay_cache.
-overlay_orphan_sentinel=/var/lib/left4me/.script-overlays-orphans-cleaned
-if [ ! -e "$overlay_orphan_sentinel" ]; then
-    $sudo_cmd rm -rf /var/lib/left4me/global_overlay_cache
-    $sudo_cmd sh -c '
-        cd /var/lib/left4me/overlays || exit 0
-        ids_in_db=$(/opt/left4me/.venv/bin/python -c "
-import sqlite3
-c = sqlite3.connect(\"/var/lib/left4me/left4me.db\")
-print(\" \".join(str(r[0]) for r in c.execute(\"SELECT id FROM overlays\")))
-")
-        for d in */; do
-            id=${d%/}
-            case " $ids_in_db " in
-                *" $id "*) ;;
-                *) echo "removing orphan overlay dir: $id"; rm -rf "$id" ;;
-            esac
-        done
-    '
-    $sudo_cmd touch "$overlay_orphan_sentinel"
-    $sudo_cmd chown left4me:left4me "$overlay_orphan_sentinel"
-fi
-
-# Seed example script overlays (cedapug, l4d2center, competitive_rework, ...)
-# as system-wide rows from /opt/left4me/examples/script-overlays/. Idempotent:
-# subsequent deploys refresh the script body in place, leaving the row id and
-# overlay directory intact.
-run_left4me_with_env env \
-    JOB_WORKER_ENABLED=false \
-    PYTHONPATH=/opt/left4me \
-    /opt/left4me/.venv/bin/flask --app l4d2web.app:create_app \
-    seed-script-overlays /opt/left4me/examples/script-overlays
-
-$sudo_cmd systemctl daemon-reload
-$sudo_cmd systemctl enable --now left4me-nft-mark.service
-$sudo_cmd systemctl enable --now left4me-cake.service
-$sudo_cmd systemctl enable --now left4me-web.service
-$sudo_cmd systemctl restart left4me-web.service
-$sudo_cmd systemctl enable --now left4me-workshop-refresh.timer
-for attempt in 1 2 3 4 5 6 7 8 9 10; do
-    if curl -fsS http://127.0.0.1:8000/health; then
-        exit 0
-    fi
-    sleep 1
-done
-
-$sudo_cmd systemctl status left4me-web.service --no-pager >&2 || true
-$sudo_cmd journalctl -u left4me-web.service -n 80 --no-pager >&2 || true
-exit 1
-REMOTE

deploy/files/etc/left4me/cake.env

@@ -1,12 +0,0 @@
-# left4me — CAKE egress shaper config. Consumed by left4me-cake.service via
-# its EnvironmentFile=. Edit then `systemctl restart left4me-cake.service`.
-# See docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md.
-
-# Uplink bandwidth in Mbit/s. Set to ~95% of the smaller of measured upload
-# and measured download. CAKE only shapes correctly when its declared
-# bandwidth sits below the real bottleneck. If unset, the shaper unit logs
-# a warning and exits 0 (no shaping).
-LEFT4ME_UPLINK_MBIT=
-
-# Egress interface. If unset, auto-detected from the IPv4 default route.
-LEFT4ME_UPLINK_IFACE=

deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft

@@ -1,12 +0,0 @@
-# left4me — uid-based DSCP/priority marking for srcds UDP egress.
-# Loaded by left4me-nft-mark.service into its own `inet` table so it cannot
-# conflict with whatever the operator already runs in /etc/nftables.conf.
-# See docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md.
-
-table inet left4me_mark {
-    chain mangle_output {
-        type filter hook output priority mangle; policy accept;
-        meta skuid "left4me" meta l4proto udp ip dscp set ef meta priority set 0006:0000
-        meta skuid "left4me" meta l4proto udp ip6 dscp set ef meta priority set 0006:0000
-    }
-}

deploy/files/usr/local/lib/systemd/system/left4me-cake.service

@@ -1,14 +0,0 @@
-[Unit]
-Description=left4me CAKE egress shaper
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=oneshot
-RemainAfterExit=yes
-EnvironmentFile=-/etc/left4me/cake.env
-ExecStart=/usr/local/libexec/left4me/left4me-apply-cake apply
-ExecStop=/usr/local/libexec/left4me/left4me-apply-cake clear
-
-[Install]
-WantedBy=multi-user.target

deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service

@@ -1,14 +0,0 @@
-[Unit]
-Description=left4me nftables packet marking (DSCP EF + priority for srcds)
-After=network-pre.target
-Before=network.target
-Wants=network-pre.target
-
-[Service]
-Type=oneshot
-RemainAfterExit=yes
-ExecStart=/usr/sbin/nft -f /usr/local/lib/left4me/nft/left4me-mark.nft
-ExecStop=/usr/sbin/nft delete table inet left4me_mark
-
-[Install]
-WantedBy=multi-user.target

deploy/files/usr/local/libexec/left4me/left4me-apply-cake

@@ -1,47 +0,0 @@
-#!/bin/sh
-# left4me — apply or clear CAKE egress shaper on the configured uplink.
-# Driven by left4me-cake.service. See spec
-# docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md.
-set -eu
-
-mode=${1:-apply}
-
-if [ -r /etc/left4me/cake.env ]; then
-    . /etc/left4me/cake.env
-fi
-
-resolve_iface() {
-    if [ -n "${LEFT4ME_UPLINK_IFACE:-}" ]; then
-        printf '%s' "$LEFT4ME_UPLINK_IFACE"
-        return
-    fi
-    ip -4 route show default | awk '/default/ {print $5; exit}'
-}
-
-case "$mode" in
-    apply)
-        if [ -z "${LEFT4ME_UPLINK_MBIT:-}" ]; then
-            echo "left4me-cake: LEFT4ME_UPLINK_MBIT unset; skipping shaper" >&2
-            exit 0
-        fi
-        iface=$(resolve_iface)
-        if [ -z "$iface" ]; then
-            echo "left4me-cake: cannot determine egress iface; skipping" >&2
-            exit 0
-        fi
-        exec tc qdisc replace dev "$iface" root cake \
-            bandwidth "${LEFT4ME_UPLINK_MBIT}mbit" \
-            internet diffserv4 dual-dsthost
-        ;;
-    clear)
-        iface=$(resolve_iface)
-        if [ -z "$iface" ]; then
-            exit 0
-        fi
-        tc qdisc del dev "$iface" root 2>/dev/null || true
-        ;;
-    *)
-        echo "usage: $0 [apply|clear]" >&2
-        exit 2
-        ;;
-esac

deploy/tests/test_deploy_artifacts.py

@@ -1,882 +0,0 @@
-import os
-import subprocess
-from pathlib import Path
-
-
-ROOT = Path(__file__).resolve().parents[2]
-DEPLOY = ROOT / "deploy"
-
-
-WEB_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-web.service"
-SERVER_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-server@.service"
-GAME_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-game.slice"
-BUILD_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-build.slice"
-SYSCTL_CONF = DEPLOY / "files/etc/sysctl.d/99-left4me.conf"
-GLOBAL_REFRESH_SERVICE = DEPLOY / "files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.service"
-GLOBAL_REFRESH_TIMER = DEPLOY / "files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer"
-NFT_MARK_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-nft-mark.service"
-CAKE_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-cake.service"
-SANDBOX_UNIT_DIR = DEPLOY / "files/usr/local/lib/systemd/system"
-SYSTEMCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-systemctl"
-JOURNALCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-journalctl"
-OVERLAY_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-overlay"
-SCRIPT_SANDBOX_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-script-sandbox"
-APPLY_CAKE_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-apply-cake"
-SANDBOX_RESOLV_CONF = DEPLOY / "files/etc/left4me/sandbox-resolv.conf"
-CAKE_ENV = DEPLOY / "files/etc/left4me/cake.env"
-SUDOERS = DEPLOY / "files/etc/sudoers.d/left4me"
-HOST_ENV = DEPLOY / "templates/etc/left4me/host.env"
-WEB_ENV_TEMPLATE = DEPLOY / "templates/etc/left4me/web.env.template"
-DEPLOY_SCRIPT = DEPLOY / "deploy-test-server.sh"
-NFT_MARK_FILE = DEPLOY / "files/usr/local/lib/left4me/nft/left4me-mark.nft"
-
-
-def test_global_unit_files_exist_at_product_level_paths():
-    assert WEB_UNIT.is_file()
-    assert SERVER_UNIT.is_file()
-
-
-def test_web_unit_contains_required_runtime_contract():
-    unit = WEB_UNIT.read_text()
-
-    assert "User=left4me" in unit
-    assert "Group=left4me" in unit
-    assert "WorkingDirectory=/opt/left4me" in unit
-    assert "Environment=PATH=/opt/left4me/.venv/bin:" in unit
-    assert "EnvironmentFile=/etc/left4me/host.env" in unit
-    assert "EnvironmentFile=/etc/left4me/web.env" in unit
-    assert "ExecStart=/opt/left4me/.venv/bin/gunicorn" in unit
-    assert "--workers 1" in unit
-    assert "--threads 32" in unit
-    # NoNewPrivileges must remain unset because sudo (used by the overlay,
-    # systemctl and journalctl helpers) is setuid.
-    assert "NoNewPrivileges=true" not in unit
-    # Restored now that fuse-overlayfs propagation is no longer the mechanism.
-    assert "PrivateTmp=true" in unit
-    assert "ProtectSystem=full" in unit
-    assert "ReadWritePaths=/var/lib/left4me" in unit
-    # Mounts now happen in PID 1's namespace via the left4me-overlay helper,
-    # so MountFlags propagation is irrelevant — and the previous assumption
-    # that MountFlags=shared made it work was incorrect.
-    assert "MountFlags=" not in unit
-
-
-def test_server_unit_contains_required_runtime_contract():
-    unit = SERVER_UNIT.read_text()
-
-    assert "User=left4me" in unit
-    assert "Group=left4me" in unit
-    assert "EnvironmentFile=/etc/left4me/host.env" in unit
-    assert "EnvironmentFile=/var/lib/left4me/instances/%i/instance.env" in unit
-    # `-` prefix: chdir failure is non-fatal so ExecStartPre can run the
-    # mount helper before the merged dir exists. ExecStart re-applies and
-    # finds the dir once the mount has landed.
-    assert "WorkingDirectory=-/var/lib/left4me/runtime/%i/merged/left4dead2" in unit
-    # ExecStart must invoke srcds_run from the *merged* overlay tree, not
-    # from installation/. srcds_run cds to its own dirname; if we point at
-    # installation/, the engine reads gameinfo.txt and addons from the lower
-    # layer and never sees overlay plugins (Metamod/SourceMod) or cfgs.
-    assert "ExecStart=/var/lib/left4me/runtime/%i/merged/srcds_run" in unit
-    assert "$L4D2_ARGS" in unit
-    assert "${L4D2_ARGS}" not in unit
-    assert "NoNewPrivileges=true" in unit
-    assert "PrivateTmp=true" in unit
-    assert "PrivateDevices=true" in unit
-    assert "ProtectHome=true" in unit
-    assert "ProtectSystem=strict" in unit
-    assert "ReadOnlyPaths=/var/lib/left4me/installation /var/lib/left4me/overlays" in unit
-    assert "ReadWritePaths=/var/lib/left4me/runtime/%i" in unit
-    assert "RestrictSUIDSGID=true" in unit
-    assert "LockPersonality=true" in unit
-
-
-def test_server_unit_mounts_overlay_via_exec_start_pre():
-    """At boot, systemd auto-starts enabled units before the web app gets a
-    chance to run start_instance's pre-start mount. The unit itself must
-    re-mount the overlay so reboots are transparent. Pairs with the helper's
-    idempotency check (test_overlay_helper_mount_is_idempotent_when_mounted).
-
-    The unit-level `nsenter --mount=/proc/1/ns/mnt --` is what makes
-    umount fast: without it, the helper Python process would inherit
-    the unit's per-service mount namespace and pin it alive, blocking
-    PID 1's umount until the helper exited. Wrapping with nsenter at
-    the Exec line puts the helper itself in PID 1's namespace.
-    """
-    unit = SERVER_UNIT.read_text()
-    # `+` prefix: runs as PID 1 (root, no sandbox). Required because
-    # the unit has NoNewPrivileges=true, which blocks sudo's setuid
-    # escalation — and the helper needs root for the mount syscall.
-    assert (
-        "ExecStartPre=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- "
-        "/usr/local/libexec/left4me/left4me-overlay mount %i"
-        in unit
-    )
-    # Bound the restart loop; without these, a CHDIR-failure (or any other
-    # pre-start error) spins indefinitely.
-    assert "StartLimitBurst=5" in unit
-    assert "StartLimitIntervalSec=60s" in unit
-
-
-def test_server_unit_unmounts_overlay_via_exec_stop_post():
-    """Single source of truth for unmount, mirroring the mount path.
-    ExecStopPost (not ExecStop) so it runs after srcds has fully exited
-    and the cgroup is cleared.
-
-    Same nsenter-at-Exec-line wrapping as ExecStartPre — without it,
-    the helper process would itself hold a reference to the unit's
-    per-service mount namespace, and umount in PID 1 would loop on
-    EBUSY until the helper gave up. With it, umount succeeds first try.
-    """
-    unit = SERVER_UNIT.read_text()
-    assert (
-        "ExecStopPost=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- "
-        "/usr/local/libexec/left4me/left4me-overlay umount %i"
-        in unit
-    )
-
-
-def test_overlay_helper_mount_is_idempotent_when_already_mounted():
-    """ExecStartPre runs on every Restart=on-failure cycle. If a previous
-    start mounted successfully but ExecStart failed afterwards, the next
-    ExecStartPre would re-mount on top -- which fails. The helper must
-    short-circuit when merged is already a mount point.
-    """
-    text = OVERLAY_HELPER.read_text()
-    # Two ismount checks now: one in cmd_mount (skip if mounted),
-    # one in cmd_umount (skip if not mounted).
-    assert text.count("os.path.ismount") >= 2
-
-
-def test_server_unit_contains_perf_baseline_directives():
-    unit = SERVER_UNIT.read_text()
-
-    # Slice membership.
-    assert "Slice=l4d2-game.slice" in unit
-
-    # CFS priority bump (no SCHED_FIFO).
-    assert "Nice=-5" in unit
-    assert "CPUSchedulingPolicy=" not in unit
-
-    # I/O priority.
-    assert "IOSchedulingClass=best-effort" in unit
-    assert "IOSchedulingPriority=4" in unit
-
-    # OOM ordering: game servers survive, sandbox dies first.
-    assert "OOMScoreAdjust=-200" in unit
-
-    # Memory caps with headroom for map-load spikes.
-    assert "MemoryHigh=1.5G" in unit
-    assert "MemoryMax=2G" in unit
-
-    # Bounded fork surface.
-    assert "TasksMax=256" in unit
-
-    # Plenty of fds for plugin-heavy setups.
-    assert "LimitNOFILE=65536" in unit
-
-    # srcds clean shutdown via SIGINT, with time to flush. With the
-    # helper running in PID 1's mount namespace (via the unit-level
-    # nsenter on ExecStopPost), umount has no race window and the
-    # default 15 s is plenty for the whole stop transition.
-    assert "KillSignal=SIGINT" in unit
-    assert "TimeoutStopSec=15s" in unit
-
-    # Per-unit override of journald rate limiting (default drops srcds output).
-    assert "LogRateLimitIntervalSec=0" in unit
-
-
-def test_l4d2_game_slice_exists_with_high_weights():
-    assert GAME_SLICE.is_file()
-    text = GAME_SLICE.read_text()
-    assert "[Slice]" in text
-    assert "CPUWeight=1000" in text
-    assert "IOWeight=1000" in text
-
-
-def test_l4d2_build_slice_exists_with_low_weights():
-    assert BUILD_SLICE.is_file()
-    text = BUILD_SLICE.read_text()
-    assert "[Slice]" in text
-    assert "CPUWeight=10" in text
-    assert "IOWeight=10" in text
-
-
-def test_sysctl_conf_present_with_perf_settings():
-    assert SYSCTL_CONF.is_file()
-    text = SYSCTL_CONF.read_text()
-    for line in (
-        "net.core.rmem_max = 8388608",
-        "net.core.wmem_max = 8388608",
-        "net.core.rmem_default = 524288",
-        "net.core.wmem_default = 524288",
-        "net.core.netdev_max_backlog = 5000",
-        "net.core.netdev_budget = 600",
-        "vm.swappiness = 10",
-        "net.ipv4.udp_rmem_min = 16384",
-        "net.ipv4.udp_wmem_min = 16384",
-        "net.core.default_qdisc = fq_codel",
-        "net.ipv4.tcp_congestion_control = bbr",
-    ):
-        assert line in text, f"missing {line!r} in 99-left4me.conf"
-
-
-def test_script_sandbox_in_build_slice_with_oom_adjust():
-    text = SCRIPT_SANDBOX_HELPER.read_text()
-
-    # Put the transient unit in the low-weight build slice so it yields to
-    # game-server instances under CPU/IO contention.
-    assert "--slice=l4d2-build.slice" in text
-
-    # Sandbox dies first if the host hits memory pressure; servers
-    # (OOMScoreAdjust=-200) survive.
-    assert "-p OOMScoreAdjust=500" in text
-
-
-def test_deploy_script_installs_perf_artifacts():
-    script = DEPLOY_SCRIPT.read_text()
-
-    # Slice files copied into the system-wide systemd unit dir.
-    assert "/usr/local/lib/systemd/system/l4d2-game.slice" in script
-    assert "/usr/local/lib/systemd/system/l4d2-build.slice" in script
-
-    # Sysctl drop-in installed under /etc/sysctl.d/.
-    assert "/etc/sysctl.d/99-left4me.conf" in script
-
-    # Values applied immediately, not on next boot.
-    assert "sysctl --system" in script
-
-
-def test_deploy_script_writes_cpuset_drop_ins():
-    script = DEPLOY_SCRIPT.read_text()
-
-    # Reads nproc and binds defaults via ${VAR:-...}.
-    assert "nproc" in script
-    assert "LEFT4ME_SYSTEM_CPUS" in script
-    assert "LEFT4ME_GAME_CPUS" in script
-    assert "${LEFT4ME_SYSTEM_CPUS:-0}" in script
-
-    # Default game-core upper bound is computed from nproc; accept either
-    # the NPROC-1 form or LEFT4ME_GAME_CPUS:-1- prefix.
-    assert (
-        "1-$((NPROC - 1))" in script
-        or "1-$((NPROC-1))" in script
-        or "1-$((nproc-1))" in script
-        or "LEFT4ME_GAME_CPUS:-1-" in script
-    )
-
-    # All four drop-in paths.
-    for slice_name in ("system", "user", "l4d2-build", "l4d2-game"):
-        assert (
-            f"/etc/systemd/system/{slice_name}.slice.d/99-left4me-cpuset.conf"
-            in script
-        )
-
-    # Drop-ins use the existing install pattern.
-    assert "install -m 0644 -o root -g root" in script
-
-    # Single-core host: skip with a warning to stderr.
-    assert ("-lt 2" in script) or ("< 2" in script) or ("-ge 2" in script)
-    assert "skipping CPU isolation" in script
-
-
-def _fake_command(tmp_path, command_name):
-    marker = tmp_path / f"{command_name}.args"
-    command = tmp_path / command_name
-    command.write_text(f"#!/bin/sh\nprintf '%s\n' \"$*\" > '{marker}'\nexit 0\n")
-    command.chmod(0o755)
-    return marker
-
-
-def _env_with_fake_commands(tmp_path):
-    env = os.environ.copy()
-    env["PATH"] = f"{tmp_path}{os.pathsep}{env.get('PATH', '')}"
-    return env
-
-
-def test_helpers_use_fixed_system_tool_paths_not_sudo_path():
-    systemctl = SYSTEMCTL_HELPER.read_text()
-    journalctl = JOURNALCTL_HELPER.read_text()
-
-    assert "command -v systemctl" not in systemctl
-    assert "command -v journalctl" not in journalctl
-    assert "/bin/systemctl" in systemctl or "/usr/bin/systemctl" in systemctl
-    assert "/bin/journalctl" in journalctl or "/usr/bin/journalctl" in journalctl
-
-
-def test_systemctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path):
-    subprocess.run(["sh", "-n", str(SYSTEMCTL_HELPER)], check=True)
-    marker = _fake_command(tmp_path, "systemctl")
-
-    for args in [
-        ["bad/action", "alpha"],
-        # `start` and `stop` are no longer accepted verbs — the lifecycle now
-        # uses `enable`/`disable` for reboot survival via WantedBy= symlinks.
-        ["start", "alpha"],
-        ["stop", "alpha"],
-        ["enable", ""],
-        ["enable", ".hidden"],
-        ["enable", "bad..name"],
-        ["enable", "bad/name"],
-        ["enable", "bad\\name"],
-        ["enable", "bad name"],
-    ]:
-        result = subprocess.run(["sh", str(SYSTEMCTL_HELPER), *args], env=_env_with_fake_commands(tmp_path), check=False)
-        assert result.returncode != 0
-        assert not marker.exists()
-
-    script = SYSTEMCTL_HELPER.read_text()
-    assert 'unit="left4me-server@${name}.service"' in script
-    assert 'enable) exec "$systemctl" enable --now "$unit"' in script
-    assert 'disable) exec "$systemctl" disable --now "$unit"' in script
-    assert "--property=ActiveState" in script
-    assert "--property=SubState" in script
-
-
-def test_journalctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path):
-    subprocess.run(["sh", "-n", str(JOURNALCTL_HELPER)], check=True)
-    marker = _fake_command(tmp_path, "journalctl")
-
-    for args in [
-        ["../evil", "--lines", "25", "--no-follow"],
-        ["alpha", "--bad", "25", "--no-follow"],
-        ["alpha", "--lines", "not-number", "--no-follow"],
-        ["alpha", "--lines", "25", "--bad-follow"],
-        ["bad/name", "--lines", "25", "--no-follow"],
-    ]:
-        result = subprocess.run(["sh", str(JOURNALCTL_HELPER), *args], env=_env_with_fake_commands(tmp_path), check=False)
-        assert result.returncode != 0
-        assert not marker.exists()
-
-    script = JOURNALCTL_HELPER.read_text()
-    assert 'unit="left4me-server@${name}.service"' in script
-    assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat "$follow_arg"' in script
-    assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat' in script
-
-
-def test_sudoers_allows_only_left4me_helpers_not_raw_system_tools():
-    sudoers = SUDOERS.read_text()
-
-    assert (
-        "left4me ALL=(root) NOPASSWD: "
-        "/usr/local/libexec/left4me/left4me-systemctl *"
-    ) in sudoers
-    assert (
-        "left4me ALL=(root) NOPASSWD: "
-        "/usr/local/libexec/left4me/left4me-journalctl *"
-    ) in sudoers
-    assert "/usr/local/libexec/left4me/left4me-overlay mount *" in sudoers
-    assert "/usr/local/libexec/left4me/left4me-overlay umount *" in sudoers
-    assert (
-        "left4me ALL=(root) NOPASSWD: "
-        "/usr/local/libexec/left4me/left4me-script-sandbox"
-    ) in sudoers
-    assert "/bin/systemctl" not in sudoers
-    assert "/usr/bin/systemctl" not in sudoers
-    assert "/bin/journalctl" not in sudoers
-    assert "/usr/bin/journalctl" not in sudoers
-    assert "/bin/mount" not in sudoers
-    assert "/bin/umount" not in sudoers
-
-
-def test_overlay_helper_is_python_with_strict_validation():
-    text = OVERLAY_HELPER.read_text()
-    assert text.startswith("#!/usr/bin/python3")
-    # Validation surface
-    assert "NAME_RE = re.compile" in text
-    assert "LOWERDIR_ALLOWLIST" in text
-    assert "user.fuseoverlayfs." in text
-    assert "MAX_LOWERDIRS = 500" in text
-    # Mounts via PID 1's mount namespace
-    assert "/proc/1/ns/mnt" in text
-    assert "nsenter" in text
-    # Verbs are mount and umount (not unmount)
-    assert '"mount"' in text and '"umount"' in text
-    assert '"unmount"' not in text
-
-
-def test_script_sandbox_uses_idmap_staging():
-    """The sandbox runs as l4d2-sandbox but writes need to land on disk as
-    left4me, so all overlay content (workshop + script-built) is uniformly
-    left4me-owned. The helper pre-creates an idmapped bind on a staging
-    path and points the sandbox's BindPaths at the staging, not at the raw
-    overlay dir. trap cleans up the staging bind on exit.
-    """
-    text = SCRIPT_SANDBOX_HELPER.read_text()
-    # Idmap mount setup uses --map-users / --map-groups.
-    assert "--map-users=" in text
-    assert "--map-groups=" in text
-    # Staging path lives under /var/lib/left4me/tmp/sandbox-idmap-<id>.
-    assert "/var/lib/left4me/tmp/sandbox-idmap-" in text
-    # BindPaths into the sandbox points at the staging path, not the
-    # raw overlay dir.
-    assert 'BindPaths="${STAGING}:/overlay"' in text
-    # trap registers cleanup so the staging bind doesn't outlive the helper.
-    assert "trap " in text and "cleanup_staging" in text
-    # The previous chown-to-l4d2-sandbox approach is gone; overlay dirs
-    # stay left4me-owned end-to-end.
-    assert "chown -R l4d2-sandbox" not in text
-
-
-def test_deploy_script_installs_overlay_helper_with_executable_mode():
-    script = DEPLOY_SCRIPT.read_text()
-    assert "/usr/local/libexec/left4me/left4me-overlay" in script
-    assert "chmod 0755" in script and "left4me-overlay" in script
-
-
-def test_deploy_script_does_not_install_fuse_overlayfs_apt_dep():
-    # fuse-overlayfs / fuse3 were the previous mount engine; kernel overlayfs
-    # replaces them. Comments in the migration block may legitimately mention
-    # the names, so scope this to the actual apt-get / dnf install lines.
-    install_lines = [
-        line for line in DEPLOY_SCRIPT.read_text().splitlines()
-        if ("apt-get install" in line or "dnf install" in line)
-    ]
-    assert install_lines, "expected at least one apt/dnf install line"
-    for line in install_lines:
-        assert "fuse-overlayfs" not in line, line
-        assert "fuse3" not in line, line
-
-
-def test_deploy_script_runs_one_shot_kernel_overlay_migration():
-    script = DEPLOY_SCRIPT.read_text()
-    assert "/var/lib/left4me/.kernel-overlay-migrated" in script
-    # Migration should stop services + force-unmount stale mounts + wipe upper/work
-    assert "systemctl stop 'left4me-server@" in script
-    assert "systemctl stop left4me-web.service" in script
-    assert "findmnt -t overlay" in script
-    assert "/runtime/" in script and "rm -rf" in script and 'upper"' in script and 'work"' in script
-
-
-def test_env_templates_contain_required_defaults():
-    host_env = HOST_ENV.read_text()
-    assert "Deployment units use fixed /var/lib/left4me paths" in host_env
-    assert host_env.endswith("LEFT4ME_ROOT=/var/lib/left4me\n")
-    web_env = WEB_ENV_TEMPLATE.read_text()
-    assert web_env.startswith(
-        "DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n"
-        "SECRET_KEY=replace-with-generated-secret\n"
-        "JOB_WORKER_THREADS=4\n"
-    )
-    assert web_env.rstrip().endswith("STEAM_WEB_API_KEY=")
-
-
-def test_deploy_script_has_safe_defaults_and_preserves_state() -> None:
-    script = DEPLOY_SCRIPT.read_text()
-
-    assert "useradd --system --home-dir /var/lib/left4me" in script
-    assert "/var/lib/left4me/installation" in script
-    assert "/var/lib/left4me/overlays" in script
-    assert "/var/lib/left4me/instances" in script
-    assert "/var/lib/left4me/runtime" in script
-    assert "tar" in script
-    assert "--exclude .venv" in script
-    assert "--exclude .claude" in script
-    assert "pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web" in script
-    assert "systemctl enable --now left4me-web.service" in script
-    assert "for attempt in" in script
-    assert "/opt/left4me/.venv" in script
-    assert "visudo -cf /etc/sudoers.d/left4me" in script
-    # Note: assertions about web.env's lifecycle (create-only-if-missing /
-    # never-sourced-from-deploy) used to live here. They became stale in
-    # commit caa8b83, which switched to "rewrite web.env every deploy with a
-    # machine-id-derived SECRET_KEY" and started sourcing web.env in the
-    # alembic + seed helper subprocesses. Removed entirely; current behavior
-    # is covered by `install -m 0640 ... /etc/left4me/web.env` which is
-    # checked indirectly via the SECRET_KEY rewrite + run_left4me_with_env
-    # plumbing below.
-    assert "run_left4me_with_env" in script
-    assert "LEFT4ME_ADMIN_USERNAME" in script
-    assert "LEFT4ME_ADMIN_PASSWORD" in script
-    assert "user already exists" in script
-    assert "deploy/files" in script
-
-
-def test_deploy_script_does_not_recurse_into_runtime_state_mounts() -> None:
-    script = DEPLOY_SCRIPT.read_text()
-
-    assert "$sudo_cmd chown -R left4me:left4me /var/lib/left4me" not in script
-    assert "$sudo_cmd chown left4me:left4me \\" in script
-    assert "/var/lib/left4me/runtime \\" in script
-    assert "$sudo_cmd chown -R left4me:left4me /opt/left4me" in script
-
-
-def test_deploy_script_runs_migrations_before_app_initialization() -> None:
-    script = DEPLOY_SCRIPT.read_text()
-
-    assert "alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head" in script
-    assert "from l4d2web.app import create_app; create_app()" not in script
-
-
-def test_deploy_script_shell_syntax() -> None:
-    subprocess.run(["sh", "-n", str(DEPLOY_SCRIPT)], check=True)
-
-
-def test_globals_refresh_units_removed():
-    """Global-overlays subsystem deleted in favor of script overlays."""
-    assert not GLOBAL_REFRESH_SERVICE.exists()
-    assert not GLOBAL_REFRESH_TIMER.exists()
-
-
-def test_deploy_script_does_not_provision_globals_subsystem():
-    script = DEPLOY_SCRIPT.read_text()
-
-    # No mkdir/install of the deleted cache dir; mention in a one-shot
-    # `rm -rf` cleanup is fine.
-    for line in script.splitlines():
-        if "/var/lib/left4me/global_overlay_cache" not in line:
-            continue
-        assert "rm -rf" in line, line
-    assert "left4me-refresh-global-overlays" not in script
-
-
-def test_deploy_script_provisions_sandbox_user():
-    script = DEPLOY_SCRIPT.read_text()
-    assert "useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox" in script
-
-
-def test_deploy_script_does_not_install_bubblewrap():
-    install_lines = [
-        line for line in DEPLOY_SCRIPT.read_text().splitlines()
-        if ("apt-get install" in line or "dnf install" in line)
-    ]
-    assert install_lines, "expected at least one apt/dnf install line"
-    for line in install_lines:
-        assert "bubblewrap" not in line, line
-        assert "bwrap" not in line, line
-
-
-def test_deploy_script_installs_script_overlay_tooling():
-    # Script overlays commonly need 7z and md5sum (e.g. l4d2center map sync).
-    # coreutils ships md5sum and is technically essential, but listing it
-    # explicitly makes the contract obvious and survives slim base images.
-    script = DEPLOY_SCRIPT.read_text().splitlines()
-    apt_lines = [l for l in script if "apt-get install" in l]
-    dnf_lines = [l for l in script if "dnf install" in l]
-    assert apt_lines, "expected an apt-get install line"
-    assert dnf_lines, "expected a dnf install line"
-    for line in apt_lines:
-        assert "p7zip-full" in line, line
-        assert "coreutils" in line, line
-    for line in dnf_lines:
-        # Fedora/RHEL split: p7zip provides 7za, p7zip-plugins provides 7z.
-        assert "p7zip" in line and "p7zip-plugins" in line, line
-        assert "coreutils" in line, line
-
-
-def test_deploy_script_tightens_left4me_db_permissions():
-    script = DEPLOY_SCRIPT.read_text()
-    # The DB and its WAL/SHM sidecars must be left4me:left4me 0640 — owner
-    # (web service) keeps rw, group is read-only, "other" (incl. l4d2-sandbox)
-    # gets nothing. The sidecars matter because SQLite in WAL mode requires
-    # write access to all three; if a sidecar ends up root-owned (e.g. from
-    # ad-hoc root-side inspection), the next write fails as "readonly db".
-    assert "chown left4me:left4me" in script
-    assert "chmod 0640" in script
-    for db_file in (
-        "/var/lib/left4me/left4me.db",
-        "/var/lib/left4me/left4me.db-wal",
-        "/var/lib/left4me/left4me.db-shm",
-    ):
-        assert db_file in script, f"deploy script must touch {db_file}"
-
-
-def test_deploy_script_installs_script_sandbox_helper():
-    script = DEPLOY_SCRIPT.read_text()
-    assert "/usr/local/libexec/left4me/left4me-script-sandbox" in script
-    assert "chmod 0755" in script and "left4me-script-sandbox" in script
-
-
-def test_script_sandbox_helper_present():
-    assert SCRIPT_SANDBOX_HELPER.is_file()
-    assert SCRIPT_SANDBOX_HELPER.read_text().startswith("#!/bin/bash")
-    mode = SCRIPT_SANDBOX_HELPER.stat().st_mode & 0o777
-    assert mode == 0o755, f"expected 0755, got {oct(mode)}"
-
-
-def test_script_sandbox_helper_passes_shell_syntax_check():
-    subprocess.run(["bash", "-n", str(SCRIPT_SANDBOX_HELPER)], check=True)
-
-
-def test_script_sandbox_helper_invokes_systemd_run_with_hardening():
-    text = SCRIPT_SANDBOX_HELPER.read_text()
-
-    # systemd-run service mode (no --scope), with synchronous I/O to caller.
-    assert "systemd-run" in text
-    assert "--scope" not in text, "v2 uses transient service units, not scopes"
-    assert "--pipe" in text
-    assert "--wait" in text
-    assert "--collect" in text
-    assert "--unit=" in text
-
-    # No bwrap.
-    assert "bwrap" not in text
-    assert "bubblewrap" not in text
-
-    # UID drop via systemd directives.
-    assert "User=l4d2-sandbox" in text
-    assert "Group=l4d2-sandbox" in text
-
-    # Cgroup limits unchanged from v1.
-    assert "MemoryMax=4G" in text
-    assert "MemorySwapMax=0" in text
-    assert "TasksMax=512" in text
-    assert "CPUQuota=200%" in text
-    assert "RuntimeMaxSec=3600" in text
-
-    # Hardening directives that v1 (scope mode) couldn't carry.
-    assert "NoNewPrivileges=yes" in text
-    assert "ProtectSystem=strict" in text
-    assert "ProtectHome=yes" in text
-    assert "PrivateTmp=yes" in text
-    assert "PrivateDevices=yes" in text
-    assert "PrivateIPC=yes" in text
-    assert "ProtectKernelTunables=yes" in text
-    assert "ProtectKernelModules=yes" in text
-    assert "ProtectKernelLogs=yes" in text
-    assert "ProtectControlGroups=yes" in text
-    assert "RestrictNamespaces=yes" in text
-    assert "RestrictSUIDSGID=yes" in text
-    assert "LockPersonality=yes" in text
-    assert "MemoryDenyWriteExecute=yes" in text
-    assert "SystemCallFilter=" in text
-    assert "@system-service" in text
-    assert "@network-io" in text
-    assert "CapabilityBoundingSet=" in text
-    assert "AmbientCapabilities=" in text
-    assert 'RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX"' in text
-
-    # Network namespace stays shared with host.
-    assert "PrivateNetwork=" not in text
-
-    # Mount setup: /etc and /var/lib masked with tmpfs; selective binds back.
-    assert 'TemporaryFileSystem="/etc /var/lib"' in text
-    assert "BindReadOnlyPaths=" in text
-    # The resolv.conf bind points at the sandbox-only file (not the host's
-    # /etc/resolv.conf, which typically references a private-IP DNS server
-    # that IPAddressDeny= blocks).
-    assert "/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf" in text
-    assert "/etc/ssl" in text
-    assert "/etc/ca-certificates" in text
-    assert "/etc/nsswitch.conf" in text
-    assert "/etc/alternatives" in text
-    assert "${SCRIPT}:/script.sh" in text
-    assert 'BindPaths="${STAGING}:/overlay"' in text
-
-    # IP egress filter: allow public, deny localhost / RFC1918 / link-local /
-    # multicast / CGNAT / ULA. systemd's "more specific rule wins" semantics
-    # mean public IPs hit the allow and listed ranges hit the deny.
-    # IPAddressDeny alone — no IPAddressAllow=any. Empirically, having both
-    # set causes the allow to win on this systemd/kernel combo regardless of
-    # the documented "more specific rule wins" behaviour. With only Deny,
-    # the kernel's default "allow all" applies to non-listed addresses.
-    assert "IPAddressDeny=" in text
-    assert "IPAddressAllow=any" not in text
-    # Explicit CIDRs — systemd-run's -p parser doesn't accept the
-    # `localhost` / `link-local` / `multicast` shorthand keywords that
-    # work in unit files (only the full strings parse).
-    for token in (
-        "127.0.0.0/8",
-        "::1/128",
-        "169.254.0.0/16",
-        "fe80::/10",
-        "224.0.0.0/4",
-        "ff00::/8",
-        "10.0.0.0/8",
-        "172.16.0.0/12",
-        "192.168.0.0/16",
-        "100.64.0.0/10",
-        "fc00::/7",
-    ):
-        assert token in text, f"missing {token!r} in IPAddressDeny set"
-
-
-def test_sandbox_resolv_conf_exists():
-    assert SANDBOX_RESOLV_CONF.is_file()
-    text = SANDBOX_RESOLV_CONF.read_text()
-    nameservers = [
-        line.split()[1]
-        for line in text.splitlines()
-        if line.startswith("nameserver ")
-    ]
-    assert len(nameservers) >= 2, "expected at least two nameservers for redundancy"
-    # Sanity: the resolvers must be public (not RFC1918 / loopback). We don't
-    # pin the exact IPs — Cloudflare/Google/Quad9 are all acceptable.
-    for ns in nameservers:
-        assert not ns.startswith("127."), ns
-        assert not ns.startswith("10."), ns
-        assert not ns.startswith("192.168."), ns
-        first_octet = int(ns.split(".")[0])
-        # Reject 172.16.0.0/12.
-        if first_octet == 172:
-            second_octet = int(ns.split(".")[1])
-            assert not (16 <= second_octet <= 31), ns
-
-
-def test_deploy_script_installs_sandbox_resolv_conf():
-    script = DEPLOY_SCRIPT.read_text()
-    assert "deploy/files/etc/left4me/sandbox-resolv.conf" in script
-    assert "/etc/left4me/sandbox-resolv.conf" in script
-
-
-def test_script_sandbox_helper_validates_overlay_id():
-    text = SCRIPT_SANDBOX_HELPER.read_text()
-    # Numeric-only overlay id
-    assert '[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]]' in text
-    # Overlay dir must exist
-    assert "/var/lib/left4me/overlays/" in text
-    assert "[[ -d $OVERLAY_DIR ]]" in text
-    # Script path must exist
-    assert "[[ -f $SCRIPT ]]" in text
-
-
-def test_script_sandbox_helper_dry_run_mode(tmp_path):
-    overlay_root = tmp_path / "var/lib/left4me/overlays/42"
-    overlay_root.mkdir(parents=True)
-    fake_script = tmp_path / "fake.sh"
-    fake_script.write_text("echo hi")
-
-    # Run in DRY_RUN mode against a fake l4d2-sandbox UID via a tiny shim that
-    # simulates `id -u l4d2-sandbox` resolving to a valid number.
-    helper_text = SCRIPT_SANDBOX_HELPER.read_text()
-    # We can't actually exec this without root + a real sandbox user; just
-    # verify the dry-run guard short-circuits before systemd-run / bwrap.
-    assert 'LEFT4ME_SCRIPT_SANDBOX_DRY_RUN' in helper_text
-    assert 'exit 0' in helper_text
-
-
-def test_nft_mark_file_marks_left4me_udp_with_dscp_ef_and_priority():
-    assert NFT_MARK_FILE.is_file()
-    text = NFT_MARK_FILE.read_text()
-
-    # Own table in the inet family so it cannot conflict with operator nftables config.
-    assert "table inet left4me_mark" in text
-    assert "chain mangle_output" in text
-    assert "type filter hook output priority mangle" in text
-
-    # Match by uid (every srcds runs as `left4me`) restricted to UDP.
-    assert 'meta skuid "left4me"' in text
-    assert "meta l4proto udp" in text
-
-    # DSCP EF for both L3 families; in `inet` tables, `ip` only fires on v4
-    # and `ip6` only on v6.
-    assert "ip dscp set ef" in text
-    assert "ip6 dscp set ef" in text
-
-    # skb->priority class 6:0, set inline alongside DSCP.
-    assert "meta priority set 0006:0000" in text
-
-
-def test_nft_mark_unit_loads_and_clears_left4me_table():
-    assert NFT_MARK_UNIT.is_file()
-    text = NFT_MARK_UNIT.read_text()
-
-    # Loads the rules early so the very first packet srcds emits is marked.
-    assert "After=network-pre.target" in text
-    assert "Before=network.target" in text
-    assert "Wants=network-pre.target" in text
-
-    # Oneshot lifecycle: load on start, drop on stop.
-    assert "Type=oneshot" in text
-    assert "RemainAfterExit=yes" in text
-    assert (
-        "ExecStart=/usr/sbin/nft -f /usr/local/lib/left4me/nft/left4me-mark.nft"
-        in text
-    )
-    assert "ExecStop=/usr/sbin/nft delete table inet left4me_mark" in text
-    assert "WantedBy=multi-user.target" in text
-
-
-def test_cake_env_template_documents_required_knobs():
-    assert CAKE_ENV.is_file()
-    text = CAKE_ENV.read_text()
-
-    # Both knobs are documented and present (commented OK; the deploy preserves
-    # operator edits, so the template must not bake in a wrong value).
-    assert "LEFT4ME_UPLINK_MBIT" in text
-    assert "LEFT4ME_UPLINK_IFACE" in text
-    # Empty defaults: shaper unit no-ops with a journal warning when unset.
-    assert "LEFT4ME_UPLINK_MBIT=" in text
-    assert "LEFT4ME_UPLINK_IFACE=" in text
-
-
-def test_apply_cake_helper_supports_apply_and_clear_modes():
-    assert APPLY_CAKE_HELPER.is_file()
-    text = APPLY_CAKE_HELPER.read_text()
-
-    assert text.startswith("#!/bin/sh")
-    # Both knobs are read from the env file.
-    assert "LEFT4ME_UPLINK_MBIT" in text
-    assert "LEFT4ME_UPLINK_IFACE" in text
-    assert ". /etc/left4me/cake.env" in text
-    # Iface fallback to default route.
-    assert "ip -4 route show default" in text
-    # Two modes; default to apply.
-    assert "mode=${1:-apply}" in text
-    assert 'apply)' in text and 'clear)' in text
-    # Apply: idempotent `tc qdisc replace` with the documented flags.
-    assert "tc qdisc replace" in text
-    assert "cake" in text
-    assert "bandwidth" in text
-    assert "internet" in text
-    assert "diffserv4" in text
-    assert "dual-dsthost" in text
-    # Clear: tolerates a missing qdisc.
-    assert "tc qdisc del" in text
-    assert "|| true" in text
-    # Fail-soft on missing config.
-    assert "LEFT4ME_UPLINK_MBIT unset" in text
-
-
-def test_apply_cake_helper_passes_shell_syntax_check():
-    subprocess.run(["sh", "-n", str(APPLY_CAKE_HELPER)], check=True)
-
-
-def test_cake_unit_runs_helper_in_apply_and_clear_modes():
-    assert CAKE_UNIT.is_file()
-    text = CAKE_UNIT.read_text()
-
-    assert "After=network-online.target" in text
-    assert "Wants=network-online.target" in text
-    assert "Type=oneshot" in text
-    assert "RemainAfterExit=yes" in text
-    # `-` prefix: missing env file is non-fatal (deploy ships one, but be safe).
-    assert "EnvironmentFile=-/etc/left4me/cake.env" in text
-    assert (
-        "ExecStart=/usr/local/libexec/left4me/left4me-apply-cake apply" in text
-    )
-    assert (
-        "ExecStop=/usr/local/libexec/left4me/left4me-apply-cake clear" in text
-    )
-    assert "WantedBy=multi-user.target" in text
-
-
-def test_deploy_script_installs_network_shaping_artifacts():
-    script = DEPLOY_SCRIPT.read_text()
-
-    # nftables: package install on both apt and dnf paths.
-    apt_lines = [l for l in script.splitlines() if "apt-get install" in l]
-    dnf_lines = [l for l in script.splitlines() if "dnf install" in l]
-    assert apt_lines and dnf_lines
-    for line in apt_lines:
-        assert "nftables" in line, line
-    for line in dnf_lines:
-        assert "nftables" in line, line
-
-    # nft rules + unit copied to system paths.
-    assert "/usr/local/lib/left4me/nft/left4me-mark.nft" in script
-    assert (
-        "/usr/local/lib/systemd/system/left4me-nft-mark.service" in script
-    )
-    assert "systemctl enable --now left4me-nft-mark.service" in script
-
-    # CAKE helper + unit copied; helper made executable.
-    assert "/usr/local/libexec/left4me/left4me-apply-cake" in script
-    assert (
-        "/usr/local/lib/systemd/system/left4me-cake.service" in script
-    )
-    assert "chmod 0755" in script and "left4me-apply-cake" in script
-    assert "systemctl enable --now left4me-cake.service" in script
-
-    # cake.env: copied only if absent (operator edits survive re-deploys).
-    assert "/etc/left4me/cake.env" in script
-    assert "[ -e /etc/left4me/cake.env ]" in script

deploy/tests/test_example_units.py

@@ -0,0 +1,233 @@
+"""Lockdown tests for the curated examples kept under `deploy/files/`.
+
+`deploy/` is reference material. The production units are emitted by
+ckn-bw's `systemd_units` reactor in `bundles/left4me/metadata.py`;
+when reactor output drifts intentionally, update these examples to match.
+"""
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[2]
+DEPLOY = ROOT / "deploy"
+
+
+WEB_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-web.service"
+SERVER_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-server@.service"
+GAME_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-game.slice"
+BUILD_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-build.slice"
+SYSCTL_CONF = DEPLOY / "files/etc/sysctl.d/99-left4me.conf"
+SANDBOX_RESOLV_CONF = DEPLOY / "files/etc/left4me/sandbox-resolv.conf"
+HOST_ENV = DEPLOY / "templates/etc/left4me/host.env"
+WEB_ENV_TEMPLATE = DEPLOY / "templates/etc/left4me/web.env.template"
+
+
+def test_global_unit_files_exist_at_product_level_paths():
+    assert WEB_UNIT.is_file()
+    assert SERVER_UNIT.is_file()
+
+
+def test_web_unit_contains_required_runtime_contract():
+    unit = WEB_UNIT.read_text()
+
+    assert "User=left4me" in unit
+    assert "Group=left4me" in unit
+    assert "WorkingDirectory=/opt/left4me" in unit
+    assert "Environment=PATH=/opt/left4me/.venv/bin:" in unit
+    assert "EnvironmentFile=/etc/left4me/host.env" in unit
+    assert "EnvironmentFile=/etc/left4me/web.env" in unit
+    assert "ExecStart=/opt/left4me/.venv/bin/gunicorn" in unit
+    assert "--workers 1" in unit
+    assert "--threads 32" in unit
+    # NoNewPrivileges must remain unset because sudo (used by the overlay,
+    # systemctl and journalctl helpers) is setuid.
+    assert "NoNewPrivileges=true" not in unit
+    # Restored now that fuse-overlayfs propagation is no longer the mechanism.
+    assert "PrivateTmp=true" in unit
+    assert "ProtectSystem=full" in unit
+    assert "ReadWritePaths=/var/lib/left4me" in unit
+    # Mounts now happen in PID 1's namespace via the left4me-overlay helper,
+    # so MountFlags propagation is irrelevant — and the previous assumption
+    # that MountFlags=shared made it work was incorrect.
+    assert "MountFlags=" not in unit
+
+
+def test_server_unit_contains_required_runtime_contract():
+    unit = SERVER_UNIT.read_text()
+
+    assert "User=left4me" in unit
+    assert "Group=left4me" in unit
+    assert "EnvironmentFile=/etc/left4me/host.env" in unit
+    assert "EnvironmentFile=/var/lib/left4me/instances/%i/instance.env" in unit
+    # `-` prefix: chdir failure is non-fatal so ExecStartPre can run the
+    # mount helper before the merged dir exists. ExecStart re-applies and
+    # finds the dir once the mount has landed.
+    assert "WorkingDirectory=-/var/lib/left4me/runtime/%i/merged/left4dead2" in unit
+    # ExecStart must invoke srcds_run from the *merged* overlay tree, not
+    # from installation/. srcds_run cds to its own dirname; if we point at
+    # installation/, the engine reads gameinfo.txt and addons from the lower
+    # layer and never sees overlay plugins (Metamod/SourceMod) or cfgs.
+    assert "ExecStart=/var/lib/left4me/runtime/%i/merged/srcds_run" in unit
+    assert "$L4D2_ARGS" in unit
+    assert "${L4D2_ARGS}" not in unit
+    assert "NoNewPrivileges=true" in unit
+    assert "PrivateTmp=true" in unit
+    assert "PrivateDevices=true" in unit
+    assert "ProtectHome=true" in unit
+    assert "ProtectSystem=strict" in unit
+    assert "ReadOnlyPaths=/var/lib/left4me/installation /var/lib/left4me/overlays" in unit
+    assert "ReadWritePaths=/var/lib/left4me/runtime/%i" in unit
+    assert "RestrictSUIDSGID=true" in unit
+    assert "LockPersonality=true" in unit
+
+
+def test_server_unit_mounts_overlay_via_exec_start_pre():
+    """At boot, systemd auto-starts enabled units before the web app gets a
+    chance to run start_instance's pre-start mount. The unit itself must
+    re-mount the overlay so reboots are transparent. Pairs with the helper's
+    idempotency check (test_overlay_helper_mount_is_idempotent_when_mounted).
+
+    The unit-level `nsenter --mount=/proc/1/ns/mnt --` is what makes
+    umount fast: without it, the helper Python process would inherit
+    the unit's per-service mount namespace and pin it alive, blocking
+    PID 1's umount until the helper exited. Wrapping with nsenter at
+    the Exec line puts the helper itself in PID 1's namespace.
+    """
+    unit = SERVER_UNIT.read_text()
+    # `+` prefix: runs as PID 1 (root, no sandbox). Required because
+    # the unit has NoNewPrivileges=true, which blocks sudo's setuid
+    # escalation — and the helper needs root for the mount syscall.
+    assert (
+        "ExecStartPre=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- "
+        "/usr/local/libexec/left4me/left4me-overlay mount %i"
+        in unit
+    )
+    # Bound the restart loop; without these, a CHDIR-failure (or any other
+    # pre-start error) spins indefinitely.
+    assert "StartLimitBurst=5" in unit
+    assert "StartLimitIntervalSec=60s" in unit
+
+
+def test_server_unit_unmounts_overlay_via_exec_stop_post():
+    """Single source of truth for unmount, mirroring the mount path.
+    ExecStopPost (not ExecStop) so it runs after srcds has fully exited
+    and the cgroup is cleared.
+
+    Same nsenter-at-Exec-line wrapping as ExecStartPre — without it,
+    the helper process would itself hold a reference to the unit's
+    per-service mount namespace, and umount in PID 1 would loop on
+    EBUSY until the helper gave up. With it, umount succeeds first try.
+    """
+    unit = SERVER_UNIT.read_text()
+    assert (
+        "ExecStopPost=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- "
+        "/usr/local/libexec/left4me/left4me-overlay umount %i"
+        in unit
+    )
+
+
+def test_server_unit_contains_perf_baseline_directives():
+    unit = SERVER_UNIT.read_text()
+
+    # Slice membership.
+    assert "Slice=l4d2-game.slice" in unit
+
+    # CFS priority bump (no SCHED_FIFO).
+    assert "Nice=-5" in unit
+    assert "CPUSchedulingPolicy=" not in unit
+
+    # I/O priority.
+    assert "IOSchedulingClass=best-effort" in unit
+    assert "IOSchedulingPriority=4" in unit
+
+    # OOM ordering: game servers survive, sandbox dies first.
+    assert "OOMScoreAdjust=-200" in unit
+
+    # Memory caps with headroom for map-load spikes.
+    assert "MemoryHigh=1.5G" in unit
+    assert "MemoryMax=2G" in unit
+
+    # Bounded fork surface.
+    assert "TasksMax=256" in unit
+
+    # Plenty of fds for plugin-heavy setups.
+    assert "LimitNOFILE=65536" in unit
+
+    # srcds clean shutdown via SIGINT, with time to flush. With the
+    # helper running in PID 1's mount namespace (via the unit-level
+    # nsenter on ExecStopPost), umount has no race window and the
+    # default 15 s is plenty for the whole stop transition.
+    assert "KillSignal=SIGINT" in unit
+    assert "TimeoutStopSec=15s" in unit
+
+    # Per-unit override of journald rate limiting (default drops srcds output).
+    assert "LogRateLimitIntervalSec=0" in unit
+
+
+def test_l4d2_game_slice_exists_with_high_weights():
+    assert GAME_SLICE.is_file()
+    text = GAME_SLICE.read_text()
+    assert "[Slice]" in text
+    assert "CPUWeight=1000" in text
+    assert "IOWeight=1000" in text
+
+
+def test_l4d2_build_slice_exists_with_low_weights():
+    assert BUILD_SLICE.is_file()
+    text = BUILD_SLICE.read_text()
+    assert "[Slice]" in text
+    assert "CPUWeight=10" in text
+    assert "IOWeight=10" in text
+
+
+def test_sysctl_conf_present_with_perf_settings():
+    assert SYSCTL_CONF.is_file()
+    text = SYSCTL_CONF.read_text()
+    for line in (
+        "net.core.rmem_max = 8388608",
+        "net.core.wmem_max = 8388608",
+        "net.core.rmem_default = 524288",
+        "net.core.wmem_default = 524288",
+        "net.core.netdev_max_backlog = 5000",
+        "net.core.netdev_budget = 600",
+        "vm.swappiness = 10",
+        "net.ipv4.udp_rmem_min = 16384",
+        "net.ipv4.udp_wmem_min = 16384",
+        "net.core.default_qdisc = fq_codel",
+        "net.ipv4.tcp_congestion_control = bbr",
+    ):
+        assert line in text, f"missing {line!r} in 99-left4me.conf"
+
+
+def test_env_templates_contain_required_defaults():
+    host_env = HOST_ENV.read_text()
+    assert "Deployment units use fixed /var/lib/left4me paths" in host_env
+    assert host_env.endswith("LEFT4ME_ROOT=/var/lib/left4me\n")
+    web_env = WEB_ENV_TEMPLATE.read_text()
+    assert web_env.startswith(
+        "DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n"
+        "SECRET_KEY=replace-with-generated-secret\n"
+        "JOB_WORKER_THREADS=4\n"
+    )
+    assert web_env.rstrip().endswith("STEAM_WEB_API_KEY=")
+
+
+def test_sandbox_resolv_conf_exists():
+    assert SANDBOX_RESOLV_CONF.is_file()
+    text = SANDBOX_RESOLV_CONF.read_text()
+    nameservers = [
+        line.split()[1]
+        for line in text.splitlines()
+        if line.startswith("nameserver ")
+    ]
+    assert len(nameservers) >= 2, "expected at least two nameservers for redundancy"
+    # Sanity: the resolvers must be public (not RFC1918 / loopback). We don't
+    # pin the exact IPs — Cloudflare/Google/Quad9 are all acceptable.
+    for ns in nameservers:
+        assert not ns.startswith("127."), ns
+        assert not ns.startswith("10."), ns
+        assert not ns.startswith("192.168."), ns
+        first_octet = int(ns.split(".")[0])
+        # Reject 172.16.0.0/12.
+        if first_octet == 172:
+            second_octet = int(ns.split(".")[1])
+            assert not (16 <= second_octet <= 31), ns

docs/superpowers/plans/2026-05-15-deploy-dir-rethink.md

@@ -0,0 +1,198 @@
+# Deploy-dir architecture rethink — implementation plan
+
+## Context
+
+Resolves the open questions in `docs/superpowers/specs/2026-05-15-deploy-dir-rethink-design.md`. After the 2026-05-15 script-consolidation work, `deploy/` ended up half-canonical / half-historical: the privileged scripts were treated as load-bearing source-of-truth there, while sudoers/sysctl/env-templates stayed duplicated against ckn-bw, and the obsolete `deploy-test-server.sh` plus a pile of dead static unit files lingered. The shape worked but couldn't be described in two sentences.
+
+This plan commits to the framing the user picked: **`deploy/` is a reference exemplar** — readable enough that a fresh consumer (ckn-bw today, hypothetical docker/ansible/manual tomorrow) could build a deployment from it, but not the live source of truth for installed binaries. The privileged scripts are **application-inherent code** and move out of `deploy/` to top-level `scripts/{libexec,sbin}/`. Dead code is deleted in the same pass. ckn-bw is updated to read scripts from the new location. The intended outcome: `deploy/` shrinks to README + example configs + a couple of curated example units, the rules for "what goes here" fit in two sentences, and the cross-repo install path becomes self-explanatory.
+
+## End state
+
+```
+left4me/
+  scripts/
+    libexec/
+      left4me-overlay              # 244-line Python helper (mount/umount)
+      left4me-script-sandbox       # 109-line bash (systemd-run sandbox)
+      left4me-systemctl            # 44-line sh wrapper
+      left4me-journalctl           # 53-line sh wrapper
+    sbin/
+      left4me                      # 17-line admin CLI wrapper
+    tests/
+      test_overlay.py
+      test_script_sandbox.py
+      test_systemctl_helper.py
+      test_journalctl_helper.py
+      test_sudoers_grants.py       # tests the contract between scripts and sudoers
+  deploy/                          # REFERENCE ONLY — see deploy/README.md
+    README.md                      # rewritten: explains target layout, points at scripts/
+    files/
+      etc/
+        sudoers.d/left4me                # example, ckn-bw ships its own verbatim copy
+        sysctl.d/99-left4me.conf         # example
+        left4me/sandbox-resolv.conf      # example
+      usr/local/lib/systemd/system/
+        left4me-server@.service          # curated example of what ckn-bw's reactor emits
+        left4me-web.service              # curated example
+        left4me-workshop-refresh.service # curated example
+        left4me-workshop-refresh.timer   # curated example
+        l4d2-game.slice                  # curated example
+        l4d2-build.slice                 # curated example
+    templates/etc/left4me/
+      host.env                           # example, ckn-bw renders its own mako version
+      web.env.template
+    tests/
+      test_example_units.py              # slimmed: just locks down the curated examples
+  l4d2host/                        # unchanged
+  l4d2web/                         # unchanged
+  docs/
+```
+
+## Step-by-step
+
+### 1. Create `scripts/` and move helpers
+
+- `mkdir -p scripts/libexec scripts/sbin scripts/tests`
+- `git mv` the four live helpers and the admin CLI:
+  - `deploy/files/usr/local/libexec/left4me/left4me-overlay` → `scripts/libexec/left4me-overlay`
+  - `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` → `scripts/libexec/left4me-script-sandbox`
+  - `deploy/files/usr/local/libexec/left4me/left4me-systemctl` → `scripts/libexec/left4me-systemctl`
+  - `deploy/files/usr/local/libexec/left4me/left4me-journalctl` → `scripts/libexec/left4me-journalctl`
+  - `deploy/files/usr/local/sbin/left4me` → `scripts/sbin/left4me`
+- The scripts' contents are unchanged. Every install-target path inside them (`/usr/local/libexec/left4me/...`, `/etc/left4me/...`, `/var/lib/left4me/...`) stays exactly as is — those are runtime paths, not source-tree paths.
+
+### 2. Delete dead code
+
+- `git rm` (truly obsolete; replacements live elsewhere or feature was retired):
+  - `deploy/files/usr/local/libexec/left4me/left4me-apply-cake` — CAKE migrated to systemd-networkd via `network/<iface>/cake` node metadata in ckn-bw.
+  - `deploy/files/usr/local/lib/systemd/system/left4me-cake.service` — same reason.
+  - `deploy/files/etc/left4me/cake.env` — bandwidth lives in node metadata, not an env file.
+  - `deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service` — central `bundles/nftables/` consumes the rules now.
+  - `deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft` — same. After the delete, the now-empty `deploy/files/usr/local/lib/left4me/` and its `nft/` child disappear (git doesn't track empty dirs).
+  - `deploy/deploy-test-server.sh` — superseded by `bw apply`; content survives in git history.
+- **Do NOT delete** `deploy/files/usr/local/lib/systemd/system/left4me-workshop-refresh.{service,timer}`. The workshop-refresh job is live (invokes `flask workshop-refresh`, defined in `l4d2web/cli.py`); ckn-bw's reactor emits these on production. They stay as curated examples, same category as `left4me-server@.service` / `left4me-web.service` / the slices. (This corrects the framing in `docs/superpowers/specs/2026-05-15-deploy-dir-rethink-design.md` and item 2 of `docs/superpowers/specs/2026-05-15-janitorial-cleanup.md`, both of which lumped workshop-refresh together with truly-dead units.)
+- Stale `__pycache__` dirs under `deploy/files/usr/local/libexec/left4me/` are deleted by the moves in step 1.
+
+### 3. Split and relocate `deploy/tests/test_deploy_artifacts.py`
+
+The current file (~880 lines) is doing four jobs. Split as follows; do not duplicate tests across files.
+
+**Concrete sequence to preserve git history where it counts**:
+
+1. `git mv deploy/tests/test_deploy_artifacts.py deploy/tests/test_example_units.py` — single rename, history follows via `git log --follow`.
+2. In the renamed file, delete every test except the "Keep in `deploy/tests/test_example_units.py`" list below. The kept tests track the unit/sysctl/env-template examples, which is what `deploy/tests/` will mean afterwards.
+3. Create new `scripts/tests/*.py` files (and `conftest.py`) by writing them fresh — pasting the relevant test functions across. The extracted tests lose direct rename history, but blame against the new files still resolves to the originals one git ref back; acceptable tradeoff.
+
+**Move to `scripts/tests/`** (tests of script behavior + the sudoers contract that gates the scripts):
+
+- `scripts/tests/test_overlay.py` — `test_overlay_helper_is_python_with_strict_validation`, `test_overlay_helper_mount_is_idempotent_when_already_mounted`
+- `scripts/tests/test_script_sandbox.py` — `test_script_sandbox_helper_present`, `test_script_sandbox_helper_passes_shell_syntax_check`, `test_script_sandbox_helper_invokes_systemd_run_with_hardening`, `test_script_sandbox_uses_idmap_staging`, `test_script_sandbox_in_build_slice_with_oom_adjust`, `test_script_sandbox_helper_validates_overlay_id`, `test_script_sandbox_helper_dry_run_mode`
+- `scripts/tests/test_systemctl_helper.py` — `test_systemctl_helper_passes_shell_syntax_check_and_rejects_bad_args`
+- `scripts/tests/test_journalctl_helper.py` — `test_journalctl_helper_passes_shell_syntax_check_and_rejects_bad_args`
+- `scripts/tests/test_helpers_use_fixed_paths.py` — `test_helpers_use_fixed_system_tool_paths_not_sudo_path`
+- `scripts/tests/test_sudoers_grants.py` — `test_sudoers_allows_only_left4me_helpers_not_raw_system_tools` (still reads `deploy/files/etc/sudoers.d/left4me` as the canonical example; comment why)
+
+The `ROOT/DEPLOY` path-prefix constants in each file get rewritten so `SCRIPTS = Path(__file__).resolve().parents[2] / "scripts"` and helpers resolve to `SCRIPTS / "libexec/left4me-overlay"` etc. Shared helpers (`_fake_command`, `_env_with_fake_commands`) move into `scripts/tests/conftest.py`.
+
+**Keep in `deploy/tests/test_example_units.py`** (locks down the curated examples; renamed from the current file):
+
+- `test_global_unit_files_exist_at_product_level_paths`
+- `test_web_unit_contains_required_runtime_contract`
+- `test_server_unit_contains_required_runtime_contract`
+- `test_server_unit_mounts_overlay_via_exec_start_pre`
+- `test_server_unit_unmounts_overlay_via_exec_stop_post`
+- `test_server_unit_contains_perf_baseline_directives`
+- `test_l4d2_game_slice_exists_with_high_weights`
+- `test_l4d2_build_slice_exists_with_low_weights`
+- `test_sysctl_conf_present_with_perf_settings`
+- `test_env_templates_contain_required_defaults`
+- `test_sandbox_resolv_conf_exists`
+
+Add a top-of-file docstring: *"These tests lock down the curated examples kept in `deploy/files/` for reference. The production units are emitted by ckn-bw's reactor in `bundles/left4me/metadata.py`; when reactor output drifts intentionally, update the examples here too."*
+
+**Delete entirely** (target removed or no longer load-bearing):
+
+- All `test_deploy_script_*` tests (12 tests; `deploy-test-server.sh` is gone)
+- `test_globals_refresh_units_removed` — file already deleted; nothing to lock down
+- `test_nft_mark_file_marks_left4me_udp_with_dscp_ef_and_priority`, `test_nft_mark_unit_loads_and_clears_left4me_table` — nft-mark moved to central nftables bundle
+- `test_cake_env_template_documents_required_knobs`, `test_apply_cake_helper_supports_apply_and_clear_modes`, `test_apply_cake_helper_passes_shell_syntax_check`, `test_cake_unit_runs_helper_in_apply_and_clear_modes` — CAKE moved to systemd-networkd
+- `test_deploy_script_installs_overlay_helper_with_executable_mode`, `test_deploy_script_installs_script_sandbox_helper` — install responsibility now lives in ckn-bw's bundle, not in any left4me-side script
+
+Final file count: `scripts/tests/` gets 6 files, `deploy/tests/test_example_units.py` is one file, `deploy/tests/test_deploy_artifacts.py` is gone (renamed).
+
+### 4. Rewrite `deploy/README.md`
+
+Reframe the top of the file as: *"This directory is a reference exemplar. The canonical deploy is [ckn-bw](https://git.sublimity.de/cronekorkn/ckn-bw)'s `bundles/left4me/` (run `bw apply ovh.left4me`). Files under `deploy/files/` and `deploy/templates/` are readable examples — not the binaries / configs ckn-bw actually installs. Read them to understand the target layout if you're building a fresh deployment by other means."*
+
+Update the file/status table:
+
+- Drop rows for files that no longer exist (apply-cake, cake.service, cake.env, nft-mark.*, workshop-refresh.*).
+- Drop the `deploy-test-server.sh` row.
+- For the privileged-scripts rows, change `files/usr/local/libexec/left4me/...` → `(moved to scripts/libexec/, installed by ckn-bw's install_left4me_scripts action)`; same for the sbin row.
+- Mark the remaining `files/etc/...` and `files/usr/local/lib/systemd/system/...` entries explicitly as **example**: ckn-bw ships its own verbatim copies of the configs, its reactor emits the units.
+
+Keep the "Target Layout" / "Runtime User" / "Overlay References" / "Performance Tuning" sections — they're useful reference prose. Strip the "Running A Test Deployment" / "Admin Bootstrap" sections that refer to the deleted shell installer; replace with a one-paragraph pointer to ckn-bw.
+
+### 5. ckn-bw cross-repo update
+
+The `install_left4me_scripts` action in `bundles/left4me/items.py` currently reads from `/opt/left4me/src/deploy/files/usr/local/{libexec,sbin}/`. Update it to read from `/opt/left4me/src/scripts/{libexec,sbin}/`. The install target is unchanged (`/usr/local/libexec/left4me/`, `/usr/local/sbin/left4me`), so nothing on the deployed host moves.
+
+This is a separate PR in the ckn-bw repo. It must land **at the same time** as the left4me move — the install action depends on the source paths existing. Coordination:
+
+1. Open both PRs simultaneously.
+2. Merge order: left4me first (scripts exist at the new path in `/opt/left4me/src/` only after a fresh `git_deploy`), then ckn-bw, then `bw apply ovh.left4me`.
+3. Alternative: have the ckn-bw PR fall back to the old path if the new path doesn't exist (one extra glob); decide during ckn-bw review whether the complexity is worth the looser coupling. Default: no fallback, coordinate the merges.
+
+Verification on the deploy target: after `bw apply`, the files under `/usr/local/libexec/left4me/` and `/usr/local/sbin/left4me` should be byte-identical to before. Sudoers, services, the web app: all unchanged.
+
+### 6. Mark adjacent specs / docs as resolved
+
+- `docs/superpowers/specs/2026-05-15-deploy-dir-rethink-design.md`: prepend a `**Resolved 2026-05-15 by docs/superpowers/plans/…</plan-name>.md.**` line at the top. Leave the body intact for archaeology.
+- `docs/superpowers/specs/2026-05-15-janitorial-cleanup.md`: cross out items 1, 5, 6 (now handled here). Item 2 needs a rewrite — the framing "all static unit files are obsolete drift" was wrong; the live reactor-emitted set (`server@`, `web`, `workshop-refresh.{service,timer}`, `l4d2-{game,build}.slice`) stays in `deploy/files/` as curated examples. The truly-dead two (`left4me-cake.service`, `left4me-nft-mark.service`) are already deleted by this plan, so item 2 collapses to "no remaining work."
+- No memory file changes needed; the project state captured here is structural and re-derivable from `deploy/README.md` after the rewrite lands.
+
+### 7. Rollback notes
+
+If `bw apply ovh.left4me` against the test server breaks something after the cross-repo merge:
+
+1. Revert the ckn-bw `install_left4me_scripts` action change to the old source path (`/opt/left4me/src/deploy/files/usr/local/{libexec,sbin}/`). Re-apply.
+2. The left4me side never needs reverting in isolation — the scripts at the new path are byte-identical to the old ones, so a stale ckn-bw install action against a *new* left4me checkout would fail at `install -t` (source path missing). That failure is loud and safe: nothing on the deployed system gets modified.
+3. The only foot-gun is **partial rollout**: ckn-bw updated but left4me not yet checked out at the right revision. The `git_deploy` step pins the revision, so as long as the two PRs reference compatible commits, the deployed `/opt/left4me/src/` always matches the action's expectation.
+
+## What does NOT change
+
+- Runtime install-target paths (`/usr/local/libexec/left4me/...`, `/usr/local/sbin/left4me`) — every reference inside `l4d2host/service_control.py:7-8`, `l4d2web/services/overlay_builders.py:34`, the sudoers file, and the systemd units stays the same.
+- The Python packages `l4d2host/` and `l4d2web/`.
+- ckn-bw's bundles for sudoers / sysctl / sandbox-resolv.conf — those keep their own verbatim copies (the user picked "deploy/ keeps configs as examples; duplication-with-ckn-bw is OK because deploy/ is explicitly reference"). Janitoring the duplication is *not* in scope for this plan.
+- The Mako env templates in ckn-bw — they stay where they are, since they need bw's metadata access for rendering.
+- The recent overlay-idmap / script-sandbox idmap-staging work — untouched.
+
+## Critical files (jump points for the implementor)
+
+- `deploy/tests/test_deploy_artifacts.py` — the source for the test split (lines 20-32 are the path constants; tests grouped roughly by helper from line 138 onward)
+- `deploy/README.md` — full rewrite of the top section, partial rewrite of the table
+- `l4d2host/service_control.py:7-8` — verify install-target paths unchanged (sanity)
+- `l4d2web/services/overlay_builders.py:34` — same
+- `deploy/files/etc/sudoers.d/left4me` — sanity-check that no path inside changed
+- `deploy/files/usr/local/lib/systemd/system/{left4me-server@.service,left4me-web.service,l4d2-{game,build}.slice}` — survive as curated examples
+- ckn-bw repo: `bundles/left4me/items.py` — the `install_left4me_scripts` action (separate PR)
+
+## Verification
+
+End-to-end:
+
+1. **Source-tree consistency.** `find scripts deploy -type f | sort` matches the layout in "End state" above (modulo `__pycache__`).
+2. **All tests pass locally.** From the repo root: `pytest scripts/tests/ deploy/tests/ l4d2host/tests/ l4d2web/tests/` — every test passes. Specifically verify `scripts/tests/test_sudoers_grants.py` still reads `deploy/files/etc/sudoers.d/left4me` correctly (path constant points across the dir boundary).
+3. **Shell syntax checks.** The split tests should still run `sh -n` / `bash -n` against the moved scripts; no script edits means no syntax regressions, but the test paths must resolve.
+4. **No accidental application breakage.** `grep -rn '/usr/local/libexec/left4me\|/usr/local/sbin/left4me' l4d2host l4d2web` returns the same hits as before (paths are install-target, source moves don't affect them).
+5. **ckn-bw dry-run.** Once the ckn-bw PR is up, `bw apply --dry-run ovh.left4me` from the ckn-bw repo: the diff should show **no changes** to files under `/usr/local/libexec/left4me/` or `/usr/local/sbin/left4me` (byte-identical content via the new path).
+6. **Production apply.** `bw apply ovh.left4me` against the real test server. After apply: `systemctl status left4me-web.service` is green, starting a game server via the web UI still works (overlay mount → srcds_run → unmount on stop), running an overlay build script through the sandbox still works.
+
+## Out of scope (handled elsewhere or deferred)
+
+- The Mako template duplication in ckn-bw — separate cleanup; the templates legitimately need bw's metadata access.
+- The 1/2/3-user uid-split decision — `docs/superpowers/specs/2026-05-15-user-uid-split-design.md`.
+- The script-sandbox → systemd template unit refactor — `docs/superpowers/specs/2026-05-15-build-overlay-unit-design.md`.
+- Remaining janitorial items: item 3 (bubblewrap→systemd-run doc drift), item 4 (stale gameserver-side idmap binds), calendar reminder for SM 1.13 stable. Items 1, 2 (partial — see step 6), 5, 6 are subsumed here.
+- Rewriting the shell helpers in Python / packaging them as console_scripts — explicitly rejected in the recent script-consolidation plan (egg-info + TOCTOU privilege concerns).
+- Historical references inside `docs/superpowers/plans/*` and `docs/superpowers/specs/*` to `deploy/files/...` or `deploy-test-server.sh` paths. Those are time-stamped snapshots of past sessions; they don't get rewritten when the underlying tree moves.

docs/superpowers/specs/2026-05-15-deploy-dir-rethink-design.md

@@ -1,5 +1,15 @@
 # Deploy directory architecture — open questions
 
+**Resolved 2026-05-15 by [`docs/superpowers/plans/2026-05-15-deploy-dir-rethink.md`](../plans/2026-05-15-deploy-dir-rethink.md).**
+Decision summary: `deploy/` is reference material; privileged scripts moved
+to top-level `scripts/{libexec,sbin}/`; `deploy-test-server.sh` deleted;
+dead static units (cake.service, nft-mark.service) deleted; reactor-emitted
+units (server@, web, workshop-refresh.{service,timer}, slices) retained as
+curated examples; ckn-bw `install_left4me_scripts` action repointed to the
+new source paths. Body below preserved for archaeology.
+
+---
+
 **Status: open questions, not a settled design.** This is a thinking-aloud
 handoff prompted by the script-consolidation change on 2026-05-15. Decisions
 deferred; a future session should pick this up, talk through the options,

docs/superpowers/specs/2026-05-15-janitorial-cleanup.md

@@ -7,9 +7,18 @@ self-contained. Knock them out individually or batch them into a
 single janitorial PR. None are urgent — the project works fine with
 all of these still present.
 
+> **2026-05-15 update**: items 1, 3, 4, and 5 resolved by
+> [`docs/superpowers/plans/2026-05-15-deploy-dir-rethink.md`](../plans/2026-05-15-deploy-dir-rethink.md).
+> Item 2 partially resolved by the same plan with a third option the
+> original enumeration didn't list: the truly-dead units (cake.service,
+> nft-mark.service) are deleted, the reactor-emitted set (server@, web,
+> workshop-refresh.{service,timer}, slices) stays as curated examples
+> under `deploy/files/`. Resolved items left in place below, marked
+> RESOLVED, for archaeology. Remaining live items: 6, 7, 8, 9, 10.
+
 ## Items
 
-### 1. `left4me-apply-cake` — dead code
+### 1. `left4me-apply-cake` — dead code [RESOLVED]
 
 **What**: `deploy/files/usr/local/libexec/left4me/left4me-apply-cake`
 (POSIX sh, ~47 lines) that applies/clears CAKE egress traffic
@@ -34,7 +43,19 @@ sudo find /var/lib/left4me /opt/left4me /usr/local -name 'left4me-apply-cake'
 # expect: empty after the rm
 ```
 
-### 2. Obsolete systemd unit files in `deploy/files/`
+### 2. Obsolete systemd unit files in `deploy/files/` [PARTIALLY RESOLVED]
+
+**Resolution path chosen**: third option not in the original enumeration —
+*only the truly-dead two* (`left4me-cake.service`, `left4me-nft-mark.service`)
+were deleted. The reactor-emitted set (`left4me-server@.service`,
+`left4me-web.service`, `left4me-workshop-refresh.{service,timer}`,
+`l4d2-game.slice`, `l4d2-build.slice`) is retained as **curated examples**
+under `deploy/files/`, locked down by `deploy/tests/test_example_units.py`.
+The framing in this item — "all six are equally drift" — was wrong: the
+reactor-emitted units carry useful signal as readable examples of what
+ckn-bw's `systemd_units` reactor emits at apply time. Original body below.
+
+
 
 **What**:
 - `deploy/files/usr/local/lib/systemd/system/left4me-cake.service`
@@ -65,7 +86,7 @@ they matter).
 **Verification**: `find deploy/files/usr/local/lib/systemd/system -type f`
 should match the README's "what's canonical" list.
 
-### 3. `deploy/files/etc/left4me/cake.env`
+### 3. `deploy/files/etc/left4me/cake.env` [RESOLVED]
 
 **What**: env file referenced by the obsolete `left4me-cake.service`.
 
@@ -75,7 +96,7 @@ read by anything live.
 
 **Action**: delete `deploy/files/etc/left4me/cake.env`.
 
-### 4. `deploy/files/usr/local/lib/left4me/nft/`
+### 4. `deploy/files/usr/local/lib/left4me/nft/` [RESOLVED]
 
 **What**: nftables fragment for `left4me-nft-mark.service`.
 
@@ -86,7 +107,11 @@ fragment isn't read.
 **Action**: delete `deploy/files/usr/local/lib/left4me/`
 recursively.
 
-### 5. `deploy-test-server.sh`'s fate
+### 5. `deploy-test-server.sh`'s fate [RESOLVED]
+
+**Resolution**: deleted entirely. Content survives in git history.
+
+
 
 **What**: `deploy/deploy-test-server.sh`, the historical one-shot
 bash deploy.

l4d2host/tests/test_overlay_helper.py

@@ -9,12 +9,8 @@ import pytest
 
 HELPER_SOURCE = (
     Path(__file__).resolve().parents[2]
-    / "deploy"
+    / "scripts"
-    / "files"
-    / "usr"
-    / "local"
     / "libexec"
-    / "left4me"
     / "left4me-overlay"
 )
 

scripts/tests/conftest.py

@@ -0,0 +1,36 @@
+"""Shared fixtures and path constants for `scripts/tests/`."""
+import os
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[2]
+SCRIPTS = ROOT / "scripts"
+LIBEXEC = SCRIPTS / "libexec"
+SBIN = SCRIPTS / "sbin"
+
+# `deploy/` is reference material; the sudoers example lives there and is
+# the canonical statement of which paths sudo grants to the `left4me` uid.
+# `scripts/tests/test_sudoers_grants.py` reads it from across the dir
+# boundary because the contract being audited is about *scripts*, not deploy.
+DEPLOY = ROOT / "deploy"
+
+
+def fake_command(tmp_path, command_name):
+    """Drop a no-op stub of `command_name` into `tmp_path`. Returns the
+    marker file the stub writes its args to, so tests can assert that the
+    helper rejected bad input before invoking the real command.
+    """
+    marker = tmp_path / f"{command_name}.args"
+    command = tmp_path / command_name
+    command.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" > '{marker}'\nexit 0\n")
+    command.chmod(0o755)
+    return marker
+
+
+def env_with_fake_commands(tmp_path):
+    """Build an environment that prepends `tmp_path` onto PATH so helpers
+    find the fake commands first.
+    """
+    env = os.environ.copy()
+    env["PATH"] = f"{tmp_path}{os.pathsep}{env.get('PATH', '')}"
+    return env

scripts/tests/test_helpers_use_fixed_paths.py

@@ -0,0 +1,15 @@
+from conftest import LIBEXEC
+
+
+SYSTEMCTL_HELPER = LIBEXEC / "left4me-systemctl"
+JOURNALCTL_HELPER = LIBEXEC / "left4me-journalctl"
+
+
+def test_helpers_use_fixed_system_tool_paths_not_sudo_path():
+    systemctl = SYSTEMCTL_HELPER.read_text()
+    journalctl = JOURNALCTL_HELPER.read_text()
+
+    assert "command -v systemctl" not in systemctl
+    assert "command -v journalctl" not in journalctl
+    assert "/bin/systemctl" in systemctl or "/usr/bin/systemctl" in systemctl
+    assert "/bin/journalctl" in journalctl or "/usr/bin/journalctl" in journalctl

scripts/tests/test_journalctl_helper.py

@@ -0,0 +1,31 @@
+import subprocess
+
+from conftest import LIBEXEC, env_with_fake_commands, fake_command
+
+
+JOURNALCTL_HELPER = LIBEXEC / "left4me-journalctl"
+
+
+def test_journalctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path):
+    subprocess.run(["sh", "-n", str(JOURNALCTL_HELPER)], check=True)
+    marker = fake_command(tmp_path, "journalctl")
+
+    for args in [
+        ["../evil", "--lines", "25", "--no-follow"],
+        ["alpha", "--bad", "25", "--no-follow"],
+        ["alpha", "--lines", "not-number", "--no-follow"],
+        ["alpha", "--lines", "25", "--bad-follow"],
+        ["bad/name", "--lines", "25", "--no-follow"],
+    ]:
+        result = subprocess.run(
+            ["sh", str(JOURNALCTL_HELPER), *args],
+            env=env_with_fake_commands(tmp_path),
+            check=False,
+        )
+        assert result.returncode != 0
+        assert not marker.exists()
+
+    script = JOURNALCTL_HELPER.read_text()
+    assert 'unit="left4me-server@${name}.service"' in script
+    assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat "$follow_arg"' in script
+    assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat' in script

scripts/tests/test_overlay.py

@@ -0,0 +1,32 @@
+from conftest import LIBEXEC
+
+
+OVERLAY_HELPER = LIBEXEC / "left4me-overlay"
+
+
+def test_overlay_helper_is_python_with_strict_validation():
+    text = OVERLAY_HELPER.read_text()
+    assert text.startswith("#!/usr/bin/python3")
+    # Validation surface
+    assert "NAME_RE = re.compile" in text
+    assert "LOWERDIR_ALLOWLIST" in text
+    assert "user.fuseoverlayfs." in text
+    assert "MAX_LOWERDIRS = 500" in text
+    # Mounts via PID 1's mount namespace
+    assert "/proc/1/ns/mnt" in text
+    assert "nsenter" in text
+    # Verbs are mount and umount (not unmount)
+    assert '"mount"' in text and '"umount"' in text
+    assert '"unmount"' not in text
+
+
+def test_overlay_helper_mount_is_idempotent_when_already_mounted():
+    """ExecStartPre runs on every Restart=on-failure cycle. If a previous
+    start mounted successfully but ExecStart failed afterwards, the next
+    ExecStartPre would re-mount on top -- which fails. The helper must
+    short-circuit when merged is already a mount point.
+    """
+    text = OVERLAY_HELPER.read_text()
+    # Two ismount checks now: one in cmd_mount (skip if mounted),
+    # one in cmd_umount (skip if not mounted).
+    assert text.count("os.path.ismount") >= 2

scripts/tests/test_script_sandbox.py

@@ -0,0 +1,171 @@
+import subprocess
+
+from conftest import LIBEXEC
+
+
+SCRIPT_SANDBOX_HELPER = LIBEXEC / "left4me-script-sandbox"
+
+
+def test_script_sandbox_helper_present():
+    assert SCRIPT_SANDBOX_HELPER.is_file()
+    assert SCRIPT_SANDBOX_HELPER.read_text().startswith("#!/bin/bash")
+    mode = SCRIPT_SANDBOX_HELPER.stat().st_mode & 0o777
+    assert mode == 0o755, f"expected 0755, got {oct(mode)}"
+
+
+def test_script_sandbox_helper_passes_shell_syntax_check():
+    subprocess.run(["bash", "-n", str(SCRIPT_SANDBOX_HELPER)], check=True)
+
+
+def test_script_sandbox_helper_invokes_systemd_run_with_hardening():
+    text = SCRIPT_SANDBOX_HELPER.read_text()
+
+    # systemd-run service mode (no --scope), with synchronous I/O to caller.
+    assert "systemd-run" in text
+    assert "--scope" not in text, "v2 uses transient service units, not scopes"
+    assert "--pipe" in text
+    assert "--wait" in text
+    assert "--collect" in text
+    assert "--unit=" in text
+
+    # No bwrap.
+    assert "bwrap" not in text
+    assert "bubblewrap" not in text
+
+    # UID drop via systemd directives.
+    assert "User=l4d2-sandbox" in text
+    assert "Group=l4d2-sandbox" in text
+
+    # Cgroup limits unchanged from v1.
+    assert "MemoryMax=4G" in text
+    assert "MemorySwapMax=0" in text
+    assert "TasksMax=512" in text
+    assert "CPUQuota=200%" in text
+    assert "RuntimeMaxSec=3600" in text
+
+    # Hardening directives that v1 (scope mode) couldn't carry.
+    assert "NoNewPrivileges=yes" in text
+    assert "ProtectSystem=strict" in text
+    assert "ProtectHome=yes" in text
+    assert "PrivateTmp=yes" in text
+    assert "PrivateDevices=yes" in text
+    assert "PrivateIPC=yes" in text
+    assert "ProtectKernelTunables=yes" in text
+    assert "ProtectKernelModules=yes" in text
+    assert "ProtectKernelLogs=yes" in text
+    assert "ProtectControlGroups=yes" in text
+    assert "RestrictNamespaces=yes" in text
+    assert "RestrictSUIDSGID=yes" in text
+    assert "LockPersonality=yes" in text
+    assert "MemoryDenyWriteExecute=yes" in text
+    assert "SystemCallFilter=" in text
+    assert "@system-service" in text
+    assert "@network-io" in text
+    assert "CapabilityBoundingSet=" in text
+    assert "AmbientCapabilities=" in text
+    assert 'RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX"' in text
+
+    # Network namespace stays shared with host.
+    assert "PrivateNetwork=" not in text
+
+    # Mount setup: /etc and /var/lib masked with tmpfs; selective binds back.
+    assert 'TemporaryFileSystem="/etc /var/lib"' in text
+    assert "BindReadOnlyPaths=" in text
+    # The resolv.conf bind points at the sandbox-only file (not the host's
+    # /etc/resolv.conf, which typically references a private-IP DNS server
+    # that IPAddressDeny= blocks).
+    assert "/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf" in text
+    assert "/etc/ssl" in text
+    assert "/etc/ca-certificates" in text
+    assert "/etc/nsswitch.conf" in text
+    assert "/etc/alternatives" in text
+    assert "${SCRIPT}:/script.sh" in text
+    assert 'BindPaths="${STAGING}:/overlay"' in text
+
+    # IP egress filter: allow public, deny localhost / RFC1918 / link-local /
+    # multicast / CGNAT / ULA. systemd's "more specific rule wins" semantics
+    # mean public IPs hit the allow and listed ranges hit the deny.
+    # IPAddressDeny alone — no IPAddressAllow=any. Empirically, having both
+    # set causes the allow to win on this systemd/kernel combo regardless of
+    # the documented "more specific rule wins" behaviour. With only Deny,
+    # the kernel's default "allow all" applies to non-listed addresses.
+    assert "IPAddressDeny=" in text
+    assert "IPAddressAllow=any" not in text
+    # Explicit CIDRs — systemd-run's -p parser doesn't accept the
+    # `localhost` / `link-local` / `multicast` shorthand keywords that
+    # work in unit files (only the full strings parse).
+    for token in (
+        "127.0.0.0/8",
+        "::1/128",
+        "169.254.0.0/16",
+        "fe80::/10",
+        "224.0.0.0/4",
+        "ff00::/8",
+        "10.0.0.0/8",
+        "172.16.0.0/12",
+        "192.168.0.0/16",
+        "100.64.0.0/10",
+        "fc00::/7",
+    ):
+        assert token in text, f"missing {token!r} in IPAddressDeny set"
+
+
+def test_script_sandbox_uses_idmap_staging():
+    """The sandbox runs as l4d2-sandbox but writes need to land on disk as
+    left4me, so all overlay content (workshop + script-built) is uniformly
+    left4me-owned. The helper pre-creates an idmapped bind on a staging
+    path and points the sandbox's BindPaths at the staging, not at the raw
+    overlay dir. trap cleans up the staging bind on exit.
+    """
+    text = SCRIPT_SANDBOX_HELPER.read_text()
+    # Idmap mount setup uses --map-users / --map-groups.
+    assert "--map-users=" in text
+    assert "--map-groups=" in text
+    # Staging path lives under /var/lib/left4me/tmp/sandbox-idmap-<id>.
+    assert "/var/lib/left4me/tmp/sandbox-idmap-" in text
+    # BindPaths into the sandbox points at the staging path, not the
+    # raw overlay dir.
+    assert 'BindPaths="${STAGING}:/overlay"' in text
+    # trap registers cleanup so the staging bind doesn't outlive the helper.
+    assert "trap " in text and "cleanup_staging" in text
+    # The previous chown-to-l4d2-sandbox approach is gone; overlay dirs
+    # stay left4me-owned end-to-end.
+    assert "chown -R l4d2-sandbox" not in text
+
+
+def test_script_sandbox_in_build_slice_with_oom_adjust():
+    text = SCRIPT_SANDBOX_HELPER.read_text()
+
+    # Put the transient unit in the low-weight build slice so it yields to
+    # game-server instances under CPU/IO contention.
+    assert "--slice=l4d2-build.slice" in text
+
+    # Sandbox dies first if the host hits memory pressure; servers
+    # (OOMScoreAdjust=-200) survive.
+    assert "-p OOMScoreAdjust=500" in text
+
+
+def test_script_sandbox_helper_validates_overlay_id():
+    text = SCRIPT_SANDBOX_HELPER.read_text()
+    # Numeric-only overlay id
+    assert '[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]]' in text
+    # Overlay dir must exist
+    assert "/var/lib/left4me/overlays/" in text
+    assert "[[ -d $OVERLAY_DIR ]]" in text
+    # Script path must exist
+    assert "[[ -f $SCRIPT ]]" in text
+
+
+def test_script_sandbox_helper_dry_run_mode(tmp_path):
+    overlay_root = tmp_path / "var/lib/left4me/overlays/42"
+    overlay_root.mkdir(parents=True)
+    fake_script = tmp_path / "fake.sh"
+    fake_script.write_text("echo hi")
+
+    # Run in DRY_RUN mode against a fake l4d2-sandbox UID via a tiny shim that
+    # simulates `id -u l4d2-sandbox` resolving to a valid number.
+    helper_text = SCRIPT_SANDBOX_HELPER.read_text()
+    # We can't actually exec this without root + a real sandbox user; just
+    # verify the dry-run guard short-circuits before systemd-run / bwrap.
+    assert 'LEFT4ME_SCRIPT_SANDBOX_DRY_RUN' in helper_text
+    assert 'exit 0' in helper_text

scripts/tests/test_sudoers_grants.py

@@ -0,0 +1,38 @@
+"""Audit the script→sudoers contract.
+
+The sudoers file in `deploy/files/etc/sudoers.d/left4me` is a reference
+example; ckn-bw ships its own verbatim copy under
+`bundles/left4me/files/etc/sudoers.d/left4me`. The two are expected to
+match. This test lives under `scripts/tests/` because the contract being
+audited is about *scripts* (which paths the `left4me` uid can sudo into)
+even though the file it reads is in `deploy/`.
+"""
+from conftest import DEPLOY
+
+
+SUDOERS = DEPLOY / "files/etc/sudoers.d/left4me"
+
+
+def test_sudoers_allows_only_left4me_helpers_not_raw_system_tools():
+    sudoers = SUDOERS.read_text()
+
+    assert (
+        "left4me ALL=(root) NOPASSWD: "
+        "/usr/local/libexec/left4me/left4me-systemctl *"
+    ) in sudoers
+    assert (
+        "left4me ALL=(root) NOPASSWD: "
+        "/usr/local/libexec/left4me/left4me-journalctl *"
+    ) in sudoers
+    assert "/usr/local/libexec/left4me/left4me-overlay mount *" in sudoers
+    assert "/usr/local/libexec/left4me/left4me-overlay umount *" in sudoers
+    assert (
+        "left4me ALL=(root) NOPASSWD: "
+        "/usr/local/libexec/left4me/left4me-script-sandbox"
+    ) in sudoers
+    assert "/bin/systemctl" not in sudoers
+    assert "/usr/bin/systemctl" not in sudoers
+    assert "/bin/journalctl" not in sudoers
+    assert "/usr/bin/journalctl" not in sudoers
+    assert "/bin/mount" not in sudoers
+    assert "/bin/umount" not in sudoers

scripts/tests/test_systemctl_helper.py

@@ -0,0 +1,39 @@
+import subprocess
+
+from conftest import LIBEXEC, env_with_fake_commands, fake_command
+
+
+SYSTEMCTL_HELPER = LIBEXEC / "left4me-systemctl"
+
+
+def test_systemctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path):
+    subprocess.run(["sh", "-n", str(SYSTEMCTL_HELPER)], check=True)
+    marker = fake_command(tmp_path, "systemctl")
+
+    for args in [
+        ["bad/action", "alpha"],
+        # `start` and `stop` are no longer accepted verbs — the lifecycle now
+        # uses `enable`/`disable` for reboot survival via WantedBy= symlinks.
+        ["start", "alpha"],
+        ["stop", "alpha"],
+        ["enable", ""],
+        ["enable", ".hidden"],
+        ["enable", "bad..name"],
+        ["enable", "bad/name"],
+        ["enable", "bad\\name"],
+        ["enable", "bad name"],
+    ]:
+        result = subprocess.run(
+            ["sh", str(SYSTEMCTL_HELPER), *args],
+            env=env_with_fake_commands(tmp_path),
+            check=False,
+        )
+        assert result.returncode != 0
+        assert not marker.exists()
+
+    script = SYSTEMCTL_HELPER.read_text()
+    assert 'unit="left4me-server@${name}.service"' in script
+    assert 'enable) exec "$systemctl" enable --now "$unit"' in script
+    assert 'disable) exec "$systemctl" disable --now "$unit"' in script
+    assert "--property=ActiveState" in script
+    assert "--property=SubState" in script