diff --git a/deploy/README.md b/deploy/README.md index b5c0c83..35778ba 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,117 +1,126 @@ -# left4me Deployment +# left4me deploy — reference exemplar -> Production provisioning of left4me on `ovh.left4me` is driven by -> [ckn-bw](https://git.sublimity.de/cronekorkn/ckn-bw) -> (`bundles/left4me/`, attached via `groups/applications/left4me.py`). -> Run `bw apply ovh.left4me` from the ckn-bw repo to deploy. +> **This directory is reference material, not the source of truth.** +> The canonical deploy of `ovh.left4me` is driven by +> [ckn-bw](https://git.sublimity.de/cronekorkn/ckn-bw)'s `bundles/left4me/` +> (attached via `groups/applications/left4me.py`); run `bw apply ovh.left4me` +> from the ckn-bw repo to deploy. > -> **What's canonical in this directory** (`deploy/files/`, `deploy/templates/`, -> `deploy/tests/`): the actual file payload ckn-bw deploys. ckn-bw fetches -> the left4me repo via `git_deploy` to `/opt/left4me/src/` and `install`s -> the privileged scripts from `deploy/files/usr/local/{libexec,sbin}/` -> directly onto the target. Sudoers, sysctl, and env-template content -> ships from `deploy/files/etc/` and `deploy/templates/etc/`. **Edit -> these files here; ckn-bw picks them up on the next apply.** No -> duplicate copy of the file content lives in ckn-bw. +> The privileged scripts the application installs live at the repo root +> under [`scripts/libexec/`](../scripts/libexec/) and +> [`scripts/sbin/`](../scripts/sbin/) — application code, not deploy +> artifacts. ckn-bw's `install_left4me_scripts` action reads them from +> `/opt/left4me/src/scripts/{libexec,sbin}/` after `git_deploy` and +> installs them into the standard FHS targets on the host. > -> **What's superseded**: the `deploy-test-server.sh` script — an older -> one-shot bash deploy that ckn-bw replaced. It's kept as a readable -> description of the install steps the bundle now performs declaratively. -> Don't run it against an ovh.left4me node managed by ckn-bw; the two -> would fight over file ownership. -> -> **What's obsolete** (kept for greppability, not currently used): CAKE -> traffic shaping (now in systemd-networkd via `network//cake` -> metadata in ckn-bw), nft marking (now in the central `nftables/output` -> set), and the systemd unit files under `files/usr/local/lib/systemd/system/` -> (emitted by the bundle's `systemd_units` reactor instead of being shipped -> as static files). The obsolete bits stay here intact so the original -> choices and tradeoffs remain greppable. +> What remains under `deploy/files/` and `deploy/templates/` is a set of +> readable **examples** — sudoers, sysctl, sandbox-resolv.conf, env +> templates, and a curated subset of the systemd units ckn-bw's reactor +> emits at apply time. They exist so a fresh consumer (other than ckn-bw) +> could read this tree and assemble an equivalent deployment. They are +> **not** the bytes ckn-bw installs; ckn-bw carries its own copies of the +> verbatim configs in `bundles/left4me/files/etc/`, and emits the live +> units from `bundles/left4me/metadata.py`'s `systemd_units` reactor. -## What lives here (and what corresponds to it in ckn-bw) +## What's here -| Path here | Status under ckn-bw | +| Path | Role | |---|---| -| `deploy-test-server.sh` | replaced by `bw apply` | -| `files/etc/sudoers.d/left4me` | shipped verbatim by `bundles/left4me/files/etc/sudoers.d/left4me` (validated with `visudo -cf` via `test_with`) | -| `files/etc/sysctl.d/99-left4me.conf` | shipped verbatim by the bundle | -| `files/etc/left4me/sandbox-resolv.conf` | shipped verbatim by the bundle | -| `files/usr/local/libexec/left4me/{left4me-systemctl,journalctl,overlay,script-sandbox}` | installed onto the target by the `install_left4me_scripts` action in `bundles/left4me/items.py`, reading directly from `/opt/left4me/src/deploy/files/usr/local/libexec/left4me/` after `git_deploy`. The bundle does **not** carry a duplicate copy. | -| `files/usr/local/sbin/left4me` | same install action; admin CLI wrapper (`sudo left4me `) | -| `files/usr/local/lib/systemd/system/left4me-web.service` | emitted by `systemd_units` reactor in `bundles/left4me/metadata.py` (intentional change: `--bind 0.0.0.0:8000` → `127.0.0.1:8000` because nginx now terminates TLS) | -| `files/usr/local/lib/systemd/system/left4me-server@.service` | emitted by the same reactor | -| `files/usr/local/lib/systemd/system/{l4d2-game,l4d2-build}.slice` | emitted by the same reactor | -| `files/usr/local/lib/systemd/system/left4me-cake.service` | **obsolete** — CAKE applied via systemd-networkd (`network//cake` metadata in `bundles/network/`) | -| `files/usr/local/libexec/left4me/left4me-apply-cake` | **obsolete** — same as above | -| `files/etc/left4me/cake.env` | **obsolete** — bandwidth lives in node metadata under `network/external/cake/Bandwidth` | -| `files/usr/local/lib/systemd/system/left4me-nft-mark.service` | **obsolete** — central `bundles/nftables/` consumes the rules from `bundles/left4me/`'s defaults | -| `files/usr/local/lib/left4me/nft/left4me-mark.nft` | **obsolete** — same as above | -| `templates/etc/left4me/host.env` | rendered as Mako by `bundles/left4me/files/etc/left4me/host.env.mako` | -| `templates/etc/left4me/web.env.template` | rendered as Mako by `bundles/left4me/files/etc/left4me/web.env.mako` (intentional change: `SESSION_COOKIE_SECURE=false` → `true`, plus `LEFT4ME_PORT_RANGE_*` are now wired through) | -| First-run admin bootstrap (`flask create-user … --admin` near the end of `deploy-test-server.sh`) | manual one-time step after `bw apply`; the bundle deliberately doesn't seed an admin to keep credentials out of the metadata pipeline | -| CPU isolation drop-ins (`/etc/systemd/system/{system,user,l4d2-game,l4d2-build}.slice.d/99-left4me-cpuset.conf`) | **not managed by the bundle** — generated dynamically based on `nproc --all` in the script; that logic doesn't fit static bundle metadata, apply manually post-deploy if needed | +| `files/etc/sudoers.d/left4me` | Example sudoers grants. Lockdown test: `scripts/tests/test_sudoers_grants.py`. | +| `files/etc/sysctl.d/99-left4me.conf` | Example sysctl perf baseline (UDP buffers, fq_codel + BBR). | +| `files/etc/left4me/sandbox-resolv.conf` | Example `/etc/resolv.conf` bound into the script-overlay sandbox. | +| `files/usr/local/lib/systemd/system/left4me-web.service` | Example of the web-app unit the reactor emits. | +| `files/usr/local/lib/systemd/system/left4me-server@.service` | Example of the per-instance gameserver unit. | +| `files/usr/local/lib/systemd/system/left4me-workshop-refresh.{service,timer}` | Example of the daily workshop-refresh cron-equivalent. | +| `files/usr/local/lib/systemd/system/l4d2-{game,build}.slice` | Example slice definitions (CPU/IO weights). | +| `templates/etc/left4me/host.env` | Example host-library env (deployment-fixed paths). | +| `templates/etc/left4me/web.env.template` | Example web-app env. ckn-bw renders the real version via the matching Mako template in `bundles/left4me/files/etc/left4me/web.env.mako`. | +| `tests/test_example_units.py` | Locks down the example units & env templates above. | ---- +The privileged scripts (`left4me-overlay`, `left4me-script-sandbox`, +`left4me-systemctl`, `left4me-journalctl`, `sbin/left4me`) used to live +under this tree at `files/usr/local/{libexec,sbin}/`; they moved to +`scripts/{libexec,sbin}/` because they are application code, not deploy +artifacts. -## Original notes (still accurate as a description of the install steps) +## Target layout -This directory contains the production-like test deployment for a Linux server. It installs the repository into a fixed host layout, configures a dedicated runtime user, installs systemd units, and wires the web app to host operations through privileged helper commands. +The deployment uses these on-host paths (FHS-aligned): -## Target Layout +- `/etc/left4me/host.env` — host library environment configuration. +- `/etc/left4me/web.env` — web app environment configuration. +- `/etc/left4me/sandbox-resolv.conf` — DNS resolv.conf bound into the + script-overlay sandbox. +- `/etc/sudoers.d/left4me` — sudoers rules letting the `left4me` uid call + the privileged helpers non-interactively. +- `/etc/sysctl.d/99-left4me.conf` — perf-baseline sysctls. +- `/opt/left4me` — deployed repository contents (via ckn-bw `git_deploy`). +- `/opt/left4me/.venv` — Python virtual environment for the web app. +- `/var/lib/left4me/left4me.db` — SQLite database used by the web app. +- `/var/lib/left4me/installation` — shared L4D2 installation. +- `/var/lib/left4me/overlays` — overlay directories. Each overlay lives + at `${overlay_id}` under here. +- `/var/lib/left4me/workshop_cache` — deduplicated cache of `.vpk` files + downloaded for workshop overlays. One file per Steam item, named + `{steam_id}.vpk`. Workshop overlays symlink into this tree. +- `/var/lib/left4me/instances` — rendered instance specifications and + per-instance state. +- `/var/lib/left4me/runtime` — per-instance runtime mount directories. +- `/var/lib/left4me/tmp` — temporary files used by deployment/runtime + operations (incl. idmap staging binds). +- `/usr/local/lib/systemd/system/` — global systemd unit files emitted + by ckn-bw's `systemd_units` reactor. +- `/usr/local/libexec/left4me/` — privileged helper commands installed + from `scripts/libexec/`. +- `/usr/local/sbin/left4me` — admin CLI wrapper installed from + `scripts/sbin/left4me`. -The deployment uses these paths: +## Runtime users -- `/etc/left4me/host.env`: host library environment configuration. -- `/etc/left4me/web.env`: web app environment configuration. -- `/opt/left4me/.venv`: Python virtual environment for deployed commands. -- `/opt/left4me`: deployed repository contents. -- `/var/lib/left4me/left4me.db`: SQLite database used by the web app. -- `/var/lib/left4me/installation`: shared L4D2 installation. -- `/var/lib/left4me/overlays`: overlay directories. Each overlay lives at `${overlay_id}` under here. -- `/var/lib/left4me/workshop_cache`: deduplicated cache of `.vpk` files downloaded for workshop overlays. One file per Steam item, named `{steam_id}.vpk`. Workshop overlays symlink into this tree. -- `/var/lib/left4me/global_overlay_cache`: cache of non-Steam map archives and extracted `.vpk` files used by managed global map overlays. -- `/var/lib/left4me/instances`: rendered instance specifications and per-instance state. -- `/var/lib/left4me/runtime`: per-instance runtime mount directories. -- `/var/lib/left4me/tmp`: temporary files used by deployment/runtime operations. -- `/usr/local/lib/systemd/system`: global systemd unit files, including `left4me-server@.service`. -- `/usr/local/libexec/left4me`: privileged helper commands, including `left4me-systemctl`, `left4me-journalctl`, and `left4me-overlay` (the latter mounts the per-instance kernel overlay in PID 1's mount namespace via `nsenter`). -- `/etc/sudoers.d/left4me`: sudoers rules allowing the web/runtime commands to call the helpers non-interactively. +Two system users are involved: -Static units are generated for `/var/lib/left4me`. If `LEFT4ME_ROOT` changes, regenerate and reinstall the unit files instead of reusing the existing static units. +- **`left4me`** (home `/var/lib/left4me`, shell `/usr/sbin/nologin`): + web app, host library, and gameserver runtime. +- **`l4d2-sandbox`** (no home, shell `/usr/sbin/nologin`): unprivileged + uid the script-overlay sandbox drops into via `systemd-run`. The + `left4me-script-sandbox` helper sets up an idmapped bind from the + sandbox uid back to `left4me` on a staging path so overlay writes + land on disk as `left4me`-owned. The split is load-bearing: a + sandbox escape would otherwise see `web.env`, the SQLite DB, and + running gameservers. -## Runtime User +(Whether the gameserver runtime should be split off into a third uid is +an open design question — see +`docs/superpowers/specs/2026-05-15-user-uid-split-design.md`.) -The deployment creates and runs host operations as the dedicated runtime user: +## Deployment -- Username: `left4me` -- Home: `/var/lib/left4me` -- Shell: `/usr/sbin/nologin` +Production deploy: -## Running A Test Deployment - -Run the deployment from the repository root: - -```bash -deploy/deploy-test-server.sh deploy-user@example-host +```sh +# In the ckn-bw repo: +bw apply ovh.left4me ``` -The SSH user must be able to run `sudo` on the target host. The deployment configures system packages, directories, environment files, helper scripts, sudoers rules, Python dependencies, and systemd units. +Admin bootstrap is a manual one-time step after the first apply +(ckn-bw deliberately doesn't seed an admin to keep credentials out of +the metadata pipeline): -## Admin Bootstrap - -Set the bootstrap credentials in the environment when creating the first admin user: - -```bash -LEFT4ME_ADMIN_USERNAME=admin \ -LEFT4ME_ADMIN_PASSWORD='change-me' \ -flask create-user "$LEFT4ME_ADMIN_USERNAME" --admin +```sh +sudo -u left4me LEFT4ME_ADMIN_USERNAME=admin \ + LEFT4ME_ADMIN_PASSWORD='change-me' \ + /opt/left4me/.venv/bin/flask --app l4d2web.app:create_app \ + create-user "$LEFT4ME_ADMIN_USERNAME" --admin ``` -Use a strong one-time password and rotate it after first login if needed. +Rotate the bootstrap password after first login. -## Overlay References +## Overlay references -Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`. With the default deployment root, they resolve under `/var/lib/left4me/overlays`. New overlays use `${overlay_id}` as their path; the digit-only form is the only one created by the web app. +Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`. +With the default deployment root, they resolve under +`/var/lib/left4me/overlays`. New overlays use `${overlay_id}` as their +path; the digit-only form is the only one created by the web app. Invalid references are rejected: @@ -122,125 +131,117 @@ Invalid references are rejected: The web app currently supports two overlay surfaces: -- `workshop` overlays (user-owned) — populated by downloading `.vpk` files from the public Steam Web API into `${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creating absolute symlinks under `${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`. -- `script` overlays — populated by an arbitrary user-authored bash script that runs inside `bubblewrap` + `systemd-run --scope` as the unprivileged `l4d2-sandbox` UID, with the overlay directory bind-mounted RW at `/overlay`. Resource caps: 1h walltime, 4 GB RAM, 512 tasks, 200% CPU, 20 GB post-build disk cap. +- **`workshop` overlays** (user-owned) — populated by downloading + `.vpk` files from the public Steam Web API into + `${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creating absolute + symlinks under + `${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`. +- **`script` overlays** — populated by an arbitrary user-authored bash + script that runs inside `systemd-run` as the unprivileged + `l4d2-sandbox` UID, with the overlay directory bind-mounted RW at + `/overlay`. Resource caps: 1h walltime, 4 GB RAM, 512 tasks, 200% CPU, + 20 GB post-build disk cap. -Both the caches and the overlay directories are owned by the `left4me` runtime user; if the web service ever runs as a different uid, ensure it shares a group with the host process and that both trees are group-readable. +Both caches and overlay directories are owned by `left4me`. If the web +service ever runs as a different uid, ensure it shares a group with the +host process and that both trees are group-readable. -## Performance Tuning +## Performance tuning -The deployment ships a host-side perf baseline (slices, unit directives, sysctls). See `docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md` for design rationale. +The deployment ships a host-side perf baseline (slices, unit directives, +sysctls). See +`docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md` +for design rationale. -The following knobs are documented escape hatches — they are **not** auto-applied. Apply only if you have measured a need and understand the failure modes. +The knobs below are documented escape hatches — **not** auto-applied. +Apply only after measuring a need and understanding the failure modes. ### Network shaping -The deploy ships three things that affect player-experience network behaviour: +Three pieces of the baseline affect player-experience network behaviour: -1. **Per-flow marking.** `left4me-nft-mark.service` loads a small nftables - table (`inet left4me_mark`) that marks every UDP packet from uid `left4me` - with DSCP EF and `skb->priority` 6. srcds doesn't set these itself, so - without this rule its UDP is indistinguishable from any other flow. +1. **Per-flow marking.** ckn-bw's central `bundles/nftables/` consumes + left4me's nftables defaults and marks every UDP packet from uid + `left4me` with DSCP EF and `skb->priority` 6. srcds doesn't set + these itself, so without this rule its UDP is indistinguishable + from any other flow. 2. **Sysctl baseline.** `99-left4me.conf` sets `udp_rmem_min=16384`, `udp_wmem_min=16384`, `default_qdisc=fq_codel`, and - `tcp_congestion_control=bbr`. Reduces head-of-line blocking when bulk - TCP egress (backups, package fetches, web responses) coexists with - game UDP. -3. **CAKE egress shaping.** `left4me-cake.service` runs - `tc qdisc replace dev root cake bandwidth Xmbit internet - diffserv4 dual-dsthost` from `/etc/left4me/cake.env`. CAKE only shapes - if its declared bandwidth is **below** the real bottleneck, so set - `LEFT4ME_UPLINK_MBIT` to ≈95% of measured uplink: - - sudoedit /etc/left4me/cake.env - # set LEFT4ME_UPLINK_MBIT=480 (or whatever ~95% of your uplink is) - sudo systemctl restart left4me-cake.service - - `LEFT4ME_UPLINK_IFACE` is auto-detected from the IPv4 default route; - override only on hosts with multi-homed setups. - - At idle 500 Mbit with no competing egress, CAKE shapes nothing — that's - expected, not a bug. The win materialises when bulk traffic on the - same uplink would otherwise bufferbloat the link the players share. - -**Production hosts running `systemd-networkd`** should NOT use the -`left4me-cake.service` oneshot. Instead, configure the equivalent in the -matching `.network` file, which systemd-networkd reapplies across iface -lifecycle events: - - # /etc/systemd/network/.network - [CAKE] - Bandwidth=480M - OverheadKeyword=internet - PriorityQueueingPreset=diffserv4 - EgressHostIsolation=yes - -The nftables marking from (1) is qdisc-installer-agnostic and ships -unchanged on production. - -**Disabling network shaping.** To turn the whole feature off on a deployed -host: - - sudo systemctl stop left4me-cake.service left4me-nft-mark.service - sudo systemctl disable left4me-cake.service left4me-nft-mark.service - -The sysctl baseline (`99-left4me.conf`) and the BBR/fq_codel defaults stay -applied; revert those by removing the file and running `sysctl --system` -if needed. + `tcp_congestion_control=bbr`. Reduces head-of-line blocking when + bulk TCP egress coexists with game UDP. +3. **CAKE egress shaping.** Configured per-interface via systemd-networkd + metadata (`network//cake` in ckn-bw's `bundles/network/`), + which reapplies the CAKE qdisc across iface lifecycle events. Set + the declared bandwidth to ≈95% of measured uplink — CAKE only shapes + if its declared bandwidth is *below* the real bottleneck. Idle links + with no competing egress see no visible CAKE effect; the win + materialises under bulk traffic that would otherwise bufferbloat the + link the players share. ### CPU governor -The performance governor squeezes a few percent off jitter under bursty load. `schedutil` is acceptable for sustained UDP workloads. +The performance governor squeezes a few percent off jitter under bursty +load. `schedutil` is acceptable for sustained UDP workloads. ```sh sudo cpupower frequency-set -g performance ``` -Install via `sudo apt install linux-cpupower` if the binary isn't present. - -Persist via your distro's CPU-frequency tooling (e.g. `/etc/default/cpufrequtils`). +Install via `sudo apt install linux-cpupower` if the binary isn't +present. Persist via your distro's CPU-frequency tooling (e.g. +`/etc/default/cpufrequtils`). ### CPU isolation (cores) -The deploy script writes four `AllowedCPUs=` drop-ins so that, by default, only `l4d2-game.slice` is allowed to run on cores 1..N-1; `system.slice`, `user.slice`, and `l4d2-build.slice` are pinned to core 0. Game servers thus get the host minus core 0 exclusively, the build sandbox and the web app stay on core 0, and a logged-in admin running CPU-heavy work in their shell can't steal cycles from a live match. +The deploy writes four `AllowedCPUs=` drop-ins so that by default only +`l4d2-game.slice` is allowed to run on cores `1..N-1`; `system.slice`, +`user.slice`, and `l4d2-build.slice` are pinned to core 0. Game servers +get the host minus core 0 exclusively; the build sandbox and the web +app stay on core 0; a logged-in admin running CPU-heavy work in their +shell can't steal cycles from a live match. Single-core hosts skip the +cpuset drop-ins entirely; the rest of the perf baseline (cgroup +weights, sysctls, OOM scores) still applies. -Override the split by setting either env var when running the deploy: - -```sh -LEFT4ME_SYSTEM_CPUS="0,1" LEFT4ME_GAME_CPUS="2-7" deploy/deploy-test-server.sh deploy-user@host -``` - -On single-core hosts the deploy skips the cpuset drop-ins entirely and prints a warning to stderr; the rest of the perf baseline (cgroup weights, sysctls, OOM scores) still applies. To force isolation on a single-core host anyway (rarely useful), set either env var explicitly. - -Per-instance `CPUAffinity=` (next subsection) composes on top of this — the per-instance value must be a subset of `l4d2-game.slice`'s `AllowedCPUs=`, which the kernel enforces. +Per-instance `CPUAffinity=` (next subsection) composes on top of this — +the per-instance value must be a subset of `l4d2-game.slice`'s +`AllowedCPUs=`, which the kernel enforces. ### Per-instance CPU affinity -`srcds` is single-threaded per instance. On a multi-core host, pinning each instance to its own core can cut jitter under contention. Drop in `/etc/systemd/system/left4me-server@.service.d/affinity.conf`: +`srcds` is single-threaded per instance. On a multi-core host, pinning +each instance to its own core can cut jitter under contention. Drop in +`/etc/systemd/system/left4me-server@.service.d/affinity.conf`: ```ini [Service] CPUAffinity=2 ``` -This pins the instance to CPU 2 specifically; per-instance values would typically be 1, 2, 3, ... so each server has its own core. - -A reasonable strategy on an N-core host: leave core 0 for the kernel + IRQs + system services, then pin one instance per remaining core. +This pins the instance to CPU 2. A reasonable strategy on an N-core +host: leave core 0 for the kernel + IRQs + system services, then pin +one instance per remaining core. ### NIC tuning -Hardware-specific (install via `sudo apt install ethtool` if not present). On a host with a single primary interface (replace `eth0`): +Hardware-specific (install via `sudo apt install ethtool` if not +present). On a host with a single primary interface (replace `eth0`): ```sh sudo ethtool -G eth0 rx 4096 tx 4096 sudo ethtool -K eth0 gro on lro off ``` -If you run a high instance count, also pin the NIC's interrupts off the cores that game servers occupy (see `/proc/interrupts` and `/proc/irq//smp_affinity`). +If you run a high instance count, also pin the NIC's interrupts off +the cores that game servers occupy (see `/proc/interrupts` and +`/proc/irq//smp_affinity`). ### Real-time scheduling (advanced, opt-in) -Source-engine servers do not need real-time scheduling, and a misbehaving `srcds` at any RT priority can starve kernel threads — even with the default `kernel.sched_rt_runtime_us=950000` throttling 5% of CPU back. Use only if you have a measured jitter problem that the baseline does not solve. +Source-engine servers do not need real-time scheduling, and a +misbehaving `srcds` at any RT priority can starve kernel threads — even +with the default `kernel.sched_rt_runtime_us=950000` throttling 5% of +CPU back. Use only if you have a measured jitter problem that the +baseline does not solve. `/etc/systemd/system/left4me-server@.service.d/realtime.conf`: @@ -252,13 +253,16 @@ LimitRTPRIO=10 AmbientCapabilities=CAP_SYS_NICE ``` -The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the service runs as `User=left4me` with `NoNewPrivileges=true`; without it some kernels/systemd combinations refuse to apply the RT policy. +The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the +service runs as `User=left4me` with `NoNewPrivileges=true`; without it +some kernels/systemd combinations refuse to apply the RT policy. ### Additional opt-in network knobs - **Ingress shaping via IFB.** Egress CAKE alone does not protect srcds - receive against ingress saturation (large workshop downloads, package - fetches arriving at line rate). One-liner: + receive against ingress saturation (large workshop downloads, + package fetches arriving at line rate). Worth flipping only when + measurement shows ingress hurting receive. sudo modprobe ifb && sudo ip link set ifb0 up sudo tc qdisc add dev handle ffff: ingress @@ -267,13 +271,11 @@ The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the service runs a sudo tc qdisc add dev ifb0 root cake bandwidth Xmbit ingress \ diffserv4 dual-srchost - Worth flipping only when measurement shows ingress hurting receive. - -- **`net.core.busy_poll = 50` / `net.core.busy_read = 50`.** Reduces UDP - receive median latency by polling for incoming packets briefly at - syscall boundaries. Cost: measurable CPU per syscall under load. Worth - flipping if a host is dedicated to game serving and CPU headroom is - plentiful. +- **`net.core.busy_poll = 50` / `net.core.busy_read = 50`.** Reduces + UDP receive median latency by polling for incoming packets briefly + at syscall boundaries. Cost: measurable CPU per syscall under load. + Worth flipping if a host is dedicated to game serving and CPU + headroom is plentiful. - **`ethtool -K gro off`.** Some Source-engine ops disable generic receive offload to avoid receive-side coalescing latency. @@ -281,7 +283,8 @@ The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the service runs a ### Applying changes to running servers -Unit-file changes do not apply to already-running services. After any change: +Unit-file changes do not apply to already-running services. After any +change: ```sh sudo systemctl daemon-reload diff --git a/deploy/deploy-test-server.sh b/deploy/deploy-test-server.sh deleted file mode 100755 index ae2a8cf..0000000 --- a/deploy/deploy-test-server.sh +++ /dev/null @@ -1,354 +0,0 @@ -#!/bin/sh -set -eu - -usage() { - printf 'Usage: %s \n' "$0" >&2 - exit 2 -} - -if [ "$#" -ne 1 ]; then - usage -fi - -target=$1 -script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) -repo_root=$(CDPATH= cd -- "$script_dir/.." && pwd) -tmp_dir=$(mktemp -d) -archive="$tmp_dir/left4me.tar.gz" - -cleanup() { - rm -rf "$tmp_dir" -} -trap cleanup EXIT INT HUP TERM - -COPYFILE_DISABLE=1 tar -czf "$archive" \ - --exclude .git \ - --exclude .claude \ - --exclude .venv \ - --exclude __pycache__ \ - --exclude .pytest_cache \ - --exclude '*.egg-info' \ - --exclude 'l4d2web.db*' \ - --exclude '._*' \ - -C "$repo_root" . - -remote_tmp=$(ssh "$target" 'mktemp -d') -scp "$archive" "$target:$remote_tmp/left4me.tar.gz" - -admin_username_file= -admin_password_file= -if [ "${LEFT4ME_ADMIN_USERNAME+x}" = x ] && [ "${LEFT4ME_ADMIN_PASSWORD+x}" = x ]; then - admin_username_file="$tmp_dir/admin_username" - admin_password_file="$tmp_dir/admin_password" - umask 077 - printf '%s' "$LEFT4ME_ADMIN_USERNAME" > "$admin_username_file" - printf '%s' "$LEFT4ME_ADMIN_PASSWORD" > "$admin_password_file" - scp "$admin_username_file" "$target:$remote_tmp/admin_username" - scp "$admin_password_file" "$target:$remote_tmp/admin_password" -fi - -ssh "$target" sh -s -- "$remote_tmp" <<'REMOTE' -set -eu - -remote_tmp=$1 -archive="$remote_tmp/left4me.tar.gz" -repo_tmp="$remote_tmp/repo" - -if [ "$(id -u)" -eq 0 ]; then - sudo_cmd= -else - sudo_cmd=sudo -fi - -run_as_left4me() { - sudo -u left4me "$@" -} - -run_left4me_with_env() { - run_as_left4me sh -c 'set -a; . /etc/left4me/host.env; . /etc/left4me/web.env; set +a; exec "$@"' sh "$@" -} - -cleanup_remote() { - rm -rf "$remote_tmp" -} -trap cleanup_remote EXIT INT HUP TERM - -if ! id left4me >/dev/null 2>&1; then - $sudo_cmd useradd --system --home-dir /var/lib/left4me --create-home --shell /usr/sbin/nologin left4me -fi - -# Sandbox uid for script-overlay builds. No home, no login shell — the bwrap -# invocation uses --uid/--gid to drop to it. -if ! id l4d2-sandbox >/dev/null 2>&1; then - $sudo_cmd useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox -fi - -if command -v apt-get >/dev/null 2>&1; then - $sudo_cmd apt-get update - $sudo_cmd apt-get install -y python3 python3-venv python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip-full nftables iproute2 -elif command -v dnf >/dev/null 2>&1; then - $sudo_cmd dnf install -y python3 python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip p7zip-plugins nftables iproute -else - printf 'Unsupported package manager: expected apt-get or dnf\n' >&2 - exit 1 -fi - -$sudo_cmd mkdir -p \ - /etc/left4me \ - /opt/left4me \ - /usr/local/lib/systemd/system \ - /usr/local/libexec/left4me \ - /usr/local/lib/left4me/nft \ - /var/lib/left4me/installation \ - /var/lib/left4me/overlays \ - /var/lib/left4me/instances \ - /var/lib/left4me/runtime \ - /var/lib/left4me/workshop_cache \ - /var/lib/left4me/tmp - -$sudo_cmd chown left4me:left4me \ - /var/lib/left4me \ - /var/lib/left4me/installation \ - /var/lib/left4me/overlays \ - /var/lib/left4me/instances \ - /var/lib/left4me/runtime \ - /var/lib/left4me/workshop_cache \ - /var/lib/left4me/tmp - -# /var/lib/left4me is left4me's home dir (mode 0700 from useradd --create-home). -# Allow other uids (notably l4d2-sandbox, used by script overlay builds) to -# traverse — but not list — so the bwrap bind-mount can resolve the overlay -# path under the dropped privilege. -$sudo_cmd chmod 0711 /var/lib/left4me -$sudo_cmd chown -R left4me:left4me /opt/left4me - -mkdir -p "$repo_tmp" -tar -xzf "$archive" -C "$repo_tmp" - -if [ -d /opt/left4me/.venv ]; then - $sudo_cmd mv /opt/left4me/.venv "$remote_tmp/venv" -fi -$sudo_cmd find /opt/left4me -mindepth 1 -maxdepth 1 -exec rm -rf {} + -$sudo_cmd cp -R "$repo_tmp"/. /opt/left4me/ -if [ -d "$remote_tmp/venv" ]; then - $sudo_cmd mv "$remote_tmp/venv" /opt/left4me/.venv -fi -$sudo_cmd chown -R left4me:left4me /opt/left4me - -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-web.service /usr/local/lib/systemd/system/left4me-web.service -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-server@.service /usr/local/lib/systemd/system/left4me-server@.service -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/l4d2-game.slice /usr/local/lib/systemd/system/l4d2-game.slice -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/l4d2-build.slice /usr/local/lib/systemd/system/l4d2-build.slice -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service /usr/local/lib/systemd/system/left4me-nft-mark.service -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-cake.service /usr/local/lib/systemd/system/left4me-cake.service -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-workshop-refresh.service /usr/local/lib/systemd/system/left4me-workshop-refresh.service -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-workshop-refresh.timer /usr/local/lib/systemd/system/left4me-workshop-refresh.timer - -# CPU isolation via cgroup-v2 AllowedCPUs= drop-ins. Pin everything that -# isn't a live game server to core 0; give game servers cores 1..N-1. -# See docs/superpowers/specs/2026-05-09-l4d2-cpu-isolation-design.md. -# `nproc --all` reports installed processors regardless of the calling -# shell's CPU affinity. Plain `nproc` honors Cpus_allowed of the calling -# process, so on a host that already has the cpuset drop-ins applied -# (system.slice → AllowedCPUs=0), the SSH login lands in user.slice with -# AllowedCPUs=0 and `nproc` would return 1 — making subsequent deploys -# wrongly think they're on a single-core box and skip CPU isolation. -NPROC=$(nproc --all) -SYSTEM_CPUS=${LEFT4ME_SYSTEM_CPUS:-0} -if [ "${LEFT4ME_GAME_CPUS+x}" = x ]; then - GAME_CPUS=$LEFT4ME_GAME_CPUS -else - GAME_CPUS="1-$((NPROC - 1))" -fi -if [ "$NPROC" -lt 2 ] && [ "${LEFT4ME_SYSTEM_CPUS+x}${LEFT4ME_GAME_CPUS+x}" = "" ]; then - printf 'left4me deploy: skipping CPU isolation (nproc=%s); cpuset drop-ins not written.\n' "$NPROC" >&2 -else - for slice_drop_in in \ - /etc/systemd/system/system.slice.d/99-left4me-cpuset.conf \ - /etc/systemd/system/user.slice.d/99-left4me-cpuset.conf \ - /etc/systemd/system/l4d2-build.slice.d/99-left4me-cpuset.conf; do - $sudo_cmd mkdir -p "$(dirname "$slice_drop_in")" - printf '[Slice]\nAllowedCPUs=%s\n' "$SYSTEM_CPUS" \ - | $sudo_cmd install -m 0644 -o root -g root /dev/stdin "$slice_drop_in" - done - $sudo_cmd mkdir -p /etc/systemd/system/l4d2-game.slice.d - printf '[Slice]\nAllowedCPUs=%s\n' "$GAME_CPUS" \ - | $sudo_cmd install -m 0644 -o root -g root /dev/stdin \ - /etc/systemd/system/l4d2-game.slice.d/99-left4me-cpuset.conf -fi - -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-systemctl -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-journalctl -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-overlay -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-script-sandbox /usr/local/libexec/left4me/left4me-script-sandbox -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-apply-cake /usr/local/libexec/left4me/left4me-apply-cake -$sudo_cmd cp /opt/left4me/deploy/files/usr/local/sbin/left4me /usr/local/sbin/left4me -$sudo_cmd chmod 0755 /usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-script-sandbox /usr/local/libexec/left4me/left4me-apply-cake /usr/local/sbin/left4me -$sudo_cmd cp /opt/left4me/deploy/files/etc/sudoers.d/left4me /etc/sudoers.d/left4me -$sudo_cmd chmod 0440 /etc/sudoers.d/left4me -$sudo_cmd visudo -cf /etc/sudoers.d/left4me - -$sudo_cmd cp /opt/left4me/deploy/templates/etc/left4me/host.env /etc/left4me/host.env -$sudo_cmd chmod 0644 /etc/left4me/host.env - -# Sandbox-only resolver config; bind-mounted into the script sandbox's /etc/resolv.conf -# so DNS still works when IPAddressDeny= blocks the host's (typically private-IP) resolver. -$sudo_cmd install -m 0644 -o root -g root \ - /opt/left4me/deploy/files/etc/left4me/sandbox-resolv.conf \ - /etc/left4me/sandbox-resolv.conf - -# Network packet marking + shaping. See spec -# docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md. -$sudo_cmd install -m 0644 -o root -g root \ - /opt/left4me/deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft \ - /usr/local/lib/left4me/nft/left4me-mark.nft - -# Host perf-baseline sysctls. Apply with `sysctl --system` so values -# take effect this deploy, not on next reboot. -$sudo_cmd install -m 0644 -o root -g root \ - /opt/left4me/deploy/files/etc/sysctl.d/99-left4me.conf \ - /etc/sysctl.d/99-left4me.conf -$sudo_cmd sysctl --system >/dev/null - -# CAKE config: ship the template only if the operator hasn't created one -# (their LEFT4ME_UPLINK_MBIT value must survive re-deploys). -if [ -e /etc/left4me/cake.env ]; then - : # operator file present; leave it intact -else - $sudo_cmd install -m 0644 -o root -g root \ - /opt/left4me/deploy/files/etc/left4me/cake.env \ - /etc/left4me/cake.env -fi - -# Stomp the file every deploy so newly added vars reach existing boxes. -# SECRET_KEY is derived from /etc/machine-id so it stays stable across -# redeploys (no session invalidation) without persisting state in /etc. -secret_key=$(sha256sum < /etc/machine-id | awk '{print $1}') -tmp_web_env="$remote_tmp/web.env" -{ - printf 'DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n' - printf 'SECRET_KEY=%s\n' "$secret_key" - printf 'JOB_WORKER_THREADS=4\n' - printf 'SESSION_COOKIE_SECURE=false\n' -} > "$tmp_web_env" -$sudo_cmd install -m 0640 -o root -g left4me "$tmp_web_env" /etc/left4me/web.env - -if [ ! -x /opt/left4me/.venv/bin/python ]; then - run_as_left4me python3 -m venv /opt/left4me/.venv -fi -run_as_left4me /opt/left4me/.venv/bin/python -m pip install --upgrade pip -run_as_left4me /opt/left4me/.venv/bin/pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web - -run_as_left4me sh -c "cd /opt/left4me/l4d2web && set -a; . /etc/left4me/host.env; . /etc/left4me/web.env; set +a; env \ - JOB_WORKER_ENABLED=false \ - PYTHONPATH=/opt/left4me \ - /opt/left4me/.venv/bin/alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head" - -# Tighten the application database to left4me:left4me 0640. The DB is -# created by the web app at first start with the default 0644 umask, which -# makes it world-readable on the host. The script-overlay sandbox runs as a -# separate system uid (l4d2-sandbox) which is NOT in the left4me group — -# 0640 blocks it via "other". The owner (left4me) keeps read+write so the -# web service can update the DB. -# -# SQLite in WAL mode (the default in this app) maintains -wal and -shm -# sidecar files; both must also be writable by the web service. If a previous -# operator opened the DB as root (e.g. for ad-hoc inspection), the sidecars -# may have ended up root-owned, which makes SQLite report "readonly database" -# on the next write. Re-chown them defensively. Idempotent on rerun. -for db_file in /var/lib/left4me/left4me.db /var/lib/left4me/left4me.db-wal /var/lib/left4me/left4me.db-shm; do - if [ -f "$db_file" ]; then - $sudo_cmd chown left4me:left4me "$db_file" - $sudo_cmd chmod 0640 "$db_file" - fi -done - -if [ -f "$remote_tmp/admin_username" ] && [ -f "$remote_tmp/admin_password" ]; then - LEFT4ME_ADMIN_USERNAME=$(cat "$remote_tmp/admin_username") - LEFT4ME_ADMIN_PASSWORD=$(cat "$remote_tmp/admin_password") - if ! create_user_output=$(run_left4me_with_env env \ - JOB_WORKER_ENABLED=false \ - LEFT4ME_ADMIN_PASSWORD="$LEFT4ME_ADMIN_PASSWORD" \ - /opt/left4me/.venv/bin/flask --app l4d2web.app:create_app create-user "$LEFT4ME_ADMIN_USERNAME" --admin 2>&1); then - case "$create_user_output" in - *'user already exists'*) printf '%s\n' "$create_user_output" ;; - *) printf '%s\n' "$create_user_output" >&2; exit 1 ;; - esac - else - printf '%s\n' "$create_user_output" - fi -fi - -# One-shot migration: fuse-overlayfs running as the left4me user used -# user.fuseoverlayfs.* xattrs for whiteouts and opaque-dir markers; kernel -# overlayfs ignores those entirely, so a pre-existing upper/ from the fuse -# era would resurrect "deleted" files. Wipe upper/ and work/ for every -# instance once, gated by a sentinel file so reruns are no-ops. -overlay_sentinel=/var/lib/left4me/.kernel-overlay-migrated -if [ ! -e "$overlay_sentinel" ]; then - $sudo_cmd sh -c "systemctl stop 'left4me-server@*.service' 2>/dev/null || true" - $sudo_cmd systemctl stop left4me-web.service 2>/dev/null || true - $sudo_cmd sh -c "findmnt -t fuse.fuse-overlayfs -o TARGET --noheadings 2>/dev/null | xargs -r -n1 umount -l 2>/dev/null || true" - $sudo_cmd sh -c "findmnt -t overlay -o TARGET --noheadings 2>/dev/null | grep '/var/lib/left4me/runtime/' | xargs -r -n1 umount -l 2>/dev/null || true" - $sudo_cmd sh -c 'for d in /var/lib/left4me/runtime/*/; do [ -d "$d" ] || continue; rm -rf "$d/upper" "$d/work"; mkdir -p "$d/upper" "$d/work"; chown left4me:left4me "$d/upper" "$d/work"; done' - $sudo_cmd touch "$overlay_sentinel" - $sudo_cmd chown left4me:left4me "$overlay_sentinel" -fi - -# One-shot migration: 0005_script_overlays drops the legacy -# l4d2center_maps / cedapug_maps overlay rows but doesn't touch their -# directories under /var/lib/left4me/overlays/{id}. Without cleanup, when -# AUTOINCREMENT (or its absence after the 0002 batch_alter_table recreate) -# re-issues an id matching one of those orphan dirs, the web app's -# create_overlay_directory(exist_ok=False) fails with FileExistsError. -# Sweep any overlay dir whose id has no matching DB row, plus the -# now-unused global_overlay_cache. -overlay_orphan_sentinel=/var/lib/left4me/.script-overlays-orphans-cleaned -if [ ! -e "$overlay_orphan_sentinel" ]; then - $sudo_cmd rm -rf /var/lib/left4me/global_overlay_cache - $sudo_cmd sh -c ' - cd /var/lib/left4me/overlays || exit 0 - ids_in_db=$(/opt/left4me/.venv/bin/python -c " -import sqlite3 -c = sqlite3.connect(\"/var/lib/left4me/left4me.db\") -print(\" \".join(str(r[0]) for r in c.execute(\"SELECT id FROM overlays\"))) -") - for d in */; do - id=${d%/} - case " $ids_in_db " in - *" $id "*) ;; - *) echo "removing orphan overlay dir: $id"; rm -rf "$id" ;; - esac - done - ' - $sudo_cmd touch "$overlay_orphan_sentinel" - $sudo_cmd chown left4me:left4me "$overlay_orphan_sentinel" -fi - -# Seed example script overlays (cedapug, l4d2center, competitive_rework, ...) -# as system-wide rows from /opt/left4me/examples/script-overlays/. Idempotent: -# subsequent deploys refresh the script body in place, leaving the row id and -# overlay directory intact. -run_left4me_with_env env \ - JOB_WORKER_ENABLED=false \ - PYTHONPATH=/opt/left4me \ - /opt/left4me/.venv/bin/flask --app l4d2web.app:create_app \ - seed-script-overlays /opt/left4me/examples/script-overlays - -$sudo_cmd systemctl daemon-reload -$sudo_cmd systemctl enable --now left4me-nft-mark.service -$sudo_cmd systemctl enable --now left4me-cake.service -$sudo_cmd systemctl enable --now left4me-web.service -$sudo_cmd systemctl restart left4me-web.service -$sudo_cmd systemctl enable --now left4me-workshop-refresh.timer -for attempt in 1 2 3 4 5 6 7 8 9 10; do - if curl -fsS http://127.0.0.1:8000/health; then - exit 0 - fi - sleep 1 -done - -$sudo_cmd systemctl status left4me-web.service --no-pager >&2 || true -$sudo_cmd journalctl -u left4me-web.service -n 80 --no-pager >&2 || true -exit 1 -REMOTE diff --git a/deploy/files/etc/left4me/cake.env b/deploy/files/etc/left4me/cake.env deleted file mode 100644 index dd5a56f..0000000 --- a/deploy/files/etc/left4me/cake.env +++ /dev/null @@ -1,12 +0,0 @@ -# left4me — CAKE egress shaper config. Consumed by left4me-cake.service via -# its EnvironmentFile=. Edit then `systemctl restart left4me-cake.service`. -# See docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md. - -# Uplink bandwidth in Mbit/s. Set to ~95% of the smaller of measured upload -# and measured download. CAKE only shapes correctly when its declared -# bandwidth sits below the real bottleneck. If unset, the shaper unit logs -# a warning and exits 0 (no shaping). -LEFT4ME_UPLINK_MBIT= - -# Egress interface. If unset, auto-detected from the IPv4 default route. -LEFT4ME_UPLINK_IFACE= diff --git a/deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft b/deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft deleted file mode 100644 index 1098266..0000000 --- a/deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft +++ /dev/null @@ -1,12 +0,0 @@ -# left4me — uid-based DSCP/priority marking for srcds UDP egress. -# Loaded by left4me-nft-mark.service into its own `inet` table so it cannot -# conflict with whatever the operator already runs in /etc/nftables.conf. -# See docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md. - -table inet left4me_mark { - chain mangle_output { - type filter hook output priority mangle; policy accept; - meta skuid "left4me" meta l4proto udp ip dscp set ef meta priority set 0006:0000 - meta skuid "left4me" meta l4proto udp ip6 dscp set ef meta priority set 0006:0000 - } -} diff --git a/deploy/files/usr/local/lib/systemd/system/left4me-cake.service b/deploy/files/usr/local/lib/systemd/system/left4me-cake.service deleted file mode 100644 index 5dce525..0000000 --- a/deploy/files/usr/local/lib/systemd/system/left4me-cake.service +++ /dev/null @@ -1,14 +0,0 @@ -[Unit] -Description=left4me CAKE egress shaper -After=network-online.target -Wants=network-online.target - -[Service] -Type=oneshot -RemainAfterExit=yes -EnvironmentFile=-/etc/left4me/cake.env -ExecStart=/usr/local/libexec/left4me/left4me-apply-cake apply -ExecStop=/usr/local/libexec/left4me/left4me-apply-cake clear - -[Install] -WantedBy=multi-user.target diff --git a/deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service b/deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service deleted file mode 100644 index b3de2f3..0000000 --- a/deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service +++ /dev/null @@ -1,14 +0,0 @@ -[Unit] -Description=left4me nftables packet marking (DSCP EF + priority for srcds) -After=network-pre.target -Before=network.target -Wants=network-pre.target - -[Service] -Type=oneshot -RemainAfterExit=yes -ExecStart=/usr/sbin/nft -f /usr/local/lib/left4me/nft/left4me-mark.nft -ExecStop=/usr/sbin/nft delete table inet left4me_mark - -[Install] -WantedBy=multi-user.target diff --git a/deploy/files/usr/local/libexec/left4me/left4me-apply-cake b/deploy/files/usr/local/libexec/left4me/left4me-apply-cake deleted file mode 100755 index 880f021..0000000 --- a/deploy/files/usr/local/libexec/left4me/left4me-apply-cake +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/sh -# left4me — apply or clear CAKE egress shaper on the configured uplink. -# Driven by left4me-cake.service. See spec -# docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md. -set -eu - -mode=${1:-apply} - -if [ -r /etc/left4me/cake.env ]; then - . /etc/left4me/cake.env -fi - -resolve_iface() { - if [ -n "${LEFT4ME_UPLINK_IFACE:-}" ]; then - printf '%s' "$LEFT4ME_UPLINK_IFACE" - return - fi - ip -4 route show default | awk '/default/ {print $5; exit}' -} - -case "$mode" in - apply) - if [ -z "${LEFT4ME_UPLINK_MBIT:-}" ]; then - echo "left4me-cake: LEFT4ME_UPLINK_MBIT unset; skipping shaper" >&2 - exit 0 - fi - iface=$(resolve_iface) - if [ -z "$iface" ]; then - echo "left4me-cake: cannot determine egress iface; skipping" >&2 - exit 0 - fi - exec tc qdisc replace dev "$iface" root cake \ - bandwidth "${LEFT4ME_UPLINK_MBIT}mbit" \ - internet diffserv4 dual-dsthost - ;; - clear) - iface=$(resolve_iface) - if [ -z "$iface" ]; then - exit 0 - fi - tc qdisc del dev "$iface" root 2>/dev/null || true - ;; - *) - echo "usage: $0 [apply|clear]" >&2 - exit 2 - ;; -esac diff --git a/deploy/tests/test_deploy_artifacts.py b/deploy/tests/test_deploy_artifacts.py deleted file mode 100644 index 29ccde1..0000000 --- a/deploy/tests/test_deploy_artifacts.py +++ /dev/null @@ -1,882 +0,0 @@ -import os -import subprocess -from pathlib import Path - - -ROOT = Path(__file__).resolve().parents[2] -DEPLOY = ROOT / "deploy" - - -WEB_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-web.service" -SERVER_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-server@.service" -GAME_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-game.slice" -BUILD_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-build.slice" -SYSCTL_CONF = DEPLOY / "files/etc/sysctl.d/99-left4me.conf" -GLOBAL_REFRESH_SERVICE = DEPLOY / "files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.service" -GLOBAL_REFRESH_TIMER = DEPLOY / "files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer" -NFT_MARK_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-nft-mark.service" -CAKE_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-cake.service" -SANDBOX_UNIT_DIR = DEPLOY / "files/usr/local/lib/systemd/system" -SYSTEMCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-systemctl" -JOURNALCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-journalctl" -OVERLAY_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-overlay" -SCRIPT_SANDBOX_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-script-sandbox" -APPLY_CAKE_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-apply-cake" -SANDBOX_RESOLV_CONF = DEPLOY / "files/etc/left4me/sandbox-resolv.conf" -CAKE_ENV = DEPLOY / "files/etc/left4me/cake.env" -SUDOERS = DEPLOY / "files/etc/sudoers.d/left4me" -HOST_ENV = DEPLOY / "templates/etc/left4me/host.env" -WEB_ENV_TEMPLATE = DEPLOY / "templates/etc/left4me/web.env.template" -DEPLOY_SCRIPT = DEPLOY / "deploy-test-server.sh" -NFT_MARK_FILE = DEPLOY / "files/usr/local/lib/left4me/nft/left4me-mark.nft" - - -def test_global_unit_files_exist_at_product_level_paths(): - assert WEB_UNIT.is_file() - assert SERVER_UNIT.is_file() - - -def test_web_unit_contains_required_runtime_contract(): - unit = WEB_UNIT.read_text() - - assert "User=left4me" in unit - assert "Group=left4me" in unit - assert "WorkingDirectory=/opt/left4me" in unit - assert "Environment=PATH=/opt/left4me/.venv/bin:" in unit - assert "EnvironmentFile=/etc/left4me/host.env" in unit - assert "EnvironmentFile=/etc/left4me/web.env" in unit - assert "ExecStart=/opt/left4me/.venv/bin/gunicorn" in unit - assert "--workers 1" in unit - assert "--threads 32" in unit - # NoNewPrivileges must remain unset because sudo (used by the overlay, - # systemctl and journalctl helpers) is setuid. - assert "NoNewPrivileges=true" not in unit - # Restored now that fuse-overlayfs propagation is no longer the mechanism. - assert "PrivateTmp=true" in unit - assert "ProtectSystem=full" in unit - assert "ReadWritePaths=/var/lib/left4me" in unit - # Mounts now happen in PID 1's namespace via the left4me-overlay helper, - # so MountFlags propagation is irrelevant — and the previous assumption - # that MountFlags=shared made it work was incorrect. - assert "MountFlags=" not in unit - - -def test_server_unit_contains_required_runtime_contract(): - unit = SERVER_UNIT.read_text() - - assert "User=left4me" in unit - assert "Group=left4me" in unit - assert "EnvironmentFile=/etc/left4me/host.env" in unit - assert "EnvironmentFile=/var/lib/left4me/instances/%i/instance.env" in unit - # `-` prefix: chdir failure is non-fatal so ExecStartPre can run the - # mount helper before the merged dir exists. ExecStart re-applies and - # finds the dir once the mount has landed. - assert "WorkingDirectory=-/var/lib/left4me/runtime/%i/merged/left4dead2" in unit - # ExecStart must invoke srcds_run from the *merged* overlay tree, not - # from installation/. srcds_run cds to its own dirname; if we point at - # installation/, the engine reads gameinfo.txt and addons from the lower - # layer and never sees overlay plugins (Metamod/SourceMod) or cfgs. - assert "ExecStart=/var/lib/left4me/runtime/%i/merged/srcds_run" in unit - assert "$L4D2_ARGS" in unit - assert "${L4D2_ARGS}" not in unit - assert "NoNewPrivileges=true" in unit - assert "PrivateTmp=true" in unit - assert "PrivateDevices=true" in unit - assert "ProtectHome=true" in unit - assert "ProtectSystem=strict" in unit - assert "ReadOnlyPaths=/var/lib/left4me/installation /var/lib/left4me/overlays" in unit - assert "ReadWritePaths=/var/lib/left4me/runtime/%i" in unit - assert "RestrictSUIDSGID=true" in unit - assert "LockPersonality=true" in unit - - -def test_server_unit_mounts_overlay_via_exec_start_pre(): - """At boot, systemd auto-starts enabled units before the web app gets a - chance to run start_instance's pre-start mount. The unit itself must - re-mount the overlay so reboots are transparent. Pairs with the helper's - idempotency check (test_overlay_helper_mount_is_idempotent_when_mounted). - - The unit-level `nsenter --mount=/proc/1/ns/mnt --` is what makes - umount fast: without it, the helper Python process would inherit - the unit's per-service mount namespace and pin it alive, blocking - PID 1's umount until the helper exited. Wrapping with nsenter at - the Exec line puts the helper itself in PID 1's namespace. - """ - unit = SERVER_UNIT.read_text() - # `+` prefix: runs as PID 1 (root, no sandbox). Required because - # the unit has NoNewPrivileges=true, which blocks sudo's setuid - # escalation — and the helper needs root for the mount syscall. - assert ( - "ExecStartPre=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- " - "/usr/local/libexec/left4me/left4me-overlay mount %i" - in unit - ) - # Bound the restart loop; without these, a CHDIR-failure (or any other - # pre-start error) spins indefinitely. - assert "StartLimitBurst=5" in unit - assert "StartLimitIntervalSec=60s" in unit - - -def test_server_unit_unmounts_overlay_via_exec_stop_post(): - """Single source of truth for unmount, mirroring the mount path. - ExecStopPost (not ExecStop) so it runs after srcds has fully exited - and the cgroup is cleared. - - Same nsenter-at-Exec-line wrapping as ExecStartPre — without it, - the helper process would itself hold a reference to the unit's - per-service mount namespace, and umount in PID 1 would loop on - EBUSY until the helper gave up. With it, umount succeeds first try. - """ - unit = SERVER_UNIT.read_text() - assert ( - "ExecStopPost=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- " - "/usr/local/libexec/left4me/left4me-overlay umount %i" - in unit - ) - - -def test_overlay_helper_mount_is_idempotent_when_already_mounted(): - """ExecStartPre runs on every Restart=on-failure cycle. If a previous - start mounted successfully but ExecStart failed afterwards, the next - ExecStartPre would re-mount on top -- which fails. The helper must - short-circuit when merged is already a mount point. - """ - text = OVERLAY_HELPER.read_text() - # Two ismount checks now: one in cmd_mount (skip if mounted), - # one in cmd_umount (skip if not mounted). - assert text.count("os.path.ismount") >= 2 - - -def test_server_unit_contains_perf_baseline_directives(): - unit = SERVER_UNIT.read_text() - - # Slice membership. - assert "Slice=l4d2-game.slice" in unit - - # CFS priority bump (no SCHED_FIFO). - assert "Nice=-5" in unit - assert "CPUSchedulingPolicy=" not in unit - - # I/O priority. - assert "IOSchedulingClass=best-effort" in unit - assert "IOSchedulingPriority=4" in unit - - # OOM ordering: game servers survive, sandbox dies first. - assert "OOMScoreAdjust=-200" in unit - - # Memory caps with headroom for map-load spikes. - assert "MemoryHigh=1.5G" in unit - assert "MemoryMax=2G" in unit - - # Bounded fork surface. - assert "TasksMax=256" in unit - - # Plenty of fds for plugin-heavy setups. - assert "LimitNOFILE=65536" in unit - - # srcds clean shutdown via SIGINT, with time to flush. With the - # helper running in PID 1's mount namespace (via the unit-level - # nsenter on ExecStopPost), umount has no race window and the - # default 15 s is plenty for the whole stop transition. - assert "KillSignal=SIGINT" in unit - assert "TimeoutStopSec=15s" in unit - - # Per-unit override of journald rate limiting (default drops srcds output). - assert "LogRateLimitIntervalSec=0" in unit - - -def test_l4d2_game_slice_exists_with_high_weights(): - assert GAME_SLICE.is_file() - text = GAME_SLICE.read_text() - assert "[Slice]" in text - assert "CPUWeight=1000" in text - assert "IOWeight=1000" in text - - -def test_l4d2_build_slice_exists_with_low_weights(): - assert BUILD_SLICE.is_file() - text = BUILD_SLICE.read_text() - assert "[Slice]" in text - assert "CPUWeight=10" in text - assert "IOWeight=10" in text - - -def test_sysctl_conf_present_with_perf_settings(): - assert SYSCTL_CONF.is_file() - text = SYSCTL_CONF.read_text() - for line in ( - "net.core.rmem_max = 8388608", - "net.core.wmem_max = 8388608", - "net.core.rmem_default = 524288", - "net.core.wmem_default = 524288", - "net.core.netdev_max_backlog = 5000", - "net.core.netdev_budget = 600", - "vm.swappiness = 10", - "net.ipv4.udp_rmem_min = 16384", - "net.ipv4.udp_wmem_min = 16384", - "net.core.default_qdisc = fq_codel", - "net.ipv4.tcp_congestion_control = bbr", - ): - assert line in text, f"missing {line!r} in 99-left4me.conf" - - -def test_script_sandbox_in_build_slice_with_oom_adjust(): - text = SCRIPT_SANDBOX_HELPER.read_text() - - # Put the transient unit in the low-weight build slice so it yields to - # game-server instances under CPU/IO contention. - assert "--slice=l4d2-build.slice" in text - - # Sandbox dies first if the host hits memory pressure; servers - # (OOMScoreAdjust=-200) survive. - assert "-p OOMScoreAdjust=500" in text - - -def test_deploy_script_installs_perf_artifacts(): - script = DEPLOY_SCRIPT.read_text() - - # Slice files copied into the system-wide systemd unit dir. - assert "/usr/local/lib/systemd/system/l4d2-game.slice" in script - assert "/usr/local/lib/systemd/system/l4d2-build.slice" in script - - # Sysctl drop-in installed under /etc/sysctl.d/. - assert "/etc/sysctl.d/99-left4me.conf" in script - - # Values applied immediately, not on next boot. - assert "sysctl --system" in script - - -def test_deploy_script_writes_cpuset_drop_ins(): - script = DEPLOY_SCRIPT.read_text() - - # Reads nproc and binds defaults via ${VAR:-...}. - assert "nproc" in script - assert "LEFT4ME_SYSTEM_CPUS" in script - assert "LEFT4ME_GAME_CPUS" in script - assert "${LEFT4ME_SYSTEM_CPUS:-0}" in script - - # Default game-core upper bound is computed from nproc; accept either - # the NPROC-1 form or LEFT4ME_GAME_CPUS:-1- prefix. - assert ( - "1-$((NPROC - 1))" in script - or "1-$((NPROC-1))" in script - or "1-$((nproc-1))" in script - or "LEFT4ME_GAME_CPUS:-1-" in script - ) - - # All four drop-in paths. - for slice_name in ("system", "user", "l4d2-build", "l4d2-game"): - assert ( - f"/etc/systemd/system/{slice_name}.slice.d/99-left4me-cpuset.conf" - in script - ) - - # Drop-ins use the existing install pattern. - assert "install -m 0644 -o root -g root" in script - - # Single-core host: skip with a warning to stderr. - assert ("-lt 2" in script) or ("< 2" in script) or ("-ge 2" in script) - assert "skipping CPU isolation" in script - - -def _fake_command(tmp_path, command_name): - marker = tmp_path / f"{command_name}.args" - command = tmp_path / command_name - command.write_text(f"#!/bin/sh\nprintf '%s\n' \"$*\" > '{marker}'\nexit 0\n") - command.chmod(0o755) - return marker - - -def _env_with_fake_commands(tmp_path): - env = os.environ.copy() - env["PATH"] = f"{tmp_path}{os.pathsep}{env.get('PATH', '')}" - return env - - -def test_helpers_use_fixed_system_tool_paths_not_sudo_path(): - systemctl = SYSTEMCTL_HELPER.read_text() - journalctl = JOURNALCTL_HELPER.read_text() - - assert "command -v systemctl" not in systemctl - assert "command -v journalctl" not in journalctl - assert "/bin/systemctl" in systemctl or "/usr/bin/systemctl" in systemctl - assert "/bin/journalctl" in journalctl or "/usr/bin/journalctl" in journalctl - - -def test_systemctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path): - subprocess.run(["sh", "-n", str(SYSTEMCTL_HELPER)], check=True) - marker = _fake_command(tmp_path, "systemctl") - - for args in [ - ["bad/action", "alpha"], - # `start` and `stop` are no longer accepted verbs — the lifecycle now - # uses `enable`/`disable` for reboot survival via WantedBy= symlinks. - ["start", "alpha"], - ["stop", "alpha"], - ["enable", ""], - ["enable", ".hidden"], - ["enable", "bad..name"], - ["enable", "bad/name"], - ["enable", "bad\\name"], - ["enable", "bad name"], - ]: - result = subprocess.run(["sh", str(SYSTEMCTL_HELPER), *args], env=_env_with_fake_commands(tmp_path), check=False) - assert result.returncode != 0 - assert not marker.exists() - - script = SYSTEMCTL_HELPER.read_text() - assert 'unit="left4me-server@${name}.service"' in script - assert 'enable) exec "$systemctl" enable --now "$unit"' in script - assert 'disable) exec "$systemctl" disable --now "$unit"' in script - assert "--property=ActiveState" in script - assert "--property=SubState" in script - - -def test_journalctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path): - subprocess.run(["sh", "-n", str(JOURNALCTL_HELPER)], check=True) - marker = _fake_command(tmp_path, "journalctl") - - for args in [ - ["../evil", "--lines", "25", "--no-follow"], - ["alpha", "--bad", "25", "--no-follow"], - ["alpha", "--lines", "not-number", "--no-follow"], - ["alpha", "--lines", "25", "--bad-follow"], - ["bad/name", "--lines", "25", "--no-follow"], - ]: - result = subprocess.run(["sh", str(JOURNALCTL_HELPER), *args], env=_env_with_fake_commands(tmp_path), check=False) - assert result.returncode != 0 - assert not marker.exists() - - script = JOURNALCTL_HELPER.read_text() - assert 'unit="left4me-server@${name}.service"' in script - assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat "$follow_arg"' in script - assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat' in script - - -def test_sudoers_allows_only_left4me_helpers_not_raw_system_tools(): - sudoers = SUDOERS.read_text() - - assert ( - "left4me ALL=(root) NOPASSWD: " - "/usr/local/libexec/left4me/left4me-systemctl *" - ) in sudoers - assert ( - "left4me ALL=(root) NOPASSWD: " - "/usr/local/libexec/left4me/left4me-journalctl *" - ) in sudoers - assert "/usr/local/libexec/left4me/left4me-overlay mount *" in sudoers - assert "/usr/local/libexec/left4me/left4me-overlay umount *" in sudoers - assert ( - "left4me ALL=(root) NOPASSWD: " - "/usr/local/libexec/left4me/left4me-script-sandbox" - ) in sudoers - assert "/bin/systemctl" not in sudoers - assert "/usr/bin/systemctl" not in sudoers - assert "/bin/journalctl" not in sudoers - assert "/usr/bin/journalctl" not in sudoers - assert "/bin/mount" not in sudoers - assert "/bin/umount" not in sudoers - - -def test_overlay_helper_is_python_with_strict_validation(): - text = OVERLAY_HELPER.read_text() - assert text.startswith("#!/usr/bin/python3") - # Validation surface - assert "NAME_RE = re.compile" in text - assert "LOWERDIR_ALLOWLIST" in text - assert "user.fuseoverlayfs." in text - assert "MAX_LOWERDIRS = 500" in text - # Mounts via PID 1's mount namespace - assert "/proc/1/ns/mnt" in text - assert "nsenter" in text - # Verbs are mount and umount (not unmount) - assert '"mount"' in text and '"umount"' in text - assert '"unmount"' not in text - - -def test_script_sandbox_uses_idmap_staging(): - """The sandbox runs as l4d2-sandbox but writes need to land on disk as - left4me, so all overlay content (workshop + script-built) is uniformly - left4me-owned. The helper pre-creates an idmapped bind on a staging - path and points the sandbox's BindPaths at the staging, not at the raw - overlay dir. trap cleans up the staging bind on exit. - """ - text = SCRIPT_SANDBOX_HELPER.read_text() - # Idmap mount setup uses --map-users / --map-groups. - assert "--map-users=" in text - assert "--map-groups=" in text - # Staging path lives under /var/lib/left4me/tmp/sandbox-idmap-. - assert "/var/lib/left4me/tmp/sandbox-idmap-" in text - # BindPaths into the sandbox points at the staging path, not the - # raw overlay dir. - assert 'BindPaths="${STAGING}:/overlay"' in text - # trap registers cleanup so the staging bind doesn't outlive the helper. - assert "trap " in text and "cleanup_staging" in text - # The previous chown-to-l4d2-sandbox approach is gone; overlay dirs - # stay left4me-owned end-to-end. - assert "chown -R l4d2-sandbox" not in text - - -def test_deploy_script_installs_overlay_helper_with_executable_mode(): - script = DEPLOY_SCRIPT.read_text() - assert "/usr/local/libexec/left4me/left4me-overlay" in script - assert "chmod 0755" in script and "left4me-overlay" in script - - -def test_deploy_script_does_not_install_fuse_overlayfs_apt_dep(): - # fuse-overlayfs / fuse3 were the previous mount engine; kernel overlayfs - # replaces them. Comments in the migration block may legitimately mention - # the names, so scope this to the actual apt-get / dnf install lines. - install_lines = [ - line for line in DEPLOY_SCRIPT.read_text().splitlines() - if ("apt-get install" in line or "dnf install" in line) - ] - assert install_lines, "expected at least one apt/dnf install line" - for line in install_lines: - assert "fuse-overlayfs" not in line, line - assert "fuse3" not in line, line - - -def test_deploy_script_runs_one_shot_kernel_overlay_migration(): - script = DEPLOY_SCRIPT.read_text() - assert "/var/lib/left4me/.kernel-overlay-migrated" in script - # Migration should stop services + force-unmount stale mounts + wipe upper/work - assert "systemctl stop 'left4me-server@" in script - assert "systemctl stop left4me-web.service" in script - assert "findmnt -t overlay" in script - assert "/runtime/" in script and "rm -rf" in script and 'upper"' in script and 'work"' in script - - -def test_env_templates_contain_required_defaults(): - host_env = HOST_ENV.read_text() - assert "Deployment units use fixed /var/lib/left4me paths" in host_env - assert host_env.endswith("LEFT4ME_ROOT=/var/lib/left4me\n") - web_env = WEB_ENV_TEMPLATE.read_text() - assert web_env.startswith( - "DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n" - "SECRET_KEY=replace-with-generated-secret\n" - "JOB_WORKER_THREADS=4\n" - ) - assert web_env.rstrip().endswith("STEAM_WEB_API_KEY=") - - -def test_deploy_script_has_safe_defaults_and_preserves_state() -> None: - script = DEPLOY_SCRIPT.read_text() - - assert "useradd --system --home-dir /var/lib/left4me" in script - assert "/var/lib/left4me/installation" in script - assert "/var/lib/left4me/overlays" in script - assert "/var/lib/left4me/instances" in script - assert "/var/lib/left4me/runtime" in script - assert "tar" in script - assert "--exclude .venv" in script - assert "--exclude .claude" in script - assert "pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web" in script - assert "systemctl enable --now left4me-web.service" in script - assert "for attempt in" in script - assert "/opt/left4me/.venv" in script - assert "visudo -cf /etc/sudoers.d/left4me" in script - # Note: assertions about web.env's lifecycle (create-only-if-missing / - # never-sourced-from-deploy) used to live here. They became stale in - # commit caa8b83, which switched to "rewrite web.env every deploy with a - # machine-id-derived SECRET_KEY" and started sourcing web.env in the - # alembic + seed helper subprocesses. Removed entirely; current behavior - # is covered by `install -m 0640 ... /etc/left4me/web.env` which is - # checked indirectly via the SECRET_KEY rewrite + run_left4me_with_env - # plumbing below. - assert "run_left4me_with_env" in script - assert "LEFT4ME_ADMIN_USERNAME" in script - assert "LEFT4ME_ADMIN_PASSWORD" in script - assert "user already exists" in script - assert "deploy/files" in script - - -def test_deploy_script_does_not_recurse_into_runtime_state_mounts() -> None: - script = DEPLOY_SCRIPT.read_text() - - assert "$sudo_cmd chown -R left4me:left4me /var/lib/left4me" not in script - assert "$sudo_cmd chown left4me:left4me \\" in script - assert "/var/lib/left4me/runtime \\" in script - assert "$sudo_cmd chown -R left4me:left4me /opt/left4me" in script - - -def test_deploy_script_runs_migrations_before_app_initialization() -> None: - script = DEPLOY_SCRIPT.read_text() - - assert "alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head" in script - assert "from l4d2web.app import create_app; create_app()" not in script - - -def test_deploy_script_shell_syntax() -> None: - subprocess.run(["sh", "-n", str(DEPLOY_SCRIPT)], check=True) - - -def test_globals_refresh_units_removed(): - """Global-overlays subsystem deleted in favor of script overlays.""" - assert not GLOBAL_REFRESH_SERVICE.exists() - assert not GLOBAL_REFRESH_TIMER.exists() - - -def test_deploy_script_does_not_provision_globals_subsystem(): - script = DEPLOY_SCRIPT.read_text() - - # No mkdir/install of the deleted cache dir; mention in a one-shot - # `rm -rf` cleanup is fine. - for line in script.splitlines(): - if "/var/lib/left4me/global_overlay_cache" not in line: - continue - assert "rm -rf" in line, line - assert "left4me-refresh-global-overlays" not in script - - -def test_deploy_script_provisions_sandbox_user(): - script = DEPLOY_SCRIPT.read_text() - assert "useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox" in script - - -def test_deploy_script_does_not_install_bubblewrap(): - install_lines = [ - line for line in DEPLOY_SCRIPT.read_text().splitlines() - if ("apt-get install" in line or "dnf install" in line) - ] - assert install_lines, "expected at least one apt/dnf install line" - for line in install_lines: - assert "bubblewrap" not in line, line - assert "bwrap" not in line, line - - -def test_deploy_script_installs_script_overlay_tooling(): - # Script overlays commonly need 7z and md5sum (e.g. l4d2center map sync). - # coreutils ships md5sum and is technically essential, but listing it - # explicitly makes the contract obvious and survives slim base images. - script = DEPLOY_SCRIPT.read_text().splitlines() - apt_lines = [l for l in script if "apt-get install" in l] - dnf_lines = [l for l in script if "dnf install" in l] - assert apt_lines, "expected an apt-get install line" - assert dnf_lines, "expected a dnf install line" - for line in apt_lines: - assert "p7zip-full" in line, line - assert "coreutils" in line, line - for line in dnf_lines: - # Fedora/RHEL split: p7zip provides 7za, p7zip-plugins provides 7z. - assert "p7zip" in line and "p7zip-plugins" in line, line - assert "coreutils" in line, line - - -def test_deploy_script_tightens_left4me_db_permissions(): - script = DEPLOY_SCRIPT.read_text() - # The DB and its WAL/SHM sidecars must be left4me:left4me 0640 — owner - # (web service) keeps rw, group is read-only, "other" (incl. l4d2-sandbox) - # gets nothing. The sidecars matter because SQLite in WAL mode requires - # write access to all three; if a sidecar ends up root-owned (e.g. from - # ad-hoc root-side inspection), the next write fails as "readonly db". - assert "chown left4me:left4me" in script - assert "chmod 0640" in script - for db_file in ( - "/var/lib/left4me/left4me.db", - "/var/lib/left4me/left4me.db-wal", - "/var/lib/left4me/left4me.db-shm", - ): - assert db_file in script, f"deploy script must touch {db_file}" - - -def test_deploy_script_installs_script_sandbox_helper(): - script = DEPLOY_SCRIPT.read_text() - assert "/usr/local/libexec/left4me/left4me-script-sandbox" in script - assert "chmod 0755" in script and "left4me-script-sandbox" in script - - -def test_script_sandbox_helper_present(): - assert SCRIPT_SANDBOX_HELPER.is_file() - assert SCRIPT_SANDBOX_HELPER.read_text().startswith("#!/bin/bash") - mode = SCRIPT_SANDBOX_HELPER.stat().st_mode & 0o777 - assert mode == 0o755, f"expected 0755, got {oct(mode)}" - - -def test_script_sandbox_helper_passes_shell_syntax_check(): - subprocess.run(["bash", "-n", str(SCRIPT_SANDBOX_HELPER)], check=True) - - -def test_script_sandbox_helper_invokes_systemd_run_with_hardening(): - text = SCRIPT_SANDBOX_HELPER.read_text() - - # systemd-run service mode (no --scope), with synchronous I/O to caller. - assert "systemd-run" in text - assert "--scope" not in text, "v2 uses transient service units, not scopes" - assert "--pipe" in text - assert "--wait" in text - assert "--collect" in text - assert "--unit=" in text - - # No bwrap. - assert "bwrap" not in text - assert "bubblewrap" not in text - - # UID drop via systemd directives. - assert "User=l4d2-sandbox" in text - assert "Group=l4d2-sandbox" in text - - # Cgroup limits unchanged from v1. - assert "MemoryMax=4G" in text - assert "MemorySwapMax=0" in text - assert "TasksMax=512" in text - assert "CPUQuota=200%" in text - assert "RuntimeMaxSec=3600" in text - - # Hardening directives that v1 (scope mode) couldn't carry. - assert "NoNewPrivileges=yes" in text - assert "ProtectSystem=strict" in text - assert "ProtectHome=yes" in text - assert "PrivateTmp=yes" in text - assert "PrivateDevices=yes" in text - assert "PrivateIPC=yes" in text - assert "ProtectKernelTunables=yes" in text - assert "ProtectKernelModules=yes" in text - assert "ProtectKernelLogs=yes" in text - assert "ProtectControlGroups=yes" in text - assert "RestrictNamespaces=yes" in text - assert "RestrictSUIDSGID=yes" in text - assert "LockPersonality=yes" in text - assert "MemoryDenyWriteExecute=yes" in text - assert "SystemCallFilter=" in text - assert "@system-service" in text - assert "@network-io" in text - assert "CapabilityBoundingSet=" in text - assert "AmbientCapabilities=" in text - assert 'RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX"' in text - - # Network namespace stays shared with host. - assert "PrivateNetwork=" not in text - - # Mount setup: /etc and /var/lib masked with tmpfs; selective binds back. - assert 'TemporaryFileSystem="/etc /var/lib"' in text - assert "BindReadOnlyPaths=" in text - # The resolv.conf bind points at the sandbox-only file (not the host's - # /etc/resolv.conf, which typically references a private-IP DNS server - # that IPAddressDeny= blocks). - assert "/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf" in text - assert "/etc/ssl" in text - assert "/etc/ca-certificates" in text - assert "/etc/nsswitch.conf" in text - assert "/etc/alternatives" in text - assert "${SCRIPT}:/script.sh" in text - assert 'BindPaths="${STAGING}:/overlay"' in text - - # IP egress filter: allow public, deny localhost / RFC1918 / link-local / - # multicast / CGNAT / ULA. systemd's "more specific rule wins" semantics - # mean public IPs hit the allow and listed ranges hit the deny. - # IPAddressDeny alone — no IPAddressAllow=any. Empirically, having both - # set causes the allow to win on this systemd/kernel combo regardless of - # the documented "more specific rule wins" behaviour. With only Deny, - # the kernel's default "allow all" applies to non-listed addresses. - assert "IPAddressDeny=" in text - assert "IPAddressAllow=any" not in text - # Explicit CIDRs — systemd-run's -p parser doesn't accept the - # `localhost` / `link-local` / `multicast` shorthand keywords that - # work in unit files (only the full strings parse). - for token in ( - "127.0.0.0/8", - "::1/128", - "169.254.0.0/16", - "fe80::/10", - "224.0.0.0/4", - "ff00::/8", - "10.0.0.0/8", - "172.16.0.0/12", - "192.168.0.0/16", - "100.64.0.0/10", - "fc00::/7", - ): - assert token in text, f"missing {token!r} in IPAddressDeny set" - - -def test_sandbox_resolv_conf_exists(): - assert SANDBOX_RESOLV_CONF.is_file() - text = SANDBOX_RESOLV_CONF.read_text() - nameservers = [ - line.split()[1] - for line in text.splitlines() - if line.startswith("nameserver ") - ] - assert len(nameservers) >= 2, "expected at least two nameservers for redundancy" - # Sanity: the resolvers must be public (not RFC1918 / loopback). We don't - # pin the exact IPs — Cloudflare/Google/Quad9 are all acceptable. - for ns in nameservers: - assert not ns.startswith("127."), ns - assert not ns.startswith("10."), ns - assert not ns.startswith("192.168."), ns - first_octet = int(ns.split(".")[0]) - # Reject 172.16.0.0/12. - if first_octet == 172: - second_octet = int(ns.split(".")[1]) - assert not (16 <= second_octet <= 31), ns - - -def test_deploy_script_installs_sandbox_resolv_conf(): - script = DEPLOY_SCRIPT.read_text() - assert "deploy/files/etc/left4me/sandbox-resolv.conf" in script - assert "/etc/left4me/sandbox-resolv.conf" in script - - -def test_script_sandbox_helper_validates_overlay_id(): - text = SCRIPT_SANDBOX_HELPER.read_text() - # Numeric-only overlay id - assert '[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]]' in text - # Overlay dir must exist - assert "/var/lib/left4me/overlays/" in text - assert "[[ -d $OVERLAY_DIR ]]" in text - # Script path must exist - assert "[[ -f $SCRIPT ]]" in text - - -def test_script_sandbox_helper_dry_run_mode(tmp_path): - overlay_root = tmp_path / "var/lib/left4me/overlays/42" - overlay_root.mkdir(parents=True) - fake_script = tmp_path / "fake.sh" - fake_script.write_text("echo hi") - - # Run in DRY_RUN mode against a fake l4d2-sandbox UID via a tiny shim that - # simulates `id -u l4d2-sandbox` resolving to a valid number. - helper_text = SCRIPT_SANDBOX_HELPER.read_text() - # We can't actually exec this without root + a real sandbox user; just - # verify the dry-run guard short-circuits before systemd-run / bwrap. - assert 'LEFT4ME_SCRIPT_SANDBOX_DRY_RUN' in helper_text - assert 'exit 0' in helper_text - - -def test_nft_mark_file_marks_left4me_udp_with_dscp_ef_and_priority(): - assert NFT_MARK_FILE.is_file() - text = NFT_MARK_FILE.read_text() - - # Own table in the inet family so it cannot conflict with operator nftables config. - assert "table inet left4me_mark" in text - assert "chain mangle_output" in text - assert "type filter hook output priority mangle" in text - - # Match by uid (every srcds runs as `left4me`) restricted to UDP. - assert 'meta skuid "left4me"' in text - assert "meta l4proto udp" in text - - # DSCP EF for both L3 families; in `inet` tables, `ip` only fires on v4 - # and `ip6` only on v6. - assert "ip dscp set ef" in text - assert "ip6 dscp set ef" in text - - # skb->priority class 6:0, set inline alongside DSCP. - assert "meta priority set 0006:0000" in text - - -def test_nft_mark_unit_loads_and_clears_left4me_table(): - assert NFT_MARK_UNIT.is_file() - text = NFT_MARK_UNIT.read_text() - - # Loads the rules early so the very first packet srcds emits is marked. - assert "After=network-pre.target" in text - assert "Before=network.target" in text - assert "Wants=network-pre.target" in text - - # Oneshot lifecycle: load on start, drop on stop. - assert "Type=oneshot" in text - assert "RemainAfterExit=yes" in text - assert ( - "ExecStart=/usr/sbin/nft -f /usr/local/lib/left4me/nft/left4me-mark.nft" - in text - ) - assert "ExecStop=/usr/sbin/nft delete table inet left4me_mark" in text - assert "WantedBy=multi-user.target" in text - - -def test_cake_env_template_documents_required_knobs(): - assert CAKE_ENV.is_file() - text = CAKE_ENV.read_text() - - # Both knobs are documented and present (commented OK; the deploy preserves - # operator edits, so the template must not bake in a wrong value). - assert "LEFT4ME_UPLINK_MBIT" in text - assert "LEFT4ME_UPLINK_IFACE" in text - # Empty defaults: shaper unit no-ops with a journal warning when unset. - assert "LEFT4ME_UPLINK_MBIT=" in text - assert "LEFT4ME_UPLINK_IFACE=" in text - - -def test_apply_cake_helper_supports_apply_and_clear_modes(): - assert APPLY_CAKE_HELPER.is_file() - text = APPLY_CAKE_HELPER.read_text() - - assert text.startswith("#!/bin/sh") - # Both knobs are read from the env file. - assert "LEFT4ME_UPLINK_MBIT" in text - assert "LEFT4ME_UPLINK_IFACE" in text - assert ". /etc/left4me/cake.env" in text - # Iface fallback to default route. - assert "ip -4 route show default" in text - # Two modes; default to apply. - assert "mode=${1:-apply}" in text - assert 'apply)' in text and 'clear)' in text - # Apply: idempotent `tc qdisc replace` with the documented flags. - assert "tc qdisc replace" in text - assert "cake" in text - assert "bandwidth" in text - assert "internet" in text - assert "diffserv4" in text - assert "dual-dsthost" in text - # Clear: tolerates a missing qdisc. - assert "tc qdisc del" in text - assert "|| true" in text - # Fail-soft on missing config. - assert "LEFT4ME_UPLINK_MBIT unset" in text - - -def test_apply_cake_helper_passes_shell_syntax_check(): - subprocess.run(["sh", "-n", str(APPLY_CAKE_HELPER)], check=True) - - -def test_cake_unit_runs_helper_in_apply_and_clear_modes(): - assert CAKE_UNIT.is_file() - text = CAKE_UNIT.read_text() - - assert "After=network-online.target" in text - assert "Wants=network-online.target" in text - assert "Type=oneshot" in text - assert "RemainAfterExit=yes" in text - # `-` prefix: missing env file is non-fatal (deploy ships one, but be safe). - assert "EnvironmentFile=-/etc/left4me/cake.env" in text - assert ( - "ExecStart=/usr/local/libexec/left4me/left4me-apply-cake apply" in text - ) - assert ( - "ExecStop=/usr/local/libexec/left4me/left4me-apply-cake clear" in text - ) - assert "WantedBy=multi-user.target" in text - - -def test_deploy_script_installs_network_shaping_artifacts(): - script = DEPLOY_SCRIPT.read_text() - - # nftables: package install on both apt and dnf paths. - apt_lines = [l for l in script.splitlines() if "apt-get install" in l] - dnf_lines = [l for l in script.splitlines() if "dnf install" in l] - assert apt_lines and dnf_lines - for line in apt_lines: - assert "nftables" in line, line - for line in dnf_lines: - assert "nftables" in line, line - - # nft rules + unit copied to system paths. - assert "/usr/local/lib/left4me/nft/left4me-mark.nft" in script - assert ( - "/usr/local/lib/systemd/system/left4me-nft-mark.service" in script - ) - assert "systemctl enable --now left4me-nft-mark.service" in script - - # CAKE helper + unit copied; helper made executable. - assert "/usr/local/libexec/left4me/left4me-apply-cake" in script - assert ( - "/usr/local/lib/systemd/system/left4me-cake.service" in script - ) - assert "chmod 0755" in script and "left4me-apply-cake" in script - assert "systemctl enable --now left4me-cake.service" in script - - # cake.env: copied only if absent (operator edits survive re-deploys). - assert "/etc/left4me/cake.env" in script - assert "[ -e /etc/left4me/cake.env ]" in script diff --git a/deploy/tests/test_example_units.py b/deploy/tests/test_example_units.py new file mode 100644 index 0000000..71f06bc --- /dev/null +++ b/deploy/tests/test_example_units.py @@ -0,0 +1,233 @@ +"""Lockdown tests for the curated examples kept under `deploy/files/`. + +`deploy/` is reference material. The production units are emitted by +ckn-bw's `systemd_units` reactor in `bundles/left4me/metadata.py`; +when reactor output drifts intentionally, update these examples to match. +""" +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +DEPLOY = ROOT / "deploy" + + +WEB_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-web.service" +SERVER_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-server@.service" +GAME_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-game.slice" +BUILD_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-build.slice" +SYSCTL_CONF = DEPLOY / "files/etc/sysctl.d/99-left4me.conf" +SANDBOX_RESOLV_CONF = DEPLOY / "files/etc/left4me/sandbox-resolv.conf" +HOST_ENV = DEPLOY / "templates/etc/left4me/host.env" +WEB_ENV_TEMPLATE = DEPLOY / "templates/etc/left4me/web.env.template" + + +def test_global_unit_files_exist_at_product_level_paths(): + assert WEB_UNIT.is_file() + assert SERVER_UNIT.is_file() + + +def test_web_unit_contains_required_runtime_contract(): + unit = WEB_UNIT.read_text() + + assert "User=left4me" in unit + assert "Group=left4me" in unit + assert "WorkingDirectory=/opt/left4me" in unit + assert "Environment=PATH=/opt/left4me/.venv/bin:" in unit + assert "EnvironmentFile=/etc/left4me/host.env" in unit + assert "EnvironmentFile=/etc/left4me/web.env" in unit + assert "ExecStart=/opt/left4me/.venv/bin/gunicorn" in unit + assert "--workers 1" in unit + assert "--threads 32" in unit + # NoNewPrivileges must remain unset because sudo (used by the overlay, + # systemctl and journalctl helpers) is setuid. + assert "NoNewPrivileges=true" not in unit + # Restored now that fuse-overlayfs propagation is no longer the mechanism. + assert "PrivateTmp=true" in unit + assert "ProtectSystem=full" in unit + assert "ReadWritePaths=/var/lib/left4me" in unit + # Mounts now happen in PID 1's namespace via the left4me-overlay helper, + # so MountFlags propagation is irrelevant — and the previous assumption + # that MountFlags=shared made it work was incorrect. + assert "MountFlags=" not in unit + + +def test_server_unit_contains_required_runtime_contract(): + unit = SERVER_UNIT.read_text() + + assert "User=left4me" in unit + assert "Group=left4me" in unit + assert "EnvironmentFile=/etc/left4me/host.env" in unit + assert "EnvironmentFile=/var/lib/left4me/instances/%i/instance.env" in unit + # `-` prefix: chdir failure is non-fatal so ExecStartPre can run the + # mount helper before the merged dir exists. ExecStart re-applies and + # finds the dir once the mount has landed. + assert "WorkingDirectory=-/var/lib/left4me/runtime/%i/merged/left4dead2" in unit + # ExecStart must invoke srcds_run from the *merged* overlay tree, not + # from installation/. srcds_run cds to its own dirname; if we point at + # installation/, the engine reads gameinfo.txt and addons from the lower + # layer and never sees overlay plugins (Metamod/SourceMod) or cfgs. + assert "ExecStart=/var/lib/left4me/runtime/%i/merged/srcds_run" in unit + assert "$L4D2_ARGS" in unit + assert "${L4D2_ARGS}" not in unit + assert "NoNewPrivileges=true" in unit + assert "PrivateTmp=true" in unit + assert "PrivateDevices=true" in unit + assert "ProtectHome=true" in unit + assert "ProtectSystem=strict" in unit + assert "ReadOnlyPaths=/var/lib/left4me/installation /var/lib/left4me/overlays" in unit + assert "ReadWritePaths=/var/lib/left4me/runtime/%i" in unit + assert "RestrictSUIDSGID=true" in unit + assert "LockPersonality=true" in unit + + +def test_server_unit_mounts_overlay_via_exec_start_pre(): + """At boot, systemd auto-starts enabled units before the web app gets a + chance to run start_instance's pre-start mount. The unit itself must + re-mount the overlay so reboots are transparent. Pairs with the helper's + idempotency check (test_overlay_helper_mount_is_idempotent_when_mounted). + + The unit-level `nsenter --mount=/proc/1/ns/mnt --` is what makes + umount fast: without it, the helper Python process would inherit + the unit's per-service mount namespace and pin it alive, blocking + PID 1's umount until the helper exited. Wrapping with nsenter at + the Exec line puts the helper itself in PID 1's namespace. + """ + unit = SERVER_UNIT.read_text() + # `+` prefix: runs as PID 1 (root, no sandbox). Required because + # the unit has NoNewPrivileges=true, which blocks sudo's setuid + # escalation — and the helper needs root for the mount syscall. + assert ( + "ExecStartPre=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- " + "/usr/local/libexec/left4me/left4me-overlay mount %i" + in unit + ) + # Bound the restart loop; without these, a CHDIR-failure (or any other + # pre-start error) spins indefinitely. + assert "StartLimitBurst=5" in unit + assert "StartLimitIntervalSec=60s" in unit + + +def test_server_unit_unmounts_overlay_via_exec_stop_post(): + """Single source of truth for unmount, mirroring the mount path. + ExecStopPost (not ExecStop) so it runs after srcds has fully exited + and the cgroup is cleared. + + Same nsenter-at-Exec-line wrapping as ExecStartPre — without it, + the helper process would itself hold a reference to the unit's + per-service mount namespace, and umount in PID 1 would loop on + EBUSY until the helper gave up. With it, umount succeeds first try. + """ + unit = SERVER_UNIT.read_text() + assert ( + "ExecStopPost=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- " + "/usr/local/libexec/left4me/left4me-overlay umount %i" + in unit + ) + + +def test_server_unit_contains_perf_baseline_directives(): + unit = SERVER_UNIT.read_text() + + # Slice membership. + assert "Slice=l4d2-game.slice" in unit + + # CFS priority bump (no SCHED_FIFO). + assert "Nice=-5" in unit + assert "CPUSchedulingPolicy=" not in unit + + # I/O priority. + assert "IOSchedulingClass=best-effort" in unit + assert "IOSchedulingPriority=4" in unit + + # OOM ordering: game servers survive, sandbox dies first. + assert "OOMScoreAdjust=-200" in unit + + # Memory caps with headroom for map-load spikes. + assert "MemoryHigh=1.5G" in unit + assert "MemoryMax=2G" in unit + + # Bounded fork surface. + assert "TasksMax=256" in unit + + # Plenty of fds for plugin-heavy setups. + assert "LimitNOFILE=65536" in unit + + # srcds clean shutdown via SIGINT, with time to flush. With the + # helper running in PID 1's mount namespace (via the unit-level + # nsenter on ExecStopPost), umount has no race window and the + # default 15 s is plenty for the whole stop transition. + assert "KillSignal=SIGINT" in unit + assert "TimeoutStopSec=15s" in unit + + # Per-unit override of journald rate limiting (default drops srcds output). + assert "LogRateLimitIntervalSec=0" in unit + + +def test_l4d2_game_slice_exists_with_high_weights(): + assert GAME_SLICE.is_file() + text = GAME_SLICE.read_text() + assert "[Slice]" in text + assert "CPUWeight=1000" in text + assert "IOWeight=1000" in text + + +def test_l4d2_build_slice_exists_with_low_weights(): + assert BUILD_SLICE.is_file() + text = BUILD_SLICE.read_text() + assert "[Slice]" in text + assert "CPUWeight=10" in text + assert "IOWeight=10" in text + + +def test_sysctl_conf_present_with_perf_settings(): + assert SYSCTL_CONF.is_file() + text = SYSCTL_CONF.read_text() + for line in ( + "net.core.rmem_max = 8388608", + "net.core.wmem_max = 8388608", + "net.core.rmem_default = 524288", + "net.core.wmem_default = 524288", + "net.core.netdev_max_backlog = 5000", + "net.core.netdev_budget = 600", + "vm.swappiness = 10", + "net.ipv4.udp_rmem_min = 16384", + "net.ipv4.udp_wmem_min = 16384", + "net.core.default_qdisc = fq_codel", + "net.ipv4.tcp_congestion_control = bbr", + ): + assert line in text, f"missing {line!r} in 99-left4me.conf" + + +def test_env_templates_contain_required_defaults(): + host_env = HOST_ENV.read_text() + assert "Deployment units use fixed /var/lib/left4me paths" in host_env + assert host_env.endswith("LEFT4ME_ROOT=/var/lib/left4me\n") + web_env = WEB_ENV_TEMPLATE.read_text() + assert web_env.startswith( + "DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n" + "SECRET_KEY=replace-with-generated-secret\n" + "JOB_WORKER_THREADS=4\n" + ) + assert web_env.rstrip().endswith("STEAM_WEB_API_KEY=") + + +def test_sandbox_resolv_conf_exists(): + assert SANDBOX_RESOLV_CONF.is_file() + text = SANDBOX_RESOLV_CONF.read_text() + nameservers = [ + line.split()[1] + for line in text.splitlines() + if line.startswith("nameserver ") + ] + assert len(nameservers) >= 2, "expected at least two nameservers for redundancy" + # Sanity: the resolvers must be public (not RFC1918 / loopback). We don't + # pin the exact IPs — Cloudflare/Google/Quad9 are all acceptable. + for ns in nameservers: + assert not ns.startswith("127."), ns + assert not ns.startswith("10."), ns + assert not ns.startswith("192.168."), ns + first_octet = int(ns.split(".")[0]) + # Reject 172.16.0.0/12. + if first_octet == 172: + second_octet = int(ns.split(".")[1]) + assert not (16 <= second_octet <= 31), ns diff --git a/l4d2host/tests/test_overlay_helper.py b/l4d2host/tests/test_overlay_helper.py index a098daf..48faf01 100644 --- a/l4d2host/tests/test_overlay_helper.py +++ b/l4d2host/tests/test_overlay_helper.py @@ -9,12 +9,8 @@ import pytest HELPER_SOURCE = ( Path(__file__).resolve().parents[2] - / "deploy" - / "files" - / "usr" - / "local" + / "scripts" / "libexec" - / "left4me" / "left4me-overlay" ) diff --git a/deploy/files/usr/local/libexec/left4me/left4me-journalctl b/scripts/libexec/left4me-journalctl similarity index 100% rename from deploy/files/usr/local/libexec/left4me/left4me-journalctl rename to scripts/libexec/left4me-journalctl diff --git a/deploy/files/usr/local/libexec/left4me/left4me-overlay b/scripts/libexec/left4me-overlay similarity index 100% rename from deploy/files/usr/local/libexec/left4me/left4me-overlay rename to scripts/libexec/left4me-overlay diff --git a/deploy/files/usr/local/libexec/left4me/left4me-script-sandbox b/scripts/libexec/left4me-script-sandbox similarity index 100% rename from deploy/files/usr/local/libexec/left4me/left4me-script-sandbox rename to scripts/libexec/left4me-script-sandbox diff --git a/deploy/files/usr/local/libexec/left4me/left4me-systemctl b/scripts/libexec/left4me-systemctl similarity index 100% rename from deploy/files/usr/local/libexec/left4me/left4me-systemctl rename to scripts/libexec/left4me-systemctl diff --git a/deploy/files/usr/local/sbin/left4me b/scripts/sbin/left4me similarity index 100% rename from deploy/files/usr/local/sbin/left4me rename to scripts/sbin/left4me diff --git a/scripts/tests/conftest.py b/scripts/tests/conftest.py new file mode 100644 index 0000000..cf89185 --- /dev/null +++ b/scripts/tests/conftest.py @@ -0,0 +1,36 @@ +"""Shared fixtures and path constants for `scripts/tests/`.""" +import os +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +SCRIPTS = ROOT / "scripts" +LIBEXEC = SCRIPTS / "libexec" +SBIN = SCRIPTS / "sbin" + +# `deploy/` is reference material; the sudoers example lives there and is +# the canonical statement of which paths sudo grants to the `left4me` uid. +# `scripts/tests/test_sudoers_grants.py` reads it from across the dir +# boundary because the contract being audited is about *scripts*, not deploy. +DEPLOY = ROOT / "deploy" + + +def fake_command(tmp_path, command_name): + """Drop a no-op stub of `command_name` into `tmp_path`. Returns the + marker file the stub writes its args to, so tests can assert that the + helper rejected bad input before invoking the real command. + """ + marker = tmp_path / f"{command_name}.args" + command = tmp_path / command_name + command.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" > '{marker}'\nexit 0\n") + command.chmod(0o755) + return marker + + +def env_with_fake_commands(tmp_path): + """Build an environment that prepends `tmp_path` onto PATH so helpers + find the fake commands first. + """ + env = os.environ.copy() + env["PATH"] = f"{tmp_path}{os.pathsep}{env.get('PATH', '')}" + return env diff --git a/scripts/tests/test_helpers_use_fixed_paths.py b/scripts/tests/test_helpers_use_fixed_paths.py new file mode 100644 index 0000000..6f80f36 --- /dev/null +++ b/scripts/tests/test_helpers_use_fixed_paths.py @@ -0,0 +1,15 @@ +from conftest import LIBEXEC + + +SYSTEMCTL_HELPER = LIBEXEC / "left4me-systemctl" +JOURNALCTL_HELPER = LIBEXEC / "left4me-journalctl" + + +def test_helpers_use_fixed_system_tool_paths_not_sudo_path(): + systemctl = SYSTEMCTL_HELPER.read_text() + journalctl = JOURNALCTL_HELPER.read_text() + + assert "command -v systemctl" not in systemctl + assert "command -v journalctl" not in journalctl + assert "/bin/systemctl" in systemctl or "/usr/bin/systemctl" in systemctl + assert "/bin/journalctl" in journalctl or "/usr/bin/journalctl" in journalctl diff --git a/scripts/tests/test_journalctl_helper.py b/scripts/tests/test_journalctl_helper.py new file mode 100644 index 0000000..8566518 --- /dev/null +++ b/scripts/tests/test_journalctl_helper.py @@ -0,0 +1,31 @@ +import subprocess + +from conftest import LIBEXEC, env_with_fake_commands, fake_command + + +JOURNALCTL_HELPER = LIBEXEC / "left4me-journalctl" + + +def test_journalctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path): + subprocess.run(["sh", "-n", str(JOURNALCTL_HELPER)], check=True) + marker = fake_command(tmp_path, "journalctl") + + for args in [ + ["../evil", "--lines", "25", "--no-follow"], + ["alpha", "--bad", "25", "--no-follow"], + ["alpha", "--lines", "not-number", "--no-follow"], + ["alpha", "--lines", "25", "--bad-follow"], + ["bad/name", "--lines", "25", "--no-follow"], + ]: + result = subprocess.run( + ["sh", str(JOURNALCTL_HELPER), *args], + env=env_with_fake_commands(tmp_path), + check=False, + ) + assert result.returncode != 0 + assert not marker.exists() + + script = JOURNALCTL_HELPER.read_text() + assert 'unit="left4me-server@${name}.service"' in script + assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat "$follow_arg"' in script + assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat' in script diff --git a/scripts/tests/test_overlay.py b/scripts/tests/test_overlay.py new file mode 100644 index 0000000..a74e32e --- /dev/null +++ b/scripts/tests/test_overlay.py @@ -0,0 +1,32 @@ +from conftest import LIBEXEC + + +OVERLAY_HELPER = LIBEXEC / "left4me-overlay" + + +def test_overlay_helper_is_python_with_strict_validation(): + text = OVERLAY_HELPER.read_text() + assert text.startswith("#!/usr/bin/python3") + # Validation surface + assert "NAME_RE = re.compile" in text + assert "LOWERDIR_ALLOWLIST" in text + assert "user.fuseoverlayfs." in text + assert "MAX_LOWERDIRS = 500" in text + # Mounts via PID 1's mount namespace + assert "/proc/1/ns/mnt" in text + assert "nsenter" in text + # Verbs are mount and umount (not unmount) + assert '"mount"' in text and '"umount"' in text + assert '"unmount"' not in text + + +def test_overlay_helper_mount_is_idempotent_when_already_mounted(): + """ExecStartPre runs on every Restart=on-failure cycle. If a previous + start mounted successfully but ExecStart failed afterwards, the next + ExecStartPre would re-mount on top -- which fails. The helper must + short-circuit when merged is already a mount point. + """ + text = OVERLAY_HELPER.read_text() + # Two ismount checks now: one in cmd_mount (skip if mounted), + # one in cmd_umount (skip if not mounted). + assert text.count("os.path.ismount") >= 2 diff --git a/scripts/tests/test_script_sandbox.py b/scripts/tests/test_script_sandbox.py new file mode 100644 index 0000000..bddb1f3 --- /dev/null +++ b/scripts/tests/test_script_sandbox.py @@ -0,0 +1,171 @@ +import subprocess + +from conftest import LIBEXEC + + +SCRIPT_SANDBOX_HELPER = LIBEXEC / "left4me-script-sandbox" + + +def test_script_sandbox_helper_present(): + assert SCRIPT_SANDBOX_HELPER.is_file() + assert SCRIPT_SANDBOX_HELPER.read_text().startswith("#!/bin/bash") + mode = SCRIPT_SANDBOX_HELPER.stat().st_mode & 0o777 + assert mode == 0o755, f"expected 0755, got {oct(mode)}" + + +def test_script_sandbox_helper_passes_shell_syntax_check(): + subprocess.run(["bash", "-n", str(SCRIPT_SANDBOX_HELPER)], check=True) + + +def test_script_sandbox_helper_invokes_systemd_run_with_hardening(): + text = SCRIPT_SANDBOX_HELPER.read_text() + + # systemd-run service mode (no --scope), with synchronous I/O to caller. + assert "systemd-run" in text + assert "--scope" not in text, "v2 uses transient service units, not scopes" + assert "--pipe" in text + assert "--wait" in text + assert "--collect" in text + assert "--unit=" in text + + # No bwrap. + assert "bwrap" not in text + assert "bubblewrap" not in text + + # UID drop via systemd directives. + assert "User=l4d2-sandbox" in text + assert "Group=l4d2-sandbox" in text + + # Cgroup limits unchanged from v1. + assert "MemoryMax=4G" in text + assert "MemorySwapMax=0" in text + assert "TasksMax=512" in text + assert "CPUQuota=200%" in text + assert "RuntimeMaxSec=3600" in text + + # Hardening directives that v1 (scope mode) couldn't carry. + assert "NoNewPrivileges=yes" in text + assert "ProtectSystem=strict" in text + assert "ProtectHome=yes" in text + assert "PrivateTmp=yes" in text + assert "PrivateDevices=yes" in text + assert "PrivateIPC=yes" in text + assert "ProtectKernelTunables=yes" in text + assert "ProtectKernelModules=yes" in text + assert "ProtectKernelLogs=yes" in text + assert "ProtectControlGroups=yes" in text + assert "RestrictNamespaces=yes" in text + assert "RestrictSUIDSGID=yes" in text + assert "LockPersonality=yes" in text + assert "MemoryDenyWriteExecute=yes" in text + assert "SystemCallFilter=" in text + assert "@system-service" in text + assert "@network-io" in text + assert "CapabilityBoundingSet=" in text + assert "AmbientCapabilities=" in text + assert 'RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX"' in text + + # Network namespace stays shared with host. + assert "PrivateNetwork=" not in text + + # Mount setup: /etc and /var/lib masked with tmpfs; selective binds back. + assert 'TemporaryFileSystem="/etc /var/lib"' in text + assert "BindReadOnlyPaths=" in text + # The resolv.conf bind points at the sandbox-only file (not the host's + # /etc/resolv.conf, which typically references a private-IP DNS server + # that IPAddressDeny= blocks). + assert "/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf" in text + assert "/etc/ssl" in text + assert "/etc/ca-certificates" in text + assert "/etc/nsswitch.conf" in text + assert "/etc/alternatives" in text + assert "${SCRIPT}:/script.sh" in text + assert 'BindPaths="${STAGING}:/overlay"' in text + + # IP egress filter: allow public, deny localhost / RFC1918 / link-local / + # multicast / CGNAT / ULA. systemd's "more specific rule wins" semantics + # mean public IPs hit the allow and listed ranges hit the deny. + # IPAddressDeny alone — no IPAddressAllow=any. Empirically, having both + # set causes the allow to win on this systemd/kernel combo regardless of + # the documented "more specific rule wins" behaviour. With only Deny, + # the kernel's default "allow all" applies to non-listed addresses. + assert "IPAddressDeny=" in text + assert "IPAddressAllow=any" not in text + # Explicit CIDRs — systemd-run's -p parser doesn't accept the + # `localhost` / `link-local` / `multicast` shorthand keywords that + # work in unit files (only the full strings parse). + for token in ( + "127.0.0.0/8", + "::1/128", + "169.254.0.0/16", + "fe80::/10", + "224.0.0.0/4", + "ff00::/8", + "10.0.0.0/8", + "172.16.0.0/12", + "192.168.0.0/16", + "100.64.0.0/10", + "fc00::/7", + ): + assert token in text, f"missing {token!r} in IPAddressDeny set" + + +def test_script_sandbox_uses_idmap_staging(): + """The sandbox runs as l4d2-sandbox but writes need to land on disk as + left4me, so all overlay content (workshop + script-built) is uniformly + left4me-owned. The helper pre-creates an idmapped bind on a staging + path and points the sandbox's BindPaths at the staging, not at the raw + overlay dir. trap cleans up the staging bind on exit. + """ + text = SCRIPT_SANDBOX_HELPER.read_text() + # Idmap mount setup uses --map-users / --map-groups. + assert "--map-users=" in text + assert "--map-groups=" in text + # Staging path lives under /var/lib/left4me/tmp/sandbox-idmap-. + assert "/var/lib/left4me/tmp/sandbox-idmap-" in text + # BindPaths into the sandbox points at the staging path, not the + # raw overlay dir. + assert 'BindPaths="${STAGING}:/overlay"' in text + # trap registers cleanup so the staging bind doesn't outlive the helper. + assert "trap " in text and "cleanup_staging" in text + # The previous chown-to-l4d2-sandbox approach is gone; overlay dirs + # stay left4me-owned end-to-end. + assert "chown -R l4d2-sandbox" not in text + + +def test_script_sandbox_in_build_slice_with_oom_adjust(): + text = SCRIPT_SANDBOX_HELPER.read_text() + + # Put the transient unit in the low-weight build slice so it yields to + # game-server instances under CPU/IO contention. + assert "--slice=l4d2-build.slice" in text + + # Sandbox dies first if the host hits memory pressure; servers + # (OOMScoreAdjust=-200) survive. + assert "-p OOMScoreAdjust=500" in text + + +def test_script_sandbox_helper_validates_overlay_id(): + text = SCRIPT_SANDBOX_HELPER.read_text() + # Numeric-only overlay id + assert '[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]]' in text + # Overlay dir must exist + assert "/var/lib/left4me/overlays/" in text + assert "[[ -d $OVERLAY_DIR ]]" in text + # Script path must exist + assert "[[ -f $SCRIPT ]]" in text + + +def test_script_sandbox_helper_dry_run_mode(tmp_path): + overlay_root = tmp_path / "var/lib/left4me/overlays/42" + overlay_root.mkdir(parents=True) + fake_script = tmp_path / "fake.sh" + fake_script.write_text("echo hi") + + # Run in DRY_RUN mode against a fake l4d2-sandbox UID via a tiny shim that + # simulates `id -u l4d2-sandbox` resolving to a valid number. + helper_text = SCRIPT_SANDBOX_HELPER.read_text() + # We can't actually exec this without root + a real sandbox user; just + # verify the dry-run guard short-circuits before systemd-run / bwrap. + assert 'LEFT4ME_SCRIPT_SANDBOX_DRY_RUN' in helper_text + assert 'exit 0' in helper_text diff --git a/scripts/tests/test_sudoers_grants.py b/scripts/tests/test_sudoers_grants.py new file mode 100644 index 0000000..d96283f --- /dev/null +++ b/scripts/tests/test_sudoers_grants.py @@ -0,0 +1,38 @@ +"""Audit the script→sudoers contract. + +The sudoers file in `deploy/files/etc/sudoers.d/left4me` is a reference +example; ckn-bw ships its own verbatim copy under +`bundles/left4me/files/etc/sudoers.d/left4me`. The two are expected to +match. This test lives under `scripts/tests/` because the contract being +audited is about *scripts* (which paths the `left4me` uid can sudo into) +even though the file it reads is in `deploy/`. +""" +from conftest import DEPLOY + + +SUDOERS = DEPLOY / "files/etc/sudoers.d/left4me" + + +def test_sudoers_allows_only_left4me_helpers_not_raw_system_tools(): + sudoers = SUDOERS.read_text() + + assert ( + "left4me ALL=(root) NOPASSWD: " + "/usr/local/libexec/left4me/left4me-systemctl *" + ) in sudoers + assert ( + "left4me ALL=(root) NOPASSWD: " + "/usr/local/libexec/left4me/left4me-journalctl *" + ) in sudoers + assert "/usr/local/libexec/left4me/left4me-overlay mount *" in sudoers + assert "/usr/local/libexec/left4me/left4me-overlay umount *" in sudoers + assert ( + "left4me ALL=(root) NOPASSWD: " + "/usr/local/libexec/left4me/left4me-script-sandbox" + ) in sudoers + assert "/bin/systemctl" not in sudoers + assert "/usr/bin/systemctl" not in sudoers + assert "/bin/journalctl" not in sudoers + assert "/usr/bin/journalctl" not in sudoers + assert "/bin/mount" not in sudoers + assert "/bin/umount" not in sudoers diff --git a/scripts/tests/test_systemctl_helper.py b/scripts/tests/test_systemctl_helper.py new file mode 100644 index 0000000..0e669c8 --- /dev/null +++ b/scripts/tests/test_systemctl_helper.py @@ -0,0 +1,39 @@ +import subprocess + +from conftest import LIBEXEC, env_with_fake_commands, fake_command + + +SYSTEMCTL_HELPER = LIBEXEC / "left4me-systemctl" + + +def test_systemctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path): + subprocess.run(["sh", "-n", str(SYSTEMCTL_HELPER)], check=True) + marker = fake_command(tmp_path, "systemctl") + + for args in [ + ["bad/action", "alpha"], + # `start` and `stop` are no longer accepted verbs — the lifecycle now + # uses `enable`/`disable` for reboot survival via WantedBy= symlinks. + ["start", "alpha"], + ["stop", "alpha"], + ["enable", ""], + ["enable", ".hidden"], + ["enable", "bad..name"], + ["enable", "bad/name"], + ["enable", "bad\\name"], + ["enable", "bad name"], + ]: + result = subprocess.run( + ["sh", str(SYSTEMCTL_HELPER), *args], + env=env_with_fake_commands(tmp_path), + check=False, + ) + assert result.returncode != 0 + assert not marker.exists() + + script = SYSTEMCTL_HELPER.read_text() + assert 'unit="left4me-server@${name}.service"' in script + assert 'enable) exec "$systemctl" enable --now "$unit"' in script + assert 'disable) exec "$systemctl" disable --now "$unit"' in script + assert "--property=ActiveState" in script + assert "--property=SubState" in script