refactor: move privileged scripts to scripts/{libexec,sbin}/; deploy/ is reference
Pulls the 5 privileged helpers out of deploy/files/usr/local/{libexec,sbin}/
into top-level scripts/{libexec,sbin}/. They are application-inherent code
(invoked at runtime via sudo from l4d2host/l4d2web), not deploy artifacts —
the previous nesting under deploy/files/ confused source-of-truth with
install-target FHS layout.
deploy/ now means "reference exemplar": README explaining the target
layout, plus example sudoers / sysctl / sandbox-resolv.conf / env
templates / curated systemd units (the ones ckn-bw's reactor emits).
Anyone building a fresh deployment (other than ckn-bw) reads this tree.
Dead static artifacts deleted: left4me-apply-cake helper, left4me-cake
+ left4me-nft-mark service units, cake.env, left4me-mark.nft, and the
superseded deploy-test-server.sh installer.
Tests split to match the new shape:
- scripts/tests/{test_overlay,test_script_sandbox,test_systemctl_helper,
test_journalctl_helper,test_helpers_use_fixed_paths,test_sudoers_grants}.py
with shared fixtures in conftest.py
- deploy/tests/test_example_units.py (renamed from test_deploy_artifacts.py)
— slimmed to lock down the curated example units, sysctl, env templates
l4d2host/tests/test_overlay_helper.py: helper-source path updated to
scripts/libexec/left4me-overlay (was building the path segment-by-segment
under deploy/files/, missed by the path-prefix grep during pre-flight).
Runtime install-target paths (/usr/local/{libexec,sbin}/) unchanged, so
l4d2host/service_control.py, l4d2web/services/overlay_builders.py, the
sudoers grants, and the systemd units all keep their existing path
references.
Requires the matching ckn-bw change to bundles/left4me/items.py
(install_left4me_scripts repointed from /opt/left4me/src/deploy/files/...
to /opt/left4me/src/scripts/...). Left4me lands first so a fresh
git_deploy exposes the new source path before the bundle apply runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
mwiegand
parent
e38b844978
commit
5284e28af7
22 changed files with 776 additions and 1517 deletions
355
deploy/README.md
355
deploy/README.md
|
|
@ -1,117 +1,126 @@
|
|||
# left4me Deployment
|
||||
# left4me deploy — reference exemplar
|
||||
|
||||
> Production provisioning of left4me on `ovh.left4me` is driven by
|
||||
> [ckn-bw](https://git.sublimity.de/cronekorkn/ckn-bw)
|
||||
> (`bundles/left4me/`, attached via `groups/applications/left4me.py`).
|
||||
> Run `bw apply ovh.left4me` from the ckn-bw repo to deploy.
|
||||
> **This directory is reference material, not the source of truth.**
|
||||
> The canonical deploy of `ovh.left4me` is driven by
|
||||
> [ckn-bw](https://git.sublimity.de/cronekorkn/ckn-bw)'s `bundles/left4me/`
|
||||
> (attached via `groups/applications/left4me.py`); run `bw apply ovh.left4me`
|
||||
> from the ckn-bw repo to deploy.
|
||||
>
|
||||
> **What's canonical in this directory** (`deploy/files/`, `deploy/templates/`,
|
||||
> `deploy/tests/`): the actual file payload ckn-bw deploys. ckn-bw fetches
|
||||
> the left4me repo via `git_deploy` to `/opt/left4me/src/` and `install`s
|
||||
> the privileged scripts from `deploy/files/usr/local/{libexec,sbin}/`
|
||||
> directly onto the target. Sudoers, sysctl, and env-template content
|
||||
> ships from `deploy/files/etc/` and `deploy/templates/etc/`. **Edit
|
||||
> these files here; ckn-bw picks them up on the next apply.** No
|
||||
> duplicate copy of the file content lives in ckn-bw.
|
||||
> The privileged scripts the application installs live at the repo root
|
||||
> under [`scripts/libexec/`](../scripts/libexec/) and
|
||||
> [`scripts/sbin/`](../scripts/sbin/) — application code, not deploy
|
||||
> artifacts. ckn-bw's `install_left4me_scripts` action reads them from
|
||||
> `/opt/left4me/src/scripts/{libexec,sbin}/` after `git_deploy` and
|
||||
> installs them into the standard FHS targets on the host.
|
||||
>
|
||||
> **What's superseded**: the `deploy-test-server.sh` script — an older
|
||||
> one-shot bash deploy that ckn-bw replaced. It's kept as a readable
|
||||
> description of the install steps the bundle now performs declaratively.
|
||||
> Don't run it against an ovh.left4me node managed by ckn-bw; the two
|
||||
> would fight over file ownership.
|
||||
>
|
||||
> **What's obsolete** (kept for greppability, not currently used): CAKE
|
||||
> traffic shaping (now in systemd-networkd via `network/<iface>/cake`
|
||||
> metadata in ckn-bw), nft marking (now in the central `nftables/output`
|
||||
> set), and the systemd unit files under `files/usr/local/lib/systemd/system/`
|
||||
> (emitted by the bundle's `systemd_units` reactor instead of being shipped
|
||||
> as static files). The obsolete bits stay here intact so the original
|
||||
> choices and tradeoffs remain greppable.
|
||||
> What remains under `deploy/files/` and `deploy/templates/` is a set of
|
||||
> readable **examples** — sudoers, sysctl, sandbox-resolv.conf, env
|
||||
> templates, and a curated subset of the systemd units ckn-bw's reactor
|
||||
> emits at apply time. They exist so a fresh consumer (other than ckn-bw)
|
||||
> could read this tree and assemble an equivalent deployment. They are
|
||||
> **not** the bytes ckn-bw installs; ckn-bw carries its own copies of the
|
||||
> verbatim configs in `bundles/left4me/files/etc/`, and emits the live
|
||||
> units from `bundles/left4me/metadata.py`'s `systemd_units` reactor.
|
||||
|
||||
## What lives here (and what corresponds to it in ckn-bw)
|
||||
## What's here
|
||||
|
||||
| Path here | Status under ckn-bw |
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| `deploy-test-server.sh` | replaced by `bw apply` |
|
||||
| `files/etc/sudoers.d/left4me` | shipped verbatim by `bundles/left4me/files/etc/sudoers.d/left4me` (validated with `visudo -cf` via `test_with`) |
|
||||
| `files/etc/sysctl.d/99-left4me.conf` | shipped verbatim by the bundle |
|
||||
| `files/etc/left4me/sandbox-resolv.conf` | shipped verbatim by the bundle |
|
||||
| `files/usr/local/libexec/left4me/{left4me-systemctl,journalctl,overlay,script-sandbox}` | installed onto the target by the `install_left4me_scripts` action in `bundles/left4me/items.py`, reading directly from `/opt/left4me/src/deploy/files/usr/local/libexec/left4me/` after `git_deploy`. The bundle does **not** carry a duplicate copy. |
|
||||
| `files/usr/local/sbin/left4me` | same install action; admin CLI wrapper (`sudo left4me <flask-subcommand>`) |
|
||||
| `files/usr/local/lib/systemd/system/left4me-web.service` | emitted by `systemd_units` reactor in `bundles/left4me/metadata.py` (intentional change: `--bind 0.0.0.0:8000` → `127.0.0.1:8000` because nginx now terminates TLS) |
|
||||
| `files/usr/local/lib/systemd/system/left4me-server@.service` | emitted by the same reactor |
|
||||
| `files/usr/local/lib/systemd/system/{l4d2-game,l4d2-build}.slice` | emitted by the same reactor |
|
||||
| `files/usr/local/lib/systemd/system/left4me-cake.service` | **obsolete** — CAKE applied via systemd-networkd (`network/<iface>/cake` metadata in `bundles/network/`) |
|
||||
| `files/usr/local/libexec/left4me/left4me-apply-cake` | **obsolete** — same as above |
|
||||
| `files/etc/left4me/cake.env` | **obsolete** — bandwidth lives in node metadata under `network/external/cake/Bandwidth` |
|
||||
| `files/usr/local/lib/systemd/system/left4me-nft-mark.service` | **obsolete** — central `bundles/nftables/` consumes the rules from `bundles/left4me/`'s defaults |
|
||||
| `files/usr/local/lib/left4me/nft/left4me-mark.nft` | **obsolete** — same as above |
|
||||
| `templates/etc/left4me/host.env` | rendered as Mako by `bundles/left4me/files/etc/left4me/host.env.mako` |
|
||||
| `templates/etc/left4me/web.env.template` | rendered as Mako by `bundles/left4me/files/etc/left4me/web.env.mako` (intentional change: `SESSION_COOKIE_SECURE=false` → `true`, plus `LEFT4ME_PORT_RANGE_*` are now wired through) |
|
||||
| First-run admin bootstrap (`flask create-user … --admin` near the end of `deploy-test-server.sh`) | manual one-time step after `bw apply`; the bundle deliberately doesn't seed an admin to keep credentials out of the metadata pipeline |
|
||||
| CPU isolation drop-ins (`/etc/systemd/system/{system,user,l4d2-game,l4d2-build}.slice.d/99-left4me-cpuset.conf`) | **not managed by the bundle** — generated dynamically based on `nproc --all` in the script; that logic doesn't fit static bundle metadata, apply manually post-deploy if needed |
|
||||
| `files/etc/sudoers.d/left4me` | Example sudoers grants. Lockdown test: `scripts/tests/test_sudoers_grants.py`. |
|
||||
| `files/etc/sysctl.d/99-left4me.conf` | Example sysctl perf baseline (UDP buffers, fq_codel + BBR). |
|
||||
| `files/etc/left4me/sandbox-resolv.conf` | Example `/etc/resolv.conf` bound into the script-overlay sandbox. |
|
||||
| `files/usr/local/lib/systemd/system/left4me-web.service` | Example of the web-app unit the reactor emits. |
|
||||
| `files/usr/local/lib/systemd/system/left4me-server@.service` | Example of the per-instance gameserver unit. |
|
||||
| `files/usr/local/lib/systemd/system/left4me-workshop-refresh.{service,timer}` | Example of the daily workshop-refresh cron-equivalent. |
|
||||
| `files/usr/local/lib/systemd/system/l4d2-{game,build}.slice` | Example slice definitions (CPU/IO weights). |
|
||||
| `templates/etc/left4me/host.env` | Example host-library env (deployment-fixed paths). |
|
||||
| `templates/etc/left4me/web.env.template` | Example web-app env. ckn-bw renders the real version via the matching Mako template in `bundles/left4me/files/etc/left4me/web.env.mako`. |
|
||||
| `tests/test_example_units.py` | Locks down the example units & env templates above. |
|
||||
|
||||
---
|
||||
The privileged scripts (`left4me-overlay`, `left4me-script-sandbox`,
|
||||
`left4me-systemctl`, `left4me-journalctl`, `sbin/left4me`) used to live
|
||||
under this tree at `files/usr/local/{libexec,sbin}/`; they moved to
|
||||
`scripts/{libexec,sbin}/` because they are application code, not deploy
|
||||
artifacts.
|
||||
|
||||
## Original notes (still accurate as a description of the install steps)
|
||||
## Target layout
|
||||
|
||||
This directory contains the production-like test deployment for a Linux server. It installs the repository into a fixed host layout, configures a dedicated runtime user, installs systemd units, and wires the web app to host operations through privileged helper commands.
|
||||
The deployment uses these on-host paths (FHS-aligned):
|
||||
|
||||
## Target Layout
|
||||
- `/etc/left4me/host.env` — host library environment configuration.
|
||||
- `/etc/left4me/web.env` — web app environment configuration.
|
||||
- `/etc/left4me/sandbox-resolv.conf` — DNS resolv.conf bound into the
|
||||
script-overlay sandbox.
|
||||
- `/etc/sudoers.d/left4me` — sudoers rules letting the `left4me` uid call
|
||||
the privileged helpers non-interactively.
|
||||
- `/etc/sysctl.d/99-left4me.conf` — perf-baseline sysctls.
|
||||
- `/opt/left4me` — deployed repository contents (via ckn-bw `git_deploy`).
|
||||
- `/opt/left4me/.venv` — Python virtual environment for the web app.
|
||||
- `/var/lib/left4me/left4me.db` — SQLite database used by the web app.
|
||||
- `/var/lib/left4me/installation` — shared L4D2 installation.
|
||||
- `/var/lib/left4me/overlays` — overlay directories. Each overlay lives
|
||||
at `${overlay_id}` under here.
|
||||
- `/var/lib/left4me/workshop_cache` — deduplicated cache of `.vpk` files
|
||||
downloaded for workshop overlays. One file per Steam item, named
|
||||
`{steam_id}.vpk`. Workshop overlays symlink into this tree.
|
||||
- `/var/lib/left4me/instances` — rendered instance specifications and
|
||||
per-instance state.
|
||||
- `/var/lib/left4me/runtime` — per-instance runtime mount directories.
|
||||
- `/var/lib/left4me/tmp` — temporary files used by deployment/runtime
|
||||
operations (incl. idmap staging binds).
|
||||
- `/usr/local/lib/systemd/system/` — global systemd unit files emitted
|
||||
by ckn-bw's `systemd_units` reactor.
|
||||
- `/usr/local/libexec/left4me/` — privileged helper commands installed
|
||||
from `scripts/libexec/`.
|
||||
- `/usr/local/sbin/left4me` — admin CLI wrapper installed from
|
||||
`scripts/sbin/left4me`.
|
||||
|
||||
The deployment uses these paths:
|
||||
## Runtime users
|
||||
|
||||
- `/etc/left4me/host.env`: host library environment configuration.
|
||||
- `/etc/left4me/web.env`: web app environment configuration.
|
||||
- `/opt/left4me/.venv`: Python virtual environment for deployed commands.
|
||||
- `/opt/left4me`: deployed repository contents.
|
||||
- `/var/lib/left4me/left4me.db`: SQLite database used by the web app.
|
||||
- `/var/lib/left4me/installation`: shared L4D2 installation.
|
||||
- `/var/lib/left4me/overlays`: overlay directories. Each overlay lives at `${overlay_id}` under here.
|
||||
- `/var/lib/left4me/workshop_cache`: deduplicated cache of `.vpk` files downloaded for workshop overlays. One file per Steam item, named `{steam_id}.vpk`. Workshop overlays symlink into this tree.
|
||||
- `/var/lib/left4me/global_overlay_cache`: cache of non-Steam map archives and extracted `.vpk` files used by managed global map overlays.
|
||||
- `/var/lib/left4me/instances`: rendered instance specifications and per-instance state.
|
||||
- `/var/lib/left4me/runtime`: per-instance runtime mount directories.
|
||||
- `/var/lib/left4me/tmp`: temporary files used by deployment/runtime operations.
|
||||
- `/usr/local/lib/systemd/system`: global systemd unit files, including `left4me-server@.service`.
|
||||
- `/usr/local/libexec/left4me`: privileged helper commands, including `left4me-systemctl`, `left4me-journalctl`, and `left4me-overlay` (the latter mounts the per-instance kernel overlay in PID 1's mount namespace via `nsenter`).
|
||||
- `/etc/sudoers.d/left4me`: sudoers rules allowing the web/runtime commands to call the helpers non-interactively.
|
||||
Two system users are involved:
|
||||
|
||||
Static units are generated for `/var/lib/left4me`. If `LEFT4ME_ROOT` changes, regenerate and reinstall the unit files instead of reusing the existing static units.
|
||||
- **`left4me`** (home `/var/lib/left4me`, shell `/usr/sbin/nologin`):
|
||||
web app, host library, and gameserver runtime.
|
||||
- **`l4d2-sandbox`** (no home, shell `/usr/sbin/nologin`): unprivileged
|
||||
uid the script-overlay sandbox drops into via `systemd-run`. The
|
||||
`left4me-script-sandbox` helper sets up an idmapped bind from the
|
||||
sandbox uid back to `left4me` on a staging path so overlay writes
|
||||
land on disk as `left4me`-owned. The split is load-bearing: a
|
||||
sandbox escape would otherwise see `web.env`, the SQLite DB, and
|
||||
running gameservers.
|
||||
|
||||
## Runtime User
|
||||
(Whether the gameserver runtime should be split off into a third uid is
|
||||
an open design question — see
|
||||
`docs/superpowers/specs/2026-05-15-user-uid-split-design.md`.)
|
||||
|
||||
The deployment creates and runs host operations as the dedicated runtime user:
|
||||
## Deployment
|
||||
|
||||
- Username: `left4me`
|
||||
- Home: `/var/lib/left4me`
|
||||
- Shell: `/usr/sbin/nologin`
|
||||
Production deploy:
|
||||
|
||||
## Running A Test Deployment
|
||||
|
||||
Run the deployment from the repository root:
|
||||
|
||||
```bash
|
||||
deploy/deploy-test-server.sh deploy-user@example-host
|
||||
```sh
|
||||
# In the ckn-bw repo:
|
||||
bw apply ovh.left4me
|
||||
```
|
||||
|
||||
The SSH user must be able to run `sudo` on the target host. The deployment configures system packages, directories, environment files, helper scripts, sudoers rules, Python dependencies, and systemd units.
|
||||
Admin bootstrap is a manual one-time step after the first apply
|
||||
(ckn-bw deliberately doesn't seed an admin to keep credentials out of
|
||||
the metadata pipeline):
|
||||
|
||||
## Admin Bootstrap
|
||||
|
||||
Set the bootstrap credentials in the environment when creating the first admin user:
|
||||
|
||||
```bash
|
||||
LEFT4ME_ADMIN_USERNAME=admin \
|
||||
```sh
|
||||
sudo -u left4me LEFT4ME_ADMIN_USERNAME=admin \
|
||||
LEFT4ME_ADMIN_PASSWORD='change-me' \
|
||||
flask create-user "$LEFT4ME_ADMIN_USERNAME" --admin
|
||||
/opt/left4me/.venv/bin/flask --app l4d2web.app:create_app \
|
||||
create-user "$LEFT4ME_ADMIN_USERNAME" --admin
|
||||
```
|
||||
|
||||
Use a strong one-time password and rotate it after first login if needed.
|
||||
Rotate the bootstrap password after first login.
|
||||
|
||||
## Overlay References
|
||||
## Overlay references
|
||||
|
||||
Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`. With the default deployment root, they resolve under `/var/lib/left4me/overlays`. New overlays use `${overlay_id}` as their path; the digit-only form is the only one created by the web app.
|
||||
Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`.
|
||||
With the default deployment root, they resolve under
|
||||
`/var/lib/left4me/overlays`. New overlays use `${overlay_id}` as their
|
||||
path; the digit-only form is the only one created by the web app.
|
||||
|
||||
Invalid references are rejected:
|
||||
|
||||
|
|
@ -122,125 +131,117 @@ Invalid references are rejected:
|
|||
|
||||
The web app currently supports two overlay surfaces:
|
||||
|
||||
- `workshop` overlays (user-owned) — populated by downloading `.vpk` files from the public Steam Web API into `${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creating absolute symlinks under `${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`.
|
||||
- `script` overlays — populated by an arbitrary user-authored bash script that runs inside `bubblewrap` + `systemd-run --scope` as the unprivileged `l4d2-sandbox` UID, with the overlay directory bind-mounted RW at `/overlay`. Resource caps: 1h walltime, 4 GB RAM, 512 tasks, 200% CPU, 20 GB post-build disk cap.
|
||||
- **`workshop` overlays** (user-owned) — populated by downloading
|
||||
`.vpk` files from the public Steam Web API into
|
||||
`${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creating absolute
|
||||
symlinks under
|
||||
`${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`.
|
||||
- **`script` overlays** — populated by an arbitrary user-authored bash
|
||||
script that runs inside `systemd-run` as the unprivileged
|
||||
`l4d2-sandbox` UID, with the overlay directory bind-mounted RW at
|
||||
`/overlay`. Resource caps: 1h walltime, 4 GB RAM, 512 tasks, 200% CPU,
|
||||
20 GB post-build disk cap.
|
||||
|
||||
Both the caches and the overlay directories are owned by the `left4me` runtime user; if the web service ever runs as a different uid, ensure it shares a group with the host process and that both trees are group-readable.
|
||||
Both caches and overlay directories are owned by `left4me`. If the web
|
||||
service ever runs as a different uid, ensure it shares a group with the
|
||||
host process and that both trees are group-readable.
|
||||
|
||||
## Performance Tuning
|
||||
## Performance tuning
|
||||
|
||||
The deployment ships a host-side perf baseline (slices, unit directives, sysctls). See `docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md` for design rationale.
|
||||
The deployment ships a host-side perf baseline (slices, unit directives,
|
||||
sysctls). See
|
||||
`docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md`
|
||||
for design rationale.
|
||||
|
||||
The following knobs are documented escape hatches — they are **not** auto-applied. Apply only if you have measured a need and understand the failure modes.
|
||||
The knobs below are documented escape hatches — **not** auto-applied.
|
||||
Apply only after measuring a need and understanding the failure modes.
|
||||
|
||||
### Network shaping
|
||||
|
||||
The deploy ships three things that affect player-experience network behaviour:
|
||||
Three pieces of the baseline affect player-experience network behaviour:
|
||||
|
||||
1. **Per-flow marking.** `left4me-nft-mark.service` loads a small nftables
|
||||
table (`inet left4me_mark`) that marks every UDP packet from uid `left4me`
|
||||
with DSCP EF and `skb->priority` 6. srcds doesn't set these itself, so
|
||||
without this rule its UDP is indistinguishable from any other flow.
|
||||
1. **Per-flow marking.** ckn-bw's central `bundles/nftables/` consumes
|
||||
left4me's nftables defaults and marks every UDP packet from uid
|
||||
`left4me` with DSCP EF and `skb->priority` 6. srcds doesn't set
|
||||
these itself, so without this rule its UDP is indistinguishable
|
||||
from any other flow.
|
||||
2. **Sysctl baseline.** `99-left4me.conf` sets `udp_rmem_min=16384`,
|
||||
`udp_wmem_min=16384`, `default_qdisc=fq_codel`, and
|
||||
`tcp_congestion_control=bbr`. Reduces head-of-line blocking when bulk
|
||||
TCP egress (backups, package fetches, web responses) coexists with
|
||||
game UDP.
|
||||
3. **CAKE egress shaping.** `left4me-cake.service` runs
|
||||
`tc qdisc replace dev <iface> root cake bandwidth Xmbit internet
|
||||
diffserv4 dual-dsthost` from `/etc/left4me/cake.env`. CAKE only shapes
|
||||
if its declared bandwidth is **below** the real bottleneck, so set
|
||||
`LEFT4ME_UPLINK_MBIT` to ≈95% of measured uplink:
|
||||
|
||||
sudoedit /etc/left4me/cake.env
|
||||
# set LEFT4ME_UPLINK_MBIT=480 (or whatever ~95% of your uplink is)
|
||||
sudo systemctl restart left4me-cake.service
|
||||
|
||||
`LEFT4ME_UPLINK_IFACE` is auto-detected from the IPv4 default route;
|
||||
override only on hosts with multi-homed setups.
|
||||
|
||||
At idle 500 Mbit with no competing egress, CAKE shapes nothing — that's
|
||||
expected, not a bug. The win materialises when bulk traffic on the
|
||||
same uplink would otherwise bufferbloat the link the players share.
|
||||
|
||||
**Production hosts running `systemd-networkd`** should NOT use the
|
||||
`left4me-cake.service` oneshot. Instead, configure the equivalent in the
|
||||
matching `.network` file, which systemd-networkd reapplies across iface
|
||||
lifecycle events:
|
||||
|
||||
# /etc/systemd/network/<your-uplink>.network
|
||||
[CAKE]
|
||||
Bandwidth=480M
|
||||
OverheadKeyword=internet
|
||||
PriorityQueueingPreset=diffserv4
|
||||
EgressHostIsolation=yes
|
||||
|
||||
The nftables marking from (1) is qdisc-installer-agnostic and ships
|
||||
unchanged on production.
|
||||
|
||||
**Disabling network shaping.** To turn the whole feature off on a deployed
|
||||
host:
|
||||
|
||||
sudo systemctl stop left4me-cake.service left4me-nft-mark.service
|
||||
sudo systemctl disable left4me-cake.service left4me-nft-mark.service
|
||||
|
||||
The sysctl baseline (`99-left4me.conf`) and the BBR/fq_codel defaults stay
|
||||
applied; revert those by removing the file and running `sysctl --system`
|
||||
if needed.
|
||||
`tcp_congestion_control=bbr`. Reduces head-of-line blocking when
|
||||
bulk TCP egress coexists with game UDP.
|
||||
3. **CAKE egress shaping.** Configured per-interface via systemd-networkd
|
||||
metadata (`network/<iface>/cake` in ckn-bw's `bundles/network/`),
|
||||
which reapplies the CAKE qdisc across iface lifecycle events. Set
|
||||
the declared bandwidth to ≈95% of measured uplink — CAKE only shapes
|
||||
if its declared bandwidth is *below* the real bottleneck. Idle links
|
||||
with no competing egress see no visible CAKE effect; the win
|
||||
materialises under bulk traffic that would otherwise bufferbloat the
|
||||
link the players share.
|
||||
|
||||
### CPU governor
|
||||
|
||||
The performance governor squeezes a few percent off jitter under bursty load. `schedutil` is acceptable for sustained UDP workloads.
|
||||
The performance governor squeezes a few percent off jitter under bursty
|
||||
load. `schedutil` is acceptable for sustained UDP workloads.
|
||||
|
||||
```sh
|
||||
sudo cpupower frequency-set -g performance
|
||||
```
|
||||
|
||||
Install via `sudo apt install linux-cpupower` if the binary isn't present.
|
||||
|
||||
Persist via your distro's CPU-frequency tooling (e.g. `/etc/default/cpufrequtils`).
|
||||
Install via `sudo apt install linux-cpupower` if the binary isn't
|
||||
present. Persist via your distro's CPU-frequency tooling (e.g.
|
||||
`/etc/default/cpufrequtils`).
|
||||
|
||||
### CPU isolation (cores)
|
||||
|
||||
The deploy script writes four `AllowedCPUs=` drop-ins so that, by default, only `l4d2-game.slice` is allowed to run on cores 1..N-1; `system.slice`, `user.slice`, and `l4d2-build.slice` are pinned to core 0. Game servers thus get the host minus core 0 exclusively, the build sandbox and the web app stay on core 0, and a logged-in admin running CPU-heavy work in their shell can't steal cycles from a live match.
|
||||
The deploy writes four `AllowedCPUs=` drop-ins so that by default only
|
||||
`l4d2-game.slice` is allowed to run on cores `1..N-1`; `system.slice`,
|
||||
`user.slice`, and `l4d2-build.slice` are pinned to core 0. Game servers
|
||||
get the host minus core 0 exclusively; the build sandbox and the web
|
||||
app stay on core 0; a logged-in admin running CPU-heavy work in their
|
||||
shell can't steal cycles from a live match. Single-core hosts skip the
|
||||
cpuset drop-ins entirely; the rest of the perf baseline (cgroup
|
||||
weights, sysctls, OOM scores) still applies.
|
||||
|
||||
Override the split by setting either env var when running the deploy:
|
||||
|
||||
```sh
|
||||
LEFT4ME_SYSTEM_CPUS="0,1" LEFT4ME_GAME_CPUS="2-7" deploy/deploy-test-server.sh deploy-user@host
|
||||
```
|
||||
|
||||
On single-core hosts the deploy skips the cpuset drop-ins entirely and prints a warning to stderr; the rest of the perf baseline (cgroup weights, sysctls, OOM scores) still applies. To force isolation on a single-core host anyway (rarely useful), set either env var explicitly.
|
||||
|
||||
Per-instance `CPUAffinity=` (next subsection) composes on top of this — the per-instance value must be a subset of `l4d2-game.slice`'s `AllowedCPUs=`, which the kernel enforces.
|
||||
Per-instance `CPUAffinity=` (next subsection) composes on top of this —
|
||||
the per-instance value must be a subset of `l4d2-game.slice`'s
|
||||
`AllowedCPUs=`, which the kernel enforces.
|
||||
|
||||
### Per-instance CPU affinity
|
||||
|
||||
`srcds` is single-threaded per instance. On a multi-core host, pinning each instance to its own core can cut jitter under contention. Drop in `/etc/systemd/system/left4me-server@<name>.service.d/affinity.conf`:
|
||||
`srcds` is single-threaded per instance. On a multi-core host, pinning
|
||||
each instance to its own core can cut jitter under contention. Drop in
|
||||
`/etc/systemd/system/left4me-server@<name>.service.d/affinity.conf`:
|
||||
|
||||
```ini
|
||||
[Service]
|
||||
CPUAffinity=2
|
||||
```
|
||||
|
||||
This pins the instance to CPU 2 specifically; per-instance values would typically be 1, 2, 3, ... so each server has its own core.
|
||||
|
||||
A reasonable strategy on an N-core host: leave core 0 for the kernel + IRQs + system services, then pin one instance per remaining core.
|
||||
This pins the instance to CPU 2. A reasonable strategy on an N-core
|
||||
host: leave core 0 for the kernel + IRQs + system services, then pin
|
||||
one instance per remaining core.
|
||||
|
||||
### NIC tuning
|
||||
|
||||
Hardware-specific (install via `sudo apt install ethtool` if not present). On a host with a single primary interface (replace `eth0`):
|
||||
Hardware-specific (install via `sudo apt install ethtool` if not
|
||||
present). On a host with a single primary interface (replace `eth0`):
|
||||
|
||||
```sh
|
||||
sudo ethtool -G eth0 rx 4096 tx 4096
|
||||
sudo ethtool -K eth0 gro on lro off
|
||||
```
|
||||
|
||||
If you run a high instance count, also pin the NIC's interrupts off the cores that game servers occupy (see `/proc/interrupts` and `/proc/irq/<n>/smp_affinity`).
|
||||
If you run a high instance count, also pin the NIC's interrupts off
|
||||
the cores that game servers occupy (see `/proc/interrupts` and
|
||||
`/proc/irq/<n>/smp_affinity`).
|
||||
|
||||
### Real-time scheduling (advanced, opt-in)
|
||||
|
||||
Source-engine servers do not need real-time scheduling, and a misbehaving `srcds` at any RT priority can starve kernel threads — even with the default `kernel.sched_rt_runtime_us=950000` throttling 5% of CPU back. Use only if you have a measured jitter problem that the baseline does not solve.
|
||||
Source-engine servers do not need real-time scheduling, and a
|
||||
misbehaving `srcds` at any RT priority can starve kernel threads — even
|
||||
with the default `kernel.sched_rt_runtime_us=950000` throttling 5% of
|
||||
CPU back. Use only if you have a measured jitter problem that the
|
||||
baseline does not solve.
|
||||
|
||||
`/etc/systemd/system/left4me-server@.service.d/realtime.conf`:
|
||||
|
||||
|
|
@ -252,13 +253,16 @@ LimitRTPRIO=10
|
|||
AmbientCapabilities=CAP_SYS_NICE
|
||||
```
|
||||
|
||||
The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the service runs as `User=left4me` with `NoNewPrivileges=true`; without it some kernels/systemd combinations refuse to apply the RT policy.
|
||||
The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the
|
||||
service runs as `User=left4me` with `NoNewPrivileges=true`; without it
|
||||
some kernels/systemd combinations refuse to apply the RT policy.
|
||||
|
||||
### Additional opt-in network knobs
|
||||
|
||||
- **Ingress shaping via IFB.** Egress CAKE alone does not protect srcds
|
||||
receive against ingress saturation (large workshop downloads, package
|
||||
fetches arriving at line rate). One-liner:
|
||||
receive against ingress saturation (large workshop downloads,
|
||||
package fetches arriving at line rate). Worth flipping only when
|
||||
measurement shows ingress hurting receive.
|
||||
|
||||
sudo modprobe ifb && sudo ip link set ifb0 up
|
||||
sudo tc qdisc add dev <uplink> handle ffff: ingress
|
||||
|
|
@ -267,13 +271,11 @@ The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the service runs a
|
|||
sudo tc qdisc add dev ifb0 root cake bandwidth Xmbit ingress \
|
||||
diffserv4 dual-srchost
|
||||
|
||||
Worth flipping only when measurement shows ingress hurting receive.
|
||||
|
||||
- **`net.core.busy_poll = 50` / `net.core.busy_read = 50`.** Reduces UDP
|
||||
receive median latency by polling for incoming packets briefly at
|
||||
syscall boundaries. Cost: measurable CPU per syscall under load. Worth
|
||||
flipping if a host is dedicated to game serving and CPU headroom is
|
||||
plentiful.
|
||||
- **`net.core.busy_poll = 50` / `net.core.busy_read = 50`.** Reduces
|
||||
UDP receive median latency by polling for incoming packets briefly
|
||||
at syscall boundaries. Cost: measurable CPU per syscall under load.
|
||||
Worth flipping if a host is dedicated to game serving and CPU
|
||||
headroom is plentiful.
|
||||
|
||||
- **`ethtool -K <iface> gro off`.** Some Source-engine ops disable
|
||||
generic receive offload to avoid receive-side coalescing latency.
|
||||
|
|
@ -281,7 +283,8 @@ The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the service runs a
|
|||
|
||||
### Applying changes to running servers
|
||||
|
||||
Unit-file changes do not apply to already-running services. After any change:
|
||||
Unit-file changes do not apply to already-running services. After any
|
||||
change:
|
||||
|
||||
```sh
|
||||
sudo systemctl daemon-reload
|
||||
|
|
|
|||
|
|
@ -1,354 +0,0 @@
|
|||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
usage() {
|
||||
printf 'Usage: %s <ssh-user@host>\n' "$0" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
if [ "$#" -ne 1 ]; then
|
||||
usage
|
||||
fi
|
||||
|
||||
target=$1
|
||||
script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
repo_root=$(CDPATH= cd -- "$script_dir/.." && pwd)
|
||||
tmp_dir=$(mktemp -d)
|
||||
archive="$tmp_dir/left4me.tar.gz"
|
||||
|
||||
cleanup() {
|
||||
rm -rf "$tmp_dir"
|
||||
}
|
||||
trap cleanup EXIT INT HUP TERM
|
||||
|
||||
COPYFILE_DISABLE=1 tar -czf "$archive" \
|
||||
--exclude .git \
|
||||
--exclude .claude \
|
||||
--exclude .venv \
|
||||
--exclude __pycache__ \
|
||||
--exclude .pytest_cache \
|
||||
--exclude '*.egg-info' \
|
||||
--exclude 'l4d2web.db*' \
|
||||
--exclude '._*' \
|
||||
-C "$repo_root" .
|
||||
|
||||
remote_tmp=$(ssh "$target" 'mktemp -d')
|
||||
scp "$archive" "$target:$remote_tmp/left4me.tar.gz"
|
||||
|
||||
admin_username_file=
|
||||
admin_password_file=
|
||||
if [ "${LEFT4ME_ADMIN_USERNAME+x}" = x ] && [ "${LEFT4ME_ADMIN_PASSWORD+x}" = x ]; then
|
||||
admin_username_file="$tmp_dir/admin_username"
|
||||
admin_password_file="$tmp_dir/admin_password"
|
||||
umask 077
|
||||
printf '%s' "$LEFT4ME_ADMIN_USERNAME" > "$admin_username_file"
|
||||
printf '%s' "$LEFT4ME_ADMIN_PASSWORD" > "$admin_password_file"
|
||||
scp "$admin_username_file" "$target:$remote_tmp/admin_username"
|
||||
scp "$admin_password_file" "$target:$remote_tmp/admin_password"
|
||||
fi
|
||||
|
||||
ssh "$target" sh -s -- "$remote_tmp" <<'REMOTE'
|
||||
set -eu
|
||||
|
||||
remote_tmp=$1
|
||||
archive="$remote_tmp/left4me.tar.gz"
|
||||
repo_tmp="$remote_tmp/repo"
|
||||
|
||||
if [ "$(id -u)" -eq 0 ]; then
|
||||
sudo_cmd=
|
||||
else
|
||||
sudo_cmd=sudo
|
||||
fi
|
||||
|
||||
run_as_left4me() {
|
||||
sudo -u left4me "$@"
|
||||
}
|
||||
|
||||
run_left4me_with_env() {
|
||||
run_as_left4me sh -c 'set -a; . /etc/left4me/host.env; . /etc/left4me/web.env; set +a; exec "$@"' sh "$@"
|
||||
}
|
||||
|
||||
cleanup_remote() {
|
||||
rm -rf "$remote_tmp"
|
||||
}
|
||||
trap cleanup_remote EXIT INT HUP TERM
|
||||
|
||||
if ! id left4me >/dev/null 2>&1; then
|
||||
$sudo_cmd useradd --system --home-dir /var/lib/left4me --create-home --shell /usr/sbin/nologin left4me
|
||||
fi
|
||||
|
||||
# Sandbox uid for script-overlay builds. No home, no login shell — the bwrap
|
||||
# invocation uses --uid/--gid to drop to it.
|
||||
if ! id l4d2-sandbox >/dev/null 2>&1; then
|
||||
$sudo_cmd useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox
|
||||
fi
|
||||
|
||||
if command -v apt-get >/dev/null 2>&1; then
|
||||
$sudo_cmd apt-get update
|
||||
$sudo_cmd apt-get install -y python3 python3-venv python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip-full nftables iproute2
|
||||
elif command -v dnf >/dev/null 2>&1; then
|
||||
$sudo_cmd dnf install -y python3 python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip p7zip-plugins nftables iproute
|
||||
else
|
||||
printf 'Unsupported package manager: expected apt-get or dnf\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
$sudo_cmd mkdir -p \
|
||||
/etc/left4me \
|
||||
/opt/left4me \
|
||||
/usr/local/lib/systemd/system \
|
||||
/usr/local/libexec/left4me \
|
||||
/usr/local/lib/left4me/nft \
|
||||
/var/lib/left4me/installation \
|
||||
/var/lib/left4me/overlays \
|
||||
/var/lib/left4me/instances \
|
||||
/var/lib/left4me/runtime \
|
||||
/var/lib/left4me/workshop_cache \
|
||||
/var/lib/left4me/tmp
|
||||
|
||||
$sudo_cmd chown left4me:left4me \
|
||||
/var/lib/left4me \
|
||||
/var/lib/left4me/installation \
|
||||
/var/lib/left4me/overlays \
|
||||
/var/lib/left4me/instances \
|
||||
/var/lib/left4me/runtime \
|
||||
/var/lib/left4me/workshop_cache \
|
||||
/var/lib/left4me/tmp
|
||||
|
||||
# /var/lib/left4me is left4me's home dir (mode 0700 from useradd --create-home).
|
||||
# Allow other uids (notably l4d2-sandbox, used by script overlay builds) to
|
||||
# traverse — but not list — so the bwrap bind-mount can resolve the overlay
|
||||
# path under the dropped privilege.
|
||||
$sudo_cmd chmod 0711 /var/lib/left4me
|
||||
$sudo_cmd chown -R left4me:left4me /opt/left4me
|
||||
|
||||
mkdir -p "$repo_tmp"
|
||||
tar -xzf "$archive" -C "$repo_tmp"
|
||||
|
||||
if [ -d /opt/left4me/.venv ]; then
|
||||
$sudo_cmd mv /opt/left4me/.venv "$remote_tmp/venv"
|
||||
fi
|
||||
$sudo_cmd find /opt/left4me -mindepth 1 -maxdepth 1 -exec rm -rf {} +
|
||||
$sudo_cmd cp -R "$repo_tmp"/. /opt/left4me/
|
||||
if [ -d "$remote_tmp/venv" ]; then
|
||||
$sudo_cmd mv "$remote_tmp/venv" /opt/left4me/.venv
|
||||
fi
|
||||
$sudo_cmd chown -R left4me:left4me /opt/left4me
|
||||
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-web.service /usr/local/lib/systemd/system/left4me-web.service
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-server@.service /usr/local/lib/systemd/system/left4me-server@.service
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/l4d2-game.slice /usr/local/lib/systemd/system/l4d2-game.slice
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/l4d2-build.slice /usr/local/lib/systemd/system/l4d2-build.slice
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service /usr/local/lib/systemd/system/left4me-nft-mark.service
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-cake.service /usr/local/lib/systemd/system/left4me-cake.service
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-workshop-refresh.service /usr/local/lib/systemd/system/left4me-workshop-refresh.service
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-workshop-refresh.timer /usr/local/lib/systemd/system/left4me-workshop-refresh.timer
|
||||
|
||||
# CPU isolation via cgroup-v2 AllowedCPUs= drop-ins. Pin everything that
|
||||
# isn't a live game server to core 0; give game servers cores 1..N-1.
|
||||
# See docs/superpowers/specs/2026-05-09-l4d2-cpu-isolation-design.md.
|
||||
# `nproc --all` reports installed processors regardless of the calling
|
||||
# shell's CPU affinity. Plain `nproc` honors Cpus_allowed of the calling
|
||||
# process, so on a host that already has the cpuset drop-ins applied
|
||||
# (system.slice → AllowedCPUs=0), the SSH login lands in user.slice with
|
||||
# AllowedCPUs=0 and `nproc` would return 1 — making subsequent deploys
|
||||
# wrongly think they're on a single-core box and skip CPU isolation.
|
||||
NPROC=$(nproc --all)
|
||||
SYSTEM_CPUS=${LEFT4ME_SYSTEM_CPUS:-0}
|
||||
if [ "${LEFT4ME_GAME_CPUS+x}" = x ]; then
|
||||
GAME_CPUS=$LEFT4ME_GAME_CPUS
|
||||
else
|
||||
GAME_CPUS="1-$((NPROC - 1))"
|
||||
fi
|
||||
if [ "$NPROC" -lt 2 ] && [ "${LEFT4ME_SYSTEM_CPUS+x}${LEFT4ME_GAME_CPUS+x}" = "" ]; then
|
||||
printf 'left4me deploy: skipping CPU isolation (nproc=%s); cpuset drop-ins not written.\n' "$NPROC" >&2
|
||||
else
|
||||
for slice_drop_in in \
|
||||
/etc/systemd/system/system.slice.d/99-left4me-cpuset.conf \
|
||||
/etc/systemd/system/user.slice.d/99-left4me-cpuset.conf \
|
||||
/etc/systemd/system/l4d2-build.slice.d/99-left4me-cpuset.conf; do
|
||||
$sudo_cmd mkdir -p "$(dirname "$slice_drop_in")"
|
||||
printf '[Slice]\nAllowedCPUs=%s\n' "$SYSTEM_CPUS" \
|
||||
| $sudo_cmd install -m 0644 -o root -g root /dev/stdin "$slice_drop_in"
|
||||
done
|
||||
$sudo_cmd mkdir -p /etc/systemd/system/l4d2-game.slice.d
|
||||
printf '[Slice]\nAllowedCPUs=%s\n' "$GAME_CPUS" \
|
||||
| $sudo_cmd install -m 0644 -o root -g root /dev/stdin \
|
||||
/etc/systemd/system/l4d2-game.slice.d/99-left4me-cpuset.conf
|
||||
fi
|
||||
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-systemctl
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-journalctl
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-overlay
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-script-sandbox /usr/local/libexec/left4me/left4me-script-sandbox
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-apply-cake /usr/local/libexec/left4me/left4me-apply-cake
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/sbin/left4me /usr/local/sbin/left4me
|
||||
$sudo_cmd chmod 0755 /usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-script-sandbox /usr/local/libexec/left4me/left4me-apply-cake /usr/local/sbin/left4me
|
||||
$sudo_cmd cp /opt/left4me/deploy/files/etc/sudoers.d/left4me /etc/sudoers.d/left4me
|
||||
$sudo_cmd chmod 0440 /etc/sudoers.d/left4me
|
||||
$sudo_cmd visudo -cf /etc/sudoers.d/left4me
|
||||
|
||||
$sudo_cmd cp /opt/left4me/deploy/templates/etc/left4me/host.env /etc/left4me/host.env
|
||||
$sudo_cmd chmod 0644 /etc/left4me/host.env
|
||||
|
||||
# Sandbox-only resolver config; bind-mounted into the script sandbox's /etc/resolv.conf
|
||||
# so DNS still works when IPAddressDeny= blocks the host's (typically private-IP) resolver.
|
||||
$sudo_cmd install -m 0644 -o root -g root \
|
||||
/opt/left4me/deploy/files/etc/left4me/sandbox-resolv.conf \
|
||||
/etc/left4me/sandbox-resolv.conf
|
||||
|
||||
# Network packet marking + shaping. See spec
|
||||
# docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md.
|
||||
$sudo_cmd install -m 0644 -o root -g root \
|
||||
/opt/left4me/deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft \
|
||||
/usr/local/lib/left4me/nft/left4me-mark.nft
|
||||
|
||||
# Host perf-baseline sysctls. Apply with `sysctl --system` so values
|
||||
# take effect this deploy, not on next reboot.
|
||||
$sudo_cmd install -m 0644 -o root -g root \
|
||||
/opt/left4me/deploy/files/etc/sysctl.d/99-left4me.conf \
|
||||
/etc/sysctl.d/99-left4me.conf
|
||||
$sudo_cmd sysctl --system >/dev/null
|
||||
|
||||
# CAKE config: ship the template only if the operator hasn't created one
|
||||
# (their LEFT4ME_UPLINK_MBIT value must survive re-deploys).
|
||||
if [ -e /etc/left4me/cake.env ]; then
|
||||
: # operator file present; leave it intact
|
||||
else
|
||||
$sudo_cmd install -m 0644 -o root -g root \
|
||||
/opt/left4me/deploy/files/etc/left4me/cake.env \
|
||||
/etc/left4me/cake.env
|
||||
fi
|
||||
|
||||
# Stomp the file every deploy so newly added vars reach existing boxes.
|
||||
# SECRET_KEY is derived from /etc/machine-id so it stays stable across
|
||||
# redeploys (no session invalidation) without persisting state in /etc.
|
||||
secret_key=$(sha256sum < /etc/machine-id | awk '{print $1}')
|
||||
tmp_web_env="$remote_tmp/web.env"
|
||||
{
|
||||
printf 'DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n'
|
||||
printf 'SECRET_KEY=%s\n' "$secret_key"
|
||||
printf 'JOB_WORKER_THREADS=4\n'
|
||||
printf 'SESSION_COOKIE_SECURE=false\n'
|
||||
} > "$tmp_web_env"
|
||||
$sudo_cmd install -m 0640 -o root -g left4me "$tmp_web_env" /etc/left4me/web.env
|
||||
|
||||
if [ ! -x /opt/left4me/.venv/bin/python ]; then
|
||||
run_as_left4me python3 -m venv /opt/left4me/.venv
|
||||
fi
|
||||
run_as_left4me /opt/left4me/.venv/bin/python -m pip install --upgrade pip
|
||||
run_as_left4me /opt/left4me/.venv/bin/pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web
|
||||
|
||||
run_as_left4me sh -c "cd /opt/left4me/l4d2web && set -a; . /etc/left4me/host.env; . /etc/left4me/web.env; set +a; env \
|
||||
JOB_WORKER_ENABLED=false \
|
||||
PYTHONPATH=/opt/left4me \
|
||||
/opt/left4me/.venv/bin/alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head"
|
||||
|
||||
# Tighten the application database to left4me:left4me 0640. The DB is
|
||||
# created by the web app at first start with the default 0644 umask, which
|
||||
# makes it world-readable on the host. The script-overlay sandbox runs as a
|
||||
# separate system uid (l4d2-sandbox) which is NOT in the left4me group —
|
||||
# 0640 blocks it via "other". The owner (left4me) keeps read+write so the
|
||||
# web service can update the DB.
|
||||
#
|
||||
# SQLite in WAL mode (the default in this app) maintains -wal and -shm
|
||||
# sidecar files; both must also be writable by the web service. If a previous
|
||||
# operator opened the DB as root (e.g. for ad-hoc inspection), the sidecars
|
||||
# may have ended up root-owned, which makes SQLite report "readonly database"
|
||||
# on the next write. Re-chown them defensively. Idempotent on rerun.
|
||||
for db_file in /var/lib/left4me/left4me.db /var/lib/left4me/left4me.db-wal /var/lib/left4me/left4me.db-shm; do
|
||||
if [ -f "$db_file" ]; then
|
||||
$sudo_cmd chown left4me:left4me "$db_file"
|
||||
$sudo_cmd chmod 0640 "$db_file"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -f "$remote_tmp/admin_username" ] && [ -f "$remote_tmp/admin_password" ]; then
|
||||
LEFT4ME_ADMIN_USERNAME=$(cat "$remote_tmp/admin_username")
|
||||
LEFT4ME_ADMIN_PASSWORD=$(cat "$remote_tmp/admin_password")
|
||||
if ! create_user_output=$(run_left4me_with_env env \
|
||||
JOB_WORKER_ENABLED=false \
|
||||
LEFT4ME_ADMIN_PASSWORD="$LEFT4ME_ADMIN_PASSWORD" \
|
||||
/opt/left4me/.venv/bin/flask --app l4d2web.app:create_app create-user "$LEFT4ME_ADMIN_USERNAME" --admin 2>&1); then
|
||||
case "$create_user_output" in
|
||||
*'user already exists'*) printf '%s\n' "$create_user_output" ;;
|
||||
*) printf '%s\n' "$create_user_output" >&2; exit 1 ;;
|
||||
esac
|
||||
else
|
||||
printf '%s\n' "$create_user_output"
|
||||
fi
|
||||
fi
|
||||
|
||||
# One-shot migration: fuse-overlayfs running as the left4me user used
|
||||
# user.fuseoverlayfs.* xattrs for whiteouts and opaque-dir markers; kernel
|
||||
# overlayfs ignores those entirely, so a pre-existing upper/ from the fuse
|
||||
# era would resurrect "deleted" files. Wipe upper/ and work/ for every
|
||||
# instance once, gated by a sentinel file so reruns are no-ops.
|
||||
overlay_sentinel=/var/lib/left4me/.kernel-overlay-migrated
|
||||
if [ ! -e "$overlay_sentinel" ]; then
|
||||
$sudo_cmd sh -c "systemctl stop 'left4me-server@*.service' 2>/dev/null || true"
|
||||
$sudo_cmd systemctl stop left4me-web.service 2>/dev/null || true
|
||||
$sudo_cmd sh -c "findmnt -t fuse.fuse-overlayfs -o TARGET --noheadings 2>/dev/null | xargs -r -n1 umount -l 2>/dev/null || true"
|
||||
$sudo_cmd sh -c "findmnt -t overlay -o TARGET --noheadings 2>/dev/null | grep '/var/lib/left4me/runtime/' | xargs -r -n1 umount -l 2>/dev/null || true"
|
||||
$sudo_cmd sh -c 'for d in /var/lib/left4me/runtime/*/; do [ -d "$d" ] || continue; rm -rf "$d/upper" "$d/work"; mkdir -p "$d/upper" "$d/work"; chown left4me:left4me "$d/upper" "$d/work"; done'
|
||||
$sudo_cmd touch "$overlay_sentinel"
|
||||
$sudo_cmd chown left4me:left4me "$overlay_sentinel"
|
||||
fi
|
||||
|
||||
# One-shot migration: 0005_script_overlays drops the legacy
|
||||
# l4d2center_maps / cedapug_maps overlay rows but doesn't touch their
|
||||
# directories under /var/lib/left4me/overlays/{id}. Without cleanup, when
|
||||
# AUTOINCREMENT (or its absence after the 0002 batch_alter_table recreate)
|
||||
# re-issues an id matching one of those orphan dirs, the web app's
|
||||
# create_overlay_directory(exist_ok=False) fails with FileExistsError.
|
||||
# Sweep any overlay dir whose id has no matching DB row, plus the
|
||||
# now-unused global_overlay_cache.
|
||||
overlay_orphan_sentinel=/var/lib/left4me/.script-overlays-orphans-cleaned
|
||||
if [ ! -e "$overlay_orphan_sentinel" ]; then
|
||||
$sudo_cmd rm -rf /var/lib/left4me/global_overlay_cache
|
||||
$sudo_cmd sh -c '
|
||||
cd /var/lib/left4me/overlays || exit 0
|
||||
ids_in_db=$(/opt/left4me/.venv/bin/python -c "
|
||||
import sqlite3
|
||||
c = sqlite3.connect(\"/var/lib/left4me/left4me.db\")
|
||||
print(\" \".join(str(r[0]) for r in c.execute(\"SELECT id FROM overlays\")))
|
||||
")
|
||||
for d in */; do
|
||||
id=${d%/}
|
||||
case " $ids_in_db " in
|
||||
*" $id "*) ;;
|
||||
*) echo "removing orphan overlay dir: $id"; rm -rf "$id" ;;
|
||||
esac
|
||||
done
|
||||
'
|
||||
$sudo_cmd touch "$overlay_orphan_sentinel"
|
||||
$sudo_cmd chown left4me:left4me "$overlay_orphan_sentinel"
|
||||
fi
|
||||
|
||||
# Seed example script overlays (cedapug, l4d2center, competitive_rework, ...)
|
||||
# as system-wide rows from /opt/left4me/examples/script-overlays/. Idempotent:
|
||||
# subsequent deploys refresh the script body in place, leaving the row id and
|
||||
# overlay directory intact.
|
||||
run_left4me_with_env env \
|
||||
JOB_WORKER_ENABLED=false \
|
||||
PYTHONPATH=/opt/left4me \
|
||||
/opt/left4me/.venv/bin/flask --app l4d2web.app:create_app \
|
||||
seed-script-overlays /opt/left4me/examples/script-overlays
|
||||
|
||||
$sudo_cmd systemctl daemon-reload
|
||||
$sudo_cmd systemctl enable --now left4me-nft-mark.service
|
||||
$sudo_cmd systemctl enable --now left4me-cake.service
|
||||
$sudo_cmd systemctl enable --now left4me-web.service
|
||||
$sudo_cmd systemctl restart left4me-web.service
|
||||
$sudo_cmd systemctl enable --now left4me-workshop-refresh.timer
|
||||
for attempt in 1 2 3 4 5 6 7 8 9 10; do
|
||||
if curl -fsS http://127.0.0.1:8000/health; then
|
||||
exit 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
$sudo_cmd systemctl status left4me-web.service --no-pager >&2 || true
|
||||
$sudo_cmd journalctl -u left4me-web.service -n 80 --no-pager >&2 || true
|
||||
exit 1
|
||||
REMOTE
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
# left4me — CAKE egress shaper config. Consumed by left4me-cake.service via
|
||||
# its EnvironmentFile=. Edit then `systemctl restart left4me-cake.service`.
|
||||
# See docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md.
|
||||
|
||||
# Uplink bandwidth in Mbit/s. Set to ~95% of the smaller of measured upload
|
||||
# and measured download. CAKE only shapes correctly when its declared
|
||||
# bandwidth sits below the real bottleneck. If unset, the shaper unit logs
|
||||
# a warning and exits 0 (no shaping).
|
||||
LEFT4ME_UPLINK_MBIT=
|
||||
|
||||
# Egress interface. If unset, auto-detected from the IPv4 default route.
|
||||
LEFT4ME_UPLINK_IFACE=
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
# left4me — uid-based DSCP/priority marking for srcds UDP egress.
|
||||
# Loaded by left4me-nft-mark.service into its own `inet` table so it cannot
|
||||
# conflict with whatever the operator already runs in /etc/nftables.conf.
|
||||
# See docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md.
|
||||
|
||||
table inet left4me_mark {
|
||||
chain mangle_output {
|
||||
type filter hook output priority mangle; policy accept;
|
||||
meta skuid "left4me" meta l4proto udp ip dscp set ef meta priority set 0006:0000
|
||||
meta skuid "left4me" meta l4proto udp ip6 dscp set ef meta priority set 0006:0000
|
||||
}
|
||||
}
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
[Unit]
|
||||
Description=left4me CAKE egress shaper
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
EnvironmentFile=-/etc/left4me/cake.env
|
||||
ExecStart=/usr/local/libexec/left4me/left4me-apply-cake apply
|
||||
ExecStop=/usr/local/libexec/left4me/left4me-apply-cake clear
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
[Unit]
|
||||
Description=left4me nftables packet marking (DSCP EF + priority for srcds)
|
||||
After=network-pre.target
|
||||
Before=network.target
|
||||
Wants=network-pre.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
ExecStart=/usr/sbin/nft -f /usr/local/lib/left4me/nft/left4me-mark.nft
|
||||
ExecStop=/usr/sbin/nft delete table inet left4me_mark
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
#!/bin/sh
|
||||
# left4me — apply or clear CAKE egress shaper on the configured uplink.
|
||||
# Driven by left4me-cake.service. See spec
|
||||
# docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md.
|
||||
set -eu
|
||||
|
||||
mode=${1:-apply}
|
||||
|
||||
if [ -r /etc/left4me/cake.env ]; then
|
||||
. /etc/left4me/cake.env
|
||||
fi
|
||||
|
||||
resolve_iface() {
|
||||
if [ -n "${LEFT4ME_UPLINK_IFACE:-}" ]; then
|
||||
printf '%s' "$LEFT4ME_UPLINK_IFACE"
|
||||
return
|
||||
fi
|
||||
ip -4 route show default | awk '/default/ {print $5; exit}'
|
||||
}
|
||||
|
||||
case "$mode" in
|
||||
apply)
|
||||
if [ -z "${LEFT4ME_UPLINK_MBIT:-}" ]; then
|
||||
echo "left4me-cake: LEFT4ME_UPLINK_MBIT unset; skipping shaper" >&2
|
||||
exit 0
|
||||
fi
|
||||
iface=$(resolve_iface)
|
||||
if [ -z "$iface" ]; then
|
||||
echo "left4me-cake: cannot determine egress iface; skipping" >&2
|
||||
exit 0
|
||||
fi
|
||||
exec tc qdisc replace dev "$iface" root cake \
|
||||
bandwidth "${LEFT4ME_UPLINK_MBIT}mbit" \
|
||||
internet diffserv4 dual-dsthost
|
||||
;;
|
||||
clear)
|
||||
iface=$(resolve_iface)
|
||||
if [ -z "$iface" ]; then
|
||||
exit 0
|
||||
fi
|
||||
tc qdisc del dev "$iface" root 2>/dev/null || true
|
||||
;;
|
||||
*)
|
||||
echo "usage: $0 [apply|clear]" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
|
@ -1,882 +0,0 @@
|
|||
import os
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
DEPLOY = ROOT / "deploy"
|
||||
|
||||
|
||||
WEB_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-web.service"
|
||||
SERVER_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-server@.service"
|
||||
GAME_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-game.slice"
|
||||
BUILD_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-build.slice"
|
||||
SYSCTL_CONF = DEPLOY / "files/etc/sysctl.d/99-left4me.conf"
|
||||
GLOBAL_REFRESH_SERVICE = DEPLOY / "files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.service"
|
||||
GLOBAL_REFRESH_TIMER = DEPLOY / "files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer"
|
||||
NFT_MARK_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-nft-mark.service"
|
||||
CAKE_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-cake.service"
|
||||
SANDBOX_UNIT_DIR = DEPLOY / "files/usr/local/lib/systemd/system"
|
||||
SYSTEMCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-systemctl"
|
||||
JOURNALCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-journalctl"
|
||||
OVERLAY_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-overlay"
|
||||
SCRIPT_SANDBOX_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-script-sandbox"
|
||||
APPLY_CAKE_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-apply-cake"
|
||||
SANDBOX_RESOLV_CONF = DEPLOY / "files/etc/left4me/sandbox-resolv.conf"
|
||||
CAKE_ENV = DEPLOY / "files/etc/left4me/cake.env"
|
||||
SUDOERS = DEPLOY / "files/etc/sudoers.d/left4me"
|
||||
HOST_ENV = DEPLOY / "templates/etc/left4me/host.env"
|
||||
WEB_ENV_TEMPLATE = DEPLOY / "templates/etc/left4me/web.env.template"
|
||||
DEPLOY_SCRIPT = DEPLOY / "deploy-test-server.sh"
|
||||
NFT_MARK_FILE = DEPLOY / "files/usr/local/lib/left4me/nft/left4me-mark.nft"
|
||||
|
||||
|
||||
def test_global_unit_files_exist_at_product_level_paths():
|
||||
assert WEB_UNIT.is_file()
|
||||
assert SERVER_UNIT.is_file()
|
||||
|
||||
|
||||
def test_web_unit_contains_required_runtime_contract():
|
||||
unit = WEB_UNIT.read_text()
|
||||
|
||||
assert "User=left4me" in unit
|
||||
assert "Group=left4me" in unit
|
||||
assert "WorkingDirectory=/opt/left4me" in unit
|
||||
assert "Environment=PATH=/opt/left4me/.venv/bin:" in unit
|
||||
assert "EnvironmentFile=/etc/left4me/host.env" in unit
|
||||
assert "EnvironmentFile=/etc/left4me/web.env" in unit
|
||||
assert "ExecStart=/opt/left4me/.venv/bin/gunicorn" in unit
|
||||
assert "--workers 1" in unit
|
||||
assert "--threads 32" in unit
|
||||
# NoNewPrivileges must remain unset because sudo (used by the overlay,
|
||||
# systemctl and journalctl helpers) is setuid.
|
||||
assert "NoNewPrivileges=true" not in unit
|
||||
# Restored now that fuse-overlayfs propagation is no longer the mechanism.
|
||||
assert "PrivateTmp=true" in unit
|
||||
assert "ProtectSystem=full" in unit
|
||||
assert "ReadWritePaths=/var/lib/left4me" in unit
|
||||
# Mounts now happen in PID 1's namespace via the left4me-overlay helper,
|
||||
# so MountFlags propagation is irrelevant — and the previous assumption
|
||||
# that MountFlags=shared made it work was incorrect.
|
||||
assert "MountFlags=" not in unit
|
||||
|
||||
|
||||
def test_server_unit_contains_required_runtime_contract():
|
||||
unit = SERVER_UNIT.read_text()
|
||||
|
||||
assert "User=left4me" in unit
|
||||
assert "Group=left4me" in unit
|
||||
assert "EnvironmentFile=/etc/left4me/host.env" in unit
|
||||
assert "EnvironmentFile=/var/lib/left4me/instances/%i/instance.env" in unit
|
||||
# `-` prefix: chdir failure is non-fatal so ExecStartPre can run the
|
||||
# mount helper before the merged dir exists. ExecStart re-applies and
|
||||
# finds the dir once the mount has landed.
|
||||
assert "WorkingDirectory=-/var/lib/left4me/runtime/%i/merged/left4dead2" in unit
|
||||
# ExecStart must invoke srcds_run from the *merged* overlay tree, not
|
||||
# from installation/. srcds_run cds to its own dirname; if we point at
|
||||
# installation/, the engine reads gameinfo.txt and addons from the lower
|
||||
# layer and never sees overlay plugins (Metamod/SourceMod) or cfgs.
|
||||
assert "ExecStart=/var/lib/left4me/runtime/%i/merged/srcds_run" in unit
|
||||
assert "$L4D2_ARGS" in unit
|
||||
assert "${L4D2_ARGS}" not in unit
|
||||
assert "NoNewPrivileges=true" in unit
|
||||
assert "PrivateTmp=true" in unit
|
||||
assert "PrivateDevices=true" in unit
|
||||
assert "ProtectHome=true" in unit
|
||||
assert "ProtectSystem=strict" in unit
|
||||
assert "ReadOnlyPaths=/var/lib/left4me/installation /var/lib/left4me/overlays" in unit
|
||||
assert "ReadWritePaths=/var/lib/left4me/runtime/%i" in unit
|
||||
assert "RestrictSUIDSGID=true" in unit
|
||||
assert "LockPersonality=true" in unit
|
||||
|
||||
|
||||
def test_server_unit_mounts_overlay_via_exec_start_pre():
|
||||
"""At boot, systemd auto-starts enabled units before the web app gets a
|
||||
chance to run start_instance's pre-start mount. The unit itself must
|
||||
re-mount the overlay so reboots are transparent. Pairs with the helper's
|
||||
idempotency check (test_overlay_helper_mount_is_idempotent_when_mounted).
|
||||
|
||||
The unit-level `nsenter --mount=/proc/1/ns/mnt --` is what makes
|
||||
umount fast: without it, the helper Python process would inherit
|
||||
the unit's per-service mount namespace and pin it alive, blocking
|
||||
PID 1's umount until the helper exited. Wrapping with nsenter at
|
||||
the Exec line puts the helper itself in PID 1's namespace.
|
||||
"""
|
||||
unit = SERVER_UNIT.read_text()
|
||||
# `+` prefix: runs as PID 1 (root, no sandbox). Required because
|
||||
# the unit has NoNewPrivileges=true, which blocks sudo's setuid
|
||||
# escalation — and the helper needs root for the mount syscall.
|
||||
assert (
|
||||
"ExecStartPre=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- "
|
||||
"/usr/local/libexec/left4me/left4me-overlay mount %i"
|
||||
in unit
|
||||
)
|
||||
# Bound the restart loop; without these, a CHDIR-failure (or any other
|
||||
# pre-start error) spins indefinitely.
|
||||
assert "StartLimitBurst=5" in unit
|
||||
assert "StartLimitIntervalSec=60s" in unit
|
||||
|
||||
|
||||
def test_server_unit_unmounts_overlay_via_exec_stop_post():
|
||||
"""Single source of truth for unmount, mirroring the mount path.
|
||||
ExecStopPost (not ExecStop) so it runs after srcds has fully exited
|
||||
and the cgroup is cleared.
|
||||
|
||||
Same nsenter-at-Exec-line wrapping as ExecStartPre — without it,
|
||||
the helper process would itself hold a reference to the unit's
|
||||
per-service mount namespace, and umount in PID 1 would loop on
|
||||
EBUSY until the helper gave up. With it, umount succeeds first try.
|
||||
"""
|
||||
unit = SERVER_UNIT.read_text()
|
||||
assert (
|
||||
"ExecStopPost=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- "
|
||||
"/usr/local/libexec/left4me/left4me-overlay umount %i"
|
||||
in unit
|
||||
)
|
||||
|
||||
|
||||
def test_overlay_helper_mount_is_idempotent_when_already_mounted():
|
||||
"""ExecStartPre runs on every Restart=on-failure cycle. If a previous
|
||||
start mounted successfully but ExecStart failed afterwards, the next
|
||||
ExecStartPre would re-mount on top -- which fails. The helper must
|
||||
short-circuit when merged is already a mount point.
|
||||
"""
|
||||
text = OVERLAY_HELPER.read_text()
|
||||
# Two ismount checks now: one in cmd_mount (skip if mounted),
|
||||
# one in cmd_umount (skip if not mounted).
|
||||
assert text.count("os.path.ismount") >= 2
|
||||
|
||||
|
||||
def test_server_unit_contains_perf_baseline_directives():
|
||||
unit = SERVER_UNIT.read_text()
|
||||
|
||||
# Slice membership.
|
||||
assert "Slice=l4d2-game.slice" in unit
|
||||
|
||||
# CFS priority bump (no SCHED_FIFO).
|
||||
assert "Nice=-5" in unit
|
||||
assert "CPUSchedulingPolicy=" not in unit
|
||||
|
||||
# I/O priority.
|
||||
assert "IOSchedulingClass=best-effort" in unit
|
||||
assert "IOSchedulingPriority=4" in unit
|
||||
|
||||
# OOM ordering: game servers survive, sandbox dies first.
|
||||
assert "OOMScoreAdjust=-200" in unit
|
||||
|
||||
# Memory caps with headroom for map-load spikes.
|
||||
assert "MemoryHigh=1.5G" in unit
|
||||
assert "MemoryMax=2G" in unit
|
||||
|
||||
# Bounded fork surface.
|
||||
assert "TasksMax=256" in unit
|
||||
|
||||
# Plenty of fds for plugin-heavy setups.
|
||||
assert "LimitNOFILE=65536" in unit
|
||||
|
||||
# srcds clean shutdown via SIGINT, with time to flush. With the
|
||||
# helper running in PID 1's mount namespace (via the unit-level
|
||||
# nsenter on ExecStopPost), umount has no race window and the
|
||||
# default 15 s is plenty for the whole stop transition.
|
||||
assert "KillSignal=SIGINT" in unit
|
||||
assert "TimeoutStopSec=15s" in unit
|
||||
|
||||
# Per-unit override of journald rate limiting (default drops srcds output).
|
||||
assert "LogRateLimitIntervalSec=0" in unit
|
||||
|
||||
|
||||
def test_l4d2_game_slice_exists_with_high_weights():
|
||||
assert GAME_SLICE.is_file()
|
||||
text = GAME_SLICE.read_text()
|
||||
assert "[Slice]" in text
|
||||
assert "CPUWeight=1000" in text
|
||||
assert "IOWeight=1000" in text
|
||||
|
||||
|
||||
def test_l4d2_build_slice_exists_with_low_weights():
|
||||
assert BUILD_SLICE.is_file()
|
||||
text = BUILD_SLICE.read_text()
|
||||
assert "[Slice]" in text
|
||||
assert "CPUWeight=10" in text
|
||||
assert "IOWeight=10" in text
|
||||
|
||||
|
||||
def test_sysctl_conf_present_with_perf_settings():
|
||||
assert SYSCTL_CONF.is_file()
|
||||
text = SYSCTL_CONF.read_text()
|
||||
for line in (
|
||||
"net.core.rmem_max = 8388608",
|
||||
"net.core.wmem_max = 8388608",
|
||||
"net.core.rmem_default = 524288",
|
||||
"net.core.wmem_default = 524288",
|
||||
"net.core.netdev_max_backlog = 5000",
|
||||
"net.core.netdev_budget = 600",
|
||||
"vm.swappiness = 10",
|
||||
"net.ipv4.udp_rmem_min = 16384",
|
||||
"net.ipv4.udp_wmem_min = 16384",
|
||||
"net.core.default_qdisc = fq_codel",
|
||||
"net.ipv4.tcp_congestion_control = bbr",
|
||||
):
|
||||
assert line in text, f"missing {line!r} in 99-left4me.conf"
|
||||
|
||||
|
||||
def test_script_sandbox_in_build_slice_with_oom_adjust():
|
||||
text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||
|
||||
# Put the transient unit in the low-weight build slice so it yields to
|
||||
# game-server instances under CPU/IO contention.
|
||||
assert "--slice=l4d2-build.slice" in text
|
||||
|
||||
# Sandbox dies first if the host hits memory pressure; servers
|
||||
# (OOMScoreAdjust=-200) survive.
|
||||
assert "-p OOMScoreAdjust=500" in text
|
||||
|
||||
|
||||
def test_deploy_script_installs_perf_artifacts():
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
|
||||
# Slice files copied into the system-wide systemd unit dir.
|
||||
assert "/usr/local/lib/systemd/system/l4d2-game.slice" in script
|
||||
assert "/usr/local/lib/systemd/system/l4d2-build.slice" in script
|
||||
|
||||
# Sysctl drop-in installed under /etc/sysctl.d/.
|
||||
assert "/etc/sysctl.d/99-left4me.conf" in script
|
||||
|
||||
# Values applied immediately, not on next boot.
|
||||
assert "sysctl --system" in script
|
||||
|
||||
|
||||
def test_deploy_script_writes_cpuset_drop_ins():
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
|
||||
# Reads nproc and binds defaults via ${VAR:-...}.
|
||||
assert "nproc" in script
|
||||
assert "LEFT4ME_SYSTEM_CPUS" in script
|
||||
assert "LEFT4ME_GAME_CPUS" in script
|
||||
assert "${LEFT4ME_SYSTEM_CPUS:-0}" in script
|
||||
|
||||
# Default game-core upper bound is computed from nproc; accept either
|
||||
# the NPROC-1 form or LEFT4ME_GAME_CPUS:-1- prefix.
|
||||
assert (
|
||||
"1-$((NPROC - 1))" in script
|
||||
or "1-$((NPROC-1))" in script
|
||||
or "1-$((nproc-1))" in script
|
||||
or "LEFT4ME_GAME_CPUS:-1-" in script
|
||||
)
|
||||
|
||||
# All four drop-in paths.
|
||||
for slice_name in ("system", "user", "l4d2-build", "l4d2-game"):
|
||||
assert (
|
||||
f"/etc/systemd/system/{slice_name}.slice.d/99-left4me-cpuset.conf"
|
||||
in script
|
||||
)
|
||||
|
||||
# Drop-ins use the existing install pattern.
|
||||
assert "install -m 0644 -o root -g root" in script
|
||||
|
||||
# Single-core host: skip with a warning to stderr.
|
||||
assert ("-lt 2" in script) or ("< 2" in script) or ("-ge 2" in script)
|
||||
assert "skipping CPU isolation" in script
|
||||
|
||||
|
||||
def _fake_command(tmp_path, command_name):
|
||||
marker = tmp_path / f"{command_name}.args"
|
||||
command = tmp_path / command_name
|
||||
command.write_text(f"#!/bin/sh\nprintf '%s\n' \"$*\" > '{marker}'\nexit 0\n")
|
||||
command.chmod(0o755)
|
||||
return marker
|
||||
|
||||
|
||||
def _env_with_fake_commands(tmp_path):
|
||||
env = os.environ.copy()
|
||||
env["PATH"] = f"{tmp_path}{os.pathsep}{env.get('PATH', '')}"
|
||||
return env
|
||||
|
||||
|
||||
def test_helpers_use_fixed_system_tool_paths_not_sudo_path():
|
||||
systemctl = SYSTEMCTL_HELPER.read_text()
|
||||
journalctl = JOURNALCTL_HELPER.read_text()
|
||||
|
||||
assert "command -v systemctl" not in systemctl
|
||||
assert "command -v journalctl" not in journalctl
|
||||
assert "/bin/systemctl" in systemctl or "/usr/bin/systemctl" in systemctl
|
||||
assert "/bin/journalctl" in journalctl or "/usr/bin/journalctl" in journalctl
|
||||
|
||||
|
||||
def test_systemctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path):
|
||||
subprocess.run(["sh", "-n", str(SYSTEMCTL_HELPER)], check=True)
|
||||
marker = _fake_command(tmp_path, "systemctl")
|
||||
|
||||
for args in [
|
||||
["bad/action", "alpha"],
|
||||
# `start` and `stop` are no longer accepted verbs — the lifecycle now
|
||||
# uses `enable`/`disable` for reboot survival via WantedBy= symlinks.
|
||||
["start", "alpha"],
|
||||
["stop", "alpha"],
|
||||
["enable", ""],
|
||||
["enable", ".hidden"],
|
||||
["enable", "bad..name"],
|
||||
["enable", "bad/name"],
|
||||
["enable", "bad\\name"],
|
||||
["enable", "bad name"],
|
||||
]:
|
||||
result = subprocess.run(["sh", str(SYSTEMCTL_HELPER), *args], env=_env_with_fake_commands(tmp_path), check=False)
|
||||
assert result.returncode != 0
|
||||
assert not marker.exists()
|
||||
|
||||
script = SYSTEMCTL_HELPER.read_text()
|
||||
assert 'unit="left4me-server@${name}.service"' in script
|
||||
assert 'enable) exec "$systemctl" enable --now "$unit"' in script
|
||||
assert 'disable) exec "$systemctl" disable --now "$unit"' in script
|
||||
assert "--property=ActiveState" in script
|
||||
assert "--property=SubState" in script
|
||||
|
||||
|
||||
def test_journalctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path):
|
||||
subprocess.run(["sh", "-n", str(JOURNALCTL_HELPER)], check=True)
|
||||
marker = _fake_command(tmp_path, "journalctl")
|
||||
|
||||
for args in [
|
||||
["../evil", "--lines", "25", "--no-follow"],
|
||||
["alpha", "--bad", "25", "--no-follow"],
|
||||
["alpha", "--lines", "not-number", "--no-follow"],
|
||||
["alpha", "--lines", "25", "--bad-follow"],
|
||||
["bad/name", "--lines", "25", "--no-follow"],
|
||||
]:
|
||||
result = subprocess.run(["sh", str(JOURNALCTL_HELPER), *args], env=_env_with_fake_commands(tmp_path), check=False)
|
||||
assert result.returncode != 0
|
||||
assert not marker.exists()
|
||||
|
||||
script = JOURNALCTL_HELPER.read_text()
|
||||
assert 'unit="left4me-server@${name}.service"' in script
|
||||
assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat "$follow_arg"' in script
|
||||
assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat' in script
|
||||
|
||||
|
||||
def test_sudoers_allows_only_left4me_helpers_not_raw_system_tools():
|
||||
sudoers = SUDOERS.read_text()
|
||||
|
||||
assert (
|
||||
"left4me ALL=(root) NOPASSWD: "
|
||||
"/usr/local/libexec/left4me/left4me-systemctl *"
|
||||
) in sudoers
|
||||
assert (
|
||||
"left4me ALL=(root) NOPASSWD: "
|
||||
"/usr/local/libexec/left4me/left4me-journalctl *"
|
||||
) in sudoers
|
||||
assert "/usr/local/libexec/left4me/left4me-overlay mount *" in sudoers
|
||||
assert "/usr/local/libexec/left4me/left4me-overlay umount *" in sudoers
|
||||
assert (
|
||||
"left4me ALL=(root) NOPASSWD: "
|
||||
"/usr/local/libexec/left4me/left4me-script-sandbox"
|
||||
) in sudoers
|
||||
assert "/bin/systemctl" not in sudoers
|
||||
assert "/usr/bin/systemctl" not in sudoers
|
||||
assert "/bin/journalctl" not in sudoers
|
||||
assert "/usr/bin/journalctl" not in sudoers
|
||||
assert "/bin/mount" not in sudoers
|
||||
assert "/bin/umount" not in sudoers
|
||||
|
||||
|
||||
def test_overlay_helper_is_python_with_strict_validation():
|
||||
text = OVERLAY_HELPER.read_text()
|
||||
assert text.startswith("#!/usr/bin/python3")
|
||||
# Validation surface
|
||||
assert "NAME_RE = re.compile" in text
|
||||
assert "LOWERDIR_ALLOWLIST" in text
|
||||
assert "user.fuseoverlayfs." in text
|
||||
assert "MAX_LOWERDIRS = 500" in text
|
||||
# Mounts via PID 1's mount namespace
|
||||
assert "/proc/1/ns/mnt" in text
|
||||
assert "nsenter" in text
|
||||
# Verbs are mount and umount (not unmount)
|
||||
assert '"mount"' in text and '"umount"' in text
|
||||
assert '"unmount"' not in text
|
||||
|
||||
|
||||
def test_script_sandbox_uses_idmap_staging():
|
||||
"""The sandbox runs as l4d2-sandbox but writes need to land on disk as
|
||||
left4me, so all overlay content (workshop + script-built) is uniformly
|
||||
left4me-owned. The helper pre-creates an idmapped bind on a staging
|
||||
path and points the sandbox's BindPaths at the staging, not at the raw
|
||||
overlay dir. trap cleans up the staging bind on exit.
|
||||
"""
|
||||
text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||
# Idmap mount setup uses --map-users / --map-groups.
|
||||
assert "--map-users=" in text
|
||||
assert "--map-groups=" in text
|
||||
# Staging path lives under /var/lib/left4me/tmp/sandbox-idmap-<id>.
|
||||
assert "/var/lib/left4me/tmp/sandbox-idmap-" in text
|
||||
# BindPaths into the sandbox points at the staging path, not the
|
||||
# raw overlay dir.
|
||||
assert 'BindPaths="${STAGING}:/overlay"' in text
|
||||
# trap registers cleanup so the staging bind doesn't outlive the helper.
|
||||
assert "trap " in text and "cleanup_staging" in text
|
||||
# The previous chown-to-l4d2-sandbox approach is gone; overlay dirs
|
||||
# stay left4me-owned end-to-end.
|
||||
assert "chown -R l4d2-sandbox" not in text
|
||||
|
||||
|
||||
def test_deploy_script_installs_overlay_helper_with_executable_mode():
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
assert "/usr/local/libexec/left4me/left4me-overlay" in script
|
||||
assert "chmod 0755" in script and "left4me-overlay" in script
|
||||
|
||||
|
||||
def test_deploy_script_does_not_install_fuse_overlayfs_apt_dep():
|
||||
# fuse-overlayfs / fuse3 were the previous mount engine; kernel overlayfs
|
||||
# replaces them. Comments in the migration block may legitimately mention
|
||||
# the names, so scope this to the actual apt-get / dnf install lines.
|
||||
install_lines = [
|
||||
line for line in DEPLOY_SCRIPT.read_text().splitlines()
|
||||
if ("apt-get install" in line or "dnf install" in line)
|
||||
]
|
||||
assert install_lines, "expected at least one apt/dnf install line"
|
||||
for line in install_lines:
|
||||
assert "fuse-overlayfs" not in line, line
|
||||
assert "fuse3" not in line, line
|
||||
|
||||
|
||||
def test_deploy_script_runs_one_shot_kernel_overlay_migration():
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
assert "/var/lib/left4me/.kernel-overlay-migrated" in script
|
||||
# Migration should stop services + force-unmount stale mounts + wipe upper/work
|
||||
assert "systemctl stop 'left4me-server@" in script
|
||||
assert "systemctl stop left4me-web.service" in script
|
||||
assert "findmnt -t overlay" in script
|
||||
assert "/runtime/" in script and "rm -rf" in script and 'upper"' in script and 'work"' in script
|
||||
|
||||
|
||||
def test_env_templates_contain_required_defaults():
|
||||
host_env = HOST_ENV.read_text()
|
||||
assert "Deployment units use fixed /var/lib/left4me paths" in host_env
|
||||
assert host_env.endswith("LEFT4ME_ROOT=/var/lib/left4me\n")
|
||||
web_env = WEB_ENV_TEMPLATE.read_text()
|
||||
assert web_env.startswith(
|
||||
"DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n"
|
||||
"SECRET_KEY=replace-with-generated-secret\n"
|
||||
"JOB_WORKER_THREADS=4\n"
|
||||
)
|
||||
assert web_env.rstrip().endswith("STEAM_WEB_API_KEY=")
|
||||
|
||||
|
||||
def test_deploy_script_has_safe_defaults_and_preserves_state() -> None:
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
|
||||
assert "useradd --system --home-dir /var/lib/left4me" in script
|
||||
assert "/var/lib/left4me/installation" in script
|
||||
assert "/var/lib/left4me/overlays" in script
|
||||
assert "/var/lib/left4me/instances" in script
|
||||
assert "/var/lib/left4me/runtime" in script
|
||||
assert "tar" in script
|
||||
assert "--exclude .venv" in script
|
||||
assert "--exclude .claude" in script
|
||||
assert "pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web" in script
|
||||
assert "systemctl enable --now left4me-web.service" in script
|
||||
assert "for attempt in" in script
|
||||
assert "/opt/left4me/.venv" in script
|
||||
assert "visudo -cf /etc/sudoers.d/left4me" in script
|
||||
# Note: assertions about web.env's lifecycle (create-only-if-missing /
|
||||
# never-sourced-from-deploy) used to live here. They became stale in
|
||||
# commit caa8b83, which switched to "rewrite web.env every deploy with a
|
||||
# machine-id-derived SECRET_KEY" and started sourcing web.env in the
|
||||
# alembic + seed helper subprocesses. Removed entirely; current behavior
|
||||
# is covered by `install -m 0640 ... /etc/left4me/web.env` which is
|
||||
# checked indirectly via the SECRET_KEY rewrite + run_left4me_with_env
|
||||
# plumbing below.
|
||||
assert "run_left4me_with_env" in script
|
||||
assert "LEFT4ME_ADMIN_USERNAME" in script
|
||||
assert "LEFT4ME_ADMIN_PASSWORD" in script
|
||||
assert "user already exists" in script
|
||||
assert "deploy/files" in script
|
||||
|
||||
|
||||
def test_deploy_script_does_not_recurse_into_runtime_state_mounts() -> None:
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
|
||||
assert "$sudo_cmd chown -R left4me:left4me /var/lib/left4me" not in script
|
||||
assert "$sudo_cmd chown left4me:left4me \\" in script
|
||||
assert "/var/lib/left4me/runtime \\" in script
|
||||
assert "$sudo_cmd chown -R left4me:left4me /opt/left4me" in script
|
||||
|
||||
|
||||
def test_deploy_script_runs_migrations_before_app_initialization() -> None:
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
|
||||
assert "alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head" in script
|
||||
assert "from l4d2web.app import create_app; create_app()" not in script
|
||||
|
||||
|
||||
def test_deploy_script_shell_syntax() -> None:
|
||||
subprocess.run(["sh", "-n", str(DEPLOY_SCRIPT)], check=True)
|
||||
|
||||
|
||||
def test_globals_refresh_units_removed():
|
||||
"""Global-overlays subsystem deleted in favor of script overlays."""
|
||||
assert not GLOBAL_REFRESH_SERVICE.exists()
|
||||
assert not GLOBAL_REFRESH_TIMER.exists()
|
||||
|
||||
|
||||
def test_deploy_script_does_not_provision_globals_subsystem():
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
|
||||
# No mkdir/install of the deleted cache dir; mention in a one-shot
|
||||
# `rm -rf` cleanup is fine.
|
||||
for line in script.splitlines():
|
||||
if "/var/lib/left4me/global_overlay_cache" not in line:
|
||||
continue
|
||||
assert "rm -rf" in line, line
|
||||
assert "left4me-refresh-global-overlays" not in script
|
||||
|
||||
|
||||
def test_deploy_script_provisions_sandbox_user():
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
assert "useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox" in script
|
||||
|
||||
|
||||
def test_deploy_script_does_not_install_bubblewrap():
|
||||
install_lines = [
|
||||
line for line in DEPLOY_SCRIPT.read_text().splitlines()
|
||||
if ("apt-get install" in line or "dnf install" in line)
|
||||
]
|
||||
assert install_lines, "expected at least one apt/dnf install line"
|
||||
for line in install_lines:
|
||||
assert "bubblewrap" not in line, line
|
||||
assert "bwrap" not in line, line
|
||||
|
||||
|
||||
def test_deploy_script_installs_script_overlay_tooling():
|
||||
# Script overlays commonly need 7z and md5sum (e.g. l4d2center map sync).
|
||||
# coreutils ships md5sum and is technically essential, but listing it
|
||||
# explicitly makes the contract obvious and survives slim base images.
|
||||
script = DEPLOY_SCRIPT.read_text().splitlines()
|
||||
apt_lines = [l for l in script if "apt-get install" in l]
|
||||
dnf_lines = [l for l in script if "dnf install" in l]
|
||||
assert apt_lines, "expected an apt-get install line"
|
||||
assert dnf_lines, "expected a dnf install line"
|
||||
for line in apt_lines:
|
||||
assert "p7zip-full" in line, line
|
||||
assert "coreutils" in line, line
|
||||
for line in dnf_lines:
|
||||
# Fedora/RHEL split: p7zip provides 7za, p7zip-plugins provides 7z.
|
||||
assert "p7zip" in line and "p7zip-plugins" in line, line
|
||||
assert "coreutils" in line, line
|
||||
|
||||
|
||||
def test_deploy_script_tightens_left4me_db_permissions():
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
# The DB and its WAL/SHM sidecars must be left4me:left4me 0640 — owner
|
||||
# (web service) keeps rw, group is read-only, "other" (incl. l4d2-sandbox)
|
||||
# gets nothing. The sidecars matter because SQLite in WAL mode requires
|
||||
# write access to all three; if a sidecar ends up root-owned (e.g. from
|
||||
# ad-hoc root-side inspection), the next write fails as "readonly db".
|
||||
assert "chown left4me:left4me" in script
|
||||
assert "chmod 0640" in script
|
||||
for db_file in (
|
||||
"/var/lib/left4me/left4me.db",
|
||||
"/var/lib/left4me/left4me.db-wal",
|
||||
"/var/lib/left4me/left4me.db-shm",
|
||||
):
|
||||
assert db_file in script, f"deploy script must touch {db_file}"
|
||||
|
||||
|
||||
def test_deploy_script_installs_script_sandbox_helper():
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
assert "/usr/local/libexec/left4me/left4me-script-sandbox" in script
|
||||
assert "chmod 0755" in script and "left4me-script-sandbox" in script
|
||||
|
||||
|
||||
def test_script_sandbox_helper_present():
|
||||
assert SCRIPT_SANDBOX_HELPER.is_file()
|
||||
assert SCRIPT_SANDBOX_HELPER.read_text().startswith("#!/bin/bash")
|
||||
mode = SCRIPT_SANDBOX_HELPER.stat().st_mode & 0o777
|
||||
assert mode == 0o755, f"expected 0755, got {oct(mode)}"
|
||||
|
||||
|
||||
def test_script_sandbox_helper_passes_shell_syntax_check():
|
||||
subprocess.run(["bash", "-n", str(SCRIPT_SANDBOX_HELPER)], check=True)
|
||||
|
||||
|
||||
def test_script_sandbox_helper_invokes_systemd_run_with_hardening():
|
||||
text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||
|
||||
# systemd-run service mode (no --scope), with synchronous I/O to caller.
|
||||
assert "systemd-run" in text
|
||||
assert "--scope" not in text, "v2 uses transient service units, not scopes"
|
||||
assert "--pipe" in text
|
||||
assert "--wait" in text
|
||||
assert "--collect" in text
|
||||
assert "--unit=" in text
|
||||
|
||||
# No bwrap.
|
||||
assert "bwrap" not in text
|
||||
assert "bubblewrap" not in text
|
||||
|
||||
# UID drop via systemd directives.
|
||||
assert "User=l4d2-sandbox" in text
|
||||
assert "Group=l4d2-sandbox" in text
|
||||
|
||||
# Cgroup limits unchanged from v1.
|
||||
assert "MemoryMax=4G" in text
|
||||
assert "MemorySwapMax=0" in text
|
||||
assert "TasksMax=512" in text
|
||||
assert "CPUQuota=200%" in text
|
||||
assert "RuntimeMaxSec=3600" in text
|
||||
|
||||
# Hardening directives that v1 (scope mode) couldn't carry.
|
||||
assert "NoNewPrivileges=yes" in text
|
||||
assert "ProtectSystem=strict" in text
|
||||
assert "ProtectHome=yes" in text
|
||||
assert "PrivateTmp=yes" in text
|
||||
assert "PrivateDevices=yes" in text
|
||||
assert "PrivateIPC=yes" in text
|
||||
assert "ProtectKernelTunables=yes" in text
|
||||
assert "ProtectKernelModules=yes" in text
|
||||
assert "ProtectKernelLogs=yes" in text
|
||||
assert "ProtectControlGroups=yes" in text
|
||||
assert "RestrictNamespaces=yes" in text
|
||||
assert "RestrictSUIDSGID=yes" in text
|
||||
assert "LockPersonality=yes" in text
|
||||
assert "MemoryDenyWriteExecute=yes" in text
|
||||
assert "SystemCallFilter=" in text
|
||||
assert "@system-service" in text
|
||||
assert "@network-io" in text
|
||||
assert "CapabilityBoundingSet=" in text
|
||||
assert "AmbientCapabilities=" in text
|
||||
assert 'RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX"' in text
|
||||
|
||||
# Network namespace stays shared with host.
|
||||
assert "PrivateNetwork=" not in text
|
||||
|
||||
# Mount setup: /etc and /var/lib masked with tmpfs; selective binds back.
|
||||
assert 'TemporaryFileSystem="/etc /var/lib"' in text
|
||||
assert "BindReadOnlyPaths=" in text
|
||||
# The resolv.conf bind points at the sandbox-only file (not the host's
|
||||
# /etc/resolv.conf, which typically references a private-IP DNS server
|
||||
# that IPAddressDeny= blocks).
|
||||
assert "/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf" in text
|
||||
assert "/etc/ssl" in text
|
||||
assert "/etc/ca-certificates" in text
|
||||
assert "/etc/nsswitch.conf" in text
|
||||
assert "/etc/alternatives" in text
|
||||
assert "${SCRIPT}:/script.sh" in text
|
||||
assert 'BindPaths="${STAGING}:/overlay"' in text
|
||||
|
||||
# IP egress filter: allow public, deny localhost / RFC1918 / link-local /
|
||||
# multicast / CGNAT / ULA. systemd's "more specific rule wins" semantics
|
||||
# mean public IPs hit the allow and listed ranges hit the deny.
|
||||
# IPAddressDeny alone — no IPAddressAllow=any. Empirically, having both
|
||||
# set causes the allow to win on this systemd/kernel combo regardless of
|
||||
# the documented "more specific rule wins" behaviour. With only Deny,
|
||||
# the kernel's default "allow all" applies to non-listed addresses.
|
||||
assert "IPAddressDeny=" in text
|
||||
assert "IPAddressAllow=any" not in text
|
||||
# Explicit CIDRs — systemd-run's -p parser doesn't accept the
|
||||
# `localhost` / `link-local` / `multicast` shorthand keywords that
|
||||
# work in unit files (only the full strings parse).
|
||||
for token in (
|
||||
"127.0.0.0/8",
|
||||
"::1/128",
|
||||
"169.254.0.0/16",
|
||||
"fe80::/10",
|
||||
"224.0.0.0/4",
|
||||
"ff00::/8",
|
||||
"10.0.0.0/8",
|
||||
"172.16.0.0/12",
|
||||
"192.168.0.0/16",
|
||||
"100.64.0.0/10",
|
||||
"fc00::/7",
|
||||
):
|
||||
assert token in text, f"missing {token!r} in IPAddressDeny set"
|
||||
|
||||
|
||||
def test_sandbox_resolv_conf_exists():
|
||||
assert SANDBOX_RESOLV_CONF.is_file()
|
||||
text = SANDBOX_RESOLV_CONF.read_text()
|
||||
nameservers = [
|
||||
line.split()[1]
|
||||
for line in text.splitlines()
|
||||
if line.startswith("nameserver ")
|
||||
]
|
||||
assert len(nameservers) >= 2, "expected at least two nameservers for redundancy"
|
||||
# Sanity: the resolvers must be public (not RFC1918 / loopback). We don't
|
||||
# pin the exact IPs — Cloudflare/Google/Quad9 are all acceptable.
|
||||
for ns in nameservers:
|
||||
assert not ns.startswith("127."), ns
|
||||
assert not ns.startswith("10."), ns
|
||||
assert not ns.startswith("192.168."), ns
|
||||
first_octet = int(ns.split(".")[0])
|
||||
# Reject 172.16.0.0/12.
|
||||
if first_octet == 172:
|
||||
second_octet = int(ns.split(".")[1])
|
||||
assert not (16 <= second_octet <= 31), ns
|
||||
|
||||
|
||||
def test_deploy_script_installs_sandbox_resolv_conf():
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
assert "deploy/files/etc/left4me/sandbox-resolv.conf" in script
|
||||
assert "/etc/left4me/sandbox-resolv.conf" in script
|
||||
|
||||
|
||||
def test_script_sandbox_helper_validates_overlay_id():
|
||||
text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||
# Numeric-only overlay id
|
||||
assert '[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]]' in text
|
||||
# Overlay dir must exist
|
||||
assert "/var/lib/left4me/overlays/" in text
|
||||
assert "[[ -d $OVERLAY_DIR ]]" in text
|
||||
# Script path must exist
|
||||
assert "[[ -f $SCRIPT ]]" in text
|
||||
|
||||
|
||||
def test_script_sandbox_helper_dry_run_mode(tmp_path):
|
||||
overlay_root = tmp_path / "var/lib/left4me/overlays/42"
|
||||
overlay_root.mkdir(parents=True)
|
||||
fake_script = tmp_path / "fake.sh"
|
||||
fake_script.write_text("echo hi")
|
||||
|
||||
# Run in DRY_RUN mode against a fake l4d2-sandbox UID via a tiny shim that
|
||||
# simulates `id -u l4d2-sandbox` resolving to a valid number.
|
||||
helper_text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||
# We can't actually exec this without root + a real sandbox user; just
|
||||
# verify the dry-run guard short-circuits before systemd-run / bwrap.
|
||||
assert 'LEFT4ME_SCRIPT_SANDBOX_DRY_RUN' in helper_text
|
||||
assert 'exit 0' in helper_text
|
||||
|
||||
|
||||
def test_nft_mark_file_marks_left4me_udp_with_dscp_ef_and_priority():
|
||||
assert NFT_MARK_FILE.is_file()
|
||||
text = NFT_MARK_FILE.read_text()
|
||||
|
||||
# Own table in the inet family so it cannot conflict with operator nftables config.
|
||||
assert "table inet left4me_mark" in text
|
||||
assert "chain mangle_output" in text
|
||||
assert "type filter hook output priority mangle" in text
|
||||
|
||||
# Match by uid (every srcds runs as `left4me`) restricted to UDP.
|
||||
assert 'meta skuid "left4me"' in text
|
||||
assert "meta l4proto udp" in text
|
||||
|
||||
# DSCP EF for both L3 families; in `inet` tables, `ip` only fires on v4
|
||||
# and `ip6` only on v6.
|
||||
assert "ip dscp set ef" in text
|
||||
assert "ip6 dscp set ef" in text
|
||||
|
||||
# skb->priority class 6:0, set inline alongside DSCP.
|
||||
assert "meta priority set 0006:0000" in text
|
||||
|
||||
|
||||
def test_nft_mark_unit_loads_and_clears_left4me_table():
|
||||
assert NFT_MARK_UNIT.is_file()
|
||||
text = NFT_MARK_UNIT.read_text()
|
||||
|
||||
# Loads the rules early so the very first packet srcds emits is marked.
|
||||
assert "After=network-pre.target" in text
|
||||
assert "Before=network.target" in text
|
||||
assert "Wants=network-pre.target" in text
|
||||
|
||||
# Oneshot lifecycle: load on start, drop on stop.
|
||||
assert "Type=oneshot" in text
|
||||
assert "RemainAfterExit=yes" in text
|
||||
assert (
|
||||
"ExecStart=/usr/sbin/nft -f /usr/local/lib/left4me/nft/left4me-mark.nft"
|
||||
in text
|
||||
)
|
||||
assert "ExecStop=/usr/sbin/nft delete table inet left4me_mark" in text
|
||||
assert "WantedBy=multi-user.target" in text
|
||||
|
||||
|
||||
def test_cake_env_template_documents_required_knobs():
|
||||
assert CAKE_ENV.is_file()
|
||||
text = CAKE_ENV.read_text()
|
||||
|
||||
# Both knobs are documented and present (commented OK; the deploy preserves
|
||||
# operator edits, so the template must not bake in a wrong value).
|
||||
assert "LEFT4ME_UPLINK_MBIT" in text
|
||||
assert "LEFT4ME_UPLINK_IFACE" in text
|
||||
# Empty defaults: shaper unit no-ops with a journal warning when unset.
|
||||
assert "LEFT4ME_UPLINK_MBIT=" in text
|
||||
assert "LEFT4ME_UPLINK_IFACE=" in text
|
||||
|
||||
|
||||
def test_apply_cake_helper_supports_apply_and_clear_modes():
|
||||
assert APPLY_CAKE_HELPER.is_file()
|
||||
text = APPLY_CAKE_HELPER.read_text()
|
||||
|
||||
assert text.startswith("#!/bin/sh")
|
||||
# Both knobs are read from the env file.
|
||||
assert "LEFT4ME_UPLINK_MBIT" in text
|
||||
assert "LEFT4ME_UPLINK_IFACE" in text
|
||||
assert ". /etc/left4me/cake.env" in text
|
||||
# Iface fallback to default route.
|
||||
assert "ip -4 route show default" in text
|
||||
# Two modes; default to apply.
|
||||
assert "mode=${1:-apply}" in text
|
||||
assert 'apply)' in text and 'clear)' in text
|
||||
# Apply: idempotent `tc qdisc replace` with the documented flags.
|
||||
assert "tc qdisc replace" in text
|
||||
assert "cake" in text
|
||||
assert "bandwidth" in text
|
||||
assert "internet" in text
|
||||
assert "diffserv4" in text
|
||||
assert "dual-dsthost" in text
|
||||
# Clear: tolerates a missing qdisc.
|
||||
assert "tc qdisc del" in text
|
||||
assert "|| true" in text
|
||||
# Fail-soft on missing config.
|
||||
assert "LEFT4ME_UPLINK_MBIT unset" in text
|
||||
|
||||
|
||||
def test_apply_cake_helper_passes_shell_syntax_check():
|
||||
subprocess.run(["sh", "-n", str(APPLY_CAKE_HELPER)], check=True)
|
||||
|
||||
|
||||
def test_cake_unit_runs_helper_in_apply_and_clear_modes():
|
||||
assert CAKE_UNIT.is_file()
|
||||
text = CAKE_UNIT.read_text()
|
||||
|
||||
assert "After=network-online.target" in text
|
||||
assert "Wants=network-online.target" in text
|
||||
assert "Type=oneshot" in text
|
||||
assert "RemainAfterExit=yes" in text
|
||||
# `-` prefix: missing env file is non-fatal (deploy ships one, but be safe).
|
||||
assert "EnvironmentFile=-/etc/left4me/cake.env" in text
|
||||
assert (
|
||||
"ExecStart=/usr/local/libexec/left4me/left4me-apply-cake apply" in text
|
||||
)
|
||||
assert (
|
||||
"ExecStop=/usr/local/libexec/left4me/left4me-apply-cake clear" in text
|
||||
)
|
||||
assert "WantedBy=multi-user.target" in text
|
||||
|
||||
|
||||
def test_deploy_script_installs_network_shaping_artifacts():
|
||||
script = DEPLOY_SCRIPT.read_text()
|
||||
|
||||
# nftables: package install on both apt and dnf paths.
|
||||
apt_lines = [l for l in script.splitlines() if "apt-get install" in l]
|
||||
dnf_lines = [l for l in script.splitlines() if "dnf install" in l]
|
||||
assert apt_lines and dnf_lines
|
||||
for line in apt_lines:
|
||||
assert "nftables" in line, line
|
||||
for line in dnf_lines:
|
||||
assert "nftables" in line, line
|
||||
|
||||
# nft rules + unit copied to system paths.
|
||||
assert "/usr/local/lib/left4me/nft/left4me-mark.nft" in script
|
||||
assert (
|
||||
"/usr/local/lib/systemd/system/left4me-nft-mark.service" in script
|
||||
)
|
||||
assert "systemctl enable --now left4me-nft-mark.service" in script
|
||||
|
||||
# CAKE helper + unit copied; helper made executable.
|
||||
assert "/usr/local/libexec/left4me/left4me-apply-cake" in script
|
||||
assert (
|
||||
"/usr/local/lib/systemd/system/left4me-cake.service" in script
|
||||
)
|
||||
assert "chmod 0755" in script and "left4me-apply-cake" in script
|
||||
assert "systemctl enable --now left4me-cake.service" in script
|
||||
|
||||
# cake.env: copied only if absent (operator edits survive re-deploys).
|
||||
assert "/etc/left4me/cake.env" in script
|
||||
assert "[ -e /etc/left4me/cake.env ]" in script
|
||||
233
deploy/tests/test_example_units.py
Normal file
233
deploy/tests/test_example_units.py
Normal file
|
|
@ -0,0 +1,233 @@
|
|||
"""Lockdown tests for the curated examples kept under `deploy/files/`.
|
||||
|
||||
`deploy/` is reference material. The production units are emitted by
|
||||
ckn-bw's `systemd_units` reactor in `bundles/left4me/metadata.py`;
|
||||
when reactor output drifts intentionally, update these examples to match.
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
DEPLOY = ROOT / "deploy"
|
||||
|
||||
|
||||
WEB_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-web.service"
|
||||
SERVER_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-server@.service"
|
||||
GAME_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-game.slice"
|
||||
BUILD_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-build.slice"
|
||||
SYSCTL_CONF = DEPLOY / "files/etc/sysctl.d/99-left4me.conf"
|
||||
SANDBOX_RESOLV_CONF = DEPLOY / "files/etc/left4me/sandbox-resolv.conf"
|
||||
HOST_ENV = DEPLOY / "templates/etc/left4me/host.env"
|
||||
WEB_ENV_TEMPLATE = DEPLOY / "templates/etc/left4me/web.env.template"
|
||||
|
||||
|
||||
def test_global_unit_files_exist_at_product_level_paths():
|
||||
assert WEB_UNIT.is_file()
|
||||
assert SERVER_UNIT.is_file()
|
||||
|
||||
|
||||
def test_web_unit_contains_required_runtime_contract():
|
||||
unit = WEB_UNIT.read_text()
|
||||
|
||||
assert "User=left4me" in unit
|
||||
assert "Group=left4me" in unit
|
||||
assert "WorkingDirectory=/opt/left4me" in unit
|
||||
assert "Environment=PATH=/opt/left4me/.venv/bin:" in unit
|
||||
assert "EnvironmentFile=/etc/left4me/host.env" in unit
|
||||
assert "EnvironmentFile=/etc/left4me/web.env" in unit
|
||||
assert "ExecStart=/opt/left4me/.venv/bin/gunicorn" in unit
|
||||
assert "--workers 1" in unit
|
||||
assert "--threads 32" in unit
|
||||
# NoNewPrivileges must remain unset because sudo (used by the overlay,
|
||||
# systemctl and journalctl helpers) is setuid.
|
||||
assert "NoNewPrivileges=true" not in unit
|
||||
# Restored now that fuse-overlayfs propagation is no longer the mechanism.
|
||||
assert "PrivateTmp=true" in unit
|
||||
assert "ProtectSystem=full" in unit
|
||||
assert "ReadWritePaths=/var/lib/left4me" in unit
|
||||
# Mounts now happen in PID 1's namespace via the left4me-overlay helper,
|
||||
# so MountFlags propagation is irrelevant — and the previous assumption
|
||||
# that MountFlags=shared made it work was incorrect.
|
||||
assert "MountFlags=" not in unit
|
||||
|
||||
|
||||
def test_server_unit_contains_required_runtime_contract():
|
||||
unit = SERVER_UNIT.read_text()
|
||||
|
||||
assert "User=left4me" in unit
|
||||
assert "Group=left4me" in unit
|
||||
assert "EnvironmentFile=/etc/left4me/host.env" in unit
|
||||
assert "EnvironmentFile=/var/lib/left4me/instances/%i/instance.env" in unit
|
||||
# `-` prefix: chdir failure is non-fatal so ExecStartPre can run the
|
||||
# mount helper before the merged dir exists. ExecStart re-applies and
|
||||
# finds the dir once the mount has landed.
|
||||
assert "WorkingDirectory=-/var/lib/left4me/runtime/%i/merged/left4dead2" in unit
|
||||
# ExecStart must invoke srcds_run from the *merged* overlay tree, not
|
||||
# from installation/. srcds_run cds to its own dirname; if we point at
|
||||
# installation/, the engine reads gameinfo.txt and addons from the lower
|
||||
# layer and never sees overlay plugins (Metamod/SourceMod) or cfgs.
|
||||
assert "ExecStart=/var/lib/left4me/runtime/%i/merged/srcds_run" in unit
|
||||
assert "$L4D2_ARGS" in unit
|
||||
assert "${L4D2_ARGS}" not in unit
|
||||
assert "NoNewPrivileges=true" in unit
|
||||
assert "PrivateTmp=true" in unit
|
||||
assert "PrivateDevices=true" in unit
|
||||
assert "ProtectHome=true" in unit
|
||||
assert "ProtectSystem=strict" in unit
|
||||
assert "ReadOnlyPaths=/var/lib/left4me/installation /var/lib/left4me/overlays" in unit
|
||||
assert "ReadWritePaths=/var/lib/left4me/runtime/%i" in unit
|
||||
assert "RestrictSUIDSGID=true" in unit
|
||||
assert "LockPersonality=true" in unit
|
||||
|
||||
|
||||
def test_server_unit_mounts_overlay_via_exec_start_pre():
|
||||
"""At boot, systemd auto-starts enabled units before the web app gets a
|
||||
chance to run start_instance's pre-start mount. The unit itself must
|
||||
re-mount the overlay so reboots are transparent. Pairs with the helper's
|
||||
idempotency check (test_overlay_helper_mount_is_idempotent_when_mounted).
|
||||
|
||||
The unit-level `nsenter --mount=/proc/1/ns/mnt --` is what makes
|
||||
umount fast: without it, the helper Python process would inherit
|
||||
the unit's per-service mount namespace and pin it alive, blocking
|
||||
PID 1's umount until the helper exited. Wrapping with nsenter at
|
||||
the Exec line puts the helper itself in PID 1's namespace.
|
||||
"""
|
||||
unit = SERVER_UNIT.read_text()
|
||||
# `+` prefix: runs as PID 1 (root, no sandbox). Required because
|
||||
# the unit has NoNewPrivileges=true, which blocks sudo's setuid
|
||||
# escalation — and the helper needs root for the mount syscall.
|
||||
assert (
|
||||
"ExecStartPre=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- "
|
||||
"/usr/local/libexec/left4me/left4me-overlay mount %i"
|
||||
in unit
|
||||
)
|
||||
# Bound the restart loop; without these, a CHDIR-failure (or any other
|
||||
# pre-start error) spins indefinitely.
|
||||
assert "StartLimitBurst=5" in unit
|
||||
assert "StartLimitIntervalSec=60s" in unit
|
||||
|
||||
|
||||
def test_server_unit_unmounts_overlay_via_exec_stop_post():
|
||||
"""Single source of truth for unmount, mirroring the mount path.
|
||||
ExecStopPost (not ExecStop) so it runs after srcds has fully exited
|
||||
and the cgroup is cleared.
|
||||
|
||||
Same nsenter-at-Exec-line wrapping as ExecStartPre — without it,
|
||||
the helper process would itself hold a reference to the unit's
|
||||
per-service mount namespace, and umount in PID 1 would loop on
|
||||
EBUSY until the helper gave up. With it, umount succeeds first try.
|
||||
"""
|
||||
unit = SERVER_UNIT.read_text()
|
||||
assert (
|
||||
"ExecStopPost=+/usr/bin/nsenter --mount=/proc/1/ns/mnt -- "
|
||||
"/usr/local/libexec/left4me/left4me-overlay umount %i"
|
||||
in unit
|
||||
)
|
||||
|
||||
|
||||
def test_server_unit_contains_perf_baseline_directives():
|
||||
unit = SERVER_UNIT.read_text()
|
||||
|
||||
# Slice membership.
|
||||
assert "Slice=l4d2-game.slice" in unit
|
||||
|
||||
# CFS priority bump (no SCHED_FIFO).
|
||||
assert "Nice=-5" in unit
|
||||
assert "CPUSchedulingPolicy=" not in unit
|
||||
|
||||
# I/O priority.
|
||||
assert "IOSchedulingClass=best-effort" in unit
|
||||
assert "IOSchedulingPriority=4" in unit
|
||||
|
||||
# OOM ordering: game servers survive, sandbox dies first.
|
||||
assert "OOMScoreAdjust=-200" in unit
|
||||
|
||||
# Memory caps with headroom for map-load spikes.
|
||||
assert "MemoryHigh=1.5G" in unit
|
||||
assert "MemoryMax=2G" in unit
|
||||
|
||||
# Bounded fork surface.
|
||||
assert "TasksMax=256" in unit
|
||||
|
||||
# Plenty of fds for plugin-heavy setups.
|
||||
assert "LimitNOFILE=65536" in unit
|
||||
|
||||
# srcds clean shutdown via SIGINT, with time to flush. With the
|
||||
# helper running in PID 1's mount namespace (via the unit-level
|
||||
# nsenter on ExecStopPost), umount has no race window and the
|
||||
# default 15 s is plenty for the whole stop transition.
|
||||
assert "KillSignal=SIGINT" in unit
|
||||
assert "TimeoutStopSec=15s" in unit
|
||||
|
||||
# Per-unit override of journald rate limiting (default drops srcds output).
|
||||
assert "LogRateLimitIntervalSec=0" in unit
|
||||
|
||||
|
||||
def test_l4d2_game_slice_exists_with_high_weights():
|
||||
assert GAME_SLICE.is_file()
|
||||
text = GAME_SLICE.read_text()
|
||||
assert "[Slice]" in text
|
||||
assert "CPUWeight=1000" in text
|
||||
assert "IOWeight=1000" in text
|
||||
|
||||
|
||||
def test_l4d2_build_slice_exists_with_low_weights():
|
||||
assert BUILD_SLICE.is_file()
|
||||
text = BUILD_SLICE.read_text()
|
||||
assert "[Slice]" in text
|
||||
assert "CPUWeight=10" in text
|
||||
assert "IOWeight=10" in text
|
||||
|
||||
|
||||
def test_sysctl_conf_present_with_perf_settings():
|
||||
assert SYSCTL_CONF.is_file()
|
||||
text = SYSCTL_CONF.read_text()
|
||||
for line in (
|
||||
"net.core.rmem_max = 8388608",
|
||||
"net.core.wmem_max = 8388608",
|
||||
"net.core.rmem_default = 524288",
|
||||
"net.core.wmem_default = 524288",
|
||||
"net.core.netdev_max_backlog = 5000",
|
||||
"net.core.netdev_budget = 600",
|
||||
"vm.swappiness = 10",
|
||||
"net.ipv4.udp_rmem_min = 16384",
|
||||
"net.ipv4.udp_wmem_min = 16384",
|
||||
"net.core.default_qdisc = fq_codel",
|
||||
"net.ipv4.tcp_congestion_control = bbr",
|
||||
):
|
||||
assert line in text, f"missing {line!r} in 99-left4me.conf"
|
||||
|
||||
|
||||
def test_env_templates_contain_required_defaults():
|
||||
host_env = HOST_ENV.read_text()
|
||||
assert "Deployment units use fixed /var/lib/left4me paths" in host_env
|
||||
assert host_env.endswith("LEFT4ME_ROOT=/var/lib/left4me\n")
|
||||
web_env = WEB_ENV_TEMPLATE.read_text()
|
||||
assert web_env.startswith(
|
||||
"DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n"
|
||||
"SECRET_KEY=replace-with-generated-secret\n"
|
||||
"JOB_WORKER_THREADS=4\n"
|
||||
)
|
||||
assert web_env.rstrip().endswith("STEAM_WEB_API_KEY=")
|
||||
|
||||
|
||||
def test_sandbox_resolv_conf_exists():
|
||||
assert SANDBOX_RESOLV_CONF.is_file()
|
||||
text = SANDBOX_RESOLV_CONF.read_text()
|
||||
nameservers = [
|
||||
line.split()[1]
|
||||
for line in text.splitlines()
|
||||
if line.startswith("nameserver ")
|
||||
]
|
||||
assert len(nameservers) >= 2, "expected at least two nameservers for redundancy"
|
||||
# Sanity: the resolvers must be public (not RFC1918 / loopback). We don't
|
||||
# pin the exact IPs — Cloudflare/Google/Quad9 are all acceptable.
|
||||
for ns in nameservers:
|
||||
assert not ns.startswith("127."), ns
|
||||
assert not ns.startswith("10."), ns
|
||||
assert not ns.startswith("192.168."), ns
|
||||
first_octet = int(ns.split(".")[0])
|
||||
# Reject 172.16.0.0/12.
|
||||
if first_octet == 172:
|
||||
second_octet = int(ns.split(".")[1])
|
||||
assert not (16 <= second_octet <= 31), ns
|
||||
|
|
@ -9,12 +9,8 @@ import pytest
|
|||
|
||||
HELPER_SOURCE = (
|
||||
Path(__file__).resolve().parents[2]
|
||||
/ "deploy"
|
||||
/ "files"
|
||||
/ "usr"
|
||||
/ "local"
|
||||
/ "scripts"
|
||||
/ "libexec"
|
||||
/ "left4me"
|
||||
/ "left4me-overlay"
|
||||
)
|
||||
|
||||
|
|
|
|||
36
scripts/tests/conftest.py
Normal file
36
scripts/tests/conftest.py
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
"""Shared fixtures and path constants for `scripts/tests/`."""
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
SCRIPTS = ROOT / "scripts"
|
||||
LIBEXEC = SCRIPTS / "libexec"
|
||||
SBIN = SCRIPTS / "sbin"
|
||||
|
||||
# `deploy/` is reference material; the sudoers example lives there and is
|
||||
# the canonical statement of which paths sudo grants to the `left4me` uid.
|
||||
# `scripts/tests/test_sudoers_grants.py` reads it from across the dir
|
||||
# boundary because the contract being audited is about *scripts*, not deploy.
|
||||
DEPLOY = ROOT / "deploy"
|
||||
|
||||
|
||||
def fake_command(tmp_path, command_name):
|
||||
"""Drop a no-op stub of `command_name` into `tmp_path`. Returns the
|
||||
marker file the stub writes its args to, so tests can assert that the
|
||||
helper rejected bad input before invoking the real command.
|
||||
"""
|
||||
marker = tmp_path / f"{command_name}.args"
|
||||
command = tmp_path / command_name
|
||||
command.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" > '{marker}'\nexit 0\n")
|
||||
command.chmod(0o755)
|
||||
return marker
|
||||
|
||||
|
||||
def env_with_fake_commands(tmp_path):
|
||||
"""Build an environment that prepends `tmp_path` onto PATH so helpers
|
||||
find the fake commands first.
|
||||
"""
|
||||
env = os.environ.copy()
|
||||
env["PATH"] = f"{tmp_path}{os.pathsep}{env.get('PATH', '')}"
|
||||
return env
|
||||
15
scripts/tests/test_helpers_use_fixed_paths.py
Normal file
15
scripts/tests/test_helpers_use_fixed_paths.py
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
from conftest import LIBEXEC
|
||||
|
||||
|
||||
SYSTEMCTL_HELPER = LIBEXEC / "left4me-systemctl"
|
||||
JOURNALCTL_HELPER = LIBEXEC / "left4me-journalctl"
|
||||
|
||||
|
||||
def test_helpers_use_fixed_system_tool_paths_not_sudo_path():
|
||||
systemctl = SYSTEMCTL_HELPER.read_text()
|
||||
journalctl = JOURNALCTL_HELPER.read_text()
|
||||
|
||||
assert "command -v systemctl" not in systemctl
|
||||
assert "command -v journalctl" not in journalctl
|
||||
assert "/bin/systemctl" in systemctl or "/usr/bin/systemctl" in systemctl
|
||||
assert "/bin/journalctl" in journalctl or "/usr/bin/journalctl" in journalctl
|
||||
31
scripts/tests/test_journalctl_helper.py
Normal file
31
scripts/tests/test_journalctl_helper.py
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
import subprocess
|
||||
|
||||
from conftest import LIBEXEC, env_with_fake_commands, fake_command
|
||||
|
||||
|
||||
JOURNALCTL_HELPER = LIBEXEC / "left4me-journalctl"
|
||||
|
||||
|
||||
def test_journalctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path):
|
||||
subprocess.run(["sh", "-n", str(JOURNALCTL_HELPER)], check=True)
|
||||
marker = fake_command(tmp_path, "journalctl")
|
||||
|
||||
for args in [
|
||||
["../evil", "--lines", "25", "--no-follow"],
|
||||
["alpha", "--bad", "25", "--no-follow"],
|
||||
["alpha", "--lines", "not-number", "--no-follow"],
|
||||
["alpha", "--lines", "25", "--bad-follow"],
|
||||
["bad/name", "--lines", "25", "--no-follow"],
|
||||
]:
|
||||
result = subprocess.run(
|
||||
["sh", str(JOURNALCTL_HELPER), *args],
|
||||
env=env_with_fake_commands(tmp_path),
|
||||
check=False,
|
||||
)
|
||||
assert result.returncode != 0
|
||||
assert not marker.exists()
|
||||
|
||||
script = JOURNALCTL_HELPER.read_text()
|
||||
assert 'unit="left4me-server@${name}.service"' in script
|
||||
assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat "$follow_arg"' in script
|
||||
assert 'exec "$journalctl" -u "$unit" -n "$lines" -o cat' in script
|
||||
32
scripts/tests/test_overlay.py
Normal file
32
scripts/tests/test_overlay.py
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
from conftest import LIBEXEC
|
||||
|
||||
|
||||
OVERLAY_HELPER = LIBEXEC / "left4me-overlay"
|
||||
|
||||
|
||||
def test_overlay_helper_is_python_with_strict_validation():
|
||||
text = OVERLAY_HELPER.read_text()
|
||||
assert text.startswith("#!/usr/bin/python3")
|
||||
# Validation surface
|
||||
assert "NAME_RE = re.compile" in text
|
||||
assert "LOWERDIR_ALLOWLIST" in text
|
||||
assert "user.fuseoverlayfs." in text
|
||||
assert "MAX_LOWERDIRS = 500" in text
|
||||
# Mounts via PID 1's mount namespace
|
||||
assert "/proc/1/ns/mnt" in text
|
||||
assert "nsenter" in text
|
||||
# Verbs are mount and umount (not unmount)
|
||||
assert '"mount"' in text and '"umount"' in text
|
||||
assert '"unmount"' not in text
|
||||
|
||||
|
||||
def test_overlay_helper_mount_is_idempotent_when_already_mounted():
|
||||
"""ExecStartPre runs on every Restart=on-failure cycle. If a previous
|
||||
start mounted successfully but ExecStart failed afterwards, the next
|
||||
ExecStartPre would re-mount on top -- which fails. The helper must
|
||||
short-circuit when merged is already a mount point.
|
||||
"""
|
||||
text = OVERLAY_HELPER.read_text()
|
||||
# Two ismount checks now: one in cmd_mount (skip if mounted),
|
||||
# one in cmd_umount (skip if not mounted).
|
||||
assert text.count("os.path.ismount") >= 2
|
||||
171
scripts/tests/test_script_sandbox.py
Normal file
171
scripts/tests/test_script_sandbox.py
Normal file
|
|
@ -0,0 +1,171 @@
|
|||
import subprocess
|
||||
|
||||
from conftest import LIBEXEC
|
||||
|
||||
|
||||
SCRIPT_SANDBOX_HELPER = LIBEXEC / "left4me-script-sandbox"
|
||||
|
||||
|
||||
def test_script_sandbox_helper_present():
|
||||
assert SCRIPT_SANDBOX_HELPER.is_file()
|
||||
assert SCRIPT_SANDBOX_HELPER.read_text().startswith("#!/bin/bash")
|
||||
mode = SCRIPT_SANDBOX_HELPER.stat().st_mode & 0o777
|
||||
assert mode == 0o755, f"expected 0755, got {oct(mode)}"
|
||||
|
||||
|
||||
def test_script_sandbox_helper_passes_shell_syntax_check():
|
||||
subprocess.run(["bash", "-n", str(SCRIPT_SANDBOX_HELPER)], check=True)
|
||||
|
||||
|
||||
def test_script_sandbox_helper_invokes_systemd_run_with_hardening():
|
||||
text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||
|
||||
# systemd-run service mode (no --scope), with synchronous I/O to caller.
|
||||
assert "systemd-run" in text
|
||||
assert "--scope" not in text, "v2 uses transient service units, not scopes"
|
||||
assert "--pipe" in text
|
||||
assert "--wait" in text
|
||||
assert "--collect" in text
|
||||
assert "--unit=" in text
|
||||
|
||||
# No bwrap.
|
||||
assert "bwrap" not in text
|
||||
assert "bubblewrap" not in text
|
||||
|
||||
# UID drop via systemd directives.
|
||||
assert "User=l4d2-sandbox" in text
|
||||
assert "Group=l4d2-sandbox" in text
|
||||
|
||||
# Cgroup limits unchanged from v1.
|
||||
assert "MemoryMax=4G" in text
|
||||
assert "MemorySwapMax=0" in text
|
||||
assert "TasksMax=512" in text
|
||||
assert "CPUQuota=200%" in text
|
||||
assert "RuntimeMaxSec=3600" in text
|
||||
|
||||
# Hardening directives that v1 (scope mode) couldn't carry.
|
||||
assert "NoNewPrivileges=yes" in text
|
||||
assert "ProtectSystem=strict" in text
|
||||
assert "ProtectHome=yes" in text
|
||||
assert "PrivateTmp=yes" in text
|
||||
assert "PrivateDevices=yes" in text
|
||||
assert "PrivateIPC=yes" in text
|
||||
assert "ProtectKernelTunables=yes" in text
|
||||
assert "ProtectKernelModules=yes" in text
|
||||
assert "ProtectKernelLogs=yes" in text
|
||||
assert "ProtectControlGroups=yes" in text
|
||||
assert "RestrictNamespaces=yes" in text
|
||||
assert "RestrictSUIDSGID=yes" in text
|
||||
assert "LockPersonality=yes" in text
|
||||
assert "MemoryDenyWriteExecute=yes" in text
|
||||
assert "SystemCallFilter=" in text
|
||||
assert "@system-service" in text
|
||||
assert "@network-io" in text
|
||||
assert "CapabilityBoundingSet=" in text
|
||||
assert "AmbientCapabilities=" in text
|
||||
assert 'RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX"' in text
|
||||
|
||||
# Network namespace stays shared with host.
|
||||
assert "PrivateNetwork=" not in text
|
||||
|
||||
# Mount setup: /etc and /var/lib masked with tmpfs; selective binds back.
|
||||
assert 'TemporaryFileSystem="/etc /var/lib"' in text
|
||||
assert "BindReadOnlyPaths=" in text
|
||||
# The resolv.conf bind points at the sandbox-only file (not the host's
|
||||
# /etc/resolv.conf, which typically references a private-IP DNS server
|
||||
# that IPAddressDeny= blocks).
|
||||
assert "/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf" in text
|
||||
assert "/etc/ssl" in text
|
||||
assert "/etc/ca-certificates" in text
|
||||
assert "/etc/nsswitch.conf" in text
|
||||
assert "/etc/alternatives" in text
|
||||
assert "${SCRIPT}:/script.sh" in text
|
||||
assert 'BindPaths="${STAGING}:/overlay"' in text
|
||||
|
||||
# IP egress filter: allow public, deny localhost / RFC1918 / link-local /
|
||||
# multicast / CGNAT / ULA. systemd's "more specific rule wins" semantics
|
||||
# mean public IPs hit the allow and listed ranges hit the deny.
|
||||
# IPAddressDeny alone — no IPAddressAllow=any. Empirically, having both
|
||||
# set causes the allow to win on this systemd/kernel combo regardless of
|
||||
# the documented "more specific rule wins" behaviour. With only Deny,
|
||||
# the kernel's default "allow all" applies to non-listed addresses.
|
||||
assert "IPAddressDeny=" in text
|
||||
assert "IPAddressAllow=any" not in text
|
||||
# Explicit CIDRs — systemd-run's -p parser doesn't accept the
|
||||
# `localhost` / `link-local` / `multicast` shorthand keywords that
|
||||
# work in unit files (only the full strings parse).
|
||||
for token in (
|
||||
"127.0.0.0/8",
|
||||
"::1/128",
|
||||
"169.254.0.0/16",
|
||||
"fe80::/10",
|
||||
"224.0.0.0/4",
|
||||
"ff00::/8",
|
||||
"10.0.0.0/8",
|
||||
"172.16.0.0/12",
|
||||
"192.168.0.0/16",
|
||||
"100.64.0.0/10",
|
||||
"fc00::/7",
|
||||
):
|
||||
assert token in text, f"missing {token!r} in IPAddressDeny set"
|
||||
|
||||
|
||||
def test_script_sandbox_uses_idmap_staging():
|
||||
"""The sandbox runs as l4d2-sandbox but writes need to land on disk as
|
||||
left4me, so all overlay content (workshop + script-built) is uniformly
|
||||
left4me-owned. The helper pre-creates an idmapped bind on a staging
|
||||
path and points the sandbox's BindPaths at the staging, not at the raw
|
||||
overlay dir. trap cleans up the staging bind on exit.
|
||||
"""
|
||||
text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||
# Idmap mount setup uses --map-users / --map-groups.
|
||||
assert "--map-users=" in text
|
||||
assert "--map-groups=" in text
|
||||
# Staging path lives under /var/lib/left4me/tmp/sandbox-idmap-<id>.
|
||||
assert "/var/lib/left4me/tmp/sandbox-idmap-" in text
|
||||
# BindPaths into the sandbox points at the staging path, not the
|
||||
# raw overlay dir.
|
||||
assert 'BindPaths="${STAGING}:/overlay"' in text
|
||||
# trap registers cleanup so the staging bind doesn't outlive the helper.
|
||||
assert "trap " in text and "cleanup_staging" in text
|
||||
# The previous chown-to-l4d2-sandbox approach is gone; overlay dirs
|
||||
# stay left4me-owned end-to-end.
|
||||
assert "chown -R l4d2-sandbox" not in text
|
||||
|
||||
|
||||
def test_script_sandbox_in_build_slice_with_oom_adjust():
|
||||
text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||
|
||||
# Put the transient unit in the low-weight build slice so it yields to
|
||||
# game-server instances under CPU/IO contention.
|
||||
assert "--slice=l4d2-build.slice" in text
|
||||
|
||||
# Sandbox dies first if the host hits memory pressure; servers
|
||||
# (OOMScoreAdjust=-200) survive.
|
||||
assert "-p OOMScoreAdjust=500" in text
|
||||
|
||||
|
||||
def test_script_sandbox_helper_validates_overlay_id():
|
||||
text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||
# Numeric-only overlay id
|
||||
assert '[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]]' in text
|
||||
# Overlay dir must exist
|
||||
assert "/var/lib/left4me/overlays/" in text
|
||||
assert "[[ -d $OVERLAY_DIR ]]" in text
|
||||
# Script path must exist
|
||||
assert "[[ -f $SCRIPT ]]" in text
|
||||
|
||||
|
||||
def test_script_sandbox_helper_dry_run_mode(tmp_path):
|
||||
overlay_root = tmp_path / "var/lib/left4me/overlays/42"
|
||||
overlay_root.mkdir(parents=True)
|
||||
fake_script = tmp_path / "fake.sh"
|
||||
fake_script.write_text("echo hi")
|
||||
|
||||
# Run in DRY_RUN mode against a fake l4d2-sandbox UID via a tiny shim that
|
||||
# simulates `id -u l4d2-sandbox` resolving to a valid number.
|
||||
helper_text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||
# We can't actually exec this without root + a real sandbox user; just
|
||||
# verify the dry-run guard short-circuits before systemd-run / bwrap.
|
||||
assert 'LEFT4ME_SCRIPT_SANDBOX_DRY_RUN' in helper_text
|
||||
assert 'exit 0' in helper_text
|
||||
38
scripts/tests/test_sudoers_grants.py
Normal file
38
scripts/tests/test_sudoers_grants.py
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
"""Audit the script→sudoers contract.
|
||||
|
||||
The sudoers file in `deploy/files/etc/sudoers.d/left4me` is a reference
|
||||
example; ckn-bw ships its own verbatim copy under
|
||||
`bundles/left4me/files/etc/sudoers.d/left4me`. The two are expected to
|
||||
match. This test lives under `scripts/tests/` because the contract being
|
||||
audited is about *scripts* (which paths the `left4me` uid can sudo into)
|
||||
even though the file it reads is in `deploy/`.
|
||||
"""
|
||||
from conftest import DEPLOY
|
||||
|
||||
|
||||
SUDOERS = DEPLOY / "files/etc/sudoers.d/left4me"
|
||||
|
||||
|
||||
def test_sudoers_allows_only_left4me_helpers_not_raw_system_tools():
|
||||
sudoers = SUDOERS.read_text()
|
||||
|
||||
assert (
|
||||
"left4me ALL=(root) NOPASSWD: "
|
||||
"/usr/local/libexec/left4me/left4me-systemctl *"
|
||||
) in sudoers
|
||||
assert (
|
||||
"left4me ALL=(root) NOPASSWD: "
|
||||
"/usr/local/libexec/left4me/left4me-journalctl *"
|
||||
) in sudoers
|
||||
assert "/usr/local/libexec/left4me/left4me-overlay mount *" in sudoers
|
||||
assert "/usr/local/libexec/left4me/left4me-overlay umount *" in sudoers
|
||||
assert (
|
||||
"left4me ALL=(root) NOPASSWD: "
|
||||
"/usr/local/libexec/left4me/left4me-script-sandbox"
|
||||
) in sudoers
|
||||
assert "/bin/systemctl" not in sudoers
|
||||
assert "/usr/bin/systemctl" not in sudoers
|
||||
assert "/bin/journalctl" not in sudoers
|
||||
assert "/usr/bin/journalctl" not in sudoers
|
||||
assert "/bin/mount" not in sudoers
|
||||
assert "/bin/umount" not in sudoers
|
||||
39
scripts/tests/test_systemctl_helper.py
Normal file
39
scripts/tests/test_systemctl_helper.py
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
import subprocess
|
||||
|
||||
from conftest import LIBEXEC, env_with_fake_commands, fake_command
|
||||
|
||||
|
||||
SYSTEMCTL_HELPER = LIBEXEC / "left4me-systemctl"
|
||||
|
||||
|
||||
def test_systemctl_helper_passes_shell_syntax_check_and_rejects_bad_args(tmp_path):
|
||||
subprocess.run(["sh", "-n", str(SYSTEMCTL_HELPER)], check=True)
|
||||
marker = fake_command(tmp_path, "systemctl")
|
||||
|
||||
for args in [
|
||||
["bad/action", "alpha"],
|
||||
# `start` and `stop` are no longer accepted verbs — the lifecycle now
|
||||
# uses `enable`/`disable` for reboot survival via WantedBy= symlinks.
|
||||
["start", "alpha"],
|
||||
["stop", "alpha"],
|
||||
["enable", ""],
|
||||
["enable", ".hidden"],
|
||||
["enable", "bad..name"],
|
||||
["enable", "bad/name"],
|
||||
["enable", "bad\\name"],
|
||||
["enable", "bad name"],
|
||||
]:
|
||||
result = subprocess.run(
|
||||
["sh", str(SYSTEMCTL_HELPER), *args],
|
||||
env=env_with_fake_commands(tmp_path),
|
||||
check=False,
|
||||
)
|
||||
assert result.returncode != 0
|
||||
assert not marker.exists()
|
||||
|
||||
script = SYSTEMCTL_HELPER.read_text()
|
||||
assert 'unit="left4me-server@${name}.service"' in script
|
||||
assert 'enable) exec "$systemctl" enable --now "$unit"' in script
|
||||
assert 'disable) exec "$systemctl" disable --now "$unit"' in script
|
||||
assert "--property=ActiveState" in script
|
||||
assert "--property=SubState" in script
|
||||
Loading…
Reference in a new issue