left4me/deploy/README.md

# left4me deploy — reference exemplar

> The canonical deploy of `ovh.left4me` is driven by
> [ckn-bw](https://git.sublimity.de/cronekorkn/ckn-bw)'s `bundles/left4me/`
> (attached via `groups/applications/left4me.py`); run `bw apply ovh.left4me`
> from the ckn-bw repo to deploy.
>
> **`deploy/files/` is the canonical source of truth** for static deployment
> artifacts — sudoers, sysctl drop-in, and hardening drop-ins for the
> systemd service units. ckn-bw delivers these via **target-side symlinks**
> from their on-host paths into `/opt/left4me/src/deploy/files/...` (safe
> because `/opt/left4me/src` is root-owned at runtime; the application cannot
> rewrite its own deployment artifacts).
>
> **`deploy/scripts/` is the canonical source of truth** for privileged
> helpers. ckn-bw creates target-side symlinks from
> `/usr/local/{libexec/left4me,sbin}/` into
> `/opt/left4me/src/deploy/scripts/{libexec,sbin}/` after `git_deploy`.
>
> What remains under `deploy/files/usr/local/lib/systemd/system/` is a set
> of **reference fixtures** — a curated subset of the systemd units ckn-bw's
> reactor emits at apply time. They exist so a fresh consumer (other than
> ckn-bw) can read this tree and understand the live unit shape, and so that
> `deploy/tests/test_example_units.py` can assert the reference matches the
> live form. The live base units are emitted by ckn-bw's `systemd/units`
> reactor with per-host CPU pinning and worker counts; the reference files
> must not include hardening directives (those live in the drop-ins, not the
> base units).

## What's here

| Path | Role |
|---|---|
| `files/etc/sudoers.d/left4me` | **Canonical** sudoers grants. Symlinked to `/etc/sudoers.d/left4me`. CI syntax test: `tests/test_sudoers.py`. |
| `files/etc/sysctl.d/99-left4me.conf` | **Canonical** sysctl drop-in (UDP buffers, fq_codel + BBR, `kernel.yama.ptrace_scope=2`). Symlinked to `/etc/sysctl.d/99-left4me.conf`. |
| `files/etc/systemd/system/left4me-web.service.d/10-hardening.conf` | **Canonical** hardening drop-in for `left4me-web.service`. Symlinked to the same on-host path. |
| `files/etc/systemd/system/left4me-server@.service.d/10-hardening.conf` | **Canonical** hardening drop-in for `left4me-server@.service`. Symlinked to the same on-host path. |
| `files/etc/left4me/sandbox-resolv.conf` | Example `/etc/resolv.conf` bound into the script-overlay sandbox (delivered as a bw `files{}` item, not a symlink). |
| `files/usr/local/lib/systemd/system/left4me-web.service` | **Reference fixture** — the web-app unit the reactor emits (per-host worker/thread counts omitted). |
| `files/usr/local/lib/systemd/system/left4me-server@.service` | **Reference fixture** — the per-instance gameserver unit template the reactor emits. |
| `files/usr/local/lib/systemd/system/left4me-workshop-refresh.{service,timer}` | **Reference fixture** — the daily workshop-refresh cron-equivalent. |
| `files/usr/local/lib/systemd/system/l4d2-{game,build}.slice` | **Reference fixture** — slice definitions (CPU/IO weights; reactor fills in `AllowedCPUs=` from host metadata). |
| `scripts/libexec/{left4me-overlay,left4me-systemctl,left4me-journalctl,left4me-script-sandbox}` | **Canonical** privileged helper commands. Symlinked under `/usr/local/libexec/left4me/`. |
| `scripts/sbin/left4me` | **Canonical** admin CLI wrapper. Symlinked to `/usr/local/sbin/left4me`. |
| `templates/etc/left4me/host.env` | Example host-library env (deployment-fixed paths). |
| `templates/etc/left4me/web.env.template` | Example web-app env. ckn-bw renders the real version via the matching Mako template in `bundles/left4me/files/etc/left4me/web.env.mako`. |
| `tests/test_example_units.py` | Locks down the reference units and env templates above; also asserts hardening drop-in shape. |
| `tests/test_sudoers.py` | Runs `visudo -cf` against the sudoers file in CI. |

## Target layout

The deployment uses these on-host paths (FHS-aligned):

- `/etc/left4me/host.env` — host library environment configuration.
- `/etc/left4me/web.env` — web app environment configuration.
- `/etc/left4me/sandbox-resolv.conf` — DNS resolv.conf bound into the
  script-overlay sandbox.
- `/etc/sudoers.d/left4me` — sudoers rules letting the `left4me` uid call
  the privileged helpers non-interactively.
- `/etc/sysctl.d/99-left4me.conf` — perf-baseline sysctls.
- `/opt/left4me/src` — deployed repository contents (via ckn-bw
  `git_deploy`). Root-owned; read-only at runtime. `/opt/left4me/`
  itself is also root-owned and contains only `src/`.
- `/var/lib/left4me/.venv` — Python virtual environment for the web app
  (non-editable install of `l4d2host` + `l4d2web`).
- `/var/lib/left4me/steam` — steamcmd install (self-updates).
- `/var/lib/left4me/left4me.db` — SQLite database used by the web app.
- `/var/lib/left4me/installation` — shared L4D2 installation.
- `/var/lib/left4me/overlays` — overlay directories. Each overlay lives
  at `${overlay_id}` under here.
- `/var/lib/left4me/workshop_cache` — deduplicated cache of `.vpk` files
  downloaded for workshop overlays. One file per Steam item, named
  `{steam_id}.vpk`. Workshop overlays symlink into this tree.
- `/var/lib/left4me/instances` — rendered instance specifications and
  per-instance state.
- `/var/lib/left4me/runtime` — per-instance runtime mount directories.
- `/var/lib/left4me/tmp` — temporary files used by deployment/runtime
  operations (incl. idmap staging binds).
- `/usr/local/lib/systemd/system/` — global systemd unit files emitted
  by ckn-bw's `systemd_units` reactor.
- `/usr/local/libexec/left4me/` — privileged helper commands, symlinked
  from `deploy/scripts/libexec/`.
- `/usr/local/sbin/left4me` — admin CLI wrapper, symlinked from
  `deploy/scripts/sbin/left4me`.

## Runtime users

One system user does everything:

- **`left4me`** (home `/var/lib/left4me`, shell `/usr/sbin/nologin`):
  web app, host library, gameserver runtime, and script-overlay
  sandbox. The sandbox unit drops privileges via `systemd-run` and
  runs the user-authored bash inside a fully hardened transient
  service (see `deploy/scripts/libexec/left4me-script-sandbox`). Same-uid
  attack surface — sandbox escape reaching `web.env`, the SQLite DB,
  or running gameservers — is closed by that hardening profile plus
  system-wide `kernel.yama.ptrace_scope=2`, rather than by a uid
  boundary.

The user-count decision and its history live in
`docs/superpowers/specs/2026-05-15-user-uid-split-design.md`.

## Deployment

Production deploy:

```sh
# In the ckn-bw repo:
bw apply ovh.left4me
```

Admin bootstrap is a manual one-time step after the first apply
(ckn-bw deliberately doesn't seed an admin to keep credentials out of
the metadata pipeline):

```sh
sudo -u left4me LEFT4ME_ADMIN_USERNAME=admin \
  LEFT4ME_ADMIN_PASSWORD='change-me' \
  /var/lib/left4me/.venv/bin/flask --app l4d2web.app:create_app \
  create-user "$LEFT4ME_ADMIN_USERNAME" --admin
```

Rotate the bootstrap password after first login.

## Overlay references

Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`.
With the default deployment root, they resolve under
`/var/lib/left4me/overlays`. New overlays use `${overlay_id}` as their
path; the digit-only form is the only one created by the web app.

Invalid references are rejected:

- Absolute paths such as `/srv/overlay`.
- Parent traversal such as `../other` or `competitive/../../base`.
- Empty path components such as `competitive//base`.
- Symlink escapes that resolve outside `${LEFT4ME_ROOT}/overlays`.

The web app currently supports two overlay surfaces:

- **`workshop` overlays** (user-owned) — populated by downloading
  `.vpk` files from the public Steam Web API into
  `${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creating absolute
  symlinks under
  `${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`.
- **`script` overlays** — populated by an arbitrary user-authored bash
  script that runs inside `systemd-run` as `left4me` (under a fully
  hardened transient service unit), with the overlay directory
  bind-mounted RW at `/overlay`. Resource caps: 1h walltime, 4 GB RAM,
  512 tasks, 200% CPU, 20 GB post-build disk cap.

Both caches and overlay directories are owned by `left4me`. If the web
service ever runs as a different uid, ensure it shares a group with the
host process and that both trees are group-readable.

## Performance tuning

The deployment ships a host-side perf baseline (slices, unit directives,
sysctls). See
`docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md`
for design rationale.

The knobs below are documented escape hatches — **not** auto-applied.
Apply only after measuring a need and understanding the failure modes.

### Network shaping

Three pieces of the baseline affect player-experience network behaviour:

1. **Per-flow marking.** ckn-bw's central `bundles/nftables/` consumes
   left4me's nftables defaults and marks every UDP packet from uid
   `left4me` with DSCP EF and `skb->priority` 6. srcds doesn't set
   these itself, so without this rule its UDP is indistinguishable
   from any other flow.
2. **Sysctl baseline.** `99-left4me.conf` sets `udp_rmem_min=16384`,
   `udp_wmem_min=16384`, `default_qdisc=fq_codel`, and
   `tcp_congestion_control=bbr`. Reduces head-of-line blocking when
   bulk TCP egress coexists with game UDP.
3. **CAKE egress shaping.** Configured per-interface via systemd-networkd
   metadata (`network/<iface>/cake` in ckn-bw's `bundles/network/`),
   which reapplies the CAKE qdisc across iface lifecycle events. Set
   the declared bandwidth to ≈95% of measured uplink — CAKE only shapes
   if its declared bandwidth is *below* the real bottleneck. Idle links
   with no competing egress see no visible CAKE effect; the win
   materialises under bulk traffic that would otherwise bufferbloat the
   link the players share.

### CPU governor

The performance governor squeezes a few percent off jitter under bursty
load. `schedutil` is acceptable for sustained UDP workloads.

```sh
sudo cpupower frequency-set -g performance
```

Install via `sudo apt install linux-cpupower` if the binary isn't
present. Persist via your distro's CPU-frequency tooling (e.g.
`/etc/default/cpufrequtils`).

### CPU isolation (cores)

The deploy writes four `AllowedCPUs=` drop-ins so that by default only
`l4d2-game.slice` is allowed to run on cores `1..N-1`; `system.slice`,
`user.slice`, and `l4d2-build.slice` are pinned to core 0. Game servers
get the host minus core 0 exclusively; the build sandbox and the web
app stay on core 0; a logged-in admin running CPU-heavy work in their
shell can't steal cycles from a live match. Single-core hosts skip the
cpuset drop-ins entirely; the rest of the perf baseline (cgroup
weights, sysctls, OOM scores) still applies.

Per-instance `CPUAffinity=` (next subsection) composes on top of this —
the per-instance value must be a subset of `l4d2-game.slice`'s
`AllowedCPUs=`, which the kernel enforces.

### Per-instance CPU affinity

`srcds` is single-threaded per instance. On a multi-core host, pinning
each instance to its own core can cut jitter under contention. Drop in
`/etc/systemd/system/left4me-server@<name>.service.d/affinity.conf`:

```ini
[Service]
CPUAffinity=2
```

This pins the instance to CPU 2. A reasonable strategy on an N-core
host: leave core 0 for the kernel + IRQs + system services, then pin
one instance per remaining core.

### NIC tuning

Hardware-specific (install via `sudo apt install ethtool` if not
present). On a host with a single primary interface (replace `eth0`):

```sh
sudo ethtool -G eth0 rx 4096 tx 4096
sudo ethtool -K eth0 gro on lro off
```

If you run a high instance count, also pin the NIC's interrupts off
the cores that game servers occupy (see `/proc/interrupts` and
`/proc/irq/<n>/smp_affinity`).

### Real-time scheduling (advanced, opt-in)

Source-engine servers do not need real-time scheduling, and a
misbehaving `srcds` at any RT priority can starve kernel threads — even
with the default `kernel.sched_rt_runtime_us=950000` throttling 5% of
CPU back. Use only if you have a measured jitter problem that the
baseline does not solve.

`/etc/systemd/system/left4me-server@.service.d/realtime.conf`:

```ini
[Service]
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=10
LimitRTPRIO=10
AmbientCapabilities=CAP_SYS_NICE
```

The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the
service runs as `User=left4me` with `NoNewPrivileges=true`; without it
some kernels/systemd combinations refuse to apply the RT policy.

### Additional opt-in network knobs

- **Ingress shaping via IFB.** Egress CAKE alone does not protect srcds
  receive against ingress saturation (large workshop downloads,
  package fetches arriving at line rate). Worth flipping only when
  measurement shows ingress hurting receive.

      sudo modprobe ifb && sudo ip link set ifb0 up
      sudo tc qdisc add dev <uplink> handle ffff: ingress
      sudo tc filter add dev <uplink> parent ffff: protocol ip u32 \
          match u32 0 0 action mirred egress redirect dev ifb0
      sudo tc qdisc add dev ifb0 root cake bandwidth Xmbit ingress \
          diffserv4 dual-srchost

- **`net.core.busy_poll = 50` / `net.core.busy_read = 50`.** Reduces
  UDP receive median latency by polling for incoming packets briefly
  at syscall boundaries. Cost: measurable CPU per syscall under load.
  Worth flipping if a host is dedicated to game serving and CPU
  headroom is plentiful.

- **`ethtool -K <iface> gro off`.** Some Source-engine ops disable
  generic receive offload to avoid receive-side coalescing latency.
  Hardware/driver dependent; document only.

### Applying changes to running servers

Unit-file changes do not apply to already-running services. After any
change:

```sh
sudo systemctl daemon-reload
# Restart each game server via the web UI's stop + start, or:
sudo systemctl restart 'left4me-server@*.service'
```