left4me/deploy/README.md

# left4me deploy — reference exemplar

> **This directory is reference material, not the source of truth.**
> The canonical deploy of `ovh.left4me` is driven by
> [ckn-bw](https://git.sublimity.de/cronekorkn/ckn-bw)'s `bundles/left4me/`
> (attached via `groups/applications/left4me.py`); run `bw apply ovh.left4me`
> from the ckn-bw repo to deploy.
>
> The privileged scripts the application installs live at the repo root
> under [`scripts/libexec/`](../scripts/libexec/) and
> [`scripts/sbin/`](../scripts/sbin/) — application code, not deploy
> artifacts. ckn-bw's `install_left4me_scripts` action reads them from
> `/opt/left4me/src/scripts/{libexec,sbin}/` after `git_deploy` and
> installs them into the standard FHS targets on the host.
>
> What remains under `deploy/files/` and `deploy/templates/` is a set of
> readable **examples** — sudoers, sysctl, sandbox-resolv.conf, env
> templates, and a curated subset of the systemd units ckn-bw's reactor
> emits at apply time. They exist so a fresh consumer (other than ckn-bw)
> could read this tree and assemble an equivalent deployment. They are
> **not** the bytes ckn-bw installs; ckn-bw carries its own copies of the
> verbatim configs in `bundles/left4me/files/etc/`, and emits the live
> units from `bundles/left4me/metadata.py`'s `systemd_units` reactor.

## What's here

| Path | Role |
|---|---|
| `files/etc/sudoers.d/left4me` | Example sudoers grants. Lockdown test: `scripts/tests/test_sudoers_grants.py`. |
| `files/etc/sysctl.d/99-left4me.conf` | Example sysctl perf baseline (UDP buffers, fq_codel + BBR). |
| `files/etc/left4me/sandbox-resolv.conf` | Example `/etc/resolv.conf` bound into the script-overlay sandbox. |
| `files/usr/local/lib/systemd/system/left4me-web.service` | Example of the web-app unit the reactor emits. |
| `files/usr/local/lib/systemd/system/left4me-server@.service` | Example of the per-instance gameserver unit. |
| `files/usr/local/lib/systemd/system/left4me-workshop-refresh.{service,timer}` | Example of the daily workshop-refresh cron-equivalent. |
| `files/usr/local/lib/systemd/system/l4d2-{game,build}.slice` | Example slice definitions (CPU/IO weights). |
| `templates/etc/left4me/host.env` | Example host-library env (deployment-fixed paths). |
| `templates/etc/left4me/web.env.template` | Example web-app env. ckn-bw renders the real version via the matching Mako template in `bundles/left4me/files/etc/left4me/web.env.mako`. |
| `tests/test_example_units.py` | Locks down the example units & env templates above. |

The privileged scripts (`left4me-overlay`, `left4me-script-sandbox`,
`left4me-systemctl`, `left4me-journalctl`, `sbin/left4me`) used to live
under this tree at `files/usr/local/{libexec,sbin}/`; they moved to
`scripts/{libexec,sbin}/` because they are application code, not deploy
artifacts.

## Target layout

The deployment uses these on-host paths (FHS-aligned):

- `/etc/left4me/host.env` — host library environment configuration.
- `/etc/left4me/web.env` — web app environment configuration.
- `/etc/left4me/sandbox-resolv.conf` — DNS resolv.conf bound into the
  script-overlay sandbox.
- `/etc/sudoers.d/left4me` — sudoers rules letting the `left4me` uid call
  the privileged helpers non-interactively.
- `/etc/sysctl.d/99-left4me.conf` — perf-baseline sysctls.
- `/opt/left4me` — deployed repository contents (via ckn-bw `git_deploy`).
- `/opt/left4me/.venv` — Python virtual environment for the web app.
- `/var/lib/left4me/left4me.db` — SQLite database used by the web app.
- `/var/lib/left4me/installation` — shared L4D2 installation.
- `/var/lib/left4me/overlays` — overlay directories. Each overlay lives
  at `${overlay_id}` under here.
- `/var/lib/left4me/workshop_cache` — deduplicated cache of `.vpk` files
  downloaded for workshop overlays. One file per Steam item, named
  `{steam_id}.vpk`. Workshop overlays symlink into this tree.
- `/var/lib/left4me/instances` — rendered instance specifications and
  per-instance state.
- `/var/lib/left4me/runtime` — per-instance runtime mount directories.
- `/var/lib/left4me/tmp` — temporary files used by deployment/runtime
  operations (incl. idmap staging binds).
- `/usr/local/lib/systemd/system/` — global systemd unit files emitted
  by ckn-bw's `systemd_units` reactor.
- `/usr/local/libexec/left4me/` — privileged helper commands installed
  from `scripts/libexec/`.
- `/usr/local/sbin/left4me` — admin CLI wrapper installed from
  `scripts/sbin/left4me`.

## Runtime users

One system user does everything:

- **`left4me`** (home `/var/lib/left4me`, shell `/usr/sbin/nologin`):
  web app, host library, gameserver runtime, and script-overlay
  sandbox. The sandbox unit drops privileges via `systemd-run` and
  runs the user-authored bash inside a fully hardened transient
  service (see `scripts/libexec/left4me-script-sandbox`). Same-uid
  attack surface — sandbox escape reaching `web.env`, the SQLite DB,
  or running gameservers — is closed by that hardening profile plus
  system-wide `kernel.yama.ptrace_scope=2`, rather than by a uid
  boundary.

The user-count decision and its history live in
`docs/superpowers/specs/2026-05-15-user-uid-split-design.md`.

## Deployment

Production deploy:

```sh
# In the ckn-bw repo:
bw apply ovh.left4me
```

Admin bootstrap is a manual one-time step after the first apply
(ckn-bw deliberately doesn't seed an admin to keep credentials out of
the metadata pipeline):

```sh
sudo -u left4me LEFT4ME_ADMIN_USERNAME=admin \
  LEFT4ME_ADMIN_PASSWORD='change-me' \
  /opt/left4me/.venv/bin/flask --app l4d2web.app:create_app \
  create-user "$LEFT4ME_ADMIN_USERNAME" --admin
```

Rotate the bootstrap password after first login.

## Overlay references

Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`.
With the default deployment root, they resolve under
`/var/lib/left4me/overlays`. New overlays use `${overlay_id}` as their
path; the digit-only form is the only one created by the web app.

Invalid references are rejected:

- Absolute paths such as `/srv/overlay`.
- Parent traversal such as `../other` or `competitive/../../base`.
- Empty path components such as `competitive//base`.
- Symlink escapes that resolve outside `${LEFT4ME_ROOT}/overlays`.

The web app currently supports two overlay surfaces:

- **`workshop` overlays** (user-owned) — populated by downloading
  `.vpk` files from the public Steam Web API into
  `${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creating absolute
  symlinks under
  `${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`.
- **`script` overlays** — populated by an arbitrary user-authored bash
  script that runs inside `systemd-run` as `left4me` (under a fully
  hardened transient service unit), with the overlay directory
  bind-mounted RW at `/overlay`. Resource caps: 1h walltime, 4 GB RAM,
  512 tasks, 200% CPU, 20 GB post-build disk cap.

Both caches and overlay directories are owned by `left4me`. If the web
service ever runs as a different uid, ensure it shares a group with the
host process and that both trees are group-readable.

## Performance tuning

The deployment ships a host-side perf baseline (slices, unit directives,
sysctls). See
`docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md`
for design rationale.

The knobs below are documented escape hatches — **not** auto-applied.
Apply only after measuring a need and understanding the failure modes.

### Network shaping

Three pieces of the baseline affect player-experience network behaviour:

1. **Per-flow marking.** ckn-bw's central `bundles/nftables/` consumes
   left4me's nftables defaults and marks every UDP packet from uid
   `left4me` with DSCP EF and `skb->priority` 6. srcds doesn't set
   these itself, so without this rule its UDP is indistinguishable
   from any other flow.
2. **Sysctl baseline.** `99-left4me.conf` sets `udp_rmem_min=16384`,
   `udp_wmem_min=16384`, `default_qdisc=fq_codel`, and
   `tcp_congestion_control=bbr`. Reduces head-of-line blocking when
   bulk TCP egress coexists with game UDP.
3. **CAKE egress shaping.** Configured per-interface via systemd-networkd
   metadata (`network/<iface>/cake` in ckn-bw's `bundles/network/`),
   which reapplies the CAKE qdisc across iface lifecycle events. Set
   the declared bandwidth to ≈95% of measured uplink — CAKE only shapes
   if its declared bandwidth is *below* the real bottleneck. Idle links
   with no competing egress see no visible CAKE effect; the win
   materialises under bulk traffic that would otherwise bufferbloat the
   link the players share.

### CPU governor

The performance governor squeezes a few percent off jitter under bursty
load. `schedutil` is acceptable for sustained UDP workloads.

```sh
sudo cpupower frequency-set -g performance
```

Install via `sudo apt install linux-cpupower` if the binary isn't
present. Persist via your distro's CPU-frequency tooling (e.g.
`/etc/default/cpufrequtils`).

### CPU isolation (cores)

The deploy writes four `AllowedCPUs=` drop-ins so that by default only
`l4d2-game.slice` is allowed to run on cores `1..N-1`; `system.slice`,
`user.slice`, and `l4d2-build.slice` are pinned to core 0. Game servers
get the host minus core 0 exclusively; the build sandbox and the web
app stay on core 0; a logged-in admin running CPU-heavy work in their
shell can't steal cycles from a live match. Single-core hosts skip the
cpuset drop-ins entirely; the rest of the perf baseline (cgroup
weights, sysctls, OOM scores) still applies.

Per-instance `CPUAffinity=` (next subsection) composes on top of this —
the per-instance value must be a subset of `l4d2-game.slice`'s
`AllowedCPUs=`, which the kernel enforces.

### Per-instance CPU affinity

`srcds` is single-threaded per instance. On a multi-core host, pinning
each instance to its own core can cut jitter under contention. Drop in
`/etc/systemd/system/left4me-server@<name>.service.d/affinity.conf`:

```ini
[Service]
CPUAffinity=2
```

This pins the instance to CPU 2. A reasonable strategy on an N-core
host: leave core 0 for the kernel + IRQs + system services, then pin
one instance per remaining core.

### NIC tuning

Hardware-specific (install via `sudo apt install ethtool` if not
present). On a host with a single primary interface (replace `eth0`):

```sh
sudo ethtool -G eth0 rx 4096 tx 4096
sudo ethtool -K eth0 gro on lro off
```

If you run a high instance count, also pin the NIC's interrupts off
the cores that game servers occupy (see `/proc/interrupts` and
`/proc/irq/<n>/smp_affinity`).

### Real-time scheduling (advanced, opt-in)

Source-engine servers do not need real-time scheduling, and a
misbehaving `srcds` at any RT priority can starve kernel threads — even
with the default `kernel.sched_rt_runtime_us=950000` throttling 5% of
CPU back. Use only if you have a measured jitter problem that the
baseline does not solve.

`/etc/systemd/system/left4me-server@.service.d/realtime.conf`:

```ini
[Service]
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=10
LimitRTPRIO=10
AmbientCapabilities=CAP_SYS_NICE
```

The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the
service runs as `User=left4me` with `NoNewPrivileges=true`; without it
some kernels/systemd combinations refuse to apply the RT policy.

### Additional opt-in network knobs

- **Ingress shaping via IFB.** Egress CAKE alone does not protect srcds
  receive against ingress saturation (large workshop downloads,
  package fetches arriving at line rate). Worth flipping only when
  measurement shows ingress hurting receive.

      sudo modprobe ifb && sudo ip link set ifb0 up
      sudo tc qdisc add dev <uplink> handle ffff: ingress
      sudo tc filter add dev <uplink> parent ffff: protocol ip u32 \
          match u32 0 0 action mirred egress redirect dev ifb0
      sudo tc qdisc add dev ifb0 root cake bandwidth Xmbit ingress \
          diffserv4 dual-srchost

- **`net.core.busy_poll = 50` / `net.core.busy_read = 50`.** Reduces
  UDP receive median latency by polling for incoming packets briefly
  at syscall boundaries. Cost: measurable CPU per syscall under load.
  Worth flipping if a host is dedicated to game serving and CPU
  headroom is plentiful.

- **`ethtool -K <iface> gro off`.** Some Source-engine ops disable
  generic receive offload to avoid receive-side coalescing latency.
  Hardware/driver dependent; document only.

### Applying changes to running servers

Unit-file changes do not apply to already-running services. After any
change:

```sh
sudo systemctl daemon-reload
# Restart each game server via the web UI's stop + start, or:
sudo systemctl restart 'left4me-server@*.service'
```