left4me/docs/superpowers/specs/2026-05-09-l4d2-cpu-isolation-design.md

# l4d2 cpu isolation — design

Date: 2026-05-09
Status: design

## Summary

Constrain every cgroup that isn't a live game server to core 0; give game servers cores 1..N-1 exclusively. Implementation is systemd cgroup-v2 `AllowedCPUs=` drop-ins, computed at deploy time from `nproc`, overridable via env vars. Lands on top of the perf baseline shipped in `851e662..e5126c8`.

## Goals

- A logged-in admin doing CPU-heavy work, the script-build sandbox, and the Flask web app cannot steal cycles from a live match.
- Layout scales automatically across host sizes (4-core, 8-core, 16-core) without per-host edits.
- Operator can override the default `0` / `1..N-1` split for NUMA boxes or hyperthread quirks.
- Single-core hosts degrade gracefully: skip CPU isolation, keep the rest of the perf baseline.

## Non-goals

- Kernel `isolcpus=` / `nohz_full=` / `rcu_nocbs=` boot parameters. True core isolation (eviction of softirqs, RCU, timer ticks) requires GRUB edits + reboot + per-host tuning. cgroup cpuset is sufficient for L4D2 tickrates; document as a future opt-in if measurement justifies it.
- NIC IRQ pinning. Hardware-specific; already documented as an escape hatch in `deploy/README.md`.
- Per-instance pinning *within* the game-core set. The slice-level cpuset is the floor; the existing per-instance `CPUAffinity=` drop-in escape hatch (already in `deploy/README.md`) composes on top — the kernel enforces "per-instance value must be a subset of slice's allowed set."
- A separate `l4d2-web.slice`. The web app is light; living in `system.slice` on core 0 is fine.
- Web-app or host-library code changes. Pure deploy-side artifact work.

## Background

The perf baseline (commit range `851e662..e5126c8`) introduced two slices (`l4d2-game.slice` weight 1000, `l4d2-build.slice` weight 10), per-instance unit directives (Nice, OOM, memory caps), and host sysctls. None of those constrain *which* CPUs cgroups run on. Under the kernel CFS, every task can move to any core; the build sandbox, ssh sessions, the web app, and game servers all compete for the same cores.

## Design

### Topology

```
                core 0           cores 1..N-1
                ─────────        ────────────
system.slice    AllowedCPUs=0
user.slice      AllowedCPUs=0
l4d2-build.slice AllowedCPUs=0
l4d2-game.slice                 AllowedCPUs=1-(N-1)
```

Everything that isn't a live game server (Flask web app, ssh sessions, journald, script-sandbox builds, cron, systemd housekeeping) is funneled to core 0. Game servers get cores 1..N-1 exclusively.

### Why slice-level `AllowedCPUs=`, not per-instance `CPUAffinity=`

- **Hierarchy does the work for free.** A cpuset on `l4d2-game.slice` propagates to every `left4me-server@*.service` automatically. No per-instance drop-ins to manage; no logic in the web app to pick cores.
- **Hot-applied.** cgroup-v2 cpuset changes apply to running cgroups; existing servers move next time the kernel schedules them. No need to restart instances after a deploy.
- **Composable.** A future operator who wants per-instance pinning *within* the game cores adds `CPUAffinity=N` via `/etc/systemd/system/left4me-server@<name>.service.d/affinity.conf` (already documented). The slice constraint and per-instance pin compose; the kernel enforces subset-of.

### Why drop-ins, not edits to the existing `.slice` files

The two slice files we ship today (`l4d2-game.slice`, `l4d2-build.slice`) are static text and host-portable. `AllowedCPUs=1-7` is true on an 8-core host and wrong on a 4-core host. Drop-ins under `<unit>.d/*.conf` are the standard systemd pattern for host-specific overrides. We already use `99-` prefixing for the sysctl drop-in so it lex-orders last; reuse that.

### Operator override

Two env vars consumed by the deploy script:

- `LEFT4ME_SYSTEM_CPUS` — defaults to `0`. Goes into `system.slice`, `user.slice`, `l4d2-build.slice` drop-ins.
- `LEFT4ME_GAME_CPUS` — defaults to `1-$((NPROC-1))`. Goes into `l4d2-game.slice` drop-in.

Operators with NUMA boxes, hyperthread quirks, or "I want core 0 *and* core 1 for system" set the vars explicitly. Defaults handle the typical case.

### Single-core fallback

If `nproc < 2`, skip CPU isolation entirely (write no drop-ins). Print a warning to stderr explaining the deploy is leaving cpuset unset. The rest of the perf baseline still applies (weights, sysctls, OOM scores).

If `LEFT4ME_GAME_CPUS` or `LEFT4ME_SYSTEM_CPUS` is set explicitly on a single-core host, honor the operator's intent — they presumably know what they're doing — but still write the drop-ins.

### Drop-in layout

Four files written to `/etc/systemd/system/`, each named `99-left4me-cpuset.conf`:

```
/etc/systemd/system/system.slice.d/99-left4me-cpuset.conf
/etc/systemd/system/user.slice.d/99-left4me-cpuset.conf
/etc/systemd/system/l4d2-build.slice.d/99-left4me-cpuset.conf
/etc/systemd/system/l4d2-game.slice.d/99-left4me-cpuset.conf
```

Each file contains:

```ini
[Slice]
AllowedCPUs=<resolved value>
```

### systemd compatibility

`AllowedCPUs=` is systemd 244+. Debian Trixie ships systemd 256+. Cgroup-v2 cpuset controller is enabled by default on Trixie; systemd auto-enables the controller when `AllowedCPUs=` is set on a unit. No additional machinery.

### Files changed / added

```
deploy/deploy-test-server.sh                   (modified — compute layout, write four drop-ins)
deploy/README.md                               (modified — new "CPU isolation" subsection inside Performance Tuning)
deploy/tests/test_deploy_artifacts.py          (modified — new tests)
```

## Tests

`deploy/tests/test_deploy_artifacts.py` additions, following the existing
`assert "X" in script` pattern:

- For `deploy-test-server.sh`, assert:
  - All four drop-in paths (`/etc/systemd/system/{system,user,l4d2-build,l4d2-game}.slice.d/99-left4me-cpuset.conf`) appear.
  - The script reads `nproc` (substring `nproc` plus a default-binding form for `LEFT4ME_GAME_CPUS`).
  - The script honors `LEFT4ME_SYSTEM_CPUS` and `LEFT4ME_GAME_CPUS` env-var overrides (substrings present, default-binding form like `${LEFT4ME_SYSTEM_CPUS:-...}`).
  - The script has a single-core fallback (substring guarding `nproc -lt 2` or equivalent, with a warning to stderr).
  - Each drop-in is written via the existing `install -m 0644 -o root -g root` heredoc pattern.

No runtime tests in this spec — verifying that systemd actually enforces `AllowedCPUs=` is operator-side via `cat /sys/fs/cgroup/<slice>/cpuset.cpus.effective` after deploy.

## Rollout

Single deploy. cgroup-v2 cpuset changes apply to running cgroups, so already-running servers move next time the kernel reschedules them — no instance restarts required. The `daemon-reload` already in the deploy script picks up the new drop-ins.

If something goes wrong (cpuset too narrow, a slice can't run any process), `systemctl status <slice>` will show the error and the operator can either fix the env vars and redeploy or `rm /etc/systemd/system/<slice>.slice.d/99-left4me-cpuset.conf` followed by `systemctl daemon-reload` to revert.

## Open questions

None blocking. Possible v2 candidates if measurement justifies them:

- Pair this with kernel `isolcpus=` boot params for true core isolation.
- Auto-pin NIC IRQs to core 0 (would compose with this isolation).
- Per-instance `CPUAffinity=` driven by a deploy-env knob, partitioning the game-core set across instances deterministically.

## References

- systemd.resource-control(5) — `AllowedCPUs=` semantics.
- Linux Documentation/admin-guide/cgroup-v2.rst — cpuset controller behavior on `cpuset.cpus` / `cpuset.cpus.effective`.
- Existing perf-baseline spec: `docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md` — sibling work that introduced the slices this spec extends.