diff --git a/docs/superpowers/plans/2026-05-09-l4d2-server-host-perf-baseline.md b/docs/superpowers/plans/2026-05-09-l4d2-server-host-perf-baseline.md new file mode 100644 index 0000000..85bd43d --- /dev/null +++ b/docs/superpowers/plans/2026-05-09-l4d2-server-host-perf-baseline.md @@ -0,0 +1,686 @@ +# L4D2 Server Host Perf Baseline Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Apply a host-side performance and resource-isolation baseline (systemd directives, slice hierarchy, host sysctls) to every L4D2 server instance, leaving game ConVars to the maintainer. + +**Architecture:** Add resource-control directives to `left4me-server@.service`; introduce two flat top-level slices (`l4d2-game.slice` weight 1000, `l4d2-build.slice` weight 10) so the build sandbox is starved by the kernel under contention; ship `/etc/sysctl.d/99-left4me.conf` for UDP buffer and netdev tuning; place the script-sandbox transient unit into `l4d2-build.slice` with `OOMScoreAdjust=500`. RT scheduling, CPU governor, CPUAffinity, NIC tuning are documentation-only escape hatches. + +**Tech Stack:** systemd unit files (service + slice), `systemd-run` properties, Linux sysctl, bash deploy script, pytest text-assertion tests under `deploy/tests/test_deploy_artifacts.py`. + +**Spec:** `docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md` + +--- + +## File Structure + +Files to create: + +- `deploy/files/usr/local/lib/systemd/system/l4d2-game.slice` — high-weight slice for game-server instances. +- `deploy/files/usr/local/lib/systemd/system/l4d2-build.slice` — low-weight slice for sandboxed script-overlay builds. +- `deploy/files/etc/sysctl.d/99-left4me.conf` — host UDP/netdev/swap sysctls. + +Files to modify: + +- `deploy/files/usr/local/lib/systemd/system/left4me-server@.service` — add resource-control directives (`Slice`, `Nice`, `IOSchedulingClass`, `IOSchedulingPriority`, `OOMScoreAdjust`, `MemoryHigh`, `MemoryMax`, `TasksMax`, `LimitNOFILE`, `KillSignal`, `TimeoutStopSec`, `LogRateLimitIntervalSec`). +- `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` — add `--slice=l4d2-build.slice` and `-p OOMScoreAdjust=500` to the `systemd-run` invocation. +- `deploy/deploy-test-server.sh` — copy the two slice files and the sysctl conf during deploy; run `sysctl --system` so values take effect immediately. +- `deploy/README.md` — append a "Performance tuning" section with the four documented escape hatches. +- `deploy/tests/test_deploy_artifacts.py` — new tests for each artifact above (text assertions following the existing `assert "X" in text` style). + +No application code (Python, Flask, host library) is touched. + +--- + +## Pre-flight + +- [ ] **Step 0a: Verify clean working tree** + +Run: `git status` +Expected: `nothing to commit, working tree clean` + +- [ ] **Step 0b: Verify the existing deploy tests pass** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py -q` +Expected: all green. + +If any test is already red, stop and surface — this plan assumes the baseline is green. + +--- + +## Task 1: Per-Instance Unit Resource-Control Directives + +Add the per-instance baseline to `left4me-server@.service`. This task is self-contained even though `Slice=l4d2-game.slice` references a slice that doesn't exist yet — systemd does not validate the reference until the unit is actually started, and the deploy artifact tests are pure text checks. + +**Files:** +- Modify: `deploy/files/usr/local/lib/systemd/system/left4me-server@.service` +- Test: `deploy/tests/test_deploy_artifacts.py` (new test function) + +- [ ] **Step 1.1: Add the failing test** + +Open `deploy/tests/test_deploy_artifacts.py` and append (after `test_server_unit_contains_required_runtime_contract`): + +```python +def test_server_unit_contains_perf_baseline_directives(): + unit = SERVER_UNIT.read_text() + + # Slice membership. + assert "Slice=l4d2-game.slice" in unit + + # CFS priority bump (no SCHED_FIFO). + assert "Nice=-5" in unit + assert "CPUSchedulingPolicy=" not in unit + + # I/O priority. + assert "IOSchedulingClass=best-effort" in unit + assert "IOSchedulingPriority=4" in unit + + # OOM ordering: game servers survive, sandbox dies first. + assert "OOMScoreAdjust=-200" in unit + + # Memory caps with headroom for map-load spikes. + assert "MemoryHigh=1.5G" in unit + assert "MemoryMax=2G" in unit + + # Bounded fork surface. + assert "TasksMax=256" in unit + + # Plenty of fds for plugin-heavy setups. + assert "LimitNOFILE=65536" in unit + + # srcds clean shutdown via SIGINT, with time to flush. + assert "KillSignal=SIGINT" in unit + assert "TimeoutStopSec=15s" in unit + + # Per-unit override of journald rate limiting (default drops srcds output). + assert "LogRateLimitIntervalSec=0" in unit +``` + +- [ ] **Step 1.2: Run the new test, verify it fails** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_server_unit_contains_perf_baseline_directives -v` +Expected: FAIL — first failing assert is on `Slice=l4d2-game.slice`. + +- [ ] **Step 1.3: Edit the unit file** + +Open `deploy/files/usr/local/lib/systemd/system/left4me-server@.service` and replace its contents with: + +```ini +[Unit] +Description=left4me server instance %i +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User=left4me +Group=left4me +EnvironmentFile=/etc/left4me/host.env +EnvironmentFile=/var/lib/left4me/instances/%i/instance.env +WorkingDirectory=/var/lib/left4me/runtime/%i/merged/left4dead2 +ExecStart=/var/lib/left4me/installation/srcds_run -game left4dead2 +hostport ${L4D2_PORT} $L4D2_ARGS +Restart=on-failure +RestartSec=5 + +# Resource control baseline — see docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md +Slice=l4d2-game.slice +Nice=-5 +IOSchedulingClass=best-effort +IOSchedulingPriority=4 +OOMScoreAdjust=-200 +MemoryHigh=1.5G +MemoryMax=2G +TasksMax=256 +LimitNOFILE=65536 +KillSignal=SIGINT +TimeoutStopSec=15s +LogRateLimitIntervalSec=0 + +# Hardening (unchanged from previous baseline). +NoNewPrivileges=true +PrivateTmp=true +PrivateDevices=true +ProtectHome=true +ProtectSystem=strict +ReadOnlyPaths=/var/lib/left4me/installation /var/lib/left4me/overlays +ReadWritePaths=/var/lib/left4me/runtime/%i +RestrictSUIDSGID=true +LockPersonality=true + +[Install] +WantedBy=multi-user.target +``` + +- [ ] **Step 1.4: Run the new test, verify it passes** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_server_unit_contains_perf_baseline_directives -v` +Expected: PASS. + +- [ ] **Step 1.5: Re-run the existing server-unit test, verify still passes** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_server_unit_contains_required_runtime_contract -v` +Expected: PASS — the existing assertions (`User=left4me`, `Group=left4me`, hardening directives, etc.) still match. + +- [ ] **Step 1.6: Commit** + +```bash +git add deploy/files/usr/local/lib/systemd/system/left4me-server@.service deploy/tests/test_deploy_artifacts.py +git commit -m "$(cat <<'EOF' +feat(deploy): perf-baseline directives on left4me-server@.service + +Slice=l4d2-game.slice, Nice=-5, IOSchedulingClass=best-effort, +OOMScoreAdjust=-200, MemoryHigh=1.5G, MemoryMax=2G, TasksMax=256, +LimitNOFILE=65536, KillSignal=SIGINT, TimeoutStopSec=15s, +LogRateLimitIntervalSec=0. Spec: +docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md +EOF +)" +``` + +--- + +## Task 2: Slice Unit Files + +Create the two slice unit files. After this task the perf unit's `Slice=l4d2-game.slice` reference is satisfied. + +**Files:** +- Create: `deploy/files/usr/local/lib/systemd/system/l4d2-game.slice` +- Create: `deploy/files/usr/local/lib/systemd/system/l4d2-build.slice` +- Test: `deploy/tests/test_deploy_artifacts.py` (new constants + new test functions) + +- [ ] **Step 2.1: Add path constants and failing tests** + +Open `deploy/tests/test_deploy_artifacts.py`. After the existing `SERVER_UNIT = ...` line, add: + +```python +GAME_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-game.slice" +BUILD_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-build.slice" +``` + +After the new `test_server_unit_contains_perf_baseline_directives`, append: + +```python +def test_l4d2_game_slice_exists_with_high_weights(): + assert GAME_SLICE.is_file() + text = GAME_SLICE.read_text() + assert "[Slice]" in text + assert "CPUWeight=1000" in text + assert "IOWeight=1000" in text + + +def test_l4d2_build_slice_exists_with_low_weights(): + assert BUILD_SLICE.is_file() + text = BUILD_SLICE.read_text() + assert "[Slice]" in text + assert "CPUWeight=10" in text + assert "IOWeight=10" in text +``` + +- [ ] **Step 2.2: Run the new tests, verify they fail** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_l4d2_game_slice_exists_with_high_weights deploy/tests/test_deploy_artifacts.py::test_l4d2_build_slice_exists_with_low_weights -v` +Expected: FAIL on `assert GAME_SLICE.is_file()` (file does not exist). + +- [ ] **Step 2.3: Create the game slice file** + +Create `deploy/files/usr/local/lib/systemd/system/l4d2-game.slice` with: + +```ini +[Unit] +Description=left4me game-server slice +Before=slices.target + +[Slice] +CPUWeight=1000 +IOWeight=1000 +``` + +- [ ] **Step 2.4: Create the build slice file** + +Create `deploy/files/usr/local/lib/systemd/system/l4d2-build.slice` with: + +```ini +[Unit] +Description=left4me script-sandbox build slice +Before=slices.target + +[Slice] +CPUWeight=10 +IOWeight=10 +``` + +- [ ] **Step 2.5: Run the new tests, verify they pass** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_l4d2_game_slice_exists_with_high_weights deploy/tests/test_deploy_artifacts.py::test_l4d2_build_slice_exists_with_low_weights -v` +Expected: PASS. + +- [ ] **Step 2.6: Commit** + +```bash +git add deploy/files/usr/local/lib/systemd/system/l4d2-game.slice deploy/files/usr/local/lib/systemd/system/l4d2-build.slice deploy/tests/test_deploy_artifacts.py +git commit -m "$(cat <<'EOF' +feat(deploy): l4d2-game.slice + l4d2-build.slice with 100:1 weight ratio + +Flat top-level slices. Game wins under contention; build still gets +the box when uncontended. Referenced by left4me-server@.service and +the script-sandbox systemd-run invocation. +EOF +)" +``` + +--- + +## Task 3: Host Sysctls + +Ship a `/etc/sysctl.d/` drop-in for UDP buffers, netdev backlog, netdev budget, and `vm.swappiness`. + +**Files:** +- Create: `deploy/files/etc/sysctl.d/99-left4me.conf` +- Test: `deploy/tests/test_deploy_artifacts.py` (new constant + new test function) + +- [ ] **Step 3.1: Add path constant and failing test** + +Open `deploy/tests/test_deploy_artifacts.py`. After the slice constants, add: + +```python +SYSCTL_CONF = DEPLOY / "files/etc/sysctl.d/99-left4me.conf" +``` + +Append a new test: + +```python +def test_sysctl_conf_present_with_perf_settings(): + assert SYSCTL_CONF.is_file() + text = SYSCTL_CONF.read_text() + for line in ( + "net.core.rmem_max = 8388608", + "net.core.wmem_max = 8388608", + "net.core.rmem_default = 524288", + "net.core.wmem_default = 524288", + "net.core.netdev_max_backlog = 5000", + "net.core.netdev_budget = 600", + "vm.swappiness = 10", + ): + assert line in text, f"missing {line!r} in 99-left4me.conf" +``` + +- [ ] **Step 3.2: Run the new test, verify it fails** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_sysctl_conf_present_with_perf_settings -v` +Expected: FAIL on `assert SYSCTL_CONF.is_file()`. + +- [ ] **Step 3.3: Create the sysctl conf file** + +Create `deploy/files/etc/sysctl.d/99-left4me.conf` with: + +``` +# Host-side perf baseline for left4me — see +# docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md +# +# UDP socket buffers: distro defaults of ~128 KiB are too small for sustained +# Source-engine UDP across multiple instances. 8 MiB matches the standard +# 1 Gbit recommendation; rmem_default/wmem_default protect sockets that don't +# explicitly enlarge their buffers. +net.core.rmem_max = 8388608 +net.core.wmem_max = 8388608 +net.core.rmem_default = 524288 +net.core.wmem_default = 524288 + +# Kernel softirq UDP path: the per-CPU backlog queue starts dropping packets +# at the default 1000 under multi-instance burst; 5000 absorbs realistic peaks. +# netdev_budget = 600 gives softirq more drain headroom per pass. +net.core.netdev_max_backlog = 5000 +net.core.netdev_budget = 600 + +# Latency-sensitive default: avoid swap unless the box is really under +# pressure. Harmless on swapless hosts. +vm.swappiness = 10 +``` + +- [ ] **Step 3.4: Run the new test, verify it passes** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_sysctl_conf_present_with_perf_settings -v` +Expected: PASS. + +- [ ] **Step 3.5: Commit** + +```bash +git add deploy/files/etc/sysctl.d/99-left4me.conf deploy/tests/test_deploy_artifacts.py +git commit -m "$(cat <<'EOF' +feat(deploy): host sysctls for UDP buffers + netdev backlog/budget + +99-left4me.conf: rmem_max/wmem_max=8M (with 512K defaults), +netdev_max_backlog=5000, netdev_budget=600, vm.swappiness=10. +EOF +)" +``` + +--- + +## Task 4: Sandbox in Build Slice + +Place the script-sandbox transient unit into `l4d2-build.slice` and give it `OOMScoreAdjust=500` so it dies first under memory pressure. + +**Files:** +- Modify: `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` +- Test: `deploy/tests/test_deploy_artifacts.py` (new test function) + +- [ ] **Step 4.1: Add the failing test** + +Open `deploy/tests/test_deploy_artifacts.py`. Append: + +```python +def test_script_sandbox_in_build_slice_with_oom_adjust(): + text = SCRIPT_SANDBOX_HELPER.read_text() + + # Put the transient unit in the low-weight build slice so it yields to + # game-server instances under CPU/IO contention. + assert "--slice=l4d2-build.slice" in text + + # Sandbox dies first if the host hits memory pressure; servers + # (OOMScoreAdjust=-200) survive. + assert "-p OOMScoreAdjust=500" in text +``` + +- [ ] **Step 4.2: Run the new test, verify it fails** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_script_sandbox_in_build_slice_with_oom_adjust -v` +Expected: FAIL — neither string is in the helper yet. + +- [ ] **Step 4.3: Edit the sandbox helper** + +Open `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox`. Locate the `systemd-run` invocation that begins with: + +``` +systemd-run --quiet --collect --wait --pipe \ + --unit="left4me-script-${OVERLAY_ID}-$$" \ +``` + +Insert two new lines immediately after the `--unit=` line, before `-p User=l4d2-sandbox`. The block becomes: + +``` +systemd-run --quiet --collect --wait --pipe \ + --unit="left4me-script-${OVERLAY_ID}-$$" \ + --slice=l4d2-build.slice \ + -p OOMScoreAdjust=500 \ + -p User=l4d2-sandbox -p Group=l4d2-sandbox \ +``` + +Leave every other `-p` line untouched. + +- [ ] **Step 4.4: Verify shell syntax still parses** + +Run: `bash -n deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` +Expected: exit 0, no output. + +- [ ] **Step 4.5: Run the new test and the existing sandbox-helper tests, verify they pass** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_script_sandbox_in_build_slice_with_oom_adjust deploy/tests/test_deploy_artifacts.py::test_script_sandbox_helper_invokes_systemd_run_with_hardening deploy/tests/test_deploy_artifacts.py::test_script_sandbox_helper_passes_shell_syntax_check -v` +Expected: PASS for all three. The hardening test still matches because it only checks for substring presence; we added strings, didn't remove any. + +- [ ] **Step 4.6: Commit** + +```bash +git add deploy/files/usr/local/libexec/left4me/left4me-script-sandbox deploy/tests/test_deploy_artifacts.py +git commit -m "$(cat <<'EOF' +feat(deploy): script-sandbox runs in l4d2-build.slice + OOMScoreAdjust=500 + +Builds yield CPU/IO to game-server instances under contention via the +slice's weight=10, and are killed first under memory pressure +(servers have OOMScoreAdjust=-200). +EOF +)" +``` + +--- + +## Task 5: Deploy Script Installs Slice + Sysctl Artifacts + +Wire the new artifacts into `deploy-test-server.sh` so a fresh deploy actually puts them on disk and applies the sysctls. + +**Files:** +- Modify: `deploy/deploy-test-server.sh` +- Test: `deploy/tests/test_deploy_artifacts.py` (new test function) + +- [ ] **Step 5.1: Add the failing test** + +Open `deploy/tests/test_deploy_artifacts.py`. Append: + +```python +def test_deploy_script_installs_perf_artifacts(): + script = DEPLOY_SCRIPT.read_text() + + # Slice files copied into the system-wide systemd unit dir. + assert "/usr/local/lib/systemd/system/l4d2-game.slice" in script + assert "/usr/local/lib/systemd/system/l4d2-build.slice" in script + + # Sysctl drop-in installed under /etc/sysctl.d/. + assert "/etc/sysctl.d/99-left4me.conf" in script + + # Values applied immediately, not on next boot. + assert "sysctl --system" in script +``` + +- [ ] **Step 5.2: Run the new test, verify it fails** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_deploy_script_installs_perf_artifacts -v` +Expected: FAIL on the first assertion. + +- [ ] **Step 5.3: Edit the deploy script — copy the slice + sysctl files** + +Open `deploy/deploy-test-server.sh`. Find the block that copies unit files (currently around line 138): + +```sh +$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-web.service /usr/local/lib/systemd/system/left4me-web.service +$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-server@.service /usr/local/lib/systemd/system/left4me-server@.service +``` + +Add two new lines immediately after the `left4me-server@.service` copy line, so the block becomes: + +```sh +$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-web.service /usr/local/lib/systemd/system/left4me-web.service +$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-server@.service /usr/local/lib/systemd/system/left4me-server@.service +$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/l4d2-game.slice /usr/local/lib/systemd/system/l4d2-game.slice +$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/l4d2-build.slice /usr/local/lib/systemd/system/l4d2-build.slice +``` + +- [ ] **Step 5.4: Edit the deploy script — install the sysctl conf and apply it** + +In `deploy/deploy-test-server.sh`, find the block that installs `/etc/left4me/sandbox-resolv.conf` (currently around lines 153–155): + +```sh +$sudo_cmd install -m 0644 -o root -g root \ + /opt/left4me/deploy/files/etc/left4me/sandbox-resolv.conf \ + /etc/left4me/sandbox-resolv.conf +``` + +Immediately after that block, add: + +```sh +# Host perf-baseline sysctls. Apply with `sysctl --system` so values +# take effect this deploy, not on next reboot. +$sudo_cmd install -m 0644 -o root -g root \ + /opt/left4me/deploy/files/etc/sysctl.d/99-left4me.conf \ + /etc/sysctl.d/99-left4me.conf +$sudo_cmd sysctl --system >/dev/null +``` + +- [ ] **Step 5.5: Verify the deploy script's shell syntax still parses** + +Run: `sh -n deploy/deploy-test-server.sh` +Expected: exit 0, no output. + +- [ ] **Step 5.6: Run the new test and the existing deploy-script tests, verify they pass** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_deploy_script_installs_perf_artifacts deploy/tests/test_deploy_artifacts.py::test_deploy_script_has_safe_defaults_and_preserves_state deploy/tests/test_deploy_artifacts.py::test_deploy_script_shell_syntax -v` +Expected: PASS for all three. + +- [ ] **Step 5.7: Commit** + +```bash +git add deploy/deploy-test-server.sh deploy/tests/test_deploy_artifacts.py +git commit -m "$(cat <<'EOF' +feat(deploy): install slice + sysctl artifacts and apply via sysctl --system + +Copies l4d2-game.slice and l4d2-build.slice into +/usr/local/lib/systemd/system/, installs 99-left4me.conf into +/etc/sysctl.d/, and runs sysctl --system so the perf baseline is +live this deploy, not on next reboot. +EOF +)" +``` + +--- + +## Task 6: Performance-Tuning Section in deploy/README.md + +Document the four escape hatches the spec lists as opt-in: CPU governor, per-instance `CPUAffinity`, NIC tuning, and SCHED_FIFO. + +**Files:** +- Modify: `deploy/README.md` + +No test for this task — README content is documentation, not contract. + +- [ ] **Step 6.1: Append the Performance Tuning section** + +Open `deploy/README.md`. Append (after the existing final paragraph) a new section: + +```markdown +## Performance Tuning + +The deployment ships a host-side perf baseline (slices, unit directives, sysctls). See `docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md` for design rationale. + +The following knobs are documented escape hatches — they are **not** auto-applied. Apply only if you have measured a need and understand the failure modes. + +### CPU governor + +The performance governor squeezes a few percent off jitter under bursty load. `schedutil` is acceptable for sustained UDP workloads. + +```sh +sudo cpupower frequency-set -g performance +``` + +Persist via your distro's CPU-frequency tooling (e.g. `/etc/default/cpufrequtils`). + +### Per-instance CPU affinity + +`srcds` is single-threaded per instance. On a multi-core host, pinning each instance to its own core can cut jitter under contention. Drop in `/etc/systemd/system/left4me-server@.service.d/affinity.conf`: + +```ini +[Service] +CPUAffinity=2 +``` + +A reasonable strategy on an N-core host: leave core 0 for the kernel + IRQs + system services, then pin one instance per remaining core. + +### NIC tuning + +Hardware-specific. On a host with a single primary interface (replace `eth0`): + +```sh +sudo ethtool -G eth0 rx 4096 tx 4096 +sudo ethtool -K eth0 gro on lro off +``` + +If you run a high instance count, also pin the NIC's interrupts off the cores that game servers occupy (see `/proc/interrupts` and `/proc/irq//smp_affinity`). + +### Real-time scheduling (advanced, opt-in) + +Source-engine servers do not need real-time scheduling, and a misbehaving `srcds` at any RT priority can starve kernel threads — even with the default `kernel.sched_rt_runtime_us=950000` throttling 5% of CPU back. Use only if you have a measured jitter problem that the baseline does not solve. + +`/etc/systemd/system/left4me-server@.service.d/realtime.conf`: + +```ini +[Service] +CPUSchedulingPolicy=fifo +CPUSchedulingPriority=10 +LimitRTPRIO=10 +``` + +### Applying changes to running servers + +Unit-file changes do not apply to already-running services. After any change: + +```sh +sudo systemctl daemon-reload +# Restart each game server via the web UI's stop + start, or: +sudo systemctl restart 'left4me-server@*.service' +``` +``` + +- [ ] **Step 6.2: Run the full deploy test suite and verify it stays green** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py -q` +Expected: all green. README changes have no test, but should not break any existing tests. + +- [ ] **Step 6.3: Commit** + +```bash +git add deploy/README.md +git commit -m "$(cat <<'EOF' +docs(deploy): performance-tuning escape-hatch section in README + +Documents CPU governor, per-instance CPUAffinity, NIC tuning, and +SCHED_FIFO opt-in patterns. None of these are auto-applied; they're +ops-side knobs for measured problems the perf baseline doesn't solve. +EOF +)" +``` + +--- + +## Final Verification + +- [ ] **Step F.1: Full deploy test suite green** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/ -q` +Expected: all green. + +- [ ] **Step F.2: Host library + web tests still green (regression check)** + +Run: `cd /Users/mwiegand/Projekte/left4me && pytest l4d2host/tests -q && pytest l4d2web/tests -q` +Expected: all green. Nothing in this plan touches host or web Python code, but a clean run rules out accidental import-time damage. + +- [ ] **Step F.3: Working tree clean and commits in order** + +Run: `git status && git log --oneline -8` +Expected: +- `git status`: `nothing to commit, working tree clean`. +- `git log`: six new commits in this order, top-most first: + 1. `docs(deploy): performance-tuning escape-hatch section in README` + 2. `feat(deploy): install slice + sysctl artifacts and apply via sysctl --system` + 3. `feat(deploy): script-sandbox runs in l4d2-build.slice + OOMScoreAdjust=500` + 4. `feat(deploy): host sysctls for UDP buffers + netdev backlog/budget` + 5. `feat(deploy): l4d2-game.slice + l4d2-build.slice with 100:1 weight ratio` + 6. `feat(deploy): perf-baseline directives on left4me-server@.service` + +If any step is missing or out of order, do not amend — diagnose, fix, and create new commits. + +- [ ] **Step F.4: Manual deploy smoke test (deferred, ops-side)** + +This plan ships artifacts. Confirming that systemd actually accepts and applies them on a real host requires running the deploy script against a test target. That validation is operator-side, not part of this implementation: + +```sh +deploy/deploy-test-server.sh deploy-user@example-host +ssh deploy-user@example-host 'systemctl cat l4d2-game.slice' +ssh deploy-user@example-host 'sysctl net.core.rmem_max' # expect 8388608 +ssh deploy-user@example-host 'systemd-analyze verify /usr/local/lib/systemd/system/left4me-server@.service' +``` + +Document any deploy-time problems back into the spec or this plan as v1.x corrections. Do not invent fixes that go beyond the spec. + +--- + +## Out of Scope (do NOT implement here) + +Listed in the spec — repeated for clarity: + +- ConVars / blueprint arguments / tickrate / sv_minrate. +- SCHED_FIFO auto-apply. +- CPU governor auto-apply. +- Per-instance `CPUAffinity` auto-apply. +- NIC ring-buffer / IRQ-pinning code. +- Job-scheduler awareness ("don't build while server X has players"). +- Hardening tightening (`ProtectKernelTunables=yes`, etc.). + +If you find yourself touching any of these, stop — they belong in a separate spec.