cronekorkn/left4me
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(plans): l4d2 server host perf baseline — implementation plan

Browse source 
Six tasks (TDD, one commit each): unit directives, slice files,
sysctl conf, sandbox slice + OOMScoreAdjust, deploy-script wiring,
README escape-hatch section. Final verification step with full
deploy + host + web pytest sweep.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
mwiegand 2026-05-09 09:39:12 +02:00
parent b6574e308b
commit 851e6629aa
No known key found for this signature in database
1 changed files with 686 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

686
docs/superpowers/plans/2026-05-09-l4d2-server-host-perf-baseline.md Normal file
View file

@ -0,0 +1,686 @@
# L4D2 Server Host Perf Baseline Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Apply a host-side performance and resource-isolation baseline (systemd directives, slice hierarchy, host sysctls) to every L4D2 server instance, leaving game ConVars to the maintainer.
**Architecture:** Add resource-control directives to `left4me-server@.service`; introduce two flat top-level slices (`l4d2-game.slice` weight 1000, `l4d2-build.slice` weight 10) so the build sandbox is starved by the kernel under contention; ship `/etc/sysctl.d/99-left4me.conf` for UDP buffer and netdev tuning; place the script-sandbox transient unit into `l4d2-build.slice` with `OOMScoreAdjust=500`. RT scheduling, CPU governor, CPUAffinity, NIC tuning are documentation-only escape hatches.
**Tech Stack:** systemd unit files (service + slice), `systemd-run` properties, Linux sysctl, bash deploy script, pytest text-assertion tests under `deploy/tests/test_deploy_artifacts.py`.
**Spec:** `docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md`
---
## File Structure
Files to create:
- `deploy/files/usr/local/lib/systemd/system/l4d2-game.slice` — high-weight slice for game-server instances.
- `deploy/files/usr/local/lib/systemd/system/l4d2-build.slice` — low-weight slice for sandboxed script-overlay builds.
- `deploy/files/etc/sysctl.d/99-left4me.conf` — host UDP/netdev/swap sysctls.
Files to modify:
- `deploy/files/usr/local/lib/systemd/system/left4me-server@.service` — add resource-control directives (`Slice`, `Nice`, `IOSchedulingClass`, `IOSchedulingPriority`, `OOMScoreAdjust`, `MemoryHigh`, `MemoryMax`, `TasksMax`, `LimitNOFILE`, `KillSignal`, `TimeoutStopSec`, `LogRateLimitIntervalSec`).
- `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` — add `--slice=l4d2-build.slice` and `-p OOMScoreAdjust=500` to the `systemd-run` invocation.
- `deploy/deploy-test-server.sh` — copy the two slice files and the sysctl conf during deploy; run `sysctl --system` so values take effect immediately.
- `deploy/README.md` — append a "Performance tuning" section with the four documented escape hatches.
- `deploy/tests/test_deploy_artifacts.py` — new tests for each artifact above (text assertions following the existing `assert "X" in text` style).
No application code (Python, Flask, host library) is touched.
---
## Pre-flight
- [ ] **Step 0a: Verify clean working tree**
Run: `git status`
Expected: `nothing to commit, working tree clean`
- [ ] **Step 0b: Verify the existing deploy tests pass**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py -q`
Expected: all green.
If any test is already red, stop and surface — this plan assumes the baseline is green.
---
## Task 1: Per-Instance Unit Resource-Control Directives
Add the per-instance baseline to `left4me-server@.service`. This task is self-contained even though `Slice=l4d2-game.slice` references a slice that doesn't exist yet — systemd does not validate the reference until the unit is actually started, and the deploy artifact tests are pure text checks.
**Files:**
- Modify: `deploy/files/usr/local/lib/systemd/system/left4me-server@.service`
- Test: `deploy/tests/test_deploy_artifacts.py` (new test function)
- [ ] **Step 1.1: Add the failing test**
Open `deploy/tests/test_deploy_artifacts.py` and append (after `test_server_unit_contains_required_runtime_contract`):
```python
def test_server_unit_contains_perf_baseline_directives():
unit = SERVER_UNIT.read_text()
# Slice membership.
assert "Slice=l4d2-game.slice" in unit
# CFS priority bump (no SCHED_FIFO).
assert "Nice=-5" in unit
assert "CPUSchedulingPolicy=" not in unit
# I/O priority.
assert "IOSchedulingClass=best-effort" in unit
assert "IOSchedulingPriority=4" in unit
# OOM ordering: game servers survive, sandbox dies first.
assert "OOMScoreAdjust=-200" in unit
# Memory caps with headroom for map-load spikes.
assert "MemoryHigh=1.5G" in unit
assert "MemoryMax=2G" in unit
# Bounded fork surface.
assert "TasksMax=256" in unit
# Plenty of fds for plugin-heavy setups.
assert "LimitNOFILE=65536" in unit
# srcds clean shutdown via SIGINT, with time to flush.
assert "KillSignal=SIGINT" in unit
assert "TimeoutStopSec=15s" in unit
# Per-unit override of journald rate limiting (default drops srcds output).
assert "LogRateLimitIntervalSec=0" in unit
```
- [ ] **Step 1.2: Run the new test, verify it fails**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_server_unit_contains_perf_baseline_directives -v`
Expected: FAIL — first failing assert is on `Slice=l4d2-game.slice`.
- [ ] **Step 1.3: Edit the unit file**
Open `deploy/files/usr/local/lib/systemd/system/left4me-server@.service` and replace its contents with:
```ini
[Unit]
Description=left4me server instance %i
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=left4me
Group=left4me
EnvironmentFile=/etc/left4me/host.env
EnvironmentFile=/var/lib/left4me/instances/%i/instance.env
WorkingDirectory=/var/lib/left4me/runtime/%i/merged/left4dead2
ExecStart=/var/lib/left4me/installation/srcds_run -game left4dead2 +hostport ${L4D2_PORT} $L4D2_ARGS
Restart=on-failure
RestartSec=5
# Resource control baseline — see docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md
Slice=l4d2-game.slice
Nice=-5
IOSchedulingClass=best-effort
IOSchedulingPriority=4
OOMScoreAdjust=-200
MemoryHigh=1.5G
MemoryMax=2G
TasksMax=256
LimitNOFILE=65536
KillSignal=SIGINT
TimeoutStopSec=15s
LogRateLimitIntervalSec=0
# Hardening (unchanged from previous baseline).
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectSystem=strict
ReadOnlyPaths=/var/lib/left4me/installation /var/lib/left4me/overlays
ReadWritePaths=/var/lib/left4me/runtime/%i
RestrictSUIDSGID=true
LockPersonality=true
[Install]
WantedBy=multi-user.target
```
- [ ] **Step 1.4: Run the new test, verify it passes**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_server_unit_contains_perf_baseline_directives -v`
Expected: PASS.
- [ ] **Step 1.5: Re-run the existing server-unit test, verify still passes**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_server_unit_contains_required_runtime_contract -v`
Expected: PASS — the existing assertions (`User=left4me`, `Group=left4me`, hardening directives, etc.) still match.
- [ ] **Step 1.6: Commit**
```bash
git add deploy/files/usr/local/lib/systemd/system/left4me-server@.service deploy/tests/test_deploy_artifacts.py
git commit -m "$(cat <<'EOF'
feat(deploy): perf-baseline directives on left4me-server@.service
Slice=l4d2-game.slice, Nice=-5, IOSchedulingClass=best-effort,
OOMScoreAdjust=-200, MemoryHigh=1.5G, MemoryMax=2G, TasksMax=256,
LimitNOFILE=65536, KillSignal=SIGINT, TimeoutStopSec=15s,
LogRateLimitIntervalSec=0. Spec:
docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md
EOF
)"
```
---
## Task 2: Slice Unit Files
Create the two slice unit files. After this task the perf unit's `Slice=l4d2-game.slice` reference is satisfied.
**Files:**
- Create: `deploy/files/usr/local/lib/systemd/system/l4d2-game.slice`
- Create: `deploy/files/usr/local/lib/systemd/system/l4d2-build.slice`
- Test: `deploy/tests/test_deploy_artifacts.py` (new constants + new test functions)
- [ ] **Step 2.1: Add path constants and failing tests**
Open `deploy/tests/test_deploy_artifacts.py`. After the existing `SERVER_UNIT = ...` line, add:
```python
GAME_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-game.slice"
BUILD_SLICE = DEPLOY / "files/usr/local/lib/systemd/system/l4d2-build.slice"
```
After the new `test_server_unit_contains_perf_baseline_directives`, append:
```python
def test_l4d2_game_slice_exists_with_high_weights():
assert GAME_SLICE.is_file()
text = GAME_SLICE.read_text()
assert "[Slice]" in text
assert "CPUWeight=1000" in text
assert "IOWeight=1000" in text
def test_l4d2_build_slice_exists_with_low_weights():
assert BUILD_SLICE.is_file()
text = BUILD_SLICE.read_text()
assert "[Slice]" in text
assert "CPUWeight=10" in text
assert "IOWeight=10" in text
```
- [ ] **Step 2.2: Run the new tests, verify they fail**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_l4d2_game_slice_exists_with_high_weights deploy/tests/test_deploy_artifacts.py::test_l4d2_build_slice_exists_with_low_weights -v`
Expected: FAIL on `assert GAME_SLICE.is_file()` (file does not exist).
- [ ] **Step 2.3: Create the game slice file**
Create `deploy/files/usr/local/lib/systemd/system/l4d2-game.slice` with:
```ini
[Unit]
Description=left4me game-server slice
Before=slices.target
[Slice]
CPUWeight=1000
IOWeight=1000
```
- [ ] **Step 2.4: Create the build slice file**
Create `deploy/files/usr/local/lib/systemd/system/l4d2-build.slice` with:
```ini
[Unit]
Description=left4me script-sandbox build slice
Before=slices.target
[Slice]
CPUWeight=10
IOWeight=10
```
- [ ] **Step 2.5: Run the new tests, verify they pass**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_l4d2_game_slice_exists_with_high_weights deploy/tests/test_deploy_artifacts.py::test_l4d2_build_slice_exists_with_low_weights -v`
Expected: PASS.
- [ ] **Step 2.6: Commit**
```bash
git add deploy/files/usr/local/lib/systemd/system/l4d2-game.slice deploy/files/usr/local/lib/systemd/system/l4d2-build.slice deploy/tests/test_deploy_artifacts.py
git commit -m "$(cat <<'EOF'
feat(deploy): l4d2-game.slice + l4d2-build.slice with 100:1 weight ratio
Flat top-level slices. Game wins under contention; build still gets
the box when uncontended. Referenced by left4me-server@.service and
the script-sandbox systemd-run invocation.
EOF
)"
```
---
## Task 3: Host Sysctls
Ship a `/etc/sysctl.d/` drop-in for UDP buffers, netdev backlog, netdev budget, and `vm.swappiness`.
**Files:**
- Create: `deploy/files/etc/sysctl.d/99-left4me.conf`
- Test: `deploy/tests/test_deploy_artifacts.py` (new constant + new test function)
- [ ] **Step 3.1: Add path constant and failing test**
Open `deploy/tests/test_deploy_artifacts.py`. After the slice constants, add:
```python
SYSCTL_CONF = DEPLOY / "files/etc/sysctl.d/99-left4me.conf"
```
Append a new test:
```python
def test_sysctl_conf_present_with_perf_settings():
assert SYSCTL_CONF.is_file()
text = SYSCTL_CONF.read_text()
for line in (
"net.core.rmem_max = 8388608",
"net.core.wmem_max = 8388608",
"net.core.rmem_default = 524288",
"net.core.wmem_default = 524288",
"net.core.netdev_max_backlog = 5000",
"net.core.netdev_budget = 600",
"vm.swappiness = 10",
):
assert line in text, f"missing {line!r} in 99-left4me.conf"
```
- [ ] **Step 3.2: Run the new test, verify it fails**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_sysctl_conf_present_with_perf_settings -v`
Expected: FAIL on `assert SYSCTL_CONF.is_file()`.
- [ ] **Step 3.3: Create the sysctl conf file**
Create `deploy/files/etc/sysctl.d/99-left4me.conf` with:
```
# Host-side perf baseline for left4me — see
# docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md
#
# UDP socket buffers: distro defaults of ~128 KiB are too small for sustained
# Source-engine UDP across multiple instances. 8 MiB matches the standard
# 1 Gbit recommendation; rmem_default/wmem_default protect sockets that don't
# explicitly enlarge their buffers.
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.rmem_default = 524288
net.core.wmem_default = 524288
# Kernel softirq UDP path: the per-CPU backlog queue starts dropping packets
# at the default 1000 under multi-instance burst; 5000 absorbs realistic peaks.
# netdev_budget = 600 gives softirq more drain headroom per pass.
net.core.netdev_max_backlog = 5000
net.core.netdev_budget = 600
# Latency-sensitive default: avoid swap unless the box is really under
# pressure. Harmless on swapless hosts.
vm.swappiness = 10
```
- [ ] **Step 3.4: Run the new test, verify it passes**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_sysctl_conf_present_with_perf_settings -v`
Expected: PASS.
- [ ] **Step 3.5: Commit**
```bash
git add deploy/files/etc/sysctl.d/99-left4me.conf deploy/tests/test_deploy_artifacts.py
git commit -m "$(cat <<'EOF'
feat(deploy): host sysctls for UDP buffers + netdev backlog/budget
99-left4me.conf: rmem_max/wmem_max=8M (with 512K defaults),
netdev_max_backlog=5000, netdev_budget=600, vm.swappiness=10.
EOF
)"
```
---
## Task 4: Sandbox in Build Slice
Place the script-sandbox transient unit into `l4d2-build.slice` and give it `OOMScoreAdjust=500` so it dies first under memory pressure.
**Files:**
- Modify: `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox`
- Test: `deploy/tests/test_deploy_artifacts.py` (new test function)
- [ ] **Step 4.1: Add the failing test**
Open `deploy/tests/test_deploy_artifacts.py`. Append:
```python
def test_script_sandbox_in_build_slice_with_oom_adjust():
text = SCRIPT_SANDBOX_HELPER.read_text()
# Put the transient unit in the low-weight build slice so it yields to
# game-server instances under CPU/IO contention.
assert "--slice=l4d2-build.slice" in text
# Sandbox dies first if the host hits memory pressure; servers
# (OOMScoreAdjust=-200) survive.
assert "-p OOMScoreAdjust=500" in text
```
- [ ] **Step 4.2: Run the new test, verify it fails**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_script_sandbox_in_build_slice_with_oom_adjust -v`
Expected: FAIL — neither string is in the helper yet.
- [ ] **Step 4.3: Edit the sandbox helper**
Open `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox`. Locate the `systemd-run` invocation that begins with:
```
systemd-run --quiet --collect --wait --pipe \
--unit="left4me-script-${OVERLAY_ID}-$$" \
```
Insert two new lines immediately after the `--unit=` line, before `-p User=l4d2-sandbox`. The block becomes:
```
systemd-run --quiet --collect --wait --pipe \
--unit="left4me-script-${OVERLAY_ID}-$$" \
--slice=l4d2-build.slice \
-p OOMScoreAdjust=500 \
-p User=l4d2-sandbox -p Group=l4d2-sandbox \
```
Leave every other `-p` line untouched.
- [ ] **Step 4.4: Verify shell syntax still parses**
Run: `bash -n deploy/files/usr/local/libexec/left4me/left4me-script-sandbox`
Expected: exit 0, no output.
- [ ] **Step 4.5: Run the new test and the existing sandbox-helper tests, verify they pass**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_script_sandbox_in_build_slice_with_oom_adjust deploy/tests/test_deploy_artifacts.py::test_script_sandbox_helper_invokes_systemd_run_with_hardening deploy/tests/test_deploy_artifacts.py::test_script_sandbox_helper_passes_shell_syntax_check -v`
Expected: PASS for all three. The hardening test still matches because it only checks for substring presence; we added strings, didn't remove any.
- [ ] **Step 4.6: Commit**
```bash
git add deploy/files/usr/local/libexec/left4me/left4me-script-sandbox deploy/tests/test_deploy_artifacts.py
git commit -m "$(cat <<'EOF'
feat(deploy): script-sandbox runs in l4d2-build.slice + OOMScoreAdjust=500
Builds yield CPU/IO to game-server instances under contention via the
slice's weight=10, and are killed first under memory pressure
(servers have OOMScoreAdjust=-200).
EOF
)"
```
---
## Task 5: Deploy Script Installs Slice + Sysctl Artifacts
Wire the new artifacts into `deploy-test-server.sh` so a fresh deploy actually puts them on disk and applies the sysctls.
**Files:**
- Modify: `deploy/deploy-test-server.sh`
- Test: `deploy/tests/test_deploy_artifacts.py` (new test function)
- [ ] **Step 5.1: Add the failing test**
Open `deploy/tests/test_deploy_artifacts.py`. Append:
```python
def test_deploy_script_installs_perf_artifacts():
script = DEPLOY_SCRIPT.read_text()
# Slice files copied into the system-wide systemd unit dir.
assert "/usr/local/lib/systemd/system/l4d2-game.slice" in script
assert "/usr/local/lib/systemd/system/l4d2-build.slice" in script
# Sysctl drop-in installed under /etc/sysctl.d/.
assert "/etc/sysctl.d/99-left4me.conf" in script
# Values applied immediately, not on next boot.
assert "sysctl --system" in script
```
- [ ] **Step 5.2: Run the new test, verify it fails**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_deploy_script_installs_perf_artifacts -v`
Expected: FAIL on the first assertion.
- [ ] **Step 5.3: Edit the deploy script — copy the slice + sysctl files**
Open `deploy/deploy-test-server.sh`. Find the block that copies unit files (currently around line 138):
```sh
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-web.service /usr/local/lib/systemd/system/left4me-web.service
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-server@.service /usr/local/lib/systemd/system/left4me-server@.service
```
Add two new lines immediately after the `left4me-server@.service` copy line, so the block becomes:
```sh
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-web.service /usr/local/lib/systemd/system/left4me-web.service
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-server@.service /usr/local/lib/systemd/system/left4me-server@.service
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/l4d2-game.slice /usr/local/lib/systemd/system/l4d2-game.slice
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/l4d2-build.slice /usr/local/lib/systemd/system/l4d2-build.slice
```
- [ ] **Step 5.4: Edit the deploy script — install the sysctl conf and apply it**
In `deploy/deploy-test-server.sh`, find the block that installs `/etc/left4me/sandbox-resolv.conf` (currently around lines 153–155):
```sh
$sudo_cmd install -m 0644 -o root -g root \
/opt/left4me/deploy/files/etc/left4me/sandbox-resolv.conf \
/etc/left4me/sandbox-resolv.conf
```
Immediately after that block, add:
```sh
# Host perf-baseline sysctls. Apply with `sysctl --system` so values
# take effect this deploy, not on next reboot.
$sudo_cmd install -m 0644 -o root -g root \
/opt/left4me/deploy/files/etc/sysctl.d/99-left4me.conf \
/etc/sysctl.d/99-left4me.conf
$sudo_cmd sysctl --system >/dev/null
```
- [ ] **Step 5.5: Verify the deploy script's shell syntax still parses**
Run: `sh -n deploy/deploy-test-server.sh`
Expected: exit 0, no output.
- [ ] **Step 5.6: Run the new test and the existing deploy-script tests, verify they pass**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py::test_deploy_script_installs_perf_artifacts deploy/tests/test_deploy_artifacts.py::test_deploy_script_has_safe_defaults_and_preserves_state deploy/tests/test_deploy_artifacts.py::test_deploy_script_shell_syntax -v`
Expected: PASS for all three.
- [ ] **Step 5.7: Commit**
```bash
git add deploy/deploy-test-server.sh deploy/tests/test_deploy_artifacts.py
git commit -m "$(cat <<'EOF'
feat(deploy): install slice + sysctl artifacts and apply via sysctl --system
Copies l4d2-game.slice and l4d2-build.slice into
/usr/local/lib/systemd/system/, installs 99-left4me.conf into
/etc/sysctl.d/, and runs sysctl --system so the perf baseline is
live this deploy, not on next reboot.
EOF
)"
```
---
## Task 6: Performance-Tuning Section in deploy/README.md
Document the four escape hatches the spec lists as opt-in: CPU governor, per-instance `CPUAffinity`, NIC tuning, and SCHED_FIFO.
**Files:**
- Modify: `deploy/README.md`
No test for this task — README content is documentation, not contract.
- [ ] **Step 6.1: Append the Performance Tuning section**
Open `deploy/README.md`. Append (after the existing final paragraph) a new section:
```markdown
## Performance Tuning
The deployment ships a host-side perf baseline (slices, unit directives, sysctls). See `docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md` for design rationale.
The following knobs are documented escape hatches — they are **not** auto-applied. Apply only if you have measured a need and understand the failure modes.
### CPU governor
The performance governor squeezes a few percent off jitter under bursty load. `schedutil` is acceptable for sustained UDP workloads.
```sh
sudo cpupower frequency-set -g performance
```
Persist via your distro's CPU-frequency tooling (e.g. `/etc/default/cpufrequtils`).
### Per-instance CPU affinity
`srcds` is single-threaded per instance. On a multi-core host, pinning each instance to its own core can cut jitter under contention. Drop in `/etc/systemd/system/left4me-server@<name>.service.d/affinity.conf`:
```ini
[Service]
CPUAffinity=2
```
A reasonable strategy on an N-core host: leave core 0 for the kernel + IRQs + system services, then pin one instance per remaining core.
### NIC tuning
Hardware-specific. On a host with a single primary interface (replace `eth0`):
```sh
sudo ethtool -G eth0 rx 4096 tx 4096
sudo ethtool -K eth0 gro on lro off
```
If you run a high instance count, also pin the NIC's interrupts off the cores that game servers occupy (see `/proc/interrupts` and `/proc/irq/<n>/smp_affinity`).
### Real-time scheduling (advanced, opt-in)
Source-engine servers do not need real-time scheduling, and a misbehaving `srcds` at any RT priority can starve kernel threads — even with the default `kernel.sched_rt_runtime_us=950000` throttling 5% of CPU back. Use only if you have a measured jitter problem that the baseline does not solve.
`/etc/systemd/system/left4me-server@.service.d/realtime.conf`:
```ini
[Service]
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=10
LimitRTPRIO=10
```
### Applying changes to running servers
Unit-file changes do not apply to already-running services. After any change:
```sh
sudo systemctl daemon-reload
# Restart each game server via the web UI's stop + start, or:
sudo systemctl restart 'left4me-server@*.service'
```
```
- [ ] **Step 6.2: Run the full deploy test suite and verify it stays green**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/test_deploy_artifacts.py -q`
Expected: all green. README changes have no test, but should not break any existing tests.
- [ ] **Step 6.3: Commit**
```bash
git add deploy/README.md
git commit -m "$(cat <<'EOF'
docs(deploy): performance-tuning escape-hatch section in README
Documents CPU governor, per-instance CPUAffinity, NIC tuning, and
SCHED_FIFO opt-in patterns. None of these are auto-applied; they're
ops-side knobs for measured problems the perf baseline doesn't solve.
EOF
)"
```
---
## Final Verification
- [ ] **Step F.1: Full deploy test suite green**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest deploy/tests/ -q`
Expected: all green.
- [ ] **Step F.2: Host library + web tests still green (regression check)**
Run: `cd /Users/mwiegand/Projekte/left4me && pytest l4d2host/tests -q && pytest l4d2web/tests -q`
Expected: all green. Nothing in this plan touches host or web Python code, but a clean run rules out accidental import-time damage.
- [ ] **Step F.3: Working tree clean and commits in order**
Run: `git status && git log --oneline -8`
Expected:
- `git status`: `nothing to commit, working tree clean`.
- `git log`: six new commits in this order, top-most first:
1. `docs(deploy): performance-tuning escape-hatch section in README`
2. `feat(deploy): install slice + sysctl artifacts and apply via sysctl --system`
3. `feat(deploy): script-sandbox runs in l4d2-build.slice + OOMScoreAdjust=500`
4. `feat(deploy): host sysctls for UDP buffers + netdev backlog/budget`
5. `feat(deploy): l4d2-game.slice + l4d2-build.slice with 100:1 weight ratio`
6. `feat(deploy): perf-baseline directives on left4me-server@.service`
If any step is missing or out of order, do not amend — diagnose, fix, and create new commits.
- [ ] **Step F.4: Manual deploy smoke test (deferred, ops-side)**
This plan ships artifacts. Confirming that systemd actually accepts and applies them on a real host requires running the deploy script against a test target. That validation is operator-side, not part of this implementation:
```sh
deploy/deploy-test-server.sh deploy-user@example-host
ssh deploy-user@example-host 'systemctl cat l4d2-game.slice'
ssh deploy-user@example-host 'sysctl net.core.rmem_max' # expect 8388608
ssh deploy-user@example-host 'systemd-analyze verify /usr/local/lib/systemd/system/left4me-server@.service'
```
Document any deploy-time problems back into the spec or this plan as v1.x corrections. Do not invent fixes that go beyond the spec.
---
## Out of Scope (do NOT implement here)
Listed in the spec — repeated for clarity:
- ConVars / blueprint arguments / tickrate / sv_minrate.
- SCHED_FIFO auto-apply.
- CPU governor auto-apply.
- Per-instance `CPUAffinity` auto-apply.
- NIC ring-buffer / IRQ-pinning code.
- Job-scheduler awareness ("don't build while server X has players").
- Hardening tightening (`ProtectKernelTunables=yes`, etc.).
If you find yourself touching any of these, stop — they belong in a separate spec.