diff --git a/docs/superpowers/plans/2026-05-10-l4d2-network-shaping.md b/docs/superpowers/plans/2026-05-10-l4d2-network-shaping.md new file mode 100644 index 0000000..facaa2b --- /dev/null +++ b/docs/superpowers/plans/2026-05-10-l4d2-network-shaping.md @@ -0,0 +1,895 @@ +# L4D2 Network Shaping & Marking Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ship a network-side player-experience baseline alongside the existing host perf baseline: nftables uid-based DSCP-EF + skb-priority marking for srcds UDP, rounding sysctls (`udp_rmem_min`/`wmem_min`, `default_qdisc=fq_codel`, `tcp_congestion_control=bbr`), and CAKE egress shaping via a systemd oneshot driven by an operator-edited env file. Production hosts running `systemd-networkd` consume an equivalent `[CAKE]` section documented in the README. + +**Architecture:** Eight ship-ready artifacts under `deploy/files/...`, wired into `deploy-test-server.sh`, asserted in `deploy/tests/test_deploy_artifacts.py`, and documented in `deploy/README.md`. Each artifact is a separate, independently-testable file. The CAKE helper takes an `apply`/`clear` mode argument so the unit's `ExecStart`/`ExecStop` are clean shell calls without escape soup. + +**Tech Stack:** sysctl, nftables (`inet` table, output hook, mangle priority), tc-cake, systemd oneshot units, POSIX `/bin/sh` for the helper, pytest substring assertions. + +**Spec:** `docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md`. + +--- + +## File Structure + +**New files (`deploy/files/...`):** +- `usr/local/lib/left4me/nft/left4me-mark.nft` — nftables ruleset, own `inet` table. +- `usr/local/lib/systemd/system/left4me-nft-mark.service` — applies/removes the table. +- `etc/left4me/cake.env` — operator-edited template (deploy preserves edits). +- `usr/local/libexec/left4me/left4me-apply-cake` — POSIX shell helper, `apply`/`clear` modes. +- `usr/local/lib/systemd/system/left4me-cake.service` — runs the helper at network-online, clears on stop. + +**Modified files:** +- `deploy/files/etc/sysctl.d/99-left4me.conf` — append four new directives. +- `deploy/deploy-test-server.sh` — add `nftables iproute2` to apt/dnf install lines, copy the new artifacts, conditional cake.env copy, enable the two new units. +- `deploy/README.md` — Network shaping subsection + three new escape hatches (IFB ingress, busy_poll, GRO). +- `deploy/tests/test_deploy_artifacts.py` — add path constants and assertions. + +Each task adds (or extends) one artifact and the matching test, ending in a commit. Order matters: sysctl extension first (smallest, isolated), then the nftables pair, then the CAKE pair, then deploy-script wiring (depends on every prior task), then README. + +--- + +### Task 1: Sysctl additions to `99-left4me.conf` + +**Files:** +- Modify: `deploy/files/etc/sysctl.d/99-left4me.conf` (append block) +- Modify: `deploy/tests/test_deploy_artifacts.py:199-211` (extend existing `test_sysctl_conf_present_with_perf_settings`) + +- [ ] **Step 1: Extend the existing sysctl test with the new lines.** + +In `deploy/tests/test_deploy_artifacts.py`, edit `test_sysctl_conf_present_with_perf_settings` to append four lines to the tuple it already iterates: + +```python +def test_sysctl_conf_present_with_perf_settings(): + assert SYSCTL_CONF.is_file() + text = SYSCTL_CONF.read_text() + for line in ( + "net.core.rmem_max = 8388608", + "net.core.wmem_max = 8388608", + "net.core.rmem_default = 524288", + "net.core.wmem_default = 524288", + "net.core.netdev_max_backlog = 5000", + "net.core.netdev_budget = 600", + "vm.swappiness = 10", + "net.ipv4.udp_rmem_min = 16384", + "net.ipv4.udp_wmem_min = 16384", + "net.core.default_qdisc = fq_codel", + "net.ipv4.tcp_congestion_control = bbr", + ): + assert line in text, f"missing {line!r} in 99-left4me.conf" +``` + +- [ ] **Step 2: Run the test to verify it fails.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_sysctl_conf_present_with_perf_settings -v +``` +Expected: FAIL — `AssertionError: missing 'net.ipv4.udp_rmem_min = 16384' in 99-left4me.conf`. + +- [ ] **Step 3: Append the new block to `99-left4me.conf`.** + +Open `deploy/files/etc/sysctl.d/99-left4me.conf` and append (after the existing `vm.swappiness = 10` line): + +``` +# Per-socket UDP buffer floors: protect game-server sockets that don't bump +# their own SO_RCVBUF/SO_SNDBUF when softirq drains lag briefly. +net.ipv4.udp_rmem_min = 16384 +net.ipv4.udp_wmem_min = 16384 + +# Default qdisc for ifaces we don't explicitly shape with CAKE. Debian Trixie +# already defaults to fq_codel; setting it explicitly is belt-and-suspenders +# and survives kernel-default churn. +net.core.default_qdisc = fq_codel + +# TCP congestion control: BBR for any bulk TCP egress on the host (admin SSH, +# backups, package fetches, web-app responses) so a long flow does not push +# the bottleneck queue ahead of game UDP. UDP srcds is unaffected. +net.ipv4.tcp_congestion_control = bbr +``` + +- [ ] **Step 4: Run the test again to verify it passes.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_sysctl_conf_present_with_perf_settings -v +``` +Expected: PASS. + +- [ ] **Step 5: Commit.** + +``` +git add deploy/files/etc/sysctl.d/99-left4me.conf deploy/tests/test_deploy_artifacts.py +git commit -m "feat(deploy): extend sysctls with udp_*_min, fq_codel default, BBR" +``` + +--- + +### Task 2: nftables marking file + +**Files:** +- Create: `deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft` +- Modify: `deploy/tests/test_deploy_artifacts.py` (add path constant + new test function) + +- [ ] **Step 1: Add the path constant and a failing test.** + +In `deploy/tests/test_deploy_artifacts.py`, add the constant near the existing path constants block (around line 26, after `DEPLOY_SCRIPT`): + +```python +NFT_MARK_FILE = DEPLOY / "files/usr/local/lib/left4me/nft/left4me-mark.nft" +``` + +Append this test function to the bottom of the file: + +```python +def test_nft_mark_file_marks_left4me_udp_with_dscp_ef_and_priority(): + assert NFT_MARK_FILE.is_file() + text = NFT_MARK_FILE.read_text() + + # Own table in the inet family so it cannot conflict with operator nftables config. + assert "table inet left4me_mark" in text + assert "chain mangle_output" in text + assert "type filter hook output priority mangle" in text + + # Match by uid (every srcds runs as `left4me`) restricted to UDP. + assert 'meta skuid "left4me"' in text + assert "meta l4proto udp" in text + + # DSCP EF for both L3 families; in `inet` tables, `ip` only fires on v4 + # and `ip6` only on v6. + assert "ip dscp set ef" in text + assert "ip6 dscp set ef" in text + + # skb->priority class 6:0, set inline alongside DSCP. + assert "meta priority set 0006:0000" in text +``` + +- [ ] **Step 2: Run the new test and confirm it fails.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_nft_mark_file_marks_left4me_udp_with_dscp_ef_and_priority -v +``` +Expected: FAIL — `AssertionError: assert False` on `NFT_MARK_FILE.is_file()`. + +- [ ] **Step 3: Create the directory and write the nftables file.** + +``` +mkdir -p deploy/files/usr/local/lib/left4me/nft +``` + +Write `deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft`: + +```nft +# left4me — uid-based DSCP/priority marking for srcds UDP egress. +# Loaded by left4me-nft-mark.service into its own `inet` table so it cannot +# conflict with whatever the operator already runs in /etc/nftables.conf. +# See docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md. + +table inet left4me_mark { + chain mangle_output { + type filter hook output priority mangle; policy accept; + meta skuid "left4me" meta l4proto udp ip dscp set ef meta priority set 0006:0000 + meta skuid "left4me" meta l4proto udp ip6 dscp set ef meta priority set 0006:0000 + } +} +``` + +- [ ] **Step 4: Re-run the test and confirm it passes.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_nft_mark_file_marks_left4me_udp_with_dscp_ef_and_priority -v +``` +Expected: PASS. + +- [ ] **Step 5: Commit.** + +``` +git add deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft deploy/tests/test_deploy_artifacts.py +git commit -m "feat(deploy): nftables uid-based DSCP-EF + skb-priority marking for srcds" +``` + +--- + +### Task 3: nftables systemd unit + +**Files:** +- Create: `deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service` +- Modify: `deploy/tests/test_deploy_artifacts.py` (path constant + test) + +- [ ] **Step 1: Add the path constant and a failing test.** + +Append the constant near the existing systemd-unit constants (around line 16): + +```python +NFT_MARK_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-nft-mark.service" +``` + +Append the test: + +```python +def test_nft_mark_unit_loads_and_clears_left4me_table(): + assert NFT_MARK_UNIT.is_file() + text = NFT_MARK_UNIT.read_text() + + # Loads the rules early so the very first packet srcds emits is marked. + assert "After=network-pre.target" in text + assert "Before=network.target" in text + assert "Wants=network-pre.target" in text + + # Oneshot lifecycle: load on start, drop on stop. + assert "Type=oneshot" in text + assert "RemainAfterExit=yes" in text + assert ( + "ExecStart=/usr/sbin/nft -f /usr/local/lib/left4me/nft/left4me-mark.nft" + in text + ) + assert "ExecStop=/usr/sbin/nft delete table inet left4me_mark" in text + assert "WantedBy=multi-user.target" in text +``` + +- [ ] **Step 2: Run the test and confirm FAIL.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_nft_mark_unit_loads_and_clears_left4me_table -v +``` +Expected: FAIL — `assert False` on `NFT_MARK_UNIT.is_file()`. + +- [ ] **Step 3: Write the unit file.** + +`deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service`: + +```ini +[Unit] +Description=left4me nftables packet marking (DSCP EF + priority for srcds) +After=network-pre.target +Before=network.target +Wants=network-pre.target + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/sbin/nft -f /usr/local/lib/left4me/nft/left4me-mark.nft +ExecStop=/usr/sbin/nft delete table inet left4me_mark + +[Install] +WantedBy=multi-user.target +``` + +- [ ] **Step 4: Re-run the test and confirm PASS.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_nft_mark_unit_loads_and_clears_left4me_table -v +``` +Expected: PASS. + +- [ ] **Step 5: Commit.** + +``` +git add deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service deploy/tests/test_deploy_artifacts.py +git commit -m "feat(deploy): systemd unit to load/clear left4me_mark nftables table" +``` + +--- + +### Task 4: CAKE env template + +**Files:** +- Create: `deploy/files/etc/left4me/cake.env` +- Modify: `deploy/tests/test_deploy_artifacts.py` (path constant + test) + +- [ ] **Step 1: Add path constant and failing test.** + +Append the constant near the other `/etc/left4me` constants (around line 22): + +```python +CAKE_ENV = DEPLOY / "files/etc/left4me/cake.env" +``` + +Append the test: + +```python +def test_cake_env_template_documents_required_knobs(): + assert CAKE_ENV.is_file() + text = CAKE_ENV.read_text() + + # Both knobs are documented and present (commented OK; the deploy preserves + # operator edits, so the template must not bake in a wrong value). + assert "LEFT4ME_UPLINK_MBIT" in text + assert "LEFT4ME_UPLINK_IFACE" in text + # Empty defaults: shaper unit no-ops with a journal warning when unset. + assert "LEFT4ME_UPLINK_MBIT=" in text + assert "LEFT4ME_UPLINK_IFACE=" in text +``` + +- [ ] **Step 2: Run and confirm FAIL.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_cake_env_template_documents_required_knobs -v +``` +Expected: FAIL on `CAKE_ENV.is_file()`. + +- [ ] **Step 3: Write the env template.** + +`deploy/files/etc/left4me/cake.env`: + +``` +# left4me — CAKE egress shaper config. Consumed by left4me-cake.service via +# its EnvironmentFile=. Edit then `systemctl restart left4me-cake.service`. +# See docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md. + +# Uplink bandwidth in Mbit/s. Set to ~95% of the smaller of measured upload +# and measured download. CAKE only shapes correctly when its declared +# bandwidth sits below the real bottleneck. If unset, the shaper unit logs +# a warning and exits 0 (no shaping). +LEFT4ME_UPLINK_MBIT= + +# Egress interface. If unset, auto-detected from the IPv4 default route. +LEFT4ME_UPLINK_IFACE= +``` + +- [ ] **Step 4: Re-run and confirm PASS.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_cake_env_template_documents_required_knobs -v +``` +Expected: PASS. + +- [ ] **Step 5: Commit.** + +``` +git add deploy/files/etc/left4me/cake.env deploy/tests/test_deploy_artifacts.py +git commit -m "feat(deploy): cake.env template with documented uplink knobs" +``` + +--- + +### Task 5: CAKE helper script + +**Files:** +- Create: `deploy/files/usr/local/libexec/left4me/left4me-apply-cake` +- Modify: `deploy/tests/test_deploy_artifacts.py` (path constant + tests) + +- [ ] **Step 1: Add path constant and failing tests.** + +Append the constant near the libexec helper constants (around line 21): + +```python +APPLY_CAKE_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-apply-cake" +``` + +Append two test functions: + +```python +def test_apply_cake_helper_supports_apply_and_clear_modes(): + assert APPLY_CAKE_HELPER.is_file() + text = APPLY_CAKE_HELPER.read_text() + + assert text.startswith("#!/bin/sh") + # Both knobs are read from the env file. + assert "LEFT4ME_UPLINK_MBIT" in text + assert "LEFT4ME_UPLINK_IFACE" in text + assert ". /etc/left4me/cake.env" in text + # Iface fallback to default route. + assert "ip -4 route show default" in text + # Two modes; default to apply. + assert "mode=${1:-apply}" in text + assert 'apply)' in text and 'clear)' in text + # Apply: idempotent `tc qdisc replace` with the documented flags. + assert "tc qdisc replace" in text + assert "cake" in text + assert "bandwidth" in text + assert "internet" in text + assert "diffserv4" in text + assert "dual-dsthost" in text + # Clear: tolerates a missing qdisc. + assert "tc qdisc del" in text + assert "|| true" in text + # Fail-soft on missing config. + assert "LEFT4ME_UPLINK_MBIT unset" in text + + +def test_apply_cake_helper_passes_shell_syntax_check(): + subprocess.run(["sh", "-n", str(APPLY_CAKE_HELPER)], check=True) +``` + +- [ ] **Step 2: Run and confirm FAIL.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_apply_cake_helper_supports_apply_and_clear_modes deploy/tests/test_deploy_artifacts.py::test_apply_cake_helper_passes_shell_syntax_check -v +``` +Expected: both FAIL. + +- [ ] **Step 3: Write the helper.** + +`deploy/files/usr/local/libexec/left4me/left4me-apply-cake`: + +```sh +#!/bin/sh +# left4me — apply or clear CAKE egress shaper on the configured uplink. +# Driven by left4me-cake.service. See spec +# docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md. +set -eu + +mode=${1:-apply} + +if [ -r /etc/left4me/cake.env ]; then + . /etc/left4me/cake.env +fi + +resolve_iface() { + if [ -n "${LEFT4ME_UPLINK_IFACE:-}" ]; then + printf '%s' "$LEFT4ME_UPLINK_IFACE" + return + fi + ip -4 route show default | awk '/default/ {print $5; exit}' +} + +case "$mode" in + apply) + if [ -z "${LEFT4ME_UPLINK_MBIT:-}" ]; then + echo "left4me-cake: LEFT4ME_UPLINK_MBIT unset; skipping shaper" >&2 + exit 0 + fi + iface=$(resolve_iface) + if [ -z "$iface" ]; then + echo "left4me-cake: cannot determine egress iface; skipping" >&2 + exit 0 + fi + exec tc qdisc replace dev "$iface" root cake \ + bandwidth "${LEFT4ME_UPLINK_MBIT}mbit" \ + internet diffserv4 dual-dsthost + ;; + clear) + iface=$(resolve_iface) + if [ -z "$iface" ]; then + exit 0 + fi + tc qdisc del dev "$iface" root 2>/dev/null || true + ;; + *) + echo "usage: $0 [apply|clear]" >&2 + exit 2 + ;; +esac +``` + +Make it executable in the repo (the deploy script also `chmod 0755`s the destination, but executable mode in the source tree is conventional here): + +``` +chmod 0755 deploy/files/usr/local/libexec/left4me/left4me-apply-cake +``` + +- [ ] **Step 4: Re-run and confirm PASS.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_apply_cake_helper_supports_apply_and_clear_modes deploy/tests/test_deploy_artifacts.py::test_apply_cake_helper_passes_shell_syntax_check -v +``` +Expected: both PASS. + +- [ ] **Step 5: Commit.** + +``` +git add deploy/files/usr/local/libexec/left4me/left4me-apply-cake deploy/tests/test_deploy_artifacts.py +git commit -m "feat(deploy): left4me-apply-cake helper with apply/clear modes" +``` + +--- + +### Task 6: CAKE systemd unit + +**Files:** +- Create: `deploy/files/usr/local/lib/systemd/system/left4me-cake.service` +- Modify: `deploy/tests/test_deploy_artifacts.py` (path constant + test) + +- [ ] **Step 1: Add path constant and failing test.** + +Append the constant near the existing systemd-unit constants (around line 16): + +```python +CAKE_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-cake.service" +``` + +Append the test: + +```python +def test_cake_unit_runs_helper_in_apply_and_clear_modes(): + assert CAKE_UNIT.is_file() + text = CAKE_UNIT.read_text() + + assert "After=network-online.target" in text + assert "Wants=network-online.target" in text + assert "Type=oneshot" in text + assert "RemainAfterExit=yes" in text + # `-` prefix: missing env file is non-fatal (deploy ships one, but be safe). + assert "EnvironmentFile=-/etc/left4me/cake.env" in text + assert ( + "ExecStart=/usr/local/libexec/left4me/left4me-apply-cake apply" in text + ) + assert ( + "ExecStop=/usr/local/libexec/left4me/left4me-apply-cake clear" in text + ) + assert "WantedBy=multi-user.target" in text +``` + +- [ ] **Step 2: Run and confirm FAIL.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_cake_unit_runs_helper_in_apply_and_clear_modes -v +``` +Expected: FAIL on `CAKE_UNIT.is_file()`. + +- [ ] **Step 3: Write the unit.** + +`deploy/files/usr/local/lib/systemd/system/left4me-cake.service`: + +```ini +[Unit] +Description=left4me CAKE egress shaper +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +RemainAfterExit=yes +EnvironmentFile=-/etc/left4me/cake.env +ExecStart=/usr/local/libexec/left4me/left4me-apply-cake apply +ExecStop=/usr/local/libexec/left4me/left4me-apply-cake clear + +[Install] +WantedBy=multi-user.target +``` + +- [ ] **Step 4: Re-run and confirm PASS.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_cake_unit_runs_helper_in_apply_and_clear_modes -v +``` +Expected: PASS. + +- [ ] **Step 5: Commit.** + +``` +git add deploy/files/usr/local/lib/systemd/system/left4me-cake.service deploy/tests/test_deploy_artifacts.py +git commit -m "feat(deploy): left4me-cake.service oneshot wrapping apply-cake helper" +``` + +--- + +### Task 7: Wire artifacts into `deploy-test-server.sh` + +**Files:** +- Modify: `deploy/deploy-test-server.sh` +- Modify: `deploy/tests/test_deploy_artifacts.py` (new test) + +This task adds: `nftables` to apt/dnf install lines, copies the four new artifact files into their target paths, conditionally copies `cake.env` only if absent, and `systemctl enable --now`s the two new units. Each piece gets its own assertion in a single new test function. + +- [ ] **Step 1: Add the new test.** + +Append to `deploy/tests/test_deploy_artifacts.py`: + +```python +def test_deploy_script_installs_network_shaping_artifacts(): + script = DEPLOY_SCRIPT.read_text() + + # nftables: package install on both apt and dnf paths. + apt_lines = [l for l in script.splitlines() if "apt-get install" in l] + dnf_lines = [l for l in script.splitlines() if "dnf install" in l] + assert apt_lines and dnf_lines + for line in apt_lines: + assert "nftables" in line, line + for line in dnf_lines: + assert "nftables" in line, line + + # nft rules + unit copied to system paths. + assert "/usr/local/lib/left4me/nft/left4me-mark.nft" in script + assert ( + "/usr/local/lib/systemd/system/left4me-nft-mark.service" in script + ) + assert "systemctl enable --now left4me-nft-mark.service" in script + + # CAKE helper + unit copied; helper made executable. + assert "/usr/local/libexec/left4me/left4me-apply-cake" in script + assert ( + "/usr/local/lib/systemd/system/left4me-cake.service" in script + ) + assert "chmod 0755" in script and "left4me-apply-cake" in script + assert "systemctl enable --now left4me-cake.service" in script + + # cake.env: copied only if absent (operator edits survive re-deploys). + assert "/etc/left4me/cake.env" in script + assert "[ -e /etc/left4me/cake.env ]" in script +``` + +- [ ] **Step 2: Run and confirm FAIL.** + +``` +pytest deploy/tests/test_deploy_artifacts.py::test_deploy_script_installs_network_shaping_artifacts -v +``` +Expected: FAIL on the first missing string. + +- [ ] **Step 3: Edit `deploy-test-server.sh`.** + +Make these targeted edits — do not rewrite the script. + +(a) **Append `nftables` to both package-install lines (line 88 and line 90 in the current file).** + +Old (line 88): +``` + $sudo_cmd apt-get install -y python3 python3-venv python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip-full +``` +New: +``` + $sudo_cmd apt-get install -y python3 python3-venv python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip-full nftables +``` + +Old (line 90): +``` + $sudo_cmd dnf install -y python3 python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip p7zip-plugins +``` +New: +``` + $sudo_cmd dnf install -y python3 python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip p7zip-plugins nftables +``` + +(b) **Add the nft-rules-dir creation to the `mkdir -p` block (currently lines 96-106).** + +Append `/usr/local/lib/left4me/nft` to the existing `mkdir -p` invocation: + +Old (lines 96-106): +``` +$sudo_cmd mkdir -p \ + /etc/left4me \ + /opt/left4me \ + /usr/local/lib/systemd/system \ + /usr/local/libexec/left4me \ + /var/lib/left4me/installation \ + /var/lib/left4me/overlays \ + /var/lib/left4me/instances \ + /var/lib/left4me/runtime \ + /var/lib/left4me/workshop_cache \ + /var/lib/left4me/tmp +``` +New (insert one line after `/usr/local/libexec/left4me`): +``` +$sudo_cmd mkdir -p \ + /etc/left4me \ + /opt/left4me \ + /usr/local/lib/systemd/system \ + /usr/local/libexec/left4me \ + /usr/local/lib/left4me/nft \ + /var/lib/left4me/installation \ + /var/lib/left4me/overlays \ + /var/lib/left4me/instances \ + /var/lib/left4me/runtime \ + /var/lib/left4me/workshop_cache \ + /var/lib/left4me/tmp +``` + +(c) **Copy the new systemd units alongside the existing ones (after line 140's `l4d2-build.slice` copy).** + +Insert immediately after the `l4d2-build.slice` copy (the existing line that ends `l4d2-build.slice`): + +``` +$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service /usr/local/lib/systemd/system/left4me-nft-mark.service +$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-cake.service /usr/local/lib/systemd/system/left4me-cake.service +``` + +(d) **Copy the nftables rules file alongside the existing `install`-mode copies (next to the sandbox-resolv.conf install at lines 189-191).** + +Insert after the sandbox-resolv install block: + +``` +# Network packet marking + shaping. See spec +# docs/superpowers/specs/2026-05-10-l4d2-network-shaping-design.md. +$sudo_cmd install -m 0644 -o root -g root \ + /opt/left4me/deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft \ + /usr/local/lib/left4me/nft/left4me-mark.nft +``` + +(e) **Copy the CAKE helper alongside the other libexec helpers (after the existing `cp` block at lines 175-179).** + +Find the existing `cp` block that copies `left4me-systemctl`, `left4me-journalctl`, `left4me-overlay`, `left4me-script-sandbox`. Add a new `cp` line for `left4me-apply-cake`, and add it to the `chmod 0755` line on line 179: + +Old (line 178): +``` +$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-script-sandbox /usr/local/libexec/left4me/left4me-script-sandbox +``` +After it, insert: +``` +$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-apply-cake /usr/local/libexec/left4me/left4me-apply-cake +``` + +Old (line 179): +``` +$sudo_cmd chmod 0755 /usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-script-sandbox +``` +New (append `left4me-apply-cake`): +``` +$sudo_cmd chmod 0755 /usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-script-sandbox /usr/local/libexec/left4me/left4me-apply-cake +``` + +(f) **Conditionally copy `cake.env` (after the existing sysctl install/apply block at lines 193-198).** + +Insert immediately after `$sudo_cmd sysctl --system >/dev/null`: + +``` +# CAKE config: ship the template only if the operator hasn't created one +# (their LEFT4ME_UPLINK_MBIT value must survive re-deploys). +if [ ! -e /etc/left4me/cake.env ]; then + $sudo_cmd install -m 0644 -o root -g root \ + /opt/left4me/deploy/files/etc/left4me/cake.env \ + /etc/left4me/cake.env +fi +``` + +(g) **Enable the new units alongside the existing `systemctl enable --now left4me-web.service`.** + +Find the existing block (around line 315-316): +``` +$sudo_cmd systemctl daemon-reload +$sudo_cmd systemctl enable --now left4me-web.service +``` + +Insert two lines between them: +``` +$sudo_cmd systemctl daemon-reload +$sudo_cmd systemctl enable --now left4me-nft-mark.service +$sudo_cmd systemctl enable --now left4me-cake.service +$sudo_cmd systemctl enable --now left4me-web.service +``` + +- [ ] **Step 4: Re-run all existing tests + the new one to make sure nothing regressed.** + +``` +pytest deploy/tests/test_deploy_artifacts.py -v +``` +Expected: every test passes, including the new `test_deploy_script_installs_network_shaping_artifacts` and the unmodified `test_deploy_script_shell_syntax` (the latter validates `sh -n` on the modified script). + +- [ ] **Step 5: Commit.** + +``` +git add deploy/deploy-test-server.sh deploy/tests/test_deploy_artifacts.py +git commit -m "feat(deploy): wire nft marking + CAKE shaper into deploy script" +``` + +--- + +### Task 8: README documentation + +**Files:** +- Modify: `deploy/README.md` + +This is documentation only — no test asserts the README contents. Run an `sh -n` of the deploy script one more time after editing, just as a hygiene check (the README change can't affect it, but the test suite is fast). + +- [ ] **Step 1: Open `deploy/README.md` and locate the existing Performance tuning section.** + +The previous perf-baseline spec added a "Performance tuning" section (entries for CPU governor, CPU affinity, NIC tuning, and real-time scheduling opt-in). Find it. + +- [ ] **Step 2: Add a "Network shaping" subsection.** + +Add this subsection at the top of "Performance tuning" (before the existing entries; network-shaping covers the universal artifacts that ship by default, while the existing entries are escape hatches): + +```markdown +### Network shaping + +The deploy ships three things that affect player-experience network behaviour: + +1. **Per-flow marking.** `left4me-nft-mark.service` loads a small nftables + table (`inet left4me_mark`) that marks every UDP packet from uid `left4me` + with DSCP EF and `skb->priority` 6. srcds doesn't set these itself, so + without this rule its UDP is indistinguishable from any other flow. +2. **Sysctl baseline.** `99-left4me.conf` sets `udp_rmem_min=16384`, + `udp_wmem_min=16384`, `default_qdisc=fq_codel`, and + `tcp_congestion_control=bbr`. Reduces head-of-line blocking when bulk + TCP egress (backups, package fetches, web responses) coexists with + game UDP. +3. **CAKE egress shaping.** `left4me-cake.service` runs + `tc qdisc replace dev root cake bandwidth Xmbit internet + diffserv4 dual-dsthost` from `/etc/left4me/cake.env`. CAKE only shapes + if its declared bandwidth is **below** the real bottleneck, so set + `LEFT4ME_UPLINK_MBIT` to ≈95% of measured uplink: + + sudoedit /etc/left4me/cake.env + # set LEFT4ME_UPLINK_MBIT=480 (or whatever ~95% of your uplink is) + sudo systemctl restart left4me-cake.service + + `LEFT4ME_UPLINK_IFACE` is auto-detected from the IPv4 default route; + override only on hosts with multi-homed setups. + + At idle 500 Mbit with no competing egress, CAKE shapes nothing — that's + expected, not a bug. The win materialises when bulk traffic on the + same uplink would otherwise bufferbloat the link the players share. + +**Production hosts running `systemd-networkd`** should NOT use the +`left4me-cake.service` oneshot. Instead, configure the equivalent in the +matching `.network` file, which systemd-networkd reapplies across iface +lifecycle events: + + # /etc/systemd/network/.network + [CAKE] + Bandwidth=480M + OverheadKeyword=internet + PriorityQueueingPreset=diffserv4 + EgressHostIsolation=yes + +The nftables marking from (1) is qdisc-installer-agnostic and ships +unchanged on production. +``` + +- [ ] **Step 3: Append the three new escape hatches to the existing Performance tuning section.** + +Add after the existing escape-hatch entries (CPU governor / CPU affinity / NIC tuning / real-time scheduling): + +```markdown +### Additional opt-in network knobs + +- **Ingress shaping via IFB.** Egress CAKE alone does not protect srcds + receive against ingress saturation (large workshop downloads, package + fetches arriving at line rate). One-liner: + + sudo modprobe ifb && sudo ip link set ifb0 up + sudo tc qdisc add dev handle ffff: ingress + sudo tc filter add dev parent ffff: protocol ip u32 \ + match u32 0 0 action mirred egress redirect dev ifb0 + sudo tc qdisc add dev ifb0 root cake bandwidth Xmbit ingress \ + diffserv4 dual-srchost + + Worth flipping only when measurement shows ingress hurting receive. + +- **`net.core.busy_poll = 50` / `net.core.busy_read = 50`.** Reduces UDP + receive median latency by polling for incoming packets briefly at + syscall boundaries. Cost: measurable CPU per syscall under load. Worth + flipping if a host is dedicated to game serving and CPU headroom is + plentiful. + +- **`ethtool -K gro off`.** Some Source-engine ops disable + generic receive offload to avoid receive-side coalescing latency. + Hardware/driver dependent; document only. +``` + +- [ ] **Step 4: Re-run the full test suite.** + +``` +pytest deploy/tests/test_deploy_artifacts.py -v +``` +Expected: every test passes, including `test_deploy_script_shell_syntax`. + +- [ ] **Step 5: Commit.** + +``` +git add deploy/README.md +git commit -m "docs(deploy): document network-shaping defaults + opt-in network knobs" +``` + +--- + +## Final verification + +After all eight tasks land, run the whole suite once more and verify the new files are tracked: + +``` +pytest deploy/tests/test_deploy_artifacts.py -v +git status +git log --oneline -10 +``` + +Every test should pass. `git status` should be clean. The last 8 commits should match the eight tasks above. + +The new files in the tree: + +``` +deploy/files/etc/left4me/cake.env +deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft +deploy/files/usr/local/lib/systemd/system/left4me-cake.service +deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service +deploy/files/usr/local/libexec/left4me/left4me-apply-cake +``` + +Modified files: + +``` +deploy/files/etc/sysctl.d/99-left4me.conf +deploy/deploy-test-server.sh +deploy/README.md +deploy/tests/test_deploy_artifacts.py +```