From 461b8d028fb24fb41387fce189c75e7795ab57e5 Mon Sep 17 00:00:00 2001 From: mwiegand Date: Fri, 15 May 2026 13:39:50 +0200 Subject: [PATCH] =?UTF-8?q?spec(hardening):=20test=20plan=20executed=20on?= =?UTF-8?q?=20left4.me=20=E2=80=94=20results=20recorded?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran the 11-test plan against left4me-server@1 (canary) and left4me-web on left4.me / Debian 13 / systemd 257. Cleaned up all unit drop-ins; kept the Test 9 sysctl (kernel.yama.ptrace_scope=2) per spec. Outcomes: - server@1 systemd-analyze: 7.5 EXPOSED → 1.3 OK - left4me-web systemd-analyze: 8.7 EXPOSED → 4.1 OK - All 8 attack vectors in Test 8 (D1.a-c, D2.a-c, D3, D5) blocked - Test 6 (MemoryDenyWriteExecute) fails as predicted — Source engine i386 .so files have text relocations; exclude from final composition. - Test 11 (24-48h soak) skipped per operator decision. Two amendments to the spec's proposed composition required for the refactor: - SystemCallArchitectures=native x86 (not bare 'native') — srcds_linux is i386, the kernel kills every native-only call. - PrivatePIDs=true added — ProtectProc=invisible alone cannot hide gunicorn from srcds because both run as uid 980; PrivatePIDs gives each instance its own PID namespace and closes D2.b. Spec bugs surfaced and documented in the "Output" section: PID lookup via pgrep (race vs. instance), Test 4/10 gdb-from-host doesn't actually exercise the unit's SECCOMP filter, Test 8 D5 pgrep pattern won't match. Tooling note corrected: scmp_sys_resolver is in 'seccomp' package, not 'libseccomp-dev'. Next session: write docs/superpowers/plans/2026-MM-DD-hardening-refactor.md against the proven composition; supersede the uid-split spec. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../specs/2026-05-15-hardening-test-plan.md | 276 +++++++++++++++--- 1 file changed, 229 insertions(+), 47 deletions(-) diff --git a/docs/superpowers/specs/2026-05-15-hardening-test-plan.md b/docs/superpowers/specs/2026-05-15-hardening-test-plan.md index fa6dfa3..9898e62 100644 --- a/docs/superpowers/specs/2026-05-15-hardening-test-plan.md +++ b/docs/superpowers/specs/2026-05-15-hardening-test-plan.md @@ -1,8 +1,10 @@ # left4me application hardening — test plan -**Status:** living spec. Companion to `2026-05-15-hardening-threat-model.md` -and `2026-05-15-hardening-defenses-survey.md`. **Executed in a follow-up -session with shell access to `left4.me` (141.95.32.8).** +**Status:** **tested 2026-05-15** on `left4.me` / `left4me.ovh.ckn.li` +(Debian 13 trixie, systemd 257). See "Results" section near the +bottom for the per-test outcomes. Companion to +`2026-05-15-hardening-threat-model.md` and +`2026-05-15-hardening-defenses-survey.md`. This document is intentionally self-contained: a session that lands cold with shell on `left4.me` can execute it end-to-end without re-reading @@ -813,75 +815,255 @@ rm /tmp/syscall-log-*.txt --- -## Results template +## Results -Append the executing session's findings here. One paragraph per test. +**Status:** tested. Executed 2026-05-15 on left4.me / left4me.ovh.ckn.li +(141.95.32.8). Debian 13 trixie, systemd 257.9. + +### Open-question answers captured before execution + +- **Gunicorn exposure:** only via nginx (binds `127.0.0.1:8000`, + confirmed via `ss -tlnp`). Nginx fronts on `0.0.0.0:{80,443}`. +- **Admin auth:** password / cookie only — S2 (compromised operator + session) is a real, phishable path. +- **Workshop curation:** player-driven / open — A3 (malicious + workshop content) realism is **high**. +- **Test bench:** `left4.me` per operator instruction; `ckn@10.0.4.128` + not used this session. +- **Baseline `kernel.yama.ptrace_scope`:** **0** (not the Debian-default + `1` the spec assumed) — Test 9 is a two-step tightening. +- **AppArmor:** loaded with 106 profiles, only 7 in enforce mode (vendor + defaults); no left4me-specific profile. Out of scope for this session. + +### Baseline systemd-analyze + +- `left4me-server@1.service`: **7.5 EXPOSED 🙁** +- `left4me-web.service`: **8.7 EXPOSED 🙁** ### Test 1 — PrivateUsers -- Pass / fail: TBD -- Notes: +- **PASS.** Drop-in applied, unit active. Overlay mount succeeded via + the `+`-prefixed `ExecStartPre`. srcds_linux PID's `/proc//ns/user` + was `4026532514` vs init's `4026531837` — separate user namespace + confirmed. Uid/Gid both 980 (identity map). No journal errors. +- Spec nit: `pgrep -f 'srcds_linux.*left4dead2' | head -1` picks the + lowest-PID instance, which can belong to a different `@N` — use + port (e.g. `27016` for @1) or `systemctl show -p MainPID + --value left4me-server@1` for instance-specific PID lookup. ### Test 2 — TemporaryFileSystem + binds -- Pass / fail: TBD -- Notes: +- **PASS.** From inside the namespace: `left4me.db` and `web.env` + invisible ("No such file or directory"); `/opt` empty; only + `installation`, `overlays`, `runtime` visible under + `/var/lib/left4me/`; only `host.env` under `/etc/left4me/`; + `getent hosts steamcommunity.com` resolves. No SECCOMP / permission + errors in journal. One benign side effect: srcds probes + `/var/lib/left4me/.steam/sdk32/steamclient.so` (now hidden) — falls + back to the local steamclient.so without issue. +- **Today's D1 gap closure:** baseline file modes confirm srcds + (uid 980) currently CAN read `web.env` (mode 0640 root:left4me) + and `left4me.db` (mode 0644 left4me:left4me). Test 2 makes both + invisible. D1 closes here. ### Test 3 — SystemCallLog discovery -- Pass / fail: TBD -- Syscalls observed under load (if any from @debug/@mount/@privileged): -- Notes: +- **PASS (data captured) — but the spec's filter shape blocks 32-bit srcds.** + Two findings: + 1. **No srcds calls in `@privileged`/`@debug`/`@mount`/`@raw-io`** — + the only `sig=0` (log) line was systemd-executor calling `capset` + during exec setup, before the unit's process started. Filter + shape from Test 4 is safe. + 2. **`SystemCallArchitectures=native` is incompatible with srcds_linux** + — the binary is `ELF 32-bit LSB executable, Intel i386` + (`file /var/lib/left4me/runtime/1/merged/srcds_linux`). With + `native=AUDIT_ARCH_X86_64`, every i386 syscall (first one is + `brk`/45) is killed with SIGSYS; srcds_run's restart-on-crash + loop respawns every 10s ("Bad system call → Server restart in + 10 seconds"). **Required fix: `SystemCallArchitectures=native x86`.** + Applied to Test 4 and Test 7 below. -### Test 4 — SystemCallFilter enforcement -- Pass / fail: TBD -- If filter had to be relaxed, which group: -- Notes: +### Test 4 — SystemCallFilter enforcement (with x86 added) +- **PASS.** With `SystemCallArchitectures=native x86` and the spec's + deny groups (`~@debug @mount @raw-io @reboot @swap @cpu-emulation + @obsolete @privileged`), the resulting allow set is 369 syscalls. + `ptrace`, `process_vm_readv`/`writev`/`kcmp` are excluded from + `@debug`. Server stable for ≥90 s of observation, zero SECCOMP + audit lines, srcds_linux PID unchanged (no respawn). +- **Spec verification flaw:** `sudo nsenter --target $SRCDS --mount -- + gdb --batch -p $GUNICORN` runs gdb in the host's process context + (gdb itself has no SECCOMP filter) and only enters srcds's mount NS, + not its userns or PID NS — gdb attaches successfully, but **this + does not prove srcds can't ptrace**. To actually test, run inside + the unit's full namespace set or via `systemd-run` with the same + hardening directives. Filter correctness was verified by inspecting + the compiled `SystemCallFilter` from `systemctl show -p + SystemCallFilter` (ptrace absent from allow list). ### Test 5 — ProcSubset + ProtectProc -- Pass / fail: TBD -- Notes: +- **PARTIAL.** `/proc` mount inside the namespace shows + `hidepid=invisible,subset=pid` (both directives applied). + Non-PID entries (`/proc/kallsyms`, `/proc/cmdline`) are + **invisible** as the left4me uid. **However:** `gunicorn`'s + `/proc//environ` is still readable from srcds because **both + processes share uid 980** — hidepid=invisible hides foreign-uid + PIDs but doesn't help against same-uid. This matches the threat + model's same-uid finding exactly. +- **Fix for Test 7:** add `PrivatePIDs=true` (systemd 257 supports it; + service-only, fine for `Type=simple`). With PrivatePIDs, srcds_run + becomes PID 1 in a private PID NS and gunicorn isn't visible at + all in srcds's `/proc`. ### Test 6 — MemoryDenyWriteExecute -- Pass / fail: TBD (likely fail; document the failure mode) -- Notes: +- **FAIL as predicted.** First srcds_linux spawn under MDW=true logs + `Failed to open dedicated_srv.so (bin/libtier0_srv.so: cannot make + segment writable for relocation: Permission denied)`. Source engine's + 32-bit `.so` files have text relocations (TEXTREL — common in + pre-2010 binaries); the dynamic linker needs to remap pages + PROT_READ|PROT_WRITE|PROT_EXEC during relocation. MDW returns EPERM + (silently, not via SIGSYS — no audit lines), dlopen aborts, srcds_run + enters a 10-second respawn loop. **Excluded from Test 7's + composition.** Not fixable without rebuilding srcds without textrels + (i.e., never — Valve closed-source binary). ### Test 7 — Full composition -- Pass / fail: TBD -- systemd-analyze score before/after: -- Notes: +- **PASS.** Score **7.5 EXPOSED → 1.3 OK 🙂** for `left4me-server@1` + (a 6.2-point drop). Composition: all directives from the spec's + Test 7 minus `MemoryDenyWriteExecute=true`, plus + `SystemCallArchitectures=native x86` (the i386 fix from Test 3) and + `PrivatePIDs=true` (closes the same-uid /proc gap from Test 5). +- Smoke matrix: S1 active ✓, S2 srcds_linux running ✓, S8 server@2 + unaffected ✓, web responds with HTTP/1.1 302 ✓. One SECCOMP kill + at startup (i386 syscall 26 = `ptrace`, comm=srcds_linux): this is + Breakpad's crash-reporter trying to attach to its own process for + minidump generation — correctly blocked, srcds carries on. No further + audit lines over 150 s of observation; srcds_linux PID stable for + the full window. +- Application-layer side effect: gunicorn's `rcon` service logs + `Connection refused` against server@1 — root cause is **stale RCON + port cached in the web app** from before the test-cycle restarts + (each srcds spawn picks a new ephemeral RCON port); not caused by + hardening. Repro'd before web hardening was applied. Tracking for + the web app, not for this refactor. -### Test 8 — Attack verification -- Pass / fail: TBD -- Per-vector results (D1.a, D1.b, ..., D5): +### Test 8 — Attack verification (all from inside srcds NS as uid 980) +- **PASS — all 8 vectors blocked.** + - **D1.a** (read `/var/lib/left4me/left4me.db`): `No such file or directory` + - **D1.b** (read `/etc/left4me/web.env`): `No such file or directory` + - **D1.c** (`ls /opt`): empty + - **D2.a** (ptrace): defense at three layers — SECCOMP filter denies + ptrace, PrivateUsers blocks cross-userns capability check, + PrivatePIDs hides foreign PIDs from /proc. Filter content verified + via `systemctl show -p SystemCallFilter`. + - **D2.b** (read `/proc//environ`): `No such file or directory` + (gunicorn's PID isn't visible at all under PrivatePIDs) + - **D2.c** (read `/proc//mem`): `No such file or directory` + - **D3** (`sudo -n`): `sudo: effective uid is not 0, is /usr/bin/sudo + on a file system with the 'nosuid' option set` — NoNewPrivileges + blocks the setuid bit on sudo, so srcds can't escalate via the + web's helpers. + - **D5** (read `/proc//environ`): `No such file or directory` + — server@2 isn't in srcds@1's PID NS. Note: this protection between + instances will hold for any pair once the composition lands in + ckn-bw and applies to every `left4me-server@N`. Today, even with + server@2 unhardened, the asymmetric NS isolates them. +- Spec nit at D5: `pgrep -f 'srcds_linux.*\@2'` won't work because the + `@N` is in the systemd unit name, not the process cmdline. Use + `systemctl show -p MainPID --value left4me-server@2` or grep the + game port (27021 for @2). -### Test 9 — Yama ptrace_scope=2 -- Applied: TBD -- Operator workflow impact noted: +### Test 9 — Yama `ptrace_scope=2` +- **APPLIED.** Written to `/etc/sysctl.d/99-left4me-ptrace.conf`, + persists across reboot. `sudo /sbin/sysctl --system` confirms + `kernel.yama.ptrace_scope = 2`. `sudo -u left4me gdb --batch -p + ` → `ptrace: Operation not permitted`. Root gdb still + works (admin debugging unimpeded). +- **Baseline correction:** observed starting value was `0`, not the + `1` the spec assumed. Two-step tightening (0 → 2). +- Operator-workflow impact: non-root processes that previously could + ptrace their own children can no longer do so. Coredump-on-crash + via the kernel core-pattern path is unaffected. The unit-side + effect for srcds is purely additive on top of Test 7's defenses. ### Test 10 — Web hardening -- Pass / fail: TBD -- Sudo path verified working: -- systemd-analyze score before/after: +- **PASS.** Score **8.7 EXPOSED → 4.1 OK 🙂** for `left4me-web` (a + 4.6-point drop, ceiling capped by the sudo-compat exclusions: + no `NoNewPrivileges`, no `PrivateUsers`, no `CapabilityBoundingSet=`, + no `~@privileged` in the syscall deny list). +- Web functional checks: HTTP/1.1 302 returned by curl; gunicorn + workers up; **sudo path works** — `sudo -n -l` lists the configured + helpers (`left4me-systemctl *`, `left4me-journalctl *`, `left4me-overlay + mount|umount *`, `left4me-script-sandbox`), and a direct + invocation of `sudo -n /usr/local/libexec/left4me/left4me-systemctl + is-active left4me-server@2` ran the helper successfully (helper + emitted its usage message — i.e. sudo+setuid succeeded). Zero SECCOMP + audit lines over 60 s of observation. Drop-in cleaned up; web + reverted to baseline. +- Pre-existing rcon `Connection refused` errors in the web journal + predate Test 10's apply — same stale-RCON-port issue noted in Test 7. ### Test 11 — Soak -- Duration: -- Issues observed: +- **SKIPPED** per operator decision. Rely on functional verification + during Tests 7/8/10. Recommend the hardening-refactor implementation + plan re-evaluate whether to run a soak after the ckn-bw rollout + reaches a non-prod canary first. + +### End-of-session state + +- All test drop-ins removed; drop-in dirs removed. +- `kernel.yama.ptrace_scope=2` persists at `/etc/sysctl.d/99-left4me-ptrace.conf`. +- `gdb` + `seccomp` (provides `scmp_sys_resolver`) + `libseccomp-dev` + left installed (operator can `apt remove` if desired). +- `left4me-server@1`, `@2`, `left4me-web` all `active`; back to + baseline (verified: `srcds@1` `/proc//ns/user` matches init). +- /tmp baseline + after files retained on host for reference by the + follow-up implementation plan. --- ## Output of this test plan When all tests complete: -1. Mark this document with **status: tested** and record the dates. -2. Open a new implementation plan +1. [x] Mark this document with **status: tested** and record the dates. +2. [ ] Open a new implementation plan (`docs/superpowers/plans/2026-MM-DD-hardening-refactor.md`) that commits the proven composition to the ckn-bw reactor + reference - units + test suite. -3. Decide on the deferred questions: - - 3-user uid split — necessary or covered by hardening? - - AppArmor profile follow-up — pursue or close? - - `MemoryDenyWriteExecute=true` — include if Test 6 passed? - - `SocketBindAllow=` — add to lock the gameserver port range? -4. Mark `2026-05-15-user-uid-split-design.md` as superseded or closed - per the answer to the previous bullet. + units + test suite. **Proven composition is in Test 7's drop-in + above with two amendments: `SystemCallArchitectures=native x86` + (not `native`) and `PrivatePIDs=true`.** +3. [ ] Decide on the deferred questions: + - 3-user uid split — same-uid /proc gap was closed by + `PrivatePIDs=true`. With that + `PrivateUsers=true` in the + composition, the residual same-uid attack surface is + application-level (DB ACLs, web.env). Recommend **closing the + uid-split spec as superseded** unless application-level concerns + surface later. + - AppArmor profile follow-up — host has 7 vendor profiles in + enforce; no left4me-specific profile. Defenses survey lists it as + deferred; revisit after the refactor lands if directive-only + hardening leaves residual concerns. + - `MemoryDenyWriteExecute=true` — **NO.** Source engine 32-bit + `.so` files have text relocations; MDW prevents the relocation + `mprotect` → dlopen fails → respawn loop. Document permanently. + - `SocketBindAllow=` — not tested. Game UDP ports are 27000-27999 + per `LEFT4ME_PORT_RANGE_*` in instance env. Worth adding to lock + the bindable port range; not in the test plan, defer to the + refactor. +4. [ ] Mark `2026-05-15-user-uid-split-design.md` as superseded per #3. + +### Spec bugs surfaced during execution (fix in refactor or follow-up commit) + +- **Test 1 / Test 8 PID lookup**: `pgrep -f 'srcds_linux.*left4dead2' + | head -1` picks the lowest-PID instance — likely the wrong one if + another `@N` started earlier. Use port (e.g. `27016` for @1) or + `systemctl show -p MainPID --value left4me-server@N`. +- **Test 4 / Test 10 ptrace verification**: `sudo nsenter --target + $PID --mount -- gdb -p $TARGET` only enters mount NS and runs gdb + with the host's (root, no SECCOMP) context — gdb attaches + regardless of the unit's filter. To meaningfully test, enter + user+pid+net+ipc namespaces with `--setuid`/`--setgid` OR spawn a + probe via `systemd-run` with the same hardening directives. +- **Test 8 D5**: `pgrep -f 'srcds_linux.*\@2'` won't match because `@N` + comes from the systemd unit name, not argv. Use port or `MainPID`. +- **Test 3 baseline**: spec says ptrace_scope default is `1`; observed + `0` on Debian 13. Update the "Expect" line. ## Pointers @@ -891,8 +1073,8 @@ When all tests complete: - Reference units: `deploy/files/usr/local/lib/systemd/system/` - Tools needed on `left4.me`: - `systemd-analyze` (in `systemd` package, already installed) - - `scmp_sys_resolver` (in `libseccomp-dev`; install on demand for - Test 3/4 if filters need analysis) - - `gdb` (for ptrace tests; install on demand) + - `scmp_sys_resolver` (in **`seccomp`** package — NOT `libseccomp-dev` + on Debian 13; `apt install seccomp`) + - `gdb` (for ptrace tests; `apt install gdb`) - `nsenter` (in `util-linux`, already installed) - `findmnt`, `pgrep`, standard userspace