spec(hardening): test plan executed on left4.me — results recorded

Ran the 11-test plan against left4me-server@1 (canary) and left4me-web
on left4.me / Debian 13 / systemd 257. Cleaned up all unit drop-ins;
kept the Test 9 sysctl (kernel.yama.ptrace_scope=2) per spec.

Outcomes:
- server@1 systemd-analyze: 7.5 EXPOSED → 1.3 OK
- left4me-web systemd-analyze: 8.7 EXPOSED → 4.1 OK
- All 8 attack vectors in Test 8 (D1.a-c, D2.a-c, D3, D5) blocked
- Test 6 (MemoryDenyWriteExecute) fails as predicted — Source engine
  i386 .so files have text relocations; exclude from final composition.
- Test 11 (24-48h soak) skipped per operator decision.

Two amendments to the spec's proposed composition required for the
refactor:
- SystemCallArchitectures=native x86 (not bare 'native') — srcds_linux
  is i386, the kernel kills every native-only call.
- PrivatePIDs=true added — ProtectProc=invisible alone cannot hide
  gunicorn from srcds because both run as uid 980; PrivatePIDs gives
  each instance its own PID namespace and closes D2.b.

Spec bugs surfaced and documented in the "Output" section: PID lookup
via pgrep (race vs. instance), Test 4/10 gdb-from-host doesn't
actually exercise the unit's SECCOMP filter, Test 8 D5 pgrep pattern
won't match. Tooling note corrected: scmp_sys_resolver is in
'seccomp' package, not 'libseccomp-dev'.

Next session: write docs/superpowers/plans/2026-MM-DD-hardening-refactor.md
against the proven composition; supersede the uid-split spec.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/superpowers/specs/2026-05-15-hardening-test-plan.md

@@ -1,8 +1,10 @@
 # left4me application hardening — test plan
 
-**Status:** living spec. Companion to `2026-05-15-hardening-threat-model.md`
-and `2026-05-15-hardening-defenses-survey.md`. **Executed in a follow-up
-session with shell access to `left4.me` (141.95.32.8).**
+**Status:** **tested 2026-05-15** on `left4.me` / `left4me.ovh.ckn.li`
+(Debian 13 trixie, systemd 257). See "Results" section near the
+bottom for the per-test outcomes. Companion to
+`2026-05-15-hardening-threat-model.md` and
+`2026-05-15-hardening-defenses-survey.md`.
 
 This document is intentionally self-contained: a session that lands cold
 with shell on `left4.me` can execute it end-to-end without re-reading
@@ -813,75 +815,255 @@ rm /tmp/syscall-log-*.txt
 
 ---
 
-## Results template
+## Results
 
-Append the executing session's findings here. One paragraph per test.
+**Status:** tested. Executed 2026-05-15 on left4.me / left4me.ovh.ckn.li
+(141.95.32.8). Debian 13 trixie, systemd 257.9.
+
+### Open-question answers captured before execution
+
+- **Gunicorn exposure:** only via nginx (binds `127.0.0.1:8000`,
+  confirmed via `ss -tlnp`). Nginx fronts on `0.0.0.0:{80,443}`.
+- **Admin auth:** password / cookie only — S2 (compromised operator
+  session) is a real, phishable path.
+- **Workshop curation:** player-driven / open — A3 (malicious
+  workshop content) realism is **high**.
+- **Test bench:** `left4.me` per operator instruction; `ckn@10.0.4.128`
+  not used this session.
+- **Baseline `kernel.yama.ptrace_scope`:** **0** (not the Debian-default
+  `1` the spec assumed) — Test 9 is a two-step tightening.
+- **AppArmor:** loaded with 106 profiles, only 7 in enforce mode (vendor
+  defaults); no left4me-specific profile. Out of scope for this session.
+
+### Baseline systemd-analyze
+
+- `left4me-server@1.service`: **7.5 EXPOSED 🙁**
+- `left4me-web.service`: **8.7 EXPOSED 🙁**
 
 ### Test 1 — PrivateUsers
-- Pass / fail: TBD
-- Notes:
+- **PASS.** Drop-in applied, unit active. Overlay mount succeeded via
+  the `+`-prefixed `ExecStartPre`. srcds_linux PID's `/proc/<pid>/ns/user`
+  was `4026532514` vs init's `4026531837` — separate user namespace
+  confirmed. Uid/Gid both 980 (identity map). No journal errors.
+- Spec nit: `pgrep -f 'srcds_linux.*left4dead2' | head -1` picks the
+  lowest-PID instance, which can belong to a different `@N` — use
+  port (e.g. `27016` for @1) or `systemctl show -p MainPID
+  --value left4me-server@1` for instance-specific PID lookup.
 
 ### Test 2 — TemporaryFileSystem + binds
-- Pass / fail: TBD
-- Notes:
+- **PASS.** From inside the namespace: `left4me.db` and `web.env`
+  invisible ("No such file or directory"); `/opt` empty; only
+  `installation`, `overlays`, `runtime` visible under
+  `/var/lib/left4me/`; only `host.env` under `/etc/left4me/`;
+  `getent hosts steamcommunity.com` resolves. No SECCOMP / permission
+  errors in journal. One benign side effect: srcds probes
+  `/var/lib/left4me/.steam/sdk32/steamclient.so` (now hidden) — falls
+  back to the local steamclient.so without issue.
+- **Today's D1 gap closure:** baseline file modes confirm srcds
+  (uid 980) currently CAN read `web.env` (mode 0640 root:left4me)
+  and `left4me.db` (mode 0644 left4me:left4me). Test 2 makes both
+  invisible. D1 closes here.
 
 ### Test 3 — SystemCallLog discovery
-- Pass / fail: TBD
-- Syscalls observed under load (if any from @debug/@mount/@privileged):
-- Notes:
+- **PASS (data captured) — but the spec's filter shape blocks 32-bit srcds.**
+  Two findings:
+  1. **No srcds calls in `@privileged`/`@debug`/`@mount`/`@raw-io`** —
+     the only `sig=0` (log) line was systemd-executor calling `capset`
+     during exec setup, before the unit's process started. Filter
+     shape from Test 4 is safe.
+  2. **`SystemCallArchitectures=native` is incompatible with srcds_linux**
+     — the binary is `ELF 32-bit LSB executable, Intel i386`
+     (`file /var/lib/left4me/runtime/1/merged/srcds_linux`). With
+     `native=AUDIT_ARCH_X86_64`, every i386 syscall (first one is
+     `brk`/45) is killed with SIGSYS; srcds_run's restart-on-crash
+     loop respawns every 10s ("Bad system call → Server restart in
+     10 seconds"). **Required fix: `SystemCallArchitectures=native x86`.**
+     Applied to Test 4 and Test 7 below.
 
-### Test 4 — SystemCallFilter enforcement
-- Pass / fail: TBD
-- If filter had to be relaxed, which group:
-- Notes:
+### Test 4 — SystemCallFilter enforcement (with x86 added)
+- **PASS.** With `SystemCallArchitectures=native x86` and the spec's
+  deny groups (`~@debug @mount @raw-io @reboot @swap @cpu-emulation
+  @obsolete @privileged`), the resulting allow set is 369 syscalls.
+  `ptrace`, `process_vm_readv`/`writev`/`kcmp` are excluded from
+  `@debug`. Server stable for ≥90 s of observation, zero SECCOMP
+  audit lines, srcds_linux PID unchanged (no respawn).
+- **Spec verification flaw:** `sudo nsenter --target $SRCDS --mount --
+  gdb --batch -p $GUNICORN` runs gdb in the host's process context
+  (gdb itself has no SECCOMP filter) and only enters srcds's mount NS,
+  not its userns or PID NS — gdb attaches successfully, but **this
+  does not prove srcds can't ptrace**. To actually test, run inside
+  the unit's full namespace set or via `systemd-run` with the same
+  hardening directives. Filter correctness was verified by inspecting
+  the compiled `SystemCallFilter` from `systemctl show -p
+  SystemCallFilter` (ptrace absent from allow list).
 
 ### Test 5 — ProcSubset + ProtectProc
-- Pass / fail: TBD
-- Notes:
+- **PARTIAL.** `/proc` mount inside the namespace shows
+  `hidepid=invisible,subset=pid` (both directives applied).
+  Non-PID entries (`/proc/kallsyms`, `/proc/cmdline`) are
+  **invisible** as the left4me uid. **However:** `gunicorn`'s
+  `/proc/<pid>/environ` is still readable from srcds because **both
+  processes share uid 980** — hidepid=invisible hides foreign-uid
+  PIDs but doesn't help against same-uid. This matches the threat
+  model's same-uid finding exactly.
+- **Fix for Test 7:** add `PrivatePIDs=true` (systemd 257 supports it;
+  service-only, fine for `Type=simple`). With PrivatePIDs, srcds_run
+  becomes PID 1 in a private PID NS and gunicorn isn't visible at
+  all in srcds's `/proc`.
 
 ### Test 6 — MemoryDenyWriteExecute
-- Pass / fail: TBD (likely fail; document the failure mode)
-- Notes:
+- **FAIL as predicted.** First srcds_linux spawn under MDW=true logs
+  `Failed to open dedicated_srv.so (bin/libtier0_srv.so: cannot make
+  segment writable for relocation: Permission denied)`. Source engine's
+  32-bit `.so` files have text relocations (TEXTREL — common in
+  pre-2010 binaries); the dynamic linker needs to remap pages
+  PROT_READ|PROT_WRITE|PROT_EXEC during relocation. MDW returns EPERM
+  (silently, not via SIGSYS — no audit lines), dlopen aborts, srcds_run
+  enters a 10-second respawn loop. **Excluded from Test 7's
+  composition.** Not fixable without rebuilding srcds without textrels
+  (i.e., never — Valve closed-source binary).
 
 ### Test 7 — Full composition
-- Pass / fail: TBD
-- systemd-analyze score before/after:
-- Notes:
+- **PASS.** Score **7.5 EXPOSED → 1.3 OK 🙂** for `left4me-server@1`
+  (a 6.2-point drop). Composition: all directives from the spec's
+  Test 7 minus `MemoryDenyWriteExecute=true`, plus
+  `SystemCallArchitectures=native x86` (the i386 fix from Test 3) and
+  `PrivatePIDs=true` (closes the same-uid /proc gap from Test 5).
+- Smoke matrix: S1 active ✓, S2 srcds_linux running ✓, S8 server@2
+  unaffected ✓, web responds with HTTP/1.1 302 ✓. One SECCOMP kill
+  at startup (i386 syscall 26 = `ptrace`, comm=srcds_linux): this is
+  Breakpad's crash-reporter trying to attach to its own process for
+  minidump generation — correctly blocked, srcds carries on. No further
+  audit lines over 150 s of observation; srcds_linux PID stable for
+  the full window.
+- Application-layer side effect: gunicorn's `rcon` service logs
+  `Connection refused` against server@1 — root cause is **stale RCON
+  port cached in the web app** from before the test-cycle restarts
+  (each srcds spawn picks a new ephemeral RCON port); not caused by
+  hardening. Repro'd before web hardening was applied. Tracking for
+  the web app, not for this refactor.
 
-### Test 8 — Attack verification
-- Pass / fail: TBD
-- Per-vector results (D1.a, D1.b, ..., D5):
+### Test 8 — Attack verification (all from inside srcds NS as uid 980)
+- **PASS — all 8 vectors blocked.**
+  - **D1.a** (read `/var/lib/left4me/left4me.db`): `No such file or directory`
+  - **D1.b** (read `/etc/left4me/web.env`): `No such file or directory`
+  - **D1.c** (`ls /opt`): empty
+  - **D2.a** (ptrace): defense at three layers — SECCOMP filter denies
+    ptrace, PrivateUsers blocks cross-userns capability check,
+    PrivatePIDs hides foreign PIDs from /proc. Filter content verified
+    via `systemctl show -p SystemCallFilter`.
+  - **D2.b** (read `/proc/<gunicorn>/environ`): `No such file or directory`
+    (gunicorn's PID isn't visible at all under PrivatePIDs)
+  - **D2.c** (read `/proc/<gunicorn>/mem`): `No such file or directory`
+  - **D3** (`sudo -n`): `sudo: effective uid is not 0, is /usr/bin/sudo
+    on a file system with the 'nosuid' option set` — NoNewPrivileges
+    blocks the setuid bit on sudo, so srcds can't escalate via the
+    web's helpers.
+  - **D5** (read `/proc/<srcds@2>/environ`): `No such file or directory`
+    — server@2 isn't in srcds@1's PID NS. Note: this protection between
+    instances will hold for any pair once the composition lands in
+    ckn-bw and applies to every `left4me-server@N`. Today, even with
+    server@2 unhardened, the asymmetric NS isolates them.
+- Spec nit at D5: `pgrep -f 'srcds_linux.*\@2'` won't work because the
+  `@N` is in the systemd unit name, not the process cmdline. Use
+  `systemctl show -p MainPID --value left4me-server@2` or grep the
+  game port (27021 for @2).
 
-### Test 9 — Yama ptrace_scope=2
-- Applied: TBD
-- Operator workflow impact noted:
+### Test 9 — Yama `ptrace_scope=2`
+- **APPLIED.** Written to `/etc/sysctl.d/99-left4me-ptrace.conf`,
+  persists across reboot. `sudo /sbin/sysctl --system` confirms
+  `kernel.yama.ptrace_scope = 2`. `sudo -u left4me gdb --batch -p
+  <gunicorn>` → `ptrace: Operation not permitted`. Root gdb still
+  works (admin debugging unimpeded).
+- **Baseline correction:** observed starting value was `0`, not the
+  `1` the spec assumed. Two-step tightening (0 → 2).
+- Operator-workflow impact: non-root processes that previously could
+  ptrace their own children can no longer do so. Coredump-on-crash
+  via the kernel core-pattern path is unaffected. The unit-side
+  effect for srcds is purely additive on top of Test 7's defenses.
 
 ### Test 10 — Web hardening
-- Pass / fail: TBD
-- Sudo path verified working:
-- systemd-analyze score before/after:
+- **PASS.** Score **8.7 EXPOSED → 4.1 OK 🙂** for `left4me-web` (a
+  4.6-point drop, ceiling capped by the sudo-compat exclusions:
+  no `NoNewPrivileges`, no `PrivateUsers`, no `CapabilityBoundingSet=`,
+  no `~@privileged` in the syscall deny list).
+- Web functional checks: HTTP/1.1 302 returned by curl; gunicorn
+  workers up; **sudo path works** — `sudo -n -l` lists the configured
+  helpers (`left4me-systemctl *`, `left4me-journalctl *`, `left4me-overlay
+  mount|umount *`, `left4me-script-sandbox`), and a direct
+  invocation of `sudo -n /usr/local/libexec/left4me/left4me-systemctl
+  is-active left4me-server@2` ran the helper successfully (helper
+  emitted its usage message — i.e. sudo+setuid succeeded). Zero SECCOMP
+  audit lines over 60 s of observation. Drop-in cleaned up; web
+  reverted to baseline.
+- Pre-existing rcon `Connection refused` errors in the web journal
+  predate Test 10's apply — same stale-RCON-port issue noted in Test 7.
 
 ### Test 11 — Soak
-- Duration:
-- Issues observed:
+- **SKIPPED** per operator decision. Rely on functional verification
+  during Tests 7/8/10. Recommend the hardening-refactor implementation
+  plan re-evaluate whether to run a soak after the ckn-bw rollout
+  reaches a non-prod canary first.
+
+### End-of-session state
+
+- All test drop-ins removed; drop-in dirs removed.
+- `kernel.yama.ptrace_scope=2` persists at `/etc/sysctl.d/99-left4me-ptrace.conf`.
+- `gdb` + `seccomp` (provides `scmp_sys_resolver`) + `libseccomp-dev`
+  left installed (operator can `apt remove` if desired).
+- `left4me-server@1`, `@2`, `left4me-web` all `active`; back to
+  baseline (verified: `srcds@1` `/proc/<pid>/ns/user` matches init).
+- /tmp baseline + after files retained on host for reference by the
+  follow-up implementation plan.
 
 ---
 
 ## Output of this test plan
 
 When all tests complete:
-1. Mark this document with **status: tested** and record the dates.
-2. Open a new implementation plan
+1. [x] Mark this document with **status: tested** and record the dates.
+2. [ ] Open a new implementation plan
    (`docs/superpowers/plans/2026-MM-DD-hardening-refactor.md`) that
    commits the proven composition to the ckn-bw reactor + reference
-   units + test suite.
-3. Decide on the deferred questions:
-   - 3-user uid split — necessary or covered by hardening?
-   - AppArmor profile follow-up — pursue or close?
-   - `MemoryDenyWriteExecute=true` — include if Test 6 passed?
-   - `SocketBindAllow=` — add to lock the gameserver port range?
-4. Mark `2026-05-15-user-uid-split-design.md` as superseded or closed
-   per the answer to the previous bullet.
+   units + test suite. **Proven composition is in Test 7's drop-in
+   above with two amendments: `SystemCallArchitectures=native x86`
+   (not `native`) and `PrivatePIDs=true`.**
+3. [ ] Decide on the deferred questions:
+   - 3-user uid split — same-uid /proc gap was closed by
+     `PrivatePIDs=true`. With that + `PrivateUsers=true` in the
+     composition, the residual same-uid attack surface is
+     application-level (DB ACLs, web.env). Recommend **closing the
+     uid-split spec as superseded** unless application-level concerns
+     surface later.
+   - AppArmor profile follow-up — host has 7 vendor profiles in
+     enforce; no left4me-specific profile. Defenses survey lists it as
+     deferred; revisit after the refactor lands if directive-only
+     hardening leaves residual concerns.
+   - `MemoryDenyWriteExecute=true` — **NO.** Source engine 32-bit
+     `.so` files have text relocations; MDW prevents the relocation
+     `mprotect` → dlopen fails → respawn loop. Document permanently.
+   - `SocketBindAllow=` — not tested. Game UDP ports are 27000-27999
+     per `LEFT4ME_PORT_RANGE_*` in instance env. Worth adding to lock
+     the bindable port range; not in the test plan, defer to the
+     refactor.
+4. [ ] Mark `2026-05-15-user-uid-split-design.md` as superseded per #3.
+
+### Spec bugs surfaced during execution (fix in refactor or follow-up commit)
+
+- **Test 1 / Test 8 PID lookup**: `pgrep -f 'srcds_linux.*left4dead2'
+  | head -1` picks the lowest-PID instance — likely the wrong one if
+  another `@N` started earlier. Use port (e.g. `27016` for @1) or
+  `systemctl show -p MainPID --value left4me-server@N`.
+- **Test 4 / Test 10 ptrace verification**: `sudo nsenter --target
+  $PID --mount -- gdb -p $TARGET` only enters mount NS and runs gdb
+  with the host's (root, no SECCOMP) context — gdb attaches
+  regardless of the unit's filter. To meaningfully test, enter
+  user+pid+net+ipc namespaces with `--setuid`/`--setgid` OR spawn a
+  probe via `systemd-run` with the same hardening directives.
+- **Test 8 D5**: `pgrep -f 'srcds_linux.*\@2'` won't match because `@N`
+  comes from the systemd unit name, not argv. Use port or `MainPID`.
+- **Test 3 baseline**: spec says ptrace_scope default is `1`; observed
+  `0` on Debian 13. Update the "Expect" line.
 
 ## Pointers
 
@@ -891,8 +1073,8 @@ When all tests complete:
 - Reference units: `deploy/files/usr/local/lib/systemd/system/`
 - Tools needed on `left4.me`:
   - `systemd-analyze` (in `systemd` package, already installed)
-  - `scmp_sys_resolver` (in `libseccomp-dev`; install on demand for
-    Test 3/4 if filters need analysis)
-  - `gdb` (for ptrace tests; install on demand)
+  - `scmp_sys_resolver` (in **`seccomp`** package — NOT `libseccomp-dev`
+    on Debian 13; `apt install seccomp`)
+  - `gdb` (for ptrace tests; `apt install gdb`)
   - `nsenter` (in `util-linux`, already installed)
   - `findmnt`, `pgrep`, standard userspace