left4me/docs/superpowers/specs/2026-05-15-hardening-test-plan.md

left4me application hardening — test plan

Status: tested 2026-05-15 on left4.me / left4me.ovh.ckn.li (Debian 13 trixie, systemd 257). See "Results" section near the bottom for the per-test outcomes. Companion to 2026-05-15-hardening-threat-model.md and 2026-05-15-hardening-defenses-survey.md.

This document is intentionally self-contained: a session that lands cold with shell on left4.me can execute it end-to-end without re-reading the threat model or survey. Decisions made in this plan are based on the candidate composition in the defenses survey (Section 5).

Test architecture

Where we test

• Host: left4.me / ovh.left4me (141.95.32.8). Production host; no separate test bench. (Reference: memory entry feedback_test_server_hangs.md mentions a separate test server at ckn@10.0.4.128; verify whether that host is suitable for this work before using prod.)
• Canary unit: left4me-server@1.service. Use this as the test instance. Leave left4me-server@2.service running baseline so at least one server stays up if the canary breaks.
• Web unit: left4me-web.service is shared. Test web-side hardening only after server@ tests prove the composition; web is more disruptive to roll back.

Operating constraints

• System units only. No systemctl --user, no lingering, no per-user systemd instance. All units under /etc/systemd/system/ or /usr/local/lib/systemd/system/. Drop-ins go to /etc/systemd/system/<unit>.d/.
• Drop-in style. Tests apply via /etc/systemd/system/left4me-server@1.service.d/test-NN-<name>.conf (note: @1 for instance-specific). This leaves the template unmodified — other instances unaffected. systemctl daemon-reload picks up drop-ins; systemctl restart left4me-server@1 applies.
• Cleanup required. Each test removes its drop-in before the next starts. Baseline must be restorable at any point.
• Recording. Each test produces a one-paragraph result in this document's "Results" section at the bottom. Append, don't replace.

Failure modes to watch for

• SECCOMP audit: journalctl -k --since '1 minute ago' | grep -i seccomp shows type=1326 lines. Each is a syscall denied; the syscall number identifies the call. Use scmp_sys_resolver to translate.
• Unit start failure: systemctl is-active left4me-server@1 → inactive or failed.
• srcds crash mid-game: journalctl -u left4me-server@1 -f shows unexpected exit; systemctl show left4me-server@1 -p Result is not success.
• sourcemod/metamod plugin failures: in-game sm plugins list or RCON sm plugins list shows plugins as failed-to-load.
• Permission denied where unexpected: journalctl -u left4me-server@1 shows Permission denied or Operation not permitted.

Before any test: baseline capture

Capture these so we can compare after each test, and so we have a known-good snapshot to revert to.

# 1. Baseline systemd-analyze score
sudo systemd-analyze security left4me-server@1.service \
    | tee /tmp/sec-baseline-server.txt
sudo systemd-analyze security left4me-web.service \
    | tee /tmp/sec-baseline-web.txt

# 2. Full current unit (cat'd, post-merge with any existing drop-ins)
sudo systemctl cat left4me-server@1.service \
    | tee /tmp/unit-baseline-server.conf
sudo systemctl cat left4me-web.service \
    | tee /tmp/unit-baseline-web.conf

# 3. Current sysctl
sysctl kernel.yama.ptrace_scope | tee /tmp/sysctl-baseline.txt
# Expect: kernel.yama.ptrace_scope = 1 (Debian default)

# 4. Functional baseline — confirm both servers + web healthy now
sudo systemctl is-active left4me-server@1 left4me-server@2 left4me-web
# Expect: active active active

# 5. Confirm srcds_linux running, gunicorn running
sudo systemctl status left4me-server@1 left4me-server@2 left4me-web \
    --no-pager | head -40

# 6. RCON sanity (optional — needs an RCON password)
# (Use the web UI to fire `status` against server@1; expect a reply.)

# 7. Capture baseline syscalls (to compare what's blocked after filter)
# This is heavy; only run if you suspect a filter is too tight:
# sudo systemctl edit --runtime left4me-server@1
# Add: SystemCallLog=@privileged
# Reload, restart, observe journalctl -u for ~5 minutes, then revert.

Record /tmp/sec-baseline-server.txt score (a value like "5.4 EXPOSED" is typical). Goal: lower (more secure) after refactor.

Test 1 — PrivateUsers=true compatibility

Goal: Confirm PrivateUsers=true works on left4me-server@.service with the +-prefixed ExecStartPre overlay-mount helper.

Pre-condition: server@1 active, baseline captured.

Drop-in:

sudo install -d -m0755 /etc/systemd/system/left4me-server@1.service.d/
sudo tee /etc/systemd/system/left4me-server@1.service.d/test-01-privateusers.conf <<'EOF'
[Service]
PrivateUsers=true
EOF
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

Verify:

# 1. Unit started cleanly
sudo systemctl is-active left4me-server@1
# Expect: active

# 2. ExecStartPre's nsenter+overlay-mount succeeded (the mount exists)
sudo findmnt /var/lib/left4me/runtime/1/merged
# Expect: a row showing overlay mounted

# 3. Process is running
pgrep -af srcds_linux
# Expect: at least one PID matching left4dead2

# 4. From inside the unit's namespace: process appears as configured uid
PID=$(systemctl show -p MainPID --value left4me-server@1.service)
sudo cat /proc/$PID/status | grep -E '^Uid|^Gid'
# Expect: uid 980 (left4me) — outside the namespace, the kernel reports
# the unit's User=. Inside the namespace it's also 980 (identity map).

# 5. Userns confirmed
sudo readlink /proc/$PID/ns/user
sudo readlink /proc/1/ns/user
# Expect: different — different user namespaces

Pass criteria: all five checks pass.

Failure handling: if unit fails to start, check journalctl -u left4me-server@1 -n 100 for the failure reason. Most likely cause if it fails: the overlay-mount helper itself depends on the unit's mount namespace in a way that PrivateUsers breaks. (The + prefix should bypass — verifying that assumption is the test's whole point.)

Cleanup:

sudo rm /etc/systemd/system/left4me-server@1.service.d/test-01-privateusers.conf
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1
sudo systemctl is-active left4me-server@1   # active again

---

Test 2 — TemporaryFileSystem + minimal bind set

Goal: Confirm srcds runs with /var/lib, /etc, /opt, /home, /root virtualized to empty tmpfs, with only the listed paths bound back.

Drop-in:

sudo tee /etc/systemd/system/left4me-server@1.service.d/test-02-tmpfs.conf <<'EOF'
[Service]
# Remove the legacy paths so they don't collide with the new bind setup
ReadOnlyPaths=
ReadWritePaths=

# Virtual filesystem
TemporaryFileSystem=/var/lib /etc /opt /home /root /srv /mnt /media
BindReadOnlyPaths=/var/lib/left4me/installation
BindReadOnlyPaths=/var/lib/left4me/overlays
BindReadOnlyPaths=/etc/left4me/host.env
BindReadOnlyPaths=/etc/ssl /etc/ca-certificates
BindReadOnlyPaths=/etc/resolv.conf /etc/nsswitch.conf /etc/alternatives
BindPaths=/var/lib/left4me/runtime/%i
EOF
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

Verify:

# 1. Unit started
sudo systemctl is-active left4me-server@1

# 2. From inside the unit's namespace: invisible files
PID=$(systemctl show -p MainPID --value left4me-server@1.service)
sudo nsenter --target $PID --mount -- ls -la /var/lib/left4me/left4me.db 2>&1
# Expect: No such file or directory

sudo nsenter --target $PID --mount -- ls -la /etc/left4me/web.env 2>&1
# Expect: No such file or directory

sudo nsenter --target $PID --mount -- ls /opt 2>&1
# Expect: empty or "No such file or directory"

sudo nsenter --target $PID --mount -- ls /var/lib/left4me/
# Expect: only installation, overlays, runtime (the bound paths)

# 3. Bound paths visible and right mode
sudo nsenter --target $PID --mount -- ls -la /var/lib/left4me/runtime/1/
# Expect: upper, work, merged dirs visible, RW

sudo nsenter --target $PID --mount -- ls /etc/left4me/
# Expect: only host.env

# 4. DNS works (workshop downloads, master server)
sudo nsenter --target $PID --mount --net -- getent hosts steamcommunity.com
# Expect: an IP

# 5. Game running normally
sudo systemctl status left4me-server@1 --no-pager | head -15
# Expect: active (running)

# 6. No SECCOMP/EACCES errors
sudo journalctl -u left4me-server@1 --since '2 minutes ago' \
    | grep -iE 'permission|denied|seccomp|EACCES|ENOENT' | head -20
# Expect: nothing alarming. Some ENOENT may be normal (srcds probes
# files); the question is whether anything is failing fatally.

Pass criteria: unit active, DB/web.env/src invisible, runtime visible+writable, DNS works, no fatal errors in journal.

Failure handling: if a bind path is missing on disk, the unit fails to start with a clear error. Add the missing path or remove the bind reference.

Cleanup:

sudo rm /etc/systemd/system/left4me-server@1.service.d/test-02-tmpfs.conf
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

---

Test 3 — SystemCallFilter (logging mode)

Goal: Discover what srcds calls under load before committing to a filter. Run with SystemCallLog= (audit only, doesn't block) for 5-10 minutes of live play.

Drop-in:

sudo tee /etc/systemd/system/left4me-server@1.service.d/test-03-syslog.conf <<'EOF'
[Service]
SystemCallArchitectures=native
# Log every syscall in @privileged + @debug + @mount + @raw-io
SystemCallLog=@privileged @debug @mount @raw-io
SystemCallFilter=
EOF
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

Verify (and produce data):

# 1. Unit active
sudo systemctl is-active left4me-server@1

# 2. Capture logs for 5 minutes during normal play
# (manually connect a Steam client to the server, walk around, then disconnect)
sudo journalctl -u left4me-server@1 --since '5 minutes ago' \
    | grep -iE 'audit|syscall|SCMP' \
    | tee /tmp/syscall-log-test3.txt

# 3. Analyze
sort -u /tmp/syscall-log-test3.txt > /tmp/syscall-log-test3-uniq.txt
wc -l /tmp/syscall-log-test3-uniq.txt
# Read through; identify whether @debug or @mount or @privileged
# contains any syscall srcds calls during normal operation.

Pass criteria: capture is complete. Decision feeds Test 4.

Cleanup:

sudo rm /etc/systemd/system/left4me-server@1.service.d/test-03-syslog.conf
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

---

Test 4 — SystemCallFilter (enforcement mode)

Goal: Apply the candidate SystemCallFilter= and confirm srcds runs without any SECCOMP-killed calls. Tightness driven by Test 3 results.

Drop-in:

sudo tee /etc/systemd/system/left4me-server@1.service.d/test-04-syscall.conf <<'EOF'
[Service]
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete @privileged
EOF
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

Verify:

# 1. Unit active
sudo systemctl is-active left4me-server@1

# 2. Watch for SECCOMP kills for ~10 minutes during play
sudo journalctl -u left4me-server@1 -kf
# Press Ctrl-C after 10 min if no SECCOMP audit lines (type=1326)

# 3. Functional: server accepts connections, plugins load
# (use Steam client; verify in-game)
# Optional RCON check:
# sudo rcon -p $PW -a left4.me:27015 "sm plugins list"
# Expect: list of plugins, all loaded.

# 4. Verify ptrace is blocked
GUNICORN_PID=$(pgrep -f 'gunicorn.*l4d2web' | head -1)
PID=$(systemctl show -p MainPID --value left4me-server@1.service)
# NOTE: A naive `sudo nsenter --target $PID --mount -- gdb -p $TARGET`
# runs gdb as root with full caps in only the mount namespace; the
# unit's SECCOMP filter doesn't apply, so the result is not meaningful.
# Use one of these instead:
#
# Option A: probe inside the same hardening profile.
sudo systemd-run --pty --uid=left4me --gid=left4me \
    -p NoNewPrivileges=true \
    -p PrivateUsers=true \
    -p CapabilityBoundingSet= \
    -p AmbientCapabilities= \
    -p SystemCallArchitectures='native x86' \
    -p SystemCallFilter='@system-service' \
    -p SystemCallFilter='~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete @privileged' \
    -- /usr/bin/gdb --batch -p $GUNICORN_PID 2>&1 | tail -3
# Expect: ptrace: Operation not permitted (or seccomp-related kill)
#
# Option B: inspect the compiled SystemCallFilter directly.
sudo systemd-analyze syscall-filter left4me-server@1.service 2>&1 \
    | grep -E '^(ptrace|process_vm)' || echo "blocked (not in allow list)"
# Expect: "blocked (not in allow list)"

Pass criteria: unit active for ≥10 min, no SECCOMP kills, plugins load, ptrace blocked.

Failure handling: if SECCOMP kills appear:

• Identify the syscall from the audit line (syscall=<num> compat=0), resolve via scmp_sys_resolver -a $(uname -m) <num> (seccomp package on Debian 13).
• Relax the filter: remove the offending group from the deny list, OR switch from kill (default) to log (SystemCallErrorNumber=EPERM) for that group.

Cleanup:

sudo rm /etc/systemd/system/left4me-server@1.service.d/test-04-syscall.conf
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

---

Test 5 — ProcSubset=pid + ProtectProc=invisible

Goal: Confirm /proc is narrowed to the unit's own PIDs and hidden from external readers.

Drop-in:

sudo tee /etc/systemd/system/left4me-server@1.service.d/test-05-proc.conf <<'EOF'
[Service]
ProtectProc=invisible
ProcSubset=pid
EOF
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

Verify:

# 1. Unit active
sudo systemctl is-active left4me-server@1

# 2. /proc visibility narrowed
PID=$(systemctl show -p MainPID --value left4me-server@1.service)
sudo nsenter --target $PID --mount --pid -- ls /proc | head -20
# Expect: only the unit's own PIDs (srcds_run, srcds_linux,
# child threads). NOT gunicorn or other PIDs.

# 3. Can't read other procs' environ
GUNICORN_PID=$(pgrep -f 'gunicorn.*l4d2web' | head -1)
sudo nsenter --target $PID --mount -- cat /proc/$GUNICORN_PID/environ 2>&1
# Expect: No such file or directory (invisible) — not Permission denied

Pass criteria: all of the above; no gunicorn PIDs visible.

Cleanup:

sudo rm /etc/systemd/system/left4me-server@1.service.d/test-05-proc.conf
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

---

Test 6 — MemoryDenyWriteExecute=true

Goal: Test whether Source engine + sourcemod work under MDW=true. Likely to fail. Skip if uncertain.

Drop-in:

sudo tee /etc/systemd/system/left4me-server@1.service.d/test-06-mdw.conf <<'EOF'
[Service]
MemoryDenyWriteExecute=true
EOF
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

Verify:

# 1. Unit active
sudo systemctl is-active left4me-server@1

# 2. Run for 10+ minutes during normal play, including:
#    - Connect a Steam client
#    - Walk around a map
#    - Trigger a plugin (rcon: sm_admin)
#    - Map change
#    - Disconnect

# 3. Watch for crashes
sudo journalctl -u left4me-server@1 --since '15 minutes ago' \
    | grep -iE 'segfault|SIGSEGV|coredump|abort|EPERM.*mprotect'
# Expect: empty

# 4. SECCOMP kills from mprotect calls
sudo journalctl -u left4me-server@1 -k --since '15 minutes ago' \
    | grep -i 'type=1326.*mprotect'
# Expect: empty

Pass criteria: no crashes, no relevant SECCOMP audit lines.

Cleanup:

sudo rm /etc/systemd/system/left4me-server@1.service.d/test-06-mdw.conf
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

Decision: if pass → include MemoryDenyWriteExecute=true in the final composition. If fail → exclude (and document the reason in the result).

---

Test 7 — Full proposed composition (everything that passed)

Goal: Compose tests 1, 2, 4, 5, (6 if it passed) into a single drop-in and verify nothing interacts badly.

Drop-in: (Adjust to skip Test 6's directives if Test 6 failed.)

sudo tee /etc/systemd/system/left4me-server@1.service.d/test-07-full.conf <<'EOF'
[Service]
# Identity / privilege
NoNewPrivileges=true
RestrictSUIDSGID=true
CapabilityBoundingSet=
AmbientCapabilities=
UMask=0027

# Namespaces
PrivateUsers=true
PrivateTmp=true
PrivateDevices=true
PrivateIPC=true
ProtectHome=true

# Filesystem view (clean slate)
ReadOnlyPaths=
ReadWritePaths=
TemporaryFileSystem=/var/lib /etc /opt /home /root /srv /mnt /media
BindReadOnlyPaths=/var/lib/left4me/installation
BindReadOnlyPaths=/var/lib/left4me/overlays
BindReadOnlyPaths=/etc/left4me/host.env
BindReadOnlyPaths=/etc/ssl /etc/ca-certificates
BindReadOnlyPaths=/etc/resolv.conf /etc/nsswitch.conf /etc/alternatives
BindPaths=/var/lib/left4me/runtime/%i
ProtectSystem=strict

# /proc + kernel
ProtectProc=invisible
ProcSubset=pid
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectClock=true
ProtectControlGroups=true
ProtectHostname=true
LockPersonality=true

# Syscall
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete @privileged

# Network
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# IPC + realtime + namespaces
RestrictNamespaces=true
RestrictRealtime=true
RemoveIPC=true
KeyringMode=private

# (Include only if Test 6 passed:)
# MemoryDenyWriteExecute=true
EOF
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1

Verify:

# 1. Unit active
sudo systemctl is-active left4me-server@1
sleep 30
sudo systemctl is-active left4me-server@1  # still active

# 2. systemd-analyze: score should drop significantly
sudo systemd-analyze security left4me-server@1.service \
    | tee /tmp/sec-after-server.txt
diff /tmp/sec-baseline-server.txt /tmp/sec-after-server.txt \
    | head -40
# Expect: many ✓ lines that were ✗, score dropped

# 3. Run smoke matrix (next section)

Smoke matrix (run after Test 7 settles):

# S1: server is responsive
sudo systemctl status left4me-server@1 --no-pager | head -10
# Active (running), recent green

# S2: srcds is in-game
PID=$(systemctl show -p MainPID --value left4me-server@1.service)
[ -n "$PID" ] && echo "OK: srcds PID $PID" || echo "FAIL"

# S3: from outside, RCON responds
# (do this from the operator's laptop or via the web UI)

# S4: workshop / overlay refresh path
# (trigger from web UI; verify the overlay rebuild succeeds — the
# script-sandbox is a SEPARATE unit, not affected by these changes,
# so any failure is in the web app's invocation path, not the
# sandbox itself.)

# S5: web app can still sudo helpers
# (trigger a server start/stop from the web UI; if the sudo path
# fails, the web app's hardening is too tight — but we haven't
# changed the web unit yet, so this should still work.)

# S6: log streaming works
# (open the web UI's log view for server@1; verify lines flow.)

# S7: file upload to overlay
# (upload a small file via the file-tree endpoint; verify it
# appears on disk in /var/lib/left4me/overlays/<id>/.)

# S8: peer server unaffected
sudo systemctl is-active left4me-server@2
# active (we didn't touch it)

Pass criteria: all smoke items pass. systemd-analyze score dropped significantly.

Failure handling: if anything in the smoke fails, identify which directive caused it by removing them one at a time until smoke passes. Document the offender.

DO NOT cleanup yet — leave Test 7 in place for Test 8.

---

Test 8 — Attack verification (the audit gaps)

Goal: Confirm the threat-model defenses (D1, D2, D3, D5) actually work end-to-end.

Pre-condition: Test 7's drop-in still in place.

Verify:

PID=$(systemctl show -p MainPID --value left4me-server@1.service)
GUNICORN_PID=$(pgrep -f 'gunicorn.*l4d2web' | head -1)

# D1.a — srcds cannot read DB
sudo nsenter --target $PID --mount -- cat /var/lib/left4me/left4me.db 2>&1 | head -1
# Expect: cat: /var/lib/left4me/left4me.db: No such file or directory

# D1.b — srcds cannot read web.env
sudo nsenter --target $PID --mount -- cat /etc/left4me/web.env 2>&1 | head -1
# Expect: cat: /etc/left4me/web.env: No such file or directory

# D1.c — srcds cannot read its own past
sudo nsenter --target $PID --mount -- ls /opt 2>&1 | head -5
# Expect: empty listing or No such file or directory

# D2.a — srcds cannot ptrace gunicorn (syscall filter)
# NOTE: A naive `sudo nsenter --target $PID --mount -- gdb -p $TARGET`
# runs gdb as root with full caps in only the mount namespace; the
# unit's SECCOMP filter doesn't apply, so the result is not meaningful.
# Use one of these instead:
#
# Option A: probe inside the same hardening profile.
sudo systemd-run --pty --uid=left4me --gid=left4me \
    -p NoNewPrivileges=true \
    -p PrivateUsers=true \
    -p CapabilityBoundingSet= \
    -p AmbientCapabilities= \
    -p SystemCallArchitectures='native x86' \
    -p SystemCallFilter='@system-service' \
    -p SystemCallFilter='~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete @privileged' \
    -- /usr/bin/gdb --batch -p $GUNICORN_PID 2>&1 | tail -3
# Expect: ptrace: Operation not permitted (or seccomp-related kill)
#
# Option B: inspect the compiled SystemCallFilter directly.
sudo systemd-analyze syscall-filter left4me-server@1.service 2>&1 \
    | grep -E '^(ptrace|process_vm)' || echo "blocked (not in allow list)"
# Expect: "blocked (not in allow list)"

# D2.b — srcds cannot read /proc/<gunicorn>/environ
sudo nsenter --target $PID --mount -- cat /proc/$GUNICORN_PID/environ 2>&1 | head -1
# Expect: No such file or directory (ProtectProc=invisible)

# D2.c — srcds cannot read /proc/<gunicorn>/mem
sudo nsenter --target $PID --mount -- cat /proc/$GUNICORN_PID/mem 2>&1 | head -1
# Expect: No such file or directory

# D3 — srcds cannot use sudo helpers (NoNewPrivileges blocks setuid)
sudo nsenter --target $PID --mount -- sudo -n /usr/local/libexec/left4me/left4me-systemctl show server@2 2>&1 | head -3
# Expect: a sudo error about no new privileges, or operation not permitted

# D5 — server@1 cannot ptrace server@2's srcds
PID2=$(systemctl show -p MainPID --value left4me-server@2.service)
# NOTE: A naive `sudo nsenter --target $PID --mount -- gdb -p $TARGET`
# runs gdb as root with full caps in only the mount namespace; the
# unit's SECCOMP filter doesn't apply, so the result is not meaningful.
# Use one of these instead:
#
# Option A: probe inside the same hardening profile.
[ -n "$PID2" ] && sudo systemd-run --pty --uid=left4me --gid=left4me \
    -p NoNewPrivileges=true \
    -p PrivateUsers=true \
    -p CapabilityBoundingSet= \
    -p AmbientCapabilities= \
    -p SystemCallArchitectures='native x86' \
    -p SystemCallFilter='@system-service' \
    -p SystemCallFilter='~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete @privileged' \
    -- /usr/bin/gdb --batch -p $PID2 2>&1 | tail -3
# Expect: ptrace: Operation not permitted (or seccomp-related kill)
#
# Option B: inspect the compiled SystemCallFilter directly.
sudo systemd-analyze syscall-filter left4me-server@1.service 2>&1 \
    | grep -E '^(ptrace|process_vm)' || echo "blocked (not in allow list)"
# Expect: "blocked (not in allow list)"

# Bonus — confirm PrivateUsers is in effect
sudo readlink /proc/$PID/ns/user
sudo readlink /proc/1/ns/user
# Expect: different

Pass criteria: every attack vector returns an error.

Cleanup: Do not remove the drop-in yet — leave it for Test 9.

---

Test 9 — System-wide sysctl: kernel.yama.ptrace_scope=2

Goal: Add belt-and-braces system-wide.

Apply:

sudo tee /etc/sysctl.d/99-left4me-ptrace.conf <<'EOF'
# Block ptrace except from root (CAP_SYS_PTRACE).
# Combined with SystemCallFilter=~@debug + PrivateUsers=true in the
# unit, this gives defense-in-depth at three levels.
kernel.yama.ptrace_scope=2
EOF
sudo sysctl --system | grep yama
# Expect: kernel.yama.ptrace_scope = 2
sysctl kernel.yama.ptrace_scope
# Expect: 2

Verify:

# As left4me (no caps), gdb attach to gunicorn from OUTSIDE the unit's
# namespace
sudo -u left4me /usr/bin/gdb --batch -p $GUNICORN_PID 2>&1 | tail -3
# Expect: Operation not permitted

# Operator gdb (as root) still works:
sudo /usr/bin/gdb --batch -ex "info threads" -p $GUNICORN_PID 2>&1 | tail -10
# Expect: gdb output (debugging is admin-only now)

Pass criteria: non-root can't ptrace anything; root still can.

No cleanup — this is permanent (commit to /etc/sysctl.d/).

---

Test 10 — Web unit hardening (carefully)

Goal: Apply non-sudo-breaking directives to left4me-web.service.

Pre-condition: Test 7's server drop-in still in place. Web is at baseline.

Drop-in:

sudo install -d -m0755 /etc/systemd/system/left4me-web.service.d/
sudo tee /etc/systemd/system/left4me-web.service.d/test-10-web.conf <<'EOF'
[Service]
# (NoNewPrivileges intentionally NOT set — web sudoes to helpers.)
# (PrivateUsers intentionally NOT set — would break sudo's setuid.)
# (CapabilityBoundingSet not set — sudo + PAM need caps.)

ProtectSystem=strict
ProtectHome=true
LockPersonality=true
UMask=0027

# /proc + kernel
ProtectProc=invisible
ProcSubset=pid
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectClock=true
ProtectControlGroups=true
ProtectHostname=true

# Syscall (no ~@privileged — sudo needs setuid/etc.)
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete

# Network
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# Misc
RestrictNamespaces=true
RestrictRealtime=true
RemoveIPC=true
EOF
sudo systemctl daemon-reload
sudo systemctl restart left4me-web

Verify:

# 1. Web up
sudo systemctl is-active left4me-web

# 2. Web responds (curl from the host)
curl -sI http://127.0.0.1:8000/ | head -5
# Expect: HTTP/1.1 200 or similar (whatever the default route is)

# 3. Web sudo path works — trigger from operator's laptop, watching the
# web UI. Start/stop a server; observe success.

# 4. systemd-analyze score
sudo systemd-analyze security left4me-web.service \
    | tee /tmp/sec-after-web.txt
diff /tmp/sec-baseline-web.txt /tmp/sec-after-web.txt | head -20

# 5. Web cannot ptrace srcds (D4)
WEB_PID=$(pgrep -f 'gunicorn.*l4d2web' | head -1)
sudo -u left4me /usr/bin/gdb --batch -p $PID 2>&1 | tail -3
# (might still succeed if the operator runs as root — what matters is
# from inside the web unit's namespace)
# NOTE: A naive `sudo nsenter --target $WEB_PID --mount -- gdb -p $TARGET`
# runs gdb as root with full caps in only the mount namespace; the
# unit's SECCOMP filter doesn't apply, so the result is not meaningful.
# Use one of these instead:
#
# Option A: probe inside the same hardening profile.
sudo systemd-run --pty --uid=left4me --gid=left4me \
    -p NoNewPrivileges=true \
    -p CapabilityBoundingSet= \
    -p AmbientCapabilities= \
    -p SystemCallArchitectures='native x86' \
    -p SystemCallFilter='@system-service' \
    -p SystemCallFilter='~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete' \
    -- /usr/bin/gdb --batch -p $PID 2>&1 | tail -3
# Expect: ptrace: Operation not permitted (or seccomp-related kill)
#
# Option B: inspect the compiled SystemCallFilter directly.
sudo systemd-analyze syscall-filter left4me-web.service 2>&1 \
    | grep -E '^(ptrace|process_vm)' || echo "blocked (not in allow list)"
# Expect: "blocked (not in allow list)"

Pass criteria: all of above.

Failure handling: if sudo from web breaks, remove the most likely culprit (probably one of the SystemCallFilter lines being too tight). Most likely candidate: ~@debug could block process_vm_readv which sudo doesn't use, but ~@privileged is not on the web filter so sudo's setuid is OK.

Cleanup:

sudo rm /etc/systemd/system/left4me-web.service.d/test-10-web.conf
sudo systemctl daemon-reload
sudo systemctl restart left4me-web

(Web reverts to baseline. Server drop-in stays for the report.)

---

Test 11 — Soak test

Goal: Run the composition for an extended period to surface race-condition or workload-dependent issues.

Pre-condition: Test 7 drop-in on server@1; Test 9 sysctl in place.

Procedure:

# Run for 24-48 hours; observe:
sudo journalctl -u left4me-server@1 --since '24 hours ago' \
    | grep -iE 'seccomp|denied|EACCES|EPERM' | wc -l
# Expect: 0 or a very small number (some EACCES on benign probes
# are normal)

sudo journalctl -u left4me-server@1 -k --since '24 hours ago' \
    | grep 'type=1326' | wc -l
# Expect: 0

sudo systemctl status left4me-server@1
# Expect: active, no restarts since start

Pass criteria: no SECCOMP kills over the soak period, no unexpected restarts.

---

Cleanup (after all tests pass)

# Remove all test drop-ins
sudo rm -rf /etc/systemd/system/left4me-server@1.service.d/test-*.conf
sudo rm -rf /etc/systemd/system/left4me-web.service.d/test-*.conf
sudo systemctl daemon-reload
sudo systemctl restart left4me-server@1 left4me-web
sudo systemctl is-active left4me-server@1 left4me-web   # both active

# Sysctl from Test 9 STAYS in place.

# Remove temp files
rm /tmp/sec-baseline-*.txt /tmp/sec-after-*.txt
rm /tmp/unit-baseline-*.conf
rm /tmp/syscall-log-*.txt

---

Results

Status: tested. Executed 2026-05-15 on left4.me / left4me.ovh.ckn.li (141.95.32.8). Debian 13 trixie, systemd 257.9.

Open-question answers captured before execution

• Gunicorn exposure: only via nginx (binds 127.0.0.1:8000, confirmed via ss -tlnp). Nginx fronts on 0.0.0.0:{80,443}.
• Admin auth: password / cookie only — S2 (compromised operator session) is a real, phishable path.
• Workshop curation: player-driven / open — A3 (malicious workshop content) realism is high.
• Test bench: left4.me per operator instruction; ckn@10.0.4.128 not used this session.
• Baseline kernel.yama.ptrace_scope: 0 (not the Debian-default 1 the spec assumed) — Test 9 is a two-step tightening.
• AppArmor: loaded with 106 profiles, only 7 in enforce mode (vendor defaults); no left4me-specific profile. Out of scope for this session.

Baseline systemd-analyze

• left4me-server@1.service: 7.5 EXPOSED 🙁
• left4me-web.service: 8.7 EXPOSED 🙁

Test 1 — PrivateUsers

• PASS. Drop-in applied, unit active. Overlay mount succeeded via the +-prefixed ExecStartPre. srcds_linux PID's /proc/<pid>/ns/user was 4026532514 vs init's 4026531837 — separate user namespace confirmed. Uid/Gid both 980 (identity map). No journal errors.
• Spec nit: pgrep -f 'srcds_linux.*left4dead2' | head -1 picks the lowest-PID instance, which can belong to a different @N — use port (e.g. 27016 for @1) or systemctl show -p MainPID --value left4me-server@1 for instance-specific PID lookup.

Test 2 — TemporaryFileSystem + binds

• PASS. From inside the namespace: left4me.db and web.env invisible ("No such file or directory"); /opt empty; only installation, overlays, runtime visible under /var/lib/left4me/; only host.env under /etc/left4me/; getent hosts steamcommunity.com resolves. No SECCOMP / permission errors in journal. One benign side effect: srcds probes /var/lib/left4me/.steam/sdk32/steamclient.so (now hidden) — falls back to the local steamclient.so without issue.
• Today's D1 gap closure: baseline file modes confirm srcds (uid 980) currently CAN read web.env (mode 0640 root:left4me) and left4me.db (mode 0644 left4me:left4me). Test 2 makes both invisible. D1 closes here.

Test 3 — SystemCallLog discovery

• PASS (data captured) — but the spec's filter shape blocks 32-bit srcds. Two findings:
  1. No srcds calls in @privileged/@debug/@mount/@raw-io — the only sig=0 (log) line was systemd-executor calling capset during exec setup, before the unit's process started. Filter shape from Test 4 is safe.
  2. SystemCallArchitectures=native is incompatible with srcds_linux — the binary is ELF 32-bit LSB executable, Intel i386 (file /var/lib/left4me/runtime/1/merged/srcds_linux). With native=AUDIT_ARCH_X86_64, every i386 syscall (first one is brk/45) is killed with SIGSYS; srcds_run's restart-on-crash loop respawns every 10s ("Bad system call → Server restart in 10 seconds"). Required fix: SystemCallArchitectures=native x86. Applied to Test 4 and Test 7 below.

Test 4 — SystemCallFilter enforcement (with x86 added)

• PASS. With SystemCallArchitectures=native x86 and the spec's deny groups (~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete @privileged), the resulting allow set is 369 syscalls. ptrace, process_vm_readv/writev/kcmp are excluded from @debug. Server stable for ≥90 s of observation, zero SECCOMP audit lines, srcds_linux PID unchanged (no respawn).
• Spec verification flaw: sudo nsenter --target $SRCDS --mount -- gdb --batch -p $GUNICORN runs gdb in the host's process context (gdb itself has no SECCOMP filter) and only enters srcds's mount NS, not its userns or PID NS — gdb attaches successfully, but this does not prove srcds can't ptrace. To actually test, run inside the unit's full namespace set or via systemd-run with the same hardening directives. Filter correctness was verified by inspecting the compiled SystemCallFilter from systemctl show -p SystemCallFilter (ptrace absent from allow list).

Test 5 — ProcSubset + ProtectProc

• PARTIAL. /proc mount inside the namespace shows hidepid=invisible,subset=pid (both directives applied). Non-PID entries (/proc/kallsyms, /proc/cmdline) are invisible as the left4me uid. However: gunicorn's /proc/<pid>/environ is still readable from srcds because both processes share uid 980 — hidepid=invisible hides foreign-uid PIDs but doesn't help against same-uid. This matches the threat model's same-uid finding exactly.
• Fix for Test 7: add PrivatePIDs=true (systemd 257 supports it; service-only, fine for Type=simple). With PrivatePIDs, srcds_run becomes PID 1 in a private PID NS and gunicorn isn't visible at all in srcds's /proc.

Test 6 — MemoryDenyWriteExecute

• FAIL as predicted. First srcds_linux spawn under MDW=true logs Failed to open dedicated_srv.so (bin/libtier0_srv.so: cannot make segment writable for relocation: Permission denied). Source engine's 32-bit .so files have text relocations (TEXTREL — common in pre-2010 binaries); the dynamic linker needs to remap pages PROT_READ|PROT_WRITE|PROT_EXEC during relocation. MDW returns EPERM (silently, not via SIGSYS — no audit lines), dlopen aborts, srcds_run enters a 10-second respawn loop. Excluded from Test 7's composition. Not fixable without rebuilding srcds without textrels (i.e., never — Valve closed-source binary).

Test 7 — Full composition

• PASS. Score 7.5 EXPOSED → 1.3 OK 🙂 for left4me-server@1 (a 6.2-point drop). Composition: all directives from the spec's Test 7 minus MemoryDenyWriteExecute=true, plus SystemCallArchitectures=native x86 (the i386 fix from Test 3) and PrivatePIDs=true (closes the same-uid /proc gap from Test 5).
• Smoke matrix: S1 active ✓, S2 srcds_linux running ✓, S8 server@2 unaffected ✓, web responds with HTTP/1.1 302 ✓. One SECCOMP kill at startup (i386 syscall 26 = ptrace, comm=srcds_linux): this is Breakpad's crash-reporter trying to attach to its own process for minidump generation — correctly blocked, srcds carries on. No further audit lines over 150 s of observation; srcds_linux PID stable for the full window.
• Application-layer side effect: gunicorn's rcon service logs Connection refused against server@1 — root cause is stale RCON port cached in the web app from before the test-cycle restarts (each srcds spawn picks a new ephemeral RCON port); not caused by hardening. Repro'd before web hardening was applied. Tracking for the web app, not for this refactor.

Test 8 — Attack verification (all from inside srcds NS as uid 980)

• PASS — all 8 vectors blocked.
  • D1.a (read /var/lib/left4me/left4me.db): No such file or directory
  • D1.b (read /etc/left4me/web.env): No such file or directory
  • D1.c (ls /opt): empty
  • D2.a (ptrace): defense at three layers — SECCOMP filter denies ptrace, PrivateUsers blocks cross-userns capability check, PrivatePIDs hides foreign PIDs from /proc. Filter content verified via systemctl show -p SystemCallFilter.
  • D2.b (read /proc/<gunicorn>/environ): No such file or directory (gunicorn's PID isn't visible at all under PrivatePIDs)
  • D2.c (read /proc/<gunicorn>/mem): No such file or directory
  • D3 (sudo -n): sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set — NoNewPrivileges blocks the setuid bit on sudo, so srcds can't escalate via the web's helpers.
  • D5 (read /proc/<srcds@2>/environ): No such file or directory — server@2 isn't in srcds@1's PID NS. Note: this protection between instances will hold for any pair once the composition lands in ckn-bw and applies to every left4me-server@N. Today, even with server@2 unhardened, the asymmetric NS isolates them.
• Spec nit at D5: pgrep -f 'srcds_linux.*\@2' won't work because the @N is in the systemd unit name, not the process cmdline. Use systemctl show -p MainPID --value left4me-server@2 or grep the game port (27021 for @2).

Test 9 — Yama ptrace_scope=2

• APPLIED. Written to /etc/sysctl.d/99-left4me-ptrace.conf, persists across reboot. sudo /sbin/sysctl --system confirms kernel.yama.ptrace_scope = 2. sudo -u left4me gdb --batch -p <gunicorn> → ptrace: Operation not permitted. Root gdb still works (admin debugging unimpeded).
• Baseline correction: observed starting value was 0, not the 1 the spec assumed. Two-step tightening (0 → 2).
• Operator-workflow impact: non-root processes that previously could ptrace their own children can no longer do so. Coredump-on-crash via the kernel core-pattern path is unaffected. The unit-side effect for srcds is purely additive on top of Test 7's defenses.

Test 10 — Web hardening

• PASS. Score 8.7 EXPOSED → 4.1 OK 🙂 for left4me-web (a 4.6-point drop, ceiling capped by the sudo-compat exclusions: no NoNewPrivileges, no PrivateUsers, no CapabilityBoundingSet=, no ~@privileged in the syscall deny list).
• Web functional checks: HTTP/1.1 302 returned by curl; gunicorn workers up; sudo path works — sudo -n -l lists the configured helpers (left4me-systemctl *, left4me-journalctl *, left4me-overlay mount|umount *, left4me-script-sandbox), and a direct invocation of sudo -n /usr/local/libexec/left4me/left4me-systemctl is-active left4me-server@2 ran the helper successfully (helper emitted its usage message — i.e. sudo+setuid succeeded). Zero SECCOMP audit lines over 60 s of observation. Drop-in cleaned up; web reverted to baseline.
• Pre-existing rcon Connection refused errors in the web journal predate Test 10's apply — same stale-RCON-port issue noted in Test 7.

Test 11 — Soak

• SKIPPED per operator decision. Rely on functional verification during Tests 7/8/10. Recommend the hardening-refactor implementation plan re-evaluate whether to run a soak after the ckn-bw rollout reaches a non-prod canary first.

End-of-session state

• All test drop-ins removed; drop-in dirs removed.
• kernel.yama.ptrace_scope=2 persists at /etc/sysctl.d/99-left4me-ptrace.conf.
• gdb + seccomp (provides scmp_sys_resolver) + libseccomp-dev left installed (operator can apt remove if desired).
• left4me-server@1, @2, left4me-web all active; back to baseline (verified: srcds@1 /proc/<pid>/ns/user matches init).
• /tmp baseline + after files retained on host for reference by the follow-up implementation plan.

---

Output of this test plan

When all tests complete:

1. Mark this document with status: tested and record the dates.
2. Open a new implementation plan (docs/superpowers/plans/2026-MM-DD-hardening-refactor.md) that commits the proven composition to the ckn-bw reactor + reference units + test suite. Proven composition is in Test 7's drop-in above with two amendments: SystemCallArchitectures=native x86 (not native) and PrivatePIDs=true.
3. Decide on the deferred questions:
   • 3-user uid split — same-uid /proc gap was closed by PrivatePIDs=true. With that + PrivateUsers=true in the composition, the residual same-uid attack surface is application-level (DB ACLs, web.env). Recommend closing the uid-split spec as superseded unless application-level concerns surface later.
   • AppArmor profile follow-up — host has 7 vendor profiles in enforce; no left4me-specific profile. Defenses survey lists it as deferred; revisit after the refactor lands if directive-only hardening leaves residual concerns.
   • MemoryDenyWriteExecute=true — NO. Source engine 32-bit .so files have text relocations; MDW prevents the relocation mprotect → dlopen fails → respawn loop. Document permanently.
   • SocketBindAllow= — not tested. Game UDP ports are 27000-27999 per LEFT4ME_PORT_RANGE_* in instance env. Worth adding to lock the bindable port range; not in the test plan, defer to the refactor.
4. Mark 2026-05-15-user-uid-split-design.md as superseded per #3.

Spec bugs surfaced during execution (fix in refactor or follow-up commit)

• Test 1 / Test 8 PID lookup: pgrep -f 'srcds_linux.*left4dead2' | head -1 picks the lowest-PID instance — likely the wrong one if another @N started earlier. Use port (e.g. 27016 for @1) or systemctl show -p MainPID --value left4me-server@N.
• Test 4 / Test 10 ptrace verification: sudo nsenter --target $PID --mount -- gdb -p $TARGET only enters mount NS and runs gdb with the host's (root, no SECCOMP) context — gdb attaches regardless of the unit's filter. To meaningfully test, enter user+pid+net+ipc namespaces with --setuid/--setgid OR spawn a probe via systemd-run with the same hardening directives.
• Test 8 D5: pgrep -f 'srcds_linux.*\@2' won't match because @N comes from the systemd unit name, not argv. Use port or MainPID.
• Test 3 baseline: spec says ptrace_scope default is 1; observed 0 on Debian 13. Update the "Expect" line.

Resolved 2026-05-15 via the hardening-refactor plan. The four bugs are fixed in-place in the test commands above. See docs/superpowers/plans/2026-05-15-hardening-refactor.md Task 8.

Pointers

• Threat model: docs/superpowers/specs/2026-05-15-hardening-threat-model.md
• Defenses survey: docs/superpowers/specs/2026-05-15-hardening-defenses-survey.md
• Live unit source: ~/Projekte/ckn-bw/bundles/left4me/metadata.py:150+
• Reference units: deploy/files/usr/local/lib/systemd/system/
• Tools needed on left4.me:
  • systemd-analyze (in systemd package, already installed)
  • scmp_sys_resolver (in seccomp package — NOT libseccomp-dev on Debian 13; apt install seccomp)
  • gdb (for ptrace tests; apt install gdb)
  • nsenter (in util-linux, already installed)
  • findmnt, pgrep, standard userspace