Reframe the queued uid-split decision into a broader hardening analysis. Audit found the same-uid attack surface (DB readable from srcds, ptrace allowed, RCON stored plaintext) is closable by either uid split or systemd directive composition; the three specs ground that choice in a threat model, survey the defenses, and lay out a self-contained test plan to run on left4.me next. uid-split spec deferred pending results. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
898 lines
28 KiB
Markdown
898 lines
28 KiB
Markdown
# left4me application hardening — test plan
|
|
|
|
**Status:** living spec. Companion to `2026-05-15-hardening-threat-model.md`
|
|
and `2026-05-15-hardening-defenses-survey.md`. **Executed in a follow-up
|
|
session with shell access to `left4.me` (141.95.32.8).**
|
|
|
|
This document is intentionally self-contained: a session that lands cold
|
|
with shell on `left4.me` can execute it end-to-end without re-reading
|
|
the threat model or survey. Decisions made in this plan are based on the
|
|
candidate composition in the defenses survey (Section 5).
|
|
|
|
## Test architecture
|
|
|
|
### Where we test
|
|
|
|
- **Host:** `left4.me` / `ovh.left4me` (141.95.32.8). Production host;
|
|
no separate test bench. (Reference: memory entry
|
|
`feedback_test_server_hangs.md` mentions a separate test server at
|
|
`ckn@10.0.4.128`; verify whether that host is suitable for this work
|
|
*before* using prod.)
|
|
- **Canary unit:** `left4me-server@1.service`. Use this as the test
|
|
instance. Leave `left4me-server@2.service` running baseline so at
|
|
least one server stays up if the canary breaks.
|
|
- **Web unit:** `left4me-web.service` is shared. Test web-side
|
|
hardening only after server@ tests prove the composition; web is
|
|
more disruptive to roll back.
|
|
|
|
### Operating constraints
|
|
|
|
- **System units only.** No `systemctl --user`, no lingering, no
|
|
per-user systemd instance. All units under `/etc/systemd/system/` or
|
|
`/usr/local/lib/systemd/system/`. Drop-ins go to
|
|
`/etc/systemd/system/<unit>.d/`.
|
|
- **Drop-in style.** Tests apply via `/etc/systemd/system/left4me-server@1.service.d/test-NN-<name>.conf`
|
|
(note: `@1` for instance-specific). This leaves the template
|
|
unmodified — other instances unaffected. `systemctl daemon-reload`
|
|
picks up drop-ins; `systemctl restart left4me-server@1` applies.
|
|
- **Cleanup required.** Each test removes its drop-in before the next
|
|
starts. Baseline must be restorable at any point.
|
|
- **Recording.** Each test produces a one-paragraph result in this
|
|
document's "Results" section at the bottom. Append, don't replace.
|
|
|
|
### Failure modes to watch for
|
|
|
|
- **SECCOMP audit:** `journalctl -k --since '1 minute ago' | grep -i seccomp`
|
|
shows `type=1326` lines. Each is a syscall denied; the syscall number
|
|
identifies the call. Use `scmp_sys_resolver` to translate.
|
|
- **Unit start failure:** `systemctl is-active left4me-server@1` → `inactive` or `failed`.
|
|
- **srcds crash mid-game:** `journalctl -u left4me-server@1 -f` shows
|
|
unexpected exit; `systemctl show left4me-server@1 -p Result` is
|
|
not `success`.
|
|
- **sourcemod/metamod plugin failures:** in-game `sm plugins list` or
|
|
RCON `sm plugins list` shows plugins as failed-to-load.
|
|
- **Permission denied where unexpected:** `journalctl -u left4me-server@1`
|
|
shows `Permission denied` or `Operation not permitted`.
|
|
|
|
## Before any test: baseline capture
|
|
|
|
Capture these so we can compare after each test, and so we have a
|
|
known-good snapshot to revert to.
|
|
|
|
```bash
|
|
# 1. Baseline systemd-analyze score
|
|
sudo systemd-analyze security left4me-server@1.service \
|
|
| tee /tmp/sec-baseline-server.txt
|
|
sudo systemd-analyze security left4me-web.service \
|
|
| tee /tmp/sec-baseline-web.txt
|
|
|
|
# 2. Full current unit (cat'd, post-merge with any existing drop-ins)
|
|
sudo systemctl cat left4me-server@1.service \
|
|
| tee /tmp/unit-baseline-server.conf
|
|
sudo systemctl cat left4me-web.service \
|
|
| tee /tmp/unit-baseline-web.conf
|
|
|
|
# 3. Current sysctl
|
|
sysctl kernel.yama.ptrace_scope | tee /tmp/sysctl-baseline.txt
|
|
# Expect: kernel.yama.ptrace_scope = 1 (Debian default)
|
|
|
|
# 4. Functional baseline — confirm both servers + web healthy now
|
|
sudo systemctl is-active left4me-server@1 left4me-server@2 left4me-web
|
|
# Expect: active active active
|
|
|
|
# 5. Confirm srcds_linux running, gunicorn running
|
|
sudo systemctl status left4me-server@1 left4me-server@2 left4me-web \
|
|
--no-pager | head -40
|
|
|
|
# 6. RCON sanity (optional — needs an RCON password)
|
|
# (Use the web UI to fire `status` against server@1; expect a reply.)
|
|
|
|
# 7. Capture baseline syscalls (to compare what's blocked after filter)
|
|
# This is heavy; only run if you suspect a filter is too tight:
|
|
# sudo systemctl edit --runtime left4me-server@1
|
|
# Add: SystemCallLog=@privileged
|
|
# Reload, restart, observe journalctl -u for ~5 minutes, then revert.
|
|
```
|
|
|
|
Record `/tmp/sec-baseline-server.txt` score (a value like "5.4 EXPOSED"
|
|
is typical). Goal: lower (more secure) after refactor.
|
|
|
|
## Test 1 — `PrivateUsers=true` compatibility
|
|
|
|
**Goal:** Confirm `PrivateUsers=true` works on `left4me-server@.service`
|
|
with the `+`-prefixed `ExecStartPre` overlay-mount helper.
|
|
|
|
**Pre-condition:** server@1 active, baseline captured.
|
|
|
|
**Drop-in:**
|
|
```bash
|
|
sudo install -d -m0755 /etc/systemd/system/left4me-server@1.service.d/
|
|
sudo tee /etc/systemd/system/left4me-server@1.service.d/test-01-privateusers.conf <<'EOF'
|
|
[Service]
|
|
PrivateUsers=true
|
|
EOF
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
**Verify:**
|
|
```bash
|
|
# 1. Unit started cleanly
|
|
sudo systemctl is-active left4me-server@1
|
|
# Expect: active
|
|
|
|
# 2. ExecStartPre's nsenter+overlay-mount succeeded (the mount exists)
|
|
sudo findmnt /var/lib/left4me/runtime/1/merged
|
|
# Expect: a row showing overlay mounted
|
|
|
|
# 3. Process is running
|
|
pgrep -af srcds_linux
|
|
# Expect: at least one PID matching left4dead2
|
|
|
|
# 4. From inside the unit's namespace: process appears as configured uid
|
|
PID=$(pgrep -f 'srcds_linux.*left4dead2' | head -1)
|
|
sudo cat /proc/$PID/status | grep -E '^Uid|^Gid'
|
|
# Expect: uid 980 (left4me) — outside the namespace, the kernel reports
|
|
# the unit's User=. Inside the namespace it's also 980 (identity map).
|
|
|
|
# 5. Userns confirmed
|
|
sudo readlink /proc/$PID/ns/user
|
|
sudo readlink /proc/1/ns/user
|
|
# Expect: different — different user namespaces
|
|
```
|
|
|
|
**Pass criteria:** all five checks pass.
|
|
|
|
**Failure handling:** if unit fails to start, check
|
|
`journalctl -u left4me-server@1 -n 100` for the failure reason. Most
|
|
likely cause if it fails: the overlay-mount helper itself depends on
|
|
the unit's mount namespace in a way that PrivateUsers breaks. (The `+`
|
|
prefix should bypass — verifying that assumption is the test's whole
|
|
point.)
|
|
|
|
**Cleanup:**
|
|
```bash
|
|
sudo rm /etc/systemd/system/left4me-server@1.service.d/test-01-privateusers.conf
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
sudo systemctl is-active left4me-server@1 # active again
|
|
```
|
|
|
|
---
|
|
|
|
## Test 2 — `TemporaryFileSystem` + minimal bind set
|
|
|
|
**Goal:** Confirm srcds runs with `/var/lib`, `/etc`, `/opt`, `/home`,
|
|
`/root` virtualized to empty tmpfs, with only the listed paths bound back.
|
|
|
|
**Drop-in:**
|
|
```bash
|
|
sudo tee /etc/systemd/system/left4me-server@1.service.d/test-02-tmpfs.conf <<'EOF'
|
|
[Service]
|
|
# Remove the legacy paths so they don't collide with the new bind setup
|
|
ReadOnlyPaths=
|
|
ReadWritePaths=
|
|
|
|
# Virtual filesystem
|
|
TemporaryFileSystem=/var/lib /etc /opt /home /root /srv /mnt /media
|
|
BindReadOnlyPaths=/var/lib/left4me/installation
|
|
BindReadOnlyPaths=/var/lib/left4me/overlays
|
|
BindReadOnlyPaths=/etc/left4me/host.env
|
|
BindReadOnlyPaths=/etc/ssl /etc/ca-certificates
|
|
BindReadOnlyPaths=/etc/resolv.conf /etc/nsswitch.conf /etc/alternatives
|
|
BindPaths=/var/lib/left4me/runtime/%i
|
|
EOF
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
**Verify:**
|
|
```bash
|
|
# 1. Unit started
|
|
sudo systemctl is-active left4me-server@1
|
|
|
|
# 2. From inside the unit's namespace: invisible files
|
|
PID=$(pgrep -f 'srcds_linux.*left4dead2' | head -1)
|
|
sudo nsenter --target $PID --mount -- ls -la /var/lib/left4me/left4me.db 2>&1
|
|
# Expect: No such file or directory
|
|
|
|
sudo nsenter --target $PID --mount -- ls -la /etc/left4me/web.env 2>&1
|
|
# Expect: No such file or directory
|
|
|
|
sudo nsenter --target $PID --mount -- ls /opt 2>&1
|
|
# Expect: empty or "No such file or directory"
|
|
|
|
sudo nsenter --target $PID --mount -- ls /var/lib/left4me/
|
|
# Expect: only installation, overlays, runtime (the bound paths)
|
|
|
|
# 3. Bound paths visible and right mode
|
|
sudo nsenter --target $PID --mount -- ls -la /var/lib/left4me/runtime/1/
|
|
# Expect: upper, work, merged dirs visible, RW
|
|
|
|
sudo nsenter --target $PID --mount -- ls /etc/left4me/
|
|
# Expect: only host.env
|
|
|
|
# 4. DNS works (workshop downloads, master server)
|
|
sudo nsenter --target $PID --mount --net -- getent hosts steamcommunity.com
|
|
# Expect: an IP
|
|
|
|
# 5. Game running normally
|
|
sudo systemctl status left4me-server@1 --no-pager | head -15
|
|
# Expect: active (running)
|
|
|
|
# 6. No SECCOMP/EACCES errors
|
|
sudo journalctl -u left4me-server@1 --since '2 minutes ago' \
|
|
| grep -iE 'permission|denied|seccomp|EACCES|ENOENT' | head -20
|
|
# Expect: nothing alarming. Some ENOENT may be normal (srcds probes
|
|
# files); the question is whether anything is failing fatally.
|
|
```
|
|
|
|
**Pass criteria:** unit active, DB/web.env/src invisible, runtime
|
|
visible+writable, DNS works, no fatal errors in journal.
|
|
|
|
**Failure handling:** if a bind path is missing on disk, the unit
|
|
fails to start with a clear error. Add the missing path or remove the
|
|
bind reference.
|
|
|
|
**Cleanup:**
|
|
```bash
|
|
sudo rm /etc/systemd/system/left4me-server@1.service.d/test-02-tmpfs.conf
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
---
|
|
|
|
## Test 3 — `SystemCallFilter` (logging mode)
|
|
|
|
**Goal:** Discover what srcds calls under load before committing to a
|
|
filter. Run with `SystemCallLog=` (audit only, doesn't block) for 5-10
|
|
minutes of live play.
|
|
|
|
**Drop-in:**
|
|
```bash
|
|
sudo tee /etc/systemd/system/left4me-server@1.service.d/test-03-syslog.conf <<'EOF'
|
|
[Service]
|
|
SystemCallArchitectures=native
|
|
# Log every syscall in @privileged + @debug + @mount + @raw-io
|
|
SystemCallLog=@privileged @debug @mount @raw-io
|
|
SystemCallFilter=
|
|
EOF
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
**Verify (and produce data):**
|
|
```bash
|
|
# 1. Unit active
|
|
sudo systemctl is-active left4me-server@1
|
|
|
|
# 2. Capture logs for 5 minutes during normal play
|
|
# (manually connect a Steam client to the server, walk around, then disconnect)
|
|
sudo journalctl -u left4me-server@1 --since '5 minutes ago' \
|
|
| grep -iE 'audit|syscall|SCMP' \
|
|
| tee /tmp/syscall-log-test3.txt
|
|
|
|
# 3. Analyze
|
|
sort -u /tmp/syscall-log-test3.txt > /tmp/syscall-log-test3-uniq.txt
|
|
wc -l /tmp/syscall-log-test3-uniq.txt
|
|
# Read through; identify whether @debug or @mount or @privileged
|
|
# contains any syscall srcds calls during normal operation.
|
|
```
|
|
|
|
**Pass criteria:** capture is complete. Decision feeds Test 4.
|
|
|
|
**Cleanup:**
|
|
```bash
|
|
sudo rm /etc/systemd/system/left4me-server@1.service.d/test-03-syslog.conf
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
---
|
|
|
|
## Test 4 — `SystemCallFilter` (enforcement mode)
|
|
|
|
**Goal:** Apply the candidate `SystemCallFilter=` and confirm srcds
|
|
runs without any SECCOMP-killed calls. Tightness driven by Test 3
|
|
results.
|
|
|
|
**Drop-in:**
|
|
```bash
|
|
sudo tee /etc/systemd/system/left4me-server@1.service.d/test-04-syscall.conf <<'EOF'
|
|
[Service]
|
|
SystemCallArchitectures=native
|
|
SystemCallFilter=@system-service
|
|
SystemCallFilter=~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete @privileged
|
|
EOF
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
**Verify:**
|
|
```bash
|
|
# 1. Unit active
|
|
sudo systemctl is-active left4me-server@1
|
|
|
|
# 2. Watch for SECCOMP kills for ~10 minutes during play
|
|
sudo journalctl -u left4me-server@1 -kf
|
|
# Press Ctrl-C after 10 min if no SECCOMP audit lines (type=1326)
|
|
|
|
# 3. Functional: server accepts connections, plugins load
|
|
# (use Steam client; verify in-game)
|
|
# Optional RCON check:
|
|
# sudo rcon -p $PW -a left4.me:27015 "sm plugins list"
|
|
# Expect: list of plugins, all loaded.
|
|
|
|
# 4. Verify ptrace is blocked
|
|
GUNICORN_PID=$(pgrep -f 'gunicorn.*l4d2web' | head -1)
|
|
PID=$(pgrep -f 'srcds_linux.*left4dead2' | head -1)
|
|
sudo nsenter --target $PID --mount -- /usr/bin/gdb --batch -p $GUNICORN_PID 2>&1 | tail -5
|
|
# Expect: ptrace: Operation not permitted (or seccomp denial)
|
|
```
|
|
|
|
**Pass criteria:** unit active for ≥10 min, no SECCOMP kills, plugins
|
|
load, ptrace blocked.
|
|
|
|
**Failure handling:** if SECCOMP kills appear:
|
|
- Identify the syscall from the audit line (`syscall=<num> compat=0`),
|
|
resolve via `scmp_sys_resolver -a $(uname -m) <num>` (libseccomp-dev).
|
|
- Relax the filter: remove the offending group from the deny list, OR
|
|
switch from kill (default) to log (`SystemCallErrorNumber=EPERM`)
|
|
for that group.
|
|
|
|
**Cleanup:**
|
|
```bash
|
|
sudo rm /etc/systemd/system/left4me-server@1.service.d/test-04-syscall.conf
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
---
|
|
|
|
## Test 5 — `ProcSubset=pid` + `ProtectProc=invisible`
|
|
|
|
**Goal:** Confirm /proc is narrowed to the unit's own PIDs and
|
|
hidden from external readers.
|
|
|
|
**Drop-in:**
|
|
```bash
|
|
sudo tee /etc/systemd/system/left4me-server@1.service.d/test-05-proc.conf <<'EOF'
|
|
[Service]
|
|
ProtectProc=invisible
|
|
ProcSubset=pid
|
|
EOF
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
**Verify:**
|
|
```bash
|
|
# 1. Unit active
|
|
sudo systemctl is-active left4me-server@1
|
|
|
|
# 2. /proc visibility narrowed
|
|
PID=$(pgrep -f 'srcds_linux.*left4dead2' | head -1)
|
|
sudo nsenter --target $PID --mount --pid -- ls /proc | head -20
|
|
# Expect: only the unit's own PIDs (srcds_run, srcds_linux,
|
|
# child threads). NOT gunicorn or other PIDs.
|
|
|
|
# 3. Can't read other procs' environ
|
|
GUNICORN_PID=$(pgrep -f 'gunicorn.*l4d2web' | head -1)
|
|
sudo nsenter --target $PID --mount -- cat /proc/$GUNICORN_PID/environ 2>&1
|
|
# Expect: No such file or directory (invisible) — not Permission denied
|
|
```
|
|
|
|
**Pass criteria:** all of the above; no gunicorn PIDs visible.
|
|
|
|
**Cleanup:**
|
|
```bash
|
|
sudo rm /etc/systemd/system/left4me-server@1.service.d/test-05-proc.conf
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
---
|
|
|
|
## Test 6 — `MemoryDenyWriteExecute=true`
|
|
|
|
**Goal:** Test whether Source engine + sourcemod work under MDW=true.
|
|
**Likely to fail.** Skip if uncertain.
|
|
|
|
**Drop-in:**
|
|
```bash
|
|
sudo tee /etc/systemd/system/left4me-server@1.service.d/test-06-mdw.conf <<'EOF'
|
|
[Service]
|
|
MemoryDenyWriteExecute=true
|
|
EOF
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
**Verify:**
|
|
```bash
|
|
# 1. Unit active
|
|
sudo systemctl is-active left4me-server@1
|
|
|
|
# 2. Run for 10+ minutes during normal play, including:
|
|
# - Connect a Steam client
|
|
# - Walk around a map
|
|
# - Trigger a plugin (rcon: sm_admin)
|
|
# - Map change
|
|
# - Disconnect
|
|
|
|
# 3. Watch for crashes
|
|
sudo journalctl -u left4me-server@1 --since '15 minutes ago' \
|
|
| grep -iE 'segfault|SIGSEGV|coredump|abort|EPERM.*mprotect'
|
|
# Expect: empty
|
|
|
|
# 4. SECCOMP kills from mprotect calls
|
|
sudo journalctl -u left4me-server@1 -k --since '15 minutes ago' \
|
|
| grep -i 'type=1326.*mprotect'
|
|
# Expect: empty
|
|
```
|
|
|
|
**Pass criteria:** no crashes, no relevant SECCOMP audit lines.
|
|
|
|
**Cleanup:**
|
|
```bash
|
|
sudo rm /etc/systemd/system/left4me-server@1.service.d/test-06-mdw.conf
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
**Decision:** if pass → include `MemoryDenyWriteExecute=true` in the
|
|
final composition. If fail → exclude (and document the reason in the
|
|
result).
|
|
|
|
---
|
|
|
|
## Test 7 — Full proposed composition (everything that passed)
|
|
|
|
**Goal:** Compose tests 1, 2, 4, 5, (6 if it passed) into a single
|
|
drop-in and verify nothing interacts badly.
|
|
|
|
**Drop-in:** (Adjust to skip Test 6's directives if Test 6 failed.)
|
|
```bash
|
|
sudo tee /etc/systemd/system/left4me-server@1.service.d/test-07-full.conf <<'EOF'
|
|
[Service]
|
|
# Identity / privilege
|
|
NoNewPrivileges=true
|
|
RestrictSUIDSGID=true
|
|
CapabilityBoundingSet=
|
|
AmbientCapabilities=
|
|
UMask=0027
|
|
|
|
# Namespaces
|
|
PrivateUsers=true
|
|
PrivateTmp=true
|
|
PrivateDevices=true
|
|
PrivateIPC=true
|
|
ProtectHome=true
|
|
|
|
# Filesystem view (clean slate)
|
|
ReadOnlyPaths=
|
|
ReadWritePaths=
|
|
TemporaryFileSystem=/var/lib /etc /opt /home /root /srv /mnt /media
|
|
BindReadOnlyPaths=/var/lib/left4me/installation
|
|
BindReadOnlyPaths=/var/lib/left4me/overlays
|
|
BindReadOnlyPaths=/etc/left4me/host.env
|
|
BindReadOnlyPaths=/etc/ssl /etc/ca-certificates
|
|
BindReadOnlyPaths=/etc/resolv.conf /etc/nsswitch.conf /etc/alternatives
|
|
BindPaths=/var/lib/left4me/runtime/%i
|
|
ProtectSystem=strict
|
|
|
|
# /proc + kernel
|
|
ProtectProc=invisible
|
|
ProcSubset=pid
|
|
ProtectKernelTunables=true
|
|
ProtectKernelModules=true
|
|
ProtectKernelLogs=true
|
|
ProtectClock=true
|
|
ProtectControlGroups=true
|
|
ProtectHostname=true
|
|
LockPersonality=true
|
|
|
|
# Syscall
|
|
SystemCallArchitectures=native
|
|
SystemCallFilter=@system-service
|
|
SystemCallFilter=~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete @privileged
|
|
|
|
# Network
|
|
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
|
|
|
|
# IPC + realtime + namespaces
|
|
RestrictNamespaces=true
|
|
RestrictRealtime=true
|
|
RemoveIPC=true
|
|
KeyringMode=private
|
|
|
|
# (Include only if Test 6 passed:)
|
|
# MemoryDenyWriteExecute=true
|
|
EOF
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1
|
|
```
|
|
|
|
**Verify:**
|
|
```bash
|
|
# 1. Unit active
|
|
sudo systemctl is-active left4me-server@1
|
|
sleep 30
|
|
sudo systemctl is-active left4me-server@1 # still active
|
|
|
|
# 2. systemd-analyze: score should drop significantly
|
|
sudo systemd-analyze security left4me-server@1.service \
|
|
| tee /tmp/sec-after-server.txt
|
|
diff /tmp/sec-baseline-server.txt /tmp/sec-after-server.txt \
|
|
| head -40
|
|
# Expect: many ✓ lines that were ✗, score dropped
|
|
|
|
# 3. Run smoke matrix (next section)
|
|
```
|
|
|
|
**Smoke matrix (run after Test 7 settles):**
|
|
|
|
```bash
|
|
# S1: server is responsive
|
|
sudo systemctl status left4me-server@1 --no-pager | head -10
|
|
# Active (running), recent green
|
|
|
|
# S2: srcds is in-game
|
|
PID=$(pgrep -f 'srcds_linux.*left4dead2' | head -1)
|
|
[ -n "$PID" ] && echo "OK: srcds PID $PID" || echo "FAIL"
|
|
|
|
# S3: from outside, RCON responds
|
|
# (do this from the operator's laptop or via the web UI)
|
|
|
|
# S4: workshop / overlay refresh path
|
|
# (trigger from web UI; verify the overlay rebuild succeeds — the
|
|
# script-sandbox is a SEPARATE unit, not affected by these changes,
|
|
# so any failure is in the web app's invocation path, not the
|
|
# sandbox itself.)
|
|
|
|
# S5: web app can still sudo helpers
|
|
# (trigger a server start/stop from the web UI; if the sudo path
|
|
# fails, the web app's hardening is too tight — but we haven't
|
|
# changed the web unit yet, so this should still work.)
|
|
|
|
# S6: log streaming works
|
|
# (open the web UI's log view for server@1; verify lines flow.)
|
|
|
|
# S7: file upload to overlay
|
|
# (upload a small file via the file-tree endpoint; verify it
|
|
# appears on disk in /var/lib/left4me/overlays/<id>/.)
|
|
|
|
# S8: peer server unaffected
|
|
sudo systemctl is-active left4me-server@2
|
|
# active (we didn't touch it)
|
|
```
|
|
|
|
**Pass criteria:** all smoke items pass. systemd-analyze score
|
|
dropped significantly.
|
|
|
|
**Failure handling:** if anything in the smoke fails, identify which
|
|
directive caused it by removing them one at a time until smoke
|
|
passes. Document the offender.
|
|
|
|
**DO NOT cleanup yet** — leave Test 7 in place for Test 8.
|
|
|
|
---
|
|
|
|
## Test 8 — Attack verification (the audit gaps)
|
|
|
|
**Goal:** Confirm the threat-model defenses (D1, D2, D3, D5) actually
|
|
work end-to-end.
|
|
|
|
**Pre-condition:** Test 7's drop-in still in place.
|
|
|
|
**Verify:**
|
|
```bash
|
|
PID=$(pgrep -f 'srcds_linux.*left4dead2' | head -1)
|
|
GUNICORN_PID=$(pgrep -f 'gunicorn.*l4d2web' | head -1)
|
|
|
|
# D1.a — srcds cannot read DB
|
|
sudo nsenter --target $PID --mount -- cat /var/lib/left4me/left4me.db 2>&1 | head -1
|
|
# Expect: cat: /var/lib/left4me/left4me.db: No such file or directory
|
|
|
|
# D1.b — srcds cannot read web.env
|
|
sudo nsenter --target $PID --mount -- cat /etc/left4me/web.env 2>&1 | head -1
|
|
# Expect: cat: /etc/left4me/web.env: No such file or directory
|
|
|
|
# D1.c — srcds cannot read its own past
|
|
sudo nsenter --target $PID --mount -- ls /opt 2>&1 | head -5
|
|
# Expect: empty listing or No such file or directory
|
|
|
|
# D2.a — srcds cannot ptrace gunicorn (syscall filter)
|
|
sudo nsenter --target $PID --mount -- /usr/bin/gdb --batch -p $GUNICORN_PID 2>&1 | tail -3
|
|
# Expect: Operation not permitted
|
|
|
|
# D2.b — srcds cannot read /proc/<gunicorn>/environ
|
|
sudo nsenter --target $PID --mount -- cat /proc/$GUNICORN_PID/environ 2>&1 | head -1
|
|
# Expect: No such file or directory (ProtectProc=invisible)
|
|
|
|
# D2.c — srcds cannot read /proc/<gunicorn>/mem
|
|
sudo nsenter --target $PID --mount -- cat /proc/$GUNICORN_PID/mem 2>&1 | head -1
|
|
# Expect: No such file or directory
|
|
|
|
# D3 — srcds cannot use sudo helpers (NoNewPrivileges blocks setuid)
|
|
sudo nsenter --target $PID --mount -- sudo -n /usr/local/libexec/left4me/left4me-systemctl show server@2 2>&1 | head -3
|
|
# Expect: a sudo error about no new privileges, or operation not permitted
|
|
|
|
# D5 — server@1 cannot ptrace server@2's srcds
|
|
PID2=$(pgrep -f 'srcds_linux.*\@2' | head -1)
|
|
[ -n "$PID2" ] && sudo nsenter --target $PID --mount -- /usr/bin/gdb --batch -p $PID2 2>&1 | tail -3
|
|
# Expect: Operation not permitted (cross-instance userns OR syscall filter)
|
|
|
|
# Bonus — confirm PrivateUsers is in effect
|
|
sudo readlink /proc/$PID/ns/user
|
|
sudo readlink /proc/1/ns/user
|
|
# Expect: different
|
|
```
|
|
|
|
**Pass criteria:** every attack vector returns an error.
|
|
|
|
**Cleanup:** **Do not remove the drop-in yet** — leave it for Test 9.
|
|
|
|
---
|
|
|
|
## Test 9 — System-wide sysctl: `kernel.yama.ptrace_scope=2`
|
|
|
|
**Goal:** Add belt-and-braces system-wide.
|
|
|
|
**Apply:**
|
|
```bash
|
|
sudo tee /etc/sysctl.d/99-left4me-ptrace.conf <<'EOF'
|
|
# Block ptrace except from root (CAP_SYS_PTRACE).
|
|
# Combined with SystemCallFilter=~@debug + PrivateUsers=true in the
|
|
# unit, this gives defense-in-depth at three levels.
|
|
kernel.yama.ptrace_scope=2
|
|
EOF
|
|
sudo sysctl --system | grep yama
|
|
# Expect: kernel.yama.ptrace_scope = 2
|
|
sysctl kernel.yama.ptrace_scope
|
|
# Expect: 2
|
|
```
|
|
|
|
**Verify:**
|
|
```bash
|
|
# As left4me (no caps), gdb attach to gunicorn from OUTSIDE the unit's
|
|
# namespace
|
|
sudo -u left4me /usr/bin/gdb --batch -p $GUNICORN_PID 2>&1 | tail -3
|
|
# Expect: Operation not permitted
|
|
|
|
# Operator gdb (as root) still works:
|
|
sudo /usr/bin/gdb --batch -ex "info threads" -p $GUNICORN_PID 2>&1 | tail -10
|
|
# Expect: gdb output (debugging is admin-only now)
|
|
```
|
|
|
|
**Pass criteria:** non-root can't ptrace anything; root still can.
|
|
|
|
**No cleanup** — this is permanent (commit to /etc/sysctl.d/).
|
|
|
|
---
|
|
|
|
## Test 10 — Web unit hardening (carefully)
|
|
|
|
**Goal:** Apply non-sudo-breaking directives to `left4me-web.service`.
|
|
|
|
**Pre-condition:** Test 7's server drop-in still in place. Web is at
|
|
baseline.
|
|
|
|
**Drop-in:**
|
|
```bash
|
|
sudo install -d -m0755 /etc/systemd/system/left4me-web.service.d/
|
|
sudo tee /etc/systemd/system/left4me-web.service.d/test-10-web.conf <<'EOF'
|
|
[Service]
|
|
# (NoNewPrivileges intentionally NOT set — web sudoes to helpers.)
|
|
# (PrivateUsers intentionally NOT set — would break sudo's setuid.)
|
|
# (CapabilityBoundingSet not set — sudo + PAM need caps.)
|
|
|
|
ProtectSystem=strict
|
|
ProtectHome=true
|
|
LockPersonality=true
|
|
UMask=0027
|
|
|
|
# /proc + kernel
|
|
ProtectProc=invisible
|
|
ProcSubset=pid
|
|
ProtectKernelTunables=true
|
|
ProtectKernelModules=true
|
|
ProtectKernelLogs=true
|
|
ProtectClock=true
|
|
ProtectControlGroups=true
|
|
ProtectHostname=true
|
|
|
|
# Syscall (no ~@privileged — sudo needs setuid/etc.)
|
|
SystemCallArchitectures=native
|
|
SystemCallFilter=@system-service
|
|
SystemCallFilter=~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete
|
|
|
|
# Network
|
|
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
|
|
|
|
# Misc
|
|
RestrictNamespaces=true
|
|
RestrictRealtime=true
|
|
RemoveIPC=true
|
|
EOF
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-web
|
|
```
|
|
|
|
**Verify:**
|
|
```bash
|
|
# 1. Web up
|
|
sudo systemctl is-active left4me-web
|
|
|
|
# 2. Web responds (curl from the host)
|
|
curl -sI http://127.0.0.1:8000/ | head -5
|
|
# Expect: HTTP/1.1 200 or similar (whatever the default route is)
|
|
|
|
# 3. Web sudo path works — trigger from operator's laptop, watching the
|
|
# web UI. Start/stop a server; observe success.
|
|
|
|
# 4. systemd-analyze score
|
|
sudo systemd-analyze security left4me-web.service \
|
|
| tee /tmp/sec-after-web.txt
|
|
diff /tmp/sec-baseline-web.txt /tmp/sec-after-web.txt | head -20
|
|
|
|
# 5. Web cannot ptrace srcds (D4)
|
|
WEB_PID=$(pgrep -f 'gunicorn.*l4d2web' | head -1)
|
|
sudo -u left4me /usr/bin/gdb --batch -p $PID 2>&1 | tail -3
|
|
# (might still succeed if the operator runs as root — what matters is
|
|
# from inside the web unit's namespace)
|
|
sudo nsenter --target $WEB_PID --mount -- /usr/bin/gdb --batch -p $PID 2>&1 | tail -3
|
|
# Expect: Operation not permitted (SystemCallFilter blocks ptrace)
|
|
```
|
|
|
|
**Pass criteria:** all of above.
|
|
|
|
**Failure handling:** if sudo from web breaks, remove the most likely
|
|
culprit (probably one of the SystemCallFilter lines being too tight).
|
|
Most likely candidate: `~@debug` could block `process_vm_readv` which
|
|
sudo doesn't use, but `~@privileged` is not on the web filter so sudo's
|
|
setuid is OK.
|
|
|
|
**Cleanup:**
|
|
```bash
|
|
sudo rm /etc/systemd/system/left4me-web.service.d/test-10-web.conf
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-web
|
|
```
|
|
|
|
(Web reverts to baseline. Server drop-in stays for the report.)
|
|
|
|
---
|
|
|
|
## Test 11 — Soak test
|
|
|
|
**Goal:** Run the composition for an extended period to surface
|
|
race-condition or workload-dependent issues.
|
|
|
|
**Pre-condition:** Test 7 drop-in on server@1; Test 9 sysctl in place.
|
|
|
|
**Procedure:**
|
|
```bash
|
|
# Run for 24-48 hours; observe:
|
|
sudo journalctl -u left4me-server@1 --since '24 hours ago' \
|
|
| grep -iE 'seccomp|denied|EACCES|EPERM' | wc -l
|
|
# Expect: 0 or a very small number (some EACCES on benign probes
|
|
# are normal)
|
|
|
|
sudo journalctl -u left4me-server@1 -k --since '24 hours ago' \
|
|
| grep 'type=1326' | wc -l
|
|
# Expect: 0
|
|
|
|
sudo systemctl status left4me-server@1
|
|
# Expect: active, no restarts since start
|
|
```
|
|
|
|
**Pass criteria:** no SECCOMP kills over the soak period, no
|
|
unexpected restarts.
|
|
|
|
---
|
|
|
|
## Cleanup (after all tests pass)
|
|
|
|
```bash
|
|
# Remove all test drop-ins
|
|
sudo rm -rf /etc/systemd/system/left4me-server@1.service.d/test-*.conf
|
|
sudo rm -rf /etc/systemd/system/left4me-web.service.d/test-*.conf
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart left4me-server@1 left4me-web
|
|
sudo systemctl is-active left4me-server@1 left4me-web # both active
|
|
|
|
# Sysctl from Test 9 STAYS in place.
|
|
|
|
# Remove temp files
|
|
rm /tmp/sec-baseline-*.txt /tmp/sec-after-*.txt
|
|
rm /tmp/unit-baseline-*.conf
|
|
rm /tmp/syscall-log-*.txt
|
|
```
|
|
|
|
---
|
|
|
|
## Results template
|
|
|
|
Append the executing session's findings here. One paragraph per test.
|
|
|
|
### Test 1 — PrivateUsers
|
|
- Pass / fail: TBD
|
|
- Notes:
|
|
|
|
### Test 2 — TemporaryFileSystem + binds
|
|
- Pass / fail: TBD
|
|
- Notes:
|
|
|
|
### Test 3 — SystemCallLog discovery
|
|
- Pass / fail: TBD
|
|
- Syscalls observed under load (if any from @debug/@mount/@privileged):
|
|
- Notes:
|
|
|
|
### Test 4 — SystemCallFilter enforcement
|
|
- Pass / fail: TBD
|
|
- If filter had to be relaxed, which group:
|
|
- Notes:
|
|
|
|
### Test 5 — ProcSubset + ProtectProc
|
|
- Pass / fail: TBD
|
|
- Notes:
|
|
|
|
### Test 6 — MemoryDenyWriteExecute
|
|
- Pass / fail: TBD (likely fail; document the failure mode)
|
|
- Notes:
|
|
|
|
### Test 7 — Full composition
|
|
- Pass / fail: TBD
|
|
- systemd-analyze score before/after:
|
|
- Notes:
|
|
|
|
### Test 8 — Attack verification
|
|
- Pass / fail: TBD
|
|
- Per-vector results (D1.a, D1.b, ..., D5):
|
|
|
|
### Test 9 — Yama ptrace_scope=2
|
|
- Applied: TBD
|
|
- Operator workflow impact noted:
|
|
|
|
### Test 10 — Web hardening
|
|
- Pass / fail: TBD
|
|
- Sudo path verified working:
|
|
- systemd-analyze score before/after:
|
|
|
|
### Test 11 — Soak
|
|
- Duration:
|
|
- Issues observed:
|
|
|
|
---
|
|
|
|
## Output of this test plan
|
|
|
|
When all tests complete:
|
|
1. Mark this document with **status: tested** and record the dates.
|
|
2. Open a new implementation plan
|
|
(`docs/superpowers/plans/2026-MM-DD-hardening-refactor.md`) that
|
|
commits the proven composition to the ckn-bw reactor + reference
|
|
units + test suite.
|
|
3. Decide on the deferred questions:
|
|
- 3-user uid split — necessary or covered by hardening?
|
|
- AppArmor profile follow-up — pursue or close?
|
|
- `MemoryDenyWriteExecute=true` — include if Test 6 passed?
|
|
- `SocketBindAllow=` — add to lock the gameserver port range?
|
|
4. Mark `2026-05-15-user-uid-split-design.md` as superseded or closed
|
|
per the answer to the previous bullet.
|
|
|
|
## Pointers
|
|
|
|
- Threat model: `docs/superpowers/specs/2026-05-15-hardening-threat-model.md`
|
|
- Defenses survey: `docs/superpowers/specs/2026-05-15-hardening-defenses-survey.md`
|
|
- Live unit source: `~/Projekte/ckn-bw/bundles/left4me/metadata.py:150+`
|
|
- Reference units: `deploy/files/usr/local/lib/systemd/system/`
|
|
- Tools needed on `left4.me`:
|
|
- `systemd-analyze` (in `systemd` package, already installed)
|
|
- `scmp_sys_resolver` (in `libseccomp-dev`; install on demand for
|
|
Test 3/4 if filters need analysis)
|
|
- `gdb` (for ptrace tests; install on demand)
|
|
- `nsenter` (in `util-linux`, already installed)
|
|
- `findmnt`, `pgrep`, standard userspace
|