Compare commits

...

58 commits

Author SHA1 Message Date
mwiegand
0d906605e9
chore: add direnv .envrc for local Python 3.13 venv
Pins to python3.13 to match the Debian Trixie production target.
Documents the dev setup in README and AGENTS.md so a fresh checkout
gets a working `python` via `direnv allow` + editable installs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 18:56:51 +02:00
mwiegand
196d2db33e
feat(l4d2-web): seed example script overlays from examples/script-overlays/
Bundles four reference script overlays (cedapug_maps, l4d2center_maps,
competitive_rework, tickrate) and adds a `flask seed-script-overlays`
CLI that upserts each *.sh as a system-wide overlay. Test deploy
invokes it after the orphan-cleanup migration so fresh test servers
come up with the same overlays the user has been maintaining by hand.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 18:41:08 +02:00
mwiegand
6b4eef22c2
feat: server Reset action — wipe runtime, keep DB row
Reset stops the systemd service, unmounts the overlay, and rm -rf's both
runtime/<name> and instances/<name>, but keeps the Server row, blueprint,
and (shared) systemd template. Next Start re-initializes from the current
blueprint, so users can clean up logs/caches/accumulated game state without
losing the server.

Implementation factors a shared _purge_instance helper out of
delete_instance; reset_instance reuses it without the existence guard. New
"reset" lifecycle op flows through the same route + worker + facade plumbing
as the other server ops.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 18:10:32 +02:00
mwiegand
c8a2d563ce
fix(l4d2-web): server delete job now removes the DB row
The delete job ran l4d2ctl delete (host-side cleanup) but never removed the
Server row, so deleted servers kept appearing on /servers. Hard-delete the
row in the worker's success path and skip the post-op status refresh, since
the systemd unit is gone.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 18:09:45 +02:00
mwiegand
fb3c6be052
feat(l4d2-web): per-overlay job list + redirect to job after build-triggering edits
Saving a script overlay or adding/removing workshop items now redirects to the
enqueued build job's detail page so logs are immediately visible. Added a new
/overlays/<id>/jobs page (linked as "all builds →" from the overlay detail
page) for browsing the full build history. Renamed the script "Save" button to
"Save and build" to make the side effect explicit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:44:22 +02:00
mwiegand
5e2c771276
chore(l4d2-web): remove orphaned 'Global map overlays' admin section
The route /admin/global-overlays/refresh was removed with the script-overlays
rewrite (migration 0005 dropped the global_overlay_* tables; the systemd
refresh units were deleted from deploy/). The admin-page form was left
behind and would 404 on submit. Drop the section and lock it out with an
assertion in the existing admin-pages test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:25:15 +02:00
mwiegand
ebddb0fab2
chore(deploy): install p7zip + coreutils for script-overlay tooling
Script overlays commonly need 7z and md5sum (e.g. the l4d2center map
sync recipe). Add p7zip-full to the apt install line, p7zip + p7zip-plugins
to dnf, and coreutils explicitly so md5sum is guaranteed even on slim base
images. Lock both in with a regression test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:23:23 +02:00
mwiegand
406f2196f8
fix(l4d2-web): write sandbox script tmpfile under LEFT4ME_ROOT, not /tmp
The web service unit has PrivateTmp=yes: its /tmp is a per-instance
namespace at /tmp/systemd-private-X-left4me-web.service-Y/tmp/ from
PID 1's perspective. When ScriptBuilder writes /tmp/tmpXXX.sh and
passes that path to the sandbox helper, systemd-run asks PID 1 to set
up BindReadOnlyPaths=${SCRIPT}:/script.sh — but PID 1 lives in the host
namespace and can't resolve the web service's PrivateTmp path. The
unit fails to start with status=226/NAMESPACE and "Failed to set up
mount namespacing: /script.sh: No such file or directory".

Move the tmpfile to ${LEFT4ME_ROOT}/sandbox-scripts/. /var/lib is not
affected by PrivateTmp (only /tmp and /var/tmp are), so PID 1 can
resolve the path. The web service has ReadWritePaths=/var/lib/left4me
already, and the directory is created on demand by Python.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:14:21 +02:00
mwiegand
023cc5c9b0
fix(deploy): chown WAL+SHM sidecars too, not just left4me.db
SQLite in WAL mode (the default for this app) maintains left4me.db-wal
and left4me.db-shm sidecar files alongside the main DB. All three must
be writable by the web service uid; if any one is root-owned, SQLite
reports "attempt to write a readonly database" on the next INSERT —
which surfaced as a 500 on POST /overlays/{id}/script after I'd done
ad-hoc root-side sqlite3.connect() inspection earlier and the resulting
root-owned WAL/SHM persisted.

Loop over all three paths in the deploy chmod step so root-owned
sidecars are corrected on every deploy. Idempotent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:11:42 +02:00
mwiegand
f6ca85fc6f
fix(deploy): chown left4me.db to left4me:left4me, not root:left4me
The v2 hardening tightened the DB to mode 0640 owned by root:left4me,
intending to block reads from the sandbox uid (l4d2-sandbox, not in the
left4me group). It did — but it also took away write access from the web
service itself, which runs as user left4me. With root owning the file,
left4me only had group-read; INSERTs into the jobs table failed with
"attempt to write a readonly database" and surfaced as a 500 on POST
/overlays/{id}/script.

Owner left4me + group left4me + mode 0640 keeps the same external
posture (l4d2-sandbox gets nothing via "other") while restoring the
web service's read+write access via "owner".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:09:47 +02:00
mwiegand
abc907b14b
docs(specs): script sandbox v3 — egress filter design + plan
Captures the v3 design: IPAddressDeny= alone (no IPAddressAllow=any
because the documented "more specific wins" semantics don't hold on
systemd 257 / kernel 6.12 — the allow trumps unconditionally), explicit
CIDRs (the -p parser rejects the localhost/link-local shorthand
keywords), and a static sandbox-only resolv.conf bind to keep DNS
reachable when private RFC1918 ranges are blocked.

Plan documents what was implemented (in 7e66936) and the lessons
surfaced during execution so the next person doesn't have to rediscover
them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:08:47 +02:00
mwiegand
7e66936d03
feat(deploy): restrict script-sandbox egress to public internet only
Adds IPAddressDeny= to the sandbox unit covering loopback (127/8 + ::1),
link-local (169.254/16 + fe80::/10), multicast (224/4 + ff00::/8), all
RFC1918 v4 (10/8, 172.16/12, 192.168/16), CGNAT (100.64/10), and ULA v6
(fc00::/7). The kernel attaches systemd's sd_fw_egress BPF program to
the unit's cgroup; egress packets matching any of the deny prefixes are
silently dropped at the cgroup boundary.

Important: do NOT pair this with `IPAddressAllow=any`. Documentation
claims "more specific rule wins" but on this systemd 257 + kernel 6.12
combo, having both set causes the allow to win unconditionally — the
deny gets ignored. Empty IPAddressAllow + populated IPAddressDeny is the
correct shape: kernel default "allow all" applies to non-listed
addresses, and the listed prefixes are blocked.

Because the host's resolv.conf typically points at a private-IP DNS
server (10.0.0.1 in the test deploy), blocking RFC1918 also kills DNS.
Adds a static /etc/left4me/sandbox-resolv.conf with public resolvers
(Cloudflare 1.1.1.1, Google 8.8.8.8) and bind-mounts that into the
sandbox at /etc/resolv.conf, replacing the host's resolver inside the
sandbox only.

Smoke-tested on ckn@10.0.4.128:
- public 1.1.1.1:443: CONNECTED
- public HTTPS via DNS (steamcommunity.com): 200
- localhost web app 127.0.0.1:8000: blocked (TimeoutError)
- localhost sshd 127.0.0.1:22: blocked
- private LAN ssh 10.0.4.128:22: blocked
- private DNS 10.0.0.1:53: blocked

AF_UNIX stays in RestrictAddressFamilies — dropping it would risk
breaking NSS / syslog for marginal gain, and the IP-level filter
addresses the primary threat (reaching the host's HTTP/SSH services).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:04:57 +02:00
mwiegand
ae443299c8
chore(deploy): drop bubblewrap apt dep + tighten left4me.db mode
bubblewrap is no longer used now that left4me-script-sandbox runs as a
systemd service unit. Remove it from the apt-get and dnf install lines.

Also tighten the application database file mode after the alembic
upgrade step: chown root:left4me, chmod 0640. The DB had been created
at default 0644 by SQLite's open() call inside the web service, which
made it world-readable on the host — i.e. readable by any uid that can
traverse /var/lib/left4me, including the sandbox's l4d2-sandbox uid.
Smoke-testing the v2 sandbox prototype on ckn@10.0.4.128 surfaced this:
the sandbox could read "SQLite format 3" from the DB until the parent
dir was masked with TemporaryFileSystem=. Tightening the file mode is
the host-level fix; the sandbox-level mask is defense in depth.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 16:48:26 +02:00
mwiegand
4ee8f6af44
refactor(deploy): rewrite left4me-script-sandbox to systemd-only — drop bwrap
Replaces the systemd-run --scope + bwrap composition with systemd-run in
service-unit mode (--pipe --wait, transient .service unit). Same cgroup
limits and walltime kill, plus the hardening directives that --scope
units cannot carry: NoNewPrivileges, ProtectSystem=strict, ProtectHome,
ProtectKernel{Tunables,Modules,Logs,ControlGroups}, RestrictNamespaces,
RestrictAddressFamilies, RestrictSUIDSGID, LockPersonality,
MemoryDenyWriteExecute, SystemCallFilter (seccomp), and an empty
CapabilityBoundingSet (drops all caps). UID drop via User=/Group=.

The TemporaryFileSystem="/etc /var/lib" pair is the gotcha:
ProtectSystem=strict makes /var/lib *read-only* but visible, so the host
DB at /var/lib/left4me/left4me.db (mode 0644) was readable from inside.
Masking /var/lib with tmpfs hides the entire subtree; the BindPaths bind
to /overlay is at a different path and unaffected.

The Python side (ScriptBuilder, run_sandboxed_script, routes) is
unchanged — same sudo-helper invocation, same argv shape.

Loses PID-namespace isolation (no PrivatePID= directive in systemd).
Host PIDs are visible via /proc and ps -ef but not signal-able due to
UID mismatch — information disclosure only, not a privilege boundary.

Smoke-tested on ckn@10.0.4.128 prior to this commit; all isolation
invariants reproduced and the hardening directives provably blocked
unshare(2), mount(2), personality(2), bpf(2), and sysctl writes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 16:47:30 +02:00
mwiegand
efaaf84cd9
docs(specs): script sandbox v2 — systemd-only design + plan
Spec captures the v2 architecture (systemd-run service mode with full
hardening directives, no bwrap), the two surfaces in scope (helper
rewrite + bubblewrap dep removal + left4me.db mode tightening), and the
gotchas surfaced by smoke-testing the prototype on ckn@10.0.4.128:
- ProtectSystem=strict makes /var/lib/left4me visible (not invisible);
  must add TemporaryFileSystem=/var/lib to mask it.
- Script bind via BindReadOnlyPaths uses ${SCRIPT}:/script.sh syntax.
- No PrivatePID= directive in systemd; host PIDs visible via /proc.
  Information disclosure only — kernel UID-mismatch blocks signals.

Plan breaks the migration into 4 tasks (helper rewrite, deploy-script
deps + DB mode, host smoke-test, drift sweep) with explicit rollback.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 16:46:13 +02:00
mwiegand
a62f26ba4a
fix(l4d2-web): normalize CRLF to LF in script overlay POST
HTML <textarea> form submission encodes line breaks as CRLF per spec.
Storing those CRLFs unchanged means every line of the script reaches
bash with a trailing \r, which bash treats as part of the argument —
turning "ls /" into "ls /\r" and failing. Normalize CRLF/CR → LF in the
/overlays/{id}/script handler so storage and the sandbox tmpfile are
LF-only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 16:20:10 +02:00
mwiegand
908bca3687
fix(l4d2-web): ScriptBuilder — chmod script tmpfile to 0644 for sandbox read
NamedTemporaryFile creates the script file at mode 0600 owned by the
left4me web user. The sandbox runs as l4d2-sandbox and bwrap bind-mounts
the file read-only at /script.sh, but the kernel still enforces the
underlying file's permissions — l4d2-sandbox can't read 0600 left4me
files, so /bin/bash /script.sh fails with "Permission denied".

Script content is not a secret (it's stored in the DB and editable by
the user), so 0644 is appropriate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 16:18:00 +02:00
mwiegand
cf865d4915
fix(deploy): one-shot cleanup of orphan overlay dirs after globals removal
Migration 0005_script_overlays drops the legacy l4d2center_maps /
cedapug_maps overlay rows but leaves their /var/lib/left4me/overlays/{id}
directories on disk. When the web app subsequently creates a new overlay
and AUTOINCREMENT issues an id matching one of those orphans,
create_overlay_directory(exist_ok=False) crashes with FileExistsError —
which surfaced as a 500 on POST /overlays the first time a script
overlay was created on a deployed test box.

Adds a sentinel-gated sweep in deploy-test-server.sh that lists overlay
ids in the DB, removes any directory under overlays/ whose id has no
matching row, and drops the now-unused global_overlay_cache. Mirrors the
.kernel-overlay-migrated sentinel pattern so reruns are no-ops.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 16:16:33 +02:00
mwiegand
06ae84fbe4
fix(deploy): script-sandbox helper — UID drop via systemd-run, --unshare-user-try, /etc/alternatives
Smoke testing on the test host revealed three issues with the helper as
shipped:

1. bwrap 0.11+ rejects --uid without --unshare-user. Switching the UID
   drop from inside bwrap to systemd-run (--uid=l4d2-sandbox
   --gid=l4d2-sandbox) sidesteps the userns UID-mapping headaches and
   keeps file ownership on the bind-mounted /overlay matching
   l4d2-sandbox on the host (which the wipe path relies on).

2. bwrap running as an unprivileged uid still needs a user namespace to
   set up its mount-namespace bind-mounts. Adding --unshare-user-try
   gives it the userns context when needed and is a no-op otherwise.

3. /etc/alternatives wasn't bind-mounted, so symlinked tools like
   /usr/bin/awk -> /etc/alternatives/awk fell over inside the sandbox.
   Adds the ro-bind.

Also: the helper now chowns the overlay dir to l4d2-sandbox before bwrap
(idempotent — needed because the web app creates the dir as left4me),
and the deploy script chmods /var/lib/left4me to 0711 so l4d2-sandbox
can traverse to the bind-mount source.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 16:12:46 +02:00
mwiegand
1e62a44c16
docs(deploy): replace globals overlay description with script overlays
deploy/README.md still described the deleted managed-global overlays as
the second overlay surface. Replace with a description of script
overlays (bubblewrap + systemd-run sandbox, resource caps).

Full test sweep: 367 passing, 2 skipped across l4d2web, l4d2host, deploy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:56:24 +02:00
mwiegand
e51a4d58a4
chore(deploy): provision l4d2-sandbox + bubblewrap; drop globals refresh timer
deploy-test-server.sh: provisions the l4d2-sandbox system user (no home,
nologin shell) and installs the bubblewrap apt/dnf package; copies the
left4me-script-sandbox helper into /usr/local/libexec/left4me with mode
0755. Drops the global_overlay_cache directory provisioning, the
refresh-global-overlays unit installation, and the timer enable.

Deletes the orphaned left4me-refresh-global-overlays.{service,timer}
files. Trims the matching paragraph from deploy/README.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:54:57 +02:00
mwiegand
75e703e1a4
feat(deploy): left4me-script-sandbox helper + sudoers fragment
Privileged bash helper that wraps user-authored scripts in
systemd-run --scope (cgroup limits + RuntimeMaxSec=3600) inside a
bubblewrap sandbox dropped to the l4d2-sandbox uid. Network is shared
with the host so scripts can fetch from Steam / l4d2center / etc.;
filesystem is RO except for /overlay (rw bind from
/var/lib/left4me/overlays/{id}) and tmpfs /tmp + /run.

Adds a sudoers rule allowing the left4me user to invoke this helper
without restrictions on its arguments. Strict argument validation is
in the helper itself.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:53:21 +02:00
mwiegand
d351bcbee5
feat(l4d2-web): script overlay UI
Adds the script type to the create-overlay modal (with an admin-only
system-wide checkbox) and a script-section to the detail page: textarea
for the bash body, Save / Rebuild / Wipe buttons, last_build_status
badge, latest-build-job link, and a Wipe confirm modal. Removes the
GlobalOverlaySource block.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:50:36 +02:00
mwiegand
be22744d54
feat(l4d2-web): script overlay routes (script update / wipe / build)
Adds POST /overlays/{id}/script, /wipe, /build under the overlay blueprint.
Generalizes /build to handle any owner/admin-editable overlay (deletes the
duplicate workshop-specific manual_build). Wipe runs the literal script
"find /overlay -mindepth 1 -delete" through run_sandboxed_script and
refuses with 409 while a build_overlay job is running. Adds an
admin-only system_wide=1 flag to POST /overlays for system-wide creation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:48:15 +02:00
mwiegand
879c54cbda
refactor(l4d2-web): drop refresh_global_overlays from scheduler
GLOBAL_OPERATIONS becomes {"install", "refresh_workshop_items"}.
Removes refresh_global_overlays_running from SchedulerState and the
_run_refresh_global_overlays dispatch. Drops dead test cases and pins
GLOBAL_OPERATIONS contents.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:45:34 +02:00
mwiegand
9f476e3456
refactor(l4d2-web): drop global-overlays subsystem in favor of script type
Deletes the global_map_sources, global_overlay_refresh, global_map_cache,
and global_overlays service modules and their tests. Removes the
refresh-global-overlays CLI command, the /admin/global-overlays/refresh
route, and the GlobalOverlaySource view in overlay_detail rendering.
Drops py7zr from dependencies — was only used by the deleted subsystem.

The job_worker scheduler still tracks refresh_global_overlays; that
cleanup is Task 4. Deploy/README references are Task 8.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:43:41 +02:00
mwiegand
d29afa41fa
feat(l4d2-web): ScriptBuilder + BUILDERS registry update
Adds ScriptBuilder that runs user-authored bash inside the
left4me-script-sandbox helper via run_command, with a 20 GB post-build
disk cap. Registry now {"workshop", "script"}.
finish_job writes Overlay.last_build_status on build_overlay completion.
Drops GlobalMapOverlayBuilder and the now-unreachable
_check_global_overlay_caches in l4d2_facade.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:39:13 +02:00
mwiegand
43dc9b0ccf
feat(l4d2-web): script overlay schema — add overlay.script + last_build_status, drop globals tables
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:33:04 +02:00
mwiegand
78ead0b41d
docs(specs): script overlay type — design + implementation plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:27:14 +02:00
mwiegand
9985ecc56c
chore(deploy): cleanup left4me-web hardening + docs for kernel overlayfs
Drop MountFlags=shared (the assumption that it propagated fuse mounts
to host was incorrect on systemd 257 with ProtectSystem+ReadWritePaths).
Restore PrivateTmp=true (was dropped in 593611e for fuse propagation
that did not work). Rewrite the comment block to describe the new
model: mounts go through the left4me-overlay helper which nsenters
into PID 1's mount namespace, so the unit's mount-ns layout is no
longer load-bearing.

Update the three user-facing READMEs (root, l4d2host, deploy) to drop
fuse-overlayfs / fusermount3 prereqs and call out the kernel overlayfs
mount path through the privileged helper.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:29:49 +02:00
mwiegand
172e574a00
chore(deploy): drop fuse-overlayfs apt dep + one-shot migrate upper/work
Drop fuse-overlayfs / fuse3 from the apt/dnf install line — the new
mount path is kernel overlayfs via the left4me-overlay helper, no
fuse userspace needed.

Add a one-shot migration block gated by /var/lib/left4me/.kernel-overlay-migrated
that runs before daemon-reload: stop gameservers + web service, force-
unmount any leftover fuse or overlay mounts under runtime/, then wipe
and recreate empty upper/ and work/ for every instance. fuse-overlayfs
running as a non-root user used user.fuseoverlayfs.* xattrs that kernel
overlayfs ignores, so a pre-existing upper/ from the fuse era would
resurrect "deleted" files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:28:00 +02:00
mwiegand
93a60befb6
refactor(l4d2-host): start/stop/delete go through OverlayMounter; drop fuse module
Replace direct fuse-overlayfs / fusermount3 subprocess calls in
start_instance / stop_instance / delete_instance with the existing
OverlayMounter abstraction, now backed by KernelOverlayFSMounter.
Adds an os.path.ismount guard at the top of start_instance so a
kernel-level overlay that survived a web-worker crash isn't double-
mounted (kernel mounts persist when the cgroup dies, unlike fuse
daemons).

Delete the unused FuseOverlayFSMounter module.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:26:28 +02:00
mwiegand
d5b321b557
feat(l4d2-host): KernelOverlayFSMounter + left4me-overlay helper
New privileged helper at /usr/local/libexec/left4me/left4me-overlay
(Python, system /usr/bin/python3, stdlib only) takes only the instance
name, parses instance.env for L4D2_LOWERDIRS, validates each lowerdir
against an allowlist (installation/, overlays/, global_overlay_cache/,
workshop_cache/), refuses upperdirs tainted with user.fuseoverlayfs.*
xattrs from the prior fuse era, and execs `nsenter --mount=/proc/1/ns/mnt
-- mount -t overlay ...` so the resulting mount lives in the host
namespace. Mirrors the existing left4me-systemctl / left4me-journalctl
pattern; sudoers entry is verb-constrained.

KernelOverlayFSMounter implements the existing OverlayMounter ABC,
deriving the instance name from the merged path. No call sites use it
yet — that's the next commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:23:58 +02:00
mwiegand
db120d77d3
docs(specs): kernel overlayfs migration design + plan
Captures the architectural fix for the mount-propagation bug: replace
fuse-overlayfs (rootless mount inside the web service's namespace, never
visible to host or to gameserver units) with kernel-native overlayfs
mounted via a privileged sudo helper that nsenters into PID 1's mount
namespace. Companion plan numbers the migration as five tasks ending in
end-to-end verification on the test box.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:19:26 +02:00
mwiegand
d5d710afa7
fix(l4d2-host): make stop_instance idempotent on the unmount step
systemctl stop is already a no-op on a stopped unit, but stop_instance
was unconditionally running fusermount3 -u and bubbling up the EINVAL
when the overlay wasn't currently mounted (e.g. server already stopped).
Mirror the established delete_instance pattern: always attempt the
unmount, swallow CalledProcessError, and label the step "(if mounted)".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 11:24:04 +02:00
mwiegand
38548ab0d7
chore(deploy): raise gunicorn thread pool to 32 for SSE headroom
Each SSE log-viewer or job-log stream holds a thread for its full
lifetime. With --threads 8, a handful of open browser tabs could
exhaust the pool. 32 keeps the same single-process scheduler invariant
(_claim_lock in job_worker is process-local) while giving SSE plenty
of headroom on the test box's user count.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 11:19:03 +02:00
mwiegand
4552af6544
fix(l4d2-web): keep SSE log stream from pinning gunicorn threads
stream_command used a blocking proc.stdout.readline() that never woke
when the underlying journalctl was silent, so Flask never delivered
GeneratorExit on client disconnect — the worker thread and the journalctl
child both leaked permanently and pinned the gunicorn thread pool.

Switch to a select-based read loop with a 15s heartbeat tick (yielded as
""), and translate the tick to an SSE keepalive comment in the log route.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 11:18:56 +02:00
mwiegand
ffc4cdbd7d
refactor(l4d2-web): remove legacy external overlay type
The workshop + managed-global overlay surface fully covers the
admin-SFTP flow that 'external' was a placeholder for. Drop the type
from the model defaults, builder registry, routes, template, and
tests, and add migration 0004 that deletes any leftover external
rows along with their blueprint and job references.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 09:31:04 +02:00
mwiegand
92d6ebbe82
feat(l4d2-web): managed global map overlays with daily refresh
Adds two managed system overlays (l4d2center-maps, cedapug-maps) that
fetch curated map archives from upstream sources and reconcile addons
symlinks for non-Steam maps. A daily systemd timer enqueues a coalesced
refresh_global_overlays worker job; downloads, extraction, and rebuilds
run in the existing job worker and surface in the job log UI.

Schema: GlobalOverlaySource / GlobalOverlayItem / GlobalOverlayItemFile
plus nullable Job.user_id so system jobs render as "system" in the UI.
The new builder reconciles symlinks against the per-source vpk cache
and leaves foreign symlinks untouched. Initialize-time guard refuses
to mount a partial overlay if any expected vpk is missing from cache.

Refresh service uses shutil.move to handle EXDEV when /tmp and the
cache live on different filesystems.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 08:05:14 +02:00
mwiegand
4f78574edd
fix(l4d2-web): keep workshop refresh responsive
Limit workshop refresh downloads to one worker and commit build-overlay enqueue work before writing the final job log so SQLite locks do not wedge the web process.
2026-05-07 17:31:12 +02:00
mwiegand
0e83ee07d7
fix(deploy): make test deployments safe to rerun
Exclude local agent state from deploy archives, avoid recursive ownership over active runtime mounts, and let Alembic own schema upgrades before app startup.
2026-05-07 17:16:58 +02:00
mwiegand
b2a8d3d5e0
feat(deploy): workshop_cache provisioning
Adds /var/lib/left4me/workshop_cache to the deploy mkdir list (owned by
the left4me runtime user). Updates deploy/README.md to document the new
directory and the workshop overlay layout: web app downloads VPKs into
the cache and symlinks them into overlays/{overlay_id}/left4dead2/addons/.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 16:53:49 +02:00
mwiegand
ac020d1e77
feat(l4d2-web): initialize-time guard for uncached workshop items
Before invoking l4d2ctl initialize, run each blueprint overlay's builder
synchronously and then verify that every workshop item attached to the
blueprint has a cache file on disk. If any are missing, raise a clear
error naming the overlay and the missing steam_ids — server start can't
silently mount a partial overlay where some maps are mysteriously absent
in-game.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 16:53:04 +02:00
mwiegand
df1ccb4cca
feat(l4d2-web): workshop overlay UI (routes + templates)
Adds workshop_routes blueprint with add-items / remove-item / manual-
build endpoints plus admin /admin/workshop/refresh. Add-items handles
single ID, single URL, multi-line batch, or a collection ID; auto-
enqueues a coalesced build_overlay job per call. Reject non-L4D2 items
with 400, duplicate associations with friendly toast, intruders with
403.

Generalizes overlay_routes: type+name only on create (no path field);
external is admin-only and system-wide, workshop is per-user and
auto-pathed. Update is name-only. Delete recursively removes the
on-disk dir only for managed paths (path == str(id)); legacy externals
are left in place. The pre-existing in-use guard is preserved.

Page routes filter the overlay listing by user permissions and load
workshop items + the latest related job for the detail view.

Templates: unified Create modal with type radio (no path field).
Type-aware overlay detail: workshop overlays show a multi-line input
+ items/collection radio + item table partial with thumbnails, manual
Rebuild button, and a small status indicator pulled from the latest
related job. Admin page gets a "Refresh all workshop items" button.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 16:50:54 +02:00
mwiegand
38a6fbbe1e
feat(l4d2-web): worker support for build_overlay and refresh_workshop_items
Extends SchedulerState with running_overlays / refresh_running /
blocked_servers_by_overlay, and updates can_start with the truth table:
install and refresh_workshop_items are global mutexes; build_overlay
serializes per-overlay; server jobs block on builds for any overlay
their blueprint references.

Adds enqueue_build_overlay coalescing helper that returns an existing
queued job for the same overlay rather than inserting a duplicate.

Adds run_job dispatch for build_overlay (BUILDERS[overlay.type].build)
and refresh_workshop_items (re-fetches metadata, re-downloads on
time_updated/filename change, enqueues coalesced rebuilds for affected
overlays).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 16:44:10 +02:00
mwiegand
700940d578
feat(l4d2-web): overlay builder registry with workshop builder
Adds l4d2web/services/overlay_builders.py with a BUILDERS dict mapping
Overlay.type to a builder class. ExternalBuilder is a no-op that just
ensures the overlay directory exists. WorkshopBuilder diff-applies
absolute symlinks under left4dead2/addons/ against the overlay's current
WorkshopItem associations: creates new ones, removes obsolete, leaves
unrelated files alone, and skips uncached items with a warning rather
than producing dangling symlinks.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 16:40:30 +02:00
mwiegand
f0230e17d3
feat(l4d2-web): overlay path helpers and creation
Adds workshop_paths.cache_path(steam_id) returning
$LEFT4ME_ROOT/workshop_cache/{steam_id}.vpk with digit-only validation.

Adds overlay_creation.generate_overlay_path(id) and
create_overlay_directory(overlay) with exist_ok=False so a stray dir from a
prior failed delete surfaces loudly instead of shadowing fresh content.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 16:38:39 +02:00
mwiegand
c6b41429ee
feat(l4d2-web): steam workshop API client and downloader
Adds l4d2web/services/steam_workshop.py: parse_workshop_input (single ID,
URL, or multi-line batch), resolve_collection (HTTPS POST to
GetCollectionDetails), fetch_metadata_batch (HTTPS POST to
GetPublishedFileDetails with consumer_app_id == 550 enforcement that
raises WorkshopValidationError in add-mode and silently skips in
refresh-mode), download_to_cache (atomic + idempotent on mtime+size),
and refresh_all (ThreadPoolExecutor with per-item error collection).

Adds requests as an explicit dependency.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 16:37:39 +02:00
mwiegand
2543a05c12
feat(l4d2-web): typed overlays + workshop schema migration
Adds Overlay.type and Overlay.user_id with two partial unique indexes
(externals globally unique by name; user overlays unique per user).
Adds WorkshopItem registry keyed on steam_id and a pure many-to-many
overlay_workshop_items association. Adds Job.overlay_id for build_overlay
job tracking. Switches overlays.id to AUTOINCREMENT so deleted IDs are
never reused.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 16:35:13 +02:00
mwiegand
b46f52258d
docs(workshop): spec and plan for steam workshop overlays
Add a typed-overlay model with workshop as the first non-external type:
deduplicated WorkshopItem registry, symlink-based overlay directories,
auto-rebuild after item changes, admin global refresh, and a unified
Create-overlay UI with web-managed paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 16:25:13 +02:00
mwiegand
d18b397330
fix(host): create ~/.steam/sdk32 and sdk64 symlinks during install
L4D2 dedicated server expects to dlopen steamclient.so via
~/.steam/sdk32 (and sdk64). Without those symlinks, srcds_run logs
'cannot open shared object file' and SteamAPI_Init fails, which means
the server is invisible to the public Steam master server, Workshop
addon downloads break, and Steam 'Join Game' / lobby joins do not
reach the server (only direct-IP connect works).

SteamInstaller.install_or_update now ensures the symlinks exist after
SteamCMD finishes. Targets prefer SteamCMD's linux32/linux64 sibling
dirs; falls back to <install_dir>/bin/ if the siblings cannot be
located. Idempotent: re-running the install repairs or leaves the
symlinks alone.

Path.home() respects HOME, which the systemd web unit sets to
/var/lib/left4me, so the symlinks land in the left4me user's home.

Existing deploys can apply the fix by re-running 'Install' from /admin
without a full redeploy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 02:11:27 +02:00
mwiegand
1968684c03
fix(deploy): MountFlags=shared on web service for fuse mount propagation
ProtectSystem=full + ReadWritePaths implicitly give the unit a private
mount namespace (systemd needs to remount /usr read-only). The default
namespace propagation is slave, so mounts the worker creates inside
never reach the host. The gameserver units (started via systemctl,
each with their own namespace) then inherit a host that lacks the
overlay, and their CHDIR into /var/lib/left4me/runtime/<name>/merged
fails.

Set MountFlags=shared so mount events propagate from the worker's
namespace back to the host, then onward to gameserver units at their
unshare time.

Verified on test box: nsenter -t <gunicorn-pid> -m mount showed the
fuse-overlayfs mount inside the worker but plain mount on the host
did not, while web unit had ProtectSystem=full + ReadWritePaths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 02:01:24 +02:00
mwiegand
593611e194
fix(deploy): drop PrivateTmp on web service so fuse mounts propagate
PrivateTmp=true gives the unit a private mount namespace. The worker's
fuse-overlayfs mount lives only inside that namespace, so the host
cannot see it and the gameserver unit (started via systemctl, with its
own namespace inherited from the host) also cannot see it. The
gameserver unit then fails CHDIR on
/var/lib/left4me/runtime/<name>/merged/left4dead2.

The mount must land in the host namespace so the gameserver unit
inherits it at unshare time. Remaining hardening: dedicated user,
ProtectSystem=full, ReadWritePaths, sudoers allowlist limited to two
helper scripts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 01:57:43 +02:00
mwiegand
56b9523d88
fix(deploy): drop NoNewPrivileges on web service so FUSE mounts work
The job worker calls fusermount3 (setuid-root) to mount per-instance
FUSE overlays and sudo to invoke the privileged systemctl wrapper.
NoNewPrivileges=true blocks both, surfacing as
"fusermount3: mount failed: Operation not permitted" the first time a
server is started. Hardening is still enforced via dedicated user,
PrivateTmp, ProtectSystem=full, ReadWritePaths, and the narrow sudoers
allowlist limited to two helper scripts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 01:51:39 +02:00
mwiegand
d14ed9c117
feat(web): blueprint-prefilled create-server flow + empty-state CTA
- Per-row "Create server" link on /blueprints navigates to
  /servers?blueprint_id=<id>; that page validates the param against
  the user's owned blueprints, pre-selects the option, and auto-opens
  the create modal.
- /servers empty-blueprint state now shows an actionable
  "Create a blueprint first ->" link (styled like the primary button)
  pointing at /blueprints, replacing the silent disabled "+ Create"
  button + muted hint.
- Drop the "Reassign blueprint" form on the server detail page
  along with the unused POST /servers/<id> form route. The JSON
  PATCH /servers/<id> endpoint is retained.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 01:47:33 +02:00
mwiegand
923a1840f4
feat(web): forms in modals, edit/delete on detail pages, port auto-assign
- Native <dialog> modal infra (CSS + ~30 LOC JS, no framework) used for
  create forms and delete confirmations.
- Index pages become listing-only: + Create button opens a modal; the
  broken blueprint Actions column and inline overlay edit cells are gone.
- Server detail gains a blueprint reassignment form; existing Delete
  button now opens a confirmation modal before tearing down the runtime.
- Blueprint detail gains a Delete button + confirmation modal (was
  unreachable from the UI before).
- New overlay detail page at /overlays/<id> with edit form, "Used by"
  blueprints list, and delete (admin only).
- Server create: port field is now optional; backend auto-assigns the
  next free port from LEFT4ME_PORT_RANGE_START/_END (default
  27015-27115). 409 on range exhaustion.
- New routes: POST /blueprints/<id>/delete (form sentinel matching
  overlays pattern), POST /servers/<id> (form-friendly blueprint
  reassign), GET /overlays/<id>.
- Server delete operation now redirects to /servers; overlay update
  redirects to /overlays/<id>.

Server rename remains unsupported pending an id-vs-name design pass for
l4d2host (the runtime directory is name-keyed; renaming would orphan
files).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 01:30:33 +02:00
mwiegand
7d9939c71d
fix(deploy): exclude macOS AppleDouble files from deploy archive
When tar runs on macOS it embeds ._* resource-fork sidecars next to each
file. These ended up under l4d2web/alembic/versions/ on the target and
alembic tried to import them as migration modules, failing with
"source code string cannot contain null bytes". Set COPYFILE_DISABLE=1
and add an --exclude '._*' so the archive is portable.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 00:58:29 +02:00
mwiegand
0210ecd301
config: allow SESSION_COOKIE_SECURE override and disable on test deploy
The HTTP-only test deployment binds gunicorn to 0.0.0.0:8000 with no TLS
terminator, so a hardcoded SESSION_COOKIE_SECURE=True breaks browser
login. Make it opt-out via env (default True outside TESTING) and set
SESSION_COOKIE_SECURE=false in the generated web.env so the test box
keeps working over HTTP.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 00:56:48 +02:00
93 changed files with 12301 additions and 297 deletions

1
.envrc Normal file
View file

@ -0,0 +1 @@
layout python python3.13

2
.gitignore vendored
View file

@ -1,5 +1,7 @@
.worktrees/ .worktrees/
.claude/
.venv/ .venv/
.direnv/
.pytest_cache/ .pytest_cache/
__pycache__/ __pycache__/
*.pyc *.pyc

View file

@ -21,6 +21,7 @@ Do not invent architecture outside these plans unless explicitly requested.
### Workspace and tools ### Workspace and tools
- Do not use git worktrees. - Do not use git worktrees.
- Local Python venv is direnv-managed via `.envrc` (Python 3.13). After fresh checkout: `direnv allow`, then `pip install -e ./l4d2host -e ./l4d2web pytest`. See README **Local development** for details.
### Naming and boundaries ### Naming and boundaries
@ -41,7 +42,7 @@ Do not invent architecture outside these plans unless explicitly requested.
- `logs <name> --lines <n> --follow/--no-follow` - `logs <name> --lines <n> --follow/--no-follow`
- Runtime paths are rooted at `LEFT4ME_ROOT`, defaulting to `/var/lib/left4me`. - Runtime paths are rooted at `LEFT4ME_ROOT`, defaulting to `/var/lib/left4me`.
- Deployment/config management owns global units under `/usr/local/lib/systemd/system` and privileged helpers under `/usr/local/libexec/left4me`. - Deployment/config management owns global units under `/usr/local/lib/systemd/system` and privileged helpers under `/usr/local/libexec/left4me`.
- Overlays are external directories (no overlay content management here). - Overlay directories are populated by the web app (workshop downloads, managed-global refresh). The host library only mounts them.
- Fail-fast subprocess behavior; pass raw stderr; propagate return code. - Fail-fast subprocess behavior; pass raw stderr; propagate return code.
- No lock manager, no rollback, no preflight runtime checks. - No lock manager, no rollback, no preflight runtime checks.
- Delete missing instance/runtime dirs must succeed (no-op). - Delete missing instance/runtime dirs must succeed (no-op).

View file

@ -27,7 +27,7 @@ Implementation plans remain the source of truth for architecture and task sequen
- `logs <name> --lines <n> --follow/--no-follow` - `logs <name> --lines <n> --follow/--no-follow`
- The web app calls host operations through `l4d2ctl`, not direct `l4d2host` imports. - The web app calls host operations through `l4d2ctl`, not direct `l4d2host` imports.
- Deployment uses `/var/lib/left4me` for runtime state, `/opt/left4me` for repository contents and the virtualenv, `/etc/left4me` for environment files, and global units under `/usr/local/lib/systemd/system`. - Deployment uses `/var/lib/left4me` for runtime state, `/opt/left4me` for repository contents and the virtualenv, `/etc/left4me` for environment files, and global units under `/usr/local/lib/systemd/system`.
- Overlay handling is directory-based and externally populated. - Overlay handling is directory-based; the web app populates each overlay (workshop downloads, managed-global refresh).
- No lock manager, no rollback, no preflight checks in host library. - No lock manager, no rollback, no preflight checks in host library.
- CLI propagates subprocess failures via stderr and return code. - CLI propagates subprocess failures via stderr and return code.
- `delete` on missing instance is no-op success. - `delete` on missing instance is no-op success.
@ -50,13 +50,22 @@ Implementation plans remain the source of truth for architecture and task sequen
See `deploy/README.md` for the Linux test deployment contract, including the runtime user, target filesystem layout, systemd units, privileged helpers, sudoers rules, admin bootstrap, and overlay reference rules. See `deploy/README.md` for the Linux test deployment contract, including the runtime user, target filesystem layout, systemd units, privileged helpers, sudoers rules, admin bootstrap, and overlay reference rules.
## Local development
This repo uses [direnv](https://direnv.net/) to auto-activate a Python 3.13 venv on `cd` (matching the Debian Trixie production target). With direnv installed and hooked into your shell:
1. `direnv allow` once per fresh checkout (and after any `.envrc` change).
2. `cd` out and back in — `.direnv/python-3.13/` is created and put on `PATH`.
3. `pip install -e ./l4d2host -e ./l4d2web` to install both packages editable.
4. `pip install pytest` to run the test suites (`pytest tests/` inside either subproject).
## Tech Stack (planned) ## Tech Stack (planned)
- Python 3.12+ - Python 3.12+
- Typer, PyYAML, pytest - Typer, PyYAML, pytest
- Flask, SQLAlchemy, Alembic - Flask, SQLAlchemy, Alembic
- HTMX (vendored locally), custom CSS, SSE - HTMX (vendored locally), custom CSS, SSE
- systemd user units, fuse-overlayfs, steamcmd - systemd units, kernel overlayfs (mounted via the `left4me-overlay` privileged helper), steamcmd
## Recommended Implementation Order ## Recommended Implementation Order

View file

@ -12,12 +12,14 @@ The deployment uses these paths:
- `/opt/left4me`: deployed repository contents. - `/opt/left4me`: deployed repository contents.
- `/var/lib/left4me/left4me.db`: SQLite database used by the web app. - `/var/lib/left4me/left4me.db`: SQLite database used by the web app.
- `/var/lib/left4me/installation`: shared L4D2 installation. - `/var/lib/left4me/installation`: shared L4D2 installation.
- `/var/lib/left4me/overlays`: externally managed overlay directories. - `/var/lib/left4me/overlays`: overlay directories. Each overlay lives at `${overlay_id}` under here.
- `/var/lib/left4me/workshop_cache`: deduplicated cache of `.vpk` files downloaded for workshop overlays. One file per Steam item, named `{steam_id}.vpk`. Workshop overlays symlink into this tree.
- `/var/lib/left4me/global_overlay_cache`: cache of non-Steam map archives and extracted `.vpk` files used by managed global map overlays.
- `/var/lib/left4me/instances`: rendered instance specifications and per-instance state. - `/var/lib/left4me/instances`: rendered instance specifications and per-instance state.
- `/var/lib/left4me/runtime`: per-instance runtime mount directories. - `/var/lib/left4me/runtime`: per-instance runtime mount directories.
- `/var/lib/left4me/tmp`: temporary files used by deployment/runtime operations. - `/var/lib/left4me/tmp`: temporary files used by deployment/runtime operations.
- `/usr/local/lib/systemd/system`: global systemd unit files, including `left4me-server@.service`. - `/usr/local/lib/systemd/system`: global systemd unit files, including `left4me-server@.service`.
- `/usr/local/libexec/left4me`: privileged helper commands, including `left4me-systemctl` and `left4me-journalctl`. - `/usr/local/libexec/left4me`: privileged helper commands, including `left4me-systemctl`, `left4me-journalctl`, and `left4me-overlay` (the latter mounts the per-instance kernel overlay in PID 1's mount namespace via `nsenter`).
- `/etc/sudoers.d/left4me`: sudoers rules allowing the web/runtime commands to call the helpers non-interactively. - `/etc/sudoers.d/left4me`: sudoers rules allowing the web/runtime commands to call the helpers non-interactively.
Static units are generated for `/var/lib/left4me`. If `LEFT4ME_ROOT` changes, regenerate and reinstall the unit files instead of reusing the existing static units. Static units are generated for `/var/lib/left4me`. If `LEFT4ME_ROOT` changes, regenerate and reinstall the unit files instead of reusing the existing static units.
@ -54,13 +56,7 @@ Use a strong one-time password and rotate it after first login if needed.
## Overlay References ## Overlay References
Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`. With the default deployment root, they resolve under `/var/lib/left4me/overlays`. Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`. With the default deployment root, they resolve under `/var/lib/left4me/overlays`. New overlays use `${overlay_id}` as their path; the digit-only form is the only one created by the web app.
Valid examples:
- `standard`
- `competitive/base`
- `users/42/custom`
Invalid references are rejected: Invalid references are rejected:
@ -69,4 +65,9 @@ Invalid references are rejected:
- Empty path components such as `competitive//base`. - Empty path components such as `competitive//base`.
- Symlink escapes that resolve outside `${LEFT4ME_ROOT}/overlays`. - Symlink escapes that resolve outside `${LEFT4ME_ROOT}/overlays`.
Overlay content is external to the host library and deployment contract. Populate overlay directories separately before referencing them from blueprints or instance specs. The web app currently supports two overlay surfaces:
- `workshop` overlays (user-owned) — populated by downloading `.vpk` files from the public Steam Web API into `${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creating absolute symlinks under `${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`.
- `script` overlays — populated by an arbitrary user-authored bash script that runs inside `bubblewrap` + `systemd-run --scope` as the unprivileged `l4d2-sandbox` UID, with the overlay directory bind-mounted RW at `/overlay`. Resource caps: 1h walltime, 4 GB RAM, 512 tasks, 200% CPU, 20 GB post-build disk cap.
Both the caches and the overlay directories are owned by the `left4me` runtime user; if the web service ever runs as a different uid, ensure it shares a group with the host process and that both trees are group-readable.

View file

@ -21,13 +21,15 @@ cleanup() {
} }
trap cleanup EXIT INT HUP TERM trap cleanup EXIT INT HUP TERM
tar -czf "$archive" \ COPYFILE_DISABLE=1 tar -czf "$archive" \
--exclude .git \ --exclude .git \
--exclude .claude \
--exclude .venv \ --exclude .venv \
--exclude __pycache__ \ --exclude __pycache__ \
--exclude .pytest_cache \ --exclude .pytest_cache \
--exclude '*.egg-info' \ --exclude '*.egg-info' \
--exclude 'l4d2web.db*' \ --exclude 'l4d2web.db*' \
--exclude '._*' \
-C "$repo_root" . -C "$repo_root" .
remote_tmp=$(ssh "$target" 'mktemp -d') remote_tmp=$(ssh "$target" 'mktemp -d')
@ -75,11 +77,17 @@ if ! id left4me >/dev/null 2>&1; then
$sudo_cmd useradd --system --home-dir /var/lib/left4me --create-home --shell /usr/sbin/nologin left4me $sudo_cmd useradd --system --home-dir /var/lib/left4me --create-home --shell /usr/sbin/nologin left4me
fi fi
# Sandbox uid for script-overlay builds. No home, no login shell — the bwrap
# invocation uses --uid/--gid to drop to it.
if ! id l4d2-sandbox >/dev/null 2>&1; then
$sudo_cmd useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox
fi
if command -v apt-get >/dev/null 2>&1; then if command -v apt-get >/dev/null 2>&1; then
$sudo_cmd apt-get update $sudo_cmd apt-get update
$sudo_cmd apt-get install -y python3 python3-venv python3-pip curl ca-certificates tar gzip fuse-overlayfs fuse3 sudo $sudo_cmd apt-get install -y python3 python3-venv python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip-full
elif command -v dnf >/dev/null 2>&1; then elif command -v dnf >/dev/null 2>&1; then
$sudo_cmd dnf install -y python3 python3-pip curl ca-certificates tar gzip fuse-overlayfs fuse3 sudo $sudo_cmd dnf install -y python3 python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip p7zip-plugins
else else
printf 'Unsupported package manager: expected apt-get or dnf\n' >&2 printf 'Unsupported package manager: expected apt-get or dnf\n' >&2
exit 1 exit 1
@ -94,9 +102,24 @@ $sudo_cmd mkdir -p \
/var/lib/left4me/overlays \ /var/lib/left4me/overlays \
/var/lib/left4me/instances \ /var/lib/left4me/instances \
/var/lib/left4me/runtime \ /var/lib/left4me/runtime \
/var/lib/left4me/workshop_cache \
/var/lib/left4me/tmp /var/lib/left4me/tmp
$sudo_cmd chown -R left4me:left4me /var/lib/left4me /opt/left4me $sudo_cmd chown left4me:left4me \
/var/lib/left4me \
/var/lib/left4me/installation \
/var/lib/left4me/overlays \
/var/lib/left4me/instances \
/var/lib/left4me/runtime \
/var/lib/left4me/workshop_cache \
/var/lib/left4me/tmp
# /var/lib/left4me is left4me's home dir (mode 0700 from useradd --create-home).
# Allow other uids (notably l4d2-sandbox, used by script overlay builds) to
# traverse — but not list — so the bwrap bind-mount can resolve the overlay
# path under the dropped privilege.
$sudo_cmd chmod 0711 /var/lib/left4me
$sudo_cmd chown -R left4me:left4me /opt/left4me
mkdir -p "$repo_tmp" mkdir -p "$repo_tmp"
tar -xzf "$archive" -C "$repo_tmp" tar -xzf "$archive" -C "$repo_tmp"
@ -115,7 +138,9 @@ $sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-web.
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-server@.service /usr/local/lib/systemd/system/left4me-server@.service $sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-server@.service /usr/local/lib/systemd/system/left4me-server@.service
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-systemctl $sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-systemctl
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-journalctl $sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-journalctl
$sudo_cmd chmod 0755 /usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-journalctl $sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-overlay
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-script-sandbox /usr/local/libexec/left4me/left4me-script-sandbox
$sudo_cmd chmod 0755 /usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-script-sandbox
$sudo_cmd cp /opt/left4me/deploy/files/etc/sudoers.d/left4me /etc/sudoers.d/left4me $sudo_cmd cp /opt/left4me/deploy/files/etc/sudoers.d/left4me /etc/sudoers.d/left4me
$sudo_cmd chmod 0440 /etc/sudoers.d/left4me $sudo_cmd chmod 0440 /etc/sudoers.d/left4me
$sudo_cmd visudo -cf /etc/sudoers.d/left4me $sudo_cmd visudo -cf /etc/sudoers.d/left4me
@ -123,6 +148,12 @@ $sudo_cmd visudo -cf /etc/sudoers.d/left4me
$sudo_cmd cp /opt/left4me/deploy/templates/etc/left4me/host.env /etc/left4me/host.env $sudo_cmd cp /opt/left4me/deploy/templates/etc/left4me/host.env /etc/left4me/host.env
$sudo_cmd chmod 0644 /etc/left4me/host.env $sudo_cmd chmod 0644 /etc/left4me/host.env
# Sandbox-only resolver config; bind-mounted into the script sandbox's /etc/resolv.conf
# so DNS still works when IPAddressDeny= blocks the host's (typically private-IP) resolver.
$sudo_cmd install -m 0644 -o root -g root \
/opt/left4me/deploy/files/etc/left4me/sandbox-resolv.conf \
/etc/left4me/sandbox-resolv.conf
if [ ! -f /etc/left4me/web.env ]; then if [ ! -f /etc/left4me/web.env ]; then
secret_key=$(python3 -c 'import secrets; print(secrets.token_hex(32))') secret_key=$(python3 -c 'import secrets; print(secrets.token_hex(32))')
tmp_web_env="$remote_tmp/web.env" tmp_web_env="$remote_tmp/web.env"
@ -130,6 +161,7 @@ if [ ! -f /etc/left4me/web.env ]; then
printf 'DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n' printf 'DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n'
printf 'SECRET_KEY=%s\n' "$secret_key" printf 'SECRET_KEY=%s\n' "$secret_key"
printf 'JOB_WORKER_THREADS=4\n' printf 'JOB_WORKER_THREADS=4\n'
printf 'SESSION_COOKIE_SECURE=false\n'
} > "$tmp_web_env" } > "$tmp_web_env"
$sudo_cmd install -m 0640 -o root -g left4me "$tmp_web_env" /etc/left4me/web.env $sudo_cmd install -m 0640 -o root -g left4me "$tmp_web_env" /etc/left4me/web.env
fi fi
@ -140,15 +172,30 @@ fi
run_as_left4me /opt/left4me/.venv/bin/python -m pip install --upgrade pip run_as_left4me /opt/left4me/.venv/bin/python -m pip install --upgrade pip
run_as_left4me /opt/left4me/.venv/bin/pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web run_as_left4me /opt/left4me/.venv/bin/pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web
run_left4me_with_env env \
JOB_WORKER_ENABLED=false \
/opt/left4me/.venv/bin/python -c "from l4d2web.app import create_app; create_app()"
run_as_left4me sh -c "cd /opt/left4me/l4d2web && set -a; . /etc/left4me/host.env; . /etc/left4me/web.env; set +a; env \ run_as_left4me sh -c "cd /opt/left4me/l4d2web && set -a; . /etc/left4me/host.env; . /etc/left4me/web.env; set +a; env \
JOB_WORKER_ENABLED=false \ JOB_WORKER_ENABLED=false \
PYTHONPATH=/opt/left4me \ PYTHONPATH=/opt/left4me \
/opt/left4me/.venv/bin/alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head" /opt/left4me/.venv/bin/alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head"
# Tighten the application database to left4me:left4me 0640. The DB is
# created by the web app at first start with the default 0644 umask, which
# makes it world-readable on the host. The script-overlay sandbox runs as a
# separate system uid (l4d2-sandbox) which is NOT in the left4me group —
# 0640 blocks it via "other". The owner (left4me) keeps read+write so the
# web service can update the DB.
#
# SQLite in WAL mode (the default in this app) maintains -wal and -shm
# sidecar files; both must also be writable by the web service. If a previous
# operator opened the DB as root (e.g. for ad-hoc inspection), the sidecars
# may have ended up root-owned, which makes SQLite report "readonly database"
# on the next write. Re-chown them defensively. Idempotent on rerun.
for db_file in /var/lib/left4me/left4me.db /var/lib/left4me/left4me.db-wal /var/lib/left4me/left4me.db-shm; do
if [ -f "$db_file" ]; then
$sudo_cmd chown left4me:left4me "$db_file"
$sudo_cmd chmod 0640 "$db_file"
fi
done
if [ -f "$remote_tmp/admin_username" ] && [ -f "$remote_tmp/admin_password" ]; then if [ -f "$remote_tmp/admin_username" ] && [ -f "$remote_tmp/admin_password" ]; then
LEFT4ME_ADMIN_USERNAME=$(cat "$remote_tmp/admin_username") LEFT4ME_ADMIN_USERNAME=$(cat "$remote_tmp/admin_username")
LEFT4ME_ADMIN_PASSWORD=$(cat "$remote_tmp/admin_password") LEFT4ME_ADMIN_PASSWORD=$(cat "$remote_tmp/admin_password")
@ -165,6 +212,62 @@ if [ -f "$remote_tmp/admin_username" ] && [ -f "$remote_tmp/admin_password" ]; t
fi fi
fi fi
# One-shot migration: fuse-overlayfs running as the left4me user used
# user.fuseoverlayfs.* xattrs for whiteouts and opaque-dir markers; kernel
# overlayfs ignores those entirely, so a pre-existing upper/ from the fuse
# era would resurrect "deleted" files. Wipe upper/ and work/ for every
# instance once, gated by a sentinel file so reruns are no-ops.
overlay_sentinel=/var/lib/left4me/.kernel-overlay-migrated
if [ ! -e "$overlay_sentinel" ]; then
$sudo_cmd sh -c "systemctl stop 'left4me-server@*.service' 2>/dev/null || true"
$sudo_cmd systemctl stop left4me-web.service 2>/dev/null || true
$sudo_cmd sh -c "findmnt -t fuse.fuse-overlayfs -o TARGET --noheadings 2>/dev/null | xargs -r -n1 umount -l 2>/dev/null || true"
$sudo_cmd sh -c "findmnt -t overlay -o TARGET --noheadings 2>/dev/null | grep '/var/lib/left4me/runtime/' | xargs -r -n1 umount -l 2>/dev/null || true"
$sudo_cmd sh -c 'for d in /var/lib/left4me/runtime/*/; do [ -d "$d" ] || continue; rm -rf "$d/upper" "$d/work"; mkdir -p "$d/upper" "$d/work"; chown left4me:left4me "$d/upper" "$d/work"; done'
$sudo_cmd touch "$overlay_sentinel"
$sudo_cmd chown left4me:left4me "$overlay_sentinel"
fi
# One-shot migration: 0005_script_overlays drops the legacy
# l4d2center_maps / cedapug_maps overlay rows but doesn't touch their
# directories under /var/lib/left4me/overlays/{id}. Without cleanup, when
# AUTOINCREMENT (or its absence after the 0002 batch_alter_table recreate)
# re-issues an id matching one of those orphan dirs, the web app's
# create_overlay_directory(exist_ok=False) fails with FileExistsError.
# Sweep any overlay dir whose id has no matching DB row, plus the
# now-unused global_overlay_cache.
overlay_orphan_sentinel=/var/lib/left4me/.script-overlays-orphans-cleaned
if [ ! -e "$overlay_orphan_sentinel" ]; then
$sudo_cmd rm -rf /var/lib/left4me/global_overlay_cache
$sudo_cmd sh -c '
cd /var/lib/left4me/overlays || exit 0
ids_in_db=$(/opt/left4me/.venv/bin/python -c "
import sqlite3
c = sqlite3.connect(\"/var/lib/left4me/left4me.db\")
print(\" \".join(str(r[0]) for r in c.execute(\"SELECT id FROM overlays\")))
")
for d in */; do
id=${d%/}
case " $ids_in_db " in
*" $id "*) ;;
*) echo "removing orphan overlay dir: $id"; rm -rf "$id" ;;
esac
done
'
$sudo_cmd touch "$overlay_orphan_sentinel"
$sudo_cmd chown left4me:left4me "$overlay_orphan_sentinel"
fi
# Seed example script overlays (cedapug, l4d2center, competitive_rework, ...)
# as system-wide rows from /opt/left4me/examples/script-overlays/. Idempotent:
# subsequent deploys refresh the script body in place, leaving the row id and
# overlay directory intact.
run_left4me_with_env env \
JOB_WORKER_ENABLED=false \
PYTHONPATH=/opt/left4me \
/opt/left4me/.venv/bin/flask --app l4d2web.app:create_app \
seed-script-overlays /opt/left4me/examples/script-overlays
$sudo_cmd systemctl daemon-reload $sudo_cmd systemctl daemon-reload
$sudo_cmd systemctl enable --now left4me-web.service $sudo_cmd systemctl enable --now left4me-web.service
$sudo_cmd systemctl restart left4me-web.service $sudo_cmd systemctl restart left4me-web.service

View file

@ -0,0 +1,6 @@
# Sandbox-only resolver config — bind-mounted into script-overlay sandboxes
# at /etc/resolv.conf. The host's resolver (often a private/LAN DNS server)
# is unreachable from inside the sandbox because IPAddressDeny= blocks
# egress to RFC1918 / loopback. Public resolvers keep DNS working.
nameserver 1.1.1.1
nameserver 8.8.8.8

View file

@ -1,3 +1,5 @@
Defaults:left4me !requiretty Defaults:left4me !requiretty
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-systemctl * left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-systemctl *
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-journalctl * left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-journalctl *
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-overlay mount *, /usr/local/libexec/left4me/left4me-overlay umount *
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-script-sandbox

View file

@ -12,13 +12,20 @@ Environment=HOME=/var/lib/left4me
Environment=PATH=/opt/left4me/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Environment=PATH=/opt/left4me/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
EnvironmentFile=/etc/left4me/host.env EnvironmentFile=/etc/left4me/host.env
EnvironmentFile=/etc/left4me/web.env EnvironmentFile=/etc/left4me/web.env
ExecStart=/opt/left4me/.venv/bin/gunicorn --workers 1 --threads 8 --bind 0.0.0.0:8000 'l4d2web.app:create_app()' ExecStart=/opt/left4me/.venv/bin/gunicorn --workers 1 --threads 32 --bind 0.0.0.0:8000 'l4d2web.app:create_app()'
Restart=on-failure Restart=on-failure
RestartSec=3 RestartSec=3
NoNewPrivileges=true # NoNewPrivileges intentionally not set: the worker invokes sudo to run
PrivateTmp=true # the left4me-systemctl, left4me-journalctl, and left4me-overlay
# privileged helpers, all setuid via sudo.
# ProtectSystem=full + ReadWritePaths implicitly give this unit a private
# mount namespace, but mount visibility no longer depends on it: overlay
# mounts are performed by the left4me-overlay helper, which nsenters into
# PID 1's mount namespace, so the resulting mount lives in the host
# namespace where the per-instance gameserver units can see it.
ProtectSystem=full ProtectSystem=full
ReadWritePaths=/var/lib/left4me ReadWritePaths=/var/lib/left4me
PrivateTmp=true
[Install] [Install]
WantedBy=multi-user.target WantedBy=multi-user.target

View file

@ -0,0 +1,188 @@
#!/usr/bin/python3
"""Privileged overlay mount helper for left4me.
Invoked via sudo by the left4me runtime user. Validates inputs strictly,
then enters PID 1's mount namespace via nsenter to perform the actual
mount/umount syscall, so the resulting mount lives in the host namespace
and is visible to the systemd-managed gameserver units.
Verbs:
mount <name> Reads ${LEFT4ME_ROOT}/instances/<name>/instance.env
for L4D2_LOWERDIRS, validates every lowerdir is
under one of installation/overlays/workshop_cache/
global_overlay_cache, then mounts the kernel
overlay at runtime/<name>/merged.
umount <name> Unmounts runtime/<name>/merged.
Set LEFT4ME_OVERLAY_PRINT_ONLY=1 to print the would-be argv (one line,
shell-quoted) and exit 0 instead of execv. Used by tests.
"""
import os
import re
import shlex
import sys
from pathlib import Path
NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{0,63}$")
DEFAULT_ROOT = "/var/lib/left4me"
LOWERDIR_ALLOWLIST = (
"installation",
"overlays",
"global_overlay_cache",
"workshop_cache",
)
MAX_LOWERDIRS = 500
NSENTER = "/usr/bin/nsenter"
MOUNT_BIN = "/bin/mount"
UMOUNT_BIN = "/bin/umount"
def die(msg: str) -> None:
sys.stderr.write(f"left4me-overlay: {msg}\n")
sys.exit(1)
def root() -> Path:
return Path(os.environ.get("LEFT4ME_ROOT") or DEFAULT_ROOT)
def validate_name(name: str) -> str:
if not NAME_RE.fullmatch(name):
die(f"invalid instance name: {name!r}")
return name
def parse_lowerdirs(env_path: Path) -> list[str]:
if not env_path.is_file():
die(f"instance.env not found: {env_path}")
raw = None
for line in env_path.read_text().splitlines():
if "=" not in line:
continue
key, value = line.split("=", 1)
if key.strip() == "L4D2_LOWERDIRS":
raw = value
break
if raw is None:
die(f"L4D2_LOWERDIRS not set in {env_path}")
if raw == "":
die(f"L4D2_LOWERDIRS is empty in {env_path}")
parts = raw.split(":")
if any(p == "" for p in parts):
die(f"L4D2_LOWERDIRS contains an empty entry: {raw!r}")
if len(parts) > MAX_LOWERDIRS:
die(f"L4D2_LOWERDIRS has {len(parts)} entries (cap {MAX_LOWERDIRS})")
return parts
def canonical_under(allowed_roots: list[Path], path: Path) -> Path:
try:
canonical = path.resolve(strict=True)
except (FileNotFoundError, RuntimeError):
die(f"path does not exist or has a symlink loop: {path}")
for r in allowed_roots:
if canonical == r or r in canonical.parents:
return canonical
die(f"path is outside the permitted roots: {path} (resolved: {canonical})")
_LISTXATTR = getattr(os, "listxattr", None)
def _entry_has_fuse_xattr(path: str) -> str | None:
if _LISTXATTR is None:
return None
try:
attrs = _LISTXATTR(path, follow_symlinks=False)
except OSError:
return None
for a in attrs:
if a.startswith("user.fuseoverlayfs."):
return a
return None
def assert_no_fuse_xattrs(upper: Path) -> None:
if not upper.exists() or _LISTXATTR is None:
return
for dirpath, dirnames, filenames in os.walk(upper):
for entry in (dirpath, *(os.path.join(dirpath, n) for n in dirnames),
*(os.path.join(dirpath, n) for n in filenames)):
tainted = _entry_has_fuse_xattr(entry)
if tainted:
die(
f"upperdir contains fuse-overlayfs xattr {tainted!r} on {entry}; "
"wipe upper/ and work/ before mounting"
)
def exec_or_print(argv: list[str]) -> None:
if os.environ.get("LEFT4ME_OVERLAY_PRINT_ONLY") == "1":
print(" ".join(shlex.quote(a) for a in argv))
sys.exit(0)
os.execv(argv[0], argv)
def cmd_mount(name: str) -> None:
name = validate_name(name)
r = root()
instance_env = r / "instances" / name / "instance.env"
raw_lowerdirs = parse_lowerdirs(instance_env)
allowed_roots = [(r / sub).resolve() for sub in LOWERDIR_ALLOWLIST]
canonical_lowerdirs = [str(canonical_under(allowed_roots, Path(p))) for p in raw_lowerdirs]
runtime_name_dir = (r / "runtime" / name).resolve(strict=True)
upper = (runtime_name_dir / "upper").resolve(strict=True)
work = (runtime_name_dir / "work").resolve(strict=True)
merged = (runtime_name_dir / "merged").resolve(strict=True)
for label, path in (("upper", upper), ("work", work), ("merged", merged)):
if path.parent != runtime_name_dir:
die(f"{label} resolved outside runtime/{name}: {path}")
assert_no_fuse_xattrs(upper)
options = f"lowerdir={':'.join(canonical_lowerdirs)},upperdir={upper},workdir={work}"
argv = [
NSENTER,
"--mount=/proc/1/ns/mnt",
"--",
MOUNT_BIN,
"-t", "overlay",
"overlay",
"-o", options,
str(merged),
]
exec_or_print(argv)
def cmd_umount(name: str) -> None:
name = validate_name(name)
r = root()
runtime_name_dir = (r / "runtime" / name).resolve(strict=True)
merged = (runtime_name_dir / "merged").resolve(strict=True)
if merged.parent != runtime_name_dir:
die(f"merged resolved outside runtime/{name}: {merged}")
argv = [
NSENTER,
"--mount=/proc/1/ns/mnt",
"--",
UMOUNT_BIN,
str(merged),
]
exec_or_print(argv)
def main(argv: list[str]) -> None:
if len(argv) != 3 or argv[1] not in ("mount", "umount"):
sys.stderr.write("usage: left4me-overlay mount|umount <name>\n")
sys.exit(2)
if argv[1] == "mount":
cmd_mount(argv[2])
else:
cmd_umount(argv[2])
if __name__ == "__main__":
main(sys.argv)

View file

@ -0,0 +1,68 @@
#!/bin/bash
# Privileged sandbox launcher for left4me script overlays.
#
# Invoked via sudo by the web user with two arguments:
# <overlay_id> numeric overlay id; bind-mounts /var/lib/left4me/overlays/<id>
# read-write at /overlay inside the sandbox.
# <script_path> absolute path to a bash file already written by the web app;
# bind-mounted read-only at /script.sh inside the sandbox.
#
# The script runs as a transient systemd .service with the full hardening
# surface: cgroup limits + walltime kill, NoNewPrivileges, ProtectSystem,
# ProtectHome, kernel-tunable / -module / -log protection, namespace
# restriction, address-family restriction, capability bounding (empty),
# seccomp filter (@system-service @network-io), MemoryDenyWriteExecute,
# LockPersonality, RestrictSUIDSGID. Network namespace is *not* restricted —
# scripts must reach the public internet to download workshop / l4d2center
# / cedapug content. PID namespace is shared with the host (no
# PrivatePID= directive in systemd); host PIDs are visible via /proc but
# not signal-able due to UID mismatch.
set -euo pipefail
[[ $# -eq 2 ]] || { echo "usage: $0 <overlay_id> <script>" >&2; exit 64; }
OVERLAY_ID=$1
SCRIPT=$2
[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]] || { echo "bad overlay id" >&2; exit 64; }
OVERLAY_DIR=/var/lib/left4me/overlays/$OVERLAY_ID
[[ -d $OVERLAY_DIR ]] || { echo "no overlay dir at $OVERLAY_DIR" >&2; exit 65; }
[[ -f $SCRIPT ]] || { echo "no script at $SCRIPT" >&2; exit 65; }
if [[ "${LEFT4ME_SCRIPT_SANDBOX_DRY_RUN:-}" == "1" ]]; then
echo "DRY RUN: overlay_id=$OVERLAY_ID script=$SCRIPT overlay_dir=$OVERLAY_DIR"
exit 0
fi
# Make sure the sandbox UID owns the overlay dir so the script can write there.
# Idempotent: a no-op when the dir is already l4d2-sandbox-owned (re-run case),
# and corrects the ownership the first time the dir was created by the web app
# under the left4me UID. World-readable so the gameserver process (left4me)
# can read the overlay contents via the kernel-overlayfs lowerdir at runtime.
chown -R l4d2-sandbox:l4d2-sandbox "$OVERLAY_DIR"
chmod 0755 "$OVERLAY_DIR"
exec systemd-run --quiet --collect --wait --pipe \
--unit="left4me-script-${OVERLAY_ID}-$$" \
-p User=l4d2-sandbox -p Group=l4d2-sandbox \
-p NoNewPrivileges=yes \
-p ProtectSystem=strict -p ProtectHome=yes \
-p PrivateTmp=yes -p PrivateDevices=yes -p PrivateIPC=yes \
-p ProtectKernelTunables=yes -p ProtectKernelModules=yes \
-p ProtectKernelLogs=yes -p ProtectControlGroups=yes \
-p RestrictNamespaces=yes \
-p RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX" \
-p RestrictSUIDSGID=yes -p LockPersonality=yes \
-p MemoryDenyWriteExecute=yes \
-p SystemCallFilter="@system-service @network-io" \
-p SystemCallArchitectures=native \
-p CapabilityBoundingSet= -p AmbientCapabilities= \
-p IPAddressDeny="127.0.0.0/8 ::1/128 169.254.0.0/16 fe80::/10 224.0.0.0/4 ff00::/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 fc00::/7" \
-p TemporaryFileSystem="/etc /var/lib" \
-p BindReadOnlyPaths="/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf /etc/ssl /etc/ca-certificates /etc/nsswitch.conf /etc/alternatives ${SCRIPT}:/script.sh" \
-p BindPaths="${OVERLAY_DIR}:/overlay" \
-p WorkingDirectory=/overlay \
-p Environment="HOME=/tmp PATH=/usr/bin:/usr/sbin OVERLAY=/overlay" \
-p MemoryMax=4G -p MemorySwapMax=0 -p TasksMax=512 \
-p CPUQuota=200% -p RuntimeMaxSec=3600 \
-- /bin/bash /script.sh

View file

@ -9,8 +9,14 @@ DEPLOY = ROOT / "deploy"
WEB_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-web.service" WEB_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-web.service"
SERVER_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-server@.service" SERVER_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-server@.service"
GLOBAL_REFRESH_SERVICE = DEPLOY / "files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.service"
GLOBAL_REFRESH_TIMER = DEPLOY / "files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer"
SANDBOX_UNIT_DIR = DEPLOY / "files/usr/local/lib/systemd/system"
SYSTEMCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-systemctl" SYSTEMCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-systemctl"
JOURNALCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-journalctl" JOURNALCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-journalctl"
OVERLAY_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-overlay"
SCRIPT_SANDBOX_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-script-sandbox"
SANDBOX_RESOLV_CONF = DEPLOY / "files/etc/left4me/sandbox-resolv.conf"
SUDOERS = DEPLOY / "files/etc/sudoers.d/left4me" SUDOERS = DEPLOY / "files/etc/sudoers.d/left4me"
HOST_ENV = DEPLOY / "templates/etc/left4me/host.env" HOST_ENV = DEPLOY / "templates/etc/left4me/host.env"
WEB_ENV_TEMPLATE = DEPLOY / "templates/etc/left4me/web.env.template" WEB_ENV_TEMPLATE = DEPLOY / "templates/etc/left4me/web.env.template"
@ -33,10 +39,18 @@ def test_web_unit_contains_required_runtime_contract():
assert "EnvironmentFile=/etc/left4me/web.env" in unit assert "EnvironmentFile=/etc/left4me/web.env" in unit
assert "ExecStart=/opt/left4me/.venv/bin/gunicorn" in unit assert "ExecStart=/opt/left4me/.venv/bin/gunicorn" in unit
assert "--workers 1" in unit assert "--workers 1" in unit
assert "NoNewPrivileges=true" in unit assert "--threads 32" in unit
# NoNewPrivileges must remain unset because sudo (used by the overlay,
# systemctl and journalctl helpers) is setuid.
assert "NoNewPrivileges=true" not in unit
# Restored now that fuse-overlayfs propagation is no longer the mechanism.
assert "PrivateTmp=true" in unit assert "PrivateTmp=true" in unit
assert "ProtectSystem=full" in unit assert "ProtectSystem=full" in unit
assert "ReadWritePaths=/var/lib/left4me" in unit assert "ReadWritePaths=/var/lib/left4me" in unit
# Mounts now happen in PID 1's namespace via the left4me-overlay helper,
# so MountFlags propagation is irrelevant — and the previous assumption
# that MountFlags=shared made it work was incorrect.
assert "MountFlags=" not in unit
def test_server_unit_contains_required_runtime_contract(): def test_server_unit_contains_required_runtime_contract():
@ -142,10 +156,64 @@ def test_sudoers_allows_only_left4me_helpers_not_raw_system_tools():
"left4me ALL=(root) NOPASSWD: " "left4me ALL=(root) NOPASSWD: "
"/usr/local/libexec/left4me/left4me-journalctl *" "/usr/local/libexec/left4me/left4me-journalctl *"
) in sudoers ) in sudoers
assert "/usr/local/libexec/left4me/left4me-overlay mount *" in sudoers
assert "/usr/local/libexec/left4me/left4me-overlay umount *" in sudoers
assert (
"left4me ALL=(root) NOPASSWD: "
"/usr/local/libexec/left4me/left4me-script-sandbox"
) in sudoers
assert "/bin/systemctl" not in sudoers assert "/bin/systemctl" not in sudoers
assert "/usr/bin/systemctl" not in sudoers assert "/usr/bin/systemctl" not in sudoers
assert "/bin/journalctl" not in sudoers assert "/bin/journalctl" not in sudoers
assert "/usr/bin/journalctl" not in sudoers assert "/usr/bin/journalctl" not in sudoers
assert "/bin/mount" not in sudoers
assert "/bin/umount" not in sudoers
def test_overlay_helper_is_python_with_strict_validation():
text = OVERLAY_HELPER.read_text()
assert text.startswith("#!/usr/bin/python3")
# Validation surface
assert "NAME_RE = re.compile" in text
assert "LOWERDIR_ALLOWLIST" in text
assert "user.fuseoverlayfs." in text
assert "MAX_LOWERDIRS = 500" in text
# Mounts via PID 1's mount namespace
assert "/proc/1/ns/mnt" in text
assert "nsenter" in text
# Verbs are mount and umount (not unmount)
assert '"mount"' in text and '"umount"' in text
assert '"unmount"' not in text
def test_deploy_script_installs_overlay_helper_with_executable_mode():
script = DEPLOY_SCRIPT.read_text()
assert "/usr/local/libexec/left4me/left4me-overlay" in script
assert "chmod 0755" in script and "left4me-overlay" in script
def test_deploy_script_does_not_install_fuse_overlayfs_apt_dep():
# fuse-overlayfs / fuse3 were the previous mount engine; kernel overlayfs
# replaces them. Comments in the migration block may legitimately mention
# the names, so scope this to the actual apt-get / dnf install lines.
install_lines = [
line for line in DEPLOY_SCRIPT.read_text().splitlines()
if ("apt-get install" in line or "dnf install" in line)
]
assert install_lines, "expected at least one apt/dnf install line"
for line in install_lines:
assert "fuse-overlayfs" not in line, line
assert "fuse3" not in line, line
def test_deploy_script_runs_one_shot_kernel_overlay_migration():
script = DEPLOY_SCRIPT.read_text()
assert "/var/lib/left4me/.kernel-overlay-migrated" in script
# Migration should stop services + force-unmount stale mounts + wipe upper/work
assert "systemctl stop 'left4me-server@" in script
assert "systemctl stop left4me-web.service" in script
assert "findmnt -t overlay" in script
assert "/runtime/" in script and "rm -rf" in script and 'upper"' in script and 'work"' in script
def test_env_templates_contain_required_defaults(): def test_env_templates_contain_required_defaults():
@ -169,6 +237,7 @@ def test_deploy_script_has_safe_defaults_and_preserves_state() -> None:
assert "/var/lib/left4me/runtime" in script assert "/var/lib/left4me/runtime" in script
assert "tar" in script assert "tar" in script
assert "--exclude .venv" in script assert "--exclude .venv" in script
assert "--exclude .claude" in script
assert "pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web" in script assert "pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web" in script
assert "systemctl enable --now left4me-web.service" in script assert "systemctl enable --now left4me-web.service" in script
assert "for attempt in" in script assert "for attempt in" in script
@ -183,5 +252,254 @@ def test_deploy_script_has_safe_defaults_and_preserves_state() -> None:
assert "deploy/files" in script assert "deploy/files" in script
def test_deploy_script_does_not_recurse_into_runtime_state_mounts() -> None:
script = DEPLOY_SCRIPT.read_text()
assert "$sudo_cmd chown -R left4me:left4me /var/lib/left4me" not in script
assert "$sudo_cmd chown left4me:left4me \\" in script
assert "/var/lib/left4me/runtime \\" in script
assert "$sudo_cmd chown -R left4me:left4me /opt/left4me" in script
def test_deploy_script_runs_migrations_before_app_initialization() -> None:
script = DEPLOY_SCRIPT.read_text()
assert "alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head" in script
assert "from l4d2web.app import create_app; create_app()" not in script
def test_deploy_script_shell_syntax() -> None: def test_deploy_script_shell_syntax() -> None:
subprocess.run(["sh", "-n", str(DEPLOY_SCRIPT)], check=True) subprocess.run(["sh", "-n", str(DEPLOY_SCRIPT)], check=True)
def test_globals_refresh_units_removed():
"""Global-overlays subsystem deleted in favor of script overlays."""
assert not GLOBAL_REFRESH_SERVICE.exists()
assert not GLOBAL_REFRESH_TIMER.exists()
def test_deploy_script_does_not_provision_globals_subsystem():
script = DEPLOY_SCRIPT.read_text()
# No mkdir/install of the deleted cache dir; mention in a one-shot
# `rm -rf` cleanup is fine.
for line in script.splitlines():
if "/var/lib/left4me/global_overlay_cache" not in line:
continue
assert "rm -rf" in line, line
assert "left4me-refresh-global-overlays" not in script
def test_deploy_script_provisions_sandbox_user():
script = DEPLOY_SCRIPT.read_text()
assert "useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox" in script
def test_deploy_script_does_not_install_bubblewrap():
install_lines = [
line for line in DEPLOY_SCRIPT.read_text().splitlines()
if ("apt-get install" in line or "dnf install" in line)
]
assert install_lines, "expected at least one apt/dnf install line"
for line in install_lines:
assert "bubblewrap" not in line, line
assert "bwrap" not in line, line
def test_deploy_script_installs_script_overlay_tooling():
# Script overlays commonly need 7z and md5sum (e.g. l4d2center map sync).
# coreutils ships md5sum and is technically essential, but listing it
# explicitly makes the contract obvious and survives slim base images.
script = DEPLOY_SCRIPT.read_text().splitlines()
apt_lines = [l for l in script if "apt-get install" in l]
dnf_lines = [l for l in script if "dnf install" in l]
assert apt_lines, "expected an apt-get install line"
assert dnf_lines, "expected a dnf install line"
for line in apt_lines:
assert "p7zip-full" in line, line
assert "coreutils" in line, line
for line in dnf_lines:
# Fedora/RHEL split: p7zip provides 7za, p7zip-plugins provides 7z.
assert "p7zip" in line and "p7zip-plugins" in line, line
assert "coreutils" in line, line
def test_deploy_script_tightens_left4me_db_permissions():
script = DEPLOY_SCRIPT.read_text()
# The DB and its WAL/SHM sidecars must be left4me:left4me 0640 — owner
# (web service) keeps rw, group is read-only, "other" (incl. l4d2-sandbox)
# gets nothing. The sidecars matter because SQLite in WAL mode requires
# write access to all three; if a sidecar ends up root-owned (e.g. from
# ad-hoc root-side inspection), the next write fails as "readonly db".
assert "chown left4me:left4me" in script
assert "chmod 0640" in script
for db_file in (
"/var/lib/left4me/left4me.db",
"/var/lib/left4me/left4me.db-wal",
"/var/lib/left4me/left4me.db-shm",
):
assert db_file in script, f"deploy script must touch {db_file}"
def test_deploy_script_installs_script_sandbox_helper():
script = DEPLOY_SCRIPT.read_text()
assert "/usr/local/libexec/left4me/left4me-script-sandbox" in script
assert "chmod 0755" in script and "left4me-script-sandbox" in script
def test_script_sandbox_helper_present():
assert SCRIPT_SANDBOX_HELPER.is_file()
assert SCRIPT_SANDBOX_HELPER.read_text().startswith("#!/bin/bash")
mode = SCRIPT_SANDBOX_HELPER.stat().st_mode & 0o777
assert mode == 0o755, f"expected 0755, got {oct(mode)}"
def test_script_sandbox_helper_passes_shell_syntax_check():
subprocess.run(["bash", "-n", str(SCRIPT_SANDBOX_HELPER)], check=True)
def test_script_sandbox_helper_invokes_systemd_run_with_hardening():
text = SCRIPT_SANDBOX_HELPER.read_text()
# systemd-run service mode (no --scope), with synchronous I/O to caller.
assert "systemd-run" in text
assert "--scope" not in text, "v2 uses transient service units, not scopes"
assert "--pipe" in text
assert "--wait" in text
assert "--collect" in text
assert "--unit=" in text
# No bwrap.
assert "bwrap" not in text
assert "bubblewrap" not in text
# UID drop via systemd directives.
assert "User=l4d2-sandbox" in text
assert "Group=l4d2-sandbox" in text
# Cgroup limits unchanged from v1.
assert "MemoryMax=4G" in text
assert "MemorySwapMax=0" in text
assert "TasksMax=512" in text
assert "CPUQuota=200%" in text
assert "RuntimeMaxSec=3600" in text
# Hardening directives that v1 (scope mode) couldn't carry.
assert "NoNewPrivileges=yes" in text
assert "ProtectSystem=strict" in text
assert "ProtectHome=yes" in text
assert "PrivateTmp=yes" in text
assert "PrivateDevices=yes" in text
assert "PrivateIPC=yes" in text
assert "ProtectKernelTunables=yes" in text
assert "ProtectKernelModules=yes" in text
assert "ProtectKernelLogs=yes" in text
assert "ProtectControlGroups=yes" in text
assert "RestrictNamespaces=yes" in text
assert "RestrictSUIDSGID=yes" in text
assert "LockPersonality=yes" in text
assert "MemoryDenyWriteExecute=yes" in text
assert "SystemCallFilter=" in text
assert "@system-service" in text
assert "@network-io" in text
assert "CapabilityBoundingSet=" in text
assert "AmbientCapabilities=" in text
assert 'RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX"' in text
# Network namespace stays shared with host.
assert "PrivateNetwork=" not in text
# Mount setup: /etc and /var/lib masked with tmpfs; selective binds back.
assert 'TemporaryFileSystem="/etc /var/lib"' in text
assert "BindReadOnlyPaths=" in text
# The resolv.conf bind points at the sandbox-only file (not the host's
# /etc/resolv.conf, which typically references a private-IP DNS server
# that IPAddressDeny= blocks).
assert "/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf" in text
assert "/etc/ssl" in text
assert "/etc/ca-certificates" in text
assert "/etc/nsswitch.conf" in text
assert "/etc/alternatives" in text
assert "${SCRIPT}:/script.sh" in text
assert 'BindPaths="${OVERLAY_DIR}:/overlay"' in text
# IP egress filter: allow public, deny localhost / RFC1918 / link-local /
# multicast / CGNAT / ULA. systemd's "more specific rule wins" semantics
# mean public IPs hit the allow and listed ranges hit the deny.
# IPAddressDeny alone — no IPAddressAllow=any. Empirically, having both
# set causes the allow to win on this systemd/kernel combo regardless of
# the documented "more specific rule wins" behaviour. With only Deny,
# the kernel's default "allow all" applies to non-listed addresses.
assert "IPAddressDeny=" in text
assert "IPAddressAllow=any" not in text
# Explicit CIDRs — systemd-run's -p parser doesn't accept the
# `localhost` / `link-local` / `multicast` shorthand keywords that
# work in unit files (only the full strings parse).
for token in (
"127.0.0.0/8",
"::1/128",
"169.254.0.0/16",
"fe80::/10",
"224.0.0.0/4",
"ff00::/8",
"10.0.0.0/8",
"172.16.0.0/12",
"192.168.0.0/16",
"100.64.0.0/10",
"fc00::/7",
):
assert token in text, f"missing {token!r} in IPAddressDeny set"
def test_sandbox_resolv_conf_exists():
assert SANDBOX_RESOLV_CONF.is_file()
text = SANDBOX_RESOLV_CONF.read_text()
nameservers = [
line.split()[1]
for line in text.splitlines()
if line.startswith("nameserver ")
]
assert len(nameservers) >= 2, "expected at least two nameservers for redundancy"
# Sanity: the resolvers must be public (not RFC1918 / loopback). We don't
# pin the exact IPs — Cloudflare/Google/Quad9 are all acceptable.
for ns in nameservers:
assert not ns.startswith("127."), ns
assert not ns.startswith("10."), ns
assert not ns.startswith("192.168."), ns
first_octet = int(ns.split(".")[0])
# Reject 172.16.0.0/12.
if first_octet == 172:
second_octet = int(ns.split(".")[1])
assert not (16 <= second_octet <= 31), ns
def test_deploy_script_installs_sandbox_resolv_conf():
script = DEPLOY_SCRIPT.read_text()
assert "deploy/files/etc/left4me/sandbox-resolv.conf" in script
assert "/etc/left4me/sandbox-resolv.conf" in script
def test_script_sandbox_helper_validates_overlay_id():
text = SCRIPT_SANDBOX_HELPER.read_text()
# Numeric-only overlay id
assert '[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]]' in text
# Overlay dir must exist
assert "/var/lib/left4me/overlays/" in text
assert "[[ -d $OVERLAY_DIR ]]" in text
# Script path must exist
assert "[[ -f $SCRIPT ]]" in text
def test_script_sandbox_helper_dry_run_mode(tmp_path):
overlay_root = tmp_path / "var/lib/left4me/overlays/42"
overlay_root.mkdir(parents=True)
fake_script = tmp_path / "fake.sh"
fake_script.write_text("echo hi")
# Run in DRY_RUN mode against a fake l4d2-sandbox UID via a tiny shim that
# simulates `id -u l4d2-sandbox` resolving to a valid number.
helper_text = SCRIPT_SANDBOX_HELPER.read_text()
# We can't actually exec this without root + a real sandbox user; just
# verify the dry-run guard short-circuits before systemd-run / bwrap.
assert 'LEFT4ME_SCRIPT_SANDBOX_DRY_RUN' in helper_text
assert 'exit 0' in helper_text

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,557 @@
# L4D2 Workshop Overlays Implementation Plan
> **Approval gate:** This plan may be written and refined without further approval. Do not implement code changes from this plan until the user explicitly approves implementation.
**Goal:** Implement the workshop overlay feature per `docs/superpowers/specs/2026-05-07-l4d2-workshop-overlays-design.md`. Add a `WorkshopItem` registry, a typed `Overlay.type` column with a builder registry, a workshop builder that downloads from the Steam Web API and manages symlinks into a deduplicated cache, and the supporting routes, templates, jobs, and tests.
**Architecture:** Keep the v1 single-process Flask architecture. New code is additive: a `WorkshopBuilder` class registered in a builder dispatcher, a `steam_workshop` service module for the Steam Web API and downloader, two new database tables and one extended one, and two new job operations on the existing in-process worker. fuse-overlayfs mount handling in `l4d2host` is unchanged — workshop content arrives at overlay paths the same way externals do today.
---
## Locked Decisions
See `docs/superpowers/specs/2026-05-07-l4d2-workshop-overlays-design.md` for the design rationale. Implementation-relevant decisions:
- Typed overlays: `external` (existing rows; no-op builder) and `workshop` (new); future types deferred.
- No JSON `source_config` blob; per-type structured data in proper tables.
- `WorkshopItem` is a global deduplicated registry keyed on `steam_id`. Cache at `/var/lib/left4me/workshop_cache/{steam_id}.vpk`.
- Overlay symlinks are absolute, named `{steam_id}.vpk`; no Steam filename in any on-disk path.
- `overlay_workshop_items` is a pure association; toggle = remove/re-add.
- Collections are atomic UI bulk-imports; DB never tracks collection attribution.
- Single global admin "Refresh all workshop items" button.
- No cache GC in v1.
- `Overlay.user_id` is the scope (NULL = system, set = private); independent of `type`.
- Workshop overlays default to private; existing externals stay system-wide.
- One unified Create-overlay button with type radio; no path field — paths are always `str(overlay_id)`.
- `consumer_app_id == 550` validated at fetch/add; not stored.
- Input field accepts numeric ID, full Workshop URL, or multi-line batch.
- Auto-rebuild after add/remove with build coalescing.
- HTTPS for all Steam Web API calls.
- `Overlay.id` uses `AUTOINCREMENT`; `create_overlay_directory` uses `exist_ok=False`.
- Two partial unique indexes for overlay names: `(name) WHERE user_id IS NULL` and `(name, user_id) WHERE user_id IS NOT NULL`.
---
## Current Gap
- `Overlay` rows have `id`, `name`, `path`, no type, no scope.
- The web app cannot download anything from Steam; users must SFTP `.vpk` files into prepared overlay directories.
- The job worker has no operations for overlay builds or workshop refreshes.
- The mount/build pipeline assumes overlay directories are externally populated.
- There is no UI affordance to add or list workshop content.
---
## Task 1: Extend Tests First — Schema Migration And Models
**Files:**
- Create: `l4d2web/tests/test_workshop_overlay_models.py`
- Modify: `l4d2web/tests/test_models.py` (extend) — partial unique index behavior
Write tests against fresh SQLite schemas asserting:
- An `Overlay` migration round-trip: existing rows acquire `type='external'` and `user_id=NULL`; their `name` values remain unique by partial index.
- After migration, two externals (both `user_id=NULL`) with the same name are rejected by the system partial unique index.
- After migration, two users may both own a workshop overlay named `"my-maps"` (per-user partial unique index).
- `WorkshopItem.steam_id` is unique; concurrent inserts of the same `steam_id` raise integrity errors.
- `overlay_workshop_items` enforces `UNIQUE(overlay_id, workshop_item_id)`.
- `Overlay` deletion cascades `overlay_workshop_items` rows but does not delete `WorkshopItem` rows (`ON DELETE RESTRICT`).
- `Job.overlay_id` is nullable and references `overlays(id)`.
- `Overlay.id` does not reuse a deleted ID after the migration (AUTOINCREMENT).
Verification command:
```bash
pytest l4d2web/tests/test_workshop_overlay_models.py l4d2web/tests/test_models.py -q
```
Expected before implementation: FAIL.
---
## Task 2: Schema Migration And ORM Mappings
**Files:**
- Create: `l4d2web/alembic/versions/0002_workshop_overlays.py`
- Modify: `l4d2web/models.py`
Migration `0002_workshop_overlays` (`down_revision = "b2c684fddbd3"`):
1. `op.batch_alter_table("overlays")`:
- Add `type VARCHAR(16) NOT NULL DEFAULT 'external'` (server_default during migration; remove after backfill).
- Add `user_id INTEGER NULL REFERENCES users(id)`.
- Drop the existing `unique=True` on `name`.
- Add index `ix_overlays_type_user_id` on `(type, user_id)`.
- Switch `id` to `AUTOINCREMENT`.
2. After batch alter, create the two partial unique indexes via raw `op.create_index(..., postgresql_where=..., sqlite_where=...)`:
- `uq_overlay_name_system` on `(name)` `WHERE user_id IS NULL`.
- `uq_overlay_name_per_user` on `(name, user_id)` `WHERE user_id IS NOT NULL`.
3. `op.create_table("workshop_items", ...)` per spec data-model section.
4. `op.create_table("overlay_workshop_items", ...)` with the unique constraint and the reverse-lookup index.
5. `op.batch_alter_table("jobs")`: add `overlay_id INTEGER NULL REFERENCES overlays(id)`.
ORM (`models.py`):
- Extend `Overlay`: add `type`, `user_id`. Drop `unique=True` on `name`. Set `__table_args__` with the two partial indexes and `ix_overlays_type_user_id`.
- Extend `Job`: add `overlay_id` mapped column with FK.
- New `WorkshopItem` and `OverlayWorkshopItem` classes per spec. Set up `Overlay.workshop_items` relationship through the association.
Verification command:
```bash
pytest l4d2web/tests/test_workshop_overlay_models.py l4d2web/tests/test_models.py -q
```
Expected after implementation: PASS.
Run alembic against a fresh test DB to verify upgrade and downgrade succeed.
---
## Task 3: Tests First — Steam Web API And Downloader
**Files:**
- Create: `l4d2web/tests/test_steam_workshop.py`
Mock HTTP with `responses` or `pytest-httpserver`. Cover:
- `parse_workshop_input` accepts a single numeric ID, a single Workshop URL (`steamcommunity.com/sharedfiles/filedetails/?id=N`), and a multi-line whitespace-separated batch of either; returns deduplicated ordered list of digit-only IDs.
- `parse_workshop_input` rejects garbage, paths outside `?id=`, non-digit IDs.
- `resolve_collection` POSTs to the HTTPS endpoint with the form-encoded payload and returns `publishedfileid` children.
- `fetch_metadata_batch` POSTs once with `itemcount=N`; returns parsed `WorkshopMetadata` per item; captures `result != 1` into `last_error`; raises `WorkshopValidationError` when any `consumer_app_id != 550` during user-add; logs and skips during refresh-mode.
- `WorkshopMetadata.preview_url` is captured.
- `download_to_cache` writes `cache_root/{steam_id}.vpk.partial`, then `os.replace` to the final name; sets `os.utime(file, (time_updated, time_updated))`.
- `download_to_cache` is idempotent: a second call where on-disk `(mtime, size)` matches `(time_updated, file_size)` is a no-op (no HTTP request issued).
- `refresh_all` runs downloads via `ThreadPoolExecutor(max_workers=8)` and reports per-item errors without aborting the batch.
- All Steam API URLs use `https://`.
Verification command:
```bash
pytest l4d2web/tests/test_steam_workshop.py -q
```
Expected before implementation: FAIL.
---
## Task 4: Steam Workshop Service Module
**Files:**
- Create: `l4d2web/services/steam_workshop.py`
Public surface:
```python
def parse_workshop_input(raw: str) -> list[str]: ...
def resolve_collection(collection_id: str) -> list[str]: ...
def fetch_metadata_batch(steam_ids: list[str], *, mode: Literal["add","refresh"]) -> list[WorkshopMetadata]: ...
def download_to_cache(meta: WorkshopMetadata, cache_root: Path, *, on_progress=None, should_cancel=None) -> Path: ...
def refresh_all(items: list[WorkshopItem], cache_root: Path, executor_workers: int = 8) -> RefreshReport: ...
```
Implementation rules:
- Endpoints are HTTPS:
- `https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/`
- `https://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1/`
- Form-encoded POSTs with `itemcount=N` / `collectioncount=N` and `publishedfileids[i]=…` per index.
- Per-request timeout 30s; per-item ceiling 5min. No retry or backoff in v1.
- `consumer_app_id != 550`:
- In `mode="add"`: raise `WorkshopValidationError` with the offending `steam_id`.
- In `mode="refresh"`: log and skip; do not abort other items.
- `result != 1`: capture Steam's result code in the item's `last_error`; do not download; do not abort siblings.
- Cooperative cancellation: `download_to_cache` checks `should_cancel()` between chunked reads; `refresh_all`'s executor checks before each task.
- `WorkshopMetadata` is a dataclass with `steam_id, title, filename, file_url, file_size, time_updated, preview_url, consumer_app_id, result`.
- `RefreshReport` aggregates per-item outcomes for the caller's job log.
- Use a single `requests.Session` per call site for connection reuse.
Verification command:
```bash
pytest l4d2web/tests/test_steam_workshop.py -q
```
Expected after implementation: PASS.
---
## Task 5: Tests First — Path Helpers And Overlay Creation
**Files:**
- Create: `l4d2web/tests/test_workshop_paths.py`
- Create: `l4d2web/tests/test_overlay_creation.py`
Cover:
- `workshop_cache_root()` returns `LEFT4ME_ROOT/workshop_cache`.
- `cache_path(steam_id)` returns `cache_root / f"{steam_id}.vpk"` for valid digit strings; rejects non-digits, slashes, dot-dot.
- `generate_overlay_path(overlay_id)` returns `str(overlay_id)`; passes `validate_overlay_ref` from `l4d2host.paths`.
- `create_overlay_directory(overlay)` creates `LEFT4ME_ROOT/overlays/{path}/` with `exist_ok=False`. Calling twice raises (DB/disk drift surfaced loudly).
Verification command:
```bash
pytest l4d2web/tests/test_workshop_paths.py l4d2web/tests/test_overlay_creation.py -q
```
Expected before implementation: FAIL.
---
## Task 6: Path Helpers And Overlay Creation
**Files:**
- Create: `l4d2web/services/workshop_paths.py`
- Create: `l4d2web/services/overlay_creation.py`
`workshop_paths`:
```python
def workshop_cache_root() -> Path: ... # LEFT4ME_ROOT/workshop_cache
def cache_path(steam_id: str) -> Path: ... # validates digits-only; returns cache_root/{steam_id}.vpk
```
`overlay_creation`:
```python
def generate_overlay_path(overlay_id: int) -> str: ... # str(overlay_id) + validate_overlay_ref
def create_overlay_directory(overlay: Overlay) -> None: # makedirs(..., exist_ok=False)
...
```
Verification command:
```bash
pytest l4d2web/tests/test_workshop_paths.py l4d2web/tests/test_overlay_creation.py -q
```
Expected after implementation: PASS.
---
## Task 7: Tests First — Overlay Builders
**Files:**
- Create: `l4d2web/tests/test_overlay_builders.py`
Cover with `tmp_path`:
- `BUILDERS` dict resolves `"external"` and `"workshop"` to instances; unknown types raise `KeyError` (caller's error).
- `ExternalBuilder.build()` is a no-op: makes the overlay directory if missing, writes one log line, returns. Existing files in the directory are untouched.
- `WorkshopBuilder.build()` against a fixture overlay with three associated `WorkshopItem` rows (two with cache files present, one without):
- Creates `left4dead2/addons/` if missing.
- Creates symlinks `addons/{steam_id_a}.vpk → cache_root/{steam_id_a}.vpk` for items with cache files. Symlinks are absolute.
- Skips the uncached item; emits a warning log line. Does not create a dangling symlink.
- On a re-run with the same associations: no FS changes; logs report `unchanged=2 skipped(uncached)=1`.
- On a re-run after one association is removed: removes the obsolete symlink only; leaves cache files alone.
- On a re-run after one item is added: adds only the new symlink.
- Files in `addons/` that aren't symlinks into the cache are left untouched.
- `should_cancel` mid-build: stops between filesystem ops; partial state is consistent and a re-run heals.
Verification command:
```bash
pytest l4d2web/tests/test_overlay_builders.py -q
```
Expected before implementation: FAIL.
---
## Task 8: Overlay Builders And Dispatcher
**Files:**
- Create: `l4d2web/services/overlay_builders.py`
```python
class OverlayBuilder(Protocol):
def build(self, overlay: Overlay, *, on_stdout, on_stderr, should_cancel) -> None: ...
class ExternalBuilder: ...
class WorkshopBuilder: ...
BUILDERS: dict[str, OverlayBuilder] = {
"external": ExternalBuilder(),
"workshop": WorkshopBuilder(),
}
```
`WorkshopBuilder.build()`:
1. Load the overlay's `WorkshopItem` rows.
2. `os.makedirs(overlay_root / "left4dead2/addons", exist_ok=True)`.
3. Compute `desired = {f"{steam_id}.vpk": cache_path(steam_id)}` for items where `last_downloaded_at IS NOT NULL` and the cache file exists. Skip and warn for items missing a cache file.
4. Inspect existing entries in `addons/` via `os.scandir`: keep entries that are not symlinks into `workshop_cache`; otherwise diff against `desired` and apply changes via `os.unlink` and `os.symlink(absolute_target, link_path)`.
5. Emit `created N, removed M, unchanged K, skipped (uncached) S` log line.
6. Check `should_cancel()` between filesystem ops.
Verification command:
```bash
pytest l4d2web/tests/test_overlay_builders.py -q
```
Expected after implementation: PASS.
---
## Task 9: Tests First — Worker Scheduler Truth Table And Coalescing
**Files:**
- Modify: `l4d2web/tests/test_job_worker.py`
Add coverage:
- Truth table for `can_start`:
- `install` not claimed while `refresh_workshop_items`, any `build_overlay`, or any server job is running.
- `refresh_workshop_items` not claimed while `install`, any `build_overlay`, or any server job is running.
- `build_overlay(N)` not claimed while `install`, `refresh_workshop_items`, or another `build_overlay(N)` is running. Two `build_overlay` jobs for **different** overlay IDs claim concurrently.
- Server start/init blocks if `refresh_workshop_items` runs or if any `build_overlay(N)` runs where N ∈ overlays of the server's blueprint.
- `enqueue_build_overlay(overlay_id)`:
- Inserts a new queued job when no pending job exists.
- Returns the existing pending job when one is already queued (coalescing).
- Does not coalesce against running jobs (a new add after build start gets a fresh queued job).
- `refresh_workshop_items` post-completion enqueues `build_overlay` only for overlays whose items had `time_updated` advance or `filename` change; each such enqueue uses the coalescing helper.
Verification command:
```bash
pytest l4d2web/tests/test_job_worker.py -q
```
Expected before implementation: FAIL.
---
## Task 10: Worker Scheduler And New Operations
**Files:**
- Modify: `l4d2web/services/job_worker.py`
Changes:
- Define `OVERLAY_OPERATIONS = {"build_overlay"}` and `GLOBAL_OPERATIONS = {"install", "refresh_workshop_items"}`. Update `malformed_server_job` to allow `server_id IS NULL` for these.
- Extend `SchedulerState` with `running_overlays: set[int]` and `refresh_running: bool`.
- Update `claim_next_job()`:
- Compute `running_overlays` from queries against `running` jobs of operation `build_overlay`.
- Apply the truth-table rules above.
- Continue using `created_at, id` ordering for deterministic claim.
- Add `enqueue_build_overlay(overlay_id: int) -> Job` helper:
- Look for `queued` `build_overlay` job with same `overlay_id`. Return it if present.
- Otherwise insert a new queued job with `overlay_id` set, `server_id=None`, `operation="build_overlay"`.
- Update `run_job` dispatch:
- `build_overlay` → load `Overlay`, dispatch to `BUILDERS[overlay.type].build(overlay, on_stdout, on_stderr, should_cancel)`.
- `refresh_workshop_items` → call `steam_workshop.refresh_all(...)`. After completion, for each affected overlay, call `enqueue_build_overlay(overlay_id)`.
Verification command:
```bash
pytest l4d2web/tests/test_job_worker.py -q
```
Expected after implementation: PASS.
---
## Task 11: Tests First — Routes, Permissions, And Auto-Rebuild
**Files:**
- Modify: `l4d2web/tests/test_overlays.py`
- Create: `l4d2web/tests/test_workshop_routes.py`
Cover:
- `POST /overlays` with `type='workshop'` and `name` succeeds for any logged-in user; `path` is auto-generated; `user_id` is set; the directory exists at `LEFT4ME_ROOT/overlays/{id}`.
- `POST /overlays` with `type='external'` succeeds only for admins; `user_id` is NULL.
- Duplicate workshop name within the same user is rejected; duplicate names across users are accepted.
- Duplicate external name is rejected.
- Non-admins see `type='external' OR user_id=current_user.id` only when listing overlays.
- `POST /overlays/{id}/items` with one numeric ID adds an association and enqueues a coalesced `build_overlay`. The response is an HTMX fragment of the updated item table.
- `POST /overlays/{id}/items` with a multi-line batch (mix of IDs and URLs) adds all and enqueues one coalesced job for the batch.
- `POST /overlays/{id}/items` with a collection ID resolves members and adds N associations.
- Adding a non-L4D2 item (`consumer_app_id != 550`) returns HTTP 400 with a useful message; no association is created.
- Adding an item already in the overlay returns "already in overlay" (no 500).
- `POST /overlays/{id}/items/{item_id}/delete` removes the association and enqueues a coalesced build.
- `POST /overlays/{id}/build` enqueues the manual rebuild and redirects to the job page.
- `POST /admin/workshop/refresh` is admin-only; non-admins receive 403.
Mock `steam_workshop` HTTP layer for these tests.
Verification command:
```bash
pytest l4d2web/tests/test_overlays.py l4d2web/tests/test_workshop_routes.py -q
```
Expected before implementation: FAIL.
---
## Task 12: Routes And Templates
**Files:**
- Modify: `l4d2web/routes/overlay_routes.py`
- Create: `l4d2web/routes/workshop_routes.py`
- Modify: `l4d2web/routes/page_routes.py`
- Modify: `l4d2web/templates/overlays.html`
- Modify: `l4d2web/templates/overlay_detail.html`
- Create: `l4d2web/templates/_overlay_item_table.html`
- Modify: `l4d2web/templates/admin.html`
- Modify: `l4d2web/app.py` (register the workshop blueprint)
`overlay_routes.py`:
- `create_overlay`: read `type` and `name` from form. No `path` field accepted.
- `type='external'`: admin-only; `user_id=NULL`. After insert, set `path = generate_overlay_path(id)`; call `create_overlay_directory(overlay)`.
- `type='workshop'`: any logged-in user; `user_id=current_user.id`. After insert, set `path = generate_overlay_path(id)`; call `create_overlay_directory(overlay)`.
- `update_overlay`: forbid changing `type` and `path`. Workshop: owner or admin can edit `name`. External: admin-only `name` edits.
- `delete_overlay`: after the row deletes, `shutil.rmtree(LEFT4ME_ROOT/overlays/{path})` only if `overlay.path == str(overlay.id)` (legacy externals are left alone). Cache untouched.
`workshop_routes.py`:
- `POST /overlays/{id}/items`: parse input via `parse_workshop_input`; if a collection ID, resolve members; batch-fetch metadata in `mode="add"`; reject non-550 with HTTP 400; upsert `WorkshopItem` via SQLite `INSERT ... ON CONFLICT DO UPDATE` on `steam_id`; bulk-add associations catching `(overlay_id, workshop_item_id)` unique violations; call `enqueue_build_overlay(overlay_id)`; return rendered `_overlay_item_table.html` fragment.
- `POST /overlays/{id}/items/{item_id}/delete`: ownership check; remove association; call `enqueue_build_overlay(overlay_id)`; return updated fragment.
- `POST /overlays/{id}/build`: ownership check; enqueue (coalesced); redirect to `/jobs/{job_id}`.
- `POST /admin/workshop/refresh`: `@require_admin`; insert a `refresh_workshop_items` queued job; redirect to `/admin/jobs`.
`page_routes.py`:
- `overlays()`: admins see all; non-admins see `type='external' OR user_id=current_user.id`.
- `overlay_detail()`: load `WorkshopItem` rows for workshop-type overlays.
Templates:
- `overlays.html`: add Type column. Modal has type radio (External | Workshop) and name field. No path field.
- `overlay_detail.html`: branch on `overlay.type`.
- External view: read-only path display, name edit (admin only).
- Workshop view: an `<textarea>` accepting one or many IDs/URLs plus a radio (Items | Collection); item table with thumbnail (`preview_url`), `steam_id` linked to Steam, title, filename, time_updated, file_size, last_error, Remove; Rebuild button; small status indicator showing the latest related job.
- `_overlay_item_table.html`: renderable standalone for HTMX swaps.
- `admin.html`: add a CSRF-protected "Refresh all workshop items" button.
Verification command:
```bash
pytest l4d2web/tests/test_overlays.py l4d2web/tests/test_workshop_routes.py -q
```
Expected after implementation: PASS.
---
## Task 13: Tests First — Initialize-Time Guard
**Files:**
- Modify: `l4d2web/tests/test_l4d2_facade.py` (or create if missing)
Cover:
- `initialize_server(server_id)` calls `BUILDERS[overlay.type].build()` for each overlay in the blueprint before writing the spec.
- For workshop overlays, when an associated `WorkshopItem` lacks a cache file (`workshop_cache/{steam_id}.vpk` missing), `initialize_server` raises a clear error containing the missing `steam_id`s and the overlay name; the spec is not written; `l4d2ctl initialize` is not invoked.
- For workshop overlays where all items have cache files, the symlinks are present and `l4d2ctl initialize` runs.
Verification command:
```bash
pytest l4d2web/tests/test_l4d2_facade.py -q
```
Expected before implementation: FAIL.
---
## Task 14: Initialize-Time Guard
**Files:**
- Modify: `l4d2web/services/l4d2_facade.py`
Implementation:
- Before writing the temp spec, iterate over the blueprint's overlays and call `BUILDERS[overlay.type].build(...)`.
- For workshop overlays, the builder logs and skips uncached items rather than failing. After all builders run, perform a second pass: query the blueprint's workshop overlays for any associated `WorkshopItem` with no cache file. If any are found, raise an exception whose message names the missing `steam_id`s and points at the overlay page (`Open overlay {name} ({id}) and click Build`).
Verification command:
```bash
pytest l4d2web/tests/test_l4d2_facade.py -q
```
Expected after implementation: PASS.
---
## Task 15: Deploy Provisioning
**Files:**
- Modify: `deploy/install.sh` (or whichever provisioning script creates `/var/lib/left4me/`)
- Modify: `deploy/README.md`
Behavior:
- Provisioning creates `/var/lib/left4me/workshop_cache/` (mode 0755), owned by the web user.
- `deploy/README.md` documents:
- The new directory and its purpose.
- Permission requirement: web user owns; host user reads (shared group with `g+r` if uids differ).
- `LEFT4ME_ROOT` layout updated with the new subtree.
No tests; verify via test deploy.
---
## Task 16: Full Verification And Manual Test Plan
Run focused suites first:
```bash
pytest l4d2web/tests/test_workshop_overlay_models.py -q
pytest l4d2web/tests/test_models.py -q
pytest l4d2web/tests/test_steam_workshop.py -q
pytest l4d2web/tests/test_workshop_paths.py l4d2web/tests/test_overlay_creation.py -q
pytest l4d2web/tests/test_overlay_builders.py -q
pytest l4d2web/tests/test_job_worker.py -q
pytest l4d2web/tests/test_overlays.py l4d2web/tests/test_workshop_routes.py -q
pytest l4d2web/tests/test_l4d2_facade.py -q
```
Then run the full web suite:
```bash
pytest l4d2web/tests -q
```
Manual test plan on the test deploy:
1. Apply migration on a copy of the prod DB; verify all existing overlays read as `type='external'`, `user_id=NULL`; names still unique by partial index; two externals with the same name are rejected.
2. As non-admin, create a workshop overlay. Add a known popular L4D2 addon by URL. Verify the build job auto-enqueues. Verify symlink + cache file. Confirm web UI shows metadata and thumbnail.
3. Paste a multi-line block of item IDs and URLs. Verify all are parsed and added; verify coalescing (only one `build_overlay` job runs).
4. Add a 50-item collection. Verify all 50 metadata rows appear and no UI mention of "from collection". Verify single coalesced build job.
5. Remove an item. Verify auto-rebuild removes the symlink while the cache file remains.
6. As admin, click Refresh All. Verify only items with newer `time_updated` re-download. Verify affected overlays get coalesced `build_overlay` jobs enqueued.
7. Boot an L4D2 server with a workshop overlay attached. Connect locally and confirm the maps appear in the map vote and load.
8. Concurrency probe: enqueue Refresh All while a `build_overlay` is queued; verify scheduler waits per truth table.
9. Initialize-time guard: manually delete a cache file for an item that's in an overlay attached to a server's blueprint. Try to start the server; verify clear error mentioning the missing `steam_id`.
10. Negative: paste a non-L4D2 workshop ID (e.g., a Skyrim mod). Expect HTTP 400 with a clear message; no row inserted.
11. Negative: simulate Steam API down (block egress). Verify add fails with clean error, not 500. Verify refresh job logs the failure.
---
## Commit Strategy
Use small commits after passing relevant tests:
1. `feat(l4d2-web): typed overlays + workshop schema migration`
2. `feat(l4d2-web): steam workshop API client and downloader`
3. `feat(l4d2-web): overlay path helpers and creation`
4. `feat(l4d2-web): overlay builder registry with workshop builder`
5. `feat(l4d2-web): worker support for build_overlay and refresh_workshop_items`
6. `feat(l4d2-web): workshop overlay UI (routes + templates)`
7. `feat(l4d2-web): initialize-time guard for uncached workshop items`
8. `feat(deploy): workshop_cache provisioning`
Do not commit unless the user explicitly asks for commits.
---
## Open Approval Gate
Before modifying implementation files, ask the user for explicit approval to proceed with the workshop-overlays implementation.

View file

@ -0,0 +1,229 @@
# Kernel Overlayfs Helper Implementation Plan
> **Approval status:** User-approved 2026-05-08. Implementation proceeds.
**Goal:** Implement the kernel-overlayfs migration per `docs/superpowers/specs/2026-05-08-kernel-overlayfs-helper-design.md`. Add a Python `left4me-overlay` privileged helper, a `KernelOverlayFSMounter` Python class, wire the existing `OverlayMounter` ABC through `l4d2host/instances.py`, drop `fuse-overlayfs` from the deploy stack, and migrate existing on-disk upper/work directories.
**Architecture:** The web app continues to call `l4d2ctl start|stop|delete <name>`; `l4d2host` continues to expose the same CLI verbs. Internally, `start_instance`/`stop_instance`/`delete_instance` move from a hardcoded subprocess call to `fuse-overlayfs`/`fusermount3` to using `KernelOverlayFSMounter`, which invokes the new sudo helper that mounts in PID 1's namespace via `nsenter`.
---
## Locked Decisions
See `docs/superpowers/specs/2026-05-08-kernel-overlayfs-helper-design.md` for the design rationale. Implementation-relevant summary:
- `left4me-overlay` Python helper in `/usr/local/libexec/left4me/`, owned root, mode 0755, system `/usr/bin/python3`, stdlib only.
- Verbs: `mount <name>`, `umount <name>`.
- Validation in helper: name regex; realpath + allowlist for each lowerdir; exact-prefix check for upper/work/merged; reject upperdir with `user.fuseoverlayfs.*` xattrs; lowerdir count ≤ 500.
- Sudoers verb-constrained: `mount *`, `umount *`.
- `KernelOverlayFSMounter` in `l4d2host/fs/kernel_overlayfs.py` — implements `OverlayMounter`. Derives `name` from the merged path's parent.
- `start_instance` adds `os.path.ismount(merged)` guard before mounting.
- Deploy migration: gated on sentinel file `/var/lib/left4me/.kernel-overlay-migrated`; stops gameservers + web, force-unmounts stale mounts, wipes upper/work, recreates empty.
- Web unit cleanup: drop `MountFlags=shared`, restore `PrivateTmp=true`, rewrite comment block. Keep `NoNewPrivileges` unset.
- Delete `l4d2host/fs/fuse_overlayfs.py` (currently unused — `start_instance` bypasses it).
- AGENTS.md contracts unchanged.
---
## Current Gap
- `l4d2host/instances.py` `start_instance` calls `fuse-overlayfs` directly (lines 85-101); `stop_instance`/`delete_instance` call `fusermount3 -u` directly. The `OverlayMounter` ABC at `l4d2host/fs/base.py` and the `FuseOverlayFSMounter` impl at `l4d2host/fs/fuse_overlayfs.py` exist but are unused.
- Mounts land in the web service's private mount namespace, invisible to host and to gameserver units. `MountFlags=shared` does not fix it.
- No privileged mount helper exists; only `left4me-systemctl` and `left4me-journalctl`.
- Deploy script installs `fuse-overlayfs` apt package and assumes it as a runtime tool.
- Existing `runtime/<name>/upper` directories may carry `user.fuseoverlayfs.*` xattrs that kernel overlayfs would silently ignore (resurrecting "deleted" files).
---
## Task 1: Helper Script + Sudoers + Mounter Class (RED-first)
**Files:**
- Create: `deploy/files/usr/local/libexec/left4me/left4me-overlay` (Python, mode 0755 after deploy)
- Modify: `deploy/files/etc/sudoers.d/left4me`
- Create: `l4d2host/fs/kernel_overlayfs.py`
- Create: `l4d2host/tests/test_kernel_overlayfs.py`
- Create: `l4d2host/tests/test_overlay_helper.py`
- Modify: `deploy/tests/test_deploy_artifacts.py` (assert helper deployed + sudoers entry)
Test plan (RED first):
1. `test_kernel_overlayfs.py::test_mount_invokes_helper_with_name` — mock `run_command`, call `KernelOverlayFSMounter().mount(lowerdirs="/x:/y", upperdir=Path("/var/lib/left4me/runtime/alpha/upper"), workdir=Path("/var/lib/left4me/runtime/alpha/work"), merged=Path("/var/lib/left4me/runtime/alpha/merged"))`, assert argv `["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", "mount", "alpha"]`.
2. `test_kernel_overlayfs.py::test_unmount_invokes_helper_with_umount_verb` — mock + call + assert argv with `umount`.
3. `test_overlay_helper.py` — drives the helper script as a subprocess with `LEFT4ME_OVERLAY_PRINT_ONLY=1` env var (helper prints the would-be `nsenter …` command line and exits 0 instead of execve), and with isolated `LEFT4ME_ROOT=tmp_path`. Cases:
- Valid mount: prints expected `nsenter --mount=/proc/1/ns/mnt -- /bin/mount -t overlay …` line.
- Valid umount: prints expected umount line.
- Bad name (`../escape`, uppercase, empty): exit non-zero, stderr matches.
- Lowerdir traversal (`/etc`, `/var/lib/left4me/../etc`, symlink escape): exit non-zero.
- Missing `instance.env`: exit non-zero.
- Tainted upperdir (with `user.fuseoverlayfs.opaque` xattr): exit non-zero with clear message. (Optional: skip if `setfattr` is unavailable on dev machine; keep test on Linux only via `pytest.mark.skipif`.)
- Lowerdir count > 500: exit non-zero.
4. `test_deploy_artifacts.py` — assert `/usr/local/libexec/left4me/left4me-overlay` is present in deployed files; sudoers includes the new lines.
Implementation:
- Helper script structure: `argparse` for the verb, then path-validation funcs, then `os.execv("/usr/bin/nsenter", [...])` (or printing it under `LEFT4ME_OVERLAY_PRINT_ONLY`).
- `KernelOverlayFSMounter`: `name = merged.parent.name` (with a one-line comment), then `run_command(["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", verb, name], on_stdout=…, on_stderr=…, passthrough=…, should_cancel=…)`.
**Verification:**
```
python3 -m pytest l4d2host/tests/test_kernel_overlayfs.py l4d2host/tests/test_overlay_helper.py deploy/tests/test_deploy_artifacts.py -q
```
Expected before implementation: FAIL on missing class/script. After: all green.
**Commit:** `feat(l4d2-host): KernelOverlayFSMounter + left4me-overlay helper`
---
## Task 2: Wire OverlayMounter Through Lifecycle + Drop Fuse Module
**Files:**
- Modify: `l4d2host/instances.py` (start/stop/delete)
- Modify: `l4d2host/tests/test_lifecycle.py` (update argv assertions, add double-mount guard test)
- Delete: `l4d2host/fs/fuse_overlayfs.py`
- Verify: `l4d2host/fs/__init__.py` does not re-export `FuseOverlayFSMounter`
Test plan (update RED, then GREEN):
1. `test_lifecycle.py::test_start_order` — change assertion: `calls[0]` is now `["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", "mount", "alpha"]`. Adjust setup so the test still creates the merged directory.
2. `test_lifecycle.py::test_stop_succeeds_when_unmount_fails``cmd[0:5] == ["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", "umount", "alpha"]`.
3. `test_lifecycle.py::test_delete_succeeds_when_unmount_fails` — same.
4. NEW `test_lifecycle.py::test_start_refuses_double_mount` — monkeypatch `os.path.ismount` to return True; expect `start_instance` to raise `subprocess.CalledProcessError`; assert NO mount command was issued.
5. `test_lifecycle.py::test_lifecycle_rejects_unsafe_instance_names` — unchanged.
6. `test_lifecycle.py::test_delete_missing_is_noop` — unchanged.
Implementation:
- `instances.py` imports `KernelOverlayFSMounter`. Module-level singleton instance (`_mounter = KernelOverlayFSMounter()`). Replace direct `run_command([...fuse-overlayfs...])` with `_mounter.mount(...)`. Replace direct `run_command([...fusermount3...])` with `_mounter.unmount(...)` (still inside the existing try/except for stop/delete).
- Add the ismount guard at the top of `start_instance` after `runtime_dir` is computed, before `emit_step("mounting runtime overlay...")`. Raise `subprocess.CalledProcessError(returncode=1, cmd=["mount-guard"], stderr="runtime overlay already mounted at <path>; refusing to double-mount")`.
- Delete `l4d2host/fs/fuse_overlayfs.py`.
- Confirm `l4d2host/fs/__init__.py` is empty (already verified to be 1 line).
**Verification:**
```
python3 -m pytest l4d2host/tests -q
python3 -m pytest l4d2web/tests -q
```
Both green. Web tests: the `"Step: mounting runtime overlay..."` log line is preserved in `start_instance`.
**Commit:** `refactor(l4d2-host): start/stop/delete go through OverlayMounter; drop FuseOverlayFSMounter`
---
## Task 3: Deploy Script Migration (Apt Deps + Wipe Upper/Work)
**Files:**
- Modify: `deploy/deploy-test-server.sh`
- Modify: `deploy/tests/test_deploy_artifacts.py` (assert deploy script contains migration lines; assert `fuse-overlayfs` no longer in apt-get install)
Test plan:
1. `test_deploy_artifacts.py::test_deploy_script_drops_fuse_overlayfs_apt_dep``assert "fuse-overlayfs" not in deploy_script` and `assert "kernel-overlay-migrated" in deploy_script`.
2. `test_deploy_artifacts.py::test_deploy_script_migration_block_uses_sentinel``assert ".kernel-overlay-migrated" in deploy_script`.
Implementation:
In `deploy/deploy-test-server.sh`, drop `fuse-overlayfs` from the apt-get and dnf lines (lines 82, 84). Insert before the existing `systemctl restart left4me-web.service` (line 182):
```sh
# One-time migration: fuse-overlayfs upperdir → kernel overlayfs upperdir.
# fuse-overlayfs running as the left4me user uses user.fuseoverlayfs.* xattrs
# for whiteouts and opaque dirs; kernel overlayfs ignores those, so any
# pre-existing upper/ from the fuse era would resurrect "deleted" files.
sentinel=/var/lib/left4me/.kernel-overlay-migrated
if [ ! -e "$sentinel" ]; then
$sudo_cmd systemctl stop 'left4me-server@*.service' 2>/dev/null || true
$sudo_cmd systemctl stop left4me-web.service 2>/dev/null || true
$sudo_cmd sh -c 'findmnt -t fuse.fuse-overlayfs -o TARGET --noheadings | xargs -r -n1 fusermount3 -u 2>/dev/null || true'
$sudo_cmd sh -c "findmnt -t overlay -o TARGET --noheadings | grep '/var/lib/left4me/runtime/' | xargs -r -n1 umount 2>/dev/null || true"
$sudo_cmd sh -c 'for d in /var/lib/left4me/runtime/*/; do [ -d "$d" ] || continue; rm -rf "$d/upper" "$d/work"; mkdir -p "$d/upper" "$d/work"; chown left4me:left4me "$d/upper" "$d/work"; done'
$sudo_cmd touch "$sentinel"
$sudo_cmd chown left4me:left4me "$sentinel"
fi
```
**Verification:**
```
python3 -m pytest deploy/tests -q
```
Green.
**Commit:** `chore(deploy): drop fuse-overlayfs apt dep + one-shot migrate upper/work`
---
## Task 4: Web Unit Hardening Cleanup + Docs
**Files:**
- Modify: `deploy/files/usr/local/lib/systemd/system/left4me-web.service`
- Modify: `deploy/tests/test_deploy_artifacts.py`
- Modify: `README.md`
- Modify: `l4d2host/README.md`
- Modify: `deploy/README.md`
Test plan:
1. `test_deploy_artifacts.py::test_web_unit_contains_required_runtime_contract` — drop `assert "MountFlags=shared" in unit` (or rather: replace with `assert "MountFlags=" not in unit`); add `assert "PrivateTmp=true" in unit`; add `assert "left4me-overlay" not in unit` (just to be precise — the unit shouldn't reference the helper directly, only via Python code).
Implementation:
Edit `left4me-web.service`:
- Drop `MountFlags=shared`.
- Restore `PrivateTmp=true`.
- Rewrite the comment block above hardening lines to explain: mounts now go through the `left4me-overlay` helper which `nsenter`s into PID 1's mount namespace, so this unit's namespace is irrelevant to gameserver visibility. `NoNewPrivileges` stays unset because sudo is setuid.
README updates:
- `README.md` (line ~59): drop fuse-overlayfs from tech-stack list; replace with "kernel overlayfs via privileged helper".
- `l4d2host/README.md`: lines 29, 52, 64 reference fuse — update to "kernel overlayfs (mount via the `left4me-overlay` helper deployed to `/usr/local/libexec/left4me/`)".
- `deploy/README.md`: add `/usr/local/libexec/left4me/left4me-overlay` to the privileged-helpers inventory.
**Verification:**
```
python3 -m pytest deploy/tests -q
```
Green. Manual readthrough of the three READMEs confirms no stale fuse references.
**Commit:** `chore(deploy): cleanup left4me-web hardening + docs for kernel overlayfs`
---
## Task 5: End-to-End Verification on `ckn@10.0.4.128`
**Pre-deploy:** branch is clean, all four prior commits land, all tests green locally.
**Deploy:**
```
deploy/deploy-test-server.sh ckn@10.0.4.128
```
**Verification commands on the box:**
1. `test -e /var/lib/left4me/.kernel-overlay-migrated && echo migrated` — sentinel created.
2. `systemctl status left4me-web.service --no-pager``active (running)`, recent invocation timestamp.
3. From the UI or via `sudo -u left4me /opt/left4me/.venv/bin/l4d2ctl start test-server` — exit 0.
4. `findmnt /var/lib/left4me/runtime/test-server/merged` — shows fstype `overlay` in the host namespace.
5. `systemctl status left4me-server@test-server --no-pager``active (running)` after the start; **not** in `activating (auto-restart)`. No `status=200/CHDIR` errors in `journalctl -u left4me-server@test-server`.
6. `sudo journalctl -k --since "5 minutes ago" | grep -i apparmor | tail` — no overlay-related denials.
7. Negative test: `sudo -u left4me sudo -n /usr/local/libexec/left4me/left4me-overlay mount '../escape'` — exits non-zero with validation error.
8. Idempotency: `l4d2ctl stop test-server && l4d2ctl stop test-server` — both succeed (per the prior `fix(l4d2-host): make stop_instance idempotent` commit, still holds).
9. Re-start: `l4d2ctl start test-server` — succeeds, `findmnt` shows the mount again.
10. Double-mount guard: while the server is running, attempting another start (not via UI; via Python REPL or a second job) — `start_instance` raises `CalledProcessError` with the "refusing to double-mount" message. Optional, can be left to the unit test.
**On failure of any step:** stop and report. Do NOT push. The deploy script is rerunnable; the migration sentinel stays so wipe doesn't repeat.
---
## Out Of Scope
- See spec's "Out Of Scope" section.
- This plan does not push commits; pushing is a separate user decision after end-to-end verification passes.

View file

@ -0,0 +1,350 @@
# L4D2 Script Overlays Implementation Plan
> **Approval status:** User-approved 2026-05-08. Implementation proceeds.
**Goal:** Implement the `script` overlay type per `docs/superpowers/specs/2026-05-08-l4d2-script-overlays-design.md`. Add an `Overlay.script` TEXT column and `Overlay.last_build_status` enum-string column, a `ScriptBuilder` that runs user bash inside a `bubblewrap` + `systemd-run --scope` sandbox via a new `left4me-script-sandbox` privileged helper, route + UI surface for editing/wiping/rebuilding, and delete the entire managed-globals (`l4d2center_maps`, `cedapug_maps`) subsystem and its daily-refresh timer/CLI.
**Architecture:** The web app continues to enqueue `build_overlay` jobs for any overlay row. The job worker dispatches via `BUILDERS[overlay.type].build(...)`. After this change `BUILDERS = {"workshop": WorkshopBuilder(), "script": ScriptBuilder()}`. The new `ScriptBuilder` writes `overlay.script` to a tmpfile and execs `sudo -n /usr/local/libexec/left4me/left4me-script-sandbox <id> <tmpfile>`, which itself execs `systemd-run --scope --collect ... -- bwrap [namespace flags] /bin/bash /script.sh`. stdout/stderr stream through the existing `run_with_streamed_output` helper into the existing job-log SSE plumbing. The job-completion path writes `Overlay.last_build_status` based on the build outcome. The kernel-overlayfs mount layer (`KernelOverlayFSMounter`) is unchanged.
---
## Locked Decisions
See `docs/superpowers/specs/2026-05-08-l4d2-script-overlays-design.md` for design rationale. Implementation-relevant summary:
- Final overlay type list: `workshop` (unchanged) + `script` (new). Drop `l4d2center_maps`, `cedapug_maps`.
- New columns on `overlays`: `script TEXT NOT NULL DEFAULT ''`, `last_build_status VARCHAR(16) NOT NULL DEFAULT ''`.
- Drop tables (FK order): `global_overlay_item_files`, `global_overlay_items`, `global_overlay_sources`.
- `ScriptBuilder` in `l4d2web/services/overlay_builders.py`, uses existing `run_with_streamed_output`.
- Privileged helper `left4me-script-sandbox` (bash, mode 0755, owned root). `systemd-run --scope --collect -p MemoryMax=4G -p MemorySwapMax=0 -p TasksMax=512 -p CPUQuota=200% -p RuntimeMaxSec=3600 -- bwrap …`. Limits 1 h walltime, 4 GB RAM, 20 GB post-build `du` cap.
- New system user `l4d2-sandbox` (`/usr/sbin/nologin`, no home). New apt dep `bubblewrap`.
- Sudoers verb-unrestricted: `left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-script-sandbox`.
- Daily refresh subsystem deleted: `left4me-refresh-global-overlays.{timer,service}` and `flask refresh-global-overlays` CLI removed. No replacement.
- Wipe is the same sandbox helper invoked with the literal script `find /overlay -mindepth 1 -delete`.
- `auto_refresh` column NOT added in this iteration.
- Test deploy DB is wiped on rollout; migration includes `DELETE FROM overlays WHERE type IN ('l4d2center_maps', 'cedapug_maps')` for safety.
---
## Current Gap
- `l4d2web/models.py` `Overlay` has no `script` or `last_build_status` columns. The 3 globals tables are present.
- `l4d2web/services/overlay_builders.py` `BUILDERS = {"workshop": WorkshopBuilder(), "l4d2center_maps": GlobalMapOverlayBuilder(), "cedapug_maps": GlobalMapOverlayBuilder()}`. No `ScriptBuilder`.
- `l4d2web/services/{global_map_sources,global_overlay_refresh,global_map_cache,global_overlays}.py` exist and are referenced by routes / CLI.
- `l4d2web/services/job_worker.py` carries `refresh_global_overlays_running` plumbing.
- `l4d2web/cli.py` defines `refresh-global-overlays`.
- `l4d2web/routes/overlay_routes.py` has no `/script`, `/wipe`, or `/build` endpoints for non-workshop types.
- `l4d2web/templates/overlays.html` create modal type radio offers only `workshop`.
- `l4d2web/templates/overlay_detail.html` has a global-source block (~lines 3446) that should not survive.
- `deploy/files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.{timer,service}` exist.
- `deploy/deploy-test-server.sh` provisions `global_overlay_cache/` and does not provision `l4d2-sandbox` or install `bubblewrap`.
- Seven `tests/test_global_*.py` files exist and reference removed code.
---
## Task 1: Schema migration (alembic 0005)
**Files:**
- Create: `l4d2web/alembic/versions/0005_script_overlays.py` (revises `0004_drop_legacy_external_overlay_type`).
- Modify: `l4d2web/models.py``Overlay` gains `script` and `last_build_status` columns; remove `GlobalOverlaySource`, `GlobalOverlayItem`, `GlobalOverlayItemFile` model classes.
- Modify: `l4d2web/tests/test_overlay_models.py` (or whichever existing test asserts the Overlay schema; create one if absent) — assert new columns present.
Test plan (RED first):
1. `tests/test_alembic_migrations.py::test_upgrade_0005_adds_script_columns` — apply migrations to a fresh in-memory SQLite, assert `script` and `last_build_status` columns present on `overlays`, assert no `global_overlay_*` tables, assert old data wipe `DELETE FROM overlays WHERE type IN (...)` is part of the upgrade.
2. `tests/test_alembic_migrations.py::test_downgrade_0005_restores_globals` (only if downgrade is supported in the project's migration policy; skip with `pytest.skip` if not — kernel-overlayfs migration is one-way, follow that precedent).
3. `tests/test_overlay_models.py::test_overlay_has_script_columns``Overlay(...)` instance has `script=''` and `last_build_status=''` defaults.
Implementation:
- Migration uses `op.drop_table('global_overlay_item_files')` etc. in correct FK order; uses `op.add_column('overlays', sa.Column('script', sa.Text(), nullable=False, server_default=''))` and similar for `last_build_status` (`sa.String(16)`).
- The `DELETE FROM overlays WHERE type IN ('l4d2center_maps','cedapug_maps')` runs *before* the column additions so the operation is straightforward — these rows do not reference the new columns.
- `models.py`: delete the three globals model classes outright; add the two new columns to `Overlay` with explicit defaults.
**Verification:**
```
python3 -m pytest l4d2web/tests/test_alembic_migrations.py l4d2web/tests/test_overlay_models.py -q
```
**Commit:** `feat(l4d2-web): script overlay schema — add overlay.script + last_build_status, drop globals tables`
---
## Task 2: ScriptBuilder + BUILDERS registry update
**Files:**
- Modify: `l4d2web/services/overlay_builders.py` — add `ScriptBuilder`, remove `GlobalMapOverlayBuilder`, change `BUILDERS` dict.
- Rewrite: `l4d2web/tests/test_overlay_builders.py` — drop globals-builder tests, add ScriptBuilder tests.
Test plan (RED first):
1. `test_overlay_builders.py::test_builders_registry``set(BUILDERS) == {"workshop", "script"}`. Assert `"l4d2center_maps"` and `"cedapug_maps"` and `"external"` are absent.
2. `test_overlay_builders.py::test_script_builder_invokes_helper` — patch `run_with_streamed_output` to capture argv; build an `Overlay(id=42, type='script', script='echo hi')`; assert argv shape `["sudo", "-n", "/usr/local/libexec/left4me/left4me-script-sandbox", "42", <script_path>]` and that the script_path file exists with content `"echo hi"` at invocation time. Verify the tmpfile is unlinked after build.
3. `test_overlay_builders.py::test_script_builder_disk_cap` — fake `subprocess.check_output` for `du` to return `25000000000`; build raises `BuildError("disk-cap-exceeded")` and `on_stderr` was called with the cap message.
4. `test_overlay_builders.py::test_script_builder_streams_output` — fake `run_with_streamed_output` invokes both `on_stdout("hello\n")` and `on_stderr("warn\n")`; both lambda lists capture the lines.
5. `test_overlay_builders.py::test_script_builder_cancel``should_cancel` returns True after the first stdout line; assert `run_with_streamed_output` propagated cancellation (the existing helper's contract — the test just ensures we pass `should_cancel` through and don't run the disk-budget check on cancel).
6. `test_overlay_builders.py::test_workshop_builder_unchanged` — smoke test that `WorkshopBuilder` still exists and is invokable (regression guard against accidental removal during refactor).
Implementation:
- Add `import os, subprocess, tempfile` at the top of `overlay_builders.py` if not present.
- `ScriptBuilder` exactly as in the spec (verbatim copy from the design doc, §Build Lifecycle).
- Define a small `BuildError` exception class if one doesn't already exist locally; reuse the existing one if `WorkshopBuilder` already raises a similar type.
- `_enforce_disk_budget` calls `subprocess.check_output(["du", "-sb", str(overlay_path(overlay_id))])`; the existing `overlay_path` helper in the module already returns the absolute Path. Parse first whitespace-delimited integer; cap is `20 * 1024**3`.
- Job-completion path: locate the existing path that handles `build_overlay` job success/failure (likely in `services/job_worker.py` or a related orchestration module). Add a single column write: on success `last_build_status='ok'`, on `BuildError` / non-zero exit / cancel `last_build_status='failed'`. Add a `tests/test_job_worker.py::test_build_overlay_writes_last_build_status` covering both branches.
- Remove `GlobalMapOverlayBuilder` class and any helper functions it owns that are not used elsewhere.
**Verification:**
```
python3 -m pytest l4d2web/tests/test_overlay_builders.py l4d2web/tests/test_job_worker.py -q
```
**Commit:** `feat(l4d2-web): ScriptBuilder + BUILDERS registry update`
---
## Task 3: Delete global-overlay services + CLI command + their tests
**Files:**
- Delete: `l4d2web/services/global_map_sources.py`
- Delete: `l4d2web/services/global_overlay_refresh.py`
- Delete: `l4d2web/services/global_map_cache.py`
- Delete: `l4d2web/services/global_overlays.py`
- Modify: `l4d2web/cli.py` — remove `refresh-global-overlays` command (lines ~4455). Drop any imports that go orphaned.
- Delete: `l4d2web/tests/test_global_map_sources.py`
- Delete: `l4d2web/tests/test_global_overlay_models.py`
- Delete: `l4d2web/tests/test_global_overlay_builders.py`
- Delete: `l4d2web/tests/test_global_overlay_cli.py`
- Delete: `l4d2web/tests/test_global_overlay_refresh.py`
- Delete: `l4d2web/tests/test_global_overlays.py`
- Delete: `l4d2web/tests/test_global_map_cache.py`
- Audit & fix: any other module that imports the deleted modules. Likely candidates: `l4d2web/app.py` (CLI registration), `routes/overlay_routes.py`, `routes/page_routes.py`. Resolve by deletion of the dead import / call site, not by stubbing.
- Modify: `pyproject.toml` — drop `py7zr` from dependencies (only used by the deleted globals subsystem).
Test plan:
1. RED-first via grep: `grep -RIn 'global_map_sources\|global_overlay_refresh\|global_map_cache\|global_overlays\|refresh_global_overlays\|GlobalMapOverlayBuilder' l4d2web/ deploy/` — should return zero hits at the end of this task. Add this as `tests/test_no_globals_references.py::test_no_globals_imports` if you want it as a permanent regression guard, otherwise spot-check.
2. Existing `tests/test_cli.py` (or whichever covers Flask CLI) loses any cases for `refresh-global-overlays`; add a `test_refresh_global_overlays_command_removed` that asserts the click command is not registered.
Implementation:
- Delete files via `git rm`.
- In `cli.py`, remove the command function and its `@app.cli.command(...)` decorator. Drop any helper imports that become orphaned.
- Remove `py7zr` from `pyproject.toml` and re-lock if a lockfile is present.
**Verification:**
```
python3 -m pytest l4d2web/tests/ -q
grep -RIn 'global_map_sources\|global_overlay_refresh\|global_map_cache\|global_overlays\|refresh_global_overlays\|GlobalMapOverlayBuilder' l4d2web/ deploy/ || echo "clean"
```
**Commit:** `refactor(l4d2-web): drop global-overlays subsystem in favor of script type`
---
## Task 4: Job worker — drop refresh_global_overlays from scheduler
**Files:**
- Modify: `l4d2web/services/job_worker.py` — remove `"refresh_global_overlays"` from `GLOBAL_OPERATIONS`; remove `refresh_global_overlays_running` field from `SchedulerState` and any references in `can_start()`; check whether `blocked_servers_by_overlay` was added solely for the globals subsystem and remove if so.
- Modify: `l4d2web/tests/test_job_worker.py` — drop `refresh_global_overlays` truth-table rows; add explicit `build_overlay` truth-table cases for `script`-type overlays (mechanically identical to workshop, but pinned by test).
Test plan:
1. `test_job_worker.py::test_global_operations_set``GLOBAL_OPERATIONS == {"install", "refresh_workshop_items"}` (or whatever subset remains; pin it).
2. `test_job_worker.py::test_build_overlay_script_type_blocks_per_overlay` — start `build_overlay(overlay_id=7)` for a `script`-type overlay; assert second `build_overlay(overlay_id=7)` cannot start; assert `build_overlay(overlay_id=8)` can.
3. `test_job_worker.py::test_build_overlay_blocks_server_init_on_blueprint_overlay` — existing test, may need re-pinning if it referenced globals.
Implementation:
- Remove the field from the dataclass / TypedDict that backs `SchedulerState`.
- Remove any update sites that flipped the flag (the worker's enqueue / on-start / on-complete paths).
- The remaining mutex rules (`install` / `refresh_workshop_items` are global; `build_overlay` per-overlay; server ops block on overlays in their blueprint) are unchanged structurally.
**Verification:**
```
python3 -m pytest l4d2web/tests/test_job_worker.py -q
```
**Commit:** `refactor(l4d2-web): drop refresh_global_overlays from scheduler`
---
## Task 5: Routes (script update / wipe / build)
**Files:**
- Modify: `l4d2web/routes/overlay_routes.py` — add three POST endpoints.
- Create: `l4d2web/tests/test_script_overlay_routes.py`.
Test plan (RED first):
1. `test_script_overlay_routes.py::test_create_script_overlay` — POST `/overlays` with form `{"name": "x", "type": "script"}` as a regular user → 302 to detail; row exists with `type='script'`, `script=''`, `last_build_status=''`, `user_id=current_user.id`, `path=str(id)`.
2. `test_script_overlay_routes.py::test_admin_creates_system_wide_script_overlay` — admin POST with system-wide flag → row has `user_id=NULL`.
3. `test_script_overlay_routes.py::test_update_script_body_enqueues_build` — POST `/overlays/{id}/script` with `{"script": "echo new"}` → row.script updated; one new `build_overlay` job enqueued for the overlay; second immediate POST coalesces (no second job inserted while first is pending).
4. `test_script_overlay_routes.py::test_manual_rebuild` — POST `/overlays/{id}/build` → enqueues `build_overlay`; coalesces.
5. `test_script_overlay_routes.py::test_wipe_runs_find_delete` — POST `/overlays/{id}/wipe` → invokes `ScriptBuilder.build` (or the underlying helper) with the literal script `find /overlay -mindepth 1 -delete`. After success, row.last_build_status `==''`. Does not enqueue a `build_overlay`.
6. `test_script_overlay_routes.py::test_wipe_refuses_during_running_build` — set scheduler state to `build_overlay(overlay_id=7)` running; POST `/overlays/7/wipe` → 409 (or whatever the existing pattern uses for scheduler conflicts), no sandbox invocation.
7. `test_script_overlay_routes.py::test_permissions_non_owner_denied` — user A creates private script overlay; user B POSTs `/overlays/{id}/script` → 403.
8. `test_script_overlay_routes.py::test_permissions_admin_can_edit_any` — admin POSTs `/overlays/{id}/script` for user A's row → 200.
Implementation:
- Mirror the existing `_can_edit_overlay()` permission helper.
- The `/wipe` endpoint can either (a) call `ScriptBuilder` directly with a synthetic `Overlay`-like object whose `.script` is the find command and whose `.id` is the real overlay id, or (b) factor a `_run_sandbox(overlay_id, script_text, on_stdout, on_stderr, should_cancel)` helper out of `ScriptBuilder.build()` and call it from both. (b) is cleaner; do (b).
- Wipe runs **synchronously** in the request thread (small, fast). It does NOT enqueue a job. Surface log output as flash messages or by streaming through the existing log infra — pick whichever matches the existing wipe-equivalent pattern (workshop overlays don't have a wipe; closest analog is the existing delete-overlay flow).
- The `/script` endpoint enqueues via the same `enqueue_build_overlay(overlay_id)` helper used by workshop overlays' add/remove flows. Coalescing is already implemented there.
**Verification:**
```
python3 -m pytest l4d2web/tests/test_script_overlay_routes.py l4d2web/tests/test_overlay_routes.py -q
```
**Commit:** `feat(l4d2-web): script overlay routes (script update / wipe / build)`
---
## Task 6: Templates (overlays.html + overlay_detail.html)
**Files:**
- Modify: `l4d2web/templates/overlays.html` — add `script` to the create-modal type radio (lines ~2949).
- Modify: `l4d2web/templates/overlay_detail.html` — add a `{% if overlay.type == 'script' %}` block with textarea + Save / Rebuild / Wipe buttons + status badge; delete the global-source block (lines ~3446).
- Modify: `l4d2web/tests/test_pages.py` — assert script-section renders for type=`script`, workshop-section renders for type=`workshop`, global-source-section is absent.
Test plan:
1. `test_pages.py::test_overlay_create_modal_offers_script_type` — GET `/overlays`; HTML contains `value="script"` radio.
2. `test_pages.py::test_overlay_detail_script_section` — create script overlay, GET `/overlays/{id}`; HTML contains `<textarea name="script">`, "Rebuild" button, "Wipe" button, status badge element.
3. `test_pages.py::test_overlay_detail_workshop_section_unchanged` — existing workshop detail still has thumbnail grid, add-item form, etc.
4. `test_pages.py::test_overlay_detail_no_global_source_block` — page HTML has no element from the deleted global-source block (check for an attribute or string unique to that block).
Implementation:
- Detail-page wipe button uses a small confirm-modal pattern (copy from the existing delete-overlay confirm modal).
- Status badge: existing CSS classes for ok/warn/error already exist in `static/`; reuse them.
- No new JS deps. Plain `<form method="post">` with HTMX `hx-post` for the script update if a streaming UX is desired (match existing patterns).
**Verification:**
```
python3 -m pytest l4d2web/tests/test_pages.py -q
```
Manual: start dev server (`flask run`), create a script overlay, paste `echo "hi" > foo`, click Save, watch log stream. Then click Wipe; confirm dir is empty. Then click Rebuild; confirm `foo` reappears.
**Commit:** `feat(l4d2-web): script overlay UI`
---
## Task 7: Libexec sandbox helper + sudoers + deploy-artifacts test
**Files:**
- Create: `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` (bash, mode 0755 after deploy, owned root).
- Modify: `deploy/files/etc/sudoers.d/left4me` — append the rule.
- Modify: `deploy/tests/test_deploy_artifacts.py` — assert helper file present + sudoers contains the new line.
Test plan (RED first):
1. `test_deploy_artifacts.py::test_script_sandbox_helper_present` — file exists, mode bits indicate 0755 (or whatever the test framework allows checking pre-deploy), shebang is `#!/bin/bash`.
2. `test_deploy_artifacts.py::test_sudoers_includes_script_sandbox_rule` — sudoers file contains the exact line `left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-script-sandbox`.
3. Optional integration test (skip on non-Linux dev): drive the helper as a subprocess with a synthesized fake `/var/lib/left4me/overlays/1/` and a no-op script, assert `bwrap` invocation happens (use a mock `systemd-run` or `LEFT4ME_SCRIPT_SANDBOX_DRY_RUN=1` env that prints the would-be invocation and exits 0). Mirrors the `LEFT4ME_OVERLAY_PRINT_ONLY=1` pattern from the kernel-overlayfs helper test.
Implementation:
- Helper script verbatim from the spec §Sandbox.
- Sudoers fragment: append (don't replace existing rules). The existing fragment has rules for `left4me-overlay`, `left4me-systemctl`, `left4me-journalctl` — match the same formatting (one rule per line, no trailing whitespace).
**Verification:**
```
python3 -m pytest deploy/tests/test_deploy_artifacts.py -q
bash -n deploy/files/usr/local/libexec/left4me/left4me-script-sandbox
```
**Commit:** `feat(deploy): left4me-script-sandbox helper + sudoers fragment`
---
## Task 8: Deploy script — provision l4d2-sandbox + bubblewrap; drop globals timer
**Files:**
- Modify: `deploy/deploy-test-server.sh` — add `useradd --system ... l4d2-sandbox`, add `apt-get install -y bubblewrap`, ensure helper installation step picks up `left4me-script-sandbox` (likely automatic if it's a glob in `deploy/files/usr/local/libexec/left4me/*`); drop the `mkdir global_overlay_cache` line if present.
- Delete: `deploy/files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer`
- Delete: `deploy/files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.service`
- Modify: `deploy/tests/test_deploy_artifacts.py` — assert the two unit files are absent; assert `useradd l4d2-sandbox` and `apt-get install ... bubblewrap` lines are present in the deploy script.
Test plan:
1. `test_deploy_artifacts.py::test_globals_refresh_units_removed` — files do not exist under `deploy/files/usr/local/lib/systemd/system/`.
2. `test_deploy_artifacts.py::test_deploy_script_provisions_sandbox_user` — grep the deploy script for the useradd line.
3. `test_deploy_artifacts.py::test_deploy_script_installs_bubblewrap` — grep for `bubblewrap` in apt invocations.
Implementation:
- `useradd` line uses `--system --no-create-home --shell /usr/sbin/nologin`. Idempotency: wrap with `id l4d2-sandbox &>/dev/null || useradd ...`.
- `apt-get install`: append `bubblewrap` to whatever package list the script already maintains.
- Globals timer/service deletions: `git rm`.
**Verification:**
```
python3 -m pytest deploy/tests/ -q
shellcheck deploy/deploy-test-server.sh deploy/files/usr/local/libexec/left4me/left4me-script-sandbox
```
**Commit:** `chore(deploy): provision l4d2-sandbox + bubblewrap; drop globals refresh timer`
---
## Task 9: Full pytest run + drift fixes
**Files:** as needed across the repo.
Test plan: run the full test suite for both packages; chase down any drift caused by removed model classes, dropped imports, or template changes.
```
python3 -m pytest l4d2web/tests/ -q
python3 -m pytest l4d2host/tests/ -q
python3 -m pytest deploy/tests/ -q
```
Implementation: fix what breaks. Common drift sources to expect:
- Tests that imported from deleted modules.
- Tests that asserted exact `BUILDERS` keyset (good — they should have been updated in Task 2).
- Tests that built fixtures with `type='l4d2center_maps'` or `type='cedapug_maps'` — those tests likely belong to the deleted set or need conversion to `type='script'`.
- Template snapshot tests (if any) that captured the deleted global-source block.
**Verification:** all three suites green.
**Commit:** `chore(l4d2-web): test suite drift fixes after script-overlays migration` (only if drift fixes needed; skip if Tasks 18 left the suite green)
---
## End-to-end deployment verification (manual, on test host)
After all tasks committed:
1. **Reset deploy:** run `deploy/deploy-test-server.sh` from clean state. Confirm `bubblewrap` installed (`dpkg -l bubblewrap`), `l4d2-sandbox` user exists (`id l4d2-sandbox`), `/usr/local/libexec/left4me/left4me-script-sandbox` is mode 0755 and root-owned, `sudo -ln` as `left4me` shows the new rule.
2. **Sandbox smoke:** as `left4me`, write `/tmp/echo.sh` containing `echo $(whoami) > /overlay/sentinel`. `mkdir -p /var/lib/left4me/overlays/1`. `sudo /usr/local/libexec/left4me/left4me-script-sandbox 1 /tmp/echo.sh`. Confirm `/var/lib/left4me/overlays/1/sentinel` contains `l4d2-sandbox` and is owned by `l4d2-sandbox`. Confirm `/etc/passwd`, `/var/lib/left4me/l4d2web.db`, and `/home` are not visible inside the sandbox by running probe scripts.
3. **Resource limits:**
- `dd if=/dev/zero of=/overlay/big bs=1M count=25000` → succeeds inside sandbox; `ScriptBuilder._enforce_disk_budget` flags the build failed; `last_build_status='failed'`.
- `sleep 7200` → killed at 1 h by `RuntimeMaxSec=3600`.
- Memory hog (`python3 -c "x=' '*(5*1024**3)"`) → OOM at 4 GB.
4. **App-level happy path:** as a non-admin user, create a script overlay via the UI, paste an old `competitive_rework`-style script, Save → build runs, succeeds, addons appear in `overlays/{id}/left4dead2/`. Stack onto a server blueprint, start the server, verify content mounts via the L4D2 admin console (`map workshop/...`).
5. **Wipe:** click Wipe → dir empty (find -delete output in log). Click Rebuild → repopulates. `last_build_status` cycles: `''``'ok'`.
6. **Scheduler:** start a server using the script overlay; in another browser tab attempt to Rebuild → 409 / scheduler-blocked. Stop server; rebuild succeeds.
7. **Audit log:** `journalctl --since "5 min ago" | grep run-` shows transient scopes per build with cgroup memory accounting visible.
These are not required for any single commit but should pass before declaring the work done.

View file

@ -0,0 +1,146 @@
# L4D2 Script Sandbox v2 Implementation Plan
> **Approval status:** User-approved 2026-05-08 after smoke-testing the v2 prototype on `ckn@10.0.4.128`.
**Goal:** Replace the bwrap-based sandbox helper with a systemd-only one per `docs/superpowers/specs/2026-05-08-l4d2-script-sandbox-v2-systemd.md`. Drop the `bubblewrap` apt dep. Tighten `left4me.db` file mode to 0640 root:left4me. Update the deploy-artifact tests to assert the new helper shape.
**Architecture:** See spec. Helper invokes `systemd-run --pipe --wait` in service-unit mode with full hardening directives. No bwrap. Web-app side (`ScriptBuilder`, `run_sandboxed_script`, routes) is unchanged.
---
## Locked Decisions
See spec §Locked Decisions for rationale. Implementation summary:
- Helper file at the same path (`deploy/files/usr/local/libexec/left4me/left4me-script-sandbox`) is rewritten in place.
- The sudoers rule is unchanged.
- `bubblewrap` dropped from `apt-get install` / `dnf install` lines.
- `left4me.db` chmod 0640 added to deploy script as a post-init step.
- Sandbox UID, system user, overlay-dir chown logic, and ScriptBuilder API stay the same.
---
## Current Gap
- `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` invokes `systemd-run --scope ... -- bwrap [namespace flags] /bin/bash /script.sh`.
- `deploy/deploy-test-server.sh` line ~84 installs `bubblewrap` via apt/dnf.
- `deploy/tests/test_deploy_artifacts.py::test_script_sandbox_helper_invokes_systemd_run_and_bwrap` asserts `bwrap`, `--unshare-pid`, `--uid=l4d2-sandbox`, etc.
- `deploy/tests/test_deploy_artifacts.py::test_deploy_script_installs_bubblewrap` asserts `bubblewrap` is in apt/dnf install lines.
- `left4me.db` is created at deploy time with the default 0644 permissions; any host user can read it.
---
## Task 1: Rewrite the sandbox helper to be systemd-only
**Files:**
- Modify: `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` — replace the `systemd-run --scope … bwrap …` invocation with `systemd-run --service --pipe --wait …` carrying the hardening directives.
Test plan:
1. `bash -n` syntax check (already covered by `test_script_sandbox_helper_passes_shell_syntax_check`).
2. `test_deploy_artifacts.py::test_script_sandbox_helper_invokes_systemd_run_and_bwrap` is replaced by a new pin: `test_script_sandbox_helper_invokes_systemd_run_with_hardening`. Asserts:
- No `bwrap` reference remains.
- `systemd-run` is invoked with `--pipe`, `--wait`, `--collect`, `--unit=` (transient service unit form, no `--scope`).
- All hardening directives present: `NoNewPrivileges=yes`, `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp=yes`, `PrivateDevices=yes`, `PrivateIPC=yes`, `ProtectKernelTunables=yes`, `ProtectKernelModules=yes`, `ProtectKernelLogs=yes`, `ProtectControlGroups=yes`, `RestrictNamespaces=yes`, `RestrictSUIDSGID=yes`, `LockPersonality=yes`, `MemoryDenyWriteExecute=yes`, `SystemCallFilter=`, `CapabilityBoundingSet=` (empty), `User=l4d2-sandbox`, `Group=l4d2-sandbox`.
- `TemporaryFileSystem=` covers `/etc` and `/var/lib`.
- `BindReadOnlyPaths=` includes `/etc/resolv.conf /etc/ssl /etc/ca-certificates /etc/nsswitch.conf /etc/alternatives` and the script bind `${SCRIPT}:/script.sh`.
- `BindPaths=` carries the overlay bind.
- Cgroup limits unchanged (`MemoryMax=4G`, `MemorySwapMax=0`, `TasksMax=512`, `CPUQuota=200%`, `RuntimeMaxSec=3600`).
3. Existing `test_script_sandbox_helper_dry_run_mode` keeps passing — the dry-run guard still short-circuits before `systemd-run`.
4. Existing `test_script_sandbox_helper_validates_overlay_id` keeps passing — argument validation is unchanged.
Implementation: helper body verbatim from the spec §Helper.
**Verification:**
```
python3 -m pytest deploy/tests/test_deploy_artifacts.py -q
bash -n deploy/files/usr/local/libexec/left4me/left4me-script-sandbox
```
**Commit:** `refactor(deploy): rewrite left4me-script-sandbox to systemd-only — drop bwrap`
---
## Task 2: Drop bubblewrap apt/dnf dep + tighten left4me.db mode
**Files:**
- Modify: `deploy/deploy-test-server.sh` — remove `bubblewrap` from `apt-get install` / `dnf install` package lists; add a post-init step that ensures `left4me.db` is mode 0640 owned `root:left4me`.
- Modify: `deploy/tests/test_deploy_artifacts.py` — replace `test_deploy_script_installs_bubblewrap` with `test_deploy_script_does_not_install_bubblewrap`; add `test_deploy_script_tightens_left4me_db_permissions`.
Test plan:
1. `test_deploy_script_does_not_install_bubblewrap` — for each `apt-get install` / `dnf install` line, `bubblewrap` is absent.
2. `test_deploy_script_tightens_left4me_db_permissions` — script contains `chmod 0640 /var/lib/left4me/left4me.db` and `chown root:left4me /var/lib/left4me/left4me.db` (in either order).
3. `test_deploy_script_shell_syntax` keeps passing (`sh -n`).
Implementation:
- Remove the bare `bubblewrap` token from the two install lines.
- After the `alembic upgrade head` step (which creates the DB if missing), add:
```
$sudo_cmd chown root:left4me /var/lib/left4me/left4me.db
$sudo_cmd chmod 0640 /var/lib/left4me/left4me.db
```
Idempotent — re-runs are no-ops.
**Verification:**
```
python3 -m pytest deploy/tests/test_deploy_artifacts.py -q
sh -n deploy/deploy-test-server.sh
```
**Commit:** `chore(deploy): drop bubblewrap apt dep + tighten left4me.db mode to 0640 root:left4me`
---
## Task 3: Deploy + smoke-test on the test host
**Files:** none.
This is an operational verification step, not a code change. Run `deploy/deploy-test-server.sh ckn@10.0.4.128`, then on the host re-run the same smoke battery used to validate the prototype:
1. **Identity / privileges**: `id` returns `uid=996 gid=985`; `/proc/self/status` shows `NoNewPrivs: 1` and `CapBnd: 0000000000000000`.
2. **Filesystem isolation**: `/etc/passwd` absent, `/etc/alternatives/awk` present, `/var/lib/left4me/left4me.db` absent, `/home` inaccessible, `/usr` not writable, `/overlay` writable.
3. **Tools + network**: `awk` resolves through `/etc/alternatives`; `curl https://steamcommunity.com/` returns 200.
4. **Cgroup limits**: while a 5s-sleep script runs, `cat /sys/fs/cgroup/.../memory.max` returns `4294967296`; `pids.max` `512`; `cpu.max` `200000 100000`.
5. **Memory cap**: 5 GB Python alloc raises `MemoryError`.
6. **Wipe**: `find /overlay -mindepth 1 -delete` empties the overlay dir.
7. **Seccomp / restriction probes**: `unshare -U`, `mount -t tmpfs`, `setarch -X`, `bpf` setsockopt all fail with EPERM/EINVAL.
8. **Build via web UI**: log in as admin, create a script overlay with `echo "hi" > foo`, click Save, confirm job succeeds and `foo` appears in `/var/lib/left4me/overlays/{id}/foo`.
9. **DB hardening**: `stat -c "%a %U:%G" /var/lib/left4me/left4me.db` returns `640 root:left4me`.
Mark this task complete only after every check passes on the live host.
**Commit:** none (operational verification — record results in conversation/PR description).
---
## Task 4: Drift sweep + push
**Files:** as needed across the repo.
Run the full test suite for all three packages; chase any drift caused by the helper rewrite or deploy-script changes.
```
python3 -m pytest l4d2web/tests/ -q
python3 -m pytest l4d2host/tests/ -q
python3 -m pytest deploy/tests/ -q
```
Implementation: fix what breaks. Expected: nothing new should break, since the Python-side contract is unchanged. If something does, treat it as a sign of an unintended coupling and address.
Push the commits to `origin/master`.
**Verification:** all three suites green; `git status` clean; commits visible on `git.sublimity.de/cronekorkn/left4me`.
**Commit:** none unless drift fixes are needed.
---
## Rollback plan
If Task 3 surfaces a blocker (a hardening directive breaks a real-world script class, seccomp filter is too narrow, BindPaths semantics differ on the host's systemd version), roll back via `git revert` of Tasks 1+2 and redeploy. Git history preserves both the v1 and v2 helper. The Python side never changed, so reverting only the deploy artifacts is sufficient — no DB migration to undo, no template change to roll back.

View file

@ -0,0 +1,89 @@
# L4D2 Script Sandbox v3 Implementation Plan
> **Approval status:** User-approved 2026-05-08; implemented and pushed in `7e66936`. This plan is recorded retrospectively for symmetry with the v1 / v2 plans.
**Goal:** Restrict the sandbox to public-internet egress per `docs/superpowers/specs/2026-05-08-l4d2-script-sandbox-v3-egress-filter.md`. Bind a static public-resolver `resolv.conf` into the sandbox.
---
## Locked Decisions (see spec for rationale)
- `IPAddressDeny=` only; no `IPAddressAllow=any`.
- Explicit CIDRs (no `localhost` / `link-local` shorthand keywords — `systemd-run -p` parser rejects them).
- Static `nameserver 1.1.1.1` + `nameserver 8.8.8.8` in a sandbox-only resolv.conf.
- `AF_UNIX` left enabled.
---
## Current Gap (at start of this iteration)
- `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` (v2) shares the host network namespace with no egress filter.
- The helper bind-mounts `/etc/resolv.conf` from the host into the sandbox (which points at private-IP DNS).
- `deploy/deploy-test-server.sh` does not install a sandbox-only resolv.conf.
- No deploy-artifact tests for `IPAddressDeny=` or for the resolv.conf shape.
---
## Task 1: Add `IPAddressDeny=`, swap resolv.conf bind, ship the static file
**Files:**
- Create: `deploy/files/etc/left4me/sandbox-resolv.conf` — two `nameserver` lines + a header comment.
- Modify: `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` — add `-p IPAddressDeny="..."` directive (11 explicit CIDRs); replace the `/etc/resolv.conf:/etc/resolv.conf` token in `BindReadOnlyPaths=` with `/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf`.
- Modify: `deploy/deploy-test-server.sh` — add an `install -m 0644 -o root -g root .../sandbox-resolv.conf /etc/left4me/sandbox-resolv.conf` line near the existing `host.env` install.
- Modify: `deploy/tests/test_deploy_artifacts.py` — extend `test_script_sandbox_helper_invokes_systemd_run_with_hardening` to assert each CIDR is present and that `IPAddressAllow=any` is **absent** (regression guard); update the BindReadOnlyPaths assertion to expect the sandbox-resolv.conf bind; add `test_sandbox_resolv_conf_exists` and `test_deploy_script_installs_sandbox_resolv_conf`.
Test plan (RED-first not used here; the work was driven by smoke-test feedback against a live host):
1. `test_script_sandbox_helper_invokes_systemd_run_with_hardening``IPAddressDeny=` present with all 11 CIDRs; no `IPAddressAllow=any`; resolv.conf bind path is `/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf`.
2. `test_sandbox_resolv_conf_exists` — file present, ≥2 nameservers, all in non-private space.
3. `test_deploy_script_installs_sandbox_resolv_conf` — deploy script references both source path under `deploy/files/etc/left4me/sandbox-resolv.conf` and target path `/etc/left4me/sandbox-resolv.conf`.
**Verification:**
```
sh -n deploy/deploy-test-server.sh
bash -n deploy/files/usr/local/libexec/left4me/left4me-script-sandbox
python3 -m pytest deploy/tests/ -q
```
**Commit:** `feat(deploy): restrict script-sandbox egress to public internet only`
---
## Task 2: Deploy + smoke-test on `ckn@10.0.4.128`
**Files:** none.
Run `deploy/deploy-test-server.sh ckn@10.0.4.128`. Then on the host, invoke the helper with a probe script that opens TCP connections to:
- `1.1.1.1:443` — must connect (public)
- `127.0.0.1:8000` — must block (web app on loopback)
- `127.0.0.1:22` — must block (sshd on loopback)
- `10.0.4.128:22` — must block (host's external SSH on private LAN)
- `10.0.0.1:53` — must block (LAN DNS resolver)
Plus `curl -m 5 https://steamcommunity.com/` end-to-end (DNS + HTTPS) → 200.
Inside the sandbox, `cat /etc/resolv.conf` must show the two public resolvers.
If any of the localhost / private targets connects, the deny is being silently overridden — see spec §Locked Decisions point 1.
**Commit:** none — operational verification.
---
## Lessons surfaced during execution
These belong in the spec but are repeated here as the "things the next person should not have to rediscover":
- **`IPAddressAllow=any` silently overrides every `IPAddressDeny=` rule** on this systemd 257 / kernel 6.12 combo, despite documentation stating "more specific rule wins". The negative test (`IPAddressAllow=any not in text`) locks this in.
- **systemd-run's `-p` parser rejects the `localhost` / `link-local` / `multicast` shorthand keywords** even though they parse fine in unit files. Use explicit CIDRs.
- **`/var/lib/left4me/.../left4me.db` is mode 0644 by default** — writing this file from the web app left it world-readable. Tightening to 0640 root:left4me happens in v2's deploy-script change; v3 does not re-touch it.
- **bpftool ships separately on Debian.** It's not needed for runtime, but `apt-get install bpftool` is useful for inspecting `sd_fw_egress` attach state when debugging filter behaviour.
---
## Rollback
`git revert 7e66936` and redeploy. The change is purely in deploy artifacts; no app code, no DB migration. Reverting reopens the previous v2 reachability.

View file

@ -0,0 +1,353 @@
# L4D2 Global Map Overlays Design
**Goal:** Add two managed, system-wide map overlays, `l4d2center-maps` and `cedapug-maps`, populated from upstream map sources and refreshed daily through the existing job system.
**Approval status:** User-approved design direction. Implementation must not start until this spec is reviewed and an implementation plan is written.
## Context
`left4me` already has typed overlays, a builder registry, global overlays through `Overlay.user_id = NULL`, and queued overlay build jobs. Steam Workshop overlays use a cache plus symlinks into `left4dead2/addons/`, and server initialization already runs overlay builders before calling `l4d2ctl initialize`.
Global map sources fit the same model. The host library remains unchanged: it receives overlay refs and mounts directories. The web app owns map-source fetching, cache management, reconciliation, and job logs.
The two upstream sources are:
- `https://l4d2center.com/maps/servers/index.csv`
- `https://cedapug.com/custom`
## Locked Decisions
1. **One general operation.** Use `refresh_global_overlays`, not source-specific cron operations.
2. **Systemd owns time.** A systemd timer runs daily and invokes a Flask CLI command. The CLI only enqueues work; the existing worker performs downloads and writes logs.
3. **System jobs are nullable-owner jobs.** `jobs.user_id` becomes nullable. `NULL` means the job was created by the system. UI displays owner as `system`. Only admins can access system jobs.
4. **Managed global overlays are auto-seeded.** The app creates or repairs exactly one `l4d2center-maps` overlay and exactly one `cedapug-maps` overlay.
5. **Global overlays are normal system overlays for users.** `Overlay.user_id = NULL` makes them visible to every authenticated user and selectable in every user's blueprint editor.
6. **Managed types are not user-creatable.** Normal overlay creation does not offer `l4d2center_maps` or `cedapug_maps`. The seeder is the only code path that creates those types.
7. **Exact reconciliation.** Refresh makes each managed overlay match its upstream manifest. Removed upstream maps are removed from the managed overlay symlink set. Foreign files are left alone and logged.
8. **No initialize-time downloads.** `initialize_server()` may run builders to repair symlinks, but it must not fetch remote manifests or download large archives. Missing cache content fails clearly.
9. **Separate cache from Workshop.** Non-Steam global maps use `${LEFT4ME_ROOT}/global_overlay_cache`, not `${LEFT4ME_ROOT}/workshop_cache`.
10. **Source-specific parsing stays explicit.** Do not introduce a generic arbitrary HTTP source framework in this phase.
## Architecture
The design extends the existing overlay-builder registry:
```python
BUILDERS = {
"external": ExternalBuilder(),
"workshop": WorkshopBuilder(),
"l4d2center_maps": GlobalMapOverlayBuilder(),
"cedapug_maps": GlobalMapOverlayBuilder(),
}
```
Both global map overlay types share the same filesystem builder. Source-specific code lives in refresh services that know how to fetch and parse upstream manifests.
High-level flow:
```text
systemd timer
-> flask refresh-global-overlays
-> ensure_global_overlays()
-> enqueue refresh_global_overlays job (coalesced)
-> worker fetches manifests
-> worker downloads/extracts cache files
-> worker records desired VPK files
-> worker rebuilds overlay symlinks directly
```
Auto-seeded overlay rows use fixed names, managed types, `user_id = NULL`, and web-generated paths:
```text
name=l4d2center-maps, type=l4d2center_maps, user_id=NULL, path=str(id)
name=cedapug-maps, type=cedapug_maps, user_id=NULL, path=str(id)
```
## Data Model
### `jobs`
Change `jobs.user_id` from required to nullable.
`NULL` means a system-created job. Authorization rules become:
- Admins can view, stream, and cancel every job, including system jobs.
- Non-admins can access only jobs where `job.user_id == current_user.id`.
- System jobs are not visible to non-admins through direct job URLs.
Job list/detail pages use outer joins to `users` and render missing owners as `system`.
### `global_overlay_sources`
One row per managed global source overlay:
```text
id INTEGER PRIMARY KEY
overlay_id INTEGER NOT NULL UNIQUE REFERENCES overlays(id) ON DELETE CASCADE
source_key VARCHAR(64) NOT NULL UNIQUE -- l4d2center-maps | cedapug-maps
source_type VARCHAR(32) NOT NULL -- l4d2center_csv | cedapug_custom_page
source_url TEXT NOT NULL
last_manifest_hash VARCHAR(64) NOT NULL DEFAULT ''
last_refreshed_at DATETIME NULL
last_error TEXT NOT NULL DEFAULT ''
created_at DATETIME NOT NULL
updated_at DATETIME NOT NULL
```
`source_key` is stable and used by the seeder to repair missing rows.
### `global_overlay_items`
One row per manifest item belonging to a global overlay source:
```text
id INTEGER PRIMARY KEY
source_id INTEGER NOT NULL REFERENCES global_overlay_sources(id) ON DELETE CASCADE
item_key VARCHAR(255) NOT NULL -- stable per source
display_name VARCHAR(255) NOT NULL DEFAULT ''
download_url TEXT NOT NULL
expected_vpk_name VARCHAR(255) NOT NULL DEFAULT ''
expected_size BIGINT NULL
expected_md5 VARCHAR(32) NOT NULL DEFAULT ''
etag VARCHAR(255) NOT NULL DEFAULT ''
last_modified VARCHAR(255) NOT NULL DEFAULT ''
content_length BIGINT NULL
last_downloaded_at DATETIME NULL
last_error TEXT NOT NULL DEFAULT ''
created_at DATETIME NOT NULL
updated_at DATETIME NOT NULL
UNIQUE(source_id, item_key)
```
For `l4d2center`, `item_key` and `expected_vpk_name` come from the CSV `Name` column, and `expected_size` / `expected_md5` come from the CSV.
For `cedapug`, `item_key` is the direct download URL path basename, normalized without query parameters. CEDAPUG does not publish checksums in the observed page, so integrity uses HTTP metadata when available and archive extraction checks.
### `global_overlay_item_files`
One row per extracted VPK file that should appear in an overlay:
```text
id INTEGER PRIMARY KEY
item_id INTEGER NOT NULL REFERENCES global_overlay_items(id) ON DELETE CASCADE
vpk_name VARCHAR(255) NOT NULL
cache_path TEXT NOT NULL -- relative path under global_overlay_cache
size BIGINT NOT NULL
md5 VARCHAR(32) NOT NULL DEFAULT ''
created_at DATETIME NOT NULL
updated_at DATETIME NOT NULL
UNIQUE(item_id, vpk_name)
```
This extra file table handles archives that contain more than one `.vpk` without overloading the item row.
## Filesystem Layout
Use a cache separate from Steam Workshop:
```text
${LEFT4ME_ROOT}/
global_overlay_cache/
l4d2center-maps/
archives/
vpks/
cedapug-maps/
archives/
vpks/
overlays/
{overlay_id}/
left4dead2/addons/
*.vpk -> absolute symlink to global_overlay_cache/.../vpks/*.vpk
```
Cache file writes are atomic: download to `*.partial`, extract to a temporary directory, verify, then `os.replace()` final VPK files.
Symlink targets are absolute, matching the existing Workshop overlay design.
## Source Parsing
### L4D2Center
Fetch `https://l4d2center.com/maps/servers/index.csv` with a normal HTTP timeout.
The CSV is semicolon-delimited and contains:
```text
Name;Size;md5;Download link
```
Each item produces:
- `item_key = Name`
- `expected_vpk_name = Name`
- `expected_size = Size`
- `expected_md5 = md5`
- `download_url = Download link`
Downloads are `.7z` archives. Extraction uses a Python 7z implementation such as `py7zr` so tests do not depend on a system `7z` binary. After extraction, the expected VPK file must exist and match both size and md5. A mismatch fails that item and leaves the prior cached file in place.
### CEDAPUG
Fetch `https://cedapug.com/custom` and parse the embedded `renderCustomMapDownloads([...])` data.
Only direct download links are managed in v1:
- Relative links like `/maps/FatalFreight.zip` are converted to absolute `https://cedapug.com/maps/FatalFreight.zip`.
- External `http` links are logged and skipped in v1.
- Entries without a download link are built-in campaigns and skipped.
Downloads are `.zip` archives extracted with Python's standard `zipfile`. Every `.vpk` in the archive becomes a managed output file for that item. If no `.vpk` is present, the item fails and the prior cached files remain in place.
Because CEDAPUG does not publish checksums in the observed page, refresh detects changes using `ETag`, `Last-Modified`, `Content-Length`, and local extracted file metadata when available. A manual refresh can force revalidation by clearing item metadata in a later maintenance path; no force-refresh UI is included in this design.
## Refresh Job
`refresh_global_overlays` is a global worker operation.
Behavior:
1. Ensure both managed global overlays and source rows exist.
2. Fetch both manifests.
3. Upsert manifest items.
4. Mark items absent from the manifest as no longer desired by deleting their item rows; cascading deletes remove their file rows.
5. Download and extract new or changed items.
6. Keep prior cache files when an item download or verification fails, but record `last_error`.
7. Rebuild symlinks for changed sources directly through the same builder interface used by `build_overlay`.
8. Emit clear job logs: manifest counts, downloads, skips, removals, verification failures, and build summaries.
`refresh_global_overlays` does not enqueue child `build_overlay` jobs. Direct builder invocation keeps the overlay in sync before the refresh job releases its global mutex, so a server job cannot start against updated cache metadata but stale overlay symlinks.
Coalescing:
- If a `refresh_global_overlays` job is queued or running, CLI/admin requests return the existing job instead of inserting a duplicate.
## Builder Reconciliation
`GlobalMapOverlayBuilder` reads desired file rows for the overlay's source and reconciles only symlinks it manages.
Managed symlink rule:
- A symlink in `left4dead2/addons/` is managed if its resolved target is under `${LEFT4ME_ROOT}/global_overlay_cache/{source_key}/vpks/`.
- Managed symlinks absent from desired files are removed.
- Desired files missing from cache are skipped and logged as errors.
- Non-symlink files and symlinks outside the source cache are left untouched and logged as foreign entries.
This mirrors `WorkshopBuilder` behavior and keeps manual files safe.
## Scheduler Rules
`refresh_global_overlays` joins the existing global mutex group.
It must not run concurrently with:
- `install`
- `refresh_workshop_items`
- any `build_overlay`
- any server job (`initialize`, `start`, `stop`, `delete`)
No server or overlay job may start while `refresh_global_overlays` is running.
This conservative rule is acceptable because daily map refreshes are rare and large downloads should not race runtime changes.
## CLI And Systemd Timer
Add Flask CLI command:
```text
flask refresh-global-overlays
```
The command:
- Loads app config and DB.
- Ensures global overlays exist.
- Enqueues or returns the existing `refresh_global_overlays` job.
- Prints the job id.
- Does not run downloads itself.
Add deployment units:
```text
left4me-refresh-global-overlays.service
left4me-refresh-global-overlays.timer
```
Service command:
```text
/opt/left4me/.venv/bin/flask --app l4d2web.app:create_app refresh-global-overlays
```
Timer policy:
```text
OnCalendar=daily
Persistent=true
```
The service runs as the `left4me` user with `/etc/left4me/host.env` and `/etc/left4me/web.env`, matching `left4me-web.service`.
## Permissions And UI
Overlay list behavior:
- Admins see all overlays, including managed global map overlays.
- Non-admin users see system overlays and their own private workshop overlays.
- Managed global overlays appear in blueprint overlay selection for every user.
Creation behavior:
- Non-admin users can create only user-creatable types, currently `workshop`.
- Admins can create normal admin-creatable types, currently `external` and `workshop`.
- No user-facing create form offers `l4d2center_maps` or `cedapug_maps`.
- Auto-seeding is the only creation path for managed global map overlay types.
Admin controls:
- Add a manual "Refresh global overlays" action in the admin area.
- The action enqueues the same coalesced `refresh_global_overlays` job as the timer.
- Managed overlay detail pages show source type, source URL, last refresh time, last error, item count, and latest related jobs.
## Error Handling
- Manifest fetch failure fails the job if no source can be processed. If one source succeeds and one fails, the job should still finish failed with partial-success logs and preserve prior content for the failed source.
- Per-item download failures do not abort sibling items.
- Verification failures keep prior cached files and record `last_error` on the item.
- Extraction rejects path traversal entries and ignores non-VPK files.
- Unsupported CEDAPUG external links are skipped with a warning.
- Initialize-time checks fail if desired global map files are missing from cache, naming the overlay and missing VPK names.
## Tests
Test coverage should include:
- Auto-seeding creates exactly one source overlay per source and repairs missing source rows.
- `jobs.user_id` nullable behavior, outer joins, and `system` display.
- Non-admins cannot access system jobs directly.
- CLI coalesces queued/running `refresh_global_overlays` jobs.
- Scheduler truth table for the new global operation.
- L4D2Center CSV parser with semicolon-delimited fixture data.
- CEDAPUG embedded JavaScript parser with fixture HTML.
- L4D2Center download/extract verifies VPK size and md5.
- CEDAPUG download/extract records every VPK in a zip archive.
- Reconcile removes obsolete managed symlinks and leaves foreign files alone.
- Overlay create UI rejects managed singleton types.
- Blueprint overlay selection includes managed global overlays for all users.
- Deployment tests cover the service and timer artifacts.
## Out Of Scope
- User-created global map source overlays.
- Arbitrary configurable HTTP manifest sources.
- Force-refresh UI for CEDAPUG items.
- Cache garbage collection for unreferenced archive files.
- Client-side map download UX.
- Steam Workshop links discovered on the CEDAPUG page; those are skipped rather than imported into workshop overlays.
- Host-library awareness of managed overlay types.
## Implementation Boundaries
- `l4d2host` remains unchanged.
- The web app continues to call host operations only through `l4d2ctl`.
- Existing blueprint semantics remain unchanged: overlays are live-linked, ordered, and first overlay has highest precedence.
- Existing workshop overlay behavior remains unchanged except scheduler interactions with the new global operation.

View file

@ -0,0 +1,226 @@
# L4D2 Workshop Overlays Design
**Goal:** Let users add Steam Workshop content (.vpk addons and maps) to L4D2 servers from the web UI. Workshop downloads run as a new typed overlay that fits the existing `Overlay` + `BlueprintOverlay` model, downloaded via the public Steam Web API and exposed through the existing fuse-overlayfs mount layer.
**Approval status:** User-approved design direction. Implementation proceeds in lockstep with the companion plan at `docs/superpowers/plans/2026-05-07-l4d2-workshop-overlays.md`.
## Context
`left4me` users today add `.vpk` content to a server only by SFTP-ing files into a manually-prepared overlay directory or by maintaining shell scripts (`competitive_rework`, `workshop_maps`, `tickrate`, etc.) that wrap `curl`/`steamcmd`. The web app exposes overlay rows but offers no way for users to populate them.
This spec adds **workshop overlays**: a user-private overlay type that downloads `.vpk` files via the public `ISteamRemoteStorage` API and surfaces them through the existing mount layer. Users keep composing blueprints by stacking overlays — workshop overlays become another row alongside today's externally-managed ones.
This is the first *typed* overlay. The design adds a `type` column and a builder-registry so future overlay types (tarball, inline, manual upload) plug in without schema churn or workflow changes.
Steam Workshop content for L4D2 (consumer_app_id 550) is downloadable via two anonymous-POST endpoints with no Steam Web API key required: `GetCollectionDetails` resolves a collection ID to its child item IDs, and `GetPublishedFileDetails` returns per-item metadata including a public `file_url` for the `.vpk`. This is the same API the user's existing `steam-workshop-download` script uses.
L4D2-specific player-side pain points (sv_consistency / RestrictAddons configuration gotchas, the inability to push workshop content via `sv_downloadurl`) are documented in **Out of scope** and tracked as separate follow-ups. This spec stays strictly on workshop content acquisition.
## Locked Decisions
1. **Typed overlays.** `Overlay.type` joins `external` (existing rows; admin-managed; no-op builder) and `workshop` (new). Future types — tarball, inline, manual upload — slot in via the same builder registry without schema churn.
2. **No JSON `source_config` blob.** Per-type structured data lives in proper relational tables. JSON is reserved for genuinely opaque diagnostic payloads.
3. **Central deduplicated `WorkshopItem` registry** keyed on `steam_id`. Cache lives at `/var/lib/left4me/workshop_cache/{steam_id}.vpk`. Multiple overlays referencing the same Steam item share the same cache file.
4. **Symlinks, not copies.** Overlay directories contain `left4dead2/addons/{steam_id}.vpk` symlinks pointing into the cache. Both the cache file and the symlink are named by `{steam_id}` only — no Steam filename in any on-disk path, so Steam can rename the upstream `.vpk` without breaking lookup.
5. **Many-to-many association is pure** (no `enabled` flag). Toggle a workshop item by removing or re-adding the association. The shared cache makes this cheap.
6. **Collections are atomic UI bulk-imports.** Pasting a collection URL/ID resolves member items and creates N item associations. The DB never tracks "this came from a collection." Re-importing a collection is idempotent on existing items and additive for new ones.
7. **Single global admin "Refresh all workshop items" button.** One Steam metadata batch call, then re-download items whose `time_updated` advanced. No per-item, per-overlay, or scheduled refresh in v1.
8. **No cache GC in v1.** Cache grows monotonically. Reference-counted cleanup is a follow-up.
9. **Globality is independent of overlay type.** `Overlay.user_id` is the scope (NULL = system-wide, set = private to that user). v1 defaults newly-created workshop overlays to private and leaves existing external overlays as system-wide. A future "publish/share" button will let owners toggle `user_id` without changing type.
10. **One unified "Create overlay" UI button.** Modal has a type radio (External | Workshop). No path field — the web app generates the path for every new overlay.
11. **Strict scope.** v1 ships only the workshop type. L4D2 server-config gotchas, client-subscription helpers, other recipe types — all deferred to follow-up specs.
12. **`consumer_app_id == 550` validation** at every Steam API response at fetch/add time; non-L4D2 items are rejected and never reach the row. The value is a fixed precondition, not data.
13. **Input field accepts numeric ID, full Workshop URL, or a multi-line batch** of either. Pasting `123456` and pasting `steamcommunity.com/sharedfiles/filedetails/?id=123456` produce the same result; pasting many of either at once works too.
14. **Web-managed overlay paths.** All new overlays (any type) get `path = str(overlay_id)` at insert time. The user never picks a path. Existing legacy external overlay rows keep their current path values; migrating them to the ID-based scheme is a follow-up. `Overlay.id` uses SQLite `AUTOINCREMENT` so deleted IDs are never reused.
15. **Auto-rebuild on item change.** Adding or removing items from a workshop overlay automatically enqueues a `build_overlay` job. The "Rebuild" button on the detail page is for manual recovery only. New build jobs for an overlay coalesce with any pending one for the same overlay (don't queue duplicates).
16. **HTTPS** for all Steam Web API calls. The reference downloader uses HTTP; we don't.
## Architecture
```
Overlay row (type=workshop)
└─refs─▶ overlay_workshop_items
└─▶ WorkshopItem (global, by steam_id)
▼ download (Steam GetPublishedFileDetails + HTTP GET)
workshop_cache/{steam_id}.vpk
overlay_dir/left4dead2/addons/{steam_id}.vpk ─symlink─┘
```
Build dispatch via a registry:
```python
BUILDERS = {"external": ExternalBuilder(), "workshop": WorkshopBuilder()}
def build_overlay(overlay_id):
overlay = db.get(Overlay, overlay_id)
BUILDERS[overlay.type].build(overlay, on_stdout, on_stderr, should_cancel)
```
`ExternalBuilder` is a no-op for legacy admin-managed dirs. `WorkshopBuilder` performs an idempotent diff-apply of `addons/` symlinks against the current associations. Future types add their own builders without changing the dispatcher, the mount layer, or the blueprint editor.
## Data Model
### `Overlay` (extended)
```
id INTEGER PK AUTOINCREMENT
name VARCHAR(255) NOT NULL
path VARCHAR(255) NOT NULL -- new overlays: str(id); legacy externals: existing values
type VARCHAR(16) NOT NULL -- 'external' | 'workshop' (extensible)
user_id INTEGER NULL REFERENCES users(id) -- NULL = system-wide
created_at, updated_at
UNIQUE INDEX on (name) WHERE user_id IS NULL -- system overlays globally unique by name
UNIQUE INDEX on (name, user_id) WHERE user_id IS NOT NULL -- per-user namespace
INDEX on (type, user_id)
```
Two partial unique indexes are required because a naive composite `UNIQUE(name, user_id)` doesn't constrain externals — SQLite treats NULL as distinct in unique constraints, so two externals could share a name. Partial indexes preserve the prior global-uniqueness invariant for system rows.
### `WorkshopItem` (new)
```
id INTEGER PK
steam_id VARCHAR(20) NOT NULL UNIQUE -- 64-bit, store as text
title VARCHAR(255) NOT NULL DEFAULT ''
filename VARCHAR(255) NOT NULL DEFAULT '' -- upstream Steam filename, display only
file_url TEXT NOT NULL DEFAULT ''
file_size BIGINT NOT NULL DEFAULT 0
time_updated INTEGER NOT NULL DEFAULT 0 -- Steam epoch
preview_url TEXT NOT NULL DEFAULT '' -- thumbnail URL hot-linked from Steam
last_downloaded_at DATETIME NULL
last_error TEXT NOT NULL DEFAULT ''
created_at, updated_at
```
`consumer_app_id` is **not** stored. It's validated at fetch time and the row never exists for non-L4D2 items.
### `overlay_workshop_items` (new, pure association)
```
id INTEGER PK
overlay_id INTEGER NOT NULL REFERENCES overlays(id) ON DELETE CASCADE
workshop_item_id INTEGER NOT NULL REFERENCES workshop_items(id) ON DELETE RESTRICT
UNIQUE (overlay_id, workshop_item_id)
INDEX (workshop_item_id) -- reverse lookup for refresh
```
No `enabled` column — toggle is remove/add, which is cheap because the cache survives.
### `Job` (extended)
Add `overlay_id INTEGER NULL REFERENCES overlays(id)` for `build_overlay` jobs.
## Filesystem Layout
```
/var/lib/left4me/
overlays/
{overlay_id}/ # flat — same shape for every type
left4dead2/addons/
{steam_id}.vpk -> /var/lib/left4me/workshop_cache/{steam_id}.vpk
workshop_cache/
{steam_id}.vpk # one file per Steam item
```
- Every new overlay (workshop, future tarball/inline/manual) lives at `overlays/{overlay_id}/`. Legacy external overlays keep their pre-migration paths (e.g. `overlays/standard/`).
- `workshop_cache/` is created during deploy provisioning, not lazily — avoids races between concurrent first downloads.
- Web user owns both trees (mode 0755). Host user (`l4d2ctl`) needs read on both. If web and host are different users, they share a group.
- Symlink targets are absolute. Relative targets resolve in the merged-mount namespace and break across the host/web boundary.
- The builder never creates a dangling symlink. If a `WorkshopItem` lacks a cache file at build time, the builder logs a warning and skips it — fuse-overlayfs surfaces broken links to L4D2 as opaque addon-scan failures.
## UI
A single "Create overlay" button on `/overlays` opens a modal with type radio (External | Workshop) and a name field. No path field. The web app generates `path = str(overlay_id)` after insert.
Workshop overlay detail page (`/overlays/{id}` when `type='workshop'`) shows:
- A multi-line input plus a radio (Items | Collection). Pasting one or many IDs/URLs adds them in order; pasting a collection ID resolves its members.
- An item table with: thumbnail (`preview_url`), `steam_id` linking to Steam, title, filename, last-updated, size, last-error if any, Remove.
- A manual "Rebuild" button (for recovery only — every add/remove auto-enqueues a coalesced `build_overlay` job).
- Status indicator pulled from the latest related `Job` row.
External overlay detail page is unchanged in shape: read-only path display, name edit (admin only). The "External" type retains the existing admin-only SFTP-to-disk workflow until a future "manual upload" type replaces it.
The blueprint editor is unchanged in structure. Workshop overlays appear alongside externals in the user's overlay picker; ordering and stacking semantics are identical.
Admin section gets one new control: "Refresh all workshop items" button on the admin landing or workshop subsection. Pressing it enqueues a single `refresh_workshop_items` job.
### Routes
| Method | Path | Purpose |
|---|---|---|
| GET | `/overlays` | List with Type column, filtered by user permissions |
| POST | `/overlays` | Create; reads `type` and `name` only |
| GET | `/overlays/{id}` | Type-aware detail page |
| POST | `/overlays/{id}/items` | Add items or collection; auto-enqueues coalesced `build_overlay` |
| POST | `/overlays/{id}/items/{item_id}/delete` | Remove association; auto-enqueues coalesced `build_overlay` |
| POST | `/overlays/{id}/build` | Manual rebuild (recovery) |
| POST | `/admin/workshop/refresh` | Admin only; enqueue `refresh_workshop_items` |
HTMX usage stays minimal: only the add-item form and per-row delete swap a fragment. Everything else is full-page POST/redirect/GET.
## Job Operations
Two new operations join the existing job worker:
- **`build_overlay(overlay_id)`** — `Job.overlay_id` is set; `server_id` is NULL. Dispatches to `BUILDERS[overlay.type].build(...)`. Cancellation between filesystem operations.
- **`refresh_workshop_items()`** — admin-only. Both `server_id` and `overlay_id` are NULL. Phases: fetch all metadata in one batched call, download items where `time_updated` advanced, enqueue (coalesced) `build_overlay` for affected overlays. v1 doesn't wait on child builds; the admin sees them in the jobs list.
### Scheduler rules
- `install` and `refresh_workshop_items` are mutually exclusive with each other, with all `build_overlay`s, and with all server jobs.
- `build_overlay(overlay_id=N)` blocks if `install_running`, `refresh_running`, or another build for the same `overlay_id` is running. Builds for *different* overlays may run concurrently.
- Server start/init blocks if `refresh_running` or any `build_overlay` for an overlay referenced by the server's blueprint is running.
Coalescing: a new `build_overlay` for an overlay that already has a queued (not-yet-running) build returns the existing job instead of inserting a new row.
`initialize_server` synchronously calls each overlay's builder before writing the spec for `l4d2ctl initialize`. If a workshop overlay references uncached items (no file in `workshop_cache/`), `initialize_server` fails fast with a clear error naming the missing IDs and pointing the user at the overlay page. It never silently mounts a partial overlay.
## Permissions
- **External overlays**: admin-only create/edit. Visible to all authenticated users (system-wide).
- **Workshop overlays**: any logged-in user can create. Owner or admin can edit and delete. Visible to the owner and admins.
- **Admin refresh**: admin-only.
The `Overlay` listing query for non-admins becomes: `type='external' OR user_id=current_user.id`.
## Risks
- **Broken symlinks across host/web boundary** — mitigated by absolute targets, build-time pre-check skipping uncached items, and `deploy/` documenting permission requirements.
- **Initialize against uncached items** — would silently mount overlays missing maps. Mitigated by `initialize_server`'s fail-fast check; tested.
- **Steam API rate limits** — refresh of 100 items is one metadata POST plus 100 downloads at 8-way parallelism. No retry/backoff in v1; 429s surface verbatim in the job log.
- **Partial failure during refresh** — each item is independent; per-item errors land on the row. Re-running refresh retries failures.
- **Concurrent same-ID adds**`WorkshopItem.steam_id` unique handles cache dedup. `(overlay_id, workshop_item_id)` unique catches double-association; the route returns "already in overlay" rather than 500.
- **Build coalescing missed** — would enqueue dozens of redundant builds during multi-item adds. Mitigated by the `enqueue_build_overlay` helper; tested.
- **Worker concurrency rule miss** — the truth-table test in `test_job_worker.py` is the only way to trust the new scheduler logic; written before dispatch.
- **DB/disk drift** — a stray directory left by a prior failed delete could shadow a fresh overlay. Mitigated by `AUTOINCREMENT` (no ID reuse) and `os.makedirs(exist_ok=False)` (loud failure on collision).
- **Partial unique gap on SQLite** — naive composite `UNIQUE(name, user_id)` doesn't constrain externals because NULL is distinct. Mitigated by two partial unique indexes; tested explicitly.
- **Cache growth without GC** — accepted v1 trade-off.
- **Item removed from Steam** — refresh marks `result != 1`; row keeps last good cache file; UI surfaces error string. Operator decides removal.
- **L4D2 containerized run** — symlink absolute targets break if the server runs in a different mount namespace. Re-evaluate when containerization comes up.
## Out Of Scope
These came up in research and dialog but stay out of v1:
- **Publish / share button on overlays.** Lets owners flip `Overlay.user_id` between their own ID and NULL without changing type. The schema already supports it; only the UI is deferred.
- **Migrate legacy external overlay paths to the ID-based scheme.** Existing external rows keep their pre-migration paths in v1; a follow-up migration moves the directories on disk and updates the rows.
- **Switch from fuse-overlayfs to kernel overlayfs via a privileged helper.** Matches the existing systemd / steam-install sudoers helper pattern under `/usr/local/libexec/left4me/`. Workshop overlays would work identically under either mount engine — symlinks resolve through normal VFS in both.
- **`sv_consistency` / `addonconfig.cfg RestrictAddons` auto-handling.** When a workshop overlay attaches to a blueprint, surface a banner with a one-click fix. Most-cited L4D2 player pain.
- **Shareable Steam Workshop collection link for clients.** Server cannot push workshop content via `sv_downloadurl`; clients must subscribe themselves. A panel-generated collection makes that one click for players. Requires Steam OAuth.
- **Other overlay types.** `tarball` (covers the old `competitive_rework` GitHub-tarball recipe), `inline` (covers `tickrate`'s inline `server.cfg`), `manual` (file manager / upload, replaces the admin-SFTP external workflow). All slot in via the builder registry without schema churn.
- **Cache GC.** Reference-counted delete or admin "Clear unreferenced" page.
- **Per-item / per-overlay / scheduled refresh.** v1 has one global admin button; revisit if users want finer control.
- **Update-aware server restart UX.** Notify users when a running server's overlay content has been refreshed underneath it.
## Implementation Boundaries
- The host library contract is unchanged. Workshop content arrives in overlay directories the same way externals do today; `l4d2host` doesn't know overlays have types.
- The job-execution model is preserved: same workers, same logs, same cancel callbacks. Only the operations table grows.
- The blueprint privacy model and desired-vs-actual server state model are unchanged.
- No new frontend dependencies. Vendored HTMX + custom CSS + small inline JS.
- No new Steam Web API key required; both endpoints used accept anonymous POSTs.
- The companion implementation plan governs task ordering and verification commands. Implementation must not start without explicit user approval per that plan's gate.

View file

@ -0,0 +1,80 @@
# Kernel Overlayfs Helper Design
**Goal:** Replace the per-instance `fuse-overlayfs` mount with kernel-native overlayfs invoked through a privileged sudo helper that mounts in PID 1's mount namespace. Restores host-namespace visibility of the merged overlay so gameserver units (`left4me-server@%i.service`) can `chdir` into it at unshare time.
**Approval status:** User-approved design direction. Implementation proceeds in lockstep with the companion plan at `docs/superpowers/plans/2026-05-08-kernel-overlayfs-helper.md`.
## Context
**Symptom.** After redeploys, starting a gameserver leaves the systemd unit in `activating (auto-restart)` with `status=200/CHDIR — Changing to the requested working directory failed: No such file or directory`. Investigation showed:
- `fuse-overlayfs` running as `left4me` user mounts in `left4me-web.service`'s mount namespace.
- `ProtectSystem=full` + `ReadWritePaths=/var/lib/left4me` forces `PrivateMounts=yes` on the unit (`systemd-analyze security` confirms).
- The unit's bind of `/var/lib/left4me` shows `shared:471 master:1` in `/proc/<pid>/mountinfo` — slave-receive-only — so mounts created beneath it never propagate back to host.
- `MountFlags=shared` (added in commit `1968684` to fix this) sets only the unit's *root* propagation; it does not override the slave-direction propagation that `ProtectSystem`/`ReadWritePaths` apply to their bind mounts. The gameserver unit, on unshare, inherits *host* mounts and sees nothing at the merged path → CHDIR fails.
The system *appeared* to work for ~1d8h before this investigation because the prior fuse daemon happened to land in the host namespace via some transient state. The mechanism documented in `1968684` does not reliably work on systemd 257 with this hardening shape.
**Out-of-scope item now in scope.** The 2026-05-07 workshop-overlays spec already lists this transition at line 211: *"Switch from fuse-overlayfs to kernel overlayfs via a privileged helper. Matches the existing systemd / steam-install sudoers helper pattern under `/usr/local/libexec/left4me/`."* The mount-propagation bug is the trigger to do it now.
## Locked Decisions
1. **Privileged helper does the mount.** New `left4me-overlay` script under `/usr/local/libexec/left4me/`, invoked via `sudo -n`. Mirrors the existing `left4me-systemctl` and `left4me-journalctl` pattern. The helper enters PID 1's mount namespace via `nsenter --mount=/proc/1/ns/mnt` and then calls `/bin/mount -t overlay …` or `/bin/umount`. Result: all overlay mounts live in the host namespace, visible to gameserver units.
2. **Kernel-native overlayfs, not fuse.** Once a privileged helper exists, fuse-overlayfs's rootless-mount-via-setuid-`fusermount3` advantage disappears. Kernel overlayfs is faster, has no long-running daemon, simpler unmount, and one fewer runtime dep.
3. **Helper is Python, not shell.** Path canonicalization, env-file parsing, and lowerdir prefix-allowlist validation are too brittle in shell. Uses system `/usr/bin/python3` (never the venv) and stdlib only. Owned by root, mode 0755.
4. **Verbs are `mount` and `umount`.** Matches the kernel/userspace utility names; reduces cognitive friction over `unmount`.
5. **Helper takes only the instance name as input.** It reads `${LEFT4ME_ROOT:-/var/lib/left4me}/instances/<name>/instance.env` for `L4D2_LOWERDIRS=` and computes `upper`/`work`/`merged` from the runtime root. Equivalent in security to taking lowerdirs as args (the user already controls instance.env), and produces a one-line audit trail in `journalctl _COMM=sudo`.
6. **Strict path validation in the helper.**
- Instance name matches `^[a-z0-9][a-z0-9_-]{0,63}$` (mirrors `validate_instance_name` in `l4d2host/paths.py`).
- Each lowerdir from `L4D2_LOWERDIRS` is `os.path.realpath`'d and must resolve under one of an allowlist: `installation/`, `overlays/`, `global_overlay_cache/`, `workshop_cache/`. Empty entries and traversals are rejected.
- `upper`/`work`/`merged` must resolve exactly to `runtime/<name>/{upper,work,merged}`.
- Lowerdir count ≤ 500 (kernel overlayfs hard cap; was 64 before kernel 5.2).
7. **Whiteout-format guard.** `fuse-overlayfs` running as non-root uses `user.fuseoverlayfs.*` xattrs for whiteouts and opaque dirs, which kernel overlayfs ignores entirely. Before mounting, the helper walks `upperdir` once and refuses if any such xattr is present. Defensive; catches a stale fuse-era upperdir that wasn't wiped during migration.
8. **One-time migration: wipe existing `upper/` and `work/`.** Deploy script runs a gated migration (sentinel file `/var/lib/left4me/.kernel-overlay-migrated`) that stops gameservers, stops web service, unmounts any stale fuse/overlay mounts, recreates empty `upper`/`work` dirs for every instance. Players' in-place edits to merged content are sacrificed; v1 accepts this for a test deployment.
9. **Sudoers verb constraints.** `left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-overlay mount *, /usr/local/libexec/left4me/left4me-overlay umount *`. Defense in depth (real validation lives in the helper); makes `sudo -l` output self-documenting.
10. **Wire the existing `OverlayMounter` ABC through.** `start_instance`/`stop_instance`/`delete_instance` today bypass the abstraction at `l4d2host/fs/base.py`. The new `KernelOverlayFSMounter` replaces the unused `FuseOverlayFSMounter` AND becomes the only path through `instances.py`. `FuseOverlayFSMounter` and the `fuse_overlayfs.py` module are deleted.
11. **Double-mount guard in `start_instance`.** Kernel mounts persist when the web worker dies (unlike fuse daemons, which die with their cgroup). `start_instance` checks `os.path.ismount(merged)` and refuses with a clear error rather than double-mounting.
12. **Hardening cleanup on `left4me-web.service`.** Drop `MountFlags=shared` (no longer the mechanism). Restore `PrivateTmp=true` (was dropped in commit `593611e` for fuse propagation that did not work). Keep `NoNewPrivileges` unset (sudo still requires setuid). Update the comment block to reflect the new model.
13. **AGENTS.md contracts unchanged.** The host library's CLI surface (`install`, `initialize`, `start`, `stop`, `delete`, `status`, `logs`) is unchanged. The web app continues to drive operations via `l4d2ctl`. The fuse-overlayfs implementation detail was never part of the public contract.
## Architecture
```
left4me-web.service (hardened, private mount namespace)
│ start_instance(name=…)
l4d2host.instances.start_instance
│ KernelOverlayFSMounter().mount(merged=…)
sudo -n /usr/local/libexec/left4me/left4me-overlay mount <name>
│ • validate name (regex)
│ • parse instance.env → L4D2_LOWERDIRS
│ • realpath each lowerdir, prefix-allowlist check
│ • compute upper/work/merged under runtime/<name>/
│ • walk upperdir, refuse if any user.fuseoverlayfs.* xattr
nsenter --mount=/proc/1/ns/mnt -- \
/bin/mount -t overlay overlay \
-o "lowerdir=…,upperdir=…,workdir=…" \
/var/lib/left4me/runtime/<name>/merged
host mount namespace now has the overlay; gameserver unit, on
unshare, inherits it and CHDIRs into …/merged/left4dead2 successfully.
```
## Operational Notes
- **Migration ordering on the test box (test-server, …).** The deploy script must, in order: (1) stop all `left4me-server@*.service`, (2) stop `left4me-web.service` (kills any lingering fuse-overlayfs daemons by reaping their cgroup), (3) `findmnt` + force-unmount any leftover fuse/overlay mounts under `/var/lib/left4me/runtime/`, (4) wipe and recreate `upper`/`work` for every instance, (5) deploy + start the new code. The sentinel file `/var/lib/left4me/.kernel-overlay-migrated` gates reruns.
- **Filesystem.** `/var/lib/left4me` is btrfs on the test box. Kernel overlayfs on btrfs is supported on kernel ≥ 5.10; the box is on 6.12 — fine. AppArmor ships enabled on Debian Trixie; verify no overlay-related denials in `journalctl -k` after first start.
- **Concurrency.** Two threads racing on `start_instance` for the same name is a latent issue unaffected by this change. The double-mount guard partly mitigates: the loser hits the existing mount and errors cleanly.
## Out Of Scope
- **Replace `sudo` with `AmbientCapabilities=CAP_SYS_ADMIN`** on a dedicated helper unit. Broader blast radius than the wrapper-script approach.
- **A `systemd-mount` per-instance mount unit.** Considered as the alternative architectural fix but adds more moving parts than the helper-script approach. The helper matches the established privileged-helper pattern in this codebase.
- **Re-enable `NoNewPrivileges` on `left4me-web.service`.** Requires removing sudo; not feasible while the helper invocation pattern stays.
- **Multi-process job-worker-claim safety.** The `_claim_lock` in `l4d2host/services/job_worker.py:131-138` is process-local; correctness depends on `--workers 1`. This change doesn't touch it.
- **Replicating the migration on production deployments.** v1 covers only the test-server deployment shape.

View file

@ -0,0 +1,323 @@
# L4D2 Script Overlays Design
**Goal:** Add a single new overlay type, `script`, that lets users author arbitrary build recipes as bash and runs them inside a `bubblewrap` + `systemd-run --scope` sandbox. The new type subsumes the existing `l4d2center_maps` and `cedapug_maps` managed-globals overlay types, both of which are removed in the same change. After this work the overlay type list is exactly `workshop` (unchanged) and `script` (new).
**Approval status:** User-approved design direction. Implementation proceeds in lockstep with the companion plan at `docs/superpowers/plans/2026-05-08-l4d2-script-overlays.md`.
## Context
`left4me` users today have two ways to add content to a server: workshop overlays (rich UI for Steam Workshop items via `WorkshopBuilder`) and a pair of managed global-map overlay types (`l4d2center_maps`, `cedapug_maps`) with bespoke parsers, per-item DB rows, ETag-based change detection, and a daily refresh timer. They cannot author arbitrary build recipes.
The user's previous setup at `ckn-bw/bundles/left4dead2/files/scripts/overlays/` expressed every recipe as a small bash file: `competitive_rework` (GitHub tarball download), `tickrate` (inline `server.cfg` + addon DLL fetch), `standard` (workshop items + admin-list write), `workshop_maps` (workshop collection import), `l4d2center_maps` (CSV-driven map sync). All five fit naturally into a single "run a sandboxed bash script that populates the overlay dir" model.
The two managed global-map types in the current codebase are over-engineered for what they do — each is essentially "fetch a manifest, download archives, extract VPKs, place in `addons/`." Folding them into the new `script` type eliminates three database tables, two source-parser modules, the `GlobalMapOverlayBuilder`, the `py7zr` dependency, the global-overlay cache root, and the managed-singleton machinery, while letting an admin paste the equivalent shell code (which the user already wrote years ago) into a normal admin-owned, system-wide script overlay.
The trust model for the sandbox is "semi-public deployment, registered users." The threat surface is one user reading another user's overlay, the application DB, or arbitrary host secrets, plus runaway scripts exhausting disk/CPU/RAM. Network access is *not* restricted — scripts must be able to download from arbitrary URLs (GitHub, l4d2center, Steam CDN). Sandbox boundaries are namespace-based (mount, PID, IPC, UTS, cgroup), not command-allowlist-based; binary-allowlist sandboxing of bash is theatre because of `eval` and `exec`.
The test deploy DB is wiped as part of rollout; no data migration is performed. Existing user blueprints that reference `l4d2center_maps` or `cedapug_maps` overlay rows do not survive the change in the test environment.
A scheduled-refresh feature (the daily timer that today drives the global-map types) is intentionally **out of scope for this iteration**. The two existing systemd units and the `flask refresh-global-overlays` CLI command are deleted with no replacement. Refresh is reintroduced in a later iteration designed against concrete needs.
## Locked Decisions
1. **Single new overlay type: `script`.** Replaces both managed-globals types. Final type list: `workshop` + `script`. No `tarball`/`inline`/`manual` types — all of those collapse into `script` (with UI templates as a future ergonomics improvement).
2. **`Overlay.script` is a DB `TEXT` column** holding the raw bash. No file storage, no revision history in v1. Empty string for `workshop` rows.
3. **Build idempotency contract: script runs against the existing overlay dir.** No automatic wipe between builds. Users write `test -f … || curl …`-style guards if they want bandwidth efficiency. A manual "Wipe overlay" button on the detail page resets the dir to empty.
4. **No left4me-aware helpers in the sandbox.** The script sees pure bash plus whatever's in `/usr` (RO bind-mount of the host). Workshop items are not exposed via a helper — users wanting workshop content create a `workshop`-type overlay, which has its own first-class UX (thumbnails, collection paste, dedup cache, refresh).
5. **Sandbox engine: `bubblewrap` (`bwrap`) inside `systemd-run --scope --collect`.** `systemd-run` provides cgroup v2 limits + walltime kill via `RuntimeMaxSec`; `bwrap` provides the namespace isolation. Both are stable, well-audited, in-tree on Debian.
6. **Resource limits (system-wide, not per-overlay):** 1 hour walltime (`RuntimeMaxSec=3600`), 4 GB RAM (`MemoryMax=4G`, `MemorySwapMax=0`), 512 tasks, 200% CPU quota, post-build 20 GB disk cap on `du -sb` of the overlay dir.
7. **Network: host-shared.** No `--unshare-net`. Scripts have full outbound. Egress filtering is not in v1; the sandbox prevents reading internal state but does not prevent talking to internal IPs. Acceptable for the current trust model.
8. **No auto-seeding of "default" overlays.** Admin manually creates the equivalents of the old `l4d2center-maps`/`cedapug-maps` post-deploy by pasting the bash. The deploy script does not insert overlay rows.
9. **Daily/scheduled refresh: out of scope for this iteration.** No `auto_refresh` flag, no timer, no CLI command. Manual rebuild via the detail-page button is the only build trigger after this change.
10. **Permissions mirror workshop overlays.** Any logged-in user can create a private (`user_id = me`) script overlay. Admin can create system-wide (`user_id = NULL`). Owner or admin can edit/delete.
11. **Failure semantics via `Overlay.last_build_status`** (`'' | 'ok' | 'failed'`). Drives a "rebuild required" badge on the list and detail pages. Server initialization does **not** auto-block on `failed` (matches workshop's current behavior).
12. **Wipe is just another sandbox invocation.** The wipe endpoint runs the literal script `find /overlay -mindepth 1 -delete` through the same `left4me-script-sandbox` helper. No second helper, no privilege/UID puzzle (files are owned by `l4d2-sandbox`, who runs the wipe). After a successful wipe, `last_build_status` is reset to `''`. Wipe does **not** auto-enqueue a rebuild — the user decides.
13. **Privileged helper: `/usr/local/libexec/left4me/left4me-script-sandbox`.** Same pattern as the existing `left4me-overlay`, `left4me-systemctl`, `left4me-journalctl` helpers. Bash, owned root, mode 0755. The web user invokes it via `sudo -n` per a sudoers fragment. Root is needed to set up the namespaces; bwrap drops to the unprivileged `l4d2-sandbox` UID immediately.
14. **Dedicated sandbox UID `l4d2-sandbox`** (system user, `/usr/sbin/nologin`, no home). Owns nothing on the host outside what bwrap binds in. UID-drop happens inside the bwrap invocation via `--uid`/`--gid`.
15. **Strict argument validation in the helper.** Overlay id matches `^[0-9]+$`; overlay dir must exist under `/var/lib/left4me/overlays/`; script path must exist. Defense in depth — the real authorization check lives in the web app.
16. **Streaming I/O via the existing `run_with_streamed_output` helper.** Same plumbing `WorkshopBuilder` already uses for `steamcmd`/`curl` invocations. No new SSE/log path.
## Architecture
```text
Overlay row (type=script, script=TEXT, last_build_status)
▼ build_overlay(overlay_id) job
▼ BUILDERS["script"].build(overlay, on_stdout, on_stderr, should_cancel)
▼ ScriptBuilder writes overlay.script → tmpfile, then:
│ sudo -n /usr/local/libexec/left4me/left4me-script-sandbox <id> <tmpfile>
▼ Helper validates args, then exec()s:
│ systemd-run --scope --collect
│ -p MemoryMax=4G -p MemorySwapMax=0
│ -p TasksMax=512 -p CPUQuota=200%
│ -p RuntimeMaxSec=3600
│ -- bwrap [namespace flags...] /bin/bash /script.sh
▼ Inside the sandbox the script sees:
│ /overlay ← /var/lib/left4me/overlays/{id} RW (the build target)
│ /tmp,/run ← fresh tmpfs RW (ephemeral)
│ /usr,/lib,/lib64,/etc/{ssl,resolv.conf,nsswitch} RO (host-curated)
│ /proc,/dev ← fresh
│ network ← shared with host
│ UID/GID ← l4d2-sandbox (no_new_privs implicit in bwrap)
▼ stdout/stderr → run_with_streamed_output → existing job-log SSE stream
▼ After exit:
│ exit 0 ∧ du -sb /overlay ≤ 20 GB → last_build_status='ok'
│ any other outcome → last_build_status='failed'
```
The host library (`l4d2host`) is unchanged. The `KernelOverlayFSMounter` already mounts whatever's at `overlays/{id}/` regardless of how it got there. The Job model and worker model are essentially unchanged — `script` is just another overlay type for the same `build_overlay` operation that today supports `workshop`.
```python
BUILDERS = {
"workshop": WorkshopBuilder(),
"script": ScriptBuilder(),
}
```
## Data Model
### `Overlay` (modified)
```text
id INTEGER PK AUTOINCREMENT
name VARCHAR(255) NOT NULL
path VARCHAR(255) NOT NULL -- str(id) for new rows
type VARCHAR(16) NOT NULL -- 'workshop' | 'script'
user_id INTEGER NULL REFERENCES users(id) -- NULL = system-wide
script TEXT NOT NULL DEFAULT '' -- new; meaningful for type='script'
last_build_status VARCHAR(16) NOT NULL DEFAULT '' -- new; '' | 'ok' | 'failed'
created_at, updated_at
UNIQUE INDEX on (name) WHERE user_id IS NULL
UNIQUE INDEX on (name, user_id) WHERE user_id IS NOT NULL
INDEX on (type, user_id)
```
### Tables removed
- `global_overlay_item_files`
- `global_overlay_items`
- `global_overlay_sources`
Drop order matters for the SQLite migration: drop `_item_files` first (FK to `_items`), then `_items` (FK to `_sources`), then `_sources` (FK to `overlays`).
### Unchanged
`WorkshopItem`, `overlay_workshop_items`, `Job` (including `Job.overlay_id` and nullable `Job.user_id`), `Server`, `Blueprint`, etc.
## Filesystem Layout
```text
${LEFT4ME_ROOT}/
overlays/
{overlay_id}/ # script writes here; mounted by host
left4dead2/... # whatever the script produces
workshop_cache/{steam_id}.vpk # workshop type only — unchanged
# removed:
# global_overlay_cache/ # was used by managed-globals types
```
Single tree per overlay. No per-overlay scratch cache (the chosen idempotency model is "script runs against existing dir," so any caching the user wants lives inside the overlay dir and is preserved between builds).
The sandbox bind-mounts `${LEFT4ME_ROOT}/overlays/{id}/` to `/overlay` (RW). Nothing else under `${LEFT4ME_ROOT}` is visible inside the sandbox.
## Sandbox
### Helper script
`deploy/files/usr/local/libexec/left4me/left4me-script-sandbox`, mode 0755, owned root:
```bash
#!/bin/bash
# args: <overlay_id> <script_path>
set -euo pipefail
[[ $# -eq 2 ]] || { echo "usage: $0 <overlay_id> <script>" >&2; exit 64; }
OVERLAY_ID=$1; SCRIPT=$2
[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]] || { echo "bad overlay id" >&2; exit 64; }
OVERLAY_DIR=/var/lib/left4me/overlays/$OVERLAY_ID
[[ -d $OVERLAY_DIR ]] || { echo "no overlay dir" >&2; exit 65; }
[[ -f $SCRIPT ]] || { echo "no script" >&2; exit 65; }
SBX_UID=$(id -u l4d2-sandbox); SBX_GID=$(id -g l4d2-sandbox)
exec systemd-run --quiet --scope --collect \
-p MemoryMax=4G -p MemorySwapMax=0 -p TasksMax=512 \
-p CPUQuota=200% -p RuntimeMaxSec=3600 \
-- bwrap \
--die-with-parent --new-session \
--unshare-pid --unshare-ipc --unshare-uts --unshare-cgroup \
--uid "$SBX_UID" --gid "$SBX_GID" \
--proc /proc --dev /dev --tmpfs /tmp --tmpfs /run \
--ro-bind /usr /usr --ro-bind /lib /lib --ro-bind /lib64 /lib64 \
--symlink usr/bin /bin --symlink usr/sbin /sbin \
--ro-bind /etc/resolv.conf /etc/resolv.conf \
--ro-bind /etc/ssl /etc/ssl \
--ro-bind /etc/ca-certificates /etc/ca-certificates \
--ro-bind /etc/nsswitch.conf /etc/nsswitch.conf \
--bind "$OVERLAY_DIR" /overlay \
--chdir /overlay \
--setenv HOME /tmp --setenv PATH /usr/bin:/usr/sbin \
--setenv OVERLAY /overlay \
--ro-bind "$SCRIPT" /script.sh \
/bin/bash /script.sh
```
Network is *not* unshared (no `--unshare-net`); the sandbox shares the host network namespace. Every transient unit is visible via `systemctl list-units --type=scope` while running and journaled afterward (`journalctl --user-unit=run-…scope` or system journal depending on invocation).
### Sudoers fragment
Append to `deploy/files/etc/sudoers.d/left4me`:
```
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-script-sandbox
```
### System user
Provisioned in `deploy/deploy-test-server.sh`:
```bash
useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox
apt-get install -y bubblewrap
```
## Build Lifecycle
`ScriptBuilder` lives in `l4d2web/services/overlay_builders.py` next to `WorkshopBuilder`:
```python
class ScriptBuilder:
def build(self, overlay, *, on_stdout, on_stderr, should_cancel):
with tempfile.NamedTemporaryFile("w", suffix=".sh", delete=False) as f:
f.write(overlay.script or "")
script_path = f.name
try:
cmd = [
"sudo", "-n",
"/usr/local/libexec/left4me/left4me-script-sandbox",
str(overlay.id), script_path,
]
run_with_streamed_output(cmd, on_stdout, on_stderr, should_cancel)
self._enforce_disk_budget(overlay.id, on_stderr)
finally:
os.unlink(script_path)
def _enforce_disk_budget(self, overlay_id, on_stderr):
size = subprocess.check_output(["du", "-sb", overlay_path(overlay_id)])
if int(size.split()[0]) > 20 * 1024**3:
on_stderr("overlay exceeded 20 GB disk cap")
raise BuildError("disk-cap-exceeded")
```
`run_with_streamed_output` is the existing helper used by `WorkshopBuilder` for `steamcmd`/`curl` invocations. The `should_cancel` callback fires `kill -TERM` on the sudo-`systemd-run` process tree; cgroup-collect tears down the whole scope on exit.
The job worker's existing job-completion path writes `Overlay.last_build_status = 'ok'` on success and `'failed'` on any non-zero exit / `BuildError` / cancel. This is a single column update inside the existing transaction; no new infrastructure.
## UI
### Create modal (`templates/overlays.html`)
The existing modal grows one option in the type radio: `Workshop | Script`. Name field unchanged. After insert, the web app generates `path = str(overlay_id)` for new rows (existing pattern).
### Detail page when `type='script'` (`templates/overlay_detail.html`)
- Plain styled `<textarea>` for `overlay.script` with a Save button → `POST /overlays/{id}/script`. No CodeMirror dependency in v1 (out of scope; keep frontend dep-light).
- "Rebuild" button → `POST /overlays/{id}/build`. Existing pattern from workshop overlays.
- "Wipe overlay" button (red, confirm-modal) → `POST /overlays/{id}/wipe`.
- `last_build_status` indicator badge: empty / "ok" / "failed".
- Live build log via existing SSE plumbing on the related Job row.
### Detail page when `type='workshop'`: unchanged.
### Sections removed
The global-source detail block (`overlay_detail.html` lines 3446) is deleted along with the managed-globals subsystem.
## Routes
`l4d2web/routes/overlay_routes.py` adds:
| Method | Path | Purpose |
|---|---|---|
| POST | `/overlays/{id}/script` | Update `script` text. Auto-enqueue coalesced `build_overlay` job. |
| POST | `/overlays/{id}/wipe` | Invoke `left4me-script-sandbox` with the literal script `find /overlay -mindepth 1 -delete`. Owner/admin only. Refuses if a `build_overlay` for this overlay is running. After success, set `last_build_status=''`. Does not auto-enqueue a rebuild. |
| POST | `/overlays/{id}/build` | Manual rebuild — same pattern as today's workshop overlay manual rebuild. |
Existing `POST /overlays` accepts `type=script` and an optional initial `script` body.
## Permissions
| Action | Who |
|---|---|
| Create script overlay (private, `user_id = me`) | Any authenticated user |
| Create script overlay (system-wide, `user_id = NULL`) | Admin |
| Edit (script body, name) | Owner or admin |
| Wipe / Rebuild | Owner or admin |
| Delete | Owner or admin |
| View | Owner, admin, or any user when `user_id IS NULL` |
These match the existing rules for workshop overlays.
## Job Worker / Scheduler
`services/job_worker.py` drops `"refresh_global_overlays"` from `GLOBAL_OPERATIONS` and removes the corresponding `refresh_global_overlays_running` and `blocked_servers_by_overlay` plumbing that exists only for the global-maps subsystem. The remaining mutex rules already cover:
- `build_overlay` per overlay (one running build per overlay).
- `install` and `refresh_workshop_items` as global mutexes.
- Server start/init blocks if any `build_overlay` for an overlay in the server's blueprint is running.
No new rules are needed for `script` — its build is mechanically identical to a `workshop` build from the scheduler's perspective.
## Daily Refresh — Removed
This iteration deletes the daily-refresh subsystem entirely:
- `deploy/files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer` and `.service` — deleted.
- `flask refresh-global-overlays` CLI command in `l4d2web/cli.py` — deleted.
- No replacement timer, no replacement CLI, no `auto_refresh` column on `Overlay`.
The only build trigger after this change is the user clicking Rebuild on the detail page (or the auto-enqueue when they Save the script body). A scheduled-refresh feature is reintroduced in a future iteration designed against concrete operational needs.
## Risks
- **Sandbox escape via kernel bug.** `bwrap` has a strong track record but is not invulnerable. Mitigated by running as `l4d2-sandbox` (no privileged capabilities), no setuid binaries reachable, `no_new_privs` implicit. A successful escape would land in an unprivileged UID with no host secrets reachable.
- **Disk fill via runaway script.** A script that writes a 20 GB+ payload to `/overlay` succeeds inside the sandbox and only fails afterward at the post-build `du` check. The 20 GB lands on disk transiently. Mitigated by the kernel's per-cgroup IO accounting being unaware of file size (no good IO-time limit), accepting this as a v1 trade-off; a future improvement is overlay-dir-on-its-own-filesystem with a quota.
- **Network exfiltration.** Script can connect to anything outbound, including internal IPs. Acceptable for the current trust model (semi-public; users have credentials). Egress firewall is out of scope.
- **Build-mid-server-running.** The scheduler refuses `build_overlay` for an overlay attached to a starting/running server (existing rule, unchanged). Good. A user can still rebuild while a server using a *different* blueprint runs concurrently.
- **Wipe race with running build.** The wipe endpoint refuses if a `build_overlay` for the overlay is running. Without this check, a wipe could blow away files mid-script and produce undefined results.
- **Stale `last_build_status`.** A row inserted via direct DB write or restored from backup could carry an `'ok'` status that no longer reflects reality. Treated as cosmetic; users can rebuild to refresh.
- **Sudoers misconfig.** A typo in the sudoers fragment could grant `left4me` more than intended. Mitigated by deploy-artifact tests asserting the exact expected lines.
- **DB row deletion racing the sandbox.** A user deleting an overlay while its build runs would invalidate the bind-mount target. Mitigated by the existing scheduler rule that tracks running overlays; delete should refuse if a build is running. (Existing pattern for workshop overlays; reuse.)
- **Migration drops globals tables.** Acceptable for the test deploy. Production rollout would need a different migration story; this spec explicitly assumes test-deploy DB wipe.
## Out Of Scope
- **Scheduled / daily refresh.** Intentionally removed in this iteration. Reintroduced later, designed against the use cases that emerge.
- **Per-overlay resource overrides.** All script overlays share the same 1 h / 4 GB / 20 GB envelope. If a real overlay needs more (l4d2center mirror at peak), revisit.
- **CodeMirror or other rich script editor.** Plain `<textarea>` in v1.
- **Egress allowlist / proxy.** No network restrictions on the sandbox in v1.
- **`$CACHE` scratch dir** persisted across builds. Users cache inside the overlay dir if they want; idempotency model is "script runs against existing dir."
- **Multi-tenant cgroup tree per user.** All sandboxes share the same cgroup-quota envelope.
- **Revision history on `script` column.** No `overlay_script_revisions` table; whatever's in the row is the current script.
- **Auto-seeding of l4d2center / cedapug equivalents.** Admin pastes the script post-deploy.
- **Migration that preserves existing global-map overlay rows.** Test deploy DB is wiped.
- **Container-per-build (podman / docker).** Heavier than `bwrap`; revisit only if multi-tenant escalates to "fully public sign-up."
- **left4me-aware helpers** (`workshop`, `download`, `extract`) inside the sandbox. Pure bash + host `/usr` only.
## Implementation Boundaries
- **`l4d2host` is unchanged.** The host library has no concept of overlay types and the mount layer (`KernelOverlayFSMounter`) doesn't care how the overlay dir got populated.
- **The `OverlayBuilder` Protocol is unchanged** — same `build(overlay, *, on_stdout, on_stderr, should_cancel)` signature. `ScriptBuilder` plugs into the existing registry.
- **The job worker model is unchanged.** Same operations, same logs, same SSE plumbing, same scheduler rules (minus the refresh_global_overlays entry).
- **No new application-level dependencies.** Vendored HTMX, no new Python packages. Two new system dependencies: `bubblewrap` apt package and the `l4d2-sandbox` system user.
- **No new config keys.** Same env files (`/etc/left4me/host.env`, `/etc/left4me/web.env`).
- **DB migration is destructive for global-maps overlay rows.** This is acceptable per the test-deploy assumption; a production-rollout follow-up would need to address it.
- The companion implementation plan governs task ordering and verification commands. Implementation must not start without explicit user approval per that plan's gate.

View file

@ -0,0 +1,138 @@
# L4D2 Script Sandbox v2 — Systemd-Only
**Goal:** Replace the bwrap-based `left4me-script-sandbox` helper with one that uses `systemd-run` in **service-unit mode** alone. Drop `bubblewrap` as a system dependency. Gain capability bounding, seccomp filtering, kernel-tunable / -module / -log protection, address-family restriction, `LockPersonality`, `MemoryDenyWriteExecute`, and `RestrictSUIDSGID` — none of which the bwrap+systemd-run-scope composition could provide. Lose PID-namespace isolation (no `PrivatePID=` directive in systemd) — judged acceptable for the current trust model.
**Approval status:** User-approved 2026-05-08 after smoke testing on `ckn@10.0.4.128`.
## Context
The v1 sandbox (see `2026-05-08-l4d2-script-overlays-design.md`) layers `bubblewrap` for namespacing inside `systemd-run --scope` for cgroup limits. That works, but `--scope` units register an existing process tree and so cannot accept service-only directives like `NoNewPrivileges=`, `ProtectSystem=`, `SystemCallFilter=`, `CapabilityBoundingSet=`, etc. Smoke testing on the deployed host confirmed bwrap covers mount/PID/IPC/UTS namespacing well, but leaves capability bounding, seccomp, and kernel-surface protection unenforced.
A switch to `systemd-run` in default (transient service) mode unlocks the full hardening surface. Smoke testing of a v2 prototype against the deployed test host confirmed:
- Every isolation invariant the bwrap version provides (filesystem masking, UID drop, network reachability, `/overlay` RW bind, host-side `l4d2-sandbox` ownership, host secret hiding) is reproducible with systemd directives.
- All cgroup limits (`memory.max=4G`, `memory.swap.max=0`, `pids.max=512`, `cpu.max=200%`, `RuntimeMaxSec=3600`) apply identically.
- `MemoryError` fires at the 4 GB cap (cgroup-enforced).
- The wipe path (`find /overlay -mindepth 1 -delete`) succeeds.
- Hardening directives the v1 design couldn't express enforce real syscall blocks: `unshare(CLONE_NEWUSER)`, `mount(2)`, `personality(2)`, `bpf(2)`, `swapoff(2)`, `sysctl -w` are all blocked.
The single behavioral regression: host process IDs are visible via `/proc` and `ps -ef` because systemd has no `PrivatePID=` directive. Sending signals to those processes is still blocked by the kernel's UID-mismatch check (`l4d2-sandbox` cannot signal `root`-owned processes). Information disclosure is the only leak; signal capability is intact.
## Locked Decisions
1. **Replace the helper body wholesale.** No `bwrap` invocation. `systemd-run` in service mode does both isolation and resource limits.
2. **Helper path, sudoers rule, ScriptBuilder API, and `l4d2-sandbox` UID are unchanged.** The Python side (`run_sandboxed_script`, route handlers, tests) does not change.
3. **`bubblewrap` apt dependency dropped from `deploy-test-server.sh`.**
4. **`left4me.db` file mode tightened to 0640 root:left4me at deploy time.** This is a host-hygiene fix that is independent of the sandbox change but was surfaced by smoke testing — without it, *any* host user (and, transitively, the sandbox) could read the application database.
5. **`TemporaryFileSystem=/var/lib` is required.** `ProtectSystem=strict` makes `/var/lib/left4me` read-only but visible; the only way to reliably hide its contents from the unit is to mask the parent with a tmpfs. The `BindPaths=…/overlays/{id}:/overlay` mount is unaffected because `/overlay` is at a different path.
6. **`PrivatePID=` is not configured.** systemd has no such directive. `ps -ef` from inside the sandbox shows host processes. The kernel's UID-based signal restriction blocks any actual interaction with them. Acceptable for the current trust model.
7. **Walltime kill remains `RuntimeMaxSec=3600`.** Same as v1.
8. **Network namespace remains shared with the host.** No `PrivateNetwork=`. Scripts must reach Steam / l4d2center / GitHub / etc.
9. **`SystemCallFilter=@system-service @network-io`** is the seccomp baseline. systemd's curated `@system-service` group is "everything a normal service does"; adding `@network-io` is explicit even though it overlaps. Build failures revealing missing syscall classes are surfaced via `journalctl` and addressed by widening the filter (`@process`, etc.) on demand.
10. **Single helper file replaces v1.** Not adding a `-v2` variant. The v1 implementation is removed in the same change.
## Architecture
```text
sudo helper
└─ systemd-run --service (default) --pipe --wait
(transient .service unit, full hardening directives)
└─ /bin/bash /script.sh
```
systemd-run in service mode:
- Opens a transient service unit on the system bus.
- Applies all `-p` properties as the unit's exec context.
- Forks; the child sets up the unit's namespaces (mount, IPC, user), drops privileges to `User=l4d2-sandbox`, applies the seccomp filter, and `execve()`s `/bin/bash /script.sh`.
- `--pipe` connects the unit's stdin/stdout/stderr to the calling helper's stdio (so the existing `run_command` harness in `ScriptBuilder` continues to capture line-by-line).
- `--wait` blocks until the unit terminates and propagates the exit code.
- `--collect` removes the unit on exit even if it failed.
- The cgroup carries the resource limits; the systemd timer enforces `RuntimeMaxSec=3600`.
### Helper
`deploy/files/usr/local/libexec/left4me/left4me-script-sandbox`, mode 0755, owned root:
```bash
#!/bin/bash
set -euo pipefail
[[ $# -eq 2 ]] || { echo "usage: $0 <overlay_id> <script>" >&2; exit 64; }
OVERLAY_ID=$1; SCRIPT=$2
[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]] || { echo "bad overlay id" >&2; exit 64; }
OVERLAY_DIR=/var/lib/left4me/overlays/$OVERLAY_ID
[[ -d $OVERLAY_DIR ]] || { echo "no overlay dir at $OVERLAY_DIR" >&2; exit 65; }
[[ -f $SCRIPT ]] || { echo "no script at $SCRIPT" >&2; exit 65; }
if [[ "${LEFT4ME_SCRIPT_SANDBOX_DRY_RUN:-}" == "1" ]]; then
echo "DRY RUN: overlay_id=$OVERLAY_ID script=$SCRIPT overlay_dir=$OVERLAY_DIR"
exit 0
fi
chown -R l4d2-sandbox:l4d2-sandbox "$OVERLAY_DIR"
chmod 0755 "$OVERLAY_DIR"
exec systemd-run --quiet --collect --wait --pipe \
--unit="left4me-script-${OVERLAY_ID}-$$" \
-p User=l4d2-sandbox -p Group=l4d2-sandbox \
-p NoNewPrivileges=yes \
-p ProtectSystem=strict -p ProtectHome=yes \
-p PrivateTmp=yes -p PrivateDevices=yes -p PrivateIPC=yes \
-p ProtectKernelTunables=yes -p ProtectKernelModules=yes \
-p ProtectKernelLogs=yes -p ProtectControlGroups=yes \
-p RestrictNamespaces=yes \
-p RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX" \
-p RestrictSUIDSGID=yes -p LockPersonality=yes \
-p MemoryDenyWriteExecute=yes \
-p SystemCallFilter="@system-service @network-io" \
-p SystemCallArchitectures=native \
-p CapabilityBoundingSet= -p AmbientCapabilities= \
-p TemporaryFileSystem="/etc /var/lib" \
-p BindReadOnlyPaths="/etc/resolv.conf /etc/ssl /etc/ca-certificates /etc/nsswitch.conf /etc/alternatives ${SCRIPT}:/script.sh" \
-p BindPaths="${OVERLAY_DIR}:/overlay" \
-p WorkingDirectory=/overlay \
-p Environment="HOME=/tmp PATH=/usr/bin:/usr/sbin OVERLAY=/overlay" \
-p MemoryMax=4G -p MemorySwapMax=0 -p TasksMax=512 \
-p CPUQuota=200% -p RuntimeMaxSec=3600 \
-- /bin/bash /script.sh
```
### Sudoers fragment
Unchanged from v1: `left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-script-sandbox`.
### System user
Unchanged from v1: `l4d2-sandbox` (`useradd --system --no-create-home --shell /usr/sbin/nologin`).
### Filesystem expectations
- `/var/lib/left4me` must be mode 0711 (left4me-owned). Already provisioned by v1 deploy script.
- `/var/lib/left4me/left4me.db` mode 0640 root:left4me. **New** — added by this change.
- Overlay directory `/var/lib/left4me/overlays/{id}/` chowned to `l4d2-sandbox:l4d2-sandbox` 0755 by the helper before each run. Unchanged from v1.
## Build Lifecycle (unchanged from v1)
`ScriptBuilder.build()` writes the script to a 0644 tmpfile, exec's `sudo -n /usr/local/libexec/left4me/left4me-script-sandbox <id> <tmpfile>` via `run_command`, then runs `_enforce_disk_budget`. The helper's internal mechanism changes; the wrapper API is identical. `Overlay.last_build_status` is written by the job worker on completion.
## Risks
- **systemd CVE landing in our directive set.** Single-tool migration removes one isolation layer. Mitigated by uid drop + cgroup limits + `NoNewPrivileges=yes` (kernel-enforced state independent of namespace setup). The escape would be an unprivileged process with no filesystem isolation but still capped on resources; same severity envelope as a hypothetical bwrap CVE in v1. The trust model (registered users) makes a single isolation layer acceptable.
- **`SystemCallFilter` rejecting a syscall a user script unexpectedly needs.** Symptom: build fails with SIGSYS. Diagnosis: `journalctl --since "1 min ago" | grep SECCOMP`. Resolution: widen the filter (`+@process`, `+@privileged` if the script genuinely needs more than a normal service). v1 had no syscall filter, so this is a new failure class.
- **`ProtectSystem=strict` masking something a script wanted to write to.** Only `/overlay`, `/tmp`, `/run` are writable inside the sandbox. Same as v1.
- **Host PID visibility (no `PrivatePID=`).** Information disclosure; not a privilege boundary.
- **`MemoryDenyWriteExecute=yes` blocking JITs.** A script that launches `node` / a JIT runtime would fail because W+X mappings are blocked. None of the recipe set the user has historically used (curl + tar + cp) needs a JIT; revisit if a real script trips this.
- **`RestrictAddressFamilies` blocking some download tools.** `curl`, `wget`, `git over https` use `AF_INET`/`AF_INET6`; `getent hosts` uses `AF_UNIX` (nss). Smoke-tested as working. A script that wanted raw sockets (`AF_PACKET`) or netlink (`AF_NETLINK`) would fail; neither is plausible for build recipes.
## Out Of Scope
- **Per-overlay UID isolation.** Cross-script-overlay write access is still possible after a hypothetical sandbox bypass (every script overlay's dir is owned by `l4d2-sandbox`). A per-overlay UID pool was discussed as the next-step hardening but is deferred.
- **`PrivateNetwork=` / egress filtering.** No change from v1.
- **systemd-nspawn or LXC.** Researched; both are heavier than necessary for transient bash builds.
- **`PrivatePID=` workaround via `unshare`.** Not pursued — would require re-introducing a wrapper inside the unit, defeating the simplification.
## Implementation Boundaries
- **Web app code is unchanged.** `ScriptBuilder`, `run_sandboxed_script`, route handlers, models, migrations — all untouched. The migration is purely in the deployed helper script and adjacent deploy artifacts.
- **`bubblewrap` apt package removed.** Already absent from production paths after this change; deploy script updated.
- **No new systemd unit files.** Each invocation is a transient unit named `left4me-script-{overlay_id}-{pid}.service`.
- **No application-level dependency changes.** No new Python packages, no template changes, no DB migration.

View file

@ -0,0 +1,113 @@
# L4D2 Script Sandbox v3 — Egress Filter (Public Internet Only)
**Goal:** Restrict the script-overlay sandbox to public-internet egress only. Block reachability to the host's own services (localhost), the LAN, and any private RFC1918 / link-local / multicast / CGNAT / ULA addresses. Public DNS is preserved by bind-mounting a sandbox-only `resolv.conf` pointing at Cloudflare + Google.
**Approval status:** User-approved 2026-05-08. Implemented and smoke-tested on `ckn@10.0.4.128`.
## Context
After the v2 (systemd-only) migration, the sandbox still shared the host's network namespace. A live probe demonstrated the script could:
- Reach the web app on `127.0.0.1:8000` (HTTP 200 from `/health`).
- Reach the host's SSH daemon on `127.0.0.1:22` (banner returned).
- Reach the host on the LAN at `10.0.4.128:22` (banner returned).
- Reach the LAN gateway / DNS server at `10.0.0.1`.
- See Unix sockets in `/run` (`AF_UNIX` allowed).
The threat model says the sandbox should reach the public internet to download Workshop / l4d2center / GitHub content, but should **not** be able to talk to the host or LAN. systemd's `IPAddressDeny=` BPF cgroup egress filter is the right tool. It attaches a BPF program (`sd_fw_egress`) to the unit's cgroup; matching packets are silently dropped at send time.
A complication: the host's `/etc/resolv.conf` typically points at a private-IP DNS server (10.0.0.1 in the test deploy). Naively blocking `10.0.0.0/8` kills DNS, which kills outbound HTTP. The fix is to give the sandbox a static `resolv.conf` with public resolvers; DNS traffic then targets allowed public IPs.
## Locked Decisions
1. **`IPAddressDeny=` alone — no `IPAddressAllow=any`.** The systemd documentation claims "more specific rule wins" when both are set, but on systemd 257 + kernel 6.12 (and likely other combos), `IPAddressAllow=any` silently overrides every `IPAddressDeny=` rule. Verified empirically. With only `IPAddressDeny=` set, the kernel's default "allow all" applies to non-listed addresses; the listed CIDRs are dropped at the egress hook. **This must not be regressed** — adding back `IPAddressAllow=any` reopens every blocked range.
2. **Explicit CIDRs, no shorthand keywords.** systemd's unit-file parser accepts `localhost`, `link-local`, `multicast` shortcuts, but the `systemd-run -p` parser rejects them with `Failed to parse IP address prefix: localhost`. Use the CIDRs directly: `127.0.0.0/8 ::1/128 169.254.0.0/16 fe80::/10 224.0.0.0/4 ff00::/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 fc00::/7`.
3. **Static `/etc/left4me/sandbox-resolv.conf` with public resolvers** (Cloudflare 1.1.1.1, Google 8.8.8.8). Bind-mounted into the sandbox at `/etc/resolv.conf` via `BindReadOnlyPaths=/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf`. Two nameservers for redundancy. Picking other public resolvers (Quad9, OpenDNS) would also be acceptable; the file is the source of truth, not the helper.
4. **`AF_UNIX` stays in `RestrictAddressFamilies=`.** Dropping it would risk breaking NSS / syslog / D-Bus introspection paths for marginal gain — the IP-level filter handles the actual threat (reaching host TCP services). The Unix-socket surface (D-Bus system bus, systemd notify) is uid-gated and `l4d2-sandbox` has no special D-Bus permissions.
5. **No `PrivateNetwork=`.** That would block all networking, including the public internet. The whole point of script overlays is reaching public download sources.
6. **No DNS-over-HTTPS or DNSSEC.** Plain UDP-53 to public resolvers is sufficient; the threat is "egress targeting", not "DNS hijacking". Revisit if the trust model relaxes.
## Architecture
```text
sudo helper (root)
└─ chown overlay dir to l4d2-sandbox
└─ systemd-run --service [...all v2 directives...]
-p IPAddressDeny="<11 CIDRs>"
-p BindReadOnlyPaths="/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf [...]"
└─ /bin/bash /script.sh
(egress to listed CIDRs dropped at sd_fw_egress BPF hook;
DNS goes to 1.1.1.1 / 8.8.8.8; everything else
reaches the public internet normally)
```
`IPAddressDeny=` blocks egress to:
| CIDR | Coverage |
|---|---|
| `127.0.0.0/8` | IPv4 loopback |
| `::1/128` | IPv6 loopback |
| `169.254.0.0/16` | IPv4 link-local (incl. AWS metadata, DHCP fallback) |
| `fe80::/10` | IPv6 link-local |
| `224.0.0.0/4` | IPv4 multicast |
| `ff00::/8` | IPv6 multicast |
| `10.0.0.0/8` | RFC1918 private |
| `172.16.0.0/12` | RFC1918 private |
| `192.168.0.0/16` | RFC1918 private |
| `100.64.0.0/10` | CGNAT (RFC6598) |
| `fc00::/7` | IPv6 ULA |
Public IPv4 / IPv6 destinations are unaffected.
## Files
- `deploy/files/etc/left4me/sandbox-resolv.conf` *(new)*`nameserver 1.1.1.1` + `nameserver 8.8.8.8`. Mode 0644 root-owned at deploy time.
- `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox``IPAddressDeny=` directive added; `BindReadOnlyPaths=` references the sandbox-resolv.conf instead of `/etc/resolv.conf`.
- `deploy/deploy-test-server.sh``install -m 0644 -o root -g root .../sandbox-resolv.conf /etc/left4me/sandbox-resolv.conf`.
- `deploy/tests/test_deploy_artifacts.py` — assert all of the above + the **negative assertion `IPAddressAllow=any not in text`** (regression guard).
The web app, ScriptBuilder, routes, models, and migrations are all unchanged. Same as v2.
## Verification
Smoke battery on the deployed host (probe script invoked through the helper as root):
| Target | Expected | Actual |
|---|---|---|
| `1.1.1.1:443` | connected | ✓ CONNECTED |
| `https://steamcommunity.com/` (DNS + HTTPS) | 200 | ✓ 200 |
| `127.0.0.1:8000` (web app) | blocked | ✓ TimeoutError |
| `127.0.0.1:22` (sshd) | blocked | ✓ TimeoutError |
| `10.0.4.128:22` (host LAN ssh) | blocked | ✓ TimeoutError |
| `10.0.0.1:53` (host's DNS resolver) | blocked | ✓ TimeoutError |
| `cat /etc/resolv.conf` inside | shows 1.1.1.1 + 8.8.8.8 | ✓ |
`bpftool cgroup show` against the unit's cgroup confirms `sd_fw_egress` and `sd_fw_ingress` are attached.
## Risks
- **`IPAddressAllow=` accidentally added back.** Reopens every blocked range silently. Mitigation: explicit negative test in `test_deploy_artifacts.py` plus a comment in the helper.
- **Public DNS resolver outage.** 1.1.1.1 and 8.8.8.8 are both down → DNS in sandbox fails → builds fail. Two resolvers from independent operators makes this very unlikely. Operator can change the file in `/etc/left4me/sandbox-resolv.conf` if they prefer different resolvers; the helper picks it up on next invocation.
- **Public DNS resolver privacy.** Cloudflare and Google see hostnames the scripts query. Acceptable for the workload (Steam Workshop, GitHub, etc. are public anyway); switch to Quad9 or self-hosted if this is a concern.
- **Future kernel/systemd that flips the documented "more specific wins" semantics.** If a future systemd version actually implements the documented behavior, a unit with only `IPAddressDeny=` continues to work; the negative test on `IPAddressAllow=any` keeps the regression-safe configuration locked in. Re-test on each major systemd upgrade.
- **Scripts that legitimately need a private IP.** E.g., a self-hosted internal mirror at 10.x. Not a use case today; if it arises, expose specific IPs via a future `IPAddressAllow=10.x.y.z/32` for that one host (not blanket).
## Out Of Scope
- **Per-overlay UID isolation.** Cross-script-overlay write access via the shared `l4d2-sandbox` UID is still possible after a hypothetical sandbox bypass. Deferred from earlier discussions.
- **Egress allowlist by hostname / domain.** Would require a forward proxy (Squid, mitmproxy). Heavier than warranted for the trust model.
- **Dropping `AF_UNIX` from `RestrictAddressFamilies=`.** Tangential to IP-level egress; risks breaking NSS / syslog.
- **DNSSEC / DoH.** Threat model is egress targeting, not DNS hijacking.
- **Network-namespace isolation (`PrivateNetwork=` + custom netns + NAT).** Heavier than `IPAddressDeny=` for equivalent outcome.
## Implementation Boundaries
- **No app code change.** Helper-side only.
- **No new systemd units.** Same transient `left4me-script-{id}-{pid}.service` pattern.
- **No new apt deps.** `bpftool` was used during smoke testing but is not required at runtime.
- **One new deploy artifact.** `sandbox-resolv.conf` shipped under `deploy/files/etc/left4me/`.

View file

@ -0,0 +1,109 @@
#!/bin/bash
# Cedapug custom-maps overlay.
# Scrapes the JS map list at https://cedapug.com/custom and keeps
# $OVERLAY/left4dead2/addons/ in sync. Re-downloads only when the zip's
# Last-Modified header changed.
set -euo pipefail
CEDAPUG_HOST="https://cedapug.com"
LIST_URL="$CEDAPUG_HOST/custom"
ADDONS_DIR="$OVERLAY/left4dead2/addons"
STATE_DIR="$OVERLAY/.cedapug"
WORK_DIR="$STATE_DIR/work"
MANIFEST="$STATE_DIR/manifest.tsv" # zip_basename \t last_modified \t vpk1,vpk2,...
mkdir -p "$ADDONS_DIR" "$STATE_DIR" "$WORK_DIR"
# --- 1. Discover the current zip list -----------------------------------
# The page embeds the map list inside renderCustomMapDownloads([...]) with
# JSON-escaped slashes (\/maps\/foo.zip). Unescape with sed before grepping.
echo ":: fetching $LIST_URL"
mapfile -t ZIP_PATHS < <(
curl -fsSL --retry 3 --retry-delay 2 "$LIST_URL" \
| sed 's@\\/@/@g' \
| grep -oE '"/maps/[A-Za-z0-9_.+-]+\.zip"' \
| tr -d '"' \
| sort -u
)
if [[ ${#ZIP_PATHS[@]} -eq 0 ]]; then
echo "!! no map zips found on $LIST_URL — page format may have changed" >&2
exit 1
fi
echo ":: ${#ZIP_PATHS[@]} custom map zip(s) listed"
# --- 2. Load previous manifest ------------------------------------------
declare -A PREV_LM PREV_VPKS
if [[ -f "$MANIFEST" ]]; then
while IFS=$'\t' read -r zname lm vpks; do
[[ -n "$zname" ]] || continue
PREV_LM[$zname]=$lm
PREV_VPKS[$zname]=$vpks
done < "$MANIFEST"
fi
# --- 3. Download / extract each desired zip -----------------------------
declare -A NEW_LM NEW_VPKS
for zpath in "${ZIP_PATHS[@]}"; do
zname="${zpath##*/}"
url="$CEDAPUG_HOST$zpath"
lm="$(curl -fsSI --retry 3 --retry-delay 2 "$url" \
| awk 'BEGIN{IGNORECASE=1} /^last-modified:/ {sub(/^[^:]*:[[:space:]]*/, ""); gsub(/\r/, ""); print; exit}')"
NEW_LM[$zname]=$lm
prev_lm="${PREV_LM[$zname]:-}"
prev_vpks="${PREV_VPKS[$zname]:-}"
# Skip if Last-Modified matches AND every recorded vpk is still on disk.
if [[ -n "$lm" && "$lm" == "$prev_lm" && -n "$prev_vpks" ]]; then
all_present=1
IFS=',' read -r -a v_arr <<<"$prev_vpks"
for v in "${v_arr[@]}"; do
[[ -f "$ADDONS_DIR/$v" ]] || { all_present=0; break; }
done
if (( all_present )); then
echo ":: $zname unchanged (Last-Modified: $lm) — skipped"
NEW_VPKS[$zname]=$prev_vpks
continue
fi
fi
echo ":: downloading $zname"
zfile="$WORK_DIR/$zname"
curl -fSL --retry 3 --retry-delay 2 -o "$zfile" "$url"
7z x -y -bd -o"$ADDONS_DIR" "$zfile" '*.vpk' >/dev/null
vpk_list="$(7z l -slt -ba "$zfile" \
| awk '/^Path = .*\.vpk$/ {sub(/^Path = /, ""); print}' \
| paste -sd, -)"
if [[ -z "$vpk_list" ]]; then
echo "!! warning: $zname contained no .vpk — leaving manifest entry empty" >&2
fi
NEW_VPKS[$zname]=$vpk_list
rm -f "$zfile"
done
# --- 4. Prune vpks from zips that disappeared from the listing ----------
for old_zname in "${!PREV_VPKS[@]}"; do
if [[ -z "${NEW_LM[$old_zname]:-}" ]]; then
IFS=',' read -r -a old_v_arr <<<"${PREV_VPKS[$old_zname]}"
for v in "${old_v_arr[@]}"; do
if [[ -n "$v" && -f "$ADDONS_DIR/$v" ]]; then
echo ":: pruning $v (zip $old_zname no longer listed)"
rm -f "$ADDONS_DIR/$v"
fi
done
fi
done
# --- 5. Rewrite manifest -------------------------------------------------
tmp_manifest="$(mktemp -p "$STATE_DIR")"
for zname in "${!NEW_LM[@]}"; do
printf '%s\t%s\t%s\n' "$zname" "${NEW_LM[$zname]}" "${NEW_VPKS[$zname]:-}" >>"$tmp_manifest"
done
mv -f "$tmp_manifest" "$MANIFEST"
rm -rf "$WORK_DIR"
echo ":: done — ${#NEW_LM[@]} zip(s) tracked, addons/ in sync"

View file

@ -0,0 +1,15 @@
#!/bin/bash
# L4D2 Competitive Rework overlay.
# Pulls the master branch of SirPlease/L4D2-Competitive-Rework as a tarball
# and extracts it into $OVERLAY/left4dead2/. Skipped on subsequent rebuilds
# once cfg/cfgogl/ is present — wipe the overlay to force a refresh.
# https://github.com/SirPlease/L4D2-Competitive-Rework
set -xeuo pipefail
DEST="$OVERLAY/left4dead2"
mkdir -p "$DEST"
if [[ ! -d "$DEST/cfg/cfgogl" ]]; then
curl -fsSL https://github.com/SirPlease/L4D2-Competitive-Rework/archive/refs/heads/master.tar.gz \
| tar -xz --strip-components=1 -C "$DEST"
fi

View file

@ -0,0 +1,48 @@
#!/bin/bash
set -xeuo pipefail
mkdir -p /overlay/left4dead2/addons
cd /overlay/left4dead2/addons
for cmd in curl md5sum 7z; do
command -v "$cmd" >/dev/null || { echo "Missing: $cmd" >&2; exit 1; }
done
CSV_URL="https://l4d2center.com/maps/servers/index.csv"
TEMP_CSV=$(mktemp)
trap 'rm -f "$TEMP_CSV"' EXIT
curl -fsSL -o "$TEMP_CSV" "$CSV_URL"
declare -A map_md5 map_links
{
IFS= read -r _header
while IFS=';' read -r Name _Size MD5 DownloadLink || [[ $Name ]]; do
Name=$(echo "$Name" | xargs)
MD5=$(echo "$MD5" | xargs)
DownloadLink=$(echo "$DownloadLink" | xargs)
map_md5["$Name"]="$MD5"
map_links["$Name"]="$DownloadLink"
done
} < "$TEMP_CSV"
shopt -s nullglob
for file in *.vpk; do
if [[ -z "${map_md5[$file]:-}" ]]; then
rm -f "$file"
elif [[ "$(md5sum "$file" | awk '{print $1}')" != "${map_md5[$file]}" ]]; then
rm -f "$file"
fi
done
for vpk in "${!map_md5[@]}"; do
[[ -f "$vpk" ]] && continue
url="${map_links[$vpk]}"
[[ -n "$url" ]] || continue
encoded=$(printf '%s' "$url" | sed 's/ /%20/g')
TEMP_7Z=$(mktemp --suffix=.7z)
curl -fsSL -o "$TEMP_7Z" "$encoded"
7z x -y "$TEMP_7Z"
rm -f "$TEMP_7Z"
done
echo "Synchronization complete."

View file

@ -0,0 +1,25 @@
#!/bin/bash
# Tickrate overlay: server.cfg with high-rate settings + the tickrate_enabler
# addon (DLL/SO/VDF) from SirPlease/L4D2-Competitive-Rework.
# https://github.com/SirPlease/L4D2-Competitive-Rework/blob/7ecc3a32a5e2180d6607a40119ff2f3c072502a9/cfg/server.cfg#L58-L69
# https://www.programmersought.com/article/513810199514/
set -xeuo pipefail
CFG_DIR="$OVERLAY/left4dead2/cfg"
ADDONS_DIR="$OVERLAY/left4dead2/addons"
mkdir -p "$CFG_DIR" "$ADDONS_DIR"
cat <<'EOF' > "$CFG_DIR/server.cfg"
# https://github.com/SirPlease/L4D2-Competitive-Rework/blob/7ecc3a32a5e2180d6607a40119ff2f3c072502a9/cfg/server.cfg#L58-L69
sv_minrate 100000
sv_maxrate 100000
nb_update_frequency 0.014
net_splitpacket_maxrate 50000
net_maxcleartime 0.0001
fps_max 0
EOF
for file in tickrate_enabler.dll tickrate_enabler.so tickrate_enabler.vdf; do
curl -fsSL "https://github.com/SirPlease/L4D2-Competitive-Rework/raw/refs/heads/master/addons/${file}" \
-o "$ADDONS_DIR/${file}"
done

View file

@ -49,7 +49,7 @@ Validated on Debian 13 during the `ckn@10.0.4.128` smoke test:
- Python 3.12+ with virtualenv/pip tooling for installing `l4d2host`. - Python 3.12+ with virtualenv/pip tooling for installing `l4d2host`.
- `steamcmd` available on `PATH` and able to self-update as the runtime user. - `steamcmd` available on `PATH` and able to self-update as the runtime user.
- 32-bit compatibility libraries for SteamCMD on amd64 Debian: `libc6-i386`, `lib32gcc-s1`, `lib32stdc++6`. - 32-bit compatibility libraries for SteamCMD on amd64 Debian: `libc6-i386`, `lib32gcc-s1`, `lib32stdc++6`.
- `fuse-overlayfs` and `fusermount3` for per-instance overlay mounts. - Kernel overlayfs (`mount -t overlay`); mount/umount go through the `left4me-overlay` privileged helper, which `nsenter`s into PID 1's mount namespace.
- `systemctl --user` and `journalctl --user` available for the runtime user. - `systemctl --user` and `journalctl --user` available for the runtime user.
- User lingering enabled when services must survive SSH sessions: `sudo loginctl enable-linger <user>`. - User lingering enabled when services must survive SSH sessions: `sudo loginctl enable-linger <user>`.
- `/var/lib/left4me` created and writable by the runtime user, unless `LEFT4ME_ROOT` is set to another deployment-managed root. - `/var/lib/left4me` created and writable by the runtime user, unless `LEFT4ME_ROOT` is set to another deployment-managed root.
@ -61,7 +61,7 @@ sudo apt-get update
sudo apt-get install -y \ sudo apt-get install -y \
python3 python3-venv python3-pip \ python3 python3-venv python3-pip \
curl ca-certificates tar gzip \ curl ca-certificates tar gzip \
fuse-overlayfs fuse3 \ util-linux \
libc6-i386 lib32gcc-s1 lib32stdc++6 libc6-i386 lib32gcc-s1 lib32stdc++6
sudo mkdir -p /opt/steamcmd /var/lib/left4me/{installation,overlays,instances,runtime,tmp} sudo mkdir -p /opt/steamcmd /var/lib/left4me/{installation,overlays,instances,runtime,tmp}

View file

@ -4,7 +4,7 @@ import subprocess
import typer import typer
from l4d2host.instances import delete_instance, initialize_instance, start_instance, stop_instance from l4d2host.instances import delete_instance, initialize_instance, reset_instance, start_instance, stop_instance
from l4d2host.logs import stream_instance_logs from l4d2host.logs import stream_instance_logs
from l4d2host.status import get_instance_status from l4d2host.status import get_instance_status
from l4d2host.steam_install import SteamInstaller from l4d2host.steam_install import SteamInstaller
@ -59,6 +59,14 @@ def delete(name: str) -> None:
_exit_from_subprocess_error(exc) _exit_from_subprocess_error(exc)
@app.command()
def reset(name: str) -> None:
try:
reset_instance(name, passthrough=True)
except subprocess.CalledProcessError as exc:
_exit_from_subprocess_error(exc)
@app.command() @app.command()
def status(name: str, json_output: bool = typer.Option(False, "--json")) -> None: def status(name: str, json_output: bool = typer.Option(False, "--json")) -> None:
instance_status = get_instance_status(name) instance_status = get_instance_status(name)

View file

@ -5,7 +5,15 @@ from l4d2host.fs.base import OverlayMounter
from l4d2host.process import run_command from l4d2host.process import run_command
class FuseOverlayFSMounter(OverlayMounter): HELPER_PATH = "/usr/local/libexec/left4me/left4me-overlay"
class KernelOverlayFSMounter(OverlayMounter):
# Delegates the actual mount/umount syscalls to the privileged
# left4me-overlay helper. The helper takes only the instance name and
# rederives lowerdirs/upper/work/merged from disk; the OverlayMounter
# ABC accepts those args for compatibility, so we extract the name
# from the merged path's parent directory.
def mount( def mount(
self, self,
*, *,
@ -18,13 +26,9 @@ class FuseOverlayFSMounter(OverlayMounter):
passthrough: bool = False, passthrough: bool = False,
should_cancel: Callable[[], bool] | None = None, should_cancel: Callable[[], bool] | None = None,
) -> None: ) -> None:
del lowerdirs, upperdir, workdir
run_command( run_command(
[ ["sudo", "-n", HELPER_PATH, "mount", merged.parent.name],
"fuse-overlayfs",
"-o",
f"lowerdir={lowerdirs},upperdir={upperdir},workdir={workdir}",
str(merged),
],
on_stdout=on_stdout, on_stdout=on_stdout,
on_stderr=on_stderr, on_stderr=on_stderr,
passthrough=passthrough, passthrough=passthrough,
@ -41,7 +45,7 @@ class FuseOverlayFSMounter(OverlayMounter):
should_cancel: Callable[[], bool] | None = None, should_cancel: Callable[[], bool] | None = None,
) -> None: ) -> None:
run_command( run_command(
["fusermount3", "-u", str(merged)], ["sudo", "-n", HELPER_PATH, "umount", merged.parent.name],
on_stdout=on_stdout, on_stdout=on_stdout,
on_stderr=on_stderr, on_stderr=on_stderr,
passthrough=passthrough, passthrough=passthrough,

View file

@ -1,10 +1,11 @@
import os
from pathlib import Path from pathlib import Path
import shutil import shutil
import subprocess import subprocess
from typing import Callable from typing import Callable
from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
from l4d2host.paths import DEFAULT_LEFT4ME_ROOT, get_left4me_root, overlay_path, validate_instance_name from l4d2host.paths import DEFAULT_LEFT4ME_ROOT, get_left4me_root, overlay_path, validate_instance_name
from l4d2host.process import run_command
from l4d2host.service_control import start_service, stop_service from l4d2host.service_control import start_service, stop_service
from l4d2host.spec import load_spec from l4d2host.spec import load_spec
@ -12,6 +13,9 @@ from l4d2host.spec import load_spec
from l4d2host.logging import emit_step from l4d2host.logging import emit_step
_mounter = KernelOverlayFSMounter()
DEFAULT_ROOT = DEFAULT_LEFT4ME_ROOT DEFAULT_ROOT = DEFAULT_LEFT4ME_ROOT
@ -82,18 +86,23 @@ def start_instance(
env = _load_instance_env(instance_dir / "instance.env") env = _load_instance_env(instance_dir / "instance.env")
merged = runtime_dir / "merged"
if os.path.ismount(merged):
# Kernel overlayfs mounts persist when the web worker dies (unlike
# fuse daemons, which were reaped with their cgroup). Refuse rather
# than double-mount.
raise subprocess.CalledProcessError(
returncode=1,
cmd=["start_instance"],
stderr=f"runtime overlay already mounted at {merged}; refusing to double-mount",
)
emit_step("mounting runtime overlay...", on_stdout, passthrough) emit_step("mounting runtime overlay...", on_stdout, passthrough)
run_command( _mounter.mount(
[ lowerdirs=env["L4D2_LOWERDIRS"],
"fuse-overlayfs", upperdir=runtime_dir / "upper",
"-o", workdir=runtime_dir / "work",
( merged=merged,
f"lowerdir={env['L4D2_LOWERDIRS']},"
f"upperdir={runtime_dir / 'upper'},"
f"workdir={runtime_dir / 'work'}"
),
str(runtime_dir / "merged"),
],
on_stdout=on_stdout, on_stdout=on_stdout,
on_stderr=on_stderr, on_stderr=on_stderr,
passthrough=passthrough, passthrough=passthrough,
@ -135,17 +144,63 @@ def stop_instance(
passthrough=passthrough, passthrough=passthrough,
should_cancel=should_cancel, should_cancel=should_cancel,
) )
emit_step("unmounting runtime overlay...", on_stdout, passthrough) emit_step("unmounting runtime overlay (if mounted)...", on_stdout, passthrough)
run_command( try:
["fusermount3", "-u", str(root / "runtime" / name / "merged")], _mounter.unmount(
on_stdout=on_stdout, merged=root / "runtime" / name / "merged",
on_stderr=on_stderr, on_stdout=on_stdout,
passthrough=passthrough, on_stderr=on_stderr,
should_cancel=should_cancel, passthrough=passthrough,
) should_cancel=should_cancel,
)
except subprocess.CalledProcessError:
pass
emit_step("stop complete.", on_stdout, passthrough) emit_step("stop complete.", on_stdout, passthrough)
def _purge_instance(
name: str,
*,
root: Path,
on_stdout: Callable[[str], None] | None,
on_stderr: Callable[[str], None] | None,
passthrough: bool,
should_cancel: Callable[[], bool] | None,
) -> None:
instance_dir = root / "instances" / name
runtime_dir = root / "runtime" / name
emit_step("stopping systemd service (if running)...", on_stdout, passthrough)
try:
stop_service(
name,
on_stdout=on_stdout,
on_stderr=on_stderr,
passthrough=passthrough,
should_cancel=should_cancel,
)
except subprocess.CalledProcessError:
pass
emit_step("unmounting runtime overlay (if mounted)...", on_stdout, passthrough)
try:
_mounter.unmount(
merged=runtime_dir / "merged",
on_stdout=on_stdout,
on_stderr=on_stderr,
passthrough=passthrough,
should_cancel=should_cancel,
)
except subprocess.CalledProcessError:
pass
emit_step("removing instance files...", on_stdout, passthrough)
if instance_dir.exists():
shutil.rmtree(instance_dir)
if runtime_dir.exists():
shutil.rmtree(runtime_dir)
def delete_instance( def delete_instance(
name: str, name: str,
*, *,
@ -163,33 +218,34 @@ def delete_instance(
if not instance_dir.exists() and not runtime_dir.exists(): if not instance_dir.exists() and not runtime_dir.exists():
return return
emit_step("stopping systemd service (if running)...", on_stdout, passthrough) _purge_instance(
try: name,
stop_service( root=root,
name, on_stdout=on_stdout,
on_stdout=on_stdout, on_stderr=on_stderr,
on_stderr=on_stderr, passthrough=passthrough,
passthrough=passthrough, should_cancel=should_cancel,
should_cancel=should_cancel, )
)
except subprocess.CalledProcessError:
pass
emit_step("unmounting runtime overlay (if mounted)...", on_stdout, passthrough)
try:
run_command(
["fusermount3", "-u", str(runtime_dir / "merged")],
on_stdout=on_stdout,
on_stderr=on_stderr,
passthrough=passthrough,
should_cancel=should_cancel,
)
except subprocess.CalledProcessError:
pass
emit_step("removing instance files...", on_stdout, passthrough)
if instance_dir.exists():
shutil.rmtree(instance_dir)
if runtime_dir.exists():
shutil.rmtree(runtime_dir)
emit_step("delete complete.", on_stdout, passthrough) emit_step("delete complete.", on_stdout, passthrough)
def reset_instance(
name: str,
*,
root: Path | None = None,
on_stdout: Callable[[str], None] | None = None,
on_stderr: Callable[[str], None] | None = None,
passthrough: bool = False,
should_cancel: Callable[[], bool] | None = None,
) -> None:
name = validate_instance_name(name)
root = get_left4me_root() if root is None else Path(root)
_purge_instance(
name,
root=root,
on_stdout=on_stdout,
on_stderr=on_stderr,
passthrough=passthrough,
should_cancel=should_cancel,
)
emit_step("reset complete; next start will reinitialize from blueprint.", on_stdout, passthrough)

View file

@ -1,3 +1,4 @@
import shutil
from pathlib import Path from pathlib import Path
from typing import Callable from typing import Callable
@ -6,6 +7,57 @@ from l4d2host.process import run_command
from l4d2host.logging import emit_step from l4d2host.logging import emit_step
def _resolve_steamcmd_siblings(steamcmd: str) -> tuple[Path, Path] | None:
resolved = shutil.which(steamcmd)
base = Path(resolved).resolve() if resolved else Path(steamcmd)
if base.is_symlink():
base = base.resolve()
parent = base.parent
linux32 = parent / "linux32"
linux64 = parent / "linux64"
if linux32.is_dir() and linux64.is_dir():
return linux32, linux64
return None
def _set_symlink(link: Path, target: Path, on_stderr: Callable[[str], None] | None) -> None:
if link.is_symlink():
try:
current = link.resolve()
except OSError:
current = None
if current == target.resolve():
return
link.unlink()
elif link.exists():
if on_stderr is not None:
on_stderr(f"refusing to replace non-symlink at {link}")
return
link.symlink_to(target)
def _ensure_steam_sdk_symlinks(
install_dir: Path,
steamcmd: str,
*,
on_stdout: Callable[[str], None] | None,
on_stderr: Callable[[str], None] | None,
passthrough: bool,
) -> None:
emit_step("ensuring steam sdk symlinks...", on_stdout, passthrough)
siblings = _resolve_steamcmd_siblings(steamcmd)
if siblings is not None:
sdk32_target, sdk64_target = siblings
else:
fallback = install_dir / "bin"
sdk32_target, sdk64_target = fallback, fallback
steam_dir = Path.home() / ".steam"
steam_dir.mkdir(parents=True, exist_ok=True)
_set_symlink(steam_dir / "sdk32", sdk32_target, on_stderr)
_set_symlink(steam_dir / "sdk64", sdk64_target, on_stderr)
class SteamInstaller: class SteamInstaller:
def __init__(self, install_dir: Path | None = None, steamcmd: str = "steamcmd"): def __init__(self, install_dir: Path | None = None, steamcmd: str = "steamcmd"):
self.install_dir = get_left4me_root() / "installation" if install_dir is None else Path(install_dir) self.install_dir = get_left4me_root() / "installation" if install_dir is None else Path(install_dir)
@ -40,4 +92,11 @@ class SteamInstaller:
passthrough=passthrough, passthrough=passthrough,
should_cancel=should_cancel, should_cancel=should_cancel,
) )
_ensure_steam_sdk_symlinks(
self.install_dir,
self.steamcmd,
on_stdout=on_stdout,
on_stderr=on_stderr,
passthrough=passthrough,
)
emit_step("installation complete.", on_stdout, passthrough) emit_step("installation complete.", on_stdout, passthrough)

View file

@ -46,8 +46,8 @@ def test_default_install_dir_uses_left4me_root(tmp_path, monkeypatch):
def test_steam_installer_emits_steps(tmp_path, monkeypatch) -> None: def test_steam_installer_emits_steps(tmp_path, monkeypatch) -> None:
from pathlib import Path
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path)) monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
monkeypatch.setenv("HOME", str(tmp_path / "home"))
monkeypatch.setattr("l4d2host.steam_install.run_command", lambda cmd, **kwargs: None) monkeypatch.setattr("l4d2host.steam_install.run_command", lambda cmd, **kwargs: None)
steps: list[str] = [] steps: list[str] = []
@ -58,5 +58,82 @@ def test_steam_installer_emits_steps(tmp_path, monkeypatch) -> None:
assert steps == [ assert steps == [
"Step: downloading windows platform payload...", "Step: downloading windows platform payload...",
"Step: downloading linux platform payload...", "Step: downloading linux platform payload...",
"Step: ensuring steam sdk symlinks...",
"Step: installation complete." "Step: installation complete."
] ]
def test_install_creates_steam_sdk_symlinks_to_steamcmd_siblings(tmp_path, monkeypatch) -> None:
home = tmp_path / "home"
home.mkdir()
monkeypatch.setenv("HOME", str(home))
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
steamcmd_dir = tmp_path / "opt" / "steamcmd"
(steamcmd_dir / "linux32").mkdir(parents=True)
(steamcmd_dir / "linux64").mkdir(parents=True)
fake_steamcmd = steamcmd_dir / "steamcmd.sh"
fake_steamcmd.write_text("#!/bin/sh\nexit 0\n")
fake_steamcmd.chmod(0o755)
monkeypatch.setattr("l4d2host.steam_install.run_command", lambda cmd, **kwargs: None)
from l4d2host.steam_install import SteamInstaller
SteamInstaller(steamcmd=str(fake_steamcmd)).install_or_update()
sdk32 = home / ".steam" / "sdk32"
sdk64 = home / ".steam" / "sdk64"
assert sdk32.is_symlink()
assert sdk64.is_symlink()
assert sdk32.resolve() == (steamcmd_dir / "linux32").resolve()
assert sdk64.resolve() == (steamcmd_dir / "linux64").resolve()
def test_install_creates_steam_sdk_symlinks_falls_back_to_install_bin(tmp_path, monkeypatch) -> None:
home = tmp_path / "home"
home.mkdir()
monkeypatch.setenv("HOME", str(home))
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
install_bin = tmp_path / "installation" / "bin"
install_bin.mkdir(parents=True)
isolated_dir = tmp_path / "no-siblings"
isolated_dir.mkdir()
fake_steamcmd = isolated_dir / "steamcmd.sh"
fake_steamcmd.write_text("#!/bin/sh\nexit 0\n")
fake_steamcmd.chmod(0o755)
monkeypatch.setattr("l4d2host.steam_install.run_command", lambda cmd, **kwargs: None)
from l4d2host.steam_install import SteamInstaller
SteamInstaller(steamcmd=str(fake_steamcmd)).install_or_update()
sdk32 = home / ".steam" / "sdk32"
sdk64 = home / ".steam" / "sdk64"
assert sdk32.resolve() == install_bin.resolve()
assert sdk64.resolve() == install_bin.resolve()
def test_install_steam_sdk_symlinks_is_idempotent(tmp_path, monkeypatch) -> None:
home = tmp_path / "home"
home.mkdir()
monkeypatch.setenv("HOME", str(home))
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
steamcmd_dir = tmp_path / "opt" / "steamcmd"
(steamcmd_dir / "linux32").mkdir(parents=True)
(steamcmd_dir / "linux64").mkdir(parents=True)
fake_steamcmd = steamcmd_dir / "steamcmd.sh"
fake_steamcmd.write_text("#!/bin/sh\nexit 0\n")
fake_steamcmd.chmod(0o755)
monkeypatch.setattr("l4d2host.steam_install.run_command", lambda cmd, **kwargs: None)
from l4d2host.steam_install import SteamInstaller
installer = SteamInstaller(steamcmd=str(fake_steamcmd))
installer.install_or_update()
installer.install_or_update()
sdk32 = home / ".steam" / "sdk32"
assert sdk32.resolve() == (steamcmd_dir / "linux32").resolve()

View file

@ -0,0 +1,76 @@
from pathlib import Path
import pytest
HELPER_PATH = "/usr/local/libexec/left4me/left4me-overlay"
def test_mount_invokes_helper_with_name_only(monkeypatch: pytest.MonkeyPatch) -> None:
from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
calls: list[list[str]] = []
def fake_run_command(cmd, **kwargs):
del kwargs
calls.append(list(cmd))
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
KernelOverlayFSMounter().mount(
lowerdirs="/var/lib/left4me/installation",
upperdir=Path("/var/lib/left4me/runtime/alpha/upper"),
workdir=Path("/var/lib/left4me/runtime/alpha/work"),
merged=Path("/var/lib/left4me/runtime/alpha/merged"),
)
assert calls == [["sudo", "-n", HELPER_PATH, "mount", "alpha"]]
def test_unmount_invokes_helper_with_umount_verb(monkeypatch: pytest.MonkeyPatch) -> None:
from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
calls: list[list[str]] = []
def fake_run_command(cmd, **kwargs):
del kwargs
calls.append(list(cmd))
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
KernelOverlayFSMounter().unmount(merged=Path("/var/lib/left4me/runtime/alpha/merged"))
assert calls == [["sudo", "-n", HELPER_PATH, "umount", "alpha"]]
def test_mount_propagates_run_command_kwargs(monkeypatch: pytest.MonkeyPatch) -> None:
from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
captured: dict = {}
def fake_run_command(cmd, **kwargs):
captured["cmd"] = list(cmd)
captured["kwargs"] = kwargs
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
out: list[str] = []
err: list[str] = []
KernelOverlayFSMounter().mount(
lowerdirs="/var/lib/left4me/installation",
upperdir=Path("/var/lib/left4me/runtime/alpha/upper"),
workdir=Path("/var/lib/left4me/runtime/alpha/work"),
merged=Path("/var/lib/left4me/runtime/alpha/merged"),
on_stdout=out.append,
on_stderr=err.append,
passthrough=False,
should_cancel=lambda: False,
)
assert captured["cmd"][0:3] == ["sudo", "-n", HELPER_PATH]
captured["kwargs"]["on_stdout"]("hi")
captured["kwargs"]["on_stderr"]("oops")
assert out == ["hi"]
assert err == ["oops"]
assert captured["kwargs"]["passthrough"] is False
assert callable(captured["kwargs"]["should_cancel"])

View file

@ -6,6 +6,7 @@ import pytest
from l4d2host.instances import ( from l4d2host.instances import (
delete_instance, delete_instance,
initialize_instance, initialize_instance,
reset_instance,
start_instance, start_instance,
stop_instance, stop_instance,
) )
@ -27,15 +28,51 @@ def test_start_order(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
) )
(instance_dir / "server.cfg").write_text("sv_consistency 1") (instance_dir / "server.cfg").write_text("sv_consistency 1")
monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command) monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command) monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
start_instance("alpha", root=tmp_path) start_instance("alpha", root=tmp_path)
assert calls[0][0] == "fuse-overlayfs" assert calls[0] == [
"sudo",
"-n",
"/usr/local/libexec/left4me/left4me-overlay",
"mount",
"alpha",
]
assert calls[1] == ["sudo", "-n", "/usr/local/libexec/left4me/left4me-systemctl", "start", "alpha"] assert calls[1] == ["sudo", "-n", "/usr/local/libexec/left4me/left4me-systemctl", "start", "alpha"]
def test_start_refuses_to_double_mount(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
calls: list[list[str]] = []
def fake_run_command(cmd, **kwargs):
del kwargs
calls.append(list(cmd))
instance_dir = tmp_path / "instances" / "alpha"
runtime_dir = tmp_path / "runtime" / "alpha"
(runtime_dir / "merged").mkdir(parents=True)
instance_dir.mkdir(parents=True)
(instance_dir / "instance.env").write_text("L4D2_PORT=27015\nL4D2_ARGS=\nL4D2_LOWERDIRS=/x\n")
(instance_dir / "server.cfg").write_text("")
merged = runtime_dir / "merged"
def fake_ismount(path):
return Path(path) == merged
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
monkeypatch.setattr("l4d2host.instances.os.path.ismount", fake_ismount)
with pytest.raises(subprocess.CalledProcessError) as exc_info:
start_instance("alpha", root=tmp_path)
assert "already mounted" in (exc_info.value.stderr or "")
assert calls == [], "no mount/start commands must be issued when refusing"
def test_delete_missing_is_noop(tmp_path: Path) -> None: def test_delete_missing_is_noop(tmp_path: Path) -> None:
delete_instance("missing", root=tmp_path) delete_instance("missing", root=tmp_path)
@ -56,7 +93,7 @@ def test_delete_succeeds_when_stop_service_fails(tmp_path: Path, monkeypatch: py
(tmp_path / "instances" / "alpha").mkdir(parents=True) (tmp_path / "instances" / "alpha").mkdir(parents=True)
(tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True) (tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command) monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command) monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
delete_instance("alpha", root=tmp_path) delete_instance("alpha", root=tmp_path)
@ -67,7 +104,7 @@ def test_delete_succeeds_when_stop_service_fails(tmp_path: Path, monkeypatch: py
@pytest.mark.parametrize("bad_name", ["..", "../escape", "foo/bar", " foo", "Foo"]) @pytest.mark.parametrize("bad_name", ["..", "../escape", "foo/bar", " foo", "Foo"])
def test_lifecycle_rejects_unsafe_instance_names(tmp_path: Path, bad_name: str) -> None: def test_lifecycle_rejects_unsafe_instance_names(tmp_path: Path, bad_name: str) -> None:
for func in (start_instance, stop_instance, delete_instance): for func in (start_instance, stop_instance, delete_instance, reset_instance):
with pytest.raises(ValueError): with pytest.raises(ValueError):
func(bad_name, root=tmp_path) func(bad_name, root=tmp_path)
with pytest.raises(ValueError): with pytest.raises(ValueError):
@ -76,6 +113,49 @@ def test_lifecycle_rejects_unsafe_instance_names(tmp_path: Path, bad_name: str)
assert not (tmp_path / "runtime").exists() assert not (tmp_path / "runtime").exists()
def test_reset_stops_unmounts_and_removes_dirs(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
calls: list[list[str]] = []
def fake_run_command(cmd, **kwargs):
del kwargs
calls.append(list(cmd))
instance_dir = tmp_path / "instances" / "alpha"
runtime_dir = tmp_path / "runtime" / "alpha"
instance_dir.mkdir(parents=True)
(runtime_dir / "merged").mkdir(parents=True)
(instance_dir / "instance.env").write_text("L4D2_PORT=27015\n")
(runtime_dir / "upper" / "logs").mkdir(parents=True)
(runtime_dir / "upper" / "logs" / "console.log").write_text("noise")
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
reset_instance("alpha", root=tmp_path)
assert not instance_dir.exists()
assert not runtime_dir.exists()
assert any("left4me-systemctl" in arg for cmd in calls for arg in cmd)
assert any("stop" in cmd for cmd in calls)
def test_reset_on_never_initialized_is_noop(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""No instance/runtime directories yet — reset should still attempt the
stop+unmount (both suppressed on failure) and not raise."""
def fake_run_command(cmd, **kwargs):
del kwargs
if "stop" in cmd:
raise subprocess.CalledProcessError(returncode=5, cmd=list(cmd), stderr="not loaded")
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
reset_instance("alpha", root=tmp_path)
assert not (tmp_path / "instances" / "alpha").exists()
assert not (tmp_path / "runtime" / "alpha").exists()
def test_delete_stopped_instance_removes_dirs(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: def test_delete_stopped_instance_removes_dirs(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
calls: list[list[str]] = [] calls: list[list[str]] = []
@ -86,7 +166,7 @@ def test_delete_stopped_instance_removes_dirs(tmp_path: Path, monkeypatch: pytes
(tmp_path / "instances" / "alpha").mkdir(parents=True) (tmp_path / "instances" / "alpha").mkdir(parents=True)
(tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True) (tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command) monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command) monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
delete_instance("alpha", root=tmp_path) delete_instance("alpha", root=tmp_path)
@ -96,27 +176,58 @@ def test_delete_stopped_instance_removes_dirs(tmp_path: Path, monkeypatch: pytes
assert ["sudo", "-n", "/usr/local/libexec/left4me/left4me-systemctl", "stop", "alpha"] in calls assert ["sudo", "-n", "/usr/local/libexec/left4me/left4me-systemctl", "stop", "alpha"] in calls
def test_delete_succeeds_when_unmount_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: def test_stop_succeeds_when_unmount_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
fusermount_calls: list[list[str]] = [] umount_calls: list[list[str]] = []
def fake_run_command(cmd, **kwargs): def fake_run_command(cmd, **kwargs):
del kwargs del kwargs
if cmd and cmd[0] == "fusermount3": if cmd[:4] == [
fusermount_calls.append(list(cmd)) "sudo",
"-n",
"/usr/local/libexec/left4me/left4me-overlay",
"umount",
]:
umount_calls.append(list(cmd))
raise subprocess.CalledProcessError( raise subprocess.CalledProcessError(
returncode=1, returncode=1,
cmd=list(cmd), cmd=list(cmd),
stderr="fusermount3: entry for merged not found in /etc/mtab", stderr="umount: /var/lib/left4me/runtime/alpha/merged: not mounted",
)
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
stop_instance("alpha", root=tmp_path)
assert umount_calls, "stop must always attempt the overlay helper (no preflight)"
def test_delete_succeeds_when_unmount_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
umount_calls: list[list[str]] = []
def fake_run_command(cmd, **kwargs):
del kwargs
if cmd[:4] == [
"sudo",
"-n",
"/usr/local/libexec/left4me/left4me-overlay",
"umount",
]:
umount_calls.append(list(cmd))
raise subprocess.CalledProcessError(
returncode=1,
cmd=list(cmd),
stderr="umount: /var/lib/left4me/runtime/alpha/merged: not mounted",
) )
(tmp_path / "instances" / "alpha").mkdir(parents=True) (tmp_path / "instances" / "alpha").mkdir(parents=True)
(tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True) (tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command) monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command) monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
delete_instance("alpha", root=tmp_path) delete_instance("alpha", root=tmp_path)
assert fusermount_calls, "delete must always attempt fusermount3 -u (no preflight)" assert umount_calls, "delete must always attempt the overlay helper (no preflight)"
assert not (tmp_path / "instances" / "alpha").exists() assert not (tmp_path / "instances" / "alpha").exists()
assert not (tmp_path / "runtime" / "alpha").exists() assert not (tmp_path / "runtime" / "alpha").exists()

View file

@ -0,0 +1,168 @@
import os
import shlex
import subprocess
import sys
from pathlib import Path
import pytest
HELPER_SOURCE = (
Path(__file__).resolve().parents[2]
/ "deploy"
/ "files"
/ "usr"
/ "local"
/ "libexec"
/ "left4me"
/ "left4me-overlay"
)
def _setup_instance(root: Path, name: str = "alpha", lowerdirs: list[str] | None = None) -> None:
"""Create the on-disk shape the helper expects."""
(root / "installation").mkdir(parents=True, exist_ok=True)
(root / "overlays" / "workshop").mkdir(parents=True, exist_ok=True)
if lowerdirs is None:
lowerdirs = [str(root / "overlays" / "workshop"), str(root / "installation")]
inst_dir = root / "instances" / name
inst_dir.mkdir(parents=True)
(inst_dir / "instance.env").write_text(
f"L4D2_PORT=27015\nL4D2_ARGS=-tickrate 100\nL4D2_LOWERDIRS={':'.join(lowerdirs)}\n"
)
runtime = root / "runtime" / name
(runtime / "upper").mkdir(parents=True)
(runtime / "work").mkdir(parents=True)
(runtime / "merged").mkdir(parents=True)
def _run(args: list[str], root: Path, extra_env: dict[str, str] | None = None) -> subprocess.CompletedProcess:
env = {
**os.environ,
"LEFT4ME_ROOT": str(root),
"LEFT4ME_OVERLAY_PRINT_ONLY": "1",
}
if extra_env:
env.update(extra_env)
return subprocess.run(
[sys.executable, str(HELPER_SOURCE), *args],
env=env,
capture_output=True,
text=True,
)
def test_mount_prints_expected_nsenter_command(tmp_path: Path) -> None:
_setup_instance(tmp_path)
result = _run(["mount", "alpha"], tmp_path)
assert result.returncode == 0, result.stderr
parts = shlex.split(result.stdout.strip())
assert parts[0] == "/usr/bin/nsenter"
assert parts[1] == "--mount=/proc/1/ns/mnt"
assert parts[2] == "--"
assert parts[3] == "/bin/mount"
assert parts[4:6] == ["-t", "overlay"]
assert parts[6] == "overlay"
assert parts[7] == "-o"
options = parts[8]
assert f"upperdir={tmp_path}/runtime/alpha/upper" in options
assert f"workdir={tmp_path}/runtime/alpha/work" in options
assert f"lowerdir={tmp_path}/overlays/workshop:{tmp_path}/installation" in options
assert parts[9] == str(tmp_path / "runtime" / "alpha" / "merged")
def test_umount_prints_expected_nsenter_command(tmp_path: Path) -> None:
_setup_instance(tmp_path)
result = _run(["umount", "alpha"], tmp_path)
assert result.returncode == 0, result.stderr
parts = shlex.split(result.stdout.strip())
assert parts == [
"/usr/bin/nsenter",
"--mount=/proc/1/ns/mnt",
"--",
"/bin/umount",
str(tmp_path / "runtime" / "alpha" / "merged"),
]
@pytest.mark.parametrize("bad_name", ["..", "../escape", "FOO", "foo bar", "foo/bar", ""])
def test_rejects_bad_instance_name(tmp_path: Path, bad_name: str) -> None:
result = _run(["mount", bad_name], tmp_path)
assert result.returncode != 0
assert "invalid instance name" in result.stderr or "usage:" in result.stderr
def test_rejects_lowerdir_outside_allowlist(tmp_path: Path) -> None:
_setup_instance(tmp_path, lowerdirs=["/etc"])
result = _run(["mount", "alpha"], tmp_path)
assert result.returncode != 0
assert "outside the permitted roots" in result.stderr
def test_rejects_lowerdir_traversal(tmp_path: Path) -> None:
# An overlay subdirectory whose path uses .. to escape the overlays root.
_setup_instance(tmp_path, lowerdirs=[str(tmp_path / "overlays" / "..") + "/etc"])
result = _run(["mount", "alpha"], tmp_path)
assert result.returncode != 0
assert "outside the permitted roots" in result.stderr or "path does not exist" in result.stderr
def test_rejects_lowerdir_symlink_escape(tmp_path: Path) -> None:
_setup_instance(tmp_path)
sneaky = tmp_path / "overlays" / "sneaky"
os.symlink("/etc", sneaky)
# rewrite instance.env to point at the symlink
inst_env = tmp_path / "instances" / "alpha" / "instance.env"
inst_env.write_text(f"L4D2_LOWERDIRS={sneaky}\n")
result = _run(["mount", "alpha"], tmp_path)
assert result.returncode != 0
assert "outside the permitted roots" in result.stderr
def test_rejects_missing_instance_env(tmp_path: Path) -> None:
(tmp_path / "instances" / "alpha").mkdir(parents=True)
runtime = tmp_path / "runtime" / "alpha"
(runtime / "upper").mkdir(parents=True)
(runtime / "work").mkdir(parents=True)
(runtime / "merged").mkdir(parents=True)
result = _run(["mount", "alpha"], tmp_path)
assert result.returncode != 0
assert "instance.env not found" in result.stderr
def test_rejects_lowerdir_count_over_cap(tmp_path: Path) -> None:
(tmp_path / "installation").mkdir()
many = [str(tmp_path / "installation")] * 501
_setup_instance(tmp_path, lowerdirs=many)
result = _run(["mount", "alpha"], tmp_path)
assert result.returncode != 0
assert "501 entries" in result.stderr
def test_rejects_empty_lowerdir_entry(tmp_path: Path) -> None:
(tmp_path / "installation").mkdir()
_setup_instance(
tmp_path,
lowerdirs=[str(tmp_path / "installation"), "", str(tmp_path / "installation")],
)
result = _run(["mount", "alpha"], tmp_path)
assert result.returncode != 0
assert "empty entry" in result.stderr
@pytest.mark.skipif(sys.platform != "linux", reason="user.* xattrs are Linux-only")
def test_rejects_upperdir_with_fuseoverlayfs_xattr(tmp_path: Path) -> None:
_setup_instance(tmp_path)
tainted = tmp_path / "runtime" / "alpha" / "upper" / "deleted-thing"
tainted.write_bytes(b"")
try:
os.setxattr(tainted, "user.fuseoverlayfs.opaque", b"y")
except OSError:
pytest.skip("filesystem doesn't support user.* xattrs")
result = _run(["mount", "alpha"], tmp_path)
assert result.returncode != 0
assert "fuse-overlayfs xattr" in result.stderr

View file

@ -0,0 +1,174 @@
"""workshop overlays
Revision ID: 0002_workshop_overlays
Revises: b2c684fddbd3
Create Date: 2026-05-07
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0002_workshop_overlays"
down_revision: Union[str, Sequence[str], None] = "b2c684fddbd3"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _existing_overlays_table() -> sa.Table:
"""Pre-migration shape used as `copy_from` so batch_alter_table rebuilds
overlays without the inline UNIQUE on `name` (replaced by partial unique
indexes after the recreate)."""
metadata = sa.MetaData()
return sa.Table(
"overlays",
metadata,
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("name", sa.String(length=128), nullable=False),
sa.Column("path", sa.String(length=512), nullable=False),
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
)
def upgrade() -> None:
# 1. Recreate `overlays` with `type`, `user_id`, autoincrement, and no inline UNIQUE on name.
with op.batch_alter_table(
"overlays",
recreate="always",
copy_from=_existing_overlays_table(),
table_kwargs={"sqlite_autoincrement": True},
) as batch_op:
batch_op.add_column(
sa.Column(
"type",
sa.String(length=16),
nullable=False,
server_default="external",
)
)
batch_op.add_column(
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", name="fk_overlays_user_id_users"),
nullable=True,
)
)
batch_op.create_index("ix_overlays_type_user_id", ["type", "user_id"])
# Drop the temporary server_default once existing rows are backfilled.
with op.batch_alter_table("overlays") as batch_op:
batch_op.alter_column("type", server_default=None)
# 2. Partial unique indexes for name uniqueness:
# - system overlays (user_id IS NULL): globally unique by name
# - user overlays (user_id IS NOT NULL): unique per user by name
op.create_index(
"uq_overlay_name_system",
"overlays",
["name"],
unique=True,
sqlite_where=sa.text("user_id IS NULL"),
)
op.create_index(
"uq_overlay_name_per_user",
"overlays",
["name", "user_id"],
unique=True,
sqlite_where=sa.text("user_id IS NOT NULL"),
)
# 3. workshop_items registry (global, deduplicated by steam_id).
op.create_table(
"workshop_items",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("steam_id", sa.String(length=20), nullable=False, unique=True),
sa.Column("title", sa.String(length=255), nullable=False, server_default=""),
sa.Column("filename", sa.String(length=255), nullable=False, server_default=""),
sa.Column("file_url", sa.Text(), nullable=False, server_default=""),
sa.Column("file_size", sa.BigInteger(), nullable=False, server_default="0"),
sa.Column("time_updated", sa.Integer(), nullable=False, server_default="0"),
sa.Column("preview_url", sa.Text(), nullable=False, server_default=""),
sa.Column("last_downloaded_at", sa.DateTime(), nullable=True),
sa.Column("last_error", sa.Text(), nullable=False, server_default=""),
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
)
# 4. overlay_workshop_items association.
op.create_table(
"overlay_workshop_items",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"overlay_id",
sa.Integer(),
sa.ForeignKey("overlays.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"workshop_item_id",
sa.Integer(),
sa.ForeignKey("workshop_items.id", ondelete="RESTRICT"),
nullable=False,
),
sa.UniqueConstraint(
"overlay_id", "workshop_item_id", name="uq_overlay_workshop_item"
),
)
op.create_index(
"ix_owi_workshop_item",
"overlay_workshop_items",
["workshop_item_id"],
)
# 5. Add overlay_id to jobs for build_overlay tracking.
with op.batch_alter_table("jobs") as batch_op:
batch_op.add_column(
sa.Column(
"overlay_id",
sa.Integer(),
sa.ForeignKey("overlays.id", name="fk_jobs_overlay_id_overlays"),
nullable=True,
)
)
def downgrade() -> None:
with op.batch_alter_table("jobs") as batch_op:
batch_op.drop_column("overlay_id")
op.drop_index("ix_owi_workshop_item", table_name="overlay_workshop_items")
op.drop_table("overlay_workshop_items")
op.drop_table("workshop_items")
op.drop_index("uq_overlay_name_per_user", table_name="overlays")
op.drop_index("uq_overlay_name_system", table_name="overlays")
op.drop_index("ix_overlays_type_user_id", table_name="overlays")
# Recreate `overlays` to drop type/user_id and restore single-column UNIQUE on name.
current_overlays = sa.Table(
"overlays",
sa.MetaData(),
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("name", sa.String(length=128), nullable=False),
sa.Column("path", sa.String(length=512), nullable=False),
sa.Column("type", sa.String(length=16), nullable=False),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", name="fk_overlays_user_id_users"),
nullable=True,
),
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
)
with op.batch_alter_table(
"overlays",
recreate="always",
copy_from=current_overlays,
) as batch_op:
batch_op.drop_column("user_id")
batch_op.drop_column("type")
batch_op.create_unique_constraint("uq_overlays_name", ["name"])

View file

@ -0,0 +1,105 @@
"""global map overlays
Revision ID: 0003_global_map_overlays
Revises: 0002_workshop_overlays
Create Date: 2026-05-07
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0003_global_map_overlays"
down_revision: Union[str, Sequence[str], None] = "0002_workshop_overlays"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.batch_alter_table("jobs") as batch_op:
batch_op.alter_column("user_id", existing_type=sa.Integer(), nullable=True)
op.create_table(
"global_overlay_sources",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"overlay_id",
sa.Integer(),
sa.ForeignKey("overlays.id", ondelete="CASCADE"),
nullable=False,
unique=True,
),
sa.Column("source_key", sa.String(length=64), nullable=False, unique=True),
sa.Column("source_type", sa.String(length=32), nullable=False),
sa.Column("source_url", sa.Text(), nullable=False),
sa.Column("last_manifest_hash", sa.String(length=64), nullable=False, server_default=""),
sa.Column("last_refreshed_at", sa.DateTime(), nullable=True),
sa.Column("last_error", sa.Text(), nullable=False, server_default=""),
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
)
op.create_index("ix_global_overlay_sources_type", "global_overlay_sources", ["source_type"])
op.create_table(
"global_overlay_items",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"source_id",
sa.Integer(),
sa.ForeignKey("global_overlay_sources.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("item_key", sa.String(length=255), nullable=False),
sa.Column("display_name", sa.String(length=255), nullable=False, server_default=""),
sa.Column("download_url", sa.Text(), nullable=False),
sa.Column("expected_vpk_name", sa.String(length=255), nullable=False, server_default=""),
sa.Column("expected_size", sa.BigInteger(), nullable=True),
sa.Column("expected_md5", sa.String(length=32), nullable=False, server_default=""),
sa.Column("etag", sa.String(length=255), nullable=False, server_default=""),
sa.Column("last_modified", sa.String(length=255), nullable=False, server_default=""),
sa.Column("content_length", sa.BigInteger(), nullable=True),
sa.Column("last_downloaded_at", sa.DateTime(), nullable=True),
sa.Column("last_error", sa.Text(), nullable=False, server_default=""),
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
sa.UniqueConstraint("source_id", "item_key", name="uq_global_overlay_item_source_key"),
)
op.create_index("ix_global_overlay_items_source", "global_overlay_items", ["source_id"])
op.create_table(
"global_overlay_item_files",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"item_id",
sa.Integer(),
sa.ForeignKey("global_overlay_items.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("vpk_name", sa.String(length=255), nullable=False),
sa.Column("cache_path", sa.Text(), nullable=False),
sa.Column("size", sa.BigInteger(), nullable=False),
sa.Column("md5", sa.String(length=32), nullable=False, server_default=""),
sa.Column("created_at", sa.DateTime(), nullable=False),
sa.Column("updated_at", sa.DateTime(), nullable=False),
sa.UniqueConstraint("item_id", "vpk_name", name="uq_global_overlay_item_file_name"),
)
op.create_index("ix_global_overlay_item_files_item", "global_overlay_item_files", ["item_id"])
def downgrade() -> None:
op.drop_index("ix_global_overlay_item_files_item", table_name="global_overlay_item_files")
op.drop_table("global_overlay_item_files")
op.drop_index("ix_global_overlay_items_source", table_name="global_overlay_items")
op.drop_table("global_overlay_items")
op.drop_index("ix_global_overlay_sources_type", table_name="global_overlay_sources")
op.drop_table("global_overlay_sources")
op.execute(
"DELETE FROM job_logs WHERE job_id IN "
"(SELECT id FROM jobs WHERE user_id IS NULL)"
)
op.execute("DELETE FROM jobs WHERE user_id IS NULL")
with op.batch_alter_table("jobs") as batch_op:
batch_op.alter_column("user_id", existing_type=sa.Integer(), nullable=False)

View file

@ -0,0 +1,32 @@
"""drop legacy external overlay type
Revision ID: 0004_drop_legacy_external_overlay_type
Revises: 0003_global_map_overlays
Create Date: 2026-05-08
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0004_drop_legacy_external_overlay_type"
down_revision: Union[str, Sequence[str], None] = "0003_global_map_overlays"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute(
"DELETE FROM jobs "
"WHERE overlay_id IN (SELECT id FROM overlays WHERE type = 'external')"
)
op.execute(
"DELETE FROM blueprint_overlays "
"WHERE overlay_id IN (SELECT id FROM overlays WHERE type = 'external')"
)
op.execute("DELETE FROM overlays WHERE type = 'external'")
def downgrade() -> None:
# data is gone; intentional one-way migration
pass

View file

@ -0,0 +1,79 @@
"""script overlays
Revision ID: 0005_script_overlays
Revises: 0004_drop_legacy_external_overlay_type
Create Date: 2026-05-08
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0005_script_overlays"
down_revision: Union[str, Sequence[str], None] = "0004_drop_legacy_external_overlay_type"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# 1. Wipe legacy global-type overlay rows and any references to them.
op.execute(
"DELETE FROM jobs "
"WHERE overlay_id IN (SELECT id FROM overlays "
"WHERE type IN ('l4d2center_maps', 'cedapug_maps'))"
)
op.execute(
"DELETE FROM blueprint_overlays "
"WHERE overlay_id IN (SELECT id FROM overlays "
"WHERE type IN ('l4d2center_maps', 'cedapug_maps'))"
)
op.execute(
"DELETE FROM overlay_workshop_items "
"WHERE overlay_id IN (SELECT id FROM overlays "
"WHERE type IN ('l4d2center_maps', 'cedapug_maps'))"
)
op.execute(
"DELETE FROM overlays WHERE type IN ('l4d2center_maps', 'cedapug_maps')"
)
# 2. Drop globals tables in FK order: item_files -> items -> sources.
op.drop_index(
"ix_global_overlay_item_files_item",
table_name="global_overlay_item_files",
)
op.drop_table("global_overlay_item_files")
op.drop_index(
"ix_global_overlay_items_source", table_name="global_overlay_items"
)
op.drop_table("global_overlay_items")
op.drop_index(
"ix_global_overlay_sources_type", table_name="global_overlay_sources"
)
op.drop_table("global_overlay_sources")
# 3. Add new columns on overlays.
with op.batch_alter_table("overlays") as batch_op:
batch_op.add_column(
sa.Column(
"script",
sa.Text(),
nullable=False,
server_default="",
)
)
batch_op.add_column(
sa.Column(
"last_build_status",
sa.String(length=16),
nullable=False,
server_default="",
)
)
def downgrade() -> None:
# data is gone; intentional one-way migration
pass

View file

@ -16,6 +16,7 @@ from l4d2web.routes.log_routes import bp as log_bp
from l4d2web.routes.overlay_routes import bp as overlay_bp from l4d2web.routes.overlay_routes import bp as overlay_bp
from l4d2web.routes.page_routes import bp as page_bp from l4d2web.routes.page_routes import bp as page_bp
from l4d2web.routes.server_routes import bp as server_bp from l4d2web.routes.server_routes import bp as server_bp
from l4d2web.routes.workshop_routes import bp as workshop_bp
from l4d2web.services.job_worker import recover_stale_jobs, start_job_workers from l4d2web.services.job_worker import recover_stale_jobs, start_job_workers
@ -38,7 +39,11 @@ def create_app(test_config: dict[str, object] | None = None) -> Flask:
if not app.config.get("TESTING") and (not secret_key or secret_key == "dev"): if not app.config.get("TESTING") and (not secret_key or secret_key == "dev"):
raise RuntimeError("SECRET_KEY must be set to a non-default value outside of testing") raise RuntimeError("SECRET_KEY must be set to a non-default value outside of testing")
app.config["SESSION_COOKIE_SECURE"] = not app.config.get("TESTING", False) secure_env = os.getenv("SESSION_COOKIE_SECURE")
if secure_env is not None:
app.config["SESSION_COOKIE_SECURE"] = secure_env.lower() not in {"0", "false", "no"}
else:
app.config["SESSION_COOKIE_SECURE"] = not app.config.get("TESTING", False)
with app.app_context(): with app.app_context():
init_db() init_db()
@ -65,6 +70,7 @@ def create_app(test_config: dict[str, object] | None = None) -> Flask:
app.before_request(load_current_user) app.before_request(load_current_user)
app.register_blueprint(auth_bp) app.register_blueprint(auth_bp)
app.register_blueprint(overlay_bp) app.register_blueprint(overlay_bp)
app.register_blueprint(workshop_bp)
app.register_blueprint(blueprint_bp) app.register_blueprint(blueprint_bp)
app.register_blueprint(server_bp) app.register_blueprint(server_bp)
app.register_blueprint(job_bp) app.register_blueprint(job_bp)

View file

@ -1,4 +1,5 @@
import os import os
from pathlib import Path
import click import click
from sqlalchemy.exc import IntegrityError from sqlalchemy.exc import IntegrityError
@ -6,7 +7,11 @@ from sqlalchemy import select
from l4d2web.auth import hash_password from l4d2web.auth import hash_password
from l4d2web.db import session_scope from l4d2web.db import session_scope
from l4d2web.models import User from l4d2web.models import Overlay, User
from l4d2web.services.overlay_creation import (
create_overlay_directory,
generate_overlay_path,
)
@click.command("promote-admin") @click.command("promote-admin")
@ -41,6 +46,50 @@ def create_user(username: str, admin: bool) -> None:
click.echo(f"created user {username}") click.echo(f"created user {username}")
@click.command("seed-script-overlays")
@click.argument(
"directory",
type=click.Path(exists=True, file_okay=False, path_type=Path),
)
def seed_script_overlays(directory: Path) -> None:
"""Upsert one system-wide script overlay per *.sh file in DIRECTORY.
Overlay name = filename stem; user_id stays NULL. Existing rows by name
have their script refreshed in place. Hard-errors if a name collides
with a non-script overlay.
"""
sh_files = sorted(p for p in directory.glob("*.sh") if p.stem)
if not sh_files:
click.echo(f"no *.sh files in {directory}", err=True)
return
with session_scope() as db:
for sh in sh_files:
name = sh.stem
content = sh.read_text()
existing = db.scalar(
select(Overlay).where(Overlay.name == name, Overlay.user_id.is_(None))
)
if existing is not None:
if existing.type != "script":
raise click.ClickException(
f"overlay {name!r} exists but is type={existing.type!r}, not script"
)
existing.script = content
click.echo(f"updated {name} (id={existing.id})")
else:
overlay = Overlay(
name=name, path="", type="script", user_id=None, script=content
)
db.add(overlay)
db.flush()
overlay.path = generate_overlay_path(overlay.id)
db.flush()
create_overlay_directory(overlay)
click.echo(f"created {name} (id={overlay.id})")
def register_cli(app) -> None: def register_cli(app) -> None:
app.cli.add_command(promote_admin) app.cli.add_command(promote_admin)
app.cli.add_command(create_user) app.cli.add_command(create_user)
app.cli.add_command(seed_script_overlays)

View file

@ -10,6 +10,8 @@ DEFAULT_CONFIG: dict[str, object] = {
"JOB_WORKER_POLL_SECONDS": 1, "JOB_WORKER_POLL_SECONDS": 1,
"JOB_LOG_REPLAY_LIMIT": 2000, "JOB_LOG_REPLAY_LIMIT": 2000,
"JOB_LOG_LINE_MAX_CHARS": 4096, "JOB_LOG_LINE_MAX_CHARS": 4096,
"PORT_RANGE_START": 27015,
"PORT_RANGE_END": 27115,
} }
@ -27,4 +29,6 @@ def load_config() -> dict[str, object]:
"JOB_WORKER_POLL_SECONDS": float(os.getenv("JOB_WORKER_POLL_SECONDS", "1")), "JOB_WORKER_POLL_SECONDS": float(os.getenv("JOB_WORKER_POLL_SECONDS", "1")),
"JOB_LOG_REPLAY_LIMIT": int(os.getenv("JOB_LOG_REPLAY_LIMIT", "2000")), "JOB_LOG_REPLAY_LIMIT": int(os.getenv("JOB_LOG_REPLAY_LIMIT", "2000")),
"JOB_LOG_LINE_MAX_CHARS": int(os.getenv("JOB_LOG_LINE_MAX_CHARS", "4096")), "JOB_LOG_LINE_MAX_CHARS": int(os.getenv("JOB_LOG_LINE_MAX_CHARS", "4096")),
"PORT_RANGE_START": int(os.getenv("LEFT4ME_PORT_RANGE_START", "27015")),
"PORT_RANGE_END": int(os.getenv("LEFT4ME_PORT_RANGE_END", "27115")),
} }

View file

@ -1,6 +1,17 @@
from datetime import UTC, datetime from datetime import UTC, datetime
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, String, Text from sqlalchemy import (
BigInteger,
Boolean,
DateTime,
ForeignKey,
Index,
Integer,
String,
Text,
UniqueConstraint,
text,
)
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
@ -25,14 +36,68 @@ class User(Base):
class Overlay(Base): class Overlay(Base):
__tablename__ = "overlays" __tablename__ = "overlays"
__table_args__ = (
Index(
"uq_overlay_name_system",
"name",
unique=True,
sqlite_where=text("user_id IS NULL"),
),
Index(
"uq_overlay_name_per_user",
"name",
"user_id",
unique=True,
sqlite_where=text("user_id IS NOT NULL"),
),
Index("ix_overlays_type_user_id", "type", "user_id"),
{"sqlite_autoincrement": True},
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
name: Mapped[str] = mapped_column(String(128), unique=True, nullable=False) name: Mapped[str] = mapped_column(String(128), nullable=False)
path: Mapped[str] = mapped_column(String(512), nullable=False) path: Mapped[str] = mapped_column(String(512), nullable=False)
type: Mapped[str] = mapped_column(String(16), nullable=False, default="workshop")
user_id: Mapped[int | None] = mapped_column(ForeignKey("users.id"), nullable=True)
script: Mapped[str] = mapped_column(Text, default="", nullable=False)
last_build_status: Mapped[str] = mapped_column(String(16), default="", nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False) created_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False) updated_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)
class WorkshopItem(Base):
__tablename__ = "workshop_items"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
steam_id: Mapped[str] = mapped_column(String(20), unique=True, nullable=False)
title: Mapped[str] = mapped_column(String(255), default="", nullable=False)
filename: Mapped[str] = mapped_column(String(255), default="", nullable=False)
file_url: Mapped[str] = mapped_column(Text, default="", nullable=False)
file_size: Mapped[int] = mapped_column(BigInteger, default=0, nullable=False)
time_updated: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
preview_url: Mapped[str] = mapped_column(Text, default="", nullable=False)
last_downloaded_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
last_error: Mapped[str] = mapped_column(Text, default="", nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)
class OverlayWorkshopItem(Base):
__tablename__ = "overlay_workshop_items"
__table_args__ = (
UniqueConstraint("overlay_id", "workshop_item_id", name="uq_overlay_workshop_item"),
Index("ix_owi_workshop_item", "workshop_item_id"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
overlay_id: Mapped[int] = mapped_column(
ForeignKey("overlays.id", ondelete="CASCADE"), nullable=False
)
workshop_item_id: Mapped[int] = mapped_column(
ForeignKey("workshop_items.id", ondelete="RESTRICT"), nullable=False
)
class Blueprint(Base): class Blueprint(Base):
__tablename__ = "blueprints" __tablename__ = "blueprints"
@ -76,8 +141,9 @@ class Job(Base):
__tablename__ = "jobs" __tablename__ = "jobs"
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False) user_id: Mapped[int | None] = mapped_column(ForeignKey("users.id"), nullable=True)
server_id: Mapped[int | None] = mapped_column(ForeignKey("servers.id"), nullable=True) server_id: Mapped[int | None] = mapped_column(ForeignKey("servers.id"), nullable=True)
overlay_id: Mapped[int | None] = mapped_column(ForeignKey("overlays.id"), nullable=True)
operation: Mapped[str] = mapped_column(String(32), nullable=False) operation: Mapped[str] = mapped_column(String(32), nullable=False)
state: Mapped[str] = mapped_column(String(16), default="queued", nullable=False) state: Mapped[str] = mapped_column(String(16), default="queued", nullable=False)
exit_code: Mapped[int | None] = mapped_column(Integer, nullable=True) exit_code: Mapped[int | None] = mapped_column(Integer, nullable=True)

View file

@ -14,6 +14,7 @@ dependencies = [
"alembic>=1.13", "alembic>=1.13",
"PyYAML>=6.0", "PyYAML>=6.0",
"gunicorn>=22.0", "gunicorn>=22.0",
"requests>=2.31",
] ]
[tool.setuptools] [tool.setuptools]

View file

@ -6,7 +6,7 @@ from sqlalchemy import delete, func, select
from l4d2web.auth import current_user, require_login from l4d2web.auth import current_user, require_login
from l4d2web.db import session_scope from l4d2web.db import session_scope
from l4d2web.models import Blueprint as BlueprintModel from l4d2web.models import Blueprint as BlueprintModel
from l4d2web.models import BlueprintOverlay, Server from l4d2web.models import BlueprintOverlay, Overlay, Server
bp = Blueprint("blueprint", __name__) bp = Blueprint("blueprint", __name__)
@ -37,6 +37,19 @@ def replace_blueprint_overlays(db, blueprint_id: int, overlay_ids: list[int]) ->
db.add(BlueprintOverlay(blueprint_id=blueprint_id, overlay_id=overlay_id, position=position)) db.add(BlueprintOverlay(blueprint_id=blueprint_id, overlay_id=overlay_id, position=position))
def overlay_ids_authorized(db, overlay_ids: list[int], user_id: int) -> bool:
unique_ids = set(overlay_ids)
if not unique_ids:
return True
allowed_count = db.scalar(
select(func.count(Overlay.id)).where(
Overlay.id.in_(unique_ids),
Overlay.user_id.is_(None) | (Overlay.user_id == user_id),
)
)
return allowed_count == len(unique_ids)
@bp.post("/blueprints") @bp.post("/blueprints")
@require_login @require_login
def create_blueprint() -> Response: def create_blueprint() -> Response:
@ -61,6 +74,8 @@ def create_blueprint() -> Response:
return Response("name is required", status=400) return Response("name is required", status=400)
with session_scope() as db: with session_scope() as db:
if not overlay_ids_authorized(db, overlay_ids, user.id):
return Response("overlay not authorized", status=403)
blueprint = BlueprintModel(user_id=user.id, name=name, arguments=json.dumps(arguments), config=json.dumps(config)) blueprint = BlueprintModel(user_id=user.id, name=name, arguments=json.dumps(arguments), config=json.dumps(config))
db.add(blueprint) db.add(blueprint)
db.flush() db.flush()
@ -87,15 +102,39 @@ def update_blueprint_form(blueprint_id: int) -> Response:
) )
if blueprint is None: if blueprint is None:
return Response(status=404) return Response(status=404)
overlay_ids = ordered_overlay_ids_from_form()
if not overlay_ids_authorized(db, overlay_ids, user.id):
return Response("overlay not authorized", status=403)
blueprint.name = name blueprint.name = name
blueprint.arguments = json.dumps(split_textarea_lines(request.form.get("arguments", ""))) blueprint.arguments = json.dumps(split_textarea_lines(request.form.get("arguments", "")))
blueprint.config = json.dumps(split_textarea_lines(request.form.get("config", ""))) blueprint.config = json.dumps(split_textarea_lines(request.form.get("config", "")))
replace_blueprint_overlays(db, blueprint.id, ordered_overlay_ids_from_form()) replace_blueprint_overlays(db, blueprint.id, overlay_ids)
return redirect(f"/blueprints/{blueprint_id}") return redirect(f"/blueprints/{blueprint_id}")
def _delete_blueprint(db, user_id: int, blueprint_id: int) -> Response | None:
blueprint = db.scalar(
select(BlueprintModel).where(
BlueprintModel.id == blueprint_id,
BlueprintModel.user_id == user_id,
)
)
if blueprint is None:
return Response(status=404)
linked_count = db.scalar(
select(func.count(Server.id)).where(Server.blueprint_id == blueprint.id)
) or 0
if linked_count > 0:
return Response("blueprint is in use", status=409)
db.execute(delete(BlueprintOverlay).where(BlueprintOverlay.blueprint_id == blueprint.id))
db.delete(blueprint)
return None
@bp.delete("/blueprints/<int:blueprint_id>") @bp.delete("/blueprints/<int:blueprint_id>")
@require_login @require_login
def delete_blueprint(blueprint_id: int) -> Response: def delete_blueprint(blueprint_id: int) -> Response:
@ -103,22 +142,22 @@ def delete_blueprint(blueprint_id: int) -> Response:
assert user is not None assert user is not None
with session_scope() as db: with session_scope() as db:
blueprint = db.scalar( error = _delete_blueprint(db, user.id, blueprint_id)
select(BlueprintModel).where( if error is not None:
BlueprintModel.id == blueprint_id, return error
BlueprintModel.user_id == user.id,
)
)
if blueprint is None:
return Response(status=404)
linked_count = db.scalar(
select(func.count(Server.id)).where(Server.blueprint_id == blueprint.id)
) or 0
if linked_count > 0:
return Response("blueprint is in use", status=409)
db.execute(delete(BlueprintOverlay).where(BlueprintOverlay.blueprint_id == blueprint.id))
db.delete(blueprint)
return Response(status=204) return Response(status=204)
@bp.post("/blueprints/<int:blueprint_id>/delete")
@require_login
def delete_blueprint_form(blueprint_id: int) -> Response:
user = current_user()
assert user is not None
with session_scope() as db:
error = _delete_blueprint(db, user.id, blueprint_id)
if error is not None:
return error
return redirect("/blueprints")

View file

@ -22,7 +22,11 @@ def format_sse_event(seq: int, event: str, data: str) -> str:
def can_access_job(job: Job, user: User) -> bool: def can_access_job(job: Job, user: User) -> bool:
return user.admin or job.user_id == user.id if user.admin:
return True
if job.user_id is None:
return False
return job.user_id == user.id
@bp.get("/jobs/<int:job_id>") @bp.get("/jobs/<int:job_id>")
@ -34,7 +38,7 @@ def job_detail(job_id: int) -> str | Response:
with session_scope() as db: with session_scope() as db:
row = db.execute( row = db.execute(
select(Job, User, Server) select(Job, User, Server)
.join(User, User.id == Job.user_id) .outerjoin(User, User.id == Job.user_id)
.outerjoin(Server, Server.id == Job.server_id) .outerjoin(Server, Server.id == Job.server_id)
.where(Job.id == job_id) .where(Job.id == job_id)
).first() ).first()

View file

@ -28,6 +28,9 @@ def stream_server_logs(server_id: int) -> Response:
def generate(): def generate():
for line in facade.stream_server_logs(server.name, lines=200, follow=True): for line in facade.stream_server_logs(server.name, lines=200, follow=True):
yield f"data: {line}\n\n" if line == "":
yield ": keepalive\n\n"
else:
yield f"data: {line}\n\n"
return Response(generate(), mimetype="text/event-stream") return Response(generate(), mimetype="text/event-stream")

View file

@ -1,72 +1,211 @@
import shutil
from flask import Blueprint, Response, redirect, request from flask import Blueprint, Response, redirect, request
from sqlalchemy import select from sqlalchemy import select
from l4d2web.auth import require_admin from l4d2host.paths import get_left4me_root
from l4d2web.auth import current_user, require_login
from l4d2web.db import session_scope from l4d2web.db import session_scope
from l4d2web.models import BlueprintOverlay, Overlay from l4d2web.models import BlueprintOverlay, Job, Overlay
from l4d2web.services.security import validate_overlay_ref from l4d2web.services import overlay_builders
from l4d2web.services.job_worker import enqueue_build_overlay
from l4d2web.services.overlay_creation import (
create_overlay_directory,
generate_overlay_path,
)
CREATABLE_OVERLAY_TYPES = {"workshop", "script"}
WIPE_SCRIPT = "find /overlay -mindepth 1 -delete"
bp = Blueprint("overlay", __name__) bp = Blueprint("overlay", __name__)
@bp.post("/overlays") def _is_managed_path(overlay: Overlay) -> bool:
@require_admin return overlay.path == str(overlay.id)
def create_overlay() -> Response:
name = request.form.get("name", "").strip()
raw_path = request.form.get("path", "")
if not name or not raw_path:
return Response("missing fields", status=400)
try:
overlay_ref = validate_overlay_ref(raw_path) def _can_edit_overlay(overlay: Overlay, user) -> bool:
except ValueError as exc: if user is None:
return Response(str(exc), status=400) return False
if user.admin:
return True
if overlay.type in {"workshop", "script"}:
return overlay.user_id == user.id
return False
def _name_already_taken(db, name: str, scope_user_id: int | None, *, except_id: int | None = None) -> bool:
query = select(Overlay).where(Overlay.name == name)
if scope_user_id is None:
query = query.where(Overlay.user_id.is_(None))
else:
query = query.where(Overlay.user_id == scope_user_id)
if except_id is not None:
query = query.where(Overlay.id != except_id)
return db.scalar(query) is not None
@bp.post("/overlays")
@require_login
def create_overlay() -> Response:
user = current_user()
assert user is not None
name = request.form.get("name", "").strip()
overlay_type = request.form.get("type", "workshop").strip().lower()
system_wide = request.form.get("system_wide") == "1"
if not name:
return Response("missing fields", status=400)
if overlay_type not in CREATABLE_OVERLAY_TYPES:
return Response(f"unknown overlay type: {overlay_type}", status=400)
scope_user_id: int | None = None if (system_wide and user.admin) else user.id
with session_scope() as db: with session_scope() as db:
existing = db.scalar(select(Overlay).where(Overlay.name == name)) if _name_already_taken(db, name, scope_user_id):
if existing is not None:
return Response("overlay already exists", status=409) return Response("overlay already exists", status=409)
db.add(Overlay(name=name, path=overlay_ref))
return redirect("/overlays") overlay = Overlay(name=name, path="", type=overlay_type, user_id=scope_user_id)
db.add(overlay)
db.flush()
overlay.path = generate_overlay_path(overlay.id)
db.flush()
create_overlay_directory(overlay)
new_id = overlay.id
return redirect(f"/overlays/{new_id}")
@bp.post("/overlays/<int:overlay_id>") @bp.post("/overlays/<int:overlay_id>")
@require_admin @require_login
def update_overlay(overlay_id: int) -> Response: def update_overlay(overlay_id: int) -> Response:
name = request.form.get("name", "").strip() user = current_user()
raw_path = request.form.get("path", "") assert user is not None
if not name or not raw_path:
return Response("missing fields", status=400)
try: name = request.form.get("name", "").strip()
overlay_ref = validate_overlay_ref(raw_path) if not name:
except ValueError as exc: return Response("missing fields", status=400)
return Response(str(exc), status=400)
with session_scope() as db: with session_scope() as db:
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id)) overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
if overlay is None: if overlay is None:
return Response(status=404) return Response(status=404)
duplicate = db.scalar(select(Overlay).where(Overlay.name == name, Overlay.id != overlay_id)) if not _can_edit_overlay(overlay, user):
if duplicate is not None: return Response(status=403)
if _name_already_taken(db, name, overlay.user_id, except_id=overlay_id):
return Response("overlay already exists", status=409) return Response("overlay already exists", status=409)
overlay.name = name overlay.name = name
overlay.path = overlay_ref
return redirect("/overlays") return redirect(f"/overlays/{overlay_id}")
@bp.post("/overlays/<int:overlay_id>/delete") @bp.post("/overlays/<int:overlay_id>/delete")
@require_admin @require_login
def delete_overlay(overlay_id: int) -> Response: def delete_overlay(overlay_id: int) -> Response:
user = current_user()
assert user is not None
with session_scope() as db: with session_scope() as db:
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id)) overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
if overlay is None: if overlay is None:
return Response(status=404) return Response(status=404)
if not _can_edit_overlay(overlay, user):
return Response(status=403)
in_use = db.scalar(select(BlueprintOverlay).where(BlueprintOverlay.overlay_id == overlay_id)) in_use = db.scalar(select(BlueprintOverlay).where(BlueprintOverlay.overlay_id == overlay_id))
if in_use is not None: if in_use is not None:
return Response("overlay is in use", status=409) return Response("overlay is in use", status=409)
path_value = overlay.path
path_is_managed = _is_managed_path(overlay)
db.delete(overlay) db.delete(overlay)
if path_is_managed and path_value:
target = get_left4me_root() / "overlays" / path_value
if target.exists():
shutil.rmtree(target)
return redirect("/overlays") return redirect("/overlays")
def _load_script_overlay(db, overlay_id: int, user) -> tuple[Overlay | None, Response | None]:
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
if overlay is None:
return None, Response(status=404)
if overlay.type != "script":
return None, Response("not a script overlay", status=400)
if not _can_edit_overlay(overlay, user):
return None, Response(status=403)
return overlay, None
@bp.post("/overlays/<int:overlay_id>/script")
@require_login
def update_script(overlay_id: int) -> Response:
user = current_user()
assert user is not None
# HTML form submission of <textarea> uses CRLF line endings per spec; bash
# treats the trailing \r as part of each argument and breaks every command.
# Normalize to LF before storage so the script is well-formed when written
# to the sandbox tmpfile.
script_text = request.form.get("script", "").replace("\r\n", "\n").replace("\r", "\n")
with session_scope() as db:
overlay, err = _load_script_overlay(db, overlay_id, user)
if err is not None:
return err
overlay.script = script_text
job = enqueue_build_overlay(db, overlay_id=overlay_id, user_id=user.id)
job_id = job.id
return redirect(f"/jobs/{job_id}")
@bp.post("/overlays/<int:overlay_id>/build")
@require_login
def manual_build(overlay_id: int) -> Response:
user = current_user()
assert user is not None
with session_scope() as db:
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
if overlay is None:
return Response(status=404)
if not _can_edit_overlay(overlay, user):
return Response(status=403)
job = enqueue_build_overlay(db, overlay_id=overlay_id, user_id=user.id)
job_id = job.id
return redirect(f"/jobs/{job_id}")
@bp.post("/overlays/<int:overlay_id>/wipe")
@require_login
def wipe_overlay(overlay_id: int) -> Response:
user = current_user()
assert user is not None
with session_scope() as db:
overlay, err = _load_script_overlay(db, overlay_id, user)
if err is not None:
return err
running = db.scalar(
select(Job).where(
Job.operation == "build_overlay",
Job.overlay_id == overlay_id,
Job.state.in_({"running", "cancelling"}),
)
)
if running is not None:
return Response("build in progress for this overlay", status=409)
overlay_builders.run_sandboxed_script(
overlay_id,
WIPE_SCRIPT,
on_stdout=lambda _line: None,
on_stderr=lambda _line: None,
should_cancel=lambda: False,
)
with session_scope() as db:
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
if overlay is not None:
overlay.last_build_status = ""
return redirect(f"/overlays/{overlay_id}")

View file

@ -1,12 +1,20 @@
import json import json
from flask import Blueprint, Response, redirect, render_template from flask import Blueprint, Response, redirect, render_template, request
from sqlalchemy import select from sqlalchemy import select
from l4d2web.auth import current_user, require_admin, require_login from l4d2web.auth import current_user, require_admin, require_login
from l4d2web.db import session_scope from l4d2web.db import session_scope
from l4d2web.models import Blueprint as BlueprintModel from l4d2web.models import Blueprint as BlueprintModel
from l4d2web.models import BlueprintOverlay, Job, Overlay, Server, User from l4d2web.models import (
BlueprintOverlay,
Job,
Overlay,
OverlayWorkshopItem,
Server,
User,
WorkshopItem,
)
bp = Blueprint("pages", __name__) bp = Blueprint("pages", __name__)
@ -48,7 +56,7 @@ def admin_jobs() -> str:
with session_scope() as db: with session_scope() as db:
rows = db.execute( rows = db.execute(
select(Job, User, Server) select(Job, User, Server)
.join(User, User.id == Job.user_id) .outerjoin(User, User.id == Job.user_id)
.outerjoin(Server, Server.id == Job.server_id) .outerjoin(Server, Server.id == Job.server_id)
.order_by(Job.created_at.desc()) .order_by(Job.created_at.desc())
).all() ).all()
@ -70,7 +78,23 @@ def servers_page() -> str:
blueprints = db.scalars( blueprints = db.scalars(
select(BlueprintModel).where(BlueprintModel.user_id == user.id).order_by(BlueprintModel.name) select(BlueprintModel).where(BlueprintModel.user_id == user.id).order_by(BlueprintModel.name)
).all() ).all()
return render_template("servers.html", rows=rows, blueprints=blueprints)
prefill_blueprint_id: int | None = None
raw_prefill = request.args.get("blueprint_id")
if raw_prefill:
try:
candidate = int(raw_prefill)
except ValueError:
candidate = None
if candidate is not None and any(b.id == candidate for b in blueprints):
prefill_blueprint_id = candidate
return render_template(
"servers.html",
rows=rows,
blueprints=blueprints,
prefill_blueprint_id=prefill_blueprint_id,
)
@bp.get("/servers/<int:server_id>") @bp.get("/servers/<int:server_id>")
@ -86,7 +110,7 @@ def server_detail(server_id: int):
blueprint = db.scalar(select(BlueprintModel).where(BlueprintModel.id == server.blueprint_id)) blueprint = db.scalar(select(BlueprintModel).where(BlueprintModel.id == server.blueprint_id))
recent_job_rows = db.execute( recent_job_rows = db.execute(
select(Job, User, Server) select(Job, User, Server)
.join(User, User.id == Job.user_id) .outerjoin(User, User.id == Job.user_id)
.outerjoin(Server, Server.id == Job.server_id) .outerjoin(Server, Server.id == Job.server_id)
.where(Job.server_id == server.id) .where(Job.server_id == server.id)
.order_by(Job.created_at.desc()) .order_by(Job.created_at.desc())
@ -113,7 +137,7 @@ def server_jobs_page(server_id: int):
return Response(status=404) return Response(status=404)
rows = db.execute( rows = db.execute(
select(Job, User, Server) select(Job, User, Server)
.join(User, User.id == Job.user_id) .outerjoin(User, User.id == Job.user_id)
.outerjoin(Server, Server.id == Job.server_id) .outerjoin(Server, Server.id == Job.server_id)
.where(Job.server_id == server.id) .where(Job.server_id == server.id)
.order_by(Job.created_at.desc()) .order_by(Job.created_at.desc())
@ -125,11 +149,85 @@ def server_jobs_page(server_id: int):
@bp.get("/overlays") @bp.get("/overlays")
@require_login @require_login
def overlays() -> str: def overlays() -> str:
user = current_user()
assert user is not None
with session_scope() as db: with session_scope() as db:
overlays = db.scalars(select(Overlay).order_by(Overlay.name)).all() query = select(Overlay).order_by(Overlay.name)
if not user.admin:
query = query.where(
Overlay.user_id.is_(None) | (Overlay.user_id == user.id)
)
overlays = db.scalars(query).all()
return render_template("overlays.html", overlays=overlays) return render_template("overlays.html", overlays=overlays)
@bp.get("/overlays/<int:overlay_id>/jobs")
@require_login
def overlay_jobs_page(overlay_id: int):
user = current_user()
assert user is not None
with session_scope() as db:
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
if overlay is None:
return Response(status=404)
if not user.admin and overlay.user_id is not None and overlay.user_id != user.id:
return Response(status=403)
rows = db.execute(
select(Job, User, Server)
.outerjoin(User, User.id == Job.user_id)
.outerjoin(Server, Server.id == Job.server_id)
.where(Job.operation == "build_overlay", Job.overlay_id == overlay.id)
.order_by(Job.created_at.desc())
).all()
return render_template("overlay_jobs.html", overlay=overlay, rows=rows)
@bp.get("/overlays/<int:overlay_id>")
@require_login
def overlay_detail(overlay_id: int):
user = current_user()
assert user is not None
with session_scope() as db:
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
if overlay is None:
return Response(status=404)
if not user.admin and overlay.user_id is not None and overlay.user_id != user.id:
return Response(status=403)
using_blueprints_query = (
select(BlueprintModel)
.join(BlueprintOverlay, BlueprintOverlay.blueprint_id == BlueprintModel.id)
.where(BlueprintOverlay.overlay_id == overlay.id)
.order_by(BlueprintModel.name)
)
if not user.admin:
using_blueprints_query = using_blueprints_query.where(BlueprintModel.user_id == user.id)
using_blueprints = db.scalars(using_blueprints_query).all()
workshop_items = []
if overlay.type == "workshop":
workshop_items = db.scalars(
select(WorkshopItem)
.join(
OverlayWorkshopItem,
OverlayWorkshopItem.workshop_item_id == WorkshopItem.id,
)
.where(OverlayWorkshopItem.overlay_id == overlay.id)
.order_by(WorkshopItem.created_at)
).all()
latest_build_job = db.scalar(
select(Job)
.where(Job.operation == "build_overlay", Job.overlay_id == overlay.id)
.order_by(Job.created_at.desc())
.limit(1)
)
return render_template(
"overlay_detail.html",
overlay=overlay,
using_blueprints=using_blueprints,
workshop_items=workshop_items,
latest_build_job=latest_build_job,
)
@bp.get("/blueprints") @bp.get("/blueprints")
@require_login @require_login
def blueprints_page() -> str: def blueprints_page() -> str:
@ -165,7 +263,11 @@ def blueprint_page(blueprint_id: int):
select(BlueprintOverlay.overlay_id, BlueprintOverlay.position) select(BlueprintOverlay.overlay_id, BlueprintOverlay.position)
.where(BlueprintOverlay.blueprint_id == blueprint.id) .where(BlueprintOverlay.blueprint_id == blueprint.id)
).all() ).all()
all_overlays = db.scalars(select(Overlay).order_by(Overlay.name)).all() all_overlays = db.scalars(
select(Overlay)
.where(Overlay.user_id.is_(None) | (Overlay.user_id == user.id))
.order_by(Overlay.name)
).all()
overlay_positions = {overlay_id: position + 1 for overlay_id, position in position_rows} overlay_positions = {overlay_id: position + 1 for overlay_id, position in position_rows}
return render_template( return render_template(

View file

@ -1,4 +1,4 @@
from flask import Blueprint, Response, jsonify, redirect, request from flask import Blueprint, Response, current_app, jsonify, redirect, request
from sqlalchemy import select from sqlalchemy import select
from sqlalchemy.exc import IntegrityError from sqlalchemy.exc import IntegrityError
@ -12,6 +12,36 @@ from l4d2web.services.security import validate_instance_name
bp = Blueprint("server", __name__) bp = Blueprint("server", __name__)
def _allocate_next_port(db) -> int | None:
start = int(current_app.config["PORT_RANGE_START"])
end = int(current_app.config["PORT_RANGE_END"])
used = set(
db.scalars(
select(Server.port).where(Server.port >= start, Server.port <= end)
).all()
)
for port in range(start, end + 1):
if port not in used:
return port
return None
def _resolve_port(payload, db) -> tuple[int | None, Response | None]:
raw = payload.get("port")
if raw is None or (isinstance(raw, str) and raw.strip() == ""):
port = _allocate_next_port(db)
if port is None:
return None, Response("no free port available", status=409)
return port, None
try:
port = int(raw)
except (TypeError, ValueError):
return None, Response("invalid port", status=400)
if not 1 <= port <= 65535:
return None, Response("invalid port", status=400)
return port, None
@bp.post("/servers") @bp.post("/servers")
@require_login @require_login
def create_server() -> Response: def create_server() -> Response:
@ -35,23 +65,27 @@ def create_server() -> Response:
if blueprint is None: if blueprint is None:
return Response("blueprint not found", status=404) return Response("blueprint not found", status=404)
port, error = _resolve_port(payload, db)
if error is not None:
return error
server = Server( server = Server(
user_id=user.id, user_id=user.id,
blueprint_id=blueprint.id, blueprint_id=blueprint.id,
name=name, name=name,
port=int(payload["port"]), port=port,
desired_state="stopped", desired_state="stopped",
actual_state="unknown", actual_state="unknown",
last_error="", last_error="",
) )
db.add(server) db.add(server)
try: try:
db.flush() db.flush()
except IntegrityError: except IntegrityError:
db.rollback() db.rollback()
return Response("port already in use", status=409) return Response("port already in use", status=409)
server_id = server.id server_id = server.id
if json_response: if json_response:
@ -85,7 +119,7 @@ def update_server(server_id: int) -> Response:
return jsonify({"id": server_id}), 200 return jsonify({"id": server_id}), 200
LIFECYCLE_OPERATIONS = {"initialize", "start", "stop", "delete"} LIFECYCLE_OPERATIONS = {"initialize", "start", "stop", "delete", "reset"}
@bp.post("/servers/<int:server_id>/<operation>") @bp.post("/servers/<int:server_id>/<operation>")
@ -103,7 +137,9 @@ def enqueue_server_operation(server_id: int, operation: str) -> Response:
db.add(Job(user_id=user.id, server_id=server.id, operation=operation, state="queued")) db.add(Job(user_id=user.id, server_id=server.id, operation=operation, state="queued"))
if operation == "start": if operation == "start":
server.desired_state = "running" server.desired_state = "running"
if operation in {"stop", "delete"}: if operation in {"stop", "delete", "reset"}:
server.desired_state = "stopped" server.desired_state = "stopped"
if operation == "delete":
return redirect("/servers")
return redirect(f"/servers/{server_id}") return redirect(f"/servers/{server_id}")

View file

@ -0,0 +1,137 @@
"""Routes for the workshop overlay type (add/remove items, manual rebuild,
admin global refresh)."""
from __future__ import annotations
from flask import Blueprint, Response, redirect, request
from sqlalchemy import delete as sa_delete
from sqlalchemy import select
from l4d2web.auth import current_user, require_admin, require_login
from l4d2web.db import session_scope
from l4d2web.models import (
Job,
Overlay,
OverlayWorkshopItem,
WorkshopItem,
)
from l4d2web.services import steam_workshop
from l4d2web.services.job_worker import enqueue_build_overlay
bp = Blueprint("workshop", __name__)
def _check_workshop_overlay_access(overlay_id: int, user, db):
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
if overlay is None:
return None, Response(status=404)
if overlay.type != "workshop":
return None, Response("not a workshop overlay", status=400)
if overlay.user_id != user.id and not user.admin:
return None, Response(status=403)
return overlay, None
@bp.post("/overlays/<int:overlay_id>/items")
@require_login
def add_items(overlay_id: int) -> Response:
user = current_user()
assert user is not None
raw_input = request.form.get("input", "").strip()
mode = request.form.get("input_mode", "items")
if not raw_input:
return Response("missing input", status=400)
try:
ids = steam_workshop.parse_workshop_input(raw_input)
except ValueError as exc:
return Response(str(exc), status=400)
if mode == "collection":
if len(ids) != 1:
return Response("collection mode expects exactly one id or url", status=400)
try:
ids = steam_workshop.resolve_collection(ids[0])
except Exception as exc:
return Response(f"failed to resolve collection: {exc}", status=502)
if not ids:
return Response("collection has no items", status=400)
try:
metas = steam_workshop.fetch_metadata_batch(ids, mode="add")
except steam_workshop.WorkshopValidationError as exc:
return Response(str(exc), status=400)
except Exception as exc:
return Response(f"steam api error: {exc}", status=502)
with session_scope() as db:
overlay, err = _check_workshop_overlay_access(overlay_id, user, db)
if err is not None:
return err
for meta in metas:
wi = db.scalar(select(WorkshopItem).where(WorkshopItem.steam_id == meta.steam_id))
if wi is None:
wi = WorkshopItem(steam_id=meta.steam_id)
db.add(wi)
wi.title = meta.title
wi.filename = meta.filename
wi.file_url = meta.file_url
wi.file_size = meta.file_size
wi.time_updated = meta.time_updated
wi.preview_url = meta.preview_url
wi.last_error = "" if meta.result == 1 else f"steam result {meta.result}"
db.flush()
existing = db.scalar(
select(OverlayWorkshopItem).where(
OverlayWorkshopItem.overlay_id == overlay_id,
OverlayWorkshopItem.workshop_item_id == wi.id,
)
)
if existing is None:
db.add(OverlayWorkshopItem(overlay_id=overlay_id, workshop_item_id=wi.id))
job = enqueue_build_overlay(db, overlay_id=overlay_id, user_id=user.id)
job_id = job.id
return redirect(f"/jobs/{job_id}")
@bp.post("/overlays/<int:overlay_id>/items/<int:item_id>/delete")
@require_login
def remove_item(overlay_id: int, item_id: int) -> Response:
user = current_user()
assert user is not None
with session_scope() as db:
overlay, err = _check_workshop_overlay_access(overlay_id, user, db)
if err is not None:
return err
result = db.execute(
sa_delete(OverlayWorkshopItem).where(
OverlayWorkshopItem.overlay_id == overlay_id,
OverlayWorkshopItem.workshop_item_id == item_id,
)
)
if result.rowcount == 0:
return Response(status=404)
job = enqueue_build_overlay(db, overlay_id=overlay_id, user_id=user.id)
job_id = job.id
return redirect(f"/jobs/{job_id}")
@bp.post("/admin/workshop/refresh")
@require_admin
def admin_refresh() -> Response:
user = current_user()
assert user is not None
with session_scope() as db:
db.add(
Job(
user_id=user.id,
server_id=None,
operation="refresh_workshop_items",
state="queued",
)
)
return redirect("/admin/jobs")

View file

@ -1,5 +1,6 @@
from dataclasses import dataclass from dataclasses import dataclass
import os import os
import select
import signal import signal
import subprocess import subprocess
import sys import sys
@ -143,19 +144,37 @@ def run_command(
return result return result
def stream_command(cmd: Sequence[str]) -> Iterator[str]: def stream_command(cmd: Sequence[str], *, heartbeat_interval: float = 15.0) -> Iterator[str]:
# An empty string yielded between real lines is a heartbeat tick: it lets
# SSE callers emit a keepalive frame so a closed peer is detected, instead
# of blocking forever inside readline() when the child is silent.
proc = subprocess.Popen( proc = subprocess.Popen(
list(cmd), list(cmd),
stdout=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, stderr=subprocess.PIPE,
text=True, bufsize=0,
bufsize=1,
) )
try: try:
if proc.stdout is None: if proc.stdout is None:
return return
for raw in iter(proc.stdout.readline, ""): fd = proc.stdout.fileno()
yield raw.rstrip("\n") buffer = b""
while True:
ready, _, _ = select.select([fd], [], [], heartbeat_interval)
if not ready:
if proc.poll() is not None:
break
yield ""
continue
chunk = os.read(fd, 4096)
if not chunk:
break
buffer += chunk
while b"\n" in buffer:
line, buffer = buffer.split(b"\n", 1)
yield line.decode("utf-8", errors="replace")
if buffer:
yield buffer.decode("utf-8", errors="replace")
finally: finally:
if proc.poll() is None: if proc.poll() is None:
proc.terminate() proc.terminate()

View file

@ -10,13 +10,25 @@ from sqlalchemy.orm import Session
from typing import Callable from typing import Callable
from l4d2web.db import session_scope from l4d2web.db import session_scope
from l4d2web.models import Job, JobLog, Server from l4d2web.models import (
Blueprint,
BlueprintOverlay,
Job,
JobLog,
Overlay,
OverlayWorkshopItem,
Server,
WorkshopItem,
)
from l4d2web.services.host_commands import CommandCancelledError from l4d2web.services.host_commands import CommandCancelledError
TERMINAL_JOB_STATES = {"succeeded", "failed", "cancelled"} TERMINAL_JOB_STATES = {"succeeded", "failed", "cancelled"}
ACTIVE_JOB_STATES = {"running", "cancelling"} ACTIVE_JOB_STATES = {"running", "cancelling"}
SERVER_OPERATIONS = {"initialize", "start", "stop", "delete"} SERVER_OPERATIONS = {"initialize", "start", "stop", "delete", "reset"}
OVERLAY_OPERATIONS = {"build_overlay"}
GLOBAL_OPERATIONS = {"install", "refresh_workshop_items"}
WORKSHOP_REFRESH_DOWNLOAD_WORKERS = 1
_claim_lock = threading.Lock() _claim_lock = threading.Lock()
_log_lock = threading.RLock() _log_lock = threading.RLock()
@ -27,17 +39,55 @@ _workers_started = False
@dataclass @dataclass
class SchedulerState: class SchedulerState:
install_running: bool = False install_running: bool = False
refresh_running: bool = False
running_servers: set[int] = field(default_factory=set) running_servers: set[int] = field(default_factory=set)
running_overlays: set[int] = field(default_factory=set)
blocked_servers_by_overlay: set[int] = field(default_factory=set)
def can_start(job, state: SchedulerState) -> bool: def can_start(job, state: SchedulerState) -> bool:
"""Truth table for the worker's claim policy.
install / refresh_workshop_items are global mutexes they block each
other, all build_overlay jobs, and all server jobs.
build_overlay(overlay_id=N) is per-overlay: blocks on install/refresh, and
on another build for the same overlay. Different overlays may build
concurrently.
Server start/init blocks on install/refresh and on a build_overlay for any
overlay referenced by the server's blueprint.
"""
if job.operation == "install": if job.operation == "install":
return (not state.install_running) and (len(state.running_servers) == 0) return (
if state.install_running: not state.install_running
and not state.refresh_running
and len(state.running_servers) == 0
and len(state.running_overlays) == 0
)
if job.operation == "refresh_workshop_items":
return (
not state.install_running
and not state.refresh_running
and len(state.running_servers) == 0
and len(state.running_overlays) == 0
)
if job.operation == "build_overlay":
if state.install_running or state.refresh_running:
return False
if job.overlay_id is None:
return False
return job.overlay_id not in state.running_overlays
# Server operations from here on.
if state.install_running or state.refresh_running:
return False return False
if job.server_id is None: if job.server_id is None:
return False return False
return job.server_id not in state.running_servers if job.server_id in state.running_servers:
return False
if job.server_id in state.blocked_servers_by_overlay:
return False
return True
def build_scheduler_state(session: Session) -> SchedulerState: def build_scheduler_state(session: Session) -> SchedulerState:
@ -46,8 +96,22 @@ def build_scheduler_state(session: Session) -> SchedulerState:
for job in running_jobs: for job in running_jobs:
if job.operation == "install": if job.operation == "install":
state.install_running = True state.install_running = True
elif job.operation == "refresh_workshop_items":
state.refresh_running = True
elif job.operation == "build_overlay" and job.overlay_id is not None:
state.running_overlays.add(job.overlay_id)
elif job.server_id is not None: elif job.server_id is not None:
state.running_servers.add(job.server_id) state.running_servers.add(job.server_id)
if state.running_overlays:
rows = session.execute(
select(Server.id)
.join(Blueprint, Blueprint.id == Server.blueprint_id)
.join(BlueprintOverlay, BlueprintOverlay.blueprint_id == Blueprint.id)
.where(BlueprintOverlay.overlay_id.in_(state.running_overlays))
).all()
state.blocked_servers_by_overlay = {row[0] for row in rows}
return state return state
@ -65,8 +129,22 @@ def claim_next_job() -> int | None:
jobs = db.scalars(select(Job).where(Job.state == "queued").order_by(Job.created_at, Job.id)).all() jobs = db.scalars(select(Job).where(Job.state == "queued").order_by(Job.created_at, Job.id)).all()
now = datetime.now(UTC) now = datetime.now(UTC)
for job in jobs: for job in jobs:
malformed_server_job = job.operation != "install" and job.server_id is None malformed_server_job = (
if not malformed_server_job and not can_start(job, state): job.operation in SERVER_OPERATIONS and job.server_id is None
)
malformed_overlay_job = (
job.operation in OVERLAY_OPERATIONS and job.overlay_id is None
)
if malformed_server_job or malformed_overlay_job:
# Mark malformed jobs failed immediately so the scheduler can move on.
job.state = "failed"
job.exit_code = 1
job.started_at = now
job.finished_at = now
job.updated_at = now
db.flush()
continue
if not can_start(job, state):
continue continue
job.state = "running" job.state = "running"
@ -78,6 +156,31 @@ def claim_next_job() -> int | None:
return None return None
def enqueue_build_overlay(session: Session, *, overlay_id: int, user_id: int) -> Job:
"""Insert a `build_overlay` job, coalescing against any already-queued
(not-yet-running) build for the same overlay. Running jobs are NOT
coalesced a fresh add after a build started gets its own job."""
existing = session.scalar(
select(Job).where(
Job.operation == "build_overlay",
Job.overlay_id == overlay_id,
Job.state == "queued",
)
)
if existing is not None:
return existing
job = Job(
user_id=user_id,
server_id=None,
overlay_id=overlay_id,
operation="build_overlay",
state="queued",
)
session.add(job)
session.flush()
return job
def run_worker_once() -> bool: def run_worker_once() -> bool:
job_id = claim_next_job() job_id = claim_next_job()
if job_id is None: if job_id is None:
@ -90,12 +193,14 @@ def run_job(job_id: int) -> None:
from l4d2web.services import l4d2_facade from l4d2web.services import l4d2_facade
server_name = "unknown" server_name = "unknown"
overlay_id_for_job: int | None = None
with session_scope() as db: with session_scope() as db:
job = db.scalar(select(Job).where(Job.id == job_id)) job = db.scalar(select(Job).where(Job.id == job_id))
if job is None: if job is None:
return return
operation = job.operation operation = job.operation
server_id = job.server_id server_id = job.server_id
overlay_id_for_job = job.overlay_id
if server_id is not None: if server_id is not None:
server = db.scalar(select(Server).where(Server.id == server_id)) server = db.scalar(select(Server).where(Server.id == server_id))
if server is not None: if server is not None:
@ -133,6 +238,27 @@ def run_job(job_id: int) -> None:
on_stderr=on_stderr, on_stderr=on_stderr,
should_cancel=should_cancel, should_cancel=should_cancel,
) )
elif operation == "refresh_workshop_items":
_run_with_boundaries(
"refresh",
"workshop items",
_run_refresh_workshop_items,
on_stdout=on_stdout,
on_stderr=on_stderr,
should_cancel=should_cancel,
)
elif operation == "build_overlay":
if overlay_id_for_job is None:
raise ValueError("build_overlay job has no overlay_id")
_run_with_boundaries(
"build",
f"overlay {overlay_id_for_job}",
_run_build_overlay,
overlay_id_for_job,
on_stdout=on_stdout,
on_stderr=on_stderr,
should_cancel=should_cancel,
)
elif operation in SERVER_OPERATIONS and server_id is None: elif operation in SERVER_OPERATIONS and server_id is None:
raise ValueError(f"{operation} job has no server_id") raise ValueError(f"{operation} job has no server_id")
elif operation == "initialize": elif operation == "initialize":
@ -185,6 +311,25 @@ def run_job(job_id: int) -> None:
on_stderr=on_stderr, on_stderr=on_stderr,
should_cancel=should_cancel, should_cancel=should_cancel,
) )
# Host-side cleanup succeeded; remove the DB row so the server
# disappears from /servers. Status refresh is skipped — the
# systemd unit is gone and querying it would just log noise.
with session_scope() as db:
server = db.scalar(select(Server).where(Server.id == server_id))
if server is not None:
db.delete(server)
finish_job(job_id, "succeeded", 0)
return
elif operation == "reset":
_run_with_boundaries(
"reset",
server_name,
l4d2_facade.reset_server,
server_id,
on_stdout=on_stdout,
on_stderr=on_stderr,
should_cancel=should_cancel,
)
else: else:
raise ValueError(f"unknown job operation: {operation}") raise ValueError(f"unknown job operation: {operation}")
@ -213,6 +358,155 @@ def run_job(job_id: int) -> None:
finish_job(job_id, "failed", 1, error=error) finish_job(job_id, "failed", 1, error=error)
def _run_build_overlay(
overlay_id: int,
*,
on_stdout: Callable[[str], None],
on_stderr: Callable[[str], None],
should_cancel: Callable[[], bool],
) -> None:
"""Dispatch a build_overlay job through the builder registry."""
from l4d2web.services.overlay_builders import BUILDERS
with session_scope() as db:
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
if overlay is None:
raise ValueError(f"overlay {overlay_id} not found")
builder = BUILDERS.get(overlay.type)
if builder is None:
raise ValueError(f"no builder registered for overlay type {overlay.type!r}")
# Detach overlay before leaving the session so the builder can read its
# attributes without a stale connection.
db.expunge(overlay)
builder.build(
overlay,
on_stdout=on_stdout,
on_stderr=on_stderr,
should_cancel=should_cancel,
)
def _run_refresh_workshop_items(
*,
on_stdout: Callable[[str], None],
on_stderr: Callable[[str], None],
should_cancel: Callable[[], bool],
) -> list[int]:
"""Refresh metadata for every WorkshopItem, redownload changed items, and
enqueue (coalesced) build_overlay jobs for any overlay whose items had
`time_updated` advance or `filename` change. Returns the affected
overlay_ids for testability."""
from l4d2web.services import steam_workshop
from l4d2web.services.workshop_paths import workshop_cache_root
# Snapshot all WorkshopItems for the metadata batch.
with session_scope() as db:
items = db.scalars(select(WorkshopItem)).all()
snapshot = [
(it.id, it.steam_id, it.time_updated, it.filename) for it in items
]
if not snapshot:
on_stdout("no workshop items registered; nothing to refresh")
return []
steam_ids = [s for _, s, _, _ in snapshot]
on_stdout(f"fetching metadata for {len(steam_ids)} items")
metas = steam_workshop.fetch_metadata_batch(steam_ids, mode="refresh")
metas_by_id = {m.steam_id: m for m in metas}
on_stdout(f"metadata phase complete (received {len(metas)} entries)")
if should_cancel():
on_stderr("refresh cancelled after metadata phase")
return []
# Update DB rows + collect items that need (re)download.
affected_workshop_item_ids: set[int] = set()
download_metas: list[steam_workshop.WorkshopMetadata] = []
with session_scope() as db:
for item_id, steam_id, prior_time_updated, prior_filename in snapshot:
meta = metas_by_id.get(steam_id)
wi = db.scalar(select(WorkshopItem).where(WorkshopItem.id == item_id))
if wi is None:
continue
if meta is None:
# Steam dropped this item from the response.
wi.last_error = "steam returned no entry for this item"
continue
wi.title = meta.title
wi.filename = meta.filename
wi.file_url = meta.file_url
wi.file_size = meta.file_size
wi.preview_url = meta.preview_url
wi.last_error = "" if meta.result == 1 else f"steam result {meta.result}"
if meta.result != 1 or not meta.file_url:
continue
if (
meta.time_updated > prior_time_updated
or meta.filename != prior_filename
or wi.last_downloaded_at is None
):
wi.time_updated = meta.time_updated
affected_workshop_item_ids.add(item_id)
download_metas.append(meta)
on_stdout(f"downloading {len(download_metas)} items")
if download_metas:
report = steam_workshop.refresh_all(
download_metas,
workshop_cache_root(),
executor_workers=WORKSHOP_REFRESH_DOWNLOAD_WORKERS,
should_cancel=should_cancel,
)
on_stdout(
f"download phase complete (downloaded={report.downloaded} errors={report.errors})"
)
if report.errors:
for steam_id, err in report.per_item_errors.items():
on_stderr(f"download {steam_id}: {err}")
# Mark successfully downloaded items.
with session_scope() as db:
for meta in download_metas:
if meta.steam_id in report.per_item_errors:
continue
wi = db.scalar(select(WorkshopItem).where(WorkshopItem.steam_id == meta.steam_id))
if wi is not None:
wi.last_downloaded_at = datetime.now(UTC)
# Enqueue (coalesced) build_overlay for affected overlays.
if not affected_workshop_item_ids:
on_stdout("no overlays needed rebuilding")
return []
with session_scope() as db:
overlay_rows = db.execute(
select(OverlayWorkshopItem.overlay_id)
.where(OverlayWorkshopItem.workshop_item_id.in_(affected_workshop_item_ids))
.distinct()
).all()
affected_overlay_ids = [row[0] for row in overlay_rows]
for ov_id in affected_overlay_ids:
# Find a sensible owner for the auto-enqueued job: the overlay's
# user_id if private, else any admin (best effort) — fall back to
# the most recent existing job's user_id.
overlay = db.scalar(select(Overlay).where(Overlay.id == ov_id))
if overlay is None:
continue
user_id = overlay.user_id
if user_id is None:
# System overlay — pick any admin user; fall back to first user.
user_id = db.scalar(
select(Job.user_id).order_by(Job.created_at.desc()).limit(1)
)
if user_id is None:
continue
enqueue_build_overlay(db, overlay_id=ov_id, user_id=user_id)
on_stdout(f"enqueued build_overlay for {len(affected_overlay_ids)} overlay(s)")
return list(affected_overlay_ids)
def finish_job(job_id: int, state: str, exit_code: int | None, error: str = "") -> None: def finish_job(job_id: int, state: str, exit_code: int | None, error: str = "") -> None:
now = datetime.now(UTC) now = datetime.now(UTC)
with session_scope() as db: with session_scope() as db:
@ -228,6 +522,11 @@ def finish_job(job_id: int, state: str, exit_code: int | None, error: str = "")
if server is not None: if server is not None:
server.last_error = "" if state == "succeeded" else error server.last_error = "" if state == "succeeded" else error
server.updated_at = now server.updated_at = now
if job.operation == "build_overlay" and job.overlay_id is not None:
overlay = db.scalar(select(Overlay).where(Overlay.id == job.overlay_id))
if overlay is not None:
overlay.last_build_status = "ok" if state == "succeeded" else "failed"
overlay.updated_at = now
def append_job_log_line(job_id: int, stream: str, line: str, max_chars: int = 4096) -> int: def append_job_log_line(job_id: int, stream: str, line: str, max_chars: int = 4096) -> int:

View file

@ -5,9 +5,17 @@ from pathlib import Path
from sqlalchemy import select from sqlalchemy import select
from l4d2web.db import session_scope from l4d2web.db import session_scope
from l4d2web.models import Blueprint, BlueprintOverlay, Overlay, Server from l4d2web.models import (
Blueprint,
BlueprintOverlay,
Overlay,
OverlayWorkshopItem,
Server,
WorkshopItem,
)
from l4d2web.services import host_commands from l4d2web.services import host_commands
from l4d2web.services.spec_yaml import write_temp_spec from l4d2web.services.spec_yaml import write_temp_spec
from l4d2web.services.workshop_paths import cache_path
@dataclass(slots=True) @dataclass(slots=True)
@ -57,6 +65,21 @@ def install_runtime(on_stdout=None, on_stderr=None, should_cancel=None) -> None:
def initialize_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None: def initialize_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
server, blueprint, overlay_refs = load_server_blueprint_bundle(server_id) server, blueprint, overlay_refs = load_server_blueprint_bundle(server_id)
# Run each overlay's builder synchronously so symlinks/dirs are present
# before l4d2ctl initialize composes the lowerdirs.
_run_blueprint_builders(
blueprint_id=blueprint.id,
on_stdout=on_stdout,
on_stderr=on_stderr,
should_cancel=should_cancel,
)
# Workshop overlays may have items not yet downloaded. The builders skip
# them, but we don't want to mount a partial overlay silently — fail
# loudly with the missing IDs.
_check_workshop_overlay_caches(blueprint_id=blueprint.id)
spec_path = write_temp_spec(build_server_spec_payload(server, blueprint, overlay_refs)) spec_path = write_temp_spec(build_server_spec_payload(server, blueprint, overlay_refs))
try: try:
host_commands.run_command( host_commands.run_command(
@ -69,6 +92,87 @@ def initialize_server(server_id: int, on_stdout=None, on_stderr=None, should_can
spec_path.unlink(missing_ok=True) spec_path.unlink(missing_ok=True)
def _run_blueprint_builders(
*,
blueprint_id: int,
on_stdout=None,
on_stderr=None,
should_cancel=None,
) -> None:
"""Synchronously invoke each overlay's builder for the given blueprint."""
from l4d2web.services.overlay_builders import BUILDERS
with session_scope() as db:
overlays = db.scalars(
select(Overlay)
.join(BlueprintOverlay, BlueprintOverlay.overlay_id == Overlay.id)
.where(BlueprintOverlay.blueprint_id == blueprint_id)
.order_by(BlueprintOverlay.position)
).all()
for overlay in overlays:
db.expunge(overlay)
log_stdout = on_stdout if on_stdout is not None else (lambda _line: None)
log_stderr = on_stderr if on_stderr is not None else (lambda _line: None)
cancel = should_cancel if should_cancel is not None else (lambda: False)
for overlay in overlays:
builder = BUILDERS.get(overlay.type)
if builder is None:
raise ValueError(f"no builder registered for overlay type {overlay.type!r}")
builder.build(
overlay,
on_stdout=log_stdout,
on_stderr=log_stderr,
should_cancel=cancel,
)
def _check_workshop_overlay_caches(*, blueprint_id: int) -> None:
"""Raise if any workshop overlay attached to this blueprint has items
that aren't yet in the workshop_cache. Mounting a partial overlay would
leave maps mysteriously missing in-game; surface the issue here instead.
"""
with session_scope() as db:
rows = db.execute(
select(Overlay.id, Overlay.name, WorkshopItem.steam_id)
.join(BlueprintOverlay, BlueprintOverlay.overlay_id == Overlay.id)
.join(
OverlayWorkshopItem,
OverlayWorkshopItem.overlay_id == Overlay.id,
)
.join(
WorkshopItem,
WorkshopItem.id == OverlayWorkshopItem.workshop_item_id,
)
.where(
BlueprintOverlay.blueprint_id == blueprint_id,
Overlay.type == "workshop",
)
).all()
missing: dict[tuple[int, str], list[str]] = {}
for overlay_id, overlay_name, steam_id in rows:
if not cache_path(steam_id).exists():
missing.setdefault((overlay_id, overlay_name), []).append(steam_id)
if not missing:
return
parts = []
for (overlay_id, overlay_name), steam_ids in missing.items():
ids = ", ".join(steam_ids)
parts.append(
f"overlay {overlay_name!r} (id={overlay_id}): items {ids} not yet downloaded"
)
detail = "; ".join(parts)
raise RuntimeError(
f"workshop content missing — {detail}. "
f"Open the overlay page and click Build (or wait for the auto-rebuild job), "
f"then retry."
)
def start_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None: def start_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
server, _, _ = load_server_blueprint_bundle(server_id) server, _, _ = load_server_blueprint_bundle(server_id)
host_commands.run_command( host_commands.run_command(
@ -99,6 +203,16 @@ def delete_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=
) )
def reset_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
server, _, _ = load_server_blueprint_bundle(server_id)
host_commands.run_command(
["l4d2ctl", "reset", server.name],
on_stdout=on_stdout,
on_stderr=on_stderr,
should_cancel=should_cancel,
)
def server_status(server_name: str) -> ServerStatus: def server_status(server_name: str) -> ServerStatus:
result = host_commands.run_command(["l4d2ctl", "status", server_name, "--json"]) result = host_commands.run_command(["l4d2ctl", "status", server_name, "--json"])
payload = json.loads(result.stdout or "{}") payload = json.loads(result.stdout or "{}")

View file

@ -0,0 +1,286 @@
"""Overlay builder registry.
Each `Overlay.type` maps to a builder. `build_overlay(overlay_id)` jobs (and
the synchronous `initialize_server` hook) dispatch through `BUILDERS`. Adding
a new overlay type means writing a new builder and registering it here no
changes to the worker, the mount layer, or the blueprint editor.
"""
from __future__ import annotations
import os
import subprocess
import tempfile
from pathlib import Path
from typing import Callable, Protocol
from sqlalchemy import select
from l4d2host.paths import get_left4me_root
from l4d2web.db import session_scope
from l4d2web.models import Overlay, OverlayWorkshopItem, WorkshopItem
from l4d2web.services.host_commands import run_command
from l4d2web.services.workshop_paths import cache_path, workshop_cache_root
CancelCheck = Callable[[], bool]
LogSink = Callable[[str], None]
SCRIPT_SANDBOX_HELPER = "/usr/local/libexec/left4me/left4me-script-sandbox"
DISK_BUDGET_BYTES = 20 * 1024**3
def _sandbox_script_dir() -> Path:
"""Where script tmpfiles live before being bind-mounted into the sandbox.
Cannot live in /tmp because the web service unit has PrivateTmp=yes:
its /tmp is a per-instance namespace that PID 1 (which actually performs
the BindReadOnlyPaths during sandbox setup) cannot resolve. /var/lib is
not affected by PrivateTmp and is visible to PID 1, so the bind-mount
succeeds.
"""
return get_left4me_root() / "sandbox-scripts"
class BuildError(RuntimeError):
"""Raised by builders when a build fails for a builder-specific reason
(e.g. disk-budget exceeded). Distinct from subprocess-level
HostCommandError / CommandCancelledError."""
class OverlayBuilder(Protocol):
def build(
self,
overlay: Overlay,
*,
on_stdout: LogSink,
on_stderr: LogSink,
should_cancel: CancelCheck,
) -> None: ...
def _overlay_root(overlay: Overlay) -> Path:
return get_left4me_root() / "overlays" / overlay.path
def overlay_path_for_id(overlay_id: int) -> Path:
return get_left4me_root() / "overlays" / str(overlay_id)
class WorkshopBuilder:
"""Diff-apply symlinks under `left4dead2/addons/` against the overlay's
current `WorkshopItem` associations. Cached items get an absolute symlink
into `workshop_cache/{steam_id}.vpk`. Items missing from cache are
skipped with a warning rather than turned into broken symlinks."""
def build(
self,
overlay: Overlay,
*,
on_stdout: LogSink,
on_stderr: LogSink,
should_cancel: CancelCheck,
) -> None:
addons_dir = _overlay_root(overlay) / "left4dead2" / "addons"
addons_dir.mkdir(parents=True, exist_ok=True)
with session_scope() as db:
items = db.scalars(
select(WorkshopItem)
.join(
OverlayWorkshopItem,
OverlayWorkshopItem.workshop_item_id == WorkshopItem.id,
)
.where(OverlayWorkshopItem.overlay_id == overlay.id)
).all()
# Detach items so we can use them outside the session.
items_data = [
(it.steam_id, it.last_downloaded_at) for it in items
]
cache_root = workshop_cache_root()
# desired: symlink-name -> absolute target path (only for cached items)
desired: dict[str, Path] = {}
skipped: list[str] = []
for steam_id, last_downloaded_at in items_data:
target = cache_path(steam_id)
if last_downloaded_at is None or not target.exists():
skipped.append(steam_id)
continue
desired[f"{steam_id}.vpk"] = target.resolve()
if should_cancel():
on_stderr("workshop build cancelled before applying symlinks")
return
# existing: symlink-name -> link target (only for symlinks pointing at our cache)
existing: dict[str, Path] = {}
for entry in os.scandir(addons_dir):
if not entry.is_symlink():
continue
try:
target = Path(os.readlink(entry.path))
except OSError:
continue
try:
resolved = target.resolve(strict=False)
except OSError:
continue
if not _is_under(resolved, cache_root):
continue
existing[entry.name] = resolved
created = 0
removed = 0
unchanged = 0
# Remove obsolete or stale symlinks first.
for name, current_target in existing.items():
if should_cancel():
on_stderr("workshop build cancelled mid-removal")
return
desired_target = desired.get(name)
if desired_target is None:
os.unlink(addons_dir / name)
removed += 1
elif current_target != desired_target:
os.unlink(addons_dir / name)
# will be recreated below
else:
unchanged += 1
# Recompute existing post-removal so the create loop knows what's left.
post_removal_existing = {
name for name in existing if name in desired and existing[name] == desired[name]
}
# Create new symlinks.
for name, target in desired.items():
if should_cancel():
on_stderr("workshop build cancelled mid-creation")
return
if name in post_removal_existing:
continue
link_path = addons_dir / name
# Defensive: if a non-symlink file collides with our name, leave it.
if link_path.exists() and not link_path.is_symlink():
on_stderr(
f"refusing to overwrite non-symlink at {link_path}; manual intervention required"
)
continue
if link_path.is_symlink():
# An obsolete symlink not in `existing` (target outside cache).
# We don't manage these — leave alone.
on_stderr(
f"refusing to overwrite foreign symlink at {link_path}"
)
continue
os.symlink(str(target), str(link_path))
created += 1
on_stdout(
f"workshop overlay {overlay.name!r}: created={created} "
f"removed={removed} unchanged={unchanged} "
f"skipped(uncached)={len(skipped)}"
)
for steam_id in skipped:
on_stderr(
f"workshop item {steam_id} skipped: not yet downloaded "
f"(refresh required before this overlay can mount it)"
)
def run_sandboxed_script(
overlay_id: int,
script_text: str,
*,
on_stdout: LogSink,
on_stderr: LogSink,
should_cancel: CancelCheck,
) -> None:
"""Write `script_text` to a tmpfile and exec it inside the privileged
sandbox helper. Used by ScriptBuilder.build and by the wipe route."""
script_dir = _sandbox_script_dir()
script_dir.mkdir(parents=True, exist_ok=True)
with tempfile.NamedTemporaryFile(
"w", suffix=".sh", delete=False, dir=str(script_dir)
) as f:
f.write(script_text or "")
script_path = f.name
# NamedTemporaryFile creates 0600 owned by the web user; the sandbox runs
# as l4d2-sandbox and needs to read it (bind-mounted at /script.sh inside
# the sandbox). Script content is not a secret — it's plain bash stored
# in the DB and editable by the user — so 0644 is appropriate.
os.chmod(script_path, 0o644)
try:
cmd = [
"sudo",
"-n",
SCRIPT_SANDBOX_HELPER,
str(overlay_id),
script_path,
]
run_command(
cmd,
on_stdout=on_stdout,
on_stderr=on_stderr,
should_cancel=should_cancel,
)
finally:
try:
os.unlink(script_path)
except FileNotFoundError:
pass
class ScriptBuilder:
"""Run an arbitrary user-authored bash script against the overlay dir
inside a bubblewrap + systemd-run sandbox. The script sees the overlay
dir as RW `/overlay` and a curated host RO mount; everything else is
isolated. After exit, enforce a 20 GB cap on `du -sb /overlay`."""
def build(
self,
overlay: Overlay,
*,
on_stdout: LogSink,
on_stderr: LogSink,
should_cancel: CancelCheck,
) -> None:
# Ensure target dir exists so the helper's bind-mount validation passes.
overlay_path_for_id(overlay.id).mkdir(parents=True, exist_ok=True)
run_sandboxed_script(
overlay.id,
overlay.script or "",
on_stdout=on_stdout,
on_stderr=on_stderr,
should_cancel=should_cancel,
)
self._enforce_disk_budget(overlay.id, on_stderr)
def _enforce_disk_budget(self, overlay_id: int, on_stderr: LogSink) -> None:
target = overlay_path_for_id(overlay_id)
size_output = subprocess.check_output(["du", "-sb", str(target)])
size_bytes = int(size_output.split()[0])
if size_bytes > DISK_BUDGET_BYTES:
on_stderr(
f"overlay exceeded 20 GB disk cap: {size_bytes} bytes > "
f"{DISK_BUDGET_BYTES} bytes"
)
raise BuildError("disk-cap-exceeded")
def _is_under(path: Path, root: Path) -> bool:
try:
path.relative_to(root)
except ValueError:
return False
return True
BUILDERS: dict[str, OverlayBuilder] = {
"workshop": WorkshopBuilder(),
"script": ScriptBuilder(),
}

View file

@ -0,0 +1,35 @@
"""Overlay path generation and on-disk directory bootstrap.
All new overlays (any type) get `path = str(overlay_id)`. The directory is
created with `exist_ok=False` so a stray folder from a prior failed delete
surfaces loudly instead of silently shadowing fresh content. Combined with
SQLite AUTOINCREMENT on `overlays.id`, that catches DB/disk drift.
"""
from __future__ import annotations
import os
from l4d2host.paths import get_left4me_root, validate_overlay_ref
from l4d2web.models import Overlay
def generate_overlay_path(overlay_id: int) -> str:
"""Return the canonical relative path for an overlay row.
Validates the result through l4d2host's overlay-ref guard. Pure numeric IDs
always pass this is just a belt-and-suspenders check that surfaces
immediately if someone changes the scheme.
"""
candidate = str(overlay_id)
return validate_overlay_ref(candidate)
def create_overlay_directory(overlay: Overlay) -> None:
"""Create `LEFT4ME_ROOT/overlays/{overlay.path}/` with `exist_ok=False`.
Raises `FileExistsError` if the directory already exists, surfacing the
rare DB/disk-drift state where a stray directory matches a fresh ID.
"""
target = get_left4me_root() / "overlays" / overlay.path
os.makedirs(target, exist_ok=False)

View file

@ -0,0 +1,295 @@
"""Steam Workshop API client + downloader.
Pure HTTP/file logic no DB writes, no Flask, no job-worker integration.
Used by the workshop overlay builder and the admin refresh job.
Endpoints:
- GetCollectionDetails: resolve a collection ID to its child item IDs.
- GetPublishedFileDetails: batch-fetch metadata for items, including a public
file_url for the .vpk.
Both endpoints accept anonymous POSTs; no Steam Web API key required.
"""
from __future__ import annotations
import os
import re
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable, Iterable, Literal
import requests
# HTTPS only (decision 16). The reference downloader uses HTTP — we don't.
GET_PUBLISHED_FILE_DETAILS_URL = (
"https://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1/"
)
GET_COLLECTION_DETAILS_URL = (
"https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/"
)
L4D2_APP_ID = 550
REQUEST_TIMEOUT_SECONDS = 30
DOWNLOAD_CHUNK_BYTES = 1_048_576
_NUMERIC_ID_RE = re.compile(r"^\d+$")
_URL_ID_RE = re.compile(r"^https?://([a-z0-9.-]*\.)?steamcommunity\.com/.*[?&]id=(\d+)", re.IGNORECASE)
_BARE_URL_ID_RE = re.compile(r"^([a-z0-9.-]*\.)?steamcommunity\.com/.*[?&]id=(\d+)", re.IGNORECASE)
_session_local = threading.local()
def _session() -> requests.Session:
"""Per-thread session for connection reuse without cross-thread leakage."""
sess = getattr(_session_local, "session", None)
if sess is None:
sess = requests.Session()
_session_local.session = sess
return sess
class WorkshopValidationError(ValueError):
"""Raised during user-add when an item fails a fixed precondition
(e.g. consumer_app_id != 550)."""
@dataclass(slots=True)
class WorkshopMetadata:
steam_id: str
title: str
filename: str
file_url: str
file_size: int
time_updated: int
preview_url: str
consumer_app_id: int
result: int
@dataclass(slots=True)
class RefreshReport:
downloaded: int = 0
skipped: int = 0
errors: int = 0
per_item_errors: dict[str, str] = field(default_factory=dict)
def parse_workshop_input(raw: str) -> list[str]:
"""Parse a single ID, a single workshop URL, or a multi-line / whitespace-
separated batch of either. Returns deduplicated digit-only IDs in order.
Raises ValueError on garbage."""
if not raw or not raw.strip():
raise ValueError("input is empty")
tokens: list[str] = []
for token in re.split(r"\s+", raw.strip()):
if not token:
continue
tokens.append(_extract_id(token))
seen: set[str] = set()
deduped: list[str] = []
for tok in tokens:
if tok not in seen:
seen.add(tok)
deduped.append(tok)
return deduped
def _extract_id(token: str) -> str:
if _NUMERIC_ID_RE.fullmatch(token):
return token
m = _URL_ID_RE.match(token)
if m:
return m.group(2)
m = _BARE_URL_ID_RE.match(token)
if m:
return m.group(2)
raise ValueError(f"could not parse a Steam workshop id from: {token!r}")
def resolve_collection(collection_id: str) -> list[str]:
"""POST GetCollectionDetails for one collection; return its non-collection
child publishedfileids in order. Nested collections (filetype != 0) are
skipped."""
if not _NUMERIC_ID_RE.fullmatch(collection_id):
raise ValueError("collection_id must be digits only")
response = _session().post(
GET_COLLECTION_DETAILS_URL,
data={
"collectioncount": 1,
"publishedfileids[0]": collection_id,
},
timeout=REQUEST_TIMEOUT_SECONDS,
)
response.raise_for_status()
payload = response.json()
children: list[str] = []
for collection in payload.get("response", {}).get("collectiondetails", []):
for child in collection.get("children", []):
if child.get("filetype", 0) != 0:
continue # nested collection, skip
child_id = child.get("publishedfileid")
if child_id is not None:
children.append(str(child_id))
return children
def fetch_metadata_batch(
steam_ids: list[str], *, mode: Literal["add", "refresh"]
) -> list[WorkshopMetadata]:
"""One POST to GetPublishedFileDetails covering all ids.
In `mode="add"`, any non-L4D2 (`consumer_app_id != 550`) raises
WorkshopValidationError so the user-add request fails cleanly.
In `mode="refresh"`, non-L4D2 entries are skipped from the result.
Items with `result != 1` are returned as-is (the caller persists the result
code into `WorkshopItem.last_error`).
"""
if not steam_ids:
return []
for sid in steam_ids:
if not _NUMERIC_ID_RE.fullmatch(sid):
raise ValueError(f"steam id must be digits only: {sid!r}")
payload: dict[str, str | int] = {"itemcount": len(steam_ids)}
for index, sid in enumerate(steam_ids):
payload[f"publishedfileids[{index}]"] = sid
response = _session().post(
GET_PUBLISHED_FILE_DETAILS_URL,
data=payload,
timeout=REQUEST_TIMEOUT_SECONDS,
)
response.raise_for_status()
body = response.json()
metas: list[WorkshopMetadata] = []
for entry in body.get("response", {}).get("publishedfiledetails", []):
meta = WorkshopMetadata(
steam_id=str(entry.get("publishedfileid", "")),
title=str(entry.get("title", "") or ""),
filename=str(entry.get("filename", "") or ""),
file_url=str(entry.get("file_url", "") or ""),
file_size=int(entry.get("file_size") or 0),
time_updated=int(entry.get("time_updated") or 0),
preview_url=str(entry.get("preview_url", "") or ""),
consumer_app_id=int(entry.get("consumer_app_id") or 0),
result=int(entry.get("result") or 0),
)
# consumer_app_id is only meaningful when the lookup itself succeeded.
if meta.result == 1 and meta.consumer_app_id != L4D2_APP_ID:
if mode == "add":
raise WorkshopValidationError(
f"item {meta.steam_id} is not a Left 4 Dead 2 workshop "
f"item (consumer_app_id={meta.consumer_app_id})"
)
# refresh mode: drop the entry silently from the batch
continue
metas.append(meta)
return metas
def download_to_cache(
meta: WorkshopMetadata,
cache_root: Path,
*,
on_progress: Callable[[int, int], None] | None = None,
should_cancel: Callable[[], bool] | None = None,
) -> Path:
"""Download `meta.file_url` to `cache_root/{steam_id}.vpk`.
Atomic via `*.partial` + `os.replace`. Idempotent: a no-op when the
existing file's `(mtime, size)` already matches `(time_updated, file_size)`.
Sets `os.utime(target, (time_updated, time_updated))` so the next run
short-circuits.
"""
if not _NUMERIC_ID_RE.fullmatch(meta.steam_id):
raise ValueError("meta.steam_id must be digits only")
cache_root.mkdir(parents=True, exist_ok=True)
target = cache_root / f"{meta.steam_id}.vpk"
if (
target.exists()
and int(target.stat().st_mtime) == int(meta.time_updated)
and int(target.stat().st_size) == int(meta.file_size)
):
return target
if not meta.file_url:
raise ValueError(f"item {meta.steam_id} has no file_url; cannot download")
partial = target.with_suffix(target.suffix + ".partial")
response = _session().get(meta.file_url, stream=True, timeout=REQUEST_TIMEOUT_SECONDS)
response.raise_for_status()
written = 0
try:
with open(partial, "wb") as f:
for chunk in response.iter_content(chunk_size=DOWNLOAD_CHUNK_BYTES):
if should_cancel is not None and should_cancel():
raise InterruptedError("download cancelled")
if not chunk:
continue
f.write(chunk)
written += len(chunk)
if on_progress is not None:
on_progress(written, int(meta.file_size))
os.replace(partial, target)
except BaseException:
partial.unlink(missing_ok=True)
raise
os.utime(target, (meta.time_updated, meta.time_updated))
return target
def refresh_all(
metas: Iterable[WorkshopMetadata],
cache_root: Path,
*,
executor_workers: int = 8,
should_cancel: Callable[[], bool] | None = None,
) -> RefreshReport:
"""Download (or skip-as-cached) every metadata item using a thread pool.
Per-item errors are collected; sibling items continue."""
metas_list = list(metas)
report = RefreshReport()
if not metas_list:
return report
cache_root.mkdir(parents=True, exist_ok=True)
with ThreadPoolExecutor(max_workers=executor_workers) as executor:
futures = {}
for meta in metas_list:
if should_cancel is not None and should_cancel():
break
future = executor.submit(
download_to_cache,
meta,
cache_root,
should_cancel=should_cancel,
)
futures[future] = meta
for future in as_completed(futures):
meta = futures[future]
try:
future.result()
except Exception as exc:
report.errors += 1
report.per_item_errors[meta.steam_id] = str(exc)
continue
report.downloaded += 1
return report

View file

@ -0,0 +1,24 @@
"""Cache-path helpers for workshop content.
The cache lives at `$LEFT4ME_ROOT/workshop_cache/{steam_id}.vpk`. Steam IDs
are validated digit-only here so callers don't need to guard separately.
"""
from __future__ import annotations
import re
from pathlib import Path
from l4d2host.paths import get_left4me_root
_NUMERIC_ID_RE = re.compile(r"^\d+$")
def workshop_cache_root() -> Path:
return get_left4me_root() / "workshop_cache"
def cache_path(steam_id: str) -> Path:
if not isinstance(steam_id, str) or not _NUMERIC_ID_RE.fullmatch(steam_id):
raise ValueError(f"steam_id must be digits only: {steam_id!r}")
return workshop_cache_root() / f"{steam_id}.vpk"

View file

@ -59,16 +59,20 @@ a:focus-visible {
outline-offset: 2px; outline-offset: 2px;
} }
button { button,
a.button {
background: var(--color-primary); background: var(--color-primary);
border: none; border: none;
border-radius: var(--radius-s); border-radius: var(--radius-s);
color: #fff; color: #fff;
padding: var(--space-s) var(--space-l); padding: var(--space-s) var(--space-l);
cursor: pointer; cursor: pointer;
display: inline-block;
text-decoration: none;
} }
button.danger { button.danger,
a.button.danger {
background: var(--color-danger); background: var(--color-danger);
} }
@ -88,3 +92,99 @@ button.danger {
.auth-panel { .auth-panel {
max-width: 28rem; max-width: 28rem;
} }
dialog.modal {
background: var(--color-surface);
color: var(--color-text);
border: var(--line);
border-radius: var(--radius-m);
padding: 0;
width: min(32rem, 90vw);
max-height: 90vh;
}
dialog.modal::backdrop {
background: rgba(0, 0, 0, 0.45);
}
.modal-header,
.modal-body,
.modal-footer {
padding: var(--space-l);
}
.modal-header {
display: flex;
justify-content: space-between;
align-items: center;
border-bottom: var(--line);
}
.modal-header h2 {
margin: 0;
font-size: 1.1rem;
}
.modal-body {
display: grid;
gap: var(--space-m);
overflow-y: auto;
}
.modal-footer {
display: flex;
justify-content: flex-end;
gap: var(--space-m);
border-top: var(--line);
}
.modal-close {
background: none;
color: var(--color-muted);
padding: var(--space-xs) var(--space-s);
font-size: 1.25rem;
line-height: 1;
}
.button-secondary {
background: var(--color-surface);
color: var(--color-text);
border: var(--line);
}
.field-hint {
color: var(--color-muted);
font-size: 0.875rem;
}
.page-heading {
display: flex;
justify-content: space-between;
align-items: center;
gap: var(--space-m);
margin-bottom: var(--space-l);
}
.page-heading h1,
.page-heading h2 {
margin: 0;
}
.button-row {
display: flex;
gap: var(--space-s);
flex-wrap: wrap;
}
.detail-grid {
display: grid;
gap: var(--space-l);
}
.used-by-list {
list-style: none;
padding: 0;
margin: 0;
display: grid;
gap: var(--space-s);
}

View file

@ -0,0 +1,27 @@
document.addEventListener("DOMContentLoaded", () => {
document.querySelectorAll("[data-modal-open]").forEach((trigger) => {
trigger.addEventListener("click", (event) => {
const targetId = trigger.getAttribute("data-modal-open");
const dialog = document.getElementById(targetId);
if (dialog && typeof dialog.showModal === "function") {
event.preventDefault();
dialog.showModal();
}
});
});
document.querySelectorAll("dialog.modal").forEach((dialog) => {
dialog.querySelectorAll("[data-modal-close]").forEach((closer) => {
closer.addEventListener("click", (event) => {
event.preventDefault();
dialog.close();
});
});
dialog.addEventListener("click", (event) => {
if (event.target === dialog) {
dialog.close();
}
});
});
});

View file

@ -17,7 +17,7 @@
<td><a href="/jobs/{{ job.id }}">#{{ job.id }}</a></td> <td><a href="/jobs/{{ job.id }}">#{{ job.id }}</a></td>
<td>{{ job.operation }}</td> <td>{{ job.operation }}</td>
<td>{{ job.state }}</td> <td>{{ job.state }}</td>
{% if show_user %}<td>{{ user.username }}</td>{% endif %} {% if show_user %}<td>{{ user.username if user else "system" }}</td>{% endif %}
{% if show_server %}<td>{% if server %}<a href="/servers/{{ server.id }}">{{ server.name }}</a>{% else %}-{% endif %}</td>{% endif %} {% if show_server %}<td>{% if server %}<a href="/servers/{{ server.id }}">{{ server.name }}</a>{% else %}-{% endif %}</td>{% endif %}
<td>{{ job.created_at }}</td> <td>{{ job.created_at }}</td>
<td>{{ job.finished_at or "-" }}</td> <td>{{ job.finished_at or "-" }}</td>

View file

@ -0,0 +1,50 @@
{% set can_edit = g.user.admin or (overlay and overlay.type == 'workshop' and overlay.user_id == g.user.id) %}
<table class="table">
<thead>
<tr>
<th></th>
<th>Steam ID</th>
<th>Title</th>
<th>Filename</th>
<th>Size</th>
<th>Updated</th>
<th>Status</th>
{% if can_edit %}<th></th>{% endif %}
</tr>
</thead>
<tbody>
{% for item in workshop_items %}
<tr>
<td>
{% if item.preview_url %}
<img src="{{ item.preview_url }}" alt="" width="48" height="48" loading="lazy">
{% endif %}
</td>
<td><a href="https://steamcommunity.com/sharedfiles/filedetails/?id={{ item.steam_id }}" target="_blank" rel="noopener">{{ item.steam_id }}</a></td>
<td>{{ item.title }}</td>
<td class="muted">{{ item.filename }}</td>
<td class="muted">{{ item.file_size }}</td>
<td class="muted">{{ item.time_updated }}</td>
<td>
{% if item.last_error %}
<span class="warning">{{ item.last_error }}</span>
{% elif item.last_downloaded_at %}
cached
{% else %}
<span class="muted">pending</span>
{% endif %}
</td>
{% if can_edit %}
<td>
<form method="post" action="/overlays/{{ overlay.id }}/items/{{ item.id }}/delete" class="inline-form">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<button type="submit" class="button-secondary">Remove</button>
</form>
</td>
{% endif %}
</tr>
{% else %}
<tr><td colspan="{% if can_edit %}8{% else %}7{% endif %}" class="muted">No workshop items yet.</td></tr>
{% endfor %}
</tbody>
</table>

View file

@ -19,4 +19,13 @@
<button type="submit">Install or update runtime</button> <button type="submit">Install or update runtime</button>
</form> </form>
</section> </section>
<section class="panel">
<h2>Workshop</h2>
<p class="muted">Re-fetch metadata for every workshop item; re-download those with newer versions on Steam, then enqueue rebuilds for the affected overlays.</p>
<form method="post" action="/admin/workshop/refresh">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<button type="submit">Refresh all workshop items</button>
</form>
</section>
{% endblock %} {% endblock %}

View file

@ -39,5 +39,6 @@
<script src="{{ url_for('static', filename='vendor/htmx.min.js') }}"></script> <script src="{{ url_for('static', filename='vendor/htmx.min.js') }}"></script>
<script src="{{ url_for('static', filename='js/csrf.js') }}"></script> <script src="{{ url_for('static', filename='js/csrf.js') }}"></script>
<script src="{{ url_for('static', filename='js/sse.js') }}"></script> <script src="{{ url_for('static', filename='js/sse.js') }}"></script>
<script src="{{ url_for('static', filename='js/modal.js') }}"></script>
</body> </body>
</html> </html>

View file

@ -4,7 +4,10 @@
{% block content %} {% block content %}
<section class="panel"> <section class="panel">
<h1>Blueprint: {{ blueprint.name }}</h1> <div class="page-heading">
<h1>Blueprint: {{ blueprint.name }}</h1>
<button type="button" class="danger" data-modal-open="delete-blueprint-modal">Delete</button>
</div>
<form method="post" action="/blueprints/{{ blueprint.id }}" class="stack"> <form method="post" action="/blueprints/{{ blueprint.id }}" class="stack">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}"> <input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<label>Name <input name="name" value="{{ blueprint.name }}" required></label> <label>Name <input name="name" value="{{ blueprint.name }}" required></label>
@ -28,4 +31,21 @@
<button type="submit">Save blueprint</button> <button type="submit">Save blueprint</button>
</form> </form>
</section> </section>
<dialog id="delete-blueprint-modal" class="modal" aria-labelledby="delete-blueprint-title">
<div class="modal-header">
<h2 id="delete-blueprint-title">Delete blueprint "{{ blueprint.name }}"?</h2>
<button type="button" class="modal-close" data-modal-close aria-label="Close">&times;</button>
</div>
<div class="modal-body">
<p>This cannot be undone. Blueprints in use by a server cannot be deleted.</p>
</div>
<div class="modal-footer">
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
<form method="post" action="/blueprints/{{ blueprint.id }}/delete" class="inline-form">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<button class="danger" type="submit">Delete</button>
</form>
</div>
</dialog>
{% endblock %} {% endblock %}

View file

@ -4,14 +4,10 @@
{% block content %} {% block content %}
<section class="panel"> <section class="panel">
<h1>Blueprints</h1> <div class="page-heading">
<form method="post" action="/blueprints" class="stack form-panel"> <h1>Blueprints</h1>
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}"> <button type="button" data-modal-open="create-blueprint-modal">+ Create</button>
<label>Name <input name="name" required></label> </div>
<label>Arguments <textarea name="arguments"></textarea></label>
<label>Config <textarea name="config"></textarea></label>
<button type="submit">Create blueprint</button>
</form>
<table class="table"> <table class="table">
<thead><tr><th>Name</th><th>Created</th><th>Updated</th><th>Actions</th></tr></thead> <thead><tr><th>Name</th><th>Created</th><th>Updated</th><th>Actions</th></tr></thead>
<tbody> <tbody>
@ -20,12 +16,7 @@
<td><a href="/blueprints/{{ blueprint.id }}">{{ blueprint.name }}</a></td> <td><a href="/blueprints/{{ blueprint.id }}">{{ blueprint.name }}</a></td>
<td>{{ blueprint.created_at }}</td> <td>{{ blueprint.created_at }}</td>
<td>{{ blueprint.updated_at }}</td> <td>{{ blueprint.updated_at }}</td>
<td> <td><a href="/servers?blueprint_id={{ blueprint.id }}">Create server</a></td>
<form method="post" action="/blueprints/{{ blueprint.id }}/delete" class="inline-form">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<button class="danger" type="submit">Delete</button>
</form>
</td>
</tr> </tr>
{% else %} {% else %}
<tr><td colspan="4" class="muted">No blueprints configured.</td></tr> <tr><td colspan="4" class="muted">No blueprints configured.</td></tr>
@ -33,4 +24,23 @@
</tbody> </tbody>
</table> </table>
</section> </section>
<dialog id="create-blueprint-modal" class="modal" aria-labelledby="create-blueprint-title">
<form method="post" action="/blueprints" class="stack">
<div class="modal-header">
<h2 id="create-blueprint-title">Create blueprint</h2>
<button type="button" class="modal-close" data-modal-close aria-label="Close">&times;</button>
</div>
<div class="modal-body">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<label>Name <input name="name" required></label>
<label>Arguments <textarea name="arguments"></textarea></label>
<label>Config <textarea name="config"></textarea></label>
</div>
<div class="modal-footer">
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
<button type="submit">Create blueprint</button>
</div>
</form>
</dialog>
{% endblock %} {% endblock %}

View file

@ -19,7 +19,7 @@
<tbody> <tbody>
<tr><th>Operation</th><td>{{ job.operation }}</td></tr> <tr><th>Operation</th><td>{{ job.operation }}</td></tr>
<tr><th>State</th><td>{{ job.state }}</td></tr> <tr><th>State</th><td>{{ job.state }}</td></tr>
<tr><th>User</th><td>{{ owner.username }}</td></tr> <tr><th>User</th><td>{{ owner.username if owner else "system" }}</td></tr>
<tr><th>Server</th><td>{% if server %}<a href="/servers/{{ server.id }}">{{ server.name }}</a>{% else %}-{% endif %}</td></tr> <tr><th>Server</th><td>{% if server %}<a href="/servers/{{ server.id }}">{{ server.name }}</a>{% else %}-{% endif %}</td></tr>
<tr><th>Created</th><td>{{ job.created_at }}</td></tr> <tr><th>Created</th><td>{{ job.created_at }}</td></tr>
<tr><th>Started</th><td>{{ job.started_at or "-" }}</td></tr> <tr><th>Started</th><td>{{ job.started_at or "-" }}</td></tr>

View file

@ -0,0 +1,182 @@
{% extends "base.html" %}
{% block title %}Overlay {{ overlay.name }} | left4me{% endblock %}
{% block content %}
<section class="panel">
<div class="page-heading">
<h1>Overlay: {{ overlay.name }}</h1>
{% set can_edit = g.user.admin or (overlay.type in ['workshop', 'script'] and overlay.user_id == g.user.id) %}
{% if can_edit %}
<button type="button" class="danger" data-modal-open="delete-overlay-modal">Delete</button>
{% endif %}
</div>
{% if can_edit %}
<form method="post" action="/overlays/{{ overlay.id }}" class="stack">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<label>Name <input name="name" value="{{ overlay.name }}" required></label>
<div>
<button type="submit">Save</button>
</div>
</form>
{% endif %}
<table class="definition-table">
<tbody>
<tr><th>Type</th><td>{{ overlay.type }}</td></tr>
<tr><th>Scope</th><td>{% if overlay.user_id %}private{% else %}system{% endif %}</td></tr>
<tr><th>Path</th><td class="muted">{{ overlay.path }}</td></tr>
<tr>
<th>Last build</th>
<td>
{% if overlay.last_build_status == 'ok' %}
<span class="badge badge-ok">ok</span>
{% elif overlay.last_build_status == 'failed' %}
<span class="badge badge-error">failed</span>
{% else %}
<span class="badge badge-muted">never</span>
{% endif %}
</td>
</tr>
</tbody>
</table>
</section>
{% if overlay.type == 'script' %}
<section class="panel">
<div class="page-heading">
<h2>Script</h2>
{% if can_edit %}
<div class="inline-form-group">
<form method="post" action="/overlays/{{ overlay.id }}/build" class="inline-form">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<button type="submit" class="button-secondary">Rebuild</button>
</form>
<button type="button" class="danger" data-modal-open="wipe-overlay-modal">Wipe</button>
</div>
{% endif %}
</div>
{% if can_edit %}
<form method="post" action="/overlays/{{ overlay.id }}/script" class="stack">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<label>Bash script
<textarea name="script" rows="20" spellcheck="false">{{ overlay.script or "" }}</textarea>
</label>
<p class="muted">Runs sandboxed against the overlay directory mounted at <code>/overlay</code>.</p>
<div>
<button type="submit">Save and build</button>
</div>
</form>
{% else %}
<pre class="script-preview">{{ overlay.script or "" }}</pre>
{% endif %}
<p>
{% if latest_build_job %}
Latest build: <a href="/jobs/{{ latest_build_job.id }}">job #{{ latest_build_job.id }}</a>
— state: <strong>{{ latest_build_job.state }}</strong>
·
{% endif %}
<a href="/overlays/{{ overlay.id }}/jobs">all builds →</a>
</p>
</section>
{% endif %}
{% if overlay.type == 'workshop' %}
<section class="panel">
<div class="page-heading">
<h2>Workshop items</h2>
{% if can_edit %}
<form method="post" action="/overlays/{{ overlay.id }}/build" class="inline-form">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<button type="submit" class="button-secondary">Rebuild</button>
</form>
{% endif %}
</div>
{% if can_edit %}
<form method="post" action="/overlays/{{ overlay.id }}/items" class="stack">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<fieldset class="workshop-input-mode">
<legend>Input mode</legend>
<label><input type="radio" name="input_mode" value="items" checked> Items (paste IDs or URLs; one or many)</label>
<label><input type="radio" name="input_mode" value="collection"> Collection (one ID or URL)</label>
</fieldset>
<label>Workshop input <textarea name="input" rows="3" placeholder="123456789&#10;https://steamcommunity.com/sharedfiles/filedetails/?id=987654321"></textarea></label>
<div>
<button type="submit">Add</button>
</div>
</form>
{% endif %}
<div id="overlay-item-table">
{% include "_overlay_item_table.html" with context %}
</div>
</section>
<section class="panel">
<h2>Builds</h2>
<p>
{% if latest_build_job %}
Latest build: <a href="/jobs/{{ latest_build_job.id }}">job #{{ latest_build_job.id }}</a>
— state: <strong>{{ latest_build_job.state }}</strong>
·
{% endif %}
<a href="/overlays/{{ overlay.id }}/jobs">all builds →</a>
</p>
</section>
{% endif %}
<section class="panel">
<h2>Used by</h2>
{% if using_blueprints %}
<ul class="used-by-list">
{% for blueprint in using_blueprints %}
<li><a href="/blueprints/{{ blueprint.id }}">{{ blueprint.name }}</a></li>
{% endfor %}
</ul>
{% else %}
<p class="muted">Not used by any blueprint.</p>
{% endif %}
</section>
{% if can_edit %}
<dialog id="delete-overlay-modal" class="modal" aria-labelledby="delete-overlay-title">
<div class="modal-header">
<h2 id="delete-overlay-title">Delete overlay "{{ overlay.name }}"?</h2>
<button type="button" class="modal-close" data-modal-close aria-label="Close">&times;</button>
</div>
<div class="modal-body">
<p>This cannot be undone. Overlays in use by a blueprint cannot be deleted.</p>
</div>
<div class="modal-footer">
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
<form method="post" action="/overlays/{{ overlay.id }}/delete" class="inline-form">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<button class="danger" type="submit">Delete</button>
</form>
</div>
</dialog>
{% if overlay.type == 'script' %}
<dialog id="wipe-overlay-modal" class="modal" aria-labelledby="wipe-overlay-title">
<div class="modal-header">
<h2 id="wipe-overlay-title">Wipe overlay "{{ overlay.name }}"?</h2>
<button type="button" class="modal-close" data-modal-close aria-label="Close">&times;</button>
</div>
<div class="modal-body">
<p>Empties the overlay directory. Use Rebuild afterward to repopulate.</p>
</div>
<div class="modal-footer">
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
<form method="post" action="/overlays/{{ overlay.id }}/wipe" class="inline-form">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<button class="danger" type="submit">Wipe</button>
</form>
</div>
</dialog>
{% endif %}
{% endif %}
{% endblock %}

View file

@ -0,0 +1,17 @@
{% extends "base.html" %}
{% block title %}Build jobs for {{ overlay.name }} | left4me{% endblock %}
{% block content %}
<section class="panel">
<div class="page-heading">
<h1>Build jobs for {{ overlay.name }}</h1>
<a href="/overlays/{{ overlay.id }}">Back to overlay</a>
</div>
{% set show_user = true %}
{% set show_server = false %}
{% set show_cancel = true %}
{% set cancel_next = "/overlays/" ~ overlay.id ~ "/jobs" %}
{% include "_job_table.html" %}
</section>
{% endblock %}

View file

@ -6,43 +6,49 @@
<section class="panel"> <section class="panel">
<div class="page-heading"> <div class="page-heading">
<h1>Overlays</h1> <h1>Overlays</h1>
<button type="button" data-modal-open="create-overlay-modal">+ Create</button>
</div> </div>
{% if g.user.admin %}
<form method="post" action="/overlays" class="stack form-panel">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<label>Name <input name="name" required></label>
<label>Path <input name="path" required placeholder="/opt/l4d2/overlays/example"></label>
<button type="submit">Add overlay</button>
</form>
{% endif %}
<table class="table"> <table class="table">
<thead><tr><th>Name</th><th>Path</th>{% if g.user.admin %}<th>Actions</th>{% endif %}</tr></thead> <thead><tr><th>Name</th><th>Type</th><th>Scope</th><th>Path</th></tr></thead>
<tbody> <tbody>
{% for overlay in overlays %} {% for overlay in overlays %}
<tr> <tr>
<td>{{ overlay.name }}</td> <td><a href="/overlays/{{ overlay.id }}">{{ overlay.name }}</a></td>
<td>{{ overlay.type }}</td>
<td class="muted">{% if overlay.user_id %}private{% else %}system{% endif %}</td>
<td class="muted">{{ overlay.path }}</td> <td class="muted">{{ overlay.path }}</td>
{% if g.user.admin %}
<td>
<form method="post" action="/overlays/{{ overlay.id }}" class="inline-form">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<input name="name" value="{{ overlay.name }}" required>
<input name="path" value="{{ overlay.path }}" required>
<button type="submit">Save</button>
</form>
<form method="post" action="/overlays/{{ overlay.id }}/delete" class="inline-form">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<button class="danger" type="submit">Delete</button>
</form>
</td>
{% endif %}
</tr> </tr>
{% else %} {% else %}
<tr><td colspan="{% if g.user.admin %}3{% else %}2{% endif %}" class="muted">No overlays configured.</td></tr> <tr><td colspan="4" class="muted">No overlays yet.</td></tr>
{% endfor %} {% endfor %}
</tbody> </tbody>
</table> </table>
</section> </section>
<dialog id="create-overlay-modal" class="modal" aria-labelledby="create-overlay-title">
<form method="post" action="/overlays" class="stack">
<div class="modal-header">
<h2 id="create-overlay-title">Create overlay</h2>
<button type="button" class="modal-close" data-modal-close aria-label="Close">&times;</button>
</div>
<div class="modal-body">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<fieldset class="overlay-type-radio">
<legend>Type</legend>
<label><input type="radio" name="type" value="workshop" checked> Workshop (downloads from Steam)</label>
<label><input type="radio" name="type" value="script"> Script (runs sandboxed bash)</label>
</fieldset>
<label>Name <input name="name" required></label>
{% if g.user and g.user.admin %}
<label><input type="checkbox" name="system_wide" value="1"> System-wide (visible to all users)</label>
{% endif %}
<p class="muted">The path is generated automatically.</p>
</div>
<div class="modal-footer">
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
<button type="submit">Create</button>
</div>
</form>
</dialog>
{% endblock %} {% endblock %}

View file

@ -13,10 +13,8 @@
<button type="submit">{{ operation }}</button> <button type="submit">{{ operation }}</button>
</form> </form>
{% endfor %} {% endfor %}
<form method="post" action="/servers/{{ server.id }}/delete" class="inline-form"> <button type="button" class="danger" data-modal-open="reset-server-modal">reset</button>
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}"> <button type="button" class="danger" data-modal-open="delete-server-modal">delete</button>
<button class="danger" type="submit">delete</button>
</form>
</div> </div>
</div> </div>
@ -49,4 +47,38 @@
<h2>Server Log</h2> <h2>Server Log</h2>
<pre class="log-stream" data-sse-url="/servers/{{ server.id }}/logs/stream"></pre> <pre class="log-stream" data-sse-url="/servers/{{ server.id }}/logs/stream"></pre>
</section> </section>
<dialog id="reset-server-modal" class="modal" aria-labelledby="reset-server-title">
<div class="modal-header">
<h2 id="reset-server-title">Reset server "{{ server.name }}"?</h2>
<button type="button" class="modal-close" data-modal-close aria-label="Close">&times;</button>
</div>
<div class="modal-body">
<p>This stops the server and wipes its runtime state (logs, caches, accumulated game state). The blueprint association is preserved; the next start rebuilds from the current blueprint.</p>
</div>
<div class="modal-footer">
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
<form method="post" action="/servers/{{ server.id }}/reset" class="inline-form">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<button class="danger" type="submit">Reset</button>
</form>
</div>
</dialog>
<dialog id="delete-server-modal" class="modal" aria-labelledby="delete-server-title">
<div class="modal-header">
<h2 id="delete-server-title">Delete server "{{ server.name }}"?</h2>
<button type="button" class="modal-close" data-modal-close aria-label="Close">&times;</button>
</div>
<div class="modal-body">
<p>This stops the server and tears down its runtime files. This cannot be undone.</p>
</div>
<div class="modal-footer">
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
<form method="post" action="/servers/{{ server.id }}/delete" class="inline-form">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<button class="danger" type="submit">Delete</button>
</form>
</div>
</dialog>
{% endblock %} {% endblock %}

View file

@ -4,24 +4,14 @@
{% block content %} {% block content %}
<section class="panel"> <section class="panel">
<h1>Servers</h1> <div class="page-heading">
{% if blueprints %} <h1>Servers</h1>
<form method="post" action="/servers" class="stack form-panel"> {% if blueprints %}
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}"> <button type="button" data-modal-open="create-server-modal">+ Create</button>
<label>Name <input name="name" required></label> {% else %}
<label>Port <input name="port" type="number" min="1" max="65535" value="27015" required></label> <a class="button" href="/blueprints">Create a blueprint first &rarr;</a>
<label>Blueprint {% endif %}
<select name="blueprint_id" required> </div>
{% for blueprint in blueprints %}
<option value="{{ blueprint.id }}">{{ blueprint.name }}</option>
{% endfor %}
</select>
</label>
<button type="submit">Create server</button>
</form>
{% else %}
<p class="muted">Create a blueprint before adding servers.</p>
{% endif %}
<table class="table"> <table class="table">
<thead><tr><th>Name</th><th>Port</th><th>Blueprint</th><th>Desired</th><th>Actual</th></tr></thead> <thead><tr><th>Name</th><th>Port</th><th>Blueprint</th><th>Desired</th><th>Actual</th></tr></thead>
<tbody> <tbody>
@ -39,4 +29,45 @@
</tbody> </tbody>
</table> </table>
</section> </section>
{% if blueprints %}
<dialog id="create-server-modal" class="modal" aria-labelledby="create-server-title">
<form method="post" action="/servers" class="stack">
<div class="modal-header">
<h2 id="create-server-title">Create server</h2>
<button type="button" class="modal-close" data-modal-close aria-label="Close">&times;</button>
</div>
<div class="modal-body">
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
<label>Name <input name="name" required></label>
<label>Port
<input name="port" type="number" min="1" max="65535" placeholder="auto">
<span class="field-hint">Leave empty for the next available port.</span>
</label>
<label>Blueprint
<select name="blueprint_id" required>
{% for blueprint in blueprints %}
<option value="{{ blueprint.id }}"{% if blueprint.id == prefill_blueprint_id %} selected{% endif %}>{{ blueprint.name }}</option>
{% endfor %}
</select>
</label>
</div>
<div class="modal-footer">
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
<button type="submit">Create server</button>
</div>
</form>
</dialog>
{% endif %}
{% if prefill_blueprint_id %}
<script>
document.addEventListener("DOMContentLoaded", () => {
const dialog = document.getElementById("create-server-modal");
if (dialog && typeof dialog.showModal === "function") {
dialog.showModal();
}
});
</script>
{% endif %}
{% endblock %} {% endblock %}

View file

@ -0,0 +1,96 @@
"""Tests for the alembic migration history.
The 0005 migration adds `script` and `last_build_status` columns to `overlays`,
drops the global_overlay_* tables, and wipes legacy l4d2center_maps/cedapug_maps
overlay rows. This module pins those behaviors.
"""
from pathlib import Path
import pytest
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine, inspect, text
_ALEMBIC_DIR = Path(__file__).resolve().parents[1] / "alembic"
def _alembic_config(db_url: str) -> Config:
cfg = Config()
cfg.set_main_option("script_location", str(_ALEMBIC_DIR))
cfg.set_main_option("sqlalchemy.url", db_url)
return cfg
@pytest.fixture
def db_url(tmp_path, monkeypatch):
path = tmp_path / "alembic.db"
url = f"sqlite:///{path}"
monkeypatch.setenv("DATABASE_URL", url)
yield url
def test_upgrade_0005_adds_script_columns(db_url) -> None:
cfg = _alembic_config(db_url)
command.upgrade(cfg, "0004_drop_legacy_external_overlay_type")
engine = create_engine(db_url)
with engine.begin() as conn:
# Seed legacy global-type overlay rows that the migration must wipe.
conn.execute(
text(
"INSERT INTO overlays (name, path, type, created_at, updated_at) "
"VALUES ('legacy-l4d2center', '1', 'l4d2center_maps', "
"'2026-01-01', '2026-01-01')"
)
)
conn.execute(
text(
"INSERT INTO overlays (name, path, type, created_at, updated_at) "
"VALUES ('legacy-cedapug', '2', 'cedapug_maps', "
"'2026-01-01', '2026-01-01')"
)
)
conn.execute(
text(
"INSERT INTO overlays (name, path, type, created_at, updated_at) "
"VALUES ('keep-workshop', '3', 'workshop', "
"'2026-01-01', '2026-01-01')"
)
)
command.upgrade(cfg, "0005_script_overlays")
inspector = inspect(engine)
overlay_cols = {c["name"]: c for c in inspector.get_columns("overlays")}
assert "script" in overlay_cols
assert "last_build_status" in overlay_cols
assert overlay_cols["script"]["nullable"] is False
assert overlay_cols["last_build_status"]["nullable"] is False
table_names = set(inspector.get_table_names())
assert "global_overlay_sources" not in table_names
assert "global_overlay_items" not in table_names
assert "global_overlay_item_files" not in table_names
with engine.connect() as conn:
rows = conn.execute(
text("SELECT name, type FROM overlays ORDER BY name")
).all()
assert rows == [("keep-workshop", "workshop")]
defaults = conn.execute(
text(
"SELECT script, last_build_status FROM overlays "
"WHERE name = 'keep-workshop'"
)
).one()
assert defaults == ("", "")
def test_downgrade_0005_skipped() -> None:
"""Per the project convention (see 0004) destructive migrations are
intentionally one-way; do not test or maintain a downgrade."""
pytest.skip("0005 is one-way: globals data is gone after upgrade")

View file

@ -79,12 +79,144 @@ def test_user_can_create_private_blueprint(user_client) -> None:
assert response.status_code == 201 assert response.status_code == 201
def _create_other_users_private_overlay() -> int:
with session_scope() as session:
other = User(username="mallory", password_digest=hash_password("secret"), admin=False)
session.add(other)
session.flush()
overlay = Overlay(
name="mallory-private",
path="mallory-private",
type="workshop",
user_id=other.id,
)
session.add(overlay)
session.flush()
return overlay.id
def test_user_cannot_create_blueprint_with_other_users_private_overlay(user_client) -> None:
foreign_overlay_id = _create_other_users_private_overlay()
payload = {
"name": "bad",
"arguments": [],
"config": [],
"overlay_ids": [foreign_overlay_id],
}
response = user_client.post(
"/blueprints",
data=json.dumps(payload),
content_type="application/json",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 403
def test_user_cannot_update_blueprint_with_other_users_private_overlay(user_client) -> None:
foreign_overlay_id = _create_other_users_private_overlay()
create = user_client.post(
"/blueprints",
data={"name": "comp", "arguments": "", "config": "", "overlay_ids": ["1"]},
headers={"X-CSRF-Token": "test-token"},
)
assert create.status_code == 302
response = user_client.post(
"/blueprints/1",
data={
"name": "edited",
"arguments": "",
"config": "",
"overlay_ids": [str(foreign_overlay_id)],
},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 403
def test_user_can_create_blueprint_with_system_overlay(user_client) -> None:
payload = {
"name": "system-ok",
"arguments": [],
"config": [],
"overlay_ids": [1],
}
response = user_client.post(
"/blueprints",
data=json.dumps(payload),
content_type="application/json",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 201
blueprint_id = response.get_json()["id"]
with session_scope() as session:
link = session.query(BlueprintOverlay).filter_by(blueprint_id=blueprint_id, overlay_id=1).one()
assert link.position == 0
def test_delete_blueprint_blocked_when_in_use(linked_blueprint) -> None: def test_delete_blueprint_blocked_when_in_use(linked_blueprint) -> None:
client, blueprint_id = linked_blueprint client, blueprint_id = linked_blueprint
response = client.delete(f"/blueprints/{blueprint_id}", headers={"X-CSRF-Token": "test-token"}) response = client.delete(f"/blueprints/{blueprint_id}", headers={"X-CSRF-Token": "test-token"})
assert response.status_code == 409 assert response.status_code == 409
def test_post_delete_blueprint_redirects_to_index(user_client) -> None:
create = user_client.post(
"/blueprints",
data={"name": "doomed", "arguments": "", "config": ""},
headers={"X-CSRF-Token": "test-token"},
)
assert create.status_code == 302
from sqlalchemy import select
from l4d2web.models import Blueprint as BlueprintModel
with session_scope() as session:
blueprint_id = session.scalars(select(BlueprintModel.id)).one()
response = user_client.post(
f"/blueprints/{blueprint_id}/delete",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
assert response.headers["Location"] == "/blueprints"
with session_scope() as session:
assert session.scalars(select(BlueprintModel)).all() == []
def test_post_delete_blueprint_blocked_when_in_use(linked_blueprint) -> None:
client, blueprint_id = linked_blueprint
response = client.post(
f"/blueprints/{blueprint_id}/delete",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 409
def test_post_delete_blueprint_returns_404_for_other_user(user_client, tmp_path) -> None:
with session_scope() as session:
other = User(username="mallory", password_digest=hash_password("secret"), admin=False)
session.add(other)
session.flush()
foreign = Blueprint(user_id=other.id, name="foreign", arguments="[]", config="[]")
session.add(foreign)
session.flush()
foreign_id = foreign.id
response = user_client.post(
f"/blueprints/{foreign_id}/delete",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 404
def test_form_update_preserves_ordered_overlays_and_multiline_fields(user_client) -> None: def test_form_update_preserves_ordered_overlays_and_multiline_fields(user_client) -> None:
create = user_client.post( create = user_client.post(
"/blueprints", "/blueprints",

View file

@ -53,6 +53,18 @@ def test_session_cookie_secure_in_production(tmp_path, monkeypatch) -> None:
assert app.config["SESSION_COOKIE_SECURE"] is True assert app.config["SESSION_COOKIE_SECURE"] is True
def test_session_cookie_secure_env_override(tmp_path, monkeypatch) -> None:
db_url = f"sqlite:///{tmp_path/'cookie-env.db'}"
monkeypatch.setenv("DATABASE_URL", db_url)
monkeypatch.setenv("SESSION_COOKIE_SECURE", "false")
monkeypatch.setattr("l4d2web.app.recover_stale_jobs", lambda: None)
monkeypatch.setattr("l4d2web.app.start_job_workers", lambda app: None)
app = create_app({"TESTING": False, "DATABASE_URL": db_url, "SECRET_KEY": "real"})
assert app.config["SESSION_COOKIE_SECURE"] is False
def test_session_cookie_secure_disabled_in_testing(tmp_path, monkeypatch) -> None: def test_session_cookie_secure_disabled_in_testing(tmp_path, monkeypatch) -> None:
db_url = f"sqlite:///{tmp_path/'cookie-test.db'}" db_url = f"sqlite:///{tmp_path/'cookie-test.db'}"
monkeypatch.setenv("DATABASE_URL", db_url) monkeypatch.setenv("DATABASE_URL", db_url)

View file

@ -72,3 +72,46 @@ def test_stream_command_yields_stdout_lines() -> None:
lines = list(stream_command(["python3", "-c", "print('one'); print('two')"])) lines = list(stream_command(["python3", "-c", "print('one'); print('two')"]))
assert lines == ["one", "two"] assert lines == ["one", "two"]
def test_stream_command_emits_heartbeat_when_subprocess_silent() -> None:
import time
from l4d2web.services.host_commands import stream_command
cmd = [
"python3",
"-c",
"import time; time.sleep(0.4); print('done')",
]
started = time.monotonic()
items: list[str] = []
for item in stream_command(cmd, heartbeat_interval=0.05):
items.append(item)
if time.monotonic() - started > 2.0:
break
assert "done" in items, items
heartbeats = [i for i in items if i == ""]
assert len(heartbeats) >= 2, f"expected ≥2 heartbeat ticks during the silent 0.4s window, got items={items!r}"
def test_stream_command_close_releases_subprocess_promptly() -> None:
import time
from l4d2web.services.host_commands import stream_command
cmd = [
"python3",
"-c",
"import time;\nwhile True:\n time.sleep(60)",
]
gen = stream_command(cmd, heartbeat_interval=0.05)
assert next(gen) == ""
started = time.monotonic()
gen.close()
elapsed = time.monotonic() - started
assert elapsed < 1.0, f"gen.close() took {elapsed:.2f}s; subprocess cleanup must not block"

View file

@ -104,3 +104,28 @@ def test_sse_js_handles_job_log_custom_events() -> None:
assert 'addEventListener("stdout"' in js assert 'addEventListener("stdout"' in js
assert 'addEventListener("stderr"' in js assert 'addEventListener("stderr"' in js
def test_system_job_logs_persist(tmp_path, monkeypatch):
from l4d2web.models import Job, JobLog
from l4d2web.services.job_worker import append_job_log
monkeypatch.setenv("DATABASE_URL", f"sqlite:///{tmp_path/'system-job-logs.db'}")
init_db()
with session_scope() as db:
job = Job(
user_id=None,
server_id=None,
operation="refresh_workshop_items",
state="queued",
)
db.add(job)
db.flush()
seq = append_job_log(db, job.id, "stdout", "queued by system timer")
db.flush()
row = db.query(JobLog).filter_by(job_id=job.id).one()
assert seq == 1
assert row.line == "queued by system timer"

View file

@ -8,16 +8,33 @@ from sqlalchemy import select
from l4d2web.auth import hash_password from l4d2web.auth import hash_password
from l4d2web.db import init_db, session_scope from l4d2web.db import init_db, session_scope
from l4d2web.models import Blueprint, Job, Server, User from l4d2web.models import (
Blueprint,
BlueprintOverlay,
Job,
Overlay,
OverlayWorkshopItem,
Server,
User,
WorkshopItem,
)
from l4d2web.services import l4d2_facade from l4d2web.services import l4d2_facade
from l4d2web.services.host_commands import CommandCancelledError from l4d2web.services.host_commands import CommandCancelledError
from l4d2web.services.job_worker import SchedulerState, can_start, recover_stale_jobs, run_worker_once from l4d2web.services.job_worker import (
SchedulerState,
build_scheduler_state,
can_start,
enqueue_build_overlay,
recover_stale_jobs,
run_worker_once,
)
@dataclass @dataclass
class DummyJob: class DummyJob:
operation: str operation: str
server_id: int | None = None server_id: int | None = None
overlay_id: int | None = None
@pytest.fixture @pytest.fixture
@ -65,12 +82,14 @@ def add_job(
server_id: int | None, server_id: int | None,
state: str = "queued", state: str = "queued",
created_at: datetime | None = None, created_at: datetime | None = None,
overlay_id: int | None = None,
) -> int: ) -> int:
now = datetime.now(UTC) now = datetime.now(UTC)
with session_scope() as session: with session_scope() as session:
job = Job( job = Job(
user_id=user_id, user_id=user_id,
server_id=server_id, server_id=server_id,
overlay_id=overlay_id,
operation=operation, operation=operation,
state=state, state=state,
created_at=created_at or now, created_at=created_at or now,
@ -241,6 +260,71 @@ def test_unexpected_exception_fails_job_with_exit_code_one(seeded_worker, monkey
job = load_job(job_id) job = load_job(job_id)
assert job.state == "failed" assert job.state == "failed"
assert job.exit_code == 1 assert job.exit_code == 1
# Failed delete must keep the Server row.
with session_scope() as session:
assert session.scalar(select(Server).where(Server.id == ids.server_one)) is not None
def test_successful_reset_keeps_server_row_and_refreshes_state(seeded_worker, monkeypatch) -> None:
app, ids = seeded_worker
job_id = add_job(ids.user, "reset", server_id=ids.server_one)
calls = []
def fake_reset(server_id, *, on_stdout=None, on_stderr=None, should_cancel=None):
del should_cancel
calls.append(server_id)
on_stdout("removing instance files")
monkeypatch.setattr(l4d2_facade, "reset_server", fake_reset)
monkeypatch.setattr(l4d2_facade, "server_status", lambda name: SimpleNamespace(state="stopped"))
with app.app_context():
assert run_worker_once() is True
assert calls == [ids.server_one]
job = load_job(job_id)
assert job.state == "succeeded"
assert job.exit_code == 0
with session_scope() as session:
server = session.scalar(select(Server).where(Server.id == ids.server_one))
assert server is not None
assert server.actual_state == "stopped"
def test_successful_delete_removes_server_row(seeded_worker, monkeypatch) -> None:
app, ids = seeded_worker
job_id = add_job(ids.user, "delete", server_id=ids.server_one)
calls = []
def fake_delete(server_id, *, on_stdout=None, on_stderr=None, should_cancel=None):
del should_cancel
calls.append(server_id)
on_stdout("removing systemd unit")
def fake_status(name): # pragma: no cover — would mean delete didn't skip refresh
pytest.fail(f"server_status must not be called after a successful delete (got {name!r})")
monkeypatch.setattr(l4d2_facade, "delete_server", fake_delete)
monkeypatch.setattr(l4d2_facade, "server_status", fake_status)
with app.app_context():
assert run_worker_once() is True
assert calls == [ids.server_one]
job = load_job(job_id)
assert job.state == "succeeded"
assert job.exit_code == 0
with session_scope() as session:
assert session.scalar(select(Server).where(Server.id == ids.server_one)) is None
# Sibling server is untouched.
assert session.scalar(select(Server).where(Server.id == ids.server_two)) is not None
# The delete job itself stays in the job log; outerjoin in views shows
# "-" for its (now-orphaned) server_id pointer.
finished_job = session.scalar(select(Job).where(Job.id == job_id))
assert finished_job is not None
assert finished_job.server_id == ids.server_one
def test_same_server_jobs_do_not_overlap(seeded_worker, monkeypatch) -> None: def test_same_server_jobs_do_not_overlap(seeded_worker, monkeypatch) -> None:
@ -379,3 +463,383 @@ def test_worker_startup_when_enabled_outside_testing(monkeypatch, tmp_path) -> N
app = app_module.create_app({"TESTING": False, "DATABASE_URL": db_url, "SECRET_KEY": "test"}) app = app_module.create_app({"TESTING": False, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
assert called == [app] assert called == [app]
# ---------------------------------------------------------------------------
# Scheduler truth table for the new operations (build_overlay,
# refresh_workshop_items) and their interaction with existing ops.
# ---------------------------------------------------------------------------
def test_install_blocks_build_overlay_and_refresh() -> None:
state = SchedulerState(install_running=True)
assert can_start(DummyJob(operation="build_overlay", overlay_id=1), state) is False
assert can_start(DummyJob(operation="refresh_workshop_items"), state) is False
def test_refresh_blocks_install_build_overlay_and_servers() -> None:
state = SchedulerState(refresh_running=True)
assert can_start(DummyJob(operation="install"), state) is False
assert can_start(DummyJob(operation="build_overlay", overlay_id=1), state) is False
assert can_start(DummyJob(operation="start", server_id=1), state) is False
def test_build_overlay_blocks_same_overlay_only() -> None:
state = SchedulerState()
state.running_overlays.add(7)
assert can_start(DummyJob(operation="build_overlay", overlay_id=7), state) is False
assert can_start(DummyJob(operation="build_overlay", overlay_id=8), state) is True
def test_install_blocked_by_active_build_overlay() -> None:
state = SchedulerState()
state.running_overlays.add(7)
assert can_start(DummyJob(operation="install"), state) is False
def test_refresh_blocked_by_active_build_overlay() -> None:
state = SchedulerState()
state.running_overlays.add(7)
assert can_start(DummyJob(operation="refresh_workshop_items"), state) is False
def test_server_job_blocked_when_blueprint_overlay_is_building() -> None:
state = SchedulerState()
state.running_overlays.add(7)
state.blocked_servers_by_overlay.add(42)
assert can_start(DummyJob(operation="start", server_id=42), state) is False
# Other servers (whose blueprints don't reference overlay 7) are NOT blocked.
assert can_start(DummyJob(operation="start", server_id=43), state) is True
@pytest.fixture
def overlay_seeded_worker(seeded_worker):
app, ids = seeded_worker
with session_scope() as s:
overlay = Overlay(name="ws", path="9", type="workshop", user_id=ids.user)
s.add(overlay)
s.flush()
# Move server_two onto a different blueprint with NO workshop overlay,
# so the test can distinguish "blocked by overlay build" from "any
# server is blocked".
bp_with_overlay = s.scalar(select(Blueprint).where(Blueprint.user_id == ids.user))
s.add(BlueprintOverlay(blueprint_id=bp_with_overlay.id, overlay_id=overlay.id, position=0))
bp_without = Blueprint(user_id=ids.user, name="no-overlay", arguments="[]", config="[]")
s.add(bp_without)
s.flush()
server_two = s.scalar(select(Server).where(Server.id == ids.server_two))
server_two.blueprint_id = bp_without.id
ids.overlay = overlay.id
return app, ids
def test_scheduler_state_finds_servers_blocked_by_running_build(overlay_seeded_worker) -> None:
app, ids = overlay_seeded_worker
add_job(ids.user, "build_overlay", server_id=None, state="running", overlay_id=ids.overlay)
with session_scope() as s:
state = build_scheduler_state(s)
assert ids.overlay in state.running_overlays
assert ids.server_one in state.blocked_servers_by_overlay
assert ids.server_two not in state.blocked_servers_by_overlay
def test_enqueue_build_overlay_creates_new_job_when_none_pending(overlay_seeded_worker) -> None:
app, ids = overlay_seeded_worker
with session_scope() as s:
job = enqueue_build_overlay(s, overlay_id=ids.overlay, user_id=ids.user)
assert job.operation == "build_overlay"
assert job.overlay_id == ids.overlay
assert job.server_id is None
assert job.state == "queued"
def test_enqueue_build_overlay_coalesces_against_pending(overlay_seeded_worker) -> None:
app, ids = overlay_seeded_worker
with session_scope() as s:
first = enqueue_build_overlay(s, overlay_id=ids.overlay, user_id=ids.user)
first_id = first.id
with session_scope() as s:
second = enqueue_build_overlay(s, overlay_id=ids.overlay, user_id=ids.user)
assert second.id == first_id, "should coalesce against the pending job"
with session_scope() as s:
n = s.query(Job).filter_by(operation="build_overlay", overlay_id=ids.overlay).count()
assert n == 1
def test_enqueue_build_overlay_does_not_coalesce_against_running(overlay_seeded_worker) -> None:
app, ids = overlay_seeded_worker
add_job(ids.user, "build_overlay", server_id=None, state="running", overlay_id=ids.overlay)
with session_scope() as s:
new_job = enqueue_build_overlay(s, overlay_id=ids.overlay, user_id=ids.user)
assert new_job.state == "queued"
with session_scope() as s:
running = s.scalars(
select(Job).where(
Job.operation == "build_overlay",
Job.overlay_id == ids.overlay,
Job.state == "running",
)
).all()
queued = s.scalars(
select(Job).where(
Job.operation == "build_overlay",
Job.overlay_id == ids.overlay,
Job.state == "queued",
)
).all()
assert len(running) == 1
assert len(queued) == 1
def test_run_worker_once_dispatches_build_overlay(overlay_seeded_worker, monkeypatch, tmp_path) -> None:
app, ids = overlay_seeded_worker
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
with session_scope() as s:
wi = WorkshopItem(steam_id="1001", title="A", filename="a.vpk", file_url="u", file_size=3, time_updated=1)
s.add(wi)
s.flush()
s.add(OverlayWorkshopItem(overlay_id=ids.overlay, workshop_item_id=wi.id))
cache = tmp_path / "workshop_cache"
cache.mkdir()
(cache / "1001.vpk").write_bytes(b"abc")
import os
os.utime(cache / "1001.vpk", (1, 1))
# Mark item as downloaded.
with session_scope() as s:
wi = s.query(WorkshopItem).filter_by(steam_id="1001").one()
wi.last_downloaded_at = datetime.now(UTC)
job_id = add_job(ids.user, "build_overlay", server_id=None, overlay_id=ids.overlay)
with app.app_context():
assert run_worker_once() is True
job = load_job(job_id)
assert job.state == "succeeded", job.exit_code
addons = tmp_path / "overlays" / "9" / "left4dead2" / "addons"
assert (addons / "1001.vpk").is_symlink()
def test_build_overlay_writes_last_build_status_ok(
overlay_seeded_worker, monkeypatch, tmp_path
) -> None:
app, ids = overlay_seeded_worker
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
from l4d2web.services import overlay_builders
class _StubBuilder:
def build(self, overlay, *, on_stdout, on_stderr, should_cancel):
on_stdout("stub build ok")
monkeypatch.setitem(overlay_builders.BUILDERS, "workshop", _StubBuilder())
job_id = add_job(ids.user, "build_overlay", server_id=None, overlay_id=ids.overlay)
with app.app_context():
assert run_worker_once() is True
assert load_job(job_id).state == "succeeded"
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=ids.overlay).one()
assert overlay.last_build_status == "ok"
def test_build_overlay_writes_last_build_status_failed(
overlay_seeded_worker, monkeypatch, tmp_path
) -> None:
app, ids = overlay_seeded_worker
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
from l4d2web.services import overlay_builders
class _FailingBuilder:
def build(self, overlay, *, on_stdout, on_stderr, should_cancel):
raise RuntimeError("synthetic build failure")
monkeypatch.setitem(overlay_builders.BUILDERS, "workshop", _FailingBuilder())
job_id = add_job(ids.user, "build_overlay", server_id=None, overlay_id=ids.overlay)
with app.app_context():
assert run_worker_once() is True
assert load_job(job_id).state == "failed"
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=ids.overlay).one()
assert overlay.last_build_status == "failed"
def test_run_worker_once_dispatches_refresh(overlay_seeded_worker, monkeypatch, tmp_path) -> None:
app, ids = overlay_seeded_worker
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
from l4d2web.services import steam_workshop, job_worker
refresh_calls = []
def fake_refresh_workshop_items(*, on_stdout, on_stderr, should_cancel):
refresh_calls.append(True)
on_stdout("refresh phase complete (downloaded=0 errors=0)")
return [] # no overlays affected
monkeypatch.setattr(job_worker, "_run_refresh_workshop_items", fake_refresh_workshop_items)
job_id = add_job(ids.user, "refresh_workshop_items", server_id=None)
with app.app_context():
assert run_worker_once() is True
assert refresh_calls == [True]
job = load_job(job_id)
assert job.state == "succeeded"
def test_refresh_downloads_serially_to_keep_web_worker_responsive(
overlay_seeded_worker, monkeypatch, tmp_path
) -> None:
app, ids = overlay_seeded_worker
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
from l4d2web.services import job_worker, steam_workshop
with session_scope() as s:
wi = WorkshopItem(
steam_id="1001",
title="A",
filename="old.vpk",
file_url="https://example.com/old.vpk",
file_size=1,
time_updated=1,
)
s.add(wi)
s.flush()
s.add(OverlayWorkshopItem(overlay_id=ids.overlay, workshop_item_id=wi.id))
meta = steam_workshop.WorkshopMetadata(
steam_id="1001",
title="A",
filename="new.vpk",
file_url="https://example.com/new.vpk",
file_size=2,
time_updated=2,
preview_url="",
consumer_app_id=550,
result=1,
)
monkeypatch.setattr(steam_workshop, "fetch_metadata_batch", lambda steam_ids, *, mode: [meta])
refresh_calls = []
def fake_refresh_all(metas, cache_root, *, executor_workers=8, should_cancel=None):
refresh_calls.append((list(metas), executor_workers, should_cancel is not None))
return steam_workshop.RefreshReport(downloaded=1, errors=0)
monkeypatch.setattr(steam_workshop, "refresh_all", fake_refresh_all)
with app.app_context():
affected = job_worker._run_refresh_workshop_items(
on_stdout=lambda _line: None,
on_stderr=lambda _line: None,
should_cancel=lambda: False,
)
assert affected == [ids.overlay]
assert refresh_calls == [([meta], 1, True)]
def test_refresh_job_enqueues_build_overlay_without_locking_its_final_log(
overlay_seeded_worker, monkeypatch, tmp_path
) -> None:
app, ids = overlay_seeded_worker
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
from l4d2web.services import steam_workshop
with session_scope() as s:
wi = WorkshopItem(
steam_id="1001",
title="A",
filename="old.vpk",
file_url="https://example.com/old.vpk",
file_size=1,
time_updated=1,
)
s.add(wi)
s.flush()
s.add(OverlayWorkshopItem(overlay_id=ids.overlay, workshop_item_id=wi.id))
meta = steam_workshop.WorkshopMetadata(
steam_id="1001",
title="A",
filename="new.vpk",
file_url="https://example.com/new.vpk",
file_size=2,
time_updated=2,
preview_url="",
consumer_app_id=550,
result=1,
)
monkeypatch.setattr(steam_workshop, "fetch_metadata_batch", lambda steam_ids, *, mode: [meta])
monkeypatch.setattr(
steam_workshop,
"refresh_all",
lambda metas, cache_root, **kwargs: steam_workshop.RefreshReport(downloaded=1, errors=0),
)
job_id = add_job(ids.user, "refresh_workshop_items", server_id=None)
with app.app_context():
assert run_worker_once() is True
with session_scope() as s:
job = s.scalar(select(Job).where(Job.id == job_id))
build_job = s.scalar(
select(Job).where(
Job.operation == "build_overlay",
Job.overlay_id == ids.overlay,
Job.state == "queued",
)
)
lines = [row.line for row in job_logs_for(s, job_id)]
assert job is not None
assert job.state == "succeeded"
assert build_job is not None
assert "enqueued build_overlay for 1 overlay(s)" in lines
def test_global_operations_set() -> None:
from l4d2web.services.job_worker import GLOBAL_OPERATIONS
assert GLOBAL_OPERATIONS == {"install", "refresh_workshop_items"}
def test_build_overlay_script_type_blocks_per_overlay(overlay_seeded_worker) -> None:
"""Mechanically identical to workshop builds, but pinned for script type."""
app, ids = overlay_seeded_worker
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=ids.overlay).one()
overlay.type = "script"
overlay.script = "echo hi"
add_job(ids.user, "build_overlay", server_id=None, state="running", overlay_id=ids.overlay)
state = SchedulerState()
state.running_overlays.add(ids.overlay)
assert (
can_start(DummyJob(operation="build_overlay", overlay_id=ids.overlay), state)
is False
)
assert (
can_start(DummyJob(operation="build_overlay", overlay_id=ids.overlay + 1), state)
is True
)

View file

@ -1,3 +1,4 @@
from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
import pytest import pytest
@ -5,7 +6,15 @@ import pytest
from l4d2web.app import create_app from l4d2web.app import create_app
from l4d2web.auth import hash_password from l4d2web.auth import hash_password
from l4d2web.db import init_db, session_scope from l4d2web.db import init_db, session_scope
from l4d2web.models import Blueprint, BlueprintOverlay, Overlay, Server, User from l4d2web.models import (
Blueprint,
BlueprintOverlay,
Overlay,
OverlayWorkshopItem,
Server,
User,
WorkshopItem,
)
from l4d2web.services.host_commands import CommandResult from l4d2web.services.host_commands import CommandResult
@ -13,6 +22,7 @@ from l4d2web.services.host_commands import CommandResult
def server_with_blueprint(tmp_path, monkeypatch): def server_with_blueprint(tmp_path, monkeypatch):
db_url = f"sqlite:///{tmp_path/'facade.db'}" db_url = f"sqlite:///{tmp_path/'facade.db'}"
monkeypatch.setenv("DATABASE_URL", db_url) monkeypatch.setenv("DATABASE_URL", db_url)
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"}) app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db() init_db()
@ -22,7 +32,7 @@ def server_with_blueprint(tmp_path, monkeypatch):
session.add(user) session.add(user)
session.flush() session.flush()
overlay = Overlay(name="Standard Overlay", path="standard") overlay = Overlay(name="Standard Overlay", path="standard", type="workshop", user_id=user.id)
session.add(overlay) session.add(overlay)
session.flush() session.flush()
@ -41,8 +51,9 @@ def server_with_blueprint(tmp_path, monkeypatch):
session.add(server) session.add(server)
session.flush() session.flush()
server_id = server.id server_id = server.id
user_id = user.id
return server_id return server_id, user_id
def test_initialize_uses_l4d2ctl_with_latest_blueprint_data( def test_initialize_uses_l4d2ctl_with_latest_blueprint_data(
@ -70,7 +81,8 @@ def test_initialize_uses_l4d2ctl_with_latest_blueprint_data(
from l4d2web.services.l4d2_facade import initialize_server from l4d2web.services.l4d2_facade import initialize_server
initialize_server(server_with_blueprint) server_id, _ = server_with_blueprint
initialize_server(server_id)
assert calls[0][:3] == ["l4d2ctl", "initialize", "alpha"] assert calls[0][:3] == ["l4d2ctl", "initialize", "alpha"]
assert calls[0][3] == "-f" assert calls[0][3] == "-f"
@ -88,24 +100,33 @@ def test_install_and_lifecycle_commands_use_l4d2ctl(
return CommandResult(returncode=0, stdout="", stderr="") return CommandResult(returncode=0, stdout="", stderr="")
monkeypatch.setattr("l4d2web.services.host_commands.run_command", fake_run_command) monkeypatch.setattr("l4d2web.services.host_commands.run_command", fake_run_command)
for name in ["SteamInstaller", "start_instance", "stop_instance", "delete_instance"]: for name in ["SteamInstaller", "start_instance", "stop_instance", "delete_instance", "reset_instance"]:
monkeypatch.setattr( monkeypatch.setattr(
f"l4d2web.services.l4d2_facade.{name}", f"l4d2web.services.l4d2_facade.{name}",
lambda *args, **kwargs: pytest.fail(f"facade must not call l4d2host {name} directly"), lambda *args, **kwargs: pytest.fail(f"facade must not call l4d2host {name} directly"),
raising=False, raising=False,
) )
from l4d2web.services.l4d2_facade import delete_server, install_runtime, start_server, stop_server from l4d2web.services.l4d2_facade import (
delete_server,
install_runtime,
reset_server,
start_server,
stop_server,
)
server_id, _ = server_with_blueprint
install_runtime() install_runtime()
start_server(server_with_blueprint) start_server(server_id)
stop_server(server_with_blueprint) stop_server(server_id)
delete_server(server_with_blueprint) reset_server(server_id)
delete_server(server_id)
assert calls == [ assert calls == [
["l4d2ctl", "install"], ["l4d2ctl", "install"],
["l4d2ctl", "start", "alpha"], ["l4d2ctl", "start", "alpha"],
["l4d2ctl", "stop", "alpha"], ["l4d2ctl", "stop", "alpha"],
["l4d2ctl", "reset", "alpha"],
["l4d2ctl", "delete", "alpha"], ["l4d2ctl", "delete", "alpha"],
] ]
@ -159,3 +180,91 @@ def test_server_logs_stream_l4d2ctl_logs(monkeypatch: pytest.MonkeyPatch) -> Non
assert calls == [["l4d2ctl", "logs", "alpha", "--lines", "10", "--no-follow"]] assert calls == [["l4d2ctl", "logs", "alpha", "--lines", "10", "--no-follow"]]
assert lines == ["one", "two"] assert lines == ["one", "two"]
def _attach_workshop_overlay_to_blueprint(
server_id: int, user_id: int, *, item_cached: bool, tmp_path: Path
) -> tuple[int, str]:
"""Add a workshop overlay with a single workshop item to the server's
blueprint. Returns (overlay_id, steam_id)."""
with session_scope() as session:
server = session.query(Server).filter_by(id=server_id).one()
overlay = Overlay(name="ws", path="placeholder", type="workshop", user_id=user_id)
session.add(overlay)
session.flush()
# Path matches id, like the production create_overlay flow does.
overlay.path = str(overlay.id)
wi = WorkshopItem(
steam_id="1001",
title="A",
filename="a.vpk",
file_url="https://example.com/a.vpk",
file_size=3,
time_updated=1700000000,
last_downloaded_at=datetime.now(UTC) if item_cached else None,
)
session.add(wi)
session.flush()
session.add(
BlueprintOverlay(blueprint_id=server.blueprint_id, overlay_id=overlay.id, position=1)
)
session.add(OverlayWorkshopItem(overlay_id=overlay.id, workshop_item_id=wi.id))
overlay_id = overlay.id
if item_cached:
cache_root = tmp_path / "workshop_cache"
cache_root.mkdir(exist_ok=True)
(cache_root / "1001.vpk").write_bytes(b"abc")
return overlay_id, "1001"
def test_initialize_runs_overlay_builders_synchronously(
monkeypatch: pytest.MonkeyPatch, server_with_blueprint, tmp_path
) -> None:
server_id, user_id = server_with_blueprint
overlay_id, _steam_id = _attach_workshop_overlay_to_blueprint(
server_id, user_id, item_cached=True, tmp_path=tmp_path
)
monkeypatch.setattr(
"l4d2web.services.host_commands.run_command",
lambda *args, **kwargs: CommandResult(returncode=0, stdout="", stderr=""),
)
from l4d2web.services.l4d2_facade import initialize_server
initialize_server(server_id)
addons = tmp_path / "overlays" / str(overlay_id) / "left4dead2" / "addons"
assert (addons / "1001.vpk").is_symlink(), "workshop builder must run before l4d2ctl initialize"
def test_initialize_fails_fast_on_uncached_workshop_items(
monkeypatch: pytest.MonkeyPatch, server_with_blueprint, tmp_path
) -> None:
server_id, user_id = server_with_blueprint
overlay_id, steam_id = _attach_workshop_overlay_to_blueprint(
server_id, user_id, item_cached=False, tmp_path=tmp_path
)
invocations: list[list[str]] = []
def fake_run_command(cmd, **kwargs):
invocations.append(list(cmd))
return CommandResult(returncode=0, stdout="", stderr="")
monkeypatch.setattr("l4d2web.services.host_commands.run_command", fake_run_command)
from l4d2web.services.l4d2_facade import initialize_server
with pytest.raises(Exception) as excinfo:
initialize_server(server_id)
msg = str(excinfo.value)
assert steam_id in msg
assert str(overlay_id) in msg or "ws" in msg
# l4d2ctl initialize MUST NOT run when uncached items are present.
assert all("initialize" not in cmd for cmd in invocations), invocations

View file

@ -0,0 +1,359 @@
"""Tests for overlay builders (registry, WorkshopBuilder, ScriptBuilder)."""
from __future__ import annotations
import os
from datetime import UTC, datetime
from pathlib import Path
from types import SimpleNamespace
import pytest
from l4d2web.db import init_db, session_scope
from l4d2web.models import Overlay, OverlayWorkshopItem, User, WorkshopItem
from l4d2web.services import overlay_builders
from l4d2web.services.host_commands import CommandCancelledError, CommandResult
@pytest.fixture
def env(tmp_path, monkeypatch):
monkeypatch.setenv("DATABASE_URL", f"sqlite:///{tmp_path/'b.db'}")
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
init_db()
yield tmp_path
def _create_user_and_overlay(name: str, type_: str) -> tuple[int, int]:
with session_scope() as s:
user = User(username="alice", password_digest="x")
s.add(user)
s.flush()
overlay = Overlay(name=name, path=str(7), type=type_, user_id=user.id)
s.add(overlay)
s.flush()
return user.id, overlay.id
def _add_workshop_item(steam_id: str, *, downloaded: bool, cache_root: Path, content: bytes = b"x") -> int:
if downloaded:
cache_root.mkdir(parents=True, exist_ok=True)
(cache_root / f"{steam_id}.vpk").write_bytes(content)
with session_scope() as s:
wi = WorkshopItem(
steam_id=steam_id,
title=f"item-{steam_id}",
filename=f"orig-{steam_id}.vpk",
file_url=f"https://example.com/{steam_id}.vpk",
file_size=len(content) if downloaded else 0,
time_updated=1700000000 if downloaded else 0,
last_downloaded_at=datetime.now(UTC) if downloaded else None,
)
s.add(wi)
s.flush()
return wi.id
def _associate(overlay_id: int, item_id: int) -> None:
with session_scope() as s:
s.add(OverlayWorkshopItem(overlay_id=overlay_id, workshop_item_id=item_id))
def _capture_logs():
out: list[str] = []
err: list[str] = []
return out, err, out.append, err.append
def test_builders_registry() -> None:
assert set(overlay_builders.BUILDERS) == {"workshop", "script"}
def test_registry_excludes_legacy_types() -> None:
for legacy in ("external", "l4d2center_maps", "cedapug_maps"):
assert legacy not in overlay_builders.BUILDERS
def test_registry_unknown_type_raises_keyerror() -> None:
with pytest.raises(KeyError):
overlay_builders.BUILDERS["nope"]
def test_workshop_builder_unchanged() -> None:
"""Regression guard against accidental removal during refactor."""
builder = overlay_builders.BUILDERS["workshop"]
assert isinstance(builder, overlay_builders.WorkshopBuilder)
assert hasattr(builder, "build")
def test_workshop_builder_creates_absolute_symlinks(env: Path) -> None:
_, overlay_id = _create_user_and_overlay("ws", "workshop")
cache_root = env / "workshop_cache"
item_a = _add_workshop_item("1001", downloaded=True, cache_root=cache_root, content=b"AAA")
item_b = _add_workshop_item("1002", downloaded=True, cache_root=cache_root, content=b"BBBB")
_associate(overlay_id, item_a)
_associate(overlay_id, item_b)
out, err, on_stdout, on_stderr = _capture_logs()
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
overlay_builders.BUILDERS["workshop"].build(
overlay, on_stdout=on_stdout, on_stderr=on_stderr, should_cancel=lambda: False
)
addons = env / "overlays" / "7" / "left4dead2" / "addons"
link_a = addons / "1001.vpk"
link_b = addons / "1002.vpk"
assert link_a.is_symlink()
assert link_b.is_symlink()
# Targets must be ABSOLUTE so they resolve in the host's namespace.
assert os.path.isabs(os.readlink(link_a))
assert os.path.isabs(os.readlink(link_b))
# And they must resolve to the cache files.
assert link_a.resolve() == (cache_root / "1001.vpk").resolve()
assert link_b.resolve() == (cache_root / "1002.vpk").resolve()
def test_workshop_builder_skips_uncached_items_with_warning(env: Path) -> None:
_, overlay_id = _create_user_and_overlay("ws", "workshop")
cache_root = env / "workshop_cache"
cached = _add_workshop_item("1001", downloaded=True, cache_root=cache_root)
uncached = _add_workshop_item("9999", downloaded=False, cache_root=cache_root)
_associate(overlay_id, cached)
_associate(overlay_id, uncached)
out, err, on_stdout, on_stderr = _capture_logs()
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
overlay_builders.BUILDERS["workshop"].build(
overlay, on_stdout=on_stdout, on_stderr=on_stderr, should_cancel=lambda: False
)
addons = env / "overlays" / "7" / "left4dead2" / "addons"
assert (addons / "1001.vpk").is_symlink()
assert not (addons / "9999.vpk").exists(), "must NOT create dangling symlink"
assert any("9999" in line and ("skip" in line.lower() or "uncached" in line.lower()) for line in err + out), err + out
def test_workshop_builder_rerun_is_idempotent(env: Path) -> None:
_, overlay_id = _create_user_and_overlay("ws", "workshop")
cache_root = env / "workshop_cache"
item = _add_workshop_item("1001", downloaded=True, cache_root=cache_root)
_associate(overlay_id, item)
addons = env / "overlays" / "7" / "left4dead2" / "addons"
# First run.
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
overlay_builders.BUILDERS["workshop"].build(overlay, on_stdout=lambda _x: None, on_stderr=lambda _x: None, should_cancel=lambda: False)
first_inode = (addons / "1001.vpk").lstat().st_ino
# Second run — no-op.
out, err, on_stdout, on_stderr = _capture_logs()
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
overlay_builders.BUILDERS["workshop"].build(overlay, on_stdout=on_stdout, on_stderr=on_stderr, should_cancel=lambda: False)
second_inode = (addons / "1001.vpk").lstat().st_ino
assert first_inode == second_inode, "symlink should be untouched on idempotent rebuild"
assert any("unchanged" in line.lower() for line in out), out
def test_workshop_builder_removes_obsolete_symlinks(env: Path) -> None:
_, overlay_id = _create_user_and_overlay("ws", "workshop")
cache_root = env / "workshop_cache"
item_a = _add_workshop_item("1001", downloaded=True, cache_root=cache_root)
item_b = _add_workshop_item("1002", downloaded=True, cache_root=cache_root)
_associate(overlay_id, item_a)
_associate(overlay_id, item_b)
addons = env / "overlays" / "7" / "left4dead2" / "addons"
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
overlay_builders.BUILDERS["workshop"].build(overlay, on_stdout=lambda _x: None, on_stderr=lambda _x: None, should_cancel=lambda: False)
assert (addons / "1002.vpk").is_symlink()
# Remove the association for 1002.
with session_scope() as s:
s.query(OverlayWorkshopItem).filter_by(workshop_item_id=item_b).delete()
out, err, on_stdout, on_stderr = _capture_logs()
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
overlay_builders.BUILDERS["workshop"].build(overlay, on_stdout=on_stdout, on_stderr=on_stderr, should_cancel=lambda: False)
assert (addons / "1001.vpk").is_symlink()
assert not (addons / "1002.vpk").exists()
# Cache file must remain — overlays are diff-applied, cache is shared.
assert (cache_root / "1002.vpk").exists()
def test_workshop_builder_leaves_unrelated_files_alone(env: Path) -> None:
_, overlay_id = _create_user_and_overlay("ws", "workshop")
cache_root = env / "workshop_cache"
item = _add_workshop_item("1001", downloaded=True, cache_root=cache_root)
_associate(overlay_id, item)
addons = env / "overlays" / "7" / "left4dead2" / "addons"
addons.mkdir(parents=True, exist_ok=True)
(addons / "manual_addon.vpk").write_bytes(b"hand-placed")
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
overlay_builders.BUILDERS["workshop"].build(overlay, on_stdout=lambda _x: None, on_stderr=lambda _x: None, should_cancel=lambda: False)
# Manual file is preserved.
assert (addons / "manual_addon.vpk").read_bytes() == b"hand-placed"
# Workshop symlink is created alongside.
assert (addons / "1001.vpk").is_symlink()
def test_workshop_builder_honors_should_cancel(env: Path) -> None:
_, overlay_id = _create_user_and_overlay("ws", "workshop")
cache_root = env / "workshop_cache"
items = [_add_workshop_item(f"100{i}", downloaded=True, cache_root=cache_root) for i in range(3)]
for it in items:
_associate(overlay_id, it)
cancel_calls = {"n": 0}
def cancel():
cancel_calls["n"] += 1
return cancel_calls["n"] > 0 # cancel immediately
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
# Should not crash; partial state is consistent (re-run heals).
overlay_builders.BUILDERS["workshop"].build(
overlay, on_stdout=lambda _x: None, on_stderr=lambda _x: None, should_cancel=cancel
)
# --- ScriptBuilder ---------------------------------------------------------
def _script_overlay(*, id_: int = 42, script: str = "echo hi") -> SimpleNamespace:
return SimpleNamespace(id=id_, type="script", path=str(id_), script=script)
def test_script_builder_invokes_helper(env, monkeypatch) -> None:
captured: dict = {}
def fake_run(cmd, *, on_stdout, on_stderr, should_cancel):
captured["cmd"] = list(cmd)
captured["script_text"] = open(cmd[-1]).read()
captured["script_path_existed"] = os.path.exists(cmd[-1])
return CommandResult(returncode=0, stdout="", stderr="")
monkeypatch.setattr(overlay_builders, "run_command", fake_run)
monkeypatch.setattr(
overlay_builders.ScriptBuilder, "_enforce_disk_budget", lambda *a, **kw: None
)
overlay = _script_overlay()
overlay_builders.ScriptBuilder().build(
overlay,
on_stdout=lambda _x: None,
on_stderr=lambda _x: None,
should_cancel=lambda: False,
)
assert captured["cmd"][:4] == [
"sudo",
"-n",
"/usr/local/libexec/left4me/left4me-script-sandbox",
"42",
]
assert captured["script_text"] == "echo hi"
assert captured["script_path_existed"] is True
# Tmpfile is unlinked after build.
assert not os.path.exists(captured["cmd"][-1])
def test_script_builder_disk_cap(env, monkeypatch) -> None:
monkeypatch.setattr(
overlay_builders,
"run_command",
lambda *a, **kw: CommandResult(returncode=0, stdout="", stderr=""),
)
monkeypatch.setattr(
overlay_builders.subprocess,
"check_output",
lambda *a, **kw: b"25000000000\t/some/path\n",
)
err: list[str] = []
overlay = _script_overlay(script="")
with pytest.raises(overlay_builders.BuildError):
overlay_builders.ScriptBuilder().build(
overlay,
on_stdout=lambda _x: None,
on_stderr=err.append,
should_cancel=lambda: False,
)
assert any("20" in line and "GB" in line for line in err), err
def test_script_builder_streams_output(env, monkeypatch) -> None:
def fake_run(cmd, *, on_stdout, on_stderr, should_cancel):
on_stdout("hello")
on_stderr("warn")
return CommandResult(returncode=0, stdout="hello", stderr="warn")
monkeypatch.setattr(overlay_builders, "run_command", fake_run)
monkeypatch.setattr(
overlay_builders.ScriptBuilder, "_enforce_disk_budget", lambda *a, **kw: None
)
out: list[str] = []
err: list[str] = []
overlay = _script_overlay(script="")
overlay_builders.ScriptBuilder().build(
overlay, on_stdout=out.append, on_stderr=err.append, should_cancel=lambda: False
)
assert out == ["hello"]
assert err == ["warn"]
def test_script_builder_passes_should_cancel_through(env, monkeypatch) -> None:
captured: dict = {}
def fake_run(cmd, *, on_stdout, on_stderr, should_cancel):
captured["should_cancel"] = should_cancel
raise CommandCancelledError(returncode=1, cmd=cmd, output="", stderr="")
monkeypatch.setattr(overlay_builders, "run_command", fake_run)
monkeypatch.setattr(
overlay_builders.ScriptBuilder, "_enforce_disk_budget", lambda *a, **kw: None
)
overlay = _script_overlay(script="")
with pytest.raises(CommandCancelledError):
overlay_builders.ScriptBuilder().build(
overlay,
on_stdout=lambda _x: None,
on_stderr=lambda _x: None,
should_cancel=lambda: True,
)
assert captured["should_cancel"]() is True
def test_script_builder_cleans_up_tmpfile_on_failure(env, monkeypatch) -> None:
captured: dict = {}
def fake_run(cmd, *, on_stdout, on_stderr, should_cancel):
captured["script_path"] = cmd[-1]
raise CommandCancelledError(returncode=1, cmd=cmd, output="", stderr="")
monkeypatch.setattr(overlay_builders, "run_command", fake_run)
overlay = _script_overlay(script="")
with pytest.raises(CommandCancelledError):
overlay_builders.ScriptBuilder().build(
overlay,
on_stdout=lambda _x: None,
on_stderr=lambda _x: None,
should_cancel=lambda: False,
)
assert not os.path.exists(captured["script_path"])

View file

@ -0,0 +1,35 @@
"""Tests for overlay path generation and directory creation."""
from pathlib import Path
import pytest
from l4d2web.models import Overlay
from l4d2web.services import overlay_creation
def test_generate_overlay_path_returns_str_id() -> None:
assert overlay_creation.generate_overlay_path(42) == "42"
def test_generate_overlay_path_validates_through_overlay_ref(monkeypatch) -> None:
# Sanity: numeric paths pass validate_overlay_ref. Anything bizarre would raise.
assert overlay_creation.generate_overlay_path(1) == "1"
def test_create_overlay_directory_makes_subtree(monkeypatch, tmp_path: Path) -> None:
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
overlay = Overlay(id=7, name="test", path="7", type="workshop", user_id=None)
overlay_creation.create_overlay_directory(overlay)
expected = tmp_path / "overlays" / "7"
assert expected.is_dir()
def test_create_overlay_directory_raises_if_already_exists(
monkeypatch, tmp_path: Path
) -> None:
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
overlay = Overlay(id=7, name="test", path="7", type="workshop", user_id=None)
(tmp_path / "overlays" / "7").mkdir(parents=True)
# exist_ok=False guards against a stray directory shadowing fresh content.
with pytest.raises(FileExistsError):
overlay_creation.create_overlay_directory(overlay)

View file

@ -10,6 +10,7 @@ from l4d2web.services.security import validate_overlay_ref
def admin_client(tmp_path, monkeypatch): def admin_client(tmp_path, monkeypatch):
db_url = f"sqlite:///{tmp_path/'admin_overlay.db'}" db_url = f"sqlite:///{tmp_path/'admin_overlay.db'}"
monkeypatch.setenv("DATABASE_URL", db_url) monkeypatch.setenv("DATABASE_URL", db_url)
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"}) app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db() init_db()
@ -30,13 +31,17 @@ def admin_client(tmp_path, monkeypatch):
def user_client_with_overlay(tmp_path, monkeypatch): def user_client_with_overlay(tmp_path, monkeypatch):
db_url = f"sqlite:///{tmp_path/'user_overlay.db'}" db_url = f"sqlite:///{tmp_path/'user_overlay.db'}"
monkeypatch.setenv("DATABASE_URL", db_url) monkeypatch.setenv("DATABASE_URL", db_url)
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"}) app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db() init_db()
with session_scope() as session: with session_scope() as session:
user = User(username="alice", password_digest=hash_password("secret"), admin=False) user = User(username="alice", password_digest=hash_password("secret"), admin=False)
session.add(user) session.add(user)
session.add(Overlay(name="standard", path="standard")) # System overlay (workshop, no user_id), pre-existing.
session.add(
Overlay(name="standard", path="standard", type="workshop", user_id=None)
)
session.flush() session.flush()
user_id = user.id user_id = user.id
@ -53,7 +58,8 @@ def test_user_can_view_overlay_catalog(user_client_with_overlay) -> None:
assert response.status_code == 200 assert response.status_code == 200
assert "standard" in text assert "standard" in text
assert "Add overlay" not in text # Non-admin users can create workshop overlays, so the Create button shows.
assert "Create overlay" in text
def test_admin_can_view_overlay_edit_controls(admin_client) -> None: def test_admin_can_view_overlay_edit_controls(admin_client) -> None:
@ -61,27 +67,29 @@ def test_admin_can_view_overlay_edit_controls(admin_client) -> None:
text = response.get_data(as_text=True) text = response.get_data(as_text=True)
assert response.status_code == 200 assert response.status_code == 200
assert "Add overlay" in text assert "Create overlay" in text
assert 'action="/overlays"' in text assert 'action="/overlays"' in text
def test_admin_can_create_overlay(admin_client) -> None: def test_admin_can_create_workshop_overlay_via_route(admin_client) -> None:
response = admin_client.post( response = admin_client.post(
"/overlays", "/overlays",
data={"name": "standard", "path": "standard"}, data={"name": "standard", "type": "workshop"},
headers={"X-CSRF-Token": "test-token"}, headers={"X-CSRF-Token": "test-token"},
) )
assert response.status_code == 302 assert response.status_code == 302
assert response.headers["Location"] == "/overlays" # Redirect to the new detail page now that paths are auto-generated.
assert response.headers["Location"].startswith("/overlays/")
def test_overlay_ref_must_be_relative(admin_client) -> None: def test_admin_cannot_create_managed_global_overlay_type(admin_client) -> None:
response = admin_client.post( response = admin_client.post(
"/overlays", "/overlays",
data={"name": "bad", "path": "/tmp/bad"}, data={"name": "managed", "type": "l4d2center_maps"},
headers={"X-CSRF-Token": "test-token"}, headers={"X-CSRF-Token": "test-token"},
) )
assert response.status_code == 400 assert response.status_code == 400
assert "unknown overlay type" in response.get_data(as_text=True)
@pytest.mark.parametrize("overlay_ref", [" standard", "standard ", "a//b", "a/", "./a", "a/.", "."]) @pytest.mark.parametrize("overlay_ref", [" standard", "standard ", "a//b", "a/", "./a", "a/.", "."])
@ -90,81 +98,241 @@ def test_overlay_ref_rejects_unsafe_components(overlay_ref: str) -> None:
validate_overlay_ref(overlay_ref) validate_overlay_ref(overlay_ref)
def test_overlay_route_rejects_whitespace_padded_ref(admin_client) -> None: def test_user_can_create_workshop_overlay(user_client_with_overlay) -> None:
response = admin_client.post(
"/overlays",
data={"name": "bad", "path": " standard"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 400
def test_non_admin_cannot_create_overlay(user_client_with_overlay) -> None:
response = user_client_with_overlay.post( response = user_client_with_overlay.post(
"/overlays", "/overlays",
data={"name": "bad", "path": "bad"}, data={"name": "my-maps", "type": "workshop"},
headers={"X-CSRF-Token": "test-token"}, headers={"X-CSRF-Token": "test-token"},
) )
assert response.status_code == 403 assert response.status_code == 302
with session_scope() as session:
overlay = session.query(Overlay).filter_by(name="my-maps").one()
assert overlay.type == "workshop"
assert overlay.user_id is not None
assert overlay.path == str(overlay.id)
def test_workshop_overlay_directory_is_created_on_disk(user_client_with_overlay, tmp_path) -> None:
response = user_client_with_overlay.post(
"/overlays",
data={"name": "my-maps", "type": "workshop"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
with session_scope() as session:
overlay = session.query(Overlay).filter_by(name="my-maps").one()
overlay_id = overlay.id
assert (tmp_path / "overlays" / str(overlay_id)).is_dir()
def test_two_users_can_have_workshop_overlay_with_same_name(tmp_path, monkeypatch) -> None:
# Set up a fresh app with two users.
db_url = f"sqlite:///{tmp_path/'shared.db'}"
monkeypatch.setenv("DATABASE_URL", db_url)
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db()
with session_scope() as session:
for username in ("alice", "bob"):
session.add(User(username=username, password_digest=hash_password("x"), admin=False))
session.flush()
alice_id, bob_id = (
session.query(User).filter_by(username="alice").one().id,
session.query(User).filter_by(username="bob").one().id,
)
def client_for(uid):
c = app.test_client()
with c.session_transaction() as sess:
sess["user_id"] = uid
sess["csrf_token"] = "test-token"
return c
for uid in (alice_id, bob_id):
r = client_for(uid).post(
"/overlays",
data={"name": "my-maps", "type": "workshop"},
headers={"X-CSRF-Token": "test-token"},
)
assert r.status_code == 302
with session_scope() as session:
rows = session.query(Overlay).filter_by(name="my-maps").all()
assert {r.user_id for r in rows} == {alice_id, bob_id}
def test_admin_can_update_and_delete_overlay(admin_client) -> None: def test_admin_can_update_and_delete_overlay(admin_client) -> None:
create = admin_client.post( create = admin_client.post(
"/overlays", "/overlays",
data={"name": "standard", "path": "standard"}, data={"name": "standard", "type": "workshop"},
headers={"X-CSRF-Token": "test-token"}, headers={"X-CSRF-Token": "test-token"},
) )
assert create.status_code == 302 assert create.status_code == 302
with session_scope() as session:
overlay_id = session.query(Overlay).filter_by(name="standard").one().id
update = admin_client.post( update = admin_client.post(
"/overlays/1", f"/overlays/{overlay_id}",
data={"name": "edited", "path": "edited"}, data={"name": "edited"},
headers={"X-CSRF-Token": "test-token"}, headers={"X-CSRF-Token": "test-token"},
) )
assert update.status_code == 302 assert update.status_code == 302
delete = admin_client.post( delete = admin_client.post(
"/overlays/1/delete", f"/overlays/{overlay_id}/delete",
headers={"X-CSRF-Token": "test-token"}, headers={"X-CSRF-Token": "test-token"},
) )
assert delete.status_code == 302 assert delete.status_code == 302
def test_update_overlay_rejects_duplicate_name(admin_client) -> None: def test_update_overlay_rejects_duplicate_name(admin_client) -> None:
for name in ["standard", "competitive"]: ids: list[int] = []
for name in ("standard", "competitive"):
response = admin_client.post( response = admin_client.post(
"/overlays", "/overlays",
data={"name": name, "path": name}, data={"name": name, "type": "workshop"},
headers={"X-CSRF-Token": "test-token"}, headers={"X-CSRF-Token": "test-token"},
) )
assert response.status_code == 302 assert response.status_code == 302
with session_scope() as session:
ids = [
session.query(Overlay).filter_by(name="standard").one().id,
session.query(Overlay).filter_by(name="competitive").one().id,
]
response = admin_client.post( response = admin_client.post(
"/overlays/2", f"/overlays/{ids[1]}",
data={"name": "standard", "path": "competitive"}, data={"name": "standard"},
headers={"X-CSRF-Token": "test-token"}, headers={"X-CSRF-Token": "test-token"},
) )
assert response.status_code == 409 assert response.status_code == 409
def test_overlay_detail_page_lists_using_blueprints(admin_client) -> None:
create = admin_client.post(
"/overlays",
data={"name": "shared", "type": "workshop"},
headers={"X-CSRF-Token": "test-token"},
)
assert create.status_code == 302
with session_scope() as session:
overlay_id = session.query(Overlay).filter_by(name="shared").one().id
with session_scope() as session:
admin = session.query(User).filter_by(username="admin").one()
bp_one = Blueprint(user_id=admin.id, name="alpha-bp", arguments="[]", config="[]")
bp_two = Blueprint(user_id=admin.id, name="beta-bp", arguments="[]", config="[]")
session.add_all([bp_one, bp_two])
session.flush()
session.add(BlueprintOverlay(blueprint_id=bp_one.id, overlay_id=overlay_id, position=0))
session.add(BlueprintOverlay(blueprint_id=bp_two.id, overlay_id=overlay_id, position=0))
response = admin_client.get(f"/overlays/{overlay_id}")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert "alpha-bp" in text
assert "beta-bp" in text
assert "Used by" in text
def test_non_admin_overlay_detail_only_lists_own_using_blueprints(user_client_with_overlay) -> None:
with session_scope() as session:
alice = session.query(User).filter_by(username="alice").one()
other = User(username="mallory", password_digest=hash_password("secret"), admin=False)
session.add(other)
session.flush()
# Use the seeded system "standard" overlay (id=1).
overlay_id = session.query(Overlay).filter_by(name="standard").one().id
own_bp = Blueprint(user_id=alice.id, name="own-bp", arguments="[]", config="[]")
other_bp = Blueprint(user_id=other.id, name="other-private-bp", arguments="[]", config="[]")
session.add_all([own_bp, other_bp])
session.flush()
session.add(BlueprintOverlay(blueprint_id=own_bp.id, overlay_id=overlay_id, position=0))
session.add(BlueprintOverlay(blueprint_id=other_bp.id, overlay_id=overlay_id, position=0))
response = user_client_with_overlay.get(f"/overlays/{overlay_id}")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert "own-bp" in text
assert "other-private-bp" not in text
def test_blueprint_edit_lists_system_and_owned_overlays_only(user_client_with_overlay) -> None:
with session_scope() as session:
alice = session.query(User).filter_by(username="alice").one()
other = User(username="mallory", password_digest=hash_password("secret"), admin=False)
session.add(other)
session.flush()
system_overlay_id = session.query(Overlay).filter_by(name="standard").one().id
foreign_overlay = Overlay(
name="other-private-workshop",
path="other-private-workshop",
type="workshop",
user_id=other.id,
)
blueprint = Blueprint(user_id=alice.id, name="alice-bp", arguments="[]", config="[]")
session.add_all([foreign_overlay, blueprint])
session.flush()
blueprint_id = blueprint.id
response = user_client_with_overlay.get(f"/blueprints/{blueprint_id}")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert "standard" in text
assert f'value="{system_overlay_id}"' in text
assert "other-private-workshop" not in text
def test_overlay_detail_page_404_when_missing(admin_client) -> None:
response = admin_client.get("/overlays/999")
assert response.status_code == 404
def test_overlay_update_redirects_to_detail(admin_client) -> None:
create = admin_client.post(
"/overlays",
data={"name": "standard", "type": "workshop"},
headers={"X-CSRF-Token": "test-token"},
)
assert create.status_code == 302
with session_scope() as session:
overlay_id = session.query(Overlay).filter_by(name="standard").one().id
response = admin_client.post(
f"/overlays/{overlay_id}",
data={"name": "renamed"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
assert response.headers["Location"] == f"/overlays/{overlay_id}"
def test_delete_overlay_rejects_in_use_overlay(admin_client) -> None: def test_delete_overlay_rejects_in_use_overlay(admin_client) -> None:
create = admin_client.post( create = admin_client.post(
"/overlays", "/overlays",
data={"name": "standard", "path": "standard"}, data={"name": "standard", "type": "workshop"},
headers={"X-CSRF-Token": "test-token"}, headers={"X-CSRF-Token": "test-token"},
) )
assert create.status_code == 302 assert create.status_code == 302
with session_scope() as session:
overlay_id = session.query(Overlay).filter_by(name="standard").one().id
with session_scope() as session: with session_scope() as session:
user = session.query(User).filter_by(username="admin").one() user = session.query(User).filter_by(username="admin").one()
blueprint = Blueprint(user_id=user.id, name="bp", arguments="[]", config="[]") blueprint = Blueprint(user_id=user.id, name="bp", arguments="[]", config="[]")
session.add(blueprint) session.add(blueprint)
session.flush() session.flush()
session.add(BlueprintOverlay(blueprint_id=blueprint.id, overlay_id=1, position=0)) session.add(BlueprintOverlay(blueprint_id=blueprint.id, overlay_id=overlay_id, position=0))
response = admin_client.post( response = admin_client.post(
"/overlays/1/delete", f"/overlays/{overlay_id}/delete",
headers={"X-CSRF-Token": "test-token"}, headers={"X-CSRF-Token": "test-token"},
) )

View file

@ -312,7 +312,10 @@ def test_admin_can_use_admin_pages(tmp_path, monkeypatch) -> None:
admin_page = client.get("/admin") admin_page = client.get("/admin")
assert admin_page.status_code == 200 assert admin_page.status_code == 200
assert 'action="/admin/install"' in admin_page.get_data(as_text=True) admin_html = admin_page.get_data(as_text=True)
assert 'action="/admin/install"' in admin_html
assert "/admin/global-overlays/refresh" not in admin_html
assert "Global map overlays" not in admin_html
assert client.get("/admin/users").status_code == 200 assert client.get("/admin/users").status_code == 200
jobs_response = client.get("/admin/jobs") jobs_response = client.get("/admin/jobs")
assert jobs_response.status_code == 200 assert jobs_response.status_code == 200
@ -457,3 +460,177 @@ def test_blueprint_detail_has_ordered_overlay_form(auth_client_with_server) -> N
assert 'name="config"' in text assert 'name="config"' in text
assert 'name="overlay_ids"' in text assert 'name="overlay_ids"' in text
assert 'name="overlay_position_1"' in text assert 'name="overlay_position_1"' in text
def test_admin_jobs_page_renders_system_job(tmp_path, monkeypatch) -> None:
db_url = f"sqlite:///{tmp_path/'admin-system-job.db'}"
monkeypatch.setenv("DATABASE_URL", db_url)
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db()
with session_scope() as session:
admin = User(username="admin", password_digest=hash_password("secret"), admin=True)
session.add(admin)
session.flush()
admin_id = admin.id
admin_client = app.test_client()
with admin_client.session_transaction() as sess:
sess["user_id"] = admin_id
with session_scope() as db:
db.add(Job(user_id=None, server_id=None, operation="refresh_workshop_items", state="queued"))
response = admin_client.get("/admin/jobs")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert "refresh_workshop_items" in text
assert "system" in text
def test_non_admin_cannot_view_system_job(tmp_path, monkeypatch) -> None:
db_url = f"sqlite:///{tmp_path/'non-admin-system-job.db'}"
monkeypatch.setenv("DATABASE_URL", db_url)
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db()
with session_scope() as session:
user = User(username="alice", password_digest=hash_password("secret"), admin=False)
session.add(user)
session.flush()
user_id = user.id
user_client = app.test_client()
with user_client.session_transaction() as sess:
sess["user_id"] = user_id
with session_scope() as db:
job = Job(user_id=None, server_id=None, operation="refresh_workshop_items", state="queued")
db.add(job)
db.flush()
job_id = job.id
response = user_client.get(f"/jobs/{job_id}")
assert response.status_code == 403
def test_overlay_create_modal_offers_script_type(auth_client_with_server) -> None:
response = auth_client_with_server.get("/overlays")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert 'value="workshop"' in text
assert 'value="script"' in text
def _seed_overlay(name: str, type_: str, user_id: int) -> int:
with session_scope() as s:
overlay = Overlay(name=name, path="", type=type_, user_id=user_id)
s.add(overlay)
s.flush()
overlay.path = str(overlay.id)
s.flush()
return overlay.id
def test_overlay_detail_script_section(auth_client_with_server) -> None:
with session_scope() as s:
user_id = s.query(User).filter_by(username="alice").one().id
overlay_id = _seed_overlay("build", "script", user_id)
response = auth_client_with_server.get(f"/overlays/{overlay_id}")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert '<textarea name="script"' in text
assert "Rebuild" in text
assert "Wipe" in text
assert "Last build" in text
def test_overlay_detail_workshop_section_unchanged(auth_client_with_server) -> None:
with session_scope() as s:
user_id = s.query(User).filter_by(username="alice").one().id
overlay_id = _seed_overlay("ws", "workshop", user_id)
response = auth_client_with_server.get(f"/overlays/{overlay_id}")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert "Workshop items" in text
def test_overlay_detail_links_to_overlay_jobs_page(auth_client_with_server) -> None:
with session_scope() as s:
user_id = s.query(User).filter_by(username="alice").one().id
overlay_id = _seed_overlay("scripted", "script", user_id)
response = auth_client_with_server.get(f"/overlays/{overlay_id}")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert f'href="/overlays/{overlay_id}/jobs"' in text
assert "Save and build" in text
def test_overlay_jobs_page_lists_overlay_builds(auth_client_with_server) -> None:
with session_scope() as s:
user_id = s.query(User).filter_by(username="alice").one().id
overlay_id = _seed_overlay("scripted", "script", user_id)
other_overlay_id = _seed_overlay("other", "script", user_id)
with session_scope() as session:
session.add(Job(user_id=user_id, overlay_id=overlay_id, operation="build_overlay", state="succeeded"))
session.add(Job(user_id=user_id, overlay_id=overlay_id, operation="build_overlay", state="queued"))
# Job for a different overlay must not appear.
session.add(Job(user_id=user_id, overlay_id=other_overlay_id, operation="build_overlay", state="succeeded"))
response = auth_client_with_server.get(f"/overlays/{overlay_id}/jobs")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert "Build jobs for scripted" in text
assert text.count("build_overlay") == 2
assert f'href="/overlays/{overlay_id}"' in text
def test_overlay_jobs_page_404_for_unknown_overlay(auth_client_with_server) -> None:
response = auth_client_with_server.get("/overlays/9999/jobs")
assert response.status_code == 404
def test_overlay_jobs_page_403_for_other_users_private_overlay(tmp_path, monkeypatch) -> None:
db_url = f"sqlite:///{tmp_path/'overlay-jobs-403.db'}"
monkeypatch.setenv("DATABASE_URL", db_url)
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db()
with session_scope() as session:
owner = User(username="owner", password_digest=hash_password("x"), admin=False)
other = User(username="other", password_digest=hash_password("x"), admin=False)
session.add_all([owner, other])
session.flush()
owner_id = owner.id
other_id = other.id
overlay_id = _seed_overlay("private", "script", owner_id)
client = app.test_client()
with client.session_transaction() as sess:
sess["user_id"] = other_id
response = client.get(f"/overlays/{overlay_id}/jobs")
assert response.status_code == 403
def test_overlay_detail_no_global_source_block(auth_client_with_server) -> None:
with session_scope() as s:
user_id = s.query(User).filter_by(username="alice").one().id
overlay_id = _seed_overlay("ws", "workshop", user_id)
response = auth_client_with_server.get(f"/overlays/{overlay_id}")
text = response.get_data(as_text=True)
assert "Global source" not in text
assert "source_url" not in text

View file

@ -0,0 +1,276 @@
"""Routes for type='script' overlays: create, /script (update body),
/wipe, /build. Permissions mirror workshop overlays (owner or admin)."""
from __future__ import annotations
import pytest
from sqlalchemy import select
from l4d2web.app import create_app
from l4d2web.auth import hash_password
from l4d2web.db import init_db, session_scope
from l4d2web.models import Job, Overlay, User
@pytest.fixture
def app(tmp_path, monkeypatch):
db_url = f"sqlite:///{tmp_path/'script-routes.db'}"
monkeypatch.setenv("DATABASE_URL", db_url)
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
flask_app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db()
return flask_app
@pytest.fixture
def alice_id(app) -> int:
with session_scope() as s:
user = User(username="alice", password_digest=hash_password("x"), admin=False)
s.add(user)
s.flush()
return user.id
@pytest.fixture
def bob_id(app) -> int:
with session_scope() as s:
user = User(username="bob", password_digest=hash_password("x"), admin=False)
s.add(user)
s.flush()
return user.id
@pytest.fixture
def admin_id(app) -> int:
with session_scope() as s:
user = User(username="admin", password_digest=hash_password("x"), admin=True)
s.add(user)
s.flush()
return user.id
def _client_for(app, user_id: int):
client = app.test_client()
with client.session_transaction() as sess:
sess["user_id"] = user_id
sess["csrf_token"] = "test-token"
return client
def _create_script_overlay(app, user_id: int, *, name: str = "x") -> int:
client = _client_for(app, user_id)
response = client.post(
"/overlays",
data={"name": name, "type": "script"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302, response.get_data(as_text=True)
with session_scope() as s:
return s.scalar(select(Overlay.id).where(Overlay.name == name))
def test_create_script_overlay(app, alice_id) -> None:
client = _client_for(app, alice_id)
response = client.post(
"/overlays",
data={"name": "first", "type": "script"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
with session_scope() as s:
overlay = s.query(Overlay).filter_by(name="first").one()
assert overlay.type == "script"
assert overlay.script == ""
assert overlay.last_build_status == ""
assert overlay.user_id == alice_id
assert overlay.path == str(overlay.id)
def test_admin_creates_system_wide_script_overlay(app, admin_id) -> None:
client = _client_for(app, admin_id)
response = client.post(
"/overlays",
data={"name": "system", "type": "script", "system_wide": "1"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
with session_scope() as s:
overlay = s.query(Overlay).filter_by(name="system").one()
assert overlay.user_id is None
def test_non_admin_system_wide_flag_is_ignored(app, alice_id) -> None:
client = _client_for(app, alice_id)
response = client.post(
"/overlays",
data={"name": "evil", "type": "script", "system_wide": "1"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
with session_scope() as s:
overlay = s.query(Overlay).filter_by(name="evil").one()
assert overlay.user_id == alice_id
def test_update_script_body_enqueues_build(app, alice_id) -> None:
overlay_id = _create_script_overlay(app, alice_id)
client = _client_for(app, alice_id)
r1 = client.post(
f"/overlays/{overlay_id}/script",
data={"script": "echo new"},
headers={"X-CSRF-Token": "test-token"},
)
assert r1.status_code == 302
assert r1.headers["Location"].startswith("/jobs/")
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
assert overlay.script == "echo new"
jobs = s.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
assert len(jobs) == 1
assert r1.headers["Location"] == f"/jobs/{jobs[0].id}"
# Coalesce against pending.
r2 = client.post(
f"/overlays/{overlay_id}/script",
data={"script": "echo newer"},
headers={"X-CSRF-Token": "test-token"},
)
assert r2.status_code == 302
assert r2.headers["Location"] == r1.headers["Location"]
with session_scope() as s:
jobs = s.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
assert len(jobs) == 1
def test_update_script_normalizes_crlf_to_lf(app, alice_id) -> None:
"""HTML <textarea> submits CRLF line endings; bash chokes on trailing \\r
in every command. Storage must be LF-only so the sandbox tmpfile is
well-formed."""
overlay_id = _create_script_overlay(app, alice_id)
client = _client_for(app, alice_id)
client.post(
f"/overlays/{overlay_id}/script",
data={"script": "ls /\r\necho hello\r\n"},
headers={"X-CSRF-Token": "test-token"},
)
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
assert overlay.script == "ls /\necho hello\n"
assert "\r" not in overlay.script
def test_manual_rebuild(app, alice_id) -> None:
overlay_id = _create_script_overlay(app, alice_id)
client = _client_for(app, alice_id)
r1 = client.post(
f"/overlays/{overlay_id}/build",
headers={"X-CSRF-Token": "test-token"},
)
assert r1.status_code == 302
with session_scope() as s:
jobs = s.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
assert len(jobs) == 1
# Coalesce.
r2 = client.post(
f"/overlays/{overlay_id}/build",
headers={"X-CSRF-Token": "test-token"},
)
assert r2.status_code == 302
with session_scope() as s:
jobs = s.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
assert len(jobs) == 1
def test_wipe_runs_find_delete(app, alice_id, monkeypatch) -> None:
overlay_id = _create_script_overlay(app, alice_id)
# Pre-set a "successful" status so we can verify wipe resets it.
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
overlay.last_build_status = "ok"
captured: dict = {}
def fake_run(overlay_id_arg, script_text, *, on_stdout, on_stderr, should_cancel):
captured["overlay_id"] = overlay_id_arg
captured["script"] = script_text
from l4d2web.services import overlay_builders
monkeypatch.setattr(overlay_builders, "run_sandboxed_script", fake_run)
client = _client_for(app, alice_id)
response = client.post(
f"/overlays/{overlay_id}/wipe",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
assert captured["overlay_id"] == overlay_id
assert captured["script"] == "find /overlay -mindepth 1 -delete"
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
assert overlay.last_build_status == ""
# Wipe does NOT auto-enqueue a rebuild.
jobs = s.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
assert len(jobs) == 0
def test_wipe_refuses_during_running_build(app, alice_id, monkeypatch) -> None:
overlay_id = _create_script_overlay(app, alice_id)
# Mark a build as running for this overlay.
with session_scope() as s:
s.add(
Job(
user_id=alice_id,
server_id=None,
overlay_id=overlay_id,
operation="build_overlay",
state="running",
)
)
invocations: list = []
def fake_run(*args, **kwargs):
invocations.append((args, kwargs))
from l4d2web.services import overlay_builders
monkeypatch.setattr(overlay_builders, "run_sandboxed_script", fake_run)
client = _client_for(app, alice_id)
response = client.post(
f"/overlays/{overlay_id}/wipe",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 409
assert invocations == []
def test_permissions_non_owner_denied(app, alice_id, bob_id) -> None:
overlay_id = _create_script_overlay(app, alice_id, name="alice-private")
bob = _client_for(app, bob_id)
r1 = bob.post(
f"/overlays/{overlay_id}/script",
data={"script": "boom"},
headers={"X-CSRF-Token": "test-token"},
)
assert r1.status_code == 403
def test_permissions_admin_can_edit_any(app, alice_id, admin_id) -> None:
overlay_id = _create_script_overlay(app, alice_id, name="alice-private")
admin = _client_for(app, admin_id)
r1 = admin.post(
f"/overlays/{overlay_id}/script",
data={"script": "echo admin"},
headers={"X-CSRF-Token": "test-token"},
)
assert r1.status_code == 302
with session_scope() as s:
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
assert overlay.script == "echo admin"

View file

@ -0,0 +1,79 @@
from pathlib import Path
from l4d2web.app import create_app
from l4d2web.db import init_db, session_scope
from l4d2web.models import Overlay
def _make_app(tmp_path: Path, monkeypatch):
db_url = f"sqlite:///{tmp_path/'seed.db'}"
monkeypatch.setenv("DATABASE_URL", db_url)
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db()
return app
def test_seed_creates_system_wide_script_overlays(tmp_path: Path, monkeypatch) -> None:
app = _make_app(tmp_path, monkeypatch)
examples = tmp_path / "examples"
examples.mkdir()
(examples / "alpha.sh").write_text("#!/bin/bash\necho alpha\n")
(examples / "beta.sh").write_text("#!/bin/bash\necho beta\n")
result = app.test_cli_runner().invoke(args=["seed-script-overlays", str(examples)])
assert result.exit_code == 0, result.output
with session_scope() as db:
rows = db.query(Overlay).order_by(Overlay.name).all()
assert [(o.name, o.type, o.user_id, o.script.strip()) for o in rows] == [
("alpha", "script", None, "#!/bin/bash\necho alpha"),
("beta", "script", None, "#!/bin/bash\necho beta"),
]
for o in rows:
assert (tmp_path / "overlays" / o.path).is_dir()
def test_seed_refreshes_existing_script_overlay(tmp_path: Path, monkeypatch) -> None:
app = _make_app(tmp_path, monkeypatch)
examples = tmp_path / "examples"
examples.mkdir()
(examples / "alpha.sh").write_text("v1\n")
runner = app.test_cli_runner()
runner.invoke(args=["seed-script-overlays", str(examples)])
(examples / "alpha.sh").write_text("v2\n")
result = runner.invoke(args=["seed-script-overlays", str(examples)])
assert result.exit_code == 0, result.output
with session_scope() as db:
rows = db.query(Overlay).filter(Overlay.name == "alpha").all()
assert len(rows) == 1
assert rows[0].script.strip() == "v2"
def test_seed_errors_on_type_collision(tmp_path: Path, monkeypatch) -> None:
app = _make_app(tmp_path, monkeypatch)
examples = tmp_path / "examples"
examples.mkdir()
(examples / "shared.sh").write_text("#!/bin/bash\n")
with session_scope() as db:
db.add(Overlay(name="shared", path="shared", type="workshop", user_id=None))
result = app.test_cli_runner().invoke(args=["seed-script-overlays", str(examples)])
assert result.exit_code != 0
assert "not script" in result.output
def test_seed_no_files_is_noop(tmp_path: Path, monkeypatch) -> None:
app = _make_app(tmp_path, monkeypatch)
examples = tmp_path / "examples"
examples.mkdir()
result = app.test_cli_runner().invoke(args=["seed-script-overlays", str(examples)])
assert result.exit_code == 0
with session_scope() as db:
assert db.query(Overlay).count() == 0

View file

@ -37,6 +37,32 @@ def user_client_with_blueprints(tmp_path, monkeypatch):
return client, payload return client, payload
def test_servers_page_without_blueprints_shows_create_blueprint_cta(tmp_path, monkeypatch) -> None:
db_url = f"sqlite:///{tmp_path/'no_blueprints.db'}"
monkeypatch.setenv("DATABASE_URL", db_url)
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db()
with session_scope() as session:
user = User(username="solo", password_digest=hash_password("secret"), admin=False)
session.add(user)
session.flush()
user_id = user.id
client = app.test_client()
with client.session_transaction() as sess:
sess["user_id"] = user_id
sess["csrf_token"] = "test-token"
response = client.get("/servers")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert 'href="/blueprints"' in text
assert "Create a blueprint first" in text
assert "disabled" not in text
def test_create_server_from_blueprint(user_client_with_blueprints) -> None: def test_create_server_from_blueprint(user_client_with_blueprints) -> None:
client, data = user_client_with_blueprints client, data = user_client_with_blueprints
payload = {"name": "alpha", "port": 27015, "blueprint_id": data["blueprint_id"]} payload = {"name": "alpha", "port": 27015, "blueprint_id": data["blueprint_id"]}
@ -132,6 +158,154 @@ def test_create_server_rejects_unsafe_names(user_client_with_blueprints, bad_nam
assert session.scalars(select(Server)).all() == [] assert session.scalars(select(Server)).all() == []
def test_create_server_with_empty_port_auto_assigns(user_client_with_blueprints) -> None:
client, data = user_client_with_blueprints
response = client.post(
"/servers",
data={"name": "alpha", "port": "", "blueprint_id": str(data["blueprint_id"])},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
from sqlalchemy import select
from l4d2web.models import Server
with session_scope() as session:
server = session.scalars(select(Server)).one()
assert server.port == 27015
def test_create_server_auto_assign_skips_taken_ports(user_client_with_blueprints) -> None:
client, data = user_client_with_blueprints
first = client.post(
"/servers",
data={"name": "alpha", "port": "27015", "blueprint_id": str(data["blueprint_id"])},
headers={"X-CSRF-Token": "test-token"},
)
assert first.status_code == 302
second = client.post(
"/servers",
data={"name": "beta", "blueprint_id": str(data["blueprint_id"])},
headers={"X-CSRF-Token": "test-token"},
)
assert second.status_code == 302
from sqlalchemy import select
from l4d2web.models import Server
with session_scope() as session:
ports = sorted(session.scalars(select(Server.port)).all())
assert ports == [27015, 27016]
def test_create_server_returns_409_when_port_range_exhausted(tmp_path, monkeypatch) -> None:
db_url = f"sqlite:///{tmp_path/'exhausted.db'}"
monkeypatch.setenv("DATABASE_URL", db_url)
monkeypatch.setenv("LEFT4ME_PORT_RANGE_START", "30000")
monkeypatch.setenv("LEFT4ME_PORT_RANGE_END", "30000")
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db()
with session_scope() as session:
user = User(username="alice", password_digest=hash_password("secret"), admin=False)
session.add(user)
session.flush()
blueprint = Blueprint(user_id=user.id, name="bp", arguments="[]", config="[]")
session.add(blueprint)
session.flush()
user_id = user.id
blueprint_id = blueprint.id
client = app.test_client()
with client.session_transaction() as sess:
sess["user_id"] = user_id
sess["csrf_token"] = "test-token"
first = client.post(
"/servers",
data={"name": "alpha", "blueprint_id": str(blueprint_id)},
headers={"X-CSRF-Token": "test-token"},
)
assert first.status_code == 302
second = client.post(
"/servers",
data={"name": "beta", "blueprint_id": str(blueprint_id)},
headers={"X-CSRF-Token": "test-token"},
)
assert second.status_code == 409
def test_delete_operation_redirects_to_index(user_client_with_blueprints) -> None:
client, data = user_client_with_blueprints
create = client.post(
"/servers",
data=json.dumps({"name": "alpha", "port": 27015, "blueprint_id": data["blueprint_id"]}),
content_type="application/json",
headers={"X-CSRF-Token": "test-token"},
)
server_id = create.get_json()["id"]
response = client.post(
f"/servers/{server_id}/delete",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
assert response.headers["Location"] == "/servers"
def test_servers_page_prefills_blueprint_when_owned(user_client_with_blueprints) -> None:
client, data = user_client_with_blueprints
response = client.get(f"/servers?blueprint_id={data['other_blueprint_id']}")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert f'<option value="{data["other_blueprint_id"]}" selected>' in text
assert "showModal" in text
def test_servers_page_ignores_foreign_blueprint_id(user_client_with_blueprints) -> None:
client, _ = user_client_with_blueprints
with session_scope() as session:
other = User(username="bob", password_digest=hash_password("secret"), admin=False)
session.add(other)
session.flush()
foreign = Blueprint(user_id=other.id, name="foreign", arguments="[]", config="[]")
session.add(foreign)
session.flush()
foreign_id = foreign.id
response = client.get(f"/servers?blueprint_id={foreign_id}")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert "selected" not in text
assert "showModal" not in text
def test_servers_page_ignores_non_integer_blueprint_id(user_client_with_blueprints) -> None:
client, _ = user_client_with_blueprints
response = client.get("/servers?blueprint_id=abc")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert "showModal" not in text
def test_servers_page_without_param_does_not_auto_open(user_client_with_blueprints) -> None:
client, _ = user_client_with_blueprints
response = client.get("/servers")
text = response.get_data(as_text=True)
assert response.status_code == 200
assert "showModal" not in text
def test_lifecycle_form_creates_queued_job(user_client_with_blueprints) -> None: def test_lifecycle_form_creates_queued_job(user_client_with_blueprints) -> None:
client, data = user_client_with_blueprints client, data = user_client_with_blueprints
create_response = client.post( create_response = client.post(
@ -149,3 +323,42 @@ def test_lifecycle_form_creates_queued_job(user_client_with_blueprints) -> None:
assert response.status_code == 302 assert response.status_code == 302
assert response.headers["Location"] == f"/servers/{server_id}" assert response.headers["Location"] == f"/servers/{server_id}"
def test_reset_operation_enqueues_job_and_stops_desired_state(user_client_with_blueprints) -> None:
from sqlalchemy import select
from l4d2web.models import Job, Server
client, data = user_client_with_blueprints
create_response = client.post(
"/servers",
data=json.dumps({"name": "alpha", "port": 27015, "blueprint_id": data["blueprint_id"]}),
content_type="application/json",
headers={"X-CSRF-Token": "test-token"},
)
server_id = create_response.get_json()["id"]
# Pretend the user already started it.
with session_scope() as session:
server = session.scalar(select(Server).where(Server.id == server_id))
assert server is not None
server.desired_state = "running"
response = client.post(
f"/servers/{server_id}/reset",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
assert response.headers["Location"] == f"/servers/{server_id}"
with session_scope() as session:
server = session.scalar(select(Server).where(Server.id == server_id))
assert server is not None
assert server.desired_state == "stopped"
jobs = session.scalars(
select(Job).where(Job.server_id == server_id, Job.operation == "reset")
).all()
assert len(jobs) == 1
assert jobs[0].state == "queued"

View file

@ -48,6 +48,23 @@ def test_owner_can_stream_server_logs(owner_client_with_server, monkeypatch) ->
assert response.status_code == 200 assert response.status_code == 200
def test_log_stream_translates_heartbeat_to_sse_keepalive(owner_client_with_server, monkeypatch) -> None:
client, server_id = owner_client_with_server
monkeypatch.setattr(
"l4d2web.services.l4d2_facade.stream_server_logs",
lambda name, lines=200, follow=True: iter(["first", "", "second"]),
)
response = client.get(f"/servers/{server_id}/logs/stream")
assert response.status_code == 200
body = response.get_data(as_text=True)
assert "data: first\n\n" in body
assert "data: second\n\n" in body
assert ": keepalive\n\n" in body
assert "data: \n\n" not in body
def test_status_precedence() -> None: def test_status_precedence() -> None:
from l4d2web.services.status import compute_display_state from l4d2web.services.status import compute_display_state

View file

@ -0,0 +1,312 @@
"""Tests for the Steam Workshop API client and downloader."""
from __future__ import annotations
import os
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
from l4d2web.services import steam_workshop
def test_parse_workshop_input_single_numeric() -> None:
assert steam_workshop.parse_workshop_input("12345") == ["12345"]
def test_parse_workshop_input_single_url() -> None:
url = "https://steamcommunity.com/sharedfiles/filedetails/?id=98765"
assert steam_workshop.parse_workshop_input(url) == ["98765"]
def test_parse_workshop_input_workshop_url_variant() -> None:
url = "steamcommunity.com/workshop/filedetails/?id=42"
assert steam_workshop.parse_workshop_input(url) == ["42"]
def test_parse_workshop_input_multiline_batch() -> None:
raw = """
12345
https://steamcommunity.com/sharedfiles/filedetails/?id=67890
99999
"""
assert steam_workshop.parse_workshop_input(raw) == ["12345", "67890", "99999"]
def test_parse_workshop_input_deduplicates_preserving_order() -> None:
raw = "100\n200\n100\n300"
assert steam_workshop.parse_workshop_input(raw) == ["100", "200", "300"]
def test_parse_workshop_input_rejects_garbage() -> None:
with pytest.raises(ValueError):
steam_workshop.parse_workshop_input("not-a-number")
def test_parse_workshop_input_rejects_empty() -> None:
with pytest.raises(ValueError):
steam_workshop.parse_workshop_input("")
def test_parse_workshop_input_rejects_non_steam_url() -> None:
with pytest.raises(ValueError):
steam_workshop.parse_workshop_input("https://example.com/?id=12345")
def test_endpoints_are_https() -> None:
assert steam_workshop.GET_PUBLISHED_FILE_DETAILS_URL.startswith("https://")
assert steam_workshop.GET_COLLECTION_DETAILS_URL.startswith("https://")
assert "api.steampowered.com" in steam_workshop.GET_PUBLISHED_FILE_DETAILS_URL
def test_resolve_collection_returns_child_ids() -> None:
fake_response = MagicMock(status_code=200)
fake_response.raise_for_status = MagicMock()
fake_response.json.return_value = {
"response": {
"collectiondetails": [
{
"publishedfileid": "555",
"result": 1,
"children": [
{"publishedfileid": "1001", "filetype": 0},
{"publishedfileid": "1002", "filetype": 0},
{"publishedfileid": "9999", "filetype": 1}, # nested collection — skip
],
}
]
}
}
with patch.object(steam_workshop, "_session", return_value=MagicMock(post=MagicMock(return_value=fake_response))):
ids = steam_workshop.resolve_collection("555")
assert ids == ["1001", "1002"]
def test_fetch_metadata_batch_parses_published_file_details() -> None:
fake_response = MagicMock(status_code=200)
fake_response.raise_for_status = MagicMock()
fake_response.json.return_value = {
"response": {
"publishedfiledetails": [
{
"publishedfileid": "1001",
"result": 1,
"consumer_app_id": 550,
"title": "Map A",
"filename": "map_a.vpk",
"file_url": "https://steamusercontent.com/abc/map_a.vpk",
"file_size": "1024",
"time_updated": 1700000000,
"preview_url": "https://steamuserimages.com/preview_a.jpg",
}
]
}
}
with patch.object(steam_workshop, "_session", return_value=MagicMock(post=MagicMock(return_value=fake_response))):
metas = steam_workshop.fetch_metadata_batch(["1001"], mode="add")
assert len(metas) == 1
m = metas[0]
assert m.steam_id == "1001"
assert m.title == "Map A"
assert m.filename == "map_a.vpk"
assert m.file_url == "https://steamusercontent.com/abc/map_a.vpk"
assert m.file_size == 1024
assert m.time_updated == 1700000000
assert m.preview_url == "https://steamuserimages.com/preview_a.jpg"
assert m.consumer_app_id == 550
assert m.result == 1
def test_fetch_metadata_batch_rejects_non_l4d2_in_add_mode() -> None:
fake_response = MagicMock(status_code=200)
fake_response.raise_for_status = MagicMock()
fake_response.json.return_value = {
"response": {
"publishedfiledetails": [
{
"publishedfileid": "1001",
"result": 1,
"consumer_app_id": 440, # TF2
"title": "Other",
"filename": "x.vpk",
"file_url": "https://example.com/x.vpk",
"file_size": "0",
"time_updated": 0,
}
]
}
}
with patch.object(steam_workshop, "_session", return_value=MagicMock(post=MagicMock(return_value=fake_response))):
with pytest.raises(steam_workshop.WorkshopValidationError):
steam_workshop.fetch_metadata_batch(["1001"], mode="add")
def test_fetch_metadata_batch_skips_non_l4d2_in_refresh_mode() -> None:
fake_response = MagicMock(status_code=200)
fake_response.raise_for_status = MagicMock()
fake_response.json.return_value = {
"response": {
"publishedfiledetails": [
{
"publishedfileid": "1001",
"result": 1,
"consumer_app_id": 440,
"title": "Other",
"filename": "x.vpk",
"file_url": "https://example.com/x.vpk",
"file_size": "0",
"time_updated": 0,
},
{
"publishedfileid": "1002",
"result": 1,
"consumer_app_id": 550,
"title": "Good",
"filename": "g.vpk",
"file_url": "https://example.com/g.vpk",
"file_size": "100",
"time_updated": 1,
},
]
}
}
with patch.object(steam_workshop, "_session", return_value=MagicMock(post=MagicMock(return_value=fake_response))):
metas = steam_workshop.fetch_metadata_batch(["1001", "1002"], mode="refresh")
# The non-L4D2 item is dropped; the L4D2 item is kept.
assert [m.steam_id for m in metas] == ["1002"]
def test_fetch_metadata_batch_captures_result_failure() -> None:
fake_response = MagicMock(status_code=200)
fake_response.raise_for_status = MagicMock()
fake_response.json.return_value = {
"response": {
"publishedfiledetails": [
{
"publishedfileid": "999",
"result": 9, # not found / hidden / etc.
}
]
}
}
with patch.object(steam_workshop, "_session", return_value=MagicMock(post=MagicMock(return_value=fake_response))):
metas = steam_workshop.fetch_metadata_batch(["999"], mode="refresh")
# Item is kept but marked with the failing result; consumer app id never validated.
assert len(metas) == 1
assert metas[0].result == 9
def test_download_to_cache_writes_atomically_and_sets_mtime(tmp_path: Path) -> None:
cache_root = tmp_path / "workshop_cache"
cache_root.mkdir()
meta = steam_workshop.WorkshopMetadata(
steam_id="1001",
title="A",
filename="a.vpk",
file_url="https://example.com/a.vpk",
file_size=11,
time_updated=1700000000,
preview_url="",
consumer_app_id=550,
result=1,
)
fake_response = MagicMock(status_code=200)
fake_response.raise_for_status = MagicMock()
fake_response.iter_content.return_value = [b"hello world"]
with patch.object(steam_workshop, "_session", return_value=MagicMock(get=MagicMock(return_value=fake_response))):
path = steam_workshop.download_to_cache(meta, cache_root)
assert path == cache_root / "1001.vpk"
assert path.read_bytes() == b"hello world"
assert int(path.stat().st_mtime) == 1700000000
# No leftover .partial file.
assert not (cache_root / "1001.vpk.partial").exists()
def test_download_to_cache_is_idempotent(tmp_path: Path) -> None:
cache_root = tmp_path / "workshop_cache"
cache_root.mkdir()
target = cache_root / "1001.vpk"
target.write_bytes(b"existing")
os.utime(target, (1700000000, 1700000000))
meta = steam_workshop.WorkshopMetadata(
steam_id="1001",
title="A",
filename="a.vpk",
file_url="https://example.com/a.vpk",
file_size=8, # matches existing
time_updated=1700000000, # matches existing mtime
preview_url="",
consumer_app_id=550,
result=1,
)
fake_session = MagicMock()
with patch.object(steam_workshop, "_session", return_value=fake_session):
steam_workshop.download_to_cache(meta, cache_root)
fake_session.get.assert_not_called()
def test_download_to_cache_redownloads_when_mtime_or_size_differ(tmp_path: Path) -> None:
cache_root = tmp_path / "workshop_cache"
cache_root.mkdir()
target = cache_root / "1001.vpk"
target.write_bytes(b"old")
os.utime(target, (1500000000, 1500000000))
meta = steam_workshop.WorkshopMetadata(
steam_id="1001",
title="A",
filename="a.vpk",
file_url="https://example.com/a.vpk",
file_size=11,
time_updated=1700000000,
preview_url="",
consumer_app_id=550,
result=1,
)
fake_response = MagicMock(status_code=200)
fake_response.raise_for_status = MagicMock()
fake_response.iter_content.return_value = [b"hello world"]
with patch.object(steam_workshop, "_session", return_value=MagicMock(get=MagicMock(return_value=fake_response))):
steam_workshop.download_to_cache(meta, cache_root)
assert target.read_bytes() == b"hello world"
assert int(target.stat().st_mtime) == 1700000000
def test_refresh_all_uses_thread_pool_and_collects_errors(tmp_path: Path) -> None:
cache_root = tmp_path / "workshop_cache"
cache_root.mkdir()
metas = [
steam_workshop.WorkshopMetadata(
steam_id=str(i),
title=f"M{i}",
filename=f"m{i}.vpk",
file_url=f"https://example.com/m{i}.vpk",
file_size=5,
time_updated=1700000000,
preview_url="",
consumer_app_id=550,
result=1,
)
for i in (1, 2, 3)
]
def fake_download(meta, cache_root_arg, **kwargs):
if meta.steam_id == "2":
raise RuntimeError("simulated download failure")
return cache_root_arg / f"{meta.steam_id}.vpk"
with patch.object(steam_workshop, "download_to_cache", side_effect=fake_download):
report = steam_workshop.refresh_all(metas, cache_root, executor_workers=4)
assert report.downloaded == 2
assert report.errors == 1
assert "2" in report.per_item_errors

View file

@ -0,0 +1,193 @@
"""Tests for the workshop-overlay schema additions: typed Overlay, partial
unique indexes, WorkshopItem registry, and overlay_workshop_items association.
"""
import pytest
from sqlalchemy.exc import IntegrityError
from l4d2web.db import init_db, session_scope
from l4d2web.models import (
Job,
Overlay,
OverlayWorkshopItem,
User,
WorkshopItem,
)
@pytest.fixture
def db(tmp_path, monkeypatch):
monkeypatch.setenv("DATABASE_URL", f"sqlite:///{tmp_path/'workshop.db'}")
init_db()
yield
def _make_user(username: str) -> int:
with session_scope() as s:
user = User(username=username, password_digest="x")
s.add(user)
s.flush()
return user.id
def test_overlay_has_type_and_user_id(db) -> None:
with session_scope() as s:
s.add(Overlay(name="standard", path="standard"))
s.flush()
row = s.query(Overlay).filter_by(name="standard").one()
assert row.type == "workshop"
assert row.user_id is None
def test_overlay_has_script_columns(db) -> None:
with session_scope() as s:
s.add(Overlay(name="defaulted", path="1"))
s.flush()
row = s.query(Overlay).filter_by(name="defaulted").one()
assert row.script == ""
assert row.last_build_status == ""
def test_two_system_overlays_with_same_name_are_rejected(db) -> None:
with session_scope() as s:
s.add(Overlay(name="shared", path="shared", type="l4d2center_maps", user_id=None))
s.flush()
with pytest.raises(IntegrityError):
with session_scope() as s:
s.add(Overlay(name="shared", path="other", type="cedapug_maps", user_id=None))
s.flush()
def test_two_users_can_share_workshop_overlay_name(db) -> None:
alice_id = _make_user("alice")
bob_id = _make_user("bob")
with session_scope() as s:
s.add(Overlay(name="my-maps", path="1", type="workshop", user_id=alice_id))
s.add(Overlay(name="my-maps", path="2", type="workshop", user_id=bob_id))
s.flush()
with session_scope() as s:
rows = s.query(Overlay).filter_by(name="my-maps").all()
assert {r.user_id for r in rows} == {alice_id, bob_id}
def test_same_user_cannot_have_duplicate_workshop_name(db) -> None:
user_id = _make_user("alice")
with session_scope() as s:
s.add(Overlay(name="dupe", path="1", type="workshop", user_id=user_id))
s.flush()
with pytest.raises(IntegrityError):
with session_scope() as s:
s.add(Overlay(name="dupe", path="2", type="workshop", user_id=user_id))
s.flush()
def test_workshop_item_steam_id_is_unique(db) -> None:
with session_scope() as s:
s.add(WorkshopItem(steam_id="123", title="Map A"))
s.flush()
with pytest.raises(IntegrityError):
with session_scope() as s:
s.add(WorkshopItem(steam_id="123", title="Map A duplicate"))
s.flush()
def test_overlay_workshop_item_unique_per_overlay(db) -> None:
user_id = _make_user("alice")
with session_scope() as s:
ov = Overlay(name="my-maps", path="1", type="workshop", user_id=user_id)
wi = WorkshopItem(steam_id="555", title="A")
s.add_all([ov, wi])
s.flush()
s.add(OverlayWorkshopItem(overlay_id=ov.id, workshop_item_id=wi.id))
s.flush()
overlay_id = ov.id
workshop_item_id = wi.id
with pytest.raises(IntegrityError):
with session_scope() as s:
s.add(
OverlayWorkshopItem(
overlay_id=overlay_id, workshop_item_id=workshop_item_id
)
)
s.flush()
def test_deleting_overlay_cascades_associations_but_not_workshop_items(db) -> None:
user_id = _make_user("alice")
with session_scope() as s:
ov = Overlay(name="my-maps", path="1", type="workshop", user_id=user_id)
wi = WorkshopItem(steam_id="777", title="A")
s.add_all([ov, wi])
s.flush()
s.add(OverlayWorkshopItem(overlay_id=ov.id, workshop_item_id=wi.id))
s.flush()
overlay_id = ov.id
# Delete via raw connection to actually exercise ON DELETE CASCADE / RESTRICT.
from l4d2web.db import get_engine
engine = get_engine()
with engine.begin() as conn:
conn.exec_driver_sql("PRAGMA foreign_keys=ON")
conn.exec_driver_sql(f"DELETE FROM overlays WHERE id = {overlay_id}")
with session_scope() as s:
assert s.query(OverlayWorkshopItem).count() == 0
assert s.query(WorkshopItem).filter_by(steam_id="777").count() == 1
def test_job_has_overlay_id_column(db) -> None:
user_id = _make_user("alice")
with session_scope() as s:
ov = Overlay(name="my-maps", path="1", type="workshop", user_id=user_id)
s.add(ov)
s.flush()
s.add(
Job(
user_id=user_id,
server_id=None,
overlay_id=ov.id,
operation="build_overlay",
state="queued",
)
)
s.flush()
with session_scope() as s:
job = s.query(Job).filter_by(operation="build_overlay").one()
assert job.overlay_id is not None
assert job.server_id is None
def test_overlay_id_does_not_reuse_after_delete(db) -> None:
"""SQLite AUTOINCREMENT must guarantee deleted IDs are never reused."""
user_id = _make_user("alice")
with session_scope() as s:
s.add(Overlay(name="first", path="1", type="workshop", user_id=user_id))
s.add(Overlay(name="second", path="2", type="workshop", user_id=user_id))
s.flush()
ids_before = sorted(o.id for o in s.query(Overlay).all())
last_id = ids_before[-1]
with session_scope() as s:
last = s.query(Overlay).filter_by(id=last_id).one()
s.delete(last)
s.flush()
with session_scope() as s:
s.add(Overlay(name="third", path="3", type="workshop", user_id=user_id))
s.flush()
new_id = s.query(Overlay).filter_by(name="third").one().id
assert new_id > last_id, (
f"AUTOINCREMENT should never reuse IDs, but got {new_id} after deleting {last_id}"
)

View file

@ -0,0 +1,23 @@
"""Tests for workshop_paths cache-resolution helpers."""
from pathlib import Path
import pytest
from l4d2web.services import workshop_paths
def test_workshop_cache_root_uses_left4me_root(monkeypatch, tmp_path: Path) -> None:
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
assert workshop_paths.workshop_cache_root() == tmp_path / "workshop_cache"
def test_cache_path_returns_id_only_filename(monkeypatch, tmp_path: Path) -> None:
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
assert workshop_paths.cache_path("12345") == tmp_path / "workshop_cache" / "12345.vpk"
@pytest.mark.parametrize("bad", ["abc", "", "12/34", "..", "../etc", "1 2", " 1"])
def test_cache_path_rejects_non_digit_steam_id(monkeypatch, tmp_path: Path, bad: str) -> None:
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
with pytest.raises(ValueError):
workshop_paths.cache_path(bad)

View file

@ -0,0 +1,283 @@
"""Tests for the workshop overlay routes (add items, remove items, build,
admin refresh)."""
from __future__ import annotations
from typing import Iterable
from unittest.mock import patch
import pytest
from l4d2web.app import create_app
from l4d2web.auth import hash_password
from l4d2web.db import init_db, session_scope
from l4d2web.models import (
Job,
Overlay,
OverlayWorkshopItem,
User,
WorkshopItem,
)
from l4d2web.services import steam_workshop
def _meta(steam_id: str, *, app_id: int = 550, result: int = 1) -> steam_workshop.WorkshopMetadata:
return steam_workshop.WorkshopMetadata(
steam_id=steam_id,
title=f"Item {steam_id}",
filename=f"{steam_id}.vpk",
file_url=f"https://example.com/{steam_id}.vpk",
file_size=42,
time_updated=1700000000,
preview_url=f"https://example.com/preview-{steam_id}.jpg",
consumer_app_id=app_id,
result=result,
)
@pytest.fixture
def env_user(tmp_path, monkeypatch):
db_url = f"sqlite:///{tmp_path/'wr.db'}"
monkeypatch.setenv("DATABASE_URL", db_url)
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
init_db()
with session_scope() as session:
user = User(username="alice", password_digest=hash_password("x"), admin=False)
admin = User(username="admin", password_digest=hash_password("x"), admin=True)
session.add_all([user, admin])
session.flush()
user_id = user.id
admin_id = admin.id
def login(uid):
c = app.test_client()
with c.session_transaction() as sess:
sess["user_id"] = uid
sess["csrf_token"] = "test-token"
return c
return app, login, user_id, admin_id
@pytest.fixture
def overlay_for(env_user):
app, login, user_id, admin_id = env_user
user_client = login(user_id)
response = user_client.post(
"/overlays",
data={"name": "my-maps", "type": "workshop"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302, response.get_data(as_text=True)
with session_scope() as session:
overlay = session.query(Overlay).filter_by(name="my-maps").one()
overlay_id = overlay.id
return app, login, user_id, admin_id, overlay_id
def _patch_steam(metas: Iterable[steam_workshop.WorkshopMetadata]):
return patch.object(steam_workshop, "fetch_metadata_batch", return_value=list(metas))
def test_add_single_item_creates_association_and_enqueues_build(overlay_for):
app, login, user_id, _admin_id, overlay_id = overlay_for
user_client = login(user_id)
with _patch_steam([_meta("1001")]):
response = user_client.post(
f"/overlays/{overlay_id}/items",
data={"input": "1001", "input_mode": "items"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
assert response.headers["Location"].startswith("/jobs/")
with session_scope() as session:
n_assoc = session.query(OverlayWorkshopItem).count()
assert n_assoc == 1
wi = session.query(WorkshopItem).filter_by(steam_id="1001").one()
assert wi.title == "Item 1001"
assert wi.preview_url.endswith("preview-1001.jpg")
# Auto-enqueued build_overlay job.
jobs = session.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
assert len(jobs) == 1
assert jobs[0].state == "queued"
assert response.headers["Location"] == f"/jobs/{jobs[0].id}"
def test_add_multiline_batch_coalesces_into_one_build_job(overlay_for):
app, login, user_id, _admin_id, overlay_id = overlay_for
user_client = login(user_id)
with _patch_steam([_meta(s) for s in ("1001", "1002", "1003")]):
response = user_client.post(
f"/overlays/{overlay_id}/items",
data={"input": "1001\n1002\n1003", "input_mode": "items"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
assert response.headers["Location"].startswith("/jobs/")
with session_scope() as session:
assert session.query(OverlayWorkshopItem).count() == 3
jobs = session.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
assert len(jobs) == 1, "multi-item add should coalesce into a single build job"
def test_add_collection_resolves_members(overlay_for):
app, login, user_id, _admin_id, overlay_id = overlay_for
user_client = login(user_id)
with patch.object(steam_workshop, "resolve_collection", return_value=["1001", "1002"]) as resolve:
with _patch_steam([_meta("1001"), _meta("1002")]):
response = user_client.post(
f"/overlays/{overlay_id}/items",
data={"input": "555", "input_mode": "collection"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
assert response.headers["Location"].startswith("/jobs/")
resolve.assert_called_once_with("555")
with session_scope() as session:
assert session.query(OverlayWorkshopItem).count() == 2
def test_add_non_l4d2_item_returns_400(overlay_for):
app, login, user_id, _admin_id, overlay_id = overlay_for
user_client = login(user_id)
def raise_validation(*args, **kwargs):
raise steam_workshop.WorkshopValidationError("not L4D2")
with patch.object(steam_workshop, "fetch_metadata_batch", side_effect=raise_validation):
response = user_client.post(
f"/overlays/{overlay_id}/items",
data={"input": "9999", "input_mode": "items"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 400
assert b"not L4D2" in response.data
with session_scope() as session:
assert session.query(WorkshopItem).count() == 0
assert session.query(OverlayWorkshopItem).count() == 0
def test_add_duplicate_item_does_not_500(overlay_for):
app, login, user_id, _admin_id, overlay_id = overlay_for
user_client = login(user_id)
with _patch_steam([_meta("1001")]):
first = user_client.post(
f"/overlays/{overlay_id}/items",
data={"input": "1001", "input_mode": "items"},
headers={"X-CSRF-Token": "test-token"},
)
assert first.status_code == 302
with _patch_steam([_meta("1001")]):
second = user_client.post(
f"/overlays/{overlay_id}/items",
data={"input": "1001", "input_mode": "items"},
headers={"X-CSRF-Token": "test-token"},
)
assert second.status_code == 302
with session_scope() as session:
assert session.query(OverlayWorkshopItem).count() == 1
def test_remove_item_drops_association_and_enqueues_rebuild(overlay_for):
app, login, user_id, _admin_id, overlay_id = overlay_for
user_client = login(user_id)
with _patch_steam([_meta("1001")]):
user_client.post(
f"/overlays/{overlay_id}/items",
data={"input": "1001", "input_mode": "items"},
headers={"X-CSRF-Token": "test-token"},
)
with session_scope() as session:
wi = session.query(WorkshopItem).filter_by(steam_id="1001").one()
item_id = wi.id
response = user_client.post(
f"/overlays/{overlay_id}/items/{item_id}/delete",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
assert response.headers["Location"].startswith("/jobs/")
with session_scope() as session:
assert session.query(OverlayWorkshopItem).count() == 0
# WorkshopItem itself remains (cache survives the association removal).
assert session.query(WorkshopItem).filter_by(steam_id="1001").one() is not None
# Coalesced into the same queued build_overlay job.
jobs = session.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
assert len(jobs) == 1
def test_manual_build_button_enqueues_job(overlay_for):
app, login, user_id, _admin_id, overlay_id = overlay_for
user_client = login(user_id)
response = user_client.post(
f"/overlays/{overlay_id}/build",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
assert response.headers["Location"].startswith("/jobs/")
with session_scope() as session:
jobs = session.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
assert len(jobs) == 1
def test_admin_refresh_enqueues_global_job(env_user):
app, login, user_id, admin_id = env_user
admin_client = login(admin_id)
response = admin_client.post(
"/admin/workshop/refresh",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 302
assert response.headers["Location"] == "/admin/jobs"
with session_scope() as session:
jobs = session.query(Job).filter_by(operation="refresh_workshop_items").all()
assert len(jobs) == 1
assert jobs[0].state == "queued"
def test_non_admin_cannot_refresh(env_user):
app, login, user_id, _admin_id = env_user
user_client = login(user_id)
response = user_client.post(
"/admin/workshop/refresh",
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 403
def test_other_user_cannot_modify_workshop_overlay(overlay_for):
app, login, user_id, _admin_id, overlay_id = overlay_for
with session_scope() as session:
intruder = User(username="bob", password_digest=hash_password("x"), admin=False)
session.add(intruder)
session.flush()
intruder_id = intruder.id
intruder_client = login(intruder_id)
response = intruder_client.post(
f"/overlays/{overlay_id}/items",
data={"input": "1001", "input_mode": "items"},
headers={"X-CSRF-Token": "test-token"},
)
assert response.status_code == 403