Compare commits
59 commits
harden-bou
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1166e13e44 | ||
|
|
0d906605e9 | ||
|
|
196d2db33e | ||
|
|
6b4eef22c2 | ||
|
|
c8a2d563ce | ||
|
|
fb3c6be052 | ||
|
|
5e2c771276 | ||
|
|
ebddb0fab2 | ||
|
|
406f2196f8 | ||
|
|
023cc5c9b0 | ||
|
|
f6ca85fc6f | ||
|
|
abc907b14b | ||
|
|
7e66936d03 | ||
|
|
ae443299c8 | ||
|
|
4ee8f6af44 | ||
|
|
efaaf84cd9 | ||
|
|
a62f26ba4a | ||
|
|
908bca3687 | ||
|
|
cf865d4915 | ||
|
|
06ae84fbe4 | ||
|
|
1e62a44c16 | ||
|
|
e51a4d58a4 | ||
|
|
75e703e1a4 | ||
|
|
d351bcbee5 | ||
|
|
be22744d54 | ||
|
|
879c54cbda | ||
|
|
9f476e3456 | ||
|
|
d29afa41fa | ||
|
|
43dc9b0ccf | ||
|
|
78ead0b41d | ||
|
|
9985ecc56c | ||
|
|
172e574a00 | ||
|
|
93a60befb6 | ||
|
|
d5b321b557 | ||
|
|
db120d77d3 | ||
|
|
d5d710afa7 | ||
|
|
38548ab0d7 | ||
|
|
4552af6544 | ||
|
|
ffc4cdbd7d | ||
|
|
92d6ebbe82 | ||
|
|
4f78574edd | ||
|
|
0e83ee07d7 | ||
|
|
b2a8d3d5e0 | ||
|
|
ac020d1e77 | ||
|
|
df1ccb4cca | ||
|
|
38a6fbbe1e | ||
|
|
700940d578 | ||
|
|
f0230e17d3 | ||
|
|
c6b41429ee | ||
|
|
2543a05c12 | ||
|
|
b46f52258d | ||
|
|
d18b397330 | ||
|
|
1968684c03 | ||
|
|
593611e194 | ||
|
|
56b9523d88 | ||
|
|
d14ed9c117 | ||
|
|
923a1840f4 | ||
|
|
7d9939c71d | ||
|
|
0210ecd301 |
97 changed files with 12544 additions and 331 deletions
1
.envrc
Normal file
1
.envrc
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
layout python python3.13
|
||||||
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -1,5 +1,7 @@
|
||||||
.worktrees/
|
.worktrees/
|
||||||
|
.claude/
|
||||||
.venv/
|
.venv/
|
||||||
|
.direnv/
|
||||||
.pytest_cache/
|
.pytest_cache/
|
||||||
__pycache__/
|
__pycache__/
|
||||||
*.pyc
|
*.pyc
|
||||||
|
|
|
||||||
|
|
@ -21,6 +21,7 @@ Do not invent architecture outside these plans unless explicitly requested.
|
||||||
### Workspace and tools
|
### Workspace and tools
|
||||||
|
|
||||||
- Do not use git worktrees.
|
- Do not use git worktrees.
|
||||||
|
- Local Python venv is direnv-managed via `.envrc` (Python 3.13). After fresh checkout: `direnv allow`, then `pip install -e ./l4d2host -e ./l4d2web pytest`. See README **Local development** for details.
|
||||||
|
|
||||||
### Naming and boundaries
|
### Naming and boundaries
|
||||||
|
|
||||||
|
|
@ -41,7 +42,7 @@ Do not invent architecture outside these plans unless explicitly requested.
|
||||||
- `logs <name> --lines <n> --follow/--no-follow`
|
- `logs <name> --lines <n> --follow/--no-follow`
|
||||||
- Runtime paths are rooted at `LEFT4ME_ROOT`, defaulting to `/var/lib/left4me`.
|
- Runtime paths are rooted at `LEFT4ME_ROOT`, defaulting to `/var/lib/left4me`.
|
||||||
- Deployment/config management owns global units under `/usr/local/lib/systemd/system` and privileged helpers under `/usr/local/libexec/left4me`.
|
- Deployment/config management owns global units under `/usr/local/lib/systemd/system` and privileged helpers under `/usr/local/libexec/left4me`.
|
||||||
- Overlays are external directories (no overlay content management here).
|
- Overlay directories are populated by the web app (workshop downloads, managed-global refresh). The host library only mounts them.
|
||||||
- Fail-fast subprocess behavior; pass raw stderr; propagate return code.
|
- Fail-fast subprocess behavior; pass raw stderr; propagate return code.
|
||||||
- No lock manager, no rollback, no preflight runtime checks.
|
- No lock manager, no rollback, no preflight runtime checks.
|
||||||
- Delete missing instance/runtime dirs must succeed (no-op).
|
- Delete missing instance/runtime dirs must succeed (no-op).
|
||||||
|
|
|
||||||
13
README.md
13
README.md
|
|
@ -27,7 +27,7 @@ Implementation plans remain the source of truth for architecture and task sequen
|
||||||
- `logs <name> --lines <n> --follow/--no-follow`
|
- `logs <name> --lines <n> --follow/--no-follow`
|
||||||
- The web app calls host operations through `l4d2ctl`, not direct `l4d2host` imports.
|
- The web app calls host operations through `l4d2ctl`, not direct `l4d2host` imports.
|
||||||
- Deployment uses `/var/lib/left4me` for runtime state, `/opt/left4me` for repository contents and the virtualenv, `/etc/left4me` for environment files, and global units under `/usr/local/lib/systemd/system`.
|
- Deployment uses `/var/lib/left4me` for runtime state, `/opt/left4me` for repository contents and the virtualenv, `/etc/left4me` for environment files, and global units under `/usr/local/lib/systemd/system`.
|
||||||
- Overlay handling is directory-based and externally populated.
|
- Overlay handling is directory-based; the web app populates each overlay (workshop downloads, managed-global refresh).
|
||||||
- No lock manager, no rollback, no preflight checks in host library.
|
- No lock manager, no rollback, no preflight checks in host library.
|
||||||
- CLI propagates subprocess failures via stderr and return code.
|
- CLI propagates subprocess failures via stderr and return code.
|
||||||
- `delete` on missing instance is no-op success.
|
- `delete` on missing instance is no-op success.
|
||||||
|
|
@ -50,13 +50,22 @@ Implementation plans remain the source of truth for architecture and task sequen
|
||||||
|
|
||||||
See `deploy/README.md` for the Linux test deployment contract, including the runtime user, target filesystem layout, systemd units, privileged helpers, sudoers rules, admin bootstrap, and overlay reference rules.
|
See `deploy/README.md` for the Linux test deployment contract, including the runtime user, target filesystem layout, systemd units, privileged helpers, sudoers rules, admin bootstrap, and overlay reference rules.
|
||||||
|
|
||||||
|
## Local development
|
||||||
|
|
||||||
|
This repo uses [direnv](https://direnv.net/) to auto-activate a Python 3.13 venv on `cd` (matching the Debian Trixie production target). With direnv installed and hooked into your shell:
|
||||||
|
|
||||||
|
1. `direnv allow` once per fresh checkout (and after any `.envrc` change).
|
||||||
|
2. `cd` out and back in — `.direnv/python-3.13/` is created and put on `PATH`.
|
||||||
|
3. `pip install -e ./l4d2host -e ./l4d2web` to install both packages editable.
|
||||||
|
4. `pip install pytest` to run the test suites (`pytest tests/` inside either subproject).
|
||||||
|
|
||||||
## Tech Stack (planned)
|
## Tech Stack (planned)
|
||||||
|
|
||||||
- Python 3.12+
|
- Python 3.12+
|
||||||
- Typer, PyYAML, pytest
|
- Typer, PyYAML, pytest
|
||||||
- Flask, SQLAlchemy, Alembic
|
- Flask, SQLAlchemy, Alembic
|
||||||
- HTMX (vendored locally), custom CSS, SSE
|
- HTMX (vendored locally), custom CSS, SSE
|
||||||
- systemd user units, fuse-overlayfs, steamcmd
|
- systemd units, kernel overlayfs (mounted via the `left4me-overlay` privileged helper), steamcmd
|
||||||
|
|
||||||
## Recommended Implementation Order
|
## Recommended Implementation Order
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -12,12 +12,14 @@ The deployment uses these paths:
|
||||||
- `/opt/left4me`: deployed repository contents.
|
- `/opt/left4me`: deployed repository contents.
|
||||||
- `/var/lib/left4me/left4me.db`: SQLite database used by the web app.
|
- `/var/lib/left4me/left4me.db`: SQLite database used by the web app.
|
||||||
- `/var/lib/left4me/installation`: shared L4D2 installation.
|
- `/var/lib/left4me/installation`: shared L4D2 installation.
|
||||||
- `/var/lib/left4me/overlays`: externally managed overlay directories.
|
- `/var/lib/left4me/overlays`: overlay directories. Each overlay lives at `${overlay_id}` under here.
|
||||||
|
- `/var/lib/left4me/workshop_cache`: deduplicated cache of `.vpk` files downloaded for workshop overlays. One file per Steam item, named `{steam_id}.vpk`. Workshop overlays symlink into this tree.
|
||||||
|
- `/var/lib/left4me/global_overlay_cache`: cache of non-Steam map archives and extracted `.vpk` files used by managed global map overlays.
|
||||||
- `/var/lib/left4me/instances`: rendered instance specifications and per-instance state.
|
- `/var/lib/left4me/instances`: rendered instance specifications and per-instance state.
|
||||||
- `/var/lib/left4me/runtime`: per-instance runtime mount directories.
|
- `/var/lib/left4me/runtime`: per-instance runtime mount directories.
|
||||||
- `/var/lib/left4me/tmp`: temporary files used by deployment/runtime operations.
|
- `/var/lib/left4me/tmp`: temporary files used by deployment/runtime operations.
|
||||||
- `/usr/local/lib/systemd/system`: global systemd unit files, including `left4me-server@.service`.
|
- `/usr/local/lib/systemd/system`: global systemd unit files, including `left4me-server@.service`.
|
||||||
- `/usr/local/libexec/left4me`: privileged helper commands, including `left4me-systemctl` and `left4me-journalctl`.
|
- `/usr/local/libexec/left4me`: privileged helper commands, including `left4me-systemctl`, `left4me-journalctl`, and `left4me-overlay` (the latter mounts the per-instance kernel overlay in PID 1's mount namespace via `nsenter`).
|
||||||
- `/etc/sudoers.d/left4me`: sudoers rules allowing the web/runtime commands to call the helpers non-interactively.
|
- `/etc/sudoers.d/left4me`: sudoers rules allowing the web/runtime commands to call the helpers non-interactively.
|
||||||
|
|
||||||
Static units are generated for `/var/lib/left4me`. If `LEFT4ME_ROOT` changes, regenerate and reinstall the unit files instead of reusing the existing static units.
|
Static units are generated for `/var/lib/left4me`. If `LEFT4ME_ROOT` changes, regenerate and reinstall the unit files instead of reusing the existing static units.
|
||||||
|
|
@ -54,13 +56,7 @@ Use a strong one-time password and rotate it after first login if needed.
|
||||||
|
|
||||||
## Overlay References
|
## Overlay References
|
||||||
|
|
||||||
Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`. With the default deployment root, they resolve under `/var/lib/left4me/overlays`.
|
Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`. With the default deployment root, they resolve under `/var/lib/left4me/overlays`. New overlays use `${overlay_id}` as their path; the digit-only form is the only one created by the web app.
|
||||||
|
|
||||||
Valid examples:
|
|
||||||
|
|
||||||
- `standard`
|
|
||||||
- `competitive/base`
|
|
||||||
- `users/42/custom`
|
|
||||||
|
|
||||||
Invalid references are rejected:
|
Invalid references are rejected:
|
||||||
|
|
||||||
|
|
@ -69,4 +65,9 @@ Invalid references are rejected:
|
||||||
- Empty path components such as `competitive//base`.
|
- Empty path components such as `competitive//base`.
|
||||||
- Symlink escapes that resolve outside `${LEFT4ME_ROOT}/overlays`.
|
- Symlink escapes that resolve outside `${LEFT4ME_ROOT}/overlays`.
|
||||||
|
|
||||||
Overlay content is external to the host library and deployment contract. Populate overlay directories separately before referencing them from blueprints or instance specs.
|
The web app currently supports two overlay surfaces:
|
||||||
|
|
||||||
|
- `workshop` overlays (user-owned) — populated by downloading `.vpk` files from the public Steam Web API into `${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creating absolute symlinks under `${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`.
|
||||||
|
- `script` overlays — populated by an arbitrary user-authored bash script that runs inside `bubblewrap` + `systemd-run --scope` as the unprivileged `l4d2-sandbox` UID, with the overlay directory bind-mounted RW at `/overlay`. Resource caps: 1h walltime, 4 GB RAM, 512 tasks, 200% CPU, 20 GB post-build disk cap.
|
||||||
|
|
||||||
|
Both the caches and the overlay directories are owned by the `left4me` runtime user; if the web service ever runs as a different uid, ensure it shares a group with the host process and that both trees are group-readable.
|
||||||
|
|
|
||||||
|
|
@ -21,13 +21,15 @@ cleanup() {
|
||||||
}
|
}
|
||||||
trap cleanup EXIT INT HUP TERM
|
trap cleanup EXIT INT HUP TERM
|
||||||
|
|
||||||
tar -czf "$archive" \
|
COPYFILE_DISABLE=1 tar -czf "$archive" \
|
||||||
--exclude .git \
|
--exclude .git \
|
||||||
|
--exclude .claude \
|
||||||
--exclude .venv \
|
--exclude .venv \
|
||||||
--exclude __pycache__ \
|
--exclude __pycache__ \
|
||||||
--exclude .pytest_cache \
|
--exclude .pytest_cache \
|
||||||
--exclude '*.egg-info' \
|
--exclude '*.egg-info' \
|
||||||
--exclude 'l4d2web.db*' \
|
--exclude 'l4d2web.db*' \
|
||||||
|
--exclude '._*' \
|
||||||
-C "$repo_root" .
|
-C "$repo_root" .
|
||||||
|
|
||||||
remote_tmp=$(ssh "$target" 'mktemp -d')
|
remote_tmp=$(ssh "$target" 'mktemp -d')
|
||||||
|
|
@ -75,11 +77,17 @@ if ! id left4me >/dev/null 2>&1; then
|
||||||
$sudo_cmd useradd --system --home-dir /var/lib/left4me --create-home --shell /usr/sbin/nologin left4me
|
$sudo_cmd useradd --system --home-dir /var/lib/left4me --create-home --shell /usr/sbin/nologin left4me
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Sandbox uid for script-overlay builds. No home, no login shell — the bwrap
|
||||||
|
# invocation uses --uid/--gid to drop to it.
|
||||||
|
if ! id l4d2-sandbox >/dev/null 2>&1; then
|
||||||
|
$sudo_cmd useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox
|
||||||
|
fi
|
||||||
|
|
||||||
if command -v apt-get >/dev/null 2>&1; then
|
if command -v apt-get >/dev/null 2>&1; then
|
||||||
$sudo_cmd apt-get update
|
$sudo_cmd apt-get update
|
||||||
$sudo_cmd apt-get install -y python3 python3-venv python3-pip curl ca-certificates tar gzip fuse-overlayfs fuse3 sudo
|
$sudo_cmd apt-get install -y python3 python3-venv python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip-full
|
||||||
elif command -v dnf >/dev/null 2>&1; then
|
elif command -v dnf >/dev/null 2>&1; then
|
||||||
$sudo_cmd dnf install -y python3 python3-pip curl ca-certificates tar gzip fuse-overlayfs fuse3 sudo
|
$sudo_cmd dnf install -y python3 python3-pip curl ca-certificates tar gzip util-linux sudo coreutils p7zip p7zip-plugins
|
||||||
else
|
else
|
||||||
printf 'Unsupported package manager: expected apt-get or dnf\n' >&2
|
printf 'Unsupported package manager: expected apt-get or dnf\n' >&2
|
||||||
exit 1
|
exit 1
|
||||||
|
|
@ -94,9 +102,24 @@ $sudo_cmd mkdir -p \
|
||||||
/var/lib/left4me/overlays \
|
/var/lib/left4me/overlays \
|
||||||
/var/lib/left4me/instances \
|
/var/lib/left4me/instances \
|
||||||
/var/lib/left4me/runtime \
|
/var/lib/left4me/runtime \
|
||||||
|
/var/lib/left4me/workshop_cache \
|
||||||
/var/lib/left4me/tmp
|
/var/lib/left4me/tmp
|
||||||
|
|
||||||
$sudo_cmd chown -R left4me:left4me /var/lib/left4me /opt/left4me
|
$sudo_cmd chown left4me:left4me \
|
||||||
|
/var/lib/left4me \
|
||||||
|
/var/lib/left4me/installation \
|
||||||
|
/var/lib/left4me/overlays \
|
||||||
|
/var/lib/left4me/instances \
|
||||||
|
/var/lib/left4me/runtime \
|
||||||
|
/var/lib/left4me/workshop_cache \
|
||||||
|
/var/lib/left4me/tmp
|
||||||
|
|
||||||
|
# /var/lib/left4me is left4me's home dir (mode 0700 from useradd --create-home).
|
||||||
|
# Allow other uids (notably l4d2-sandbox, used by script overlay builds) to
|
||||||
|
# traverse — but not list — so the bwrap bind-mount can resolve the overlay
|
||||||
|
# path under the dropped privilege.
|
||||||
|
$sudo_cmd chmod 0711 /var/lib/left4me
|
||||||
|
$sudo_cmd chown -R left4me:left4me /opt/left4me
|
||||||
|
|
||||||
mkdir -p "$repo_tmp"
|
mkdir -p "$repo_tmp"
|
||||||
tar -xzf "$archive" -C "$repo_tmp"
|
tar -xzf "$archive" -C "$repo_tmp"
|
||||||
|
|
@ -115,7 +138,9 @@ $sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-web.
|
||||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-server@.service /usr/local/lib/systemd/system/left4me-server@.service
|
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-server@.service /usr/local/lib/systemd/system/left4me-server@.service
|
||||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-systemctl
|
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-systemctl
|
||||||
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-journalctl
|
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-journalctl
|
||||||
$sudo_cmd chmod 0755 /usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-journalctl
|
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-overlay
|
||||||
|
$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-script-sandbox /usr/local/libexec/left4me/left4me-script-sandbox
|
||||||
|
$sudo_cmd chmod 0755 /usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-script-sandbox
|
||||||
$sudo_cmd cp /opt/left4me/deploy/files/etc/sudoers.d/left4me /etc/sudoers.d/left4me
|
$sudo_cmd cp /opt/left4me/deploy/files/etc/sudoers.d/left4me /etc/sudoers.d/left4me
|
||||||
$sudo_cmd chmod 0440 /etc/sudoers.d/left4me
|
$sudo_cmd chmod 0440 /etc/sudoers.d/left4me
|
||||||
$sudo_cmd visudo -cf /etc/sudoers.d/left4me
|
$sudo_cmd visudo -cf /etc/sudoers.d/left4me
|
||||||
|
|
@ -123,6 +148,12 @@ $sudo_cmd visudo -cf /etc/sudoers.d/left4me
|
||||||
$sudo_cmd cp /opt/left4me/deploy/templates/etc/left4me/host.env /etc/left4me/host.env
|
$sudo_cmd cp /opt/left4me/deploy/templates/etc/left4me/host.env /etc/left4me/host.env
|
||||||
$sudo_cmd chmod 0644 /etc/left4me/host.env
|
$sudo_cmd chmod 0644 /etc/left4me/host.env
|
||||||
|
|
||||||
|
# Sandbox-only resolver config; bind-mounted into the script sandbox's /etc/resolv.conf
|
||||||
|
# so DNS still works when IPAddressDeny= blocks the host's (typically private-IP) resolver.
|
||||||
|
$sudo_cmd install -m 0644 -o root -g root \
|
||||||
|
/opt/left4me/deploy/files/etc/left4me/sandbox-resolv.conf \
|
||||||
|
/etc/left4me/sandbox-resolv.conf
|
||||||
|
|
||||||
if [ ! -f /etc/left4me/web.env ]; then
|
if [ ! -f /etc/left4me/web.env ]; then
|
||||||
secret_key=$(python3 -c 'import secrets; print(secrets.token_hex(32))')
|
secret_key=$(python3 -c 'import secrets; print(secrets.token_hex(32))')
|
||||||
tmp_web_env="$remote_tmp/web.env"
|
tmp_web_env="$remote_tmp/web.env"
|
||||||
|
|
@ -130,6 +161,7 @@ if [ ! -f /etc/left4me/web.env ]; then
|
||||||
printf 'DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n'
|
printf 'DATABASE_URL=sqlite:////var/lib/left4me/left4me.db\n'
|
||||||
printf 'SECRET_KEY=%s\n' "$secret_key"
|
printf 'SECRET_KEY=%s\n' "$secret_key"
|
||||||
printf 'JOB_WORKER_THREADS=4\n'
|
printf 'JOB_WORKER_THREADS=4\n'
|
||||||
|
printf 'SESSION_COOKIE_SECURE=false\n'
|
||||||
} > "$tmp_web_env"
|
} > "$tmp_web_env"
|
||||||
$sudo_cmd install -m 0640 -o root -g left4me "$tmp_web_env" /etc/left4me/web.env
|
$sudo_cmd install -m 0640 -o root -g left4me "$tmp_web_env" /etc/left4me/web.env
|
||||||
fi
|
fi
|
||||||
|
|
@ -140,15 +172,30 @@ fi
|
||||||
run_as_left4me /opt/left4me/.venv/bin/python -m pip install --upgrade pip
|
run_as_left4me /opt/left4me/.venv/bin/python -m pip install --upgrade pip
|
||||||
run_as_left4me /opt/left4me/.venv/bin/pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web
|
run_as_left4me /opt/left4me/.venv/bin/pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web
|
||||||
|
|
||||||
run_left4me_with_env env \
|
|
||||||
JOB_WORKER_ENABLED=false \
|
|
||||||
/opt/left4me/.venv/bin/python -c "from l4d2web.app import create_app; create_app()"
|
|
||||||
|
|
||||||
run_as_left4me sh -c "cd /opt/left4me/l4d2web && set -a; . /etc/left4me/host.env; . /etc/left4me/web.env; set +a; env \
|
run_as_left4me sh -c "cd /opt/left4me/l4d2web && set -a; . /etc/left4me/host.env; . /etc/left4me/web.env; set +a; env \
|
||||||
JOB_WORKER_ENABLED=false \
|
JOB_WORKER_ENABLED=false \
|
||||||
PYTHONPATH=/opt/left4me \
|
PYTHONPATH=/opt/left4me \
|
||||||
/opt/left4me/.venv/bin/alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head"
|
/opt/left4me/.venv/bin/alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head"
|
||||||
|
|
||||||
|
# Tighten the application database to left4me:left4me 0640. The DB is
|
||||||
|
# created by the web app at first start with the default 0644 umask, which
|
||||||
|
# makes it world-readable on the host. The script-overlay sandbox runs as a
|
||||||
|
# separate system uid (l4d2-sandbox) which is NOT in the left4me group —
|
||||||
|
# 0640 blocks it via "other". The owner (left4me) keeps read+write so the
|
||||||
|
# web service can update the DB.
|
||||||
|
#
|
||||||
|
# SQLite in WAL mode (the default in this app) maintains -wal and -shm
|
||||||
|
# sidecar files; both must also be writable by the web service. If a previous
|
||||||
|
# operator opened the DB as root (e.g. for ad-hoc inspection), the sidecars
|
||||||
|
# may have ended up root-owned, which makes SQLite report "readonly database"
|
||||||
|
# on the next write. Re-chown them defensively. Idempotent on rerun.
|
||||||
|
for db_file in /var/lib/left4me/left4me.db /var/lib/left4me/left4me.db-wal /var/lib/left4me/left4me.db-shm; do
|
||||||
|
if [ -f "$db_file" ]; then
|
||||||
|
$sudo_cmd chown left4me:left4me "$db_file"
|
||||||
|
$sudo_cmd chmod 0640 "$db_file"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
if [ -f "$remote_tmp/admin_username" ] && [ -f "$remote_tmp/admin_password" ]; then
|
if [ -f "$remote_tmp/admin_username" ] && [ -f "$remote_tmp/admin_password" ]; then
|
||||||
LEFT4ME_ADMIN_USERNAME=$(cat "$remote_tmp/admin_username")
|
LEFT4ME_ADMIN_USERNAME=$(cat "$remote_tmp/admin_username")
|
||||||
LEFT4ME_ADMIN_PASSWORD=$(cat "$remote_tmp/admin_password")
|
LEFT4ME_ADMIN_PASSWORD=$(cat "$remote_tmp/admin_password")
|
||||||
|
|
@ -165,6 +212,62 @@ if [ -f "$remote_tmp/admin_username" ] && [ -f "$remote_tmp/admin_password" ]; t
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# One-shot migration: fuse-overlayfs running as the left4me user used
|
||||||
|
# user.fuseoverlayfs.* xattrs for whiteouts and opaque-dir markers; kernel
|
||||||
|
# overlayfs ignores those entirely, so a pre-existing upper/ from the fuse
|
||||||
|
# era would resurrect "deleted" files. Wipe upper/ and work/ for every
|
||||||
|
# instance once, gated by a sentinel file so reruns are no-ops.
|
||||||
|
overlay_sentinel=/var/lib/left4me/.kernel-overlay-migrated
|
||||||
|
if [ ! -e "$overlay_sentinel" ]; then
|
||||||
|
$sudo_cmd sh -c "systemctl stop 'left4me-server@*.service' 2>/dev/null || true"
|
||||||
|
$sudo_cmd systemctl stop left4me-web.service 2>/dev/null || true
|
||||||
|
$sudo_cmd sh -c "findmnt -t fuse.fuse-overlayfs -o TARGET --noheadings 2>/dev/null | xargs -r -n1 umount -l 2>/dev/null || true"
|
||||||
|
$sudo_cmd sh -c "findmnt -t overlay -o TARGET --noheadings 2>/dev/null | grep '/var/lib/left4me/runtime/' | xargs -r -n1 umount -l 2>/dev/null || true"
|
||||||
|
$sudo_cmd sh -c 'for d in /var/lib/left4me/runtime/*/; do [ -d "$d" ] || continue; rm -rf "$d/upper" "$d/work"; mkdir -p "$d/upper" "$d/work"; chown left4me:left4me "$d/upper" "$d/work"; done'
|
||||||
|
$sudo_cmd touch "$overlay_sentinel"
|
||||||
|
$sudo_cmd chown left4me:left4me "$overlay_sentinel"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# One-shot migration: 0005_script_overlays drops the legacy
|
||||||
|
# l4d2center_maps / cedapug_maps overlay rows but doesn't touch their
|
||||||
|
# directories under /var/lib/left4me/overlays/{id}. Without cleanup, when
|
||||||
|
# AUTOINCREMENT (or its absence after the 0002 batch_alter_table recreate)
|
||||||
|
# re-issues an id matching one of those orphan dirs, the web app's
|
||||||
|
# create_overlay_directory(exist_ok=False) fails with FileExistsError.
|
||||||
|
# Sweep any overlay dir whose id has no matching DB row, plus the
|
||||||
|
# now-unused global_overlay_cache.
|
||||||
|
overlay_orphan_sentinel=/var/lib/left4me/.script-overlays-orphans-cleaned
|
||||||
|
if [ ! -e "$overlay_orphan_sentinel" ]; then
|
||||||
|
$sudo_cmd rm -rf /var/lib/left4me/global_overlay_cache
|
||||||
|
$sudo_cmd sh -c '
|
||||||
|
cd /var/lib/left4me/overlays || exit 0
|
||||||
|
ids_in_db=$(/opt/left4me/.venv/bin/python -c "
|
||||||
|
import sqlite3
|
||||||
|
c = sqlite3.connect(\"/var/lib/left4me/left4me.db\")
|
||||||
|
print(\" \".join(str(r[0]) for r in c.execute(\"SELECT id FROM overlays\")))
|
||||||
|
")
|
||||||
|
for d in */; do
|
||||||
|
id=${d%/}
|
||||||
|
case " $ids_in_db " in
|
||||||
|
*" $id "*) ;;
|
||||||
|
*) echo "removing orphan overlay dir: $id"; rm -rf "$id" ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
'
|
||||||
|
$sudo_cmd touch "$overlay_orphan_sentinel"
|
||||||
|
$sudo_cmd chown left4me:left4me "$overlay_orphan_sentinel"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Seed example script overlays (cedapug, l4d2center, competitive_rework, ...)
|
||||||
|
# as system-wide rows from /opt/left4me/examples/script-overlays/. Idempotent:
|
||||||
|
# subsequent deploys refresh the script body in place, leaving the row id and
|
||||||
|
# overlay directory intact.
|
||||||
|
run_left4me_with_env env \
|
||||||
|
JOB_WORKER_ENABLED=false \
|
||||||
|
PYTHONPATH=/opt/left4me \
|
||||||
|
/opt/left4me/.venv/bin/flask --app l4d2web.app:create_app \
|
||||||
|
seed-script-overlays /opt/left4me/examples/script-overlays
|
||||||
|
|
||||||
$sudo_cmd systemctl daemon-reload
|
$sudo_cmd systemctl daemon-reload
|
||||||
$sudo_cmd systemctl enable --now left4me-web.service
|
$sudo_cmd systemctl enable --now left4me-web.service
|
||||||
$sudo_cmd systemctl restart left4me-web.service
|
$sudo_cmd systemctl restart left4me-web.service
|
||||||
|
|
|
||||||
6
deploy/files/etc/left4me/sandbox-resolv.conf
Normal file
6
deploy/files/etc/left4me/sandbox-resolv.conf
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
# Sandbox-only resolver config — bind-mounted into script-overlay sandboxes
|
||||||
|
# at /etc/resolv.conf. The host's resolver (often a private/LAN DNS server)
|
||||||
|
# is unreachable from inside the sandbox because IPAddressDeny= blocks
|
||||||
|
# egress to RFC1918 / loopback. Public resolvers keep DNS working.
|
||||||
|
nameserver 1.1.1.1
|
||||||
|
nameserver 8.8.8.8
|
||||||
|
|
@ -1,3 +1,5 @@
|
||||||
Defaults:left4me !requiretty
|
Defaults:left4me !requiretty
|
||||||
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-systemctl *
|
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-systemctl *
|
||||||
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-journalctl *
|
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-journalctl *
|
||||||
|
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-overlay mount *, /usr/local/libexec/left4me/left4me-overlay umount *
|
||||||
|
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-script-sandbox
|
||||||
|
|
|
||||||
|
|
@ -12,13 +12,20 @@ Environment=HOME=/var/lib/left4me
|
||||||
Environment=PATH=/opt/left4me/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
Environment=PATH=/opt/left4me/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||||
EnvironmentFile=/etc/left4me/host.env
|
EnvironmentFile=/etc/left4me/host.env
|
||||||
EnvironmentFile=/etc/left4me/web.env
|
EnvironmentFile=/etc/left4me/web.env
|
||||||
ExecStart=/opt/left4me/.venv/bin/gunicorn --workers 1 --threads 8 --bind 0.0.0.0:8000 'l4d2web.app:create_app()'
|
ExecStart=/opt/left4me/.venv/bin/gunicorn --workers 1 --threads 32 --bind 0.0.0.0:8000 'l4d2web.app:create_app()'
|
||||||
Restart=on-failure
|
Restart=on-failure
|
||||||
RestartSec=3
|
RestartSec=3
|
||||||
NoNewPrivileges=true
|
# NoNewPrivileges intentionally not set: the worker invokes sudo to run
|
||||||
PrivateTmp=true
|
# the left4me-systemctl, left4me-journalctl, and left4me-overlay
|
||||||
|
# privileged helpers, all setuid via sudo.
|
||||||
|
# ProtectSystem=full + ReadWritePaths implicitly give this unit a private
|
||||||
|
# mount namespace, but mount visibility no longer depends on it: overlay
|
||||||
|
# mounts are performed by the left4me-overlay helper, which nsenters into
|
||||||
|
# PID 1's mount namespace, so the resulting mount lives in the host
|
||||||
|
# namespace where the per-instance gameserver units can see it.
|
||||||
ProtectSystem=full
|
ProtectSystem=full
|
||||||
ReadWritePaths=/var/lib/left4me
|
ReadWritePaths=/var/lib/left4me
|
||||||
|
PrivateTmp=true
|
||||||
|
|
||||||
[Install]
|
[Install]
|
||||||
WantedBy=multi-user.target
|
WantedBy=multi-user.target
|
||||||
|
|
|
||||||
188
deploy/files/usr/local/libexec/left4me/left4me-overlay
Normal file
188
deploy/files/usr/local/libexec/left4me/left4me-overlay
Normal file
|
|
@ -0,0 +1,188 @@
|
||||||
|
#!/usr/bin/python3
|
||||||
|
"""Privileged overlay mount helper for left4me.
|
||||||
|
|
||||||
|
Invoked via sudo by the left4me runtime user. Validates inputs strictly,
|
||||||
|
then enters PID 1's mount namespace via nsenter to perform the actual
|
||||||
|
mount/umount syscall, so the resulting mount lives in the host namespace
|
||||||
|
and is visible to the systemd-managed gameserver units.
|
||||||
|
|
||||||
|
Verbs:
|
||||||
|
mount <name> Reads ${LEFT4ME_ROOT}/instances/<name>/instance.env
|
||||||
|
for L4D2_LOWERDIRS, validates every lowerdir is
|
||||||
|
under one of installation/overlays/workshop_cache/
|
||||||
|
global_overlay_cache, then mounts the kernel
|
||||||
|
overlay at runtime/<name>/merged.
|
||||||
|
umount <name> Unmounts runtime/<name>/merged.
|
||||||
|
|
||||||
|
Set LEFT4ME_OVERLAY_PRINT_ONLY=1 to print the would-be argv (one line,
|
||||||
|
shell-quoted) and exit 0 instead of execv. Used by tests.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import shlex
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{0,63}$")
|
||||||
|
DEFAULT_ROOT = "/var/lib/left4me"
|
||||||
|
LOWERDIR_ALLOWLIST = (
|
||||||
|
"installation",
|
||||||
|
"overlays",
|
||||||
|
"global_overlay_cache",
|
||||||
|
"workshop_cache",
|
||||||
|
)
|
||||||
|
MAX_LOWERDIRS = 500
|
||||||
|
NSENTER = "/usr/bin/nsenter"
|
||||||
|
MOUNT_BIN = "/bin/mount"
|
||||||
|
UMOUNT_BIN = "/bin/umount"
|
||||||
|
|
||||||
|
|
||||||
|
def die(msg: str) -> None:
|
||||||
|
sys.stderr.write(f"left4me-overlay: {msg}\n")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
def root() -> Path:
|
||||||
|
return Path(os.environ.get("LEFT4ME_ROOT") or DEFAULT_ROOT)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_name(name: str) -> str:
|
||||||
|
if not NAME_RE.fullmatch(name):
|
||||||
|
die(f"invalid instance name: {name!r}")
|
||||||
|
return name
|
||||||
|
|
||||||
|
|
||||||
|
def parse_lowerdirs(env_path: Path) -> list[str]:
|
||||||
|
if not env_path.is_file():
|
||||||
|
die(f"instance.env not found: {env_path}")
|
||||||
|
raw = None
|
||||||
|
for line in env_path.read_text().splitlines():
|
||||||
|
if "=" not in line:
|
||||||
|
continue
|
||||||
|
key, value = line.split("=", 1)
|
||||||
|
if key.strip() == "L4D2_LOWERDIRS":
|
||||||
|
raw = value
|
||||||
|
break
|
||||||
|
if raw is None:
|
||||||
|
die(f"L4D2_LOWERDIRS not set in {env_path}")
|
||||||
|
if raw == "":
|
||||||
|
die(f"L4D2_LOWERDIRS is empty in {env_path}")
|
||||||
|
parts = raw.split(":")
|
||||||
|
if any(p == "" for p in parts):
|
||||||
|
die(f"L4D2_LOWERDIRS contains an empty entry: {raw!r}")
|
||||||
|
if len(parts) > MAX_LOWERDIRS:
|
||||||
|
die(f"L4D2_LOWERDIRS has {len(parts)} entries (cap {MAX_LOWERDIRS})")
|
||||||
|
return parts
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_under(allowed_roots: list[Path], path: Path) -> Path:
|
||||||
|
try:
|
||||||
|
canonical = path.resolve(strict=True)
|
||||||
|
except (FileNotFoundError, RuntimeError):
|
||||||
|
die(f"path does not exist or has a symlink loop: {path}")
|
||||||
|
for r in allowed_roots:
|
||||||
|
if canonical == r or r in canonical.parents:
|
||||||
|
return canonical
|
||||||
|
die(f"path is outside the permitted roots: {path} (resolved: {canonical})")
|
||||||
|
|
||||||
|
|
||||||
|
_LISTXATTR = getattr(os, "listxattr", None)
|
||||||
|
|
||||||
|
|
||||||
|
def _entry_has_fuse_xattr(path: str) -> str | None:
|
||||||
|
if _LISTXATTR is None:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
attrs = _LISTXATTR(path, follow_symlinks=False)
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
for a in attrs:
|
||||||
|
if a.startswith("user.fuseoverlayfs."):
|
||||||
|
return a
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def assert_no_fuse_xattrs(upper: Path) -> None:
|
||||||
|
if not upper.exists() or _LISTXATTR is None:
|
||||||
|
return
|
||||||
|
for dirpath, dirnames, filenames in os.walk(upper):
|
||||||
|
for entry in (dirpath, *(os.path.join(dirpath, n) for n in dirnames),
|
||||||
|
*(os.path.join(dirpath, n) for n in filenames)):
|
||||||
|
tainted = _entry_has_fuse_xattr(entry)
|
||||||
|
if tainted:
|
||||||
|
die(
|
||||||
|
f"upperdir contains fuse-overlayfs xattr {tainted!r} on {entry}; "
|
||||||
|
"wipe upper/ and work/ before mounting"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def exec_or_print(argv: list[str]) -> None:
|
||||||
|
if os.environ.get("LEFT4ME_OVERLAY_PRINT_ONLY") == "1":
|
||||||
|
print(" ".join(shlex.quote(a) for a in argv))
|
||||||
|
sys.exit(0)
|
||||||
|
os.execv(argv[0], argv)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_mount(name: str) -> None:
|
||||||
|
name = validate_name(name)
|
||||||
|
r = root()
|
||||||
|
instance_env = r / "instances" / name / "instance.env"
|
||||||
|
raw_lowerdirs = parse_lowerdirs(instance_env)
|
||||||
|
|
||||||
|
allowed_roots = [(r / sub).resolve() for sub in LOWERDIR_ALLOWLIST]
|
||||||
|
canonical_lowerdirs = [str(canonical_under(allowed_roots, Path(p))) for p in raw_lowerdirs]
|
||||||
|
|
||||||
|
runtime_name_dir = (r / "runtime" / name).resolve(strict=True)
|
||||||
|
upper = (runtime_name_dir / "upper").resolve(strict=True)
|
||||||
|
work = (runtime_name_dir / "work").resolve(strict=True)
|
||||||
|
merged = (runtime_name_dir / "merged").resolve(strict=True)
|
||||||
|
for label, path in (("upper", upper), ("work", work), ("merged", merged)):
|
||||||
|
if path.parent != runtime_name_dir:
|
||||||
|
die(f"{label} resolved outside runtime/{name}: {path}")
|
||||||
|
|
||||||
|
assert_no_fuse_xattrs(upper)
|
||||||
|
|
||||||
|
options = f"lowerdir={':'.join(canonical_lowerdirs)},upperdir={upper},workdir={work}"
|
||||||
|
argv = [
|
||||||
|
NSENTER,
|
||||||
|
"--mount=/proc/1/ns/mnt",
|
||||||
|
"--",
|
||||||
|
MOUNT_BIN,
|
||||||
|
"-t", "overlay",
|
||||||
|
"overlay",
|
||||||
|
"-o", options,
|
||||||
|
str(merged),
|
||||||
|
]
|
||||||
|
exec_or_print(argv)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_umount(name: str) -> None:
|
||||||
|
name = validate_name(name)
|
||||||
|
r = root()
|
||||||
|
runtime_name_dir = (r / "runtime" / name).resolve(strict=True)
|
||||||
|
merged = (runtime_name_dir / "merged").resolve(strict=True)
|
||||||
|
if merged.parent != runtime_name_dir:
|
||||||
|
die(f"merged resolved outside runtime/{name}: {merged}")
|
||||||
|
argv = [
|
||||||
|
NSENTER,
|
||||||
|
"--mount=/proc/1/ns/mnt",
|
||||||
|
"--",
|
||||||
|
UMOUNT_BIN,
|
||||||
|
str(merged),
|
||||||
|
]
|
||||||
|
exec_or_print(argv)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str]) -> None:
|
||||||
|
if len(argv) != 3 or argv[1] not in ("mount", "umount"):
|
||||||
|
sys.stderr.write("usage: left4me-overlay mount|umount <name>\n")
|
||||||
|
sys.exit(2)
|
||||||
|
if argv[1] == "mount":
|
||||||
|
cmd_mount(argv[2])
|
||||||
|
else:
|
||||||
|
cmd_umount(argv[2])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main(sys.argv)
|
||||||
68
deploy/files/usr/local/libexec/left4me/left4me-script-sandbox
Executable file
68
deploy/files/usr/local/libexec/left4me/left4me-script-sandbox
Executable file
|
|
@ -0,0 +1,68 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Privileged sandbox launcher for left4me script overlays.
|
||||||
|
#
|
||||||
|
# Invoked via sudo by the web user with two arguments:
|
||||||
|
# <overlay_id> numeric overlay id; bind-mounts /var/lib/left4me/overlays/<id>
|
||||||
|
# read-write at /overlay inside the sandbox.
|
||||||
|
# <script_path> absolute path to a bash file already written by the web app;
|
||||||
|
# bind-mounted read-only at /script.sh inside the sandbox.
|
||||||
|
#
|
||||||
|
# The script runs as a transient systemd .service with the full hardening
|
||||||
|
# surface: cgroup limits + walltime kill, NoNewPrivileges, ProtectSystem,
|
||||||
|
# ProtectHome, kernel-tunable / -module / -log protection, namespace
|
||||||
|
# restriction, address-family restriction, capability bounding (empty),
|
||||||
|
# seccomp filter (@system-service @network-io), MemoryDenyWriteExecute,
|
||||||
|
# LockPersonality, RestrictSUIDSGID. Network namespace is *not* restricted —
|
||||||
|
# scripts must reach the public internet to download workshop / l4d2center
|
||||||
|
# / cedapug content. PID namespace is shared with the host (no
|
||||||
|
# PrivatePID= directive in systemd); host PIDs are visible via /proc but
|
||||||
|
# not signal-able due to UID mismatch.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
[[ $# -eq 2 ]] || { echo "usage: $0 <overlay_id> <script>" >&2; exit 64; }
|
||||||
|
|
||||||
|
OVERLAY_ID=$1
|
||||||
|
SCRIPT=$2
|
||||||
|
|
||||||
|
[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]] || { echo "bad overlay id" >&2; exit 64; }
|
||||||
|
OVERLAY_DIR=/var/lib/left4me/overlays/$OVERLAY_ID
|
||||||
|
[[ -d $OVERLAY_DIR ]] || { echo "no overlay dir at $OVERLAY_DIR" >&2; exit 65; }
|
||||||
|
[[ -f $SCRIPT ]] || { echo "no script at $SCRIPT" >&2; exit 65; }
|
||||||
|
|
||||||
|
if [[ "${LEFT4ME_SCRIPT_SANDBOX_DRY_RUN:-}" == "1" ]]; then
|
||||||
|
echo "DRY RUN: overlay_id=$OVERLAY_ID script=$SCRIPT overlay_dir=$OVERLAY_DIR"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Make sure the sandbox UID owns the overlay dir so the script can write there.
|
||||||
|
# Idempotent: a no-op when the dir is already l4d2-sandbox-owned (re-run case),
|
||||||
|
# and corrects the ownership the first time the dir was created by the web app
|
||||||
|
# under the left4me UID. World-readable so the gameserver process (left4me)
|
||||||
|
# can read the overlay contents via the kernel-overlayfs lowerdir at runtime.
|
||||||
|
chown -R l4d2-sandbox:l4d2-sandbox "$OVERLAY_DIR"
|
||||||
|
chmod 0755 "$OVERLAY_DIR"
|
||||||
|
|
||||||
|
exec systemd-run --quiet --collect --wait --pipe \
|
||||||
|
--unit="left4me-script-${OVERLAY_ID}-$$" \
|
||||||
|
-p User=l4d2-sandbox -p Group=l4d2-sandbox \
|
||||||
|
-p NoNewPrivileges=yes \
|
||||||
|
-p ProtectSystem=strict -p ProtectHome=yes \
|
||||||
|
-p PrivateTmp=yes -p PrivateDevices=yes -p PrivateIPC=yes \
|
||||||
|
-p ProtectKernelTunables=yes -p ProtectKernelModules=yes \
|
||||||
|
-p ProtectKernelLogs=yes -p ProtectControlGroups=yes \
|
||||||
|
-p RestrictNamespaces=yes \
|
||||||
|
-p RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX" \
|
||||||
|
-p RestrictSUIDSGID=yes -p LockPersonality=yes \
|
||||||
|
-p MemoryDenyWriteExecute=yes \
|
||||||
|
-p SystemCallFilter="@system-service @network-io" \
|
||||||
|
-p SystemCallArchitectures=native \
|
||||||
|
-p CapabilityBoundingSet= -p AmbientCapabilities= \
|
||||||
|
-p IPAddressDeny="127.0.0.0/8 ::1/128 169.254.0.0/16 fe80::/10 224.0.0.0/4 ff00::/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 fc00::/7" \
|
||||||
|
-p TemporaryFileSystem="/etc /var/lib" \
|
||||||
|
-p BindReadOnlyPaths="/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf /etc/ssl /etc/ca-certificates /etc/nsswitch.conf /etc/alternatives ${SCRIPT}:/script.sh" \
|
||||||
|
-p BindPaths="${OVERLAY_DIR}:/overlay" \
|
||||||
|
-p WorkingDirectory=/overlay \
|
||||||
|
-p Environment="HOME=/tmp PATH=/usr/bin:/usr/sbin OVERLAY=/overlay" \
|
||||||
|
-p MemoryMax=4G -p MemorySwapMax=0 -p TasksMax=512 \
|
||||||
|
-p CPUQuota=200% -p RuntimeMaxSec=3600 \
|
||||||
|
-- /bin/bash /script.sh
|
||||||
|
|
@ -9,8 +9,14 @@ DEPLOY = ROOT / "deploy"
|
||||||
|
|
||||||
WEB_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-web.service"
|
WEB_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-web.service"
|
||||||
SERVER_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-server@.service"
|
SERVER_UNIT = DEPLOY / "files/usr/local/lib/systemd/system/left4me-server@.service"
|
||||||
|
GLOBAL_REFRESH_SERVICE = DEPLOY / "files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.service"
|
||||||
|
GLOBAL_REFRESH_TIMER = DEPLOY / "files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer"
|
||||||
|
SANDBOX_UNIT_DIR = DEPLOY / "files/usr/local/lib/systemd/system"
|
||||||
SYSTEMCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-systemctl"
|
SYSTEMCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-systemctl"
|
||||||
JOURNALCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-journalctl"
|
JOURNALCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-journalctl"
|
||||||
|
OVERLAY_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-overlay"
|
||||||
|
SCRIPT_SANDBOX_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-script-sandbox"
|
||||||
|
SANDBOX_RESOLV_CONF = DEPLOY / "files/etc/left4me/sandbox-resolv.conf"
|
||||||
SUDOERS = DEPLOY / "files/etc/sudoers.d/left4me"
|
SUDOERS = DEPLOY / "files/etc/sudoers.d/left4me"
|
||||||
HOST_ENV = DEPLOY / "templates/etc/left4me/host.env"
|
HOST_ENV = DEPLOY / "templates/etc/left4me/host.env"
|
||||||
WEB_ENV_TEMPLATE = DEPLOY / "templates/etc/left4me/web.env.template"
|
WEB_ENV_TEMPLATE = DEPLOY / "templates/etc/left4me/web.env.template"
|
||||||
|
|
@ -33,10 +39,18 @@ def test_web_unit_contains_required_runtime_contract():
|
||||||
assert "EnvironmentFile=/etc/left4me/web.env" in unit
|
assert "EnvironmentFile=/etc/left4me/web.env" in unit
|
||||||
assert "ExecStart=/opt/left4me/.venv/bin/gunicorn" in unit
|
assert "ExecStart=/opt/left4me/.venv/bin/gunicorn" in unit
|
||||||
assert "--workers 1" in unit
|
assert "--workers 1" in unit
|
||||||
assert "NoNewPrivileges=true" in unit
|
assert "--threads 32" in unit
|
||||||
|
# NoNewPrivileges must remain unset because sudo (used by the overlay,
|
||||||
|
# systemctl and journalctl helpers) is setuid.
|
||||||
|
assert "NoNewPrivileges=true" not in unit
|
||||||
|
# Restored now that fuse-overlayfs propagation is no longer the mechanism.
|
||||||
assert "PrivateTmp=true" in unit
|
assert "PrivateTmp=true" in unit
|
||||||
assert "ProtectSystem=full" in unit
|
assert "ProtectSystem=full" in unit
|
||||||
assert "ReadWritePaths=/var/lib/left4me" in unit
|
assert "ReadWritePaths=/var/lib/left4me" in unit
|
||||||
|
# Mounts now happen in PID 1's namespace via the left4me-overlay helper,
|
||||||
|
# so MountFlags propagation is irrelevant — and the previous assumption
|
||||||
|
# that MountFlags=shared made it work was incorrect.
|
||||||
|
assert "MountFlags=" not in unit
|
||||||
|
|
||||||
|
|
||||||
def test_server_unit_contains_required_runtime_contract():
|
def test_server_unit_contains_required_runtime_contract():
|
||||||
|
|
@ -142,10 +156,64 @@ def test_sudoers_allows_only_left4me_helpers_not_raw_system_tools():
|
||||||
"left4me ALL=(root) NOPASSWD: "
|
"left4me ALL=(root) NOPASSWD: "
|
||||||
"/usr/local/libexec/left4me/left4me-journalctl *"
|
"/usr/local/libexec/left4me/left4me-journalctl *"
|
||||||
) in sudoers
|
) in sudoers
|
||||||
|
assert "/usr/local/libexec/left4me/left4me-overlay mount *" in sudoers
|
||||||
|
assert "/usr/local/libexec/left4me/left4me-overlay umount *" in sudoers
|
||||||
|
assert (
|
||||||
|
"left4me ALL=(root) NOPASSWD: "
|
||||||
|
"/usr/local/libexec/left4me/left4me-script-sandbox"
|
||||||
|
) in sudoers
|
||||||
assert "/bin/systemctl" not in sudoers
|
assert "/bin/systemctl" not in sudoers
|
||||||
assert "/usr/bin/systemctl" not in sudoers
|
assert "/usr/bin/systemctl" not in sudoers
|
||||||
assert "/bin/journalctl" not in sudoers
|
assert "/bin/journalctl" not in sudoers
|
||||||
assert "/usr/bin/journalctl" not in sudoers
|
assert "/usr/bin/journalctl" not in sudoers
|
||||||
|
assert "/bin/mount" not in sudoers
|
||||||
|
assert "/bin/umount" not in sudoers
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_helper_is_python_with_strict_validation():
|
||||||
|
text = OVERLAY_HELPER.read_text()
|
||||||
|
assert text.startswith("#!/usr/bin/python3")
|
||||||
|
# Validation surface
|
||||||
|
assert "NAME_RE = re.compile" in text
|
||||||
|
assert "LOWERDIR_ALLOWLIST" in text
|
||||||
|
assert "user.fuseoverlayfs." in text
|
||||||
|
assert "MAX_LOWERDIRS = 500" in text
|
||||||
|
# Mounts via PID 1's mount namespace
|
||||||
|
assert "/proc/1/ns/mnt" in text
|
||||||
|
assert "nsenter" in text
|
||||||
|
# Verbs are mount and umount (not unmount)
|
||||||
|
assert '"mount"' in text and '"umount"' in text
|
||||||
|
assert '"unmount"' not in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_installs_overlay_helper_with_executable_mode():
|
||||||
|
script = DEPLOY_SCRIPT.read_text()
|
||||||
|
assert "/usr/local/libexec/left4me/left4me-overlay" in script
|
||||||
|
assert "chmod 0755" in script and "left4me-overlay" in script
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_does_not_install_fuse_overlayfs_apt_dep():
|
||||||
|
# fuse-overlayfs / fuse3 were the previous mount engine; kernel overlayfs
|
||||||
|
# replaces them. Comments in the migration block may legitimately mention
|
||||||
|
# the names, so scope this to the actual apt-get / dnf install lines.
|
||||||
|
install_lines = [
|
||||||
|
line for line in DEPLOY_SCRIPT.read_text().splitlines()
|
||||||
|
if ("apt-get install" in line or "dnf install" in line)
|
||||||
|
]
|
||||||
|
assert install_lines, "expected at least one apt/dnf install line"
|
||||||
|
for line in install_lines:
|
||||||
|
assert "fuse-overlayfs" not in line, line
|
||||||
|
assert "fuse3" not in line, line
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_runs_one_shot_kernel_overlay_migration():
|
||||||
|
script = DEPLOY_SCRIPT.read_text()
|
||||||
|
assert "/var/lib/left4me/.kernel-overlay-migrated" in script
|
||||||
|
# Migration should stop services + force-unmount stale mounts + wipe upper/work
|
||||||
|
assert "systemctl stop 'left4me-server@" in script
|
||||||
|
assert "systemctl stop left4me-web.service" in script
|
||||||
|
assert "findmnt -t overlay" in script
|
||||||
|
assert "/runtime/" in script and "rm -rf" in script and 'upper"' in script and 'work"' in script
|
||||||
|
|
||||||
|
|
||||||
def test_env_templates_contain_required_defaults():
|
def test_env_templates_contain_required_defaults():
|
||||||
|
|
@ -169,6 +237,7 @@ def test_deploy_script_has_safe_defaults_and_preserves_state() -> None:
|
||||||
assert "/var/lib/left4me/runtime" in script
|
assert "/var/lib/left4me/runtime" in script
|
||||||
assert "tar" in script
|
assert "tar" in script
|
||||||
assert "--exclude .venv" in script
|
assert "--exclude .venv" in script
|
||||||
|
assert "--exclude .claude" in script
|
||||||
assert "pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web" in script
|
assert "pip install -e /opt/left4me/l4d2host -e /opt/left4me/l4d2web" in script
|
||||||
assert "systemctl enable --now left4me-web.service" in script
|
assert "systemctl enable --now left4me-web.service" in script
|
||||||
assert "for attempt in" in script
|
assert "for attempt in" in script
|
||||||
|
|
@ -183,5 +252,254 @@ def test_deploy_script_has_safe_defaults_and_preserves_state() -> None:
|
||||||
assert "deploy/files" in script
|
assert "deploy/files" in script
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_does_not_recurse_into_runtime_state_mounts() -> None:
|
||||||
|
script = DEPLOY_SCRIPT.read_text()
|
||||||
|
|
||||||
|
assert "$sudo_cmd chown -R left4me:left4me /var/lib/left4me" not in script
|
||||||
|
assert "$sudo_cmd chown left4me:left4me \\" in script
|
||||||
|
assert "/var/lib/left4me/runtime \\" in script
|
||||||
|
assert "$sudo_cmd chown -R left4me:left4me /opt/left4me" in script
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_runs_migrations_before_app_initialization() -> None:
|
||||||
|
script = DEPLOY_SCRIPT.read_text()
|
||||||
|
|
||||||
|
assert "alembic -c /opt/left4me/l4d2web/alembic.ini upgrade head" in script
|
||||||
|
assert "from l4d2web.app import create_app; create_app()" not in script
|
||||||
|
|
||||||
|
|
||||||
def test_deploy_script_shell_syntax() -> None:
|
def test_deploy_script_shell_syntax() -> None:
|
||||||
subprocess.run(["sh", "-n", str(DEPLOY_SCRIPT)], check=True)
|
subprocess.run(["sh", "-n", str(DEPLOY_SCRIPT)], check=True)
|
||||||
|
|
||||||
|
|
||||||
|
def test_globals_refresh_units_removed():
|
||||||
|
"""Global-overlays subsystem deleted in favor of script overlays."""
|
||||||
|
assert not GLOBAL_REFRESH_SERVICE.exists()
|
||||||
|
assert not GLOBAL_REFRESH_TIMER.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_does_not_provision_globals_subsystem():
|
||||||
|
script = DEPLOY_SCRIPT.read_text()
|
||||||
|
|
||||||
|
# No mkdir/install of the deleted cache dir; mention in a one-shot
|
||||||
|
# `rm -rf` cleanup is fine.
|
||||||
|
for line in script.splitlines():
|
||||||
|
if "/var/lib/left4me/global_overlay_cache" not in line:
|
||||||
|
continue
|
||||||
|
assert "rm -rf" in line, line
|
||||||
|
assert "left4me-refresh-global-overlays" not in script
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_provisions_sandbox_user():
|
||||||
|
script = DEPLOY_SCRIPT.read_text()
|
||||||
|
assert "useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox" in script
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_does_not_install_bubblewrap():
|
||||||
|
install_lines = [
|
||||||
|
line for line in DEPLOY_SCRIPT.read_text().splitlines()
|
||||||
|
if ("apt-get install" in line or "dnf install" in line)
|
||||||
|
]
|
||||||
|
assert install_lines, "expected at least one apt/dnf install line"
|
||||||
|
for line in install_lines:
|
||||||
|
assert "bubblewrap" not in line, line
|
||||||
|
assert "bwrap" not in line, line
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_installs_script_overlay_tooling():
|
||||||
|
# Script overlays commonly need 7z and md5sum (e.g. l4d2center map sync).
|
||||||
|
# coreutils ships md5sum and is technically essential, but listing it
|
||||||
|
# explicitly makes the contract obvious and survives slim base images.
|
||||||
|
script = DEPLOY_SCRIPT.read_text().splitlines()
|
||||||
|
apt_lines = [l for l in script if "apt-get install" in l]
|
||||||
|
dnf_lines = [l for l in script if "dnf install" in l]
|
||||||
|
assert apt_lines, "expected an apt-get install line"
|
||||||
|
assert dnf_lines, "expected a dnf install line"
|
||||||
|
for line in apt_lines:
|
||||||
|
assert "p7zip-full" in line, line
|
||||||
|
assert "coreutils" in line, line
|
||||||
|
for line in dnf_lines:
|
||||||
|
# Fedora/RHEL split: p7zip provides 7za, p7zip-plugins provides 7z.
|
||||||
|
assert "p7zip" in line and "p7zip-plugins" in line, line
|
||||||
|
assert "coreutils" in line, line
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_tightens_left4me_db_permissions():
|
||||||
|
script = DEPLOY_SCRIPT.read_text()
|
||||||
|
# The DB and its WAL/SHM sidecars must be left4me:left4me 0640 — owner
|
||||||
|
# (web service) keeps rw, group is read-only, "other" (incl. l4d2-sandbox)
|
||||||
|
# gets nothing. The sidecars matter because SQLite in WAL mode requires
|
||||||
|
# write access to all three; if a sidecar ends up root-owned (e.g. from
|
||||||
|
# ad-hoc root-side inspection), the next write fails as "readonly db".
|
||||||
|
assert "chown left4me:left4me" in script
|
||||||
|
assert "chmod 0640" in script
|
||||||
|
for db_file in (
|
||||||
|
"/var/lib/left4me/left4me.db",
|
||||||
|
"/var/lib/left4me/left4me.db-wal",
|
||||||
|
"/var/lib/left4me/left4me.db-shm",
|
||||||
|
):
|
||||||
|
assert db_file in script, f"deploy script must touch {db_file}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_installs_script_sandbox_helper():
|
||||||
|
script = DEPLOY_SCRIPT.read_text()
|
||||||
|
assert "/usr/local/libexec/left4me/left4me-script-sandbox" in script
|
||||||
|
assert "chmod 0755" in script and "left4me-script-sandbox" in script
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_sandbox_helper_present():
|
||||||
|
assert SCRIPT_SANDBOX_HELPER.is_file()
|
||||||
|
assert SCRIPT_SANDBOX_HELPER.read_text().startswith("#!/bin/bash")
|
||||||
|
mode = SCRIPT_SANDBOX_HELPER.stat().st_mode & 0o777
|
||||||
|
assert mode == 0o755, f"expected 0755, got {oct(mode)}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_sandbox_helper_passes_shell_syntax_check():
|
||||||
|
subprocess.run(["bash", "-n", str(SCRIPT_SANDBOX_HELPER)], check=True)
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_sandbox_helper_invokes_systemd_run_with_hardening():
|
||||||
|
text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||||
|
|
||||||
|
# systemd-run service mode (no --scope), with synchronous I/O to caller.
|
||||||
|
assert "systemd-run" in text
|
||||||
|
assert "--scope" not in text, "v2 uses transient service units, not scopes"
|
||||||
|
assert "--pipe" in text
|
||||||
|
assert "--wait" in text
|
||||||
|
assert "--collect" in text
|
||||||
|
assert "--unit=" in text
|
||||||
|
|
||||||
|
# No bwrap.
|
||||||
|
assert "bwrap" not in text
|
||||||
|
assert "bubblewrap" not in text
|
||||||
|
|
||||||
|
# UID drop via systemd directives.
|
||||||
|
assert "User=l4d2-sandbox" in text
|
||||||
|
assert "Group=l4d2-sandbox" in text
|
||||||
|
|
||||||
|
# Cgroup limits unchanged from v1.
|
||||||
|
assert "MemoryMax=4G" in text
|
||||||
|
assert "MemorySwapMax=0" in text
|
||||||
|
assert "TasksMax=512" in text
|
||||||
|
assert "CPUQuota=200%" in text
|
||||||
|
assert "RuntimeMaxSec=3600" in text
|
||||||
|
|
||||||
|
# Hardening directives that v1 (scope mode) couldn't carry.
|
||||||
|
assert "NoNewPrivileges=yes" in text
|
||||||
|
assert "ProtectSystem=strict" in text
|
||||||
|
assert "ProtectHome=yes" in text
|
||||||
|
assert "PrivateTmp=yes" in text
|
||||||
|
assert "PrivateDevices=yes" in text
|
||||||
|
assert "PrivateIPC=yes" in text
|
||||||
|
assert "ProtectKernelTunables=yes" in text
|
||||||
|
assert "ProtectKernelModules=yes" in text
|
||||||
|
assert "ProtectKernelLogs=yes" in text
|
||||||
|
assert "ProtectControlGroups=yes" in text
|
||||||
|
assert "RestrictNamespaces=yes" in text
|
||||||
|
assert "RestrictSUIDSGID=yes" in text
|
||||||
|
assert "LockPersonality=yes" in text
|
||||||
|
assert "MemoryDenyWriteExecute=yes" in text
|
||||||
|
assert "SystemCallFilter=" in text
|
||||||
|
assert "@system-service" in text
|
||||||
|
assert "@network-io" in text
|
||||||
|
assert "CapabilityBoundingSet=" in text
|
||||||
|
assert "AmbientCapabilities=" in text
|
||||||
|
assert 'RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX"' in text
|
||||||
|
|
||||||
|
# Network namespace stays shared with host.
|
||||||
|
assert "PrivateNetwork=" not in text
|
||||||
|
|
||||||
|
# Mount setup: /etc and /var/lib masked with tmpfs; selective binds back.
|
||||||
|
assert 'TemporaryFileSystem="/etc /var/lib"' in text
|
||||||
|
assert "BindReadOnlyPaths=" in text
|
||||||
|
# The resolv.conf bind points at the sandbox-only file (not the host's
|
||||||
|
# /etc/resolv.conf, which typically references a private-IP DNS server
|
||||||
|
# that IPAddressDeny= blocks).
|
||||||
|
assert "/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf" in text
|
||||||
|
assert "/etc/ssl" in text
|
||||||
|
assert "/etc/ca-certificates" in text
|
||||||
|
assert "/etc/nsswitch.conf" in text
|
||||||
|
assert "/etc/alternatives" in text
|
||||||
|
assert "${SCRIPT}:/script.sh" in text
|
||||||
|
assert 'BindPaths="${OVERLAY_DIR}:/overlay"' in text
|
||||||
|
|
||||||
|
# IP egress filter: allow public, deny localhost / RFC1918 / link-local /
|
||||||
|
# multicast / CGNAT / ULA. systemd's "more specific rule wins" semantics
|
||||||
|
# mean public IPs hit the allow and listed ranges hit the deny.
|
||||||
|
# IPAddressDeny alone — no IPAddressAllow=any. Empirically, having both
|
||||||
|
# set causes the allow to win on this systemd/kernel combo regardless of
|
||||||
|
# the documented "more specific rule wins" behaviour. With only Deny,
|
||||||
|
# the kernel's default "allow all" applies to non-listed addresses.
|
||||||
|
assert "IPAddressDeny=" in text
|
||||||
|
assert "IPAddressAllow=any" not in text
|
||||||
|
# Explicit CIDRs — systemd-run's -p parser doesn't accept the
|
||||||
|
# `localhost` / `link-local` / `multicast` shorthand keywords that
|
||||||
|
# work in unit files (only the full strings parse).
|
||||||
|
for token in (
|
||||||
|
"127.0.0.0/8",
|
||||||
|
"::1/128",
|
||||||
|
"169.254.0.0/16",
|
||||||
|
"fe80::/10",
|
||||||
|
"224.0.0.0/4",
|
||||||
|
"ff00::/8",
|
||||||
|
"10.0.0.0/8",
|
||||||
|
"172.16.0.0/12",
|
||||||
|
"192.168.0.0/16",
|
||||||
|
"100.64.0.0/10",
|
||||||
|
"fc00::/7",
|
||||||
|
):
|
||||||
|
assert token in text, f"missing {token!r} in IPAddressDeny set"
|
||||||
|
|
||||||
|
|
||||||
|
def test_sandbox_resolv_conf_exists():
|
||||||
|
assert SANDBOX_RESOLV_CONF.is_file()
|
||||||
|
text = SANDBOX_RESOLV_CONF.read_text()
|
||||||
|
nameservers = [
|
||||||
|
line.split()[1]
|
||||||
|
for line in text.splitlines()
|
||||||
|
if line.startswith("nameserver ")
|
||||||
|
]
|
||||||
|
assert len(nameservers) >= 2, "expected at least two nameservers for redundancy"
|
||||||
|
# Sanity: the resolvers must be public (not RFC1918 / loopback). We don't
|
||||||
|
# pin the exact IPs — Cloudflare/Google/Quad9 are all acceptable.
|
||||||
|
for ns in nameservers:
|
||||||
|
assert not ns.startswith("127."), ns
|
||||||
|
assert not ns.startswith("10."), ns
|
||||||
|
assert not ns.startswith("192.168."), ns
|
||||||
|
first_octet = int(ns.split(".")[0])
|
||||||
|
# Reject 172.16.0.0/12.
|
||||||
|
if first_octet == 172:
|
||||||
|
second_octet = int(ns.split(".")[1])
|
||||||
|
assert not (16 <= second_octet <= 31), ns
|
||||||
|
|
||||||
|
|
||||||
|
def test_deploy_script_installs_sandbox_resolv_conf():
|
||||||
|
script = DEPLOY_SCRIPT.read_text()
|
||||||
|
assert "deploy/files/etc/left4me/sandbox-resolv.conf" in script
|
||||||
|
assert "/etc/left4me/sandbox-resolv.conf" in script
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_sandbox_helper_validates_overlay_id():
|
||||||
|
text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||||
|
# Numeric-only overlay id
|
||||||
|
assert '[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]]' in text
|
||||||
|
# Overlay dir must exist
|
||||||
|
assert "/var/lib/left4me/overlays/" in text
|
||||||
|
assert "[[ -d $OVERLAY_DIR ]]" in text
|
||||||
|
# Script path must exist
|
||||||
|
assert "[[ -f $SCRIPT ]]" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_sandbox_helper_dry_run_mode(tmp_path):
|
||||||
|
overlay_root = tmp_path / "var/lib/left4me/overlays/42"
|
||||||
|
overlay_root.mkdir(parents=True)
|
||||||
|
fake_script = tmp_path / "fake.sh"
|
||||||
|
fake_script.write_text("echo hi")
|
||||||
|
|
||||||
|
# Run in DRY_RUN mode against a fake l4d2-sandbox UID via a tiny shim that
|
||||||
|
# simulates `id -u l4d2-sandbox` resolving to a valid number.
|
||||||
|
helper_text = SCRIPT_SANDBOX_HELPER.read_text()
|
||||||
|
# We can't actually exec this without root + a real sandbox user; just
|
||||||
|
# verify the dry-run guard short-circuits before systemd-run / bwrap.
|
||||||
|
assert 'LEFT4ME_SCRIPT_SANDBOX_DRY_RUN' in helper_text
|
||||||
|
assert 'exit 0' in helper_text
|
||||||
|
|
|
||||||
2389
docs/superpowers/plans/2026-05-07-l4d2-global-map-overlays.md
Normal file
2389
docs/superpowers/plans/2026-05-07-l4d2-global-map-overlays.md
Normal file
File diff suppressed because it is too large
Load diff
557
docs/superpowers/plans/2026-05-07-l4d2-workshop-overlays.md
Normal file
557
docs/superpowers/plans/2026-05-07-l4d2-workshop-overlays.md
Normal file
|
|
@ -0,0 +1,557 @@
|
||||||
|
# L4D2 Workshop Overlays Implementation Plan
|
||||||
|
|
||||||
|
> **Approval gate:** This plan may be written and refined without further approval. Do not implement code changes from this plan until the user explicitly approves implementation.
|
||||||
|
|
||||||
|
**Goal:** Implement the workshop overlay feature per `docs/superpowers/specs/2026-05-07-l4d2-workshop-overlays-design.md`. Add a `WorkshopItem` registry, a typed `Overlay.type` column with a builder registry, a workshop builder that downloads from the Steam Web API and manages symlinks into a deduplicated cache, and the supporting routes, templates, jobs, and tests.
|
||||||
|
|
||||||
|
**Architecture:** Keep the v1 single-process Flask architecture. New code is additive: a `WorkshopBuilder` class registered in a builder dispatcher, a `steam_workshop` service module for the Steam Web API and downloader, two new database tables and one extended one, and two new job operations on the existing in-process worker. fuse-overlayfs mount handling in `l4d2host` is unchanged — workshop content arrives at overlay paths the same way externals do today.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
See `docs/superpowers/specs/2026-05-07-l4d2-workshop-overlays-design.md` for the design rationale. Implementation-relevant decisions:
|
||||||
|
|
||||||
|
- Typed overlays: `external` (existing rows; no-op builder) and `workshop` (new); future types deferred.
|
||||||
|
- No JSON `source_config` blob; per-type structured data in proper tables.
|
||||||
|
- `WorkshopItem` is a global deduplicated registry keyed on `steam_id`. Cache at `/var/lib/left4me/workshop_cache/{steam_id}.vpk`.
|
||||||
|
- Overlay symlinks are absolute, named `{steam_id}.vpk`; no Steam filename in any on-disk path.
|
||||||
|
- `overlay_workshop_items` is a pure association; toggle = remove/re-add.
|
||||||
|
- Collections are atomic UI bulk-imports; DB never tracks collection attribution.
|
||||||
|
- Single global admin "Refresh all workshop items" button.
|
||||||
|
- No cache GC in v1.
|
||||||
|
- `Overlay.user_id` is the scope (NULL = system, set = private); independent of `type`.
|
||||||
|
- Workshop overlays default to private; existing externals stay system-wide.
|
||||||
|
- One unified Create-overlay button with type radio; no path field — paths are always `str(overlay_id)`.
|
||||||
|
- `consumer_app_id == 550` validated at fetch/add; not stored.
|
||||||
|
- Input field accepts numeric ID, full Workshop URL, or multi-line batch.
|
||||||
|
- Auto-rebuild after add/remove with build coalescing.
|
||||||
|
- HTTPS for all Steam Web API calls.
|
||||||
|
- `Overlay.id` uses `AUTOINCREMENT`; `create_overlay_directory` uses `exist_ok=False`.
|
||||||
|
- Two partial unique indexes for overlay names: `(name) WHERE user_id IS NULL` and `(name, user_id) WHERE user_id IS NOT NULL`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Gap
|
||||||
|
|
||||||
|
- `Overlay` rows have `id`, `name`, `path`, no type, no scope.
|
||||||
|
- The web app cannot download anything from Steam; users must SFTP `.vpk` files into prepared overlay directories.
|
||||||
|
- The job worker has no operations for overlay builds or workshop refreshes.
|
||||||
|
- The mount/build pipeline assumes overlay directories are externally populated.
|
||||||
|
- There is no UI affordance to add or list workshop content.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: Extend Tests First — Schema Migration And Models
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `l4d2web/tests/test_workshop_overlay_models.py`
|
||||||
|
- Modify: `l4d2web/tests/test_models.py` (extend) — partial unique index behavior
|
||||||
|
|
||||||
|
Write tests against fresh SQLite schemas asserting:
|
||||||
|
|
||||||
|
- An `Overlay` migration round-trip: existing rows acquire `type='external'` and `user_id=NULL`; their `name` values remain unique by partial index.
|
||||||
|
- After migration, two externals (both `user_id=NULL`) with the same name are rejected by the system partial unique index.
|
||||||
|
- After migration, two users may both own a workshop overlay named `"my-maps"` (per-user partial unique index).
|
||||||
|
- `WorkshopItem.steam_id` is unique; concurrent inserts of the same `steam_id` raise integrity errors.
|
||||||
|
- `overlay_workshop_items` enforces `UNIQUE(overlay_id, workshop_item_id)`.
|
||||||
|
- `Overlay` deletion cascades `overlay_workshop_items` rows but does not delete `WorkshopItem` rows (`ON DELETE RESTRICT`).
|
||||||
|
- `Job.overlay_id` is nullable and references `overlays(id)`.
|
||||||
|
- `Overlay.id` does not reuse a deleted ID after the migration (AUTOINCREMENT).
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_workshop_overlay_models.py l4d2web/tests/test_models.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected before implementation: FAIL.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: Schema Migration And ORM Mappings
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `l4d2web/alembic/versions/0002_workshop_overlays.py`
|
||||||
|
- Modify: `l4d2web/models.py`
|
||||||
|
|
||||||
|
Migration `0002_workshop_overlays` (`down_revision = "b2c684fddbd3"`):
|
||||||
|
|
||||||
|
1. `op.batch_alter_table("overlays")`:
|
||||||
|
- Add `type VARCHAR(16) NOT NULL DEFAULT 'external'` (server_default during migration; remove after backfill).
|
||||||
|
- Add `user_id INTEGER NULL REFERENCES users(id)`.
|
||||||
|
- Drop the existing `unique=True` on `name`.
|
||||||
|
- Add index `ix_overlays_type_user_id` on `(type, user_id)`.
|
||||||
|
- Switch `id` to `AUTOINCREMENT`.
|
||||||
|
2. After batch alter, create the two partial unique indexes via raw `op.create_index(..., postgresql_where=..., sqlite_where=...)`:
|
||||||
|
- `uq_overlay_name_system` on `(name)` `WHERE user_id IS NULL`.
|
||||||
|
- `uq_overlay_name_per_user` on `(name, user_id)` `WHERE user_id IS NOT NULL`.
|
||||||
|
3. `op.create_table("workshop_items", ...)` per spec data-model section.
|
||||||
|
4. `op.create_table("overlay_workshop_items", ...)` with the unique constraint and the reverse-lookup index.
|
||||||
|
5. `op.batch_alter_table("jobs")`: add `overlay_id INTEGER NULL REFERENCES overlays(id)`.
|
||||||
|
|
||||||
|
ORM (`models.py`):
|
||||||
|
|
||||||
|
- Extend `Overlay`: add `type`, `user_id`. Drop `unique=True` on `name`. Set `__table_args__` with the two partial indexes and `ix_overlays_type_user_id`.
|
||||||
|
- Extend `Job`: add `overlay_id` mapped column with FK.
|
||||||
|
- New `WorkshopItem` and `OverlayWorkshopItem` classes per spec. Set up `Overlay.workshop_items` relationship through the association.
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_workshop_overlay_models.py l4d2web/tests/test_models.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected after implementation: PASS.
|
||||||
|
|
||||||
|
Run alembic against a fresh test DB to verify upgrade and downgrade succeed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: Tests First — Steam Web API And Downloader
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `l4d2web/tests/test_steam_workshop.py`
|
||||||
|
|
||||||
|
Mock HTTP with `responses` or `pytest-httpserver`. Cover:
|
||||||
|
|
||||||
|
- `parse_workshop_input` accepts a single numeric ID, a single Workshop URL (`steamcommunity.com/sharedfiles/filedetails/?id=N`), and a multi-line whitespace-separated batch of either; returns deduplicated ordered list of digit-only IDs.
|
||||||
|
- `parse_workshop_input` rejects garbage, paths outside `?id=`, non-digit IDs.
|
||||||
|
- `resolve_collection` POSTs to the HTTPS endpoint with the form-encoded payload and returns `publishedfileid` children.
|
||||||
|
- `fetch_metadata_batch` POSTs once with `itemcount=N`; returns parsed `WorkshopMetadata` per item; captures `result != 1` into `last_error`; raises `WorkshopValidationError` when any `consumer_app_id != 550` during user-add; logs and skips during refresh-mode.
|
||||||
|
- `WorkshopMetadata.preview_url` is captured.
|
||||||
|
- `download_to_cache` writes `cache_root/{steam_id}.vpk.partial`, then `os.replace` to the final name; sets `os.utime(file, (time_updated, time_updated))`.
|
||||||
|
- `download_to_cache` is idempotent: a second call where on-disk `(mtime, size)` matches `(time_updated, file_size)` is a no-op (no HTTP request issued).
|
||||||
|
- `refresh_all` runs downloads via `ThreadPoolExecutor(max_workers=8)` and reports per-item errors without aborting the batch.
|
||||||
|
- All Steam API URLs use `https://`.
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_steam_workshop.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected before implementation: FAIL.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 4: Steam Workshop Service Module
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `l4d2web/services/steam_workshop.py`
|
||||||
|
|
||||||
|
Public surface:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def parse_workshop_input(raw: str) -> list[str]: ...
|
||||||
|
def resolve_collection(collection_id: str) -> list[str]: ...
|
||||||
|
def fetch_metadata_batch(steam_ids: list[str], *, mode: Literal["add","refresh"]) -> list[WorkshopMetadata]: ...
|
||||||
|
def download_to_cache(meta: WorkshopMetadata, cache_root: Path, *, on_progress=None, should_cancel=None) -> Path: ...
|
||||||
|
def refresh_all(items: list[WorkshopItem], cache_root: Path, executor_workers: int = 8) -> RefreshReport: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
Implementation rules:
|
||||||
|
|
||||||
|
- Endpoints are HTTPS:
|
||||||
|
- `https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/`
|
||||||
|
- `https://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1/`
|
||||||
|
- Form-encoded POSTs with `itemcount=N` / `collectioncount=N` and `publishedfileids[i]=…` per index.
|
||||||
|
- Per-request timeout 30s; per-item ceiling 5min. No retry or backoff in v1.
|
||||||
|
- `consumer_app_id != 550`:
|
||||||
|
- In `mode="add"`: raise `WorkshopValidationError` with the offending `steam_id`.
|
||||||
|
- In `mode="refresh"`: log and skip; do not abort other items.
|
||||||
|
- `result != 1`: capture Steam's result code in the item's `last_error`; do not download; do not abort siblings.
|
||||||
|
- Cooperative cancellation: `download_to_cache` checks `should_cancel()` between chunked reads; `refresh_all`'s executor checks before each task.
|
||||||
|
- `WorkshopMetadata` is a dataclass with `steam_id, title, filename, file_url, file_size, time_updated, preview_url, consumer_app_id, result`.
|
||||||
|
- `RefreshReport` aggregates per-item outcomes for the caller's job log.
|
||||||
|
- Use a single `requests.Session` per call site for connection reuse.
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_steam_workshop.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected after implementation: PASS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 5: Tests First — Path Helpers And Overlay Creation
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `l4d2web/tests/test_workshop_paths.py`
|
||||||
|
- Create: `l4d2web/tests/test_overlay_creation.py`
|
||||||
|
|
||||||
|
Cover:
|
||||||
|
|
||||||
|
- `workshop_cache_root()` returns `LEFT4ME_ROOT/workshop_cache`.
|
||||||
|
- `cache_path(steam_id)` returns `cache_root / f"{steam_id}.vpk"` for valid digit strings; rejects non-digits, slashes, dot-dot.
|
||||||
|
- `generate_overlay_path(overlay_id)` returns `str(overlay_id)`; passes `validate_overlay_ref` from `l4d2host.paths`.
|
||||||
|
- `create_overlay_directory(overlay)` creates `LEFT4ME_ROOT/overlays/{path}/` with `exist_ok=False`. Calling twice raises (DB/disk drift surfaced loudly).
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_workshop_paths.py l4d2web/tests/test_overlay_creation.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected before implementation: FAIL.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 6: Path Helpers And Overlay Creation
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `l4d2web/services/workshop_paths.py`
|
||||||
|
- Create: `l4d2web/services/overlay_creation.py`
|
||||||
|
|
||||||
|
`workshop_paths`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def workshop_cache_root() -> Path: ... # LEFT4ME_ROOT/workshop_cache
|
||||||
|
def cache_path(steam_id: str) -> Path: ... # validates digits-only; returns cache_root/{steam_id}.vpk
|
||||||
|
```
|
||||||
|
|
||||||
|
`overlay_creation`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def generate_overlay_path(overlay_id: int) -> str: ... # str(overlay_id) + validate_overlay_ref
|
||||||
|
def create_overlay_directory(overlay: Overlay) -> None: # makedirs(..., exist_ok=False)
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_workshop_paths.py l4d2web/tests/test_overlay_creation.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected after implementation: PASS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 7: Tests First — Overlay Builders
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `l4d2web/tests/test_overlay_builders.py`
|
||||||
|
|
||||||
|
Cover with `tmp_path`:
|
||||||
|
|
||||||
|
- `BUILDERS` dict resolves `"external"` and `"workshop"` to instances; unknown types raise `KeyError` (caller's error).
|
||||||
|
- `ExternalBuilder.build()` is a no-op: makes the overlay directory if missing, writes one log line, returns. Existing files in the directory are untouched.
|
||||||
|
- `WorkshopBuilder.build()` against a fixture overlay with three associated `WorkshopItem` rows (two with cache files present, one without):
|
||||||
|
- Creates `left4dead2/addons/` if missing.
|
||||||
|
- Creates symlinks `addons/{steam_id_a}.vpk → cache_root/{steam_id_a}.vpk` for items with cache files. Symlinks are absolute.
|
||||||
|
- Skips the uncached item; emits a warning log line. Does not create a dangling symlink.
|
||||||
|
- On a re-run with the same associations: no FS changes; logs report `unchanged=2 skipped(uncached)=1`.
|
||||||
|
- On a re-run after one association is removed: removes the obsolete symlink only; leaves cache files alone.
|
||||||
|
- On a re-run after one item is added: adds only the new symlink.
|
||||||
|
- Files in `addons/` that aren't symlinks into the cache are left untouched.
|
||||||
|
- `should_cancel` mid-build: stops between filesystem ops; partial state is consistent and a re-run heals.
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_overlay_builders.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected before implementation: FAIL.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 8: Overlay Builders And Dispatcher
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `l4d2web/services/overlay_builders.py`
|
||||||
|
|
||||||
|
```python
|
||||||
|
class OverlayBuilder(Protocol):
|
||||||
|
def build(self, overlay: Overlay, *, on_stdout, on_stderr, should_cancel) -> None: ...
|
||||||
|
|
||||||
|
class ExternalBuilder: ...
|
||||||
|
class WorkshopBuilder: ...
|
||||||
|
|
||||||
|
BUILDERS: dict[str, OverlayBuilder] = {
|
||||||
|
"external": ExternalBuilder(),
|
||||||
|
"workshop": WorkshopBuilder(),
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`WorkshopBuilder.build()`:
|
||||||
|
|
||||||
|
1. Load the overlay's `WorkshopItem` rows.
|
||||||
|
2. `os.makedirs(overlay_root / "left4dead2/addons", exist_ok=True)`.
|
||||||
|
3. Compute `desired = {f"{steam_id}.vpk": cache_path(steam_id)}` for items where `last_downloaded_at IS NOT NULL` and the cache file exists. Skip and warn for items missing a cache file.
|
||||||
|
4. Inspect existing entries in `addons/` via `os.scandir`: keep entries that are not symlinks into `workshop_cache`; otherwise diff against `desired` and apply changes via `os.unlink` and `os.symlink(absolute_target, link_path)`.
|
||||||
|
5. Emit `created N, removed M, unchanged K, skipped (uncached) S` log line.
|
||||||
|
6. Check `should_cancel()` between filesystem ops.
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_overlay_builders.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected after implementation: PASS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 9: Tests First — Worker Scheduler Truth Table And Coalescing
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `l4d2web/tests/test_job_worker.py`
|
||||||
|
|
||||||
|
Add coverage:
|
||||||
|
|
||||||
|
- Truth table for `can_start`:
|
||||||
|
- `install` not claimed while `refresh_workshop_items`, any `build_overlay`, or any server job is running.
|
||||||
|
- `refresh_workshop_items` not claimed while `install`, any `build_overlay`, or any server job is running.
|
||||||
|
- `build_overlay(N)` not claimed while `install`, `refresh_workshop_items`, or another `build_overlay(N)` is running. Two `build_overlay` jobs for **different** overlay IDs claim concurrently.
|
||||||
|
- Server start/init blocks if `refresh_workshop_items` runs or if any `build_overlay(N)` runs where N ∈ overlays of the server's blueprint.
|
||||||
|
- `enqueue_build_overlay(overlay_id)`:
|
||||||
|
- Inserts a new queued job when no pending job exists.
|
||||||
|
- Returns the existing pending job when one is already queued (coalescing).
|
||||||
|
- Does not coalesce against running jobs (a new add after build start gets a fresh queued job).
|
||||||
|
- `refresh_workshop_items` post-completion enqueues `build_overlay` only for overlays whose items had `time_updated` advance or `filename` change; each such enqueue uses the coalescing helper.
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_job_worker.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected before implementation: FAIL.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 10: Worker Scheduler And New Operations
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `l4d2web/services/job_worker.py`
|
||||||
|
|
||||||
|
Changes:
|
||||||
|
|
||||||
|
- Define `OVERLAY_OPERATIONS = {"build_overlay"}` and `GLOBAL_OPERATIONS = {"install", "refresh_workshop_items"}`. Update `malformed_server_job` to allow `server_id IS NULL` for these.
|
||||||
|
- Extend `SchedulerState` with `running_overlays: set[int]` and `refresh_running: bool`.
|
||||||
|
- Update `claim_next_job()`:
|
||||||
|
- Compute `running_overlays` from queries against `running` jobs of operation `build_overlay`.
|
||||||
|
- Apply the truth-table rules above.
|
||||||
|
- Continue using `created_at, id` ordering for deterministic claim.
|
||||||
|
- Add `enqueue_build_overlay(overlay_id: int) -> Job` helper:
|
||||||
|
- Look for `queued` `build_overlay` job with same `overlay_id`. Return it if present.
|
||||||
|
- Otherwise insert a new queued job with `overlay_id` set, `server_id=None`, `operation="build_overlay"`.
|
||||||
|
- Update `run_job` dispatch:
|
||||||
|
- `build_overlay` → load `Overlay`, dispatch to `BUILDERS[overlay.type].build(overlay, on_stdout, on_stderr, should_cancel)`.
|
||||||
|
- `refresh_workshop_items` → call `steam_workshop.refresh_all(...)`. After completion, for each affected overlay, call `enqueue_build_overlay(overlay_id)`.
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_job_worker.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected after implementation: PASS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 11: Tests First — Routes, Permissions, And Auto-Rebuild
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `l4d2web/tests/test_overlays.py`
|
||||||
|
- Create: `l4d2web/tests/test_workshop_routes.py`
|
||||||
|
|
||||||
|
Cover:
|
||||||
|
|
||||||
|
- `POST /overlays` with `type='workshop'` and `name` succeeds for any logged-in user; `path` is auto-generated; `user_id` is set; the directory exists at `LEFT4ME_ROOT/overlays/{id}`.
|
||||||
|
- `POST /overlays` with `type='external'` succeeds only for admins; `user_id` is NULL.
|
||||||
|
- Duplicate workshop name within the same user is rejected; duplicate names across users are accepted.
|
||||||
|
- Duplicate external name is rejected.
|
||||||
|
- Non-admins see `type='external' OR user_id=current_user.id` only when listing overlays.
|
||||||
|
- `POST /overlays/{id}/items` with one numeric ID adds an association and enqueues a coalesced `build_overlay`. The response is an HTMX fragment of the updated item table.
|
||||||
|
- `POST /overlays/{id}/items` with a multi-line batch (mix of IDs and URLs) adds all and enqueues one coalesced job for the batch.
|
||||||
|
- `POST /overlays/{id}/items` with a collection ID resolves members and adds N associations.
|
||||||
|
- Adding a non-L4D2 item (`consumer_app_id != 550`) returns HTTP 400 with a useful message; no association is created.
|
||||||
|
- Adding an item already in the overlay returns "already in overlay" (no 500).
|
||||||
|
- `POST /overlays/{id}/items/{item_id}/delete` removes the association and enqueues a coalesced build.
|
||||||
|
- `POST /overlays/{id}/build` enqueues the manual rebuild and redirects to the job page.
|
||||||
|
- `POST /admin/workshop/refresh` is admin-only; non-admins receive 403.
|
||||||
|
|
||||||
|
Mock `steam_workshop` HTTP layer for these tests.
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_overlays.py l4d2web/tests/test_workshop_routes.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected before implementation: FAIL.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 12: Routes And Templates
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `l4d2web/routes/overlay_routes.py`
|
||||||
|
- Create: `l4d2web/routes/workshop_routes.py`
|
||||||
|
- Modify: `l4d2web/routes/page_routes.py`
|
||||||
|
- Modify: `l4d2web/templates/overlays.html`
|
||||||
|
- Modify: `l4d2web/templates/overlay_detail.html`
|
||||||
|
- Create: `l4d2web/templates/_overlay_item_table.html`
|
||||||
|
- Modify: `l4d2web/templates/admin.html`
|
||||||
|
- Modify: `l4d2web/app.py` (register the workshop blueprint)
|
||||||
|
|
||||||
|
`overlay_routes.py`:
|
||||||
|
|
||||||
|
- `create_overlay`: read `type` and `name` from form. No `path` field accepted.
|
||||||
|
- `type='external'`: admin-only; `user_id=NULL`. After insert, set `path = generate_overlay_path(id)`; call `create_overlay_directory(overlay)`.
|
||||||
|
- `type='workshop'`: any logged-in user; `user_id=current_user.id`. After insert, set `path = generate_overlay_path(id)`; call `create_overlay_directory(overlay)`.
|
||||||
|
- `update_overlay`: forbid changing `type` and `path`. Workshop: owner or admin can edit `name`. External: admin-only `name` edits.
|
||||||
|
- `delete_overlay`: after the row deletes, `shutil.rmtree(LEFT4ME_ROOT/overlays/{path})` only if `overlay.path == str(overlay.id)` (legacy externals are left alone). Cache untouched.
|
||||||
|
|
||||||
|
`workshop_routes.py`:
|
||||||
|
|
||||||
|
- `POST /overlays/{id}/items`: parse input via `parse_workshop_input`; if a collection ID, resolve members; batch-fetch metadata in `mode="add"`; reject non-550 with HTTP 400; upsert `WorkshopItem` via SQLite `INSERT ... ON CONFLICT DO UPDATE` on `steam_id`; bulk-add associations catching `(overlay_id, workshop_item_id)` unique violations; call `enqueue_build_overlay(overlay_id)`; return rendered `_overlay_item_table.html` fragment.
|
||||||
|
- `POST /overlays/{id}/items/{item_id}/delete`: ownership check; remove association; call `enqueue_build_overlay(overlay_id)`; return updated fragment.
|
||||||
|
- `POST /overlays/{id}/build`: ownership check; enqueue (coalesced); redirect to `/jobs/{job_id}`.
|
||||||
|
- `POST /admin/workshop/refresh`: `@require_admin`; insert a `refresh_workshop_items` queued job; redirect to `/admin/jobs`.
|
||||||
|
|
||||||
|
`page_routes.py`:
|
||||||
|
|
||||||
|
- `overlays()`: admins see all; non-admins see `type='external' OR user_id=current_user.id`.
|
||||||
|
- `overlay_detail()`: load `WorkshopItem` rows for workshop-type overlays.
|
||||||
|
|
||||||
|
Templates:
|
||||||
|
|
||||||
|
- `overlays.html`: add Type column. Modal has type radio (External | Workshop) and name field. No path field.
|
||||||
|
- `overlay_detail.html`: branch on `overlay.type`.
|
||||||
|
- External view: read-only path display, name edit (admin only).
|
||||||
|
- Workshop view: an `<textarea>` accepting one or many IDs/URLs plus a radio (Items | Collection); item table with thumbnail (`preview_url`), `steam_id` linked to Steam, title, filename, time_updated, file_size, last_error, Remove; Rebuild button; small status indicator showing the latest related job.
|
||||||
|
- `_overlay_item_table.html`: renderable standalone for HTMX swaps.
|
||||||
|
- `admin.html`: add a CSRF-protected "Refresh all workshop items" button.
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_overlays.py l4d2web/tests/test_workshop_routes.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected after implementation: PASS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 13: Tests First — Initialize-Time Guard
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `l4d2web/tests/test_l4d2_facade.py` (or create if missing)
|
||||||
|
|
||||||
|
Cover:
|
||||||
|
|
||||||
|
- `initialize_server(server_id)` calls `BUILDERS[overlay.type].build()` for each overlay in the blueprint before writing the spec.
|
||||||
|
- For workshop overlays, when an associated `WorkshopItem` lacks a cache file (`workshop_cache/{steam_id}.vpk` missing), `initialize_server` raises a clear error containing the missing `steam_id`s and the overlay name; the spec is not written; `l4d2ctl initialize` is not invoked.
|
||||||
|
- For workshop overlays where all items have cache files, the symlinks are present and `l4d2ctl initialize` runs.
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_l4d2_facade.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected before implementation: FAIL.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 14: Initialize-Time Guard
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `l4d2web/services/l4d2_facade.py`
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Before writing the temp spec, iterate over the blueprint's overlays and call `BUILDERS[overlay.type].build(...)`.
|
||||||
|
- For workshop overlays, the builder logs and skips uncached items rather than failing. After all builders run, perform a second pass: query the blueprint's workshop overlays for any associated `WorkshopItem` with no cache file. If any are found, raise an exception whose message names the missing `steam_id`s and points at the overlay page (`Open overlay {name} ({id}) and click Build`).
|
||||||
|
|
||||||
|
Verification command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_l4d2_facade.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected after implementation: PASS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 15: Deploy Provisioning
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `deploy/install.sh` (or whichever provisioning script creates `/var/lib/left4me/`)
|
||||||
|
- Modify: `deploy/README.md`
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- Provisioning creates `/var/lib/left4me/workshop_cache/` (mode 0755), owned by the web user.
|
||||||
|
- `deploy/README.md` documents:
|
||||||
|
- The new directory and its purpose.
|
||||||
|
- Permission requirement: web user owns; host user reads (shared group with `g+r` if uids differ).
|
||||||
|
- `LEFT4ME_ROOT` layout updated with the new subtree.
|
||||||
|
|
||||||
|
No tests; verify via test deploy.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 16: Full Verification And Manual Test Plan
|
||||||
|
|
||||||
|
Run focused suites first:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests/test_workshop_overlay_models.py -q
|
||||||
|
pytest l4d2web/tests/test_models.py -q
|
||||||
|
pytest l4d2web/tests/test_steam_workshop.py -q
|
||||||
|
pytest l4d2web/tests/test_workshop_paths.py l4d2web/tests/test_overlay_creation.py -q
|
||||||
|
pytest l4d2web/tests/test_overlay_builders.py -q
|
||||||
|
pytest l4d2web/tests/test_job_worker.py -q
|
||||||
|
pytest l4d2web/tests/test_overlays.py l4d2web/tests/test_workshop_routes.py -q
|
||||||
|
pytest l4d2web/tests/test_l4d2_facade.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run the full web suite:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest l4d2web/tests -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Manual test plan on the test deploy:
|
||||||
|
|
||||||
|
1. Apply migration on a copy of the prod DB; verify all existing overlays read as `type='external'`, `user_id=NULL`; names still unique by partial index; two externals with the same name are rejected.
|
||||||
|
2. As non-admin, create a workshop overlay. Add a known popular L4D2 addon by URL. Verify the build job auto-enqueues. Verify symlink + cache file. Confirm web UI shows metadata and thumbnail.
|
||||||
|
3. Paste a multi-line block of item IDs and URLs. Verify all are parsed and added; verify coalescing (only one `build_overlay` job runs).
|
||||||
|
4. Add a 50-item collection. Verify all 50 metadata rows appear and no UI mention of "from collection". Verify single coalesced build job.
|
||||||
|
5. Remove an item. Verify auto-rebuild removes the symlink while the cache file remains.
|
||||||
|
6. As admin, click Refresh All. Verify only items with newer `time_updated` re-download. Verify affected overlays get coalesced `build_overlay` jobs enqueued.
|
||||||
|
7. Boot an L4D2 server with a workshop overlay attached. Connect locally and confirm the maps appear in the map vote and load.
|
||||||
|
8. Concurrency probe: enqueue Refresh All while a `build_overlay` is queued; verify scheduler waits per truth table.
|
||||||
|
9. Initialize-time guard: manually delete a cache file for an item that's in an overlay attached to a server's blueprint. Try to start the server; verify clear error mentioning the missing `steam_id`.
|
||||||
|
10. Negative: paste a non-L4D2 workshop ID (e.g., a Skyrim mod). Expect HTTP 400 with a clear message; no row inserted.
|
||||||
|
11. Negative: simulate Steam API down (block egress). Verify add fails with clean error, not 500. Verify refresh job logs the failure.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Commit Strategy
|
||||||
|
|
||||||
|
Use small commits after passing relevant tests:
|
||||||
|
|
||||||
|
1. `feat(l4d2-web): typed overlays + workshop schema migration`
|
||||||
|
2. `feat(l4d2-web): steam workshop API client and downloader`
|
||||||
|
3. `feat(l4d2-web): overlay path helpers and creation`
|
||||||
|
4. `feat(l4d2-web): overlay builder registry with workshop builder`
|
||||||
|
5. `feat(l4d2-web): worker support for build_overlay and refresh_workshop_items`
|
||||||
|
6. `feat(l4d2-web): workshop overlay UI (routes + templates)`
|
||||||
|
7. `feat(l4d2-web): initialize-time guard for uncached workshop items`
|
||||||
|
8. `feat(deploy): workshop_cache provisioning`
|
||||||
|
|
||||||
|
Do not commit unless the user explicitly asks for commits.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open Approval Gate
|
||||||
|
|
||||||
|
Before modifying implementation files, ask the user for explicit approval to proceed with the workshop-overlays implementation.
|
||||||
229
docs/superpowers/plans/2026-05-08-kernel-overlayfs-helper.md
Normal file
229
docs/superpowers/plans/2026-05-08-kernel-overlayfs-helper.md
Normal file
|
|
@ -0,0 +1,229 @@
|
||||||
|
# Kernel Overlayfs Helper Implementation Plan
|
||||||
|
|
||||||
|
> **Approval status:** User-approved 2026-05-08. Implementation proceeds.
|
||||||
|
|
||||||
|
**Goal:** Implement the kernel-overlayfs migration per `docs/superpowers/specs/2026-05-08-kernel-overlayfs-helper-design.md`. Add a Python `left4me-overlay` privileged helper, a `KernelOverlayFSMounter` Python class, wire the existing `OverlayMounter` ABC through `l4d2host/instances.py`, drop `fuse-overlayfs` from the deploy stack, and migrate existing on-disk upper/work directories.
|
||||||
|
|
||||||
|
**Architecture:** The web app continues to call `l4d2ctl start|stop|delete <name>`; `l4d2host` continues to expose the same CLI verbs. Internally, `start_instance`/`stop_instance`/`delete_instance` move from a hardcoded subprocess call to `fuse-overlayfs`/`fusermount3` to using `KernelOverlayFSMounter`, which invokes the new sudo helper that mounts in PID 1's namespace via `nsenter`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
See `docs/superpowers/specs/2026-05-08-kernel-overlayfs-helper-design.md` for the design rationale. Implementation-relevant summary:
|
||||||
|
|
||||||
|
- `left4me-overlay` Python helper in `/usr/local/libexec/left4me/`, owned root, mode 0755, system `/usr/bin/python3`, stdlib only.
|
||||||
|
- Verbs: `mount <name>`, `umount <name>`.
|
||||||
|
- Validation in helper: name regex; realpath + allowlist for each lowerdir; exact-prefix check for upper/work/merged; reject upperdir with `user.fuseoverlayfs.*` xattrs; lowerdir count ≤ 500.
|
||||||
|
- Sudoers verb-constrained: `mount *`, `umount *`.
|
||||||
|
- `KernelOverlayFSMounter` in `l4d2host/fs/kernel_overlayfs.py` — implements `OverlayMounter`. Derives `name` from the merged path's parent.
|
||||||
|
- `start_instance` adds `os.path.ismount(merged)` guard before mounting.
|
||||||
|
- Deploy migration: gated on sentinel file `/var/lib/left4me/.kernel-overlay-migrated`; stops gameservers + web, force-unmounts stale mounts, wipes upper/work, recreates empty.
|
||||||
|
- Web unit cleanup: drop `MountFlags=shared`, restore `PrivateTmp=true`, rewrite comment block. Keep `NoNewPrivileges` unset.
|
||||||
|
- Delete `l4d2host/fs/fuse_overlayfs.py` (currently unused — `start_instance` bypasses it).
|
||||||
|
- AGENTS.md contracts unchanged.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Gap
|
||||||
|
|
||||||
|
- `l4d2host/instances.py` `start_instance` calls `fuse-overlayfs` directly (lines 85-101); `stop_instance`/`delete_instance` call `fusermount3 -u` directly. The `OverlayMounter` ABC at `l4d2host/fs/base.py` and the `FuseOverlayFSMounter` impl at `l4d2host/fs/fuse_overlayfs.py` exist but are unused.
|
||||||
|
- Mounts land in the web service's private mount namespace, invisible to host and to gameserver units. `MountFlags=shared` does not fix it.
|
||||||
|
- No privileged mount helper exists; only `left4me-systemctl` and `left4me-journalctl`.
|
||||||
|
- Deploy script installs `fuse-overlayfs` apt package and assumes it as a runtime tool.
|
||||||
|
- Existing `runtime/<name>/upper` directories may carry `user.fuseoverlayfs.*` xattrs that kernel overlayfs would silently ignore (resurrecting "deleted" files).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: Helper Script + Sudoers + Mounter Class (RED-first)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `deploy/files/usr/local/libexec/left4me/left4me-overlay` (Python, mode 0755 after deploy)
|
||||||
|
- Modify: `deploy/files/etc/sudoers.d/left4me`
|
||||||
|
- Create: `l4d2host/fs/kernel_overlayfs.py`
|
||||||
|
- Create: `l4d2host/tests/test_kernel_overlayfs.py`
|
||||||
|
- Create: `l4d2host/tests/test_overlay_helper.py`
|
||||||
|
- Modify: `deploy/tests/test_deploy_artifacts.py` (assert helper deployed + sudoers entry)
|
||||||
|
|
||||||
|
Test plan (RED first):
|
||||||
|
|
||||||
|
1. `test_kernel_overlayfs.py::test_mount_invokes_helper_with_name` — mock `run_command`, call `KernelOverlayFSMounter().mount(lowerdirs="/x:/y", upperdir=Path("/var/lib/left4me/runtime/alpha/upper"), workdir=Path("/var/lib/left4me/runtime/alpha/work"), merged=Path("/var/lib/left4me/runtime/alpha/merged"))`, assert argv `["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", "mount", "alpha"]`.
|
||||||
|
2. `test_kernel_overlayfs.py::test_unmount_invokes_helper_with_umount_verb` — mock + call + assert argv with `umount`.
|
||||||
|
3. `test_overlay_helper.py` — drives the helper script as a subprocess with `LEFT4ME_OVERLAY_PRINT_ONLY=1` env var (helper prints the would-be `nsenter …` command line and exits 0 instead of execve), and with isolated `LEFT4ME_ROOT=tmp_path`. Cases:
|
||||||
|
- Valid mount: prints expected `nsenter --mount=/proc/1/ns/mnt -- /bin/mount -t overlay …` line.
|
||||||
|
- Valid umount: prints expected umount line.
|
||||||
|
- Bad name (`../escape`, uppercase, empty): exit non-zero, stderr matches.
|
||||||
|
- Lowerdir traversal (`/etc`, `/var/lib/left4me/../etc`, symlink escape): exit non-zero.
|
||||||
|
- Missing `instance.env`: exit non-zero.
|
||||||
|
- Tainted upperdir (with `user.fuseoverlayfs.opaque` xattr): exit non-zero with clear message. (Optional: skip if `setfattr` is unavailable on dev machine; keep test on Linux only via `pytest.mark.skipif`.)
|
||||||
|
- Lowerdir count > 500: exit non-zero.
|
||||||
|
4. `test_deploy_artifacts.py` — assert `/usr/local/libexec/left4me/left4me-overlay` is present in deployed files; sudoers includes the new lines.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Helper script structure: `argparse` for the verb, then path-validation funcs, then `os.execv("/usr/bin/nsenter", [...])` (or printing it under `LEFT4ME_OVERLAY_PRINT_ONLY`).
|
||||||
|
- `KernelOverlayFSMounter`: `name = merged.parent.name` (with a one-line comment), then `run_command(["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", verb, name], on_stdout=…, on_stderr=…, passthrough=…, should_cancel=…)`.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest l4d2host/tests/test_kernel_overlayfs.py l4d2host/tests/test_overlay_helper.py deploy/tests/test_deploy_artifacts.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected before implementation: FAIL on missing class/script. After: all green.
|
||||||
|
|
||||||
|
**Commit:** `feat(l4d2-host): KernelOverlayFSMounter + left4me-overlay helper`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: Wire OverlayMounter Through Lifecycle + Drop Fuse Module
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `l4d2host/instances.py` (start/stop/delete)
|
||||||
|
- Modify: `l4d2host/tests/test_lifecycle.py` (update argv assertions, add double-mount guard test)
|
||||||
|
- Delete: `l4d2host/fs/fuse_overlayfs.py`
|
||||||
|
- Verify: `l4d2host/fs/__init__.py` does not re-export `FuseOverlayFSMounter`
|
||||||
|
|
||||||
|
Test plan (update RED, then GREEN):
|
||||||
|
|
||||||
|
1. `test_lifecycle.py::test_start_order` — change assertion: `calls[0]` is now `["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", "mount", "alpha"]`. Adjust setup so the test still creates the merged directory.
|
||||||
|
2. `test_lifecycle.py::test_stop_succeeds_when_unmount_fails` — `cmd[0:5] == ["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", "umount", "alpha"]`.
|
||||||
|
3. `test_lifecycle.py::test_delete_succeeds_when_unmount_fails` — same.
|
||||||
|
4. NEW `test_lifecycle.py::test_start_refuses_double_mount` — monkeypatch `os.path.ismount` to return True; expect `start_instance` to raise `subprocess.CalledProcessError`; assert NO mount command was issued.
|
||||||
|
5. `test_lifecycle.py::test_lifecycle_rejects_unsafe_instance_names` — unchanged.
|
||||||
|
6. `test_lifecycle.py::test_delete_missing_is_noop` — unchanged.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- `instances.py` imports `KernelOverlayFSMounter`. Module-level singleton instance (`_mounter = KernelOverlayFSMounter()`). Replace direct `run_command([...fuse-overlayfs...])` with `_mounter.mount(...)`. Replace direct `run_command([...fusermount3...])` with `_mounter.unmount(...)` (still inside the existing try/except for stop/delete).
|
||||||
|
- Add the ismount guard at the top of `start_instance` after `runtime_dir` is computed, before `emit_step("mounting runtime overlay...")`. Raise `subprocess.CalledProcessError(returncode=1, cmd=["mount-guard"], stderr="runtime overlay already mounted at <path>; refusing to double-mount")`.
|
||||||
|
- Delete `l4d2host/fs/fuse_overlayfs.py`.
|
||||||
|
- Confirm `l4d2host/fs/__init__.py` is empty (already verified to be 1 line).
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest l4d2host/tests -q
|
||||||
|
python3 -m pytest l4d2web/tests -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Both green. Web tests: the `"Step: mounting runtime overlay..."` log line is preserved in `start_instance`.
|
||||||
|
|
||||||
|
**Commit:** `refactor(l4d2-host): start/stop/delete go through OverlayMounter; drop FuseOverlayFSMounter`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: Deploy Script Migration (Apt Deps + Wipe Upper/Work)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `deploy/deploy-test-server.sh`
|
||||||
|
- Modify: `deploy/tests/test_deploy_artifacts.py` (assert deploy script contains migration lines; assert `fuse-overlayfs` no longer in apt-get install)
|
||||||
|
|
||||||
|
Test plan:
|
||||||
|
|
||||||
|
1. `test_deploy_artifacts.py::test_deploy_script_drops_fuse_overlayfs_apt_dep` — `assert "fuse-overlayfs" not in deploy_script` and `assert "kernel-overlay-migrated" in deploy_script`.
|
||||||
|
2. `test_deploy_artifacts.py::test_deploy_script_migration_block_uses_sentinel` — `assert ".kernel-overlay-migrated" in deploy_script`.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
In `deploy/deploy-test-server.sh`, drop `fuse-overlayfs` from the apt-get and dnf lines (lines 82, 84). Insert before the existing `systemctl restart left4me-web.service` (line 182):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# One-time migration: fuse-overlayfs upperdir → kernel overlayfs upperdir.
|
||||||
|
# fuse-overlayfs running as the left4me user uses user.fuseoverlayfs.* xattrs
|
||||||
|
# for whiteouts and opaque dirs; kernel overlayfs ignores those, so any
|
||||||
|
# pre-existing upper/ from the fuse era would resurrect "deleted" files.
|
||||||
|
sentinel=/var/lib/left4me/.kernel-overlay-migrated
|
||||||
|
if [ ! -e "$sentinel" ]; then
|
||||||
|
$sudo_cmd systemctl stop 'left4me-server@*.service' 2>/dev/null || true
|
||||||
|
$sudo_cmd systemctl stop left4me-web.service 2>/dev/null || true
|
||||||
|
$sudo_cmd sh -c 'findmnt -t fuse.fuse-overlayfs -o TARGET --noheadings | xargs -r -n1 fusermount3 -u 2>/dev/null || true'
|
||||||
|
$sudo_cmd sh -c "findmnt -t overlay -o TARGET --noheadings | grep '/var/lib/left4me/runtime/' | xargs -r -n1 umount 2>/dev/null || true"
|
||||||
|
$sudo_cmd sh -c 'for d in /var/lib/left4me/runtime/*/; do [ -d "$d" ] || continue; rm -rf "$d/upper" "$d/work"; mkdir -p "$d/upper" "$d/work"; chown left4me:left4me "$d/upper" "$d/work"; done'
|
||||||
|
$sudo_cmd touch "$sentinel"
|
||||||
|
$sudo_cmd chown left4me:left4me "$sentinel"
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest deploy/tests -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Green.
|
||||||
|
|
||||||
|
**Commit:** `chore(deploy): drop fuse-overlayfs apt dep + one-shot migrate upper/work`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 4: Web Unit Hardening Cleanup + Docs
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `deploy/files/usr/local/lib/systemd/system/left4me-web.service`
|
||||||
|
- Modify: `deploy/tests/test_deploy_artifacts.py`
|
||||||
|
- Modify: `README.md`
|
||||||
|
- Modify: `l4d2host/README.md`
|
||||||
|
- Modify: `deploy/README.md`
|
||||||
|
|
||||||
|
Test plan:
|
||||||
|
|
||||||
|
1. `test_deploy_artifacts.py::test_web_unit_contains_required_runtime_contract` — drop `assert "MountFlags=shared" in unit` (or rather: replace with `assert "MountFlags=" not in unit`); add `assert "PrivateTmp=true" in unit`; add `assert "left4me-overlay" not in unit` (just to be precise — the unit shouldn't reference the helper directly, only via Python code).
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
Edit `left4me-web.service`:
|
||||||
|
|
||||||
|
- Drop `MountFlags=shared`.
|
||||||
|
- Restore `PrivateTmp=true`.
|
||||||
|
- Rewrite the comment block above hardening lines to explain: mounts now go through the `left4me-overlay` helper which `nsenter`s into PID 1's mount namespace, so this unit's namespace is irrelevant to gameserver visibility. `NoNewPrivileges` stays unset because sudo is setuid.
|
||||||
|
|
||||||
|
README updates:
|
||||||
|
|
||||||
|
- `README.md` (line ~59): drop fuse-overlayfs from tech-stack list; replace with "kernel overlayfs via privileged helper".
|
||||||
|
- `l4d2host/README.md`: lines 29, 52, 64 reference fuse — update to "kernel overlayfs (mount via the `left4me-overlay` helper deployed to `/usr/local/libexec/left4me/`)".
|
||||||
|
- `deploy/README.md`: add `/usr/local/libexec/left4me/left4me-overlay` to the privileged-helpers inventory.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest deploy/tests -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Green. Manual readthrough of the three READMEs confirms no stale fuse references.
|
||||||
|
|
||||||
|
**Commit:** `chore(deploy): cleanup left4me-web hardening + docs for kernel overlayfs`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 5: End-to-End Verification on `ckn@10.0.4.128`
|
||||||
|
|
||||||
|
**Pre-deploy:** branch is clean, all four prior commits land, all tests green locally.
|
||||||
|
|
||||||
|
**Deploy:**
|
||||||
|
|
||||||
|
```
|
||||||
|
deploy/deploy-test-server.sh ckn@10.0.4.128
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verification commands on the box:**
|
||||||
|
|
||||||
|
1. `test -e /var/lib/left4me/.kernel-overlay-migrated && echo migrated` — sentinel created.
|
||||||
|
2. `systemctl status left4me-web.service --no-pager` — `active (running)`, recent invocation timestamp.
|
||||||
|
3. From the UI or via `sudo -u left4me /opt/left4me/.venv/bin/l4d2ctl start test-server` — exit 0.
|
||||||
|
4. `findmnt /var/lib/left4me/runtime/test-server/merged` — shows fstype `overlay` in the host namespace.
|
||||||
|
5. `systemctl status left4me-server@test-server --no-pager` — `active (running)` after the start; **not** in `activating (auto-restart)`. No `status=200/CHDIR` errors in `journalctl -u left4me-server@test-server`.
|
||||||
|
6. `sudo journalctl -k --since "5 minutes ago" | grep -i apparmor | tail` — no overlay-related denials.
|
||||||
|
7. Negative test: `sudo -u left4me sudo -n /usr/local/libexec/left4me/left4me-overlay mount '../escape'` — exits non-zero with validation error.
|
||||||
|
8. Idempotency: `l4d2ctl stop test-server && l4d2ctl stop test-server` — both succeed (per the prior `fix(l4d2-host): make stop_instance idempotent` commit, still holds).
|
||||||
|
9. Re-start: `l4d2ctl start test-server` — succeeds, `findmnt` shows the mount again.
|
||||||
|
10. Double-mount guard: while the server is running, attempting another start (not via UI; via Python REPL or a second job) — `start_instance` raises `CalledProcessError` with the "refusing to double-mount" message. Optional, can be left to the unit test.
|
||||||
|
|
||||||
|
**On failure of any step:** stop and report. Do NOT push. The deploy script is rerunnable; the migration sentinel stays so wipe doesn't repeat.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- See spec's "Out Of Scope" section.
|
||||||
|
- This plan does not push commits; pushing is a separate user decision after end-to-end verification passes.
|
||||||
350
docs/superpowers/plans/2026-05-08-l4d2-script-overlays.md
Normal file
350
docs/superpowers/plans/2026-05-08-l4d2-script-overlays.md
Normal file
|
|
@ -0,0 +1,350 @@
|
||||||
|
# L4D2 Script Overlays Implementation Plan
|
||||||
|
|
||||||
|
> **Approval status:** User-approved 2026-05-08. Implementation proceeds.
|
||||||
|
|
||||||
|
**Goal:** Implement the `script` overlay type per `docs/superpowers/specs/2026-05-08-l4d2-script-overlays-design.md`. Add an `Overlay.script` TEXT column and `Overlay.last_build_status` enum-string column, a `ScriptBuilder` that runs user bash inside a `bubblewrap` + `systemd-run --scope` sandbox via a new `left4me-script-sandbox` privileged helper, route + UI surface for editing/wiping/rebuilding, and delete the entire managed-globals (`l4d2center_maps`, `cedapug_maps`) subsystem and its daily-refresh timer/CLI.
|
||||||
|
|
||||||
|
**Architecture:** The web app continues to enqueue `build_overlay` jobs for any overlay row. The job worker dispatches via `BUILDERS[overlay.type].build(...)`. After this change `BUILDERS = {"workshop": WorkshopBuilder(), "script": ScriptBuilder()}`. The new `ScriptBuilder` writes `overlay.script` to a tmpfile and execs `sudo -n /usr/local/libexec/left4me/left4me-script-sandbox <id> <tmpfile>`, which itself execs `systemd-run --scope --collect ... -- bwrap [namespace flags] /bin/bash /script.sh`. stdout/stderr stream through the existing `run_with_streamed_output` helper into the existing job-log SSE plumbing. The job-completion path writes `Overlay.last_build_status` based on the build outcome. The kernel-overlayfs mount layer (`KernelOverlayFSMounter`) is unchanged.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
See `docs/superpowers/specs/2026-05-08-l4d2-script-overlays-design.md` for design rationale. Implementation-relevant summary:
|
||||||
|
|
||||||
|
- Final overlay type list: `workshop` (unchanged) + `script` (new). Drop `l4d2center_maps`, `cedapug_maps`.
|
||||||
|
- New columns on `overlays`: `script TEXT NOT NULL DEFAULT ''`, `last_build_status VARCHAR(16) NOT NULL DEFAULT ''`.
|
||||||
|
- Drop tables (FK order): `global_overlay_item_files`, `global_overlay_items`, `global_overlay_sources`.
|
||||||
|
- `ScriptBuilder` in `l4d2web/services/overlay_builders.py`, uses existing `run_with_streamed_output`.
|
||||||
|
- Privileged helper `left4me-script-sandbox` (bash, mode 0755, owned root). `systemd-run --scope --collect -p MemoryMax=4G -p MemorySwapMax=0 -p TasksMax=512 -p CPUQuota=200% -p RuntimeMaxSec=3600 -- bwrap …`. Limits 1 h walltime, 4 GB RAM, 20 GB post-build `du` cap.
|
||||||
|
- New system user `l4d2-sandbox` (`/usr/sbin/nologin`, no home). New apt dep `bubblewrap`.
|
||||||
|
- Sudoers verb-unrestricted: `left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-script-sandbox`.
|
||||||
|
- Daily refresh subsystem deleted: `left4me-refresh-global-overlays.{timer,service}` and `flask refresh-global-overlays` CLI removed. No replacement.
|
||||||
|
- Wipe is the same sandbox helper invoked with the literal script `find /overlay -mindepth 1 -delete`.
|
||||||
|
- `auto_refresh` column NOT added in this iteration.
|
||||||
|
- Test deploy DB is wiped on rollout; migration includes `DELETE FROM overlays WHERE type IN ('l4d2center_maps', 'cedapug_maps')` for safety.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Gap
|
||||||
|
|
||||||
|
- `l4d2web/models.py` `Overlay` has no `script` or `last_build_status` columns. The 3 globals tables are present.
|
||||||
|
- `l4d2web/services/overlay_builders.py` `BUILDERS = {"workshop": WorkshopBuilder(), "l4d2center_maps": GlobalMapOverlayBuilder(), "cedapug_maps": GlobalMapOverlayBuilder()}`. No `ScriptBuilder`.
|
||||||
|
- `l4d2web/services/{global_map_sources,global_overlay_refresh,global_map_cache,global_overlays}.py` exist and are referenced by routes / CLI.
|
||||||
|
- `l4d2web/services/job_worker.py` carries `refresh_global_overlays_running` plumbing.
|
||||||
|
- `l4d2web/cli.py` defines `refresh-global-overlays`.
|
||||||
|
- `l4d2web/routes/overlay_routes.py` has no `/script`, `/wipe`, or `/build` endpoints for non-workshop types.
|
||||||
|
- `l4d2web/templates/overlays.html` create modal type radio offers only `workshop`.
|
||||||
|
- `l4d2web/templates/overlay_detail.html` has a global-source block (~lines 34–46) that should not survive.
|
||||||
|
- `deploy/files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.{timer,service}` exist.
|
||||||
|
- `deploy/deploy-test-server.sh` provisions `global_overlay_cache/` and does not provision `l4d2-sandbox` or install `bubblewrap`.
|
||||||
|
- Seven `tests/test_global_*.py` files exist and reference removed code.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: Schema migration (alembic 0005)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `l4d2web/alembic/versions/0005_script_overlays.py` (revises `0004_drop_legacy_external_overlay_type`).
|
||||||
|
- Modify: `l4d2web/models.py` — `Overlay` gains `script` and `last_build_status` columns; remove `GlobalOverlaySource`, `GlobalOverlayItem`, `GlobalOverlayItemFile` model classes.
|
||||||
|
- Modify: `l4d2web/tests/test_overlay_models.py` (or whichever existing test asserts the Overlay schema; create one if absent) — assert new columns present.
|
||||||
|
|
||||||
|
Test plan (RED first):
|
||||||
|
|
||||||
|
1. `tests/test_alembic_migrations.py::test_upgrade_0005_adds_script_columns` — apply migrations to a fresh in-memory SQLite, assert `script` and `last_build_status` columns present on `overlays`, assert no `global_overlay_*` tables, assert old data wipe `DELETE FROM overlays WHERE type IN (...)` is part of the upgrade.
|
||||||
|
2. `tests/test_alembic_migrations.py::test_downgrade_0005_restores_globals` (only if downgrade is supported in the project's migration policy; skip with `pytest.skip` if not — kernel-overlayfs migration is one-way, follow that precedent).
|
||||||
|
3. `tests/test_overlay_models.py::test_overlay_has_script_columns` — `Overlay(...)` instance has `script=''` and `last_build_status=''` defaults.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Migration uses `op.drop_table('global_overlay_item_files')` etc. in correct FK order; uses `op.add_column('overlays', sa.Column('script', sa.Text(), nullable=False, server_default=''))` and similar for `last_build_status` (`sa.String(16)`).
|
||||||
|
- The `DELETE FROM overlays WHERE type IN ('l4d2center_maps','cedapug_maps')` runs *before* the column additions so the operation is straightforward — these rows do not reference the new columns.
|
||||||
|
- `models.py`: delete the three globals model classes outright; add the two new columns to `Overlay` with explicit defaults.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest l4d2web/tests/test_alembic_migrations.py l4d2web/tests/test_overlay_models.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit:** `feat(l4d2-web): script overlay schema — add overlay.script + last_build_status, drop globals tables`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: ScriptBuilder + BUILDERS registry update
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `l4d2web/services/overlay_builders.py` — add `ScriptBuilder`, remove `GlobalMapOverlayBuilder`, change `BUILDERS` dict.
|
||||||
|
- Rewrite: `l4d2web/tests/test_overlay_builders.py` — drop globals-builder tests, add ScriptBuilder tests.
|
||||||
|
|
||||||
|
Test plan (RED first):
|
||||||
|
|
||||||
|
1. `test_overlay_builders.py::test_builders_registry` — `set(BUILDERS) == {"workshop", "script"}`. Assert `"l4d2center_maps"` and `"cedapug_maps"` and `"external"` are absent.
|
||||||
|
2. `test_overlay_builders.py::test_script_builder_invokes_helper` — patch `run_with_streamed_output` to capture argv; build an `Overlay(id=42, type='script', script='echo hi')`; assert argv shape `["sudo", "-n", "/usr/local/libexec/left4me/left4me-script-sandbox", "42", <script_path>]` and that the script_path file exists with content `"echo hi"` at invocation time. Verify the tmpfile is unlinked after build.
|
||||||
|
3. `test_overlay_builders.py::test_script_builder_disk_cap` — fake `subprocess.check_output` for `du` to return `25000000000`; build raises `BuildError("disk-cap-exceeded")` and `on_stderr` was called with the cap message.
|
||||||
|
4. `test_overlay_builders.py::test_script_builder_streams_output` — fake `run_with_streamed_output` invokes both `on_stdout("hello\n")` and `on_stderr("warn\n")`; both lambda lists capture the lines.
|
||||||
|
5. `test_overlay_builders.py::test_script_builder_cancel` — `should_cancel` returns True after the first stdout line; assert `run_with_streamed_output` propagated cancellation (the existing helper's contract — the test just ensures we pass `should_cancel` through and don't run the disk-budget check on cancel).
|
||||||
|
6. `test_overlay_builders.py::test_workshop_builder_unchanged` — smoke test that `WorkshopBuilder` still exists and is invokable (regression guard against accidental removal during refactor).
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Add `import os, subprocess, tempfile` at the top of `overlay_builders.py` if not present.
|
||||||
|
- `ScriptBuilder` exactly as in the spec (verbatim copy from the design doc, §Build Lifecycle).
|
||||||
|
- Define a small `BuildError` exception class if one doesn't already exist locally; reuse the existing one if `WorkshopBuilder` already raises a similar type.
|
||||||
|
- `_enforce_disk_budget` calls `subprocess.check_output(["du", "-sb", str(overlay_path(overlay_id))])`; the existing `overlay_path` helper in the module already returns the absolute Path. Parse first whitespace-delimited integer; cap is `20 * 1024**3`.
|
||||||
|
- Job-completion path: locate the existing path that handles `build_overlay` job success/failure (likely in `services/job_worker.py` or a related orchestration module). Add a single column write: on success `last_build_status='ok'`, on `BuildError` / non-zero exit / cancel `last_build_status='failed'`. Add a `tests/test_job_worker.py::test_build_overlay_writes_last_build_status` covering both branches.
|
||||||
|
- Remove `GlobalMapOverlayBuilder` class and any helper functions it owns that are not used elsewhere.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest l4d2web/tests/test_overlay_builders.py l4d2web/tests/test_job_worker.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit:** `feat(l4d2-web): ScriptBuilder + BUILDERS registry update`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: Delete global-overlay services + CLI command + their tests
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Delete: `l4d2web/services/global_map_sources.py`
|
||||||
|
- Delete: `l4d2web/services/global_overlay_refresh.py`
|
||||||
|
- Delete: `l4d2web/services/global_map_cache.py`
|
||||||
|
- Delete: `l4d2web/services/global_overlays.py`
|
||||||
|
- Modify: `l4d2web/cli.py` — remove `refresh-global-overlays` command (lines ~44–55). Drop any imports that go orphaned.
|
||||||
|
- Delete: `l4d2web/tests/test_global_map_sources.py`
|
||||||
|
- Delete: `l4d2web/tests/test_global_overlay_models.py`
|
||||||
|
- Delete: `l4d2web/tests/test_global_overlay_builders.py`
|
||||||
|
- Delete: `l4d2web/tests/test_global_overlay_cli.py`
|
||||||
|
- Delete: `l4d2web/tests/test_global_overlay_refresh.py`
|
||||||
|
- Delete: `l4d2web/tests/test_global_overlays.py`
|
||||||
|
- Delete: `l4d2web/tests/test_global_map_cache.py`
|
||||||
|
- Audit & fix: any other module that imports the deleted modules. Likely candidates: `l4d2web/app.py` (CLI registration), `routes/overlay_routes.py`, `routes/page_routes.py`. Resolve by deletion of the dead import / call site, not by stubbing.
|
||||||
|
- Modify: `pyproject.toml` — drop `py7zr` from dependencies (only used by the deleted globals subsystem).
|
||||||
|
|
||||||
|
Test plan:
|
||||||
|
|
||||||
|
1. RED-first via grep: `grep -RIn 'global_map_sources\|global_overlay_refresh\|global_map_cache\|global_overlays\|refresh_global_overlays\|GlobalMapOverlayBuilder' l4d2web/ deploy/` — should return zero hits at the end of this task. Add this as `tests/test_no_globals_references.py::test_no_globals_imports` if you want it as a permanent regression guard, otherwise spot-check.
|
||||||
|
2. Existing `tests/test_cli.py` (or whichever covers Flask CLI) loses any cases for `refresh-global-overlays`; add a `test_refresh_global_overlays_command_removed` that asserts the click command is not registered.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Delete files via `git rm`.
|
||||||
|
- In `cli.py`, remove the command function and its `@app.cli.command(...)` decorator. Drop any helper imports that become orphaned.
|
||||||
|
- Remove `py7zr` from `pyproject.toml` and re-lock if a lockfile is present.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest l4d2web/tests/ -q
|
||||||
|
grep -RIn 'global_map_sources\|global_overlay_refresh\|global_map_cache\|global_overlays\|refresh_global_overlays\|GlobalMapOverlayBuilder' l4d2web/ deploy/ || echo "clean"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit:** `refactor(l4d2-web): drop global-overlays subsystem in favor of script type`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 4: Job worker — drop refresh_global_overlays from scheduler
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `l4d2web/services/job_worker.py` — remove `"refresh_global_overlays"` from `GLOBAL_OPERATIONS`; remove `refresh_global_overlays_running` field from `SchedulerState` and any references in `can_start()`; check whether `blocked_servers_by_overlay` was added solely for the globals subsystem and remove if so.
|
||||||
|
- Modify: `l4d2web/tests/test_job_worker.py` — drop `refresh_global_overlays` truth-table rows; add explicit `build_overlay` truth-table cases for `script`-type overlays (mechanically identical to workshop, but pinned by test).
|
||||||
|
|
||||||
|
Test plan:
|
||||||
|
|
||||||
|
1. `test_job_worker.py::test_global_operations_set` — `GLOBAL_OPERATIONS == {"install", "refresh_workshop_items"}` (or whatever subset remains; pin it).
|
||||||
|
2. `test_job_worker.py::test_build_overlay_script_type_blocks_per_overlay` — start `build_overlay(overlay_id=7)` for a `script`-type overlay; assert second `build_overlay(overlay_id=7)` cannot start; assert `build_overlay(overlay_id=8)` can.
|
||||||
|
3. `test_job_worker.py::test_build_overlay_blocks_server_init_on_blueprint_overlay` — existing test, may need re-pinning if it referenced globals.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Remove the field from the dataclass / TypedDict that backs `SchedulerState`.
|
||||||
|
- Remove any update sites that flipped the flag (the worker's enqueue / on-start / on-complete paths).
|
||||||
|
- The remaining mutex rules (`install` / `refresh_workshop_items` are global; `build_overlay` per-overlay; server ops block on overlays in their blueprint) are unchanged structurally.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest l4d2web/tests/test_job_worker.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit:** `refactor(l4d2-web): drop refresh_global_overlays from scheduler`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 5: Routes (script update / wipe / build)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `l4d2web/routes/overlay_routes.py` — add three POST endpoints.
|
||||||
|
- Create: `l4d2web/tests/test_script_overlay_routes.py`.
|
||||||
|
|
||||||
|
Test plan (RED first):
|
||||||
|
|
||||||
|
1. `test_script_overlay_routes.py::test_create_script_overlay` — POST `/overlays` with form `{"name": "x", "type": "script"}` as a regular user → 302 to detail; row exists with `type='script'`, `script=''`, `last_build_status=''`, `user_id=current_user.id`, `path=str(id)`.
|
||||||
|
2. `test_script_overlay_routes.py::test_admin_creates_system_wide_script_overlay` — admin POST with system-wide flag → row has `user_id=NULL`.
|
||||||
|
3. `test_script_overlay_routes.py::test_update_script_body_enqueues_build` — POST `/overlays/{id}/script` with `{"script": "echo new"}` → row.script updated; one new `build_overlay` job enqueued for the overlay; second immediate POST coalesces (no second job inserted while first is pending).
|
||||||
|
4. `test_script_overlay_routes.py::test_manual_rebuild` — POST `/overlays/{id}/build` → enqueues `build_overlay`; coalesces.
|
||||||
|
5. `test_script_overlay_routes.py::test_wipe_runs_find_delete` — POST `/overlays/{id}/wipe` → invokes `ScriptBuilder.build` (or the underlying helper) with the literal script `find /overlay -mindepth 1 -delete`. After success, row.last_build_status `==''`. Does not enqueue a `build_overlay`.
|
||||||
|
6. `test_script_overlay_routes.py::test_wipe_refuses_during_running_build` — set scheduler state to `build_overlay(overlay_id=7)` running; POST `/overlays/7/wipe` → 409 (or whatever the existing pattern uses for scheduler conflicts), no sandbox invocation.
|
||||||
|
7. `test_script_overlay_routes.py::test_permissions_non_owner_denied` — user A creates private script overlay; user B POSTs `/overlays/{id}/script` → 403.
|
||||||
|
8. `test_script_overlay_routes.py::test_permissions_admin_can_edit_any` — admin POSTs `/overlays/{id}/script` for user A's row → 200.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Mirror the existing `_can_edit_overlay()` permission helper.
|
||||||
|
- The `/wipe` endpoint can either (a) call `ScriptBuilder` directly with a synthetic `Overlay`-like object whose `.script` is the find command and whose `.id` is the real overlay id, or (b) factor a `_run_sandbox(overlay_id, script_text, on_stdout, on_stderr, should_cancel)` helper out of `ScriptBuilder.build()` and call it from both. (b) is cleaner; do (b).
|
||||||
|
- Wipe runs **synchronously** in the request thread (small, fast). It does NOT enqueue a job. Surface log output as flash messages or by streaming through the existing log infra — pick whichever matches the existing wipe-equivalent pattern (workshop overlays don't have a wipe; closest analog is the existing delete-overlay flow).
|
||||||
|
- The `/script` endpoint enqueues via the same `enqueue_build_overlay(overlay_id)` helper used by workshop overlays' add/remove flows. Coalescing is already implemented there.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest l4d2web/tests/test_script_overlay_routes.py l4d2web/tests/test_overlay_routes.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit:** `feat(l4d2-web): script overlay routes (script update / wipe / build)`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 6: Templates (overlays.html + overlay_detail.html)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `l4d2web/templates/overlays.html` — add `script` to the create-modal type radio (lines ~29–49).
|
||||||
|
- Modify: `l4d2web/templates/overlay_detail.html` — add a `{% if overlay.type == 'script' %}` block with textarea + Save / Rebuild / Wipe buttons + status badge; delete the global-source block (lines ~34–46).
|
||||||
|
- Modify: `l4d2web/tests/test_pages.py` — assert script-section renders for type=`script`, workshop-section renders for type=`workshop`, global-source-section is absent.
|
||||||
|
|
||||||
|
Test plan:
|
||||||
|
|
||||||
|
1. `test_pages.py::test_overlay_create_modal_offers_script_type` — GET `/overlays`; HTML contains `value="script"` radio.
|
||||||
|
2. `test_pages.py::test_overlay_detail_script_section` — create script overlay, GET `/overlays/{id}`; HTML contains `<textarea name="script">`, "Rebuild" button, "Wipe" button, status badge element.
|
||||||
|
3. `test_pages.py::test_overlay_detail_workshop_section_unchanged` — existing workshop detail still has thumbnail grid, add-item form, etc.
|
||||||
|
4. `test_pages.py::test_overlay_detail_no_global_source_block` — page HTML has no element from the deleted global-source block (check for an attribute or string unique to that block).
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Detail-page wipe button uses a small confirm-modal pattern (copy from the existing delete-overlay confirm modal).
|
||||||
|
- Status badge: existing CSS classes for ok/warn/error already exist in `static/`; reuse them.
|
||||||
|
- No new JS deps. Plain `<form method="post">` with HTMX `hx-post` for the script update if a streaming UX is desired (match existing patterns).
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest l4d2web/tests/test_pages.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Manual: start dev server (`flask run`), create a script overlay, paste `echo "hi" > foo`, click Save, watch log stream. Then click Wipe; confirm dir is empty. Then click Rebuild; confirm `foo` reappears.
|
||||||
|
|
||||||
|
**Commit:** `feat(l4d2-web): script overlay UI`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 7: Libexec sandbox helper + sudoers + deploy-artifacts test
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` (bash, mode 0755 after deploy, owned root).
|
||||||
|
- Modify: `deploy/files/etc/sudoers.d/left4me` — append the rule.
|
||||||
|
- Modify: `deploy/tests/test_deploy_artifacts.py` — assert helper file present + sudoers contains the new line.
|
||||||
|
|
||||||
|
Test plan (RED first):
|
||||||
|
|
||||||
|
1. `test_deploy_artifacts.py::test_script_sandbox_helper_present` — file exists, mode bits indicate 0755 (or whatever the test framework allows checking pre-deploy), shebang is `#!/bin/bash`.
|
||||||
|
2. `test_deploy_artifacts.py::test_sudoers_includes_script_sandbox_rule` — sudoers file contains the exact line `left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-script-sandbox`.
|
||||||
|
3. Optional integration test (skip on non-Linux dev): drive the helper as a subprocess with a synthesized fake `/var/lib/left4me/overlays/1/` and a no-op script, assert `bwrap` invocation happens (use a mock `systemd-run` or `LEFT4ME_SCRIPT_SANDBOX_DRY_RUN=1` env that prints the would-be invocation and exits 0). Mirrors the `LEFT4ME_OVERLAY_PRINT_ONLY=1` pattern from the kernel-overlayfs helper test.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Helper script verbatim from the spec §Sandbox.
|
||||||
|
- Sudoers fragment: append (don't replace existing rules). The existing fragment has rules for `left4me-overlay`, `left4me-systemctl`, `left4me-journalctl` — match the same formatting (one rule per line, no trailing whitespace).
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest deploy/tests/test_deploy_artifacts.py -q
|
||||||
|
bash -n deploy/files/usr/local/libexec/left4me/left4me-script-sandbox
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit:** `feat(deploy): left4me-script-sandbox helper + sudoers fragment`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 8: Deploy script — provision l4d2-sandbox + bubblewrap; drop globals timer
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `deploy/deploy-test-server.sh` — add `useradd --system ... l4d2-sandbox`, add `apt-get install -y bubblewrap`, ensure helper installation step picks up `left4me-script-sandbox` (likely automatic if it's a glob in `deploy/files/usr/local/libexec/left4me/*`); drop the `mkdir global_overlay_cache` line if present.
|
||||||
|
- Delete: `deploy/files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer`
|
||||||
|
- Delete: `deploy/files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.service`
|
||||||
|
- Modify: `deploy/tests/test_deploy_artifacts.py` — assert the two unit files are absent; assert `useradd l4d2-sandbox` and `apt-get install ... bubblewrap` lines are present in the deploy script.
|
||||||
|
|
||||||
|
Test plan:
|
||||||
|
|
||||||
|
1. `test_deploy_artifacts.py::test_globals_refresh_units_removed` — files do not exist under `deploy/files/usr/local/lib/systemd/system/`.
|
||||||
|
2. `test_deploy_artifacts.py::test_deploy_script_provisions_sandbox_user` — grep the deploy script for the useradd line.
|
||||||
|
3. `test_deploy_artifacts.py::test_deploy_script_installs_bubblewrap` — grep for `bubblewrap` in apt invocations.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- `useradd` line uses `--system --no-create-home --shell /usr/sbin/nologin`. Idempotency: wrap with `id l4d2-sandbox &>/dev/null || useradd ...`.
|
||||||
|
- `apt-get install`: append `bubblewrap` to whatever package list the script already maintains.
|
||||||
|
- Globals timer/service deletions: `git rm`.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest deploy/tests/ -q
|
||||||
|
shellcheck deploy/deploy-test-server.sh deploy/files/usr/local/libexec/left4me/left4me-script-sandbox
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit:** `chore(deploy): provision l4d2-sandbox + bubblewrap; drop globals refresh timer`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 9: Full pytest run + drift fixes
|
||||||
|
|
||||||
|
**Files:** as needed across the repo.
|
||||||
|
|
||||||
|
Test plan: run the full test suite for both packages; chase down any drift caused by removed model classes, dropped imports, or template changes.
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest l4d2web/tests/ -q
|
||||||
|
python3 -m pytest l4d2host/tests/ -q
|
||||||
|
python3 -m pytest deploy/tests/ -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Implementation: fix what breaks. Common drift sources to expect:
|
||||||
|
|
||||||
|
- Tests that imported from deleted modules.
|
||||||
|
- Tests that asserted exact `BUILDERS` keyset (good — they should have been updated in Task 2).
|
||||||
|
- Tests that built fixtures with `type='l4d2center_maps'` or `type='cedapug_maps'` — those tests likely belong to the deleted set or need conversion to `type='script'`.
|
||||||
|
- Template snapshot tests (if any) that captured the deleted global-source block.
|
||||||
|
|
||||||
|
**Verification:** all three suites green.
|
||||||
|
|
||||||
|
**Commit:** `chore(l4d2-web): test suite drift fixes after script-overlays migration` (only if drift fixes needed; skip if Tasks 1–8 left the suite green)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## End-to-end deployment verification (manual, on test host)
|
||||||
|
|
||||||
|
After all tasks committed:
|
||||||
|
|
||||||
|
1. **Reset deploy:** run `deploy/deploy-test-server.sh` from clean state. Confirm `bubblewrap` installed (`dpkg -l bubblewrap`), `l4d2-sandbox` user exists (`id l4d2-sandbox`), `/usr/local/libexec/left4me/left4me-script-sandbox` is mode 0755 and root-owned, `sudo -ln` as `left4me` shows the new rule.
|
||||||
|
2. **Sandbox smoke:** as `left4me`, write `/tmp/echo.sh` containing `echo $(whoami) > /overlay/sentinel`. `mkdir -p /var/lib/left4me/overlays/1`. `sudo /usr/local/libexec/left4me/left4me-script-sandbox 1 /tmp/echo.sh`. Confirm `/var/lib/left4me/overlays/1/sentinel` contains `l4d2-sandbox` and is owned by `l4d2-sandbox`. Confirm `/etc/passwd`, `/var/lib/left4me/l4d2web.db`, and `/home` are not visible inside the sandbox by running probe scripts.
|
||||||
|
3. **Resource limits:**
|
||||||
|
- `dd if=/dev/zero of=/overlay/big bs=1M count=25000` → succeeds inside sandbox; `ScriptBuilder._enforce_disk_budget` flags the build failed; `last_build_status='failed'`.
|
||||||
|
- `sleep 7200` → killed at 1 h by `RuntimeMaxSec=3600`.
|
||||||
|
- Memory hog (`python3 -c "x=' '*(5*1024**3)"`) → OOM at 4 GB.
|
||||||
|
4. **App-level happy path:** as a non-admin user, create a script overlay via the UI, paste an old `competitive_rework`-style script, Save → build runs, succeeds, addons appear in `overlays/{id}/left4dead2/`. Stack onto a server blueprint, start the server, verify content mounts via the L4D2 admin console (`map workshop/...`).
|
||||||
|
5. **Wipe:** click Wipe → dir empty (find -delete output in log). Click Rebuild → repopulates. `last_build_status` cycles: `''` → `'ok'`.
|
||||||
|
6. **Scheduler:** start a server using the script overlay; in another browser tab attempt to Rebuild → 409 / scheduler-blocked. Stop server; rebuild succeeds.
|
||||||
|
7. **Audit log:** `journalctl --since "5 min ago" | grep run-` shows transient scopes per build with cgroup memory accounting visible.
|
||||||
|
|
||||||
|
These are not required for any single commit but should pass before declaring the work done.
|
||||||
|
|
@ -0,0 +1,146 @@
|
||||||
|
# L4D2 Script Sandbox v2 Implementation Plan
|
||||||
|
|
||||||
|
> **Approval status:** User-approved 2026-05-08 after smoke-testing the v2 prototype on `ckn@10.0.4.128`.
|
||||||
|
|
||||||
|
**Goal:** Replace the bwrap-based sandbox helper with a systemd-only one per `docs/superpowers/specs/2026-05-08-l4d2-script-sandbox-v2-systemd.md`. Drop the `bubblewrap` apt dep. Tighten `left4me.db` file mode to 0640 root:left4me. Update the deploy-artifact tests to assert the new helper shape.
|
||||||
|
|
||||||
|
**Architecture:** See spec. Helper invokes `systemd-run --pipe --wait` in service-unit mode with full hardening directives. No bwrap. Web-app side (`ScriptBuilder`, `run_sandboxed_script`, routes) is unchanged.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
See spec §Locked Decisions for rationale. Implementation summary:
|
||||||
|
|
||||||
|
- Helper file at the same path (`deploy/files/usr/local/libexec/left4me/left4me-script-sandbox`) is rewritten in place.
|
||||||
|
- The sudoers rule is unchanged.
|
||||||
|
- `bubblewrap` dropped from `apt-get install` / `dnf install` lines.
|
||||||
|
- `left4me.db` chmod 0640 added to deploy script as a post-init step.
|
||||||
|
- Sandbox UID, system user, overlay-dir chown logic, and ScriptBuilder API stay the same.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Gap
|
||||||
|
|
||||||
|
- `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` invokes `systemd-run --scope ... -- bwrap [namespace flags] /bin/bash /script.sh`.
|
||||||
|
- `deploy/deploy-test-server.sh` line ~84 installs `bubblewrap` via apt/dnf.
|
||||||
|
- `deploy/tests/test_deploy_artifacts.py::test_script_sandbox_helper_invokes_systemd_run_and_bwrap` asserts `bwrap`, `--unshare-pid`, `--uid=l4d2-sandbox`, etc.
|
||||||
|
- `deploy/tests/test_deploy_artifacts.py::test_deploy_script_installs_bubblewrap` asserts `bubblewrap` is in apt/dnf install lines.
|
||||||
|
- `left4me.db` is created at deploy time with the default 0644 permissions; any host user can read it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: Rewrite the sandbox helper to be systemd-only
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` — replace the `systemd-run --scope … bwrap …` invocation with `systemd-run --service --pipe --wait …` carrying the hardening directives.
|
||||||
|
|
||||||
|
Test plan:
|
||||||
|
|
||||||
|
1. `bash -n` syntax check (already covered by `test_script_sandbox_helper_passes_shell_syntax_check`).
|
||||||
|
2. `test_deploy_artifacts.py::test_script_sandbox_helper_invokes_systemd_run_and_bwrap` is replaced by a new pin: `test_script_sandbox_helper_invokes_systemd_run_with_hardening`. Asserts:
|
||||||
|
- No `bwrap` reference remains.
|
||||||
|
- `systemd-run` is invoked with `--pipe`, `--wait`, `--collect`, `--unit=` (transient service unit form, no `--scope`).
|
||||||
|
- All hardening directives present: `NoNewPrivileges=yes`, `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp=yes`, `PrivateDevices=yes`, `PrivateIPC=yes`, `ProtectKernelTunables=yes`, `ProtectKernelModules=yes`, `ProtectKernelLogs=yes`, `ProtectControlGroups=yes`, `RestrictNamespaces=yes`, `RestrictSUIDSGID=yes`, `LockPersonality=yes`, `MemoryDenyWriteExecute=yes`, `SystemCallFilter=`, `CapabilityBoundingSet=` (empty), `User=l4d2-sandbox`, `Group=l4d2-sandbox`.
|
||||||
|
- `TemporaryFileSystem=` covers `/etc` and `/var/lib`.
|
||||||
|
- `BindReadOnlyPaths=` includes `/etc/resolv.conf /etc/ssl /etc/ca-certificates /etc/nsswitch.conf /etc/alternatives` and the script bind `${SCRIPT}:/script.sh`.
|
||||||
|
- `BindPaths=` carries the overlay bind.
|
||||||
|
- Cgroup limits unchanged (`MemoryMax=4G`, `MemorySwapMax=0`, `TasksMax=512`, `CPUQuota=200%`, `RuntimeMaxSec=3600`).
|
||||||
|
3. Existing `test_script_sandbox_helper_dry_run_mode` keeps passing — the dry-run guard still short-circuits before `systemd-run`.
|
||||||
|
4. Existing `test_script_sandbox_helper_validates_overlay_id` keeps passing — argument validation is unchanged.
|
||||||
|
|
||||||
|
Implementation: helper body verbatim from the spec §Helper.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest deploy/tests/test_deploy_artifacts.py -q
|
||||||
|
bash -n deploy/files/usr/local/libexec/left4me/left4me-script-sandbox
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit:** `refactor(deploy): rewrite left4me-script-sandbox to systemd-only — drop bwrap`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: Drop bubblewrap apt/dnf dep + tighten left4me.db mode
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `deploy/deploy-test-server.sh` — remove `bubblewrap` from `apt-get install` / `dnf install` package lists; add a post-init step that ensures `left4me.db` is mode 0640 owned `root:left4me`.
|
||||||
|
- Modify: `deploy/tests/test_deploy_artifacts.py` — replace `test_deploy_script_installs_bubblewrap` with `test_deploy_script_does_not_install_bubblewrap`; add `test_deploy_script_tightens_left4me_db_permissions`.
|
||||||
|
|
||||||
|
Test plan:
|
||||||
|
|
||||||
|
1. `test_deploy_script_does_not_install_bubblewrap` — for each `apt-get install` / `dnf install` line, `bubblewrap` is absent.
|
||||||
|
2. `test_deploy_script_tightens_left4me_db_permissions` — script contains `chmod 0640 /var/lib/left4me/left4me.db` and `chown root:left4me /var/lib/left4me/left4me.db` (in either order).
|
||||||
|
3. `test_deploy_script_shell_syntax` keeps passing (`sh -n`).
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Remove the bare `bubblewrap` token from the two install lines.
|
||||||
|
- After the `alembic upgrade head` step (which creates the DB if missing), add:
|
||||||
|
```
|
||||||
|
$sudo_cmd chown root:left4me /var/lib/left4me/left4me.db
|
||||||
|
$sudo_cmd chmod 0640 /var/lib/left4me/left4me.db
|
||||||
|
```
|
||||||
|
Idempotent — re-runs are no-ops.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest deploy/tests/test_deploy_artifacts.py -q
|
||||||
|
sh -n deploy/deploy-test-server.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit:** `chore(deploy): drop bubblewrap apt dep + tighten left4me.db mode to 0640 root:left4me`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: Deploy + smoke-test on the test host
|
||||||
|
|
||||||
|
**Files:** none.
|
||||||
|
|
||||||
|
This is an operational verification step, not a code change. Run `deploy/deploy-test-server.sh ckn@10.0.4.128`, then on the host re-run the same smoke battery used to validate the prototype:
|
||||||
|
|
||||||
|
1. **Identity / privileges**: `id` returns `uid=996 gid=985`; `/proc/self/status` shows `NoNewPrivs: 1` and `CapBnd: 0000000000000000`.
|
||||||
|
2. **Filesystem isolation**: `/etc/passwd` absent, `/etc/alternatives/awk` present, `/var/lib/left4me/left4me.db` absent, `/home` inaccessible, `/usr` not writable, `/overlay` writable.
|
||||||
|
3. **Tools + network**: `awk` resolves through `/etc/alternatives`; `curl https://steamcommunity.com/` returns 200.
|
||||||
|
4. **Cgroup limits**: while a 5s-sleep script runs, `cat /sys/fs/cgroup/.../memory.max` returns `4294967296`; `pids.max` `512`; `cpu.max` `200000 100000`.
|
||||||
|
5. **Memory cap**: 5 GB Python alloc raises `MemoryError`.
|
||||||
|
6. **Wipe**: `find /overlay -mindepth 1 -delete` empties the overlay dir.
|
||||||
|
7. **Seccomp / restriction probes**: `unshare -U`, `mount -t tmpfs`, `setarch -X`, `bpf` setsockopt all fail with EPERM/EINVAL.
|
||||||
|
8. **Build via web UI**: log in as admin, create a script overlay with `echo "hi" > foo`, click Save, confirm job succeeds and `foo` appears in `/var/lib/left4me/overlays/{id}/foo`.
|
||||||
|
9. **DB hardening**: `stat -c "%a %U:%G" /var/lib/left4me/left4me.db` returns `640 root:left4me`.
|
||||||
|
|
||||||
|
Mark this task complete only after every check passes on the live host.
|
||||||
|
|
||||||
|
**Commit:** none (operational verification — record results in conversation/PR description).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 4: Drift sweep + push
|
||||||
|
|
||||||
|
**Files:** as needed across the repo.
|
||||||
|
|
||||||
|
Run the full test suite for all three packages; chase any drift caused by the helper rewrite or deploy-script changes.
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m pytest l4d2web/tests/ -q
|
||||||
|
python3 -m pytest l4d2host/tests/ -q
|
||||||
|
python3 -m pytest deploy/tests/ -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Implementation: fix what breaks. Expected: nothing new should break, since the Python-side contract is unchanged. If something does, treat it as a sign of an unintended coupling and address.
|
||||||
|
|
||||||
|
Push the commits to `origin/master`.
|
||||||
|
|
||||||
|
**Verification:** all three suites green; `git status` clean; commits visible on `git.sublimity.de/cronekorkn/left4me`.
|
||||||
|
|
||||||
|
**Commit:** none unless drift fixes are needed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rollback plan
|
||||||
|
|
||||||
|
If Task 3 surfaces a blocker (a hardening directive breaks a real-world script class, seccomp filter is too narrow, BindPaths semantics differ on the host's systemd version), roll back via `git revert` of Tasks 1+2 and redeploy. Git history preserves both the v1 and v2 helper. The Python side never changed, so reverting only the deploy artifacts is sufficient — no DB migration to undo, no template change to roll back.
|
||||||
|
|
@ -0,0 +1,89 @@
|
||||||
|
# L4D2 Script Sandbox v3 Implementation Plan
|
||||||
|
|
||||||
|
> **Approval status:** User-approved 2026-05-08; implemented and pushed in `7e66936`. This plan is recorded retrospectively for symmetry with the v1 / v2 plans.
|
||||||
|
|
||||||
|
**Goal:** Restrict the sandbox to public-internet egress per `docs/superpowers/specs/2026-05-08-l4d2-script-sandbox-v3-egress-filter.md`. Bind a static public-resolver `resolv.conf` into the sandbox.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Locked Decisions (see spec for rationale)
|
||||||
|
|
||||||
|
- `IPAddressDeny=` only; no `IPAddressAllow=any`.
|
||||||
|
- Explicit CIDRs (no `localhost` / `link-local` shorthand keywords — `systemd-run -p` parser rejects them).
|
||||||
|
- Static `nameserver 1.1.1.1` + `nameserver 8.8.8.8` in a sandbox-only resolv.conf.
|
||||||
|
- `AF_UNIX` left enabled.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Gap (at start of this iteration)
|
||||||
|
|
||||||
|
- `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` (v2) shares the host network namespace with no egress filter.
|
||||||
|
- The helper bind-mounts `/etc/resolv.conf` from the host into the sandbox (which points at private-IP DNS).
|
||||||
|
- `deploy/deploy-test-server.sh` does not install a sandbox-only resolv.conf.
|
||||||
|
- No deploy-artifact tests for `IPAddressDeny=` or for the resolv.conf shape.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: Add `IPAddressDeny=`, swap resolv.conf bind, ship the static file
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `deploy/files/etc/left4me/sandbox-resolv.conf` — two `nameserver` lines + a header comment.
|
||||||
|
- Modify: `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` — add `-p IPAddressDeny="..."` directive (11 explicit CIDRs); replace the `/etc/resolv.conf:/etc/resolv.conf` token in `BindReadOnlyPaths=` with `/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf`.
|
||||||
|
- Modify: `deploy/deploy-test-server.sh` — add an `install -m 0644 -o root -g root .../sandbox-resolv.conf /etc/left4me/sandbox-resolv.conf` line near the existing `host.env` install.
|
||||||
|
- Modify: `deploy/tests/test_deploy_artifacts.py` — extend `test_script_sandbox_helper_invokes_systemd_run_with_hardening` to assert each CIDR is present and that `IPAddressAllow=any` is **absent** (regression guard); update the BindReadOnlyPaths assertion to expect the sandbox-resolv.conf bind; add `test_sandbox_resolv_conf_exists` and `test_deploy_script_installs_sandbox_resolv_conf`.
|
||||||
|
|
||||||
|
Test plan (RED-first not used here; the work was driven by smoke-test feedback against a live host):
|
||||||
|
|
||||||
|
1. `test_script_sandbox_helper_invokes_systemd_run_with_hardening` — `IPAddressDeny=` present with all 11 CIDRs; no `IPAddressAllow=any`; resolv.conf bind path is `/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf`.
|
||||||
|
2. `test_sandbox_resolv_conf_exists` — file present, ≥2 nameservers, all in non-private space.
|
||||||
|
3. `test_deploy_script_installs_sandbox_resolv_conf` — deploy script references both source path under `deploy/files/etc/left4me/sandbox-resolv.conf` and target path `/etc/left4me/sandbox-resolv.conf`.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```
|
||||||
|
sh -n deploy/deploy-test-server.sh
|
||||||
|
bash -n deploy/files/usr/local/libexec/left4me/left4me-script-sandbox
|
||||||
|
python3 -m pytest deploy/tests/ -q
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit:** `feat(deploy): restrict script-sandbox egress to public internet only`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: Deploy + smoke-test on `ckn@10.0.4.128`
|
||||||
|
|
||||||
|
**Files:** none.
|
||||||
|
|
||||||
|
Run `deploy/deploy-test-server.sh ckn@10.0.4.128`. Then on the host, invoke the helper with a probe script that opens TCP connections to:
|
||||||
|
|
||||||
|
- `1.1.1.1:443` — must connect (public)
|
||||||
|
- `127.0.0.1:8000` — must block (web app on loopback)
|
||||||
|
- `127.0.0.1:22` — must block (sshd on loopback)
|
||||||
|
- `10.0.4.128:22` — must block (host's external SSH on private LAN)
|
||||||
|
- `10.0.0.1:53` — must block (LAN DNS resolver)
|
||||||
|
|
||||||
|
Plus `curl -m 5 https://steamcommunity.com/` end-to-end (DNS + HTTPS) → 200.
|
||||||
|
|
||||||
|
Inside the sandbox, `cat /etc/resolv.conf` must show the two public resolvers.
|
||||||
|
|
||||||
|
If any of the localhost / private targets connects, the deny is being silently overridden — see spec §Locked Decisions point 1.
|
||||||
|
|
||||||
|
**Commit:** none — operational verification.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Lessons surfaced during execution
|
||||||
|
|
||||||
|
These belong in the spec but are repeated here as the "things the next person should not have to rediscover":
|
||||||
|
|
||||||
|
- **`IPAddressAllow=any` silently overrides every `IPAddressDeny=` rule** on this systemd 257 / kernel 6.12 combo, despite documentation stating "more specific rule wins". The negative test (`IPAddressAllow=any not in text`) locks this in.
|
||||||
|
- **systemd-run's `-p` parser rejects the `localhost` / `link-local` / `multicast` shorthand keywords** even though they parse fine in unit files. Use explicit CIDRs.
|
||||||
|
- **`/var/lib/left4me/.../left4me.db` is mode 0644 by default** — writing this file from the web app left it world-readable. Tightening to 0640 root:left4me happens in v2's deploy-script change; v3 does not re-touch it.
|
||||||
|
- **bpftool ships separately on Debian.** It's not needed for runtime, but `apt-get install bpftool` is useful for inspecting `sd_fw_egress` attach state when debugging filter behaviour.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
`git revert 7e66936` and redeploy. The change is purely in deploy artifacts; no app code, no DB migration. Reverting reopens the previous v2 reachability.
|
||||||
|
|
@ -0,0 +1,353 @@
|
||||||
|
# L4D2 Global Map Overlays Design
|
||||||
|
|
||||||
|
**Goal:** Add two managed, system-wide map overlays, `l4d2center-maps` and `cedapug-maps`, populated from upstream map sources and refreshed daily through the existing job system.
|
||||||
|
|
||||||
|
**Approval status:** User-approved design direction. Implementation must not start until this spec is reviewed and an implementation plan is written.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
`left4me` already has typed overlays, a builder registry, global overlays through `Overlay.user_id = NULL`, and queued overlay build jobs. Steam Workshop overlays use a cache plus symlinks into `left4dead2/addons/`, and server initialization already runs overlay builders before calling `l4d2ctl initialize`.
|
||||||
|
|
||||||
|
Global map sources fit the same model. The host library remains unchanged: it receives overlay refs and mounts directories. The web app owns map-source fetching, cache management, reconciliation, and job logs.
|
||||||
|
|
||||||
|
The two upstream sources are:
|
||||||
|
|
||||||
|
- `https://l4d2center.com/maps/servers/index.csv`
|
||||||
|
- `https://cedapug.com/custom`
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
1. **One general operation.** Use `refresh_global_overlays`, not source-specific cron operations.
|
||||||
|
2. **Systemd owns time.** A systemd timer runs daily and invokes a Flask CLI command. The CLI only enqueues work; the existing worker performs downloads and writes logs.
|
||||||
|
3. **System jobs are nullable-owner jobs.** `jobs.user_id` becomes nullable. `NULL` means the job was created by the system. UI displays owner as `system`. Only admins can access system jobs.
|
||||||
|
4. **Managed global overlays are auto-seeded.** The app creates or repairs exactly one `l4d2center-maps` overlay and exactly one `cedapug-maps` overlay.
|
||||||
|
5. **Global overlays are normal system overlays for users.** `Overlay.user_id = NULL` makes them visible to every authenticated user and selectable in every user's blueprint editor.
|
||||||
|
6. **Managed types are not user-creatable.** Normal overlay creation does not offer `l4d2center_maps` or `cedapug_maps`. The seeder is the only code path that creates those types.
|
||||||
|
7. **Exact reconciliation.** Refresh makes each managed overlay match its upstream manifest. Removed upstream maps are removed from the managed overlay symlink set. Foreign files are left alone and logged.
|
||||||
|
8. **No initialize-time downloads.** `initialize_server()` may run builders to repair symlinks, but it must not fetch remote manifests or download large archives. Missing cache content fails clearly.
|
||||||
|
9. **Separate cache from Workshop.** Non-Steam global maps use `${LEFT4ME_ROOT}/global_overlay_cache`, not `${LEFT4ME_ROOT}/workshop_cache`.
|
||||||
|
10. **Source-specific parsing stays explicit.** Do not introduce a generic arbitrary HTTP source framework in this phase.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
The design extends the existing overlay-builder registry:
|
||||||
|
|
||||||
|
```python
|
||||||
|
BUILDERS = {
|
||||||
|
"external": ExternalBuilder(),
|
||||||
|
"workshop": WorkshopBuilder(),
|
||||||
|
"l4d2center_maps": GlobalMapOverlayBuilder(),
|
||||||
|
"cedapug_maps": GlobalMapOverlayBuilder(),
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Both global map overlay types share the same filesystem builder. Source-specific code lives in refresh services that know how to fetch and parse upstream manifests.
|
||||||
|
|
||||||
|
High-level flow:
|
||||||
|
|
||||||
|
```text
|
||||||
|
systemd timer
|
||||||
|
-> flask refresh-global-overlays
|
||||||
|
-> ensure_global_overlays()
|
||||||
|
-> enqueue refresh_global_overlays job (coalesced)
|
||||||
|
-> worker fetches manifests
|
||||||
|
-> worker downloads/extracts cache files
|
||||||
|
-> worker records desired VPK files
|
||||||
|
-> worker rebuilds overlay symlinks directly
|
||||||
|
```
|
||||||
|
|
||||||
|
Auto-seeded overlay rows use fixed names, managed types, `user_id = NULL`, and web-generated paths:
|
||||||
|
|
||||||
|
```text
|
||||||
|
name=l4d2center-maps, type=l4d2center_maps, user_id=NULL, path=str(id)
|
||||||
|
name=cedapug-maps, type=cedapug_maps, user_id=NULL, path=str(id)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### `jobs`
|
||||||
|
|
||||||
|
Change `jobs.user_id` from required to nullable.
|
||||||
|
|
||||||
|
`NULL` means a system-created job. Authorization rules become:
|
||||||
|
|
||||||
|
- Admins can view, stream, and cancel every job, including system jobs.
|
||||||
|
- Non-admins can access only jobs where `job.user_id == current_user.id`.
|
||||||
|
- System jobs are not visible to non-admins through direct job URLs.
|
||||||
|
|
||||||
|
Job list/detail pages use outer joins to `users` and render missing owners as `system`.
|
||||||
|
|
||||||
|
### `global_overlay_sources`
|
||||||
|
|
||||||
|
One row per managed global source overlay:
|
||||||
|
|
||||||
|
```text
|
||||||
|
id INTEGER PRIMARY KEY
|
||||||
|
overlay_id INTEGER NOT NULL UNIQUE REFERENCES overlays(id) ON DELETE CASCADE
|
||||||
|
source_key VARCHAR(64) NOT NULL UNIQUE -- l4d2center-maps | cedapug-maps
|
||||||
|
source_type VARCHAR(32) NOT NULL -- l4d2center_csv | cedapug_custom_page
|
||||||
|
source_url TEXT NOT NULL
|
||||||
|
last_manifest_hash VARCHAR(64) NOT NULL DEFAULT ''
|
||||||
|
last_refreshed_at DATETIME NULL
|
||||||
|
last_error TEXT NOT NULL DEFAULT ''
|
||||||
|
created_at DATETIME NOT NULL
|
||||||
|
updated_at DATETIME NOT NULL
|
||||||
|
```
|
||||||
|
|
||||||
|
`source_key` is stable and used by the seeder to repair missing rows.
|
||||||
|
|
||||||
|
### `global_overlay_items`
|
||||||
|
|
||||||
|
One row per manifest item belonging to a global overlay source:
|
||||||
|
|
||||||
|
```text
|
||||||
|
id INTEGER PRIMARY KEY
|
||||||
|
source_id INTEGER NOT NULL REFERENCES global_overlay_sources(id) ON DELETE CASCADE
|
||||||
|
item_key VARCHAR(255) NOT NULL -- stable per source
|
||||||
|
display_name VARCHAR(255) NOT NULL DEFAULT ''
|
||||||
|
download_url TEXT NOT NULL
|
||||||
|
expected_vpk_name VARCHAR(255) NOT NULL DEFAULT ''
|
||||||
|
expected_size BIGINT NULL
|
||||||
|
expected_md5 VARCHAR(32) NOT NULL DEFAULT ''
|
||||||
|
etag VARCHAR(255) NOT NULL DEFAULT ''
|
||||||
|
last_modified VARCHAR(255) NOT NULL DEFAULT ''
|
||||||
|
content_length BIGINT NULL
|
||||||
|
last_downloaded_at DATETIME NULL
|
||||||
|
last_error TEXT NOT NULL DEFAULT ''
|
||||||
|
created_at DATETIME NOT NULL
|
||||||
|
updated_at DATETIME NOT NULL
|
||||||
|
UNIQUE(source_id, item_key)
|
||||||
|
```
|
||||||
|
|
||||||
|
For `l4d2center`, `item_key` and `expected_vpk_name` come from the CSV `Name` column, and `expected_size` / `expected_md5` come from the CSV.
|
||||||
|
|
||||||
|
For `cedapug`, `item_key` is the direct download URL path basename, normalized without query parameters. CEDAPUG does not publish checksums in the observed page, so integrity uses HTTP metadata when available and archive extraction checks.
|
||||||
|
|
||||||
|
### `global_overlay_item_files`
|
||||||
|
|
||||||
|
One row per extracted VPK file that should appear in an overlay:
|
||||||
|
|
||||||
|
```text
|
||||||
|
id INTEGER PRIMARY KEY
|
||||||
|
item_id INTEGER NOT NULL REFERENCES global_overlay_items(id) ON DELETE CASCADE
|
||||||
|
vpk_name VARCHAR(255) NOT NULL
|
||||||
|
cache_path TEXT NOT NULL -- relative path under global_overlay_cache
|
||||||
|
size BIGINT NOT NULL
|
||||||
|
md5 VARCHAR(32) NOT NULL DEFAULT ''
|
||||||
|
created_at DATETIME NOT NULL
|
||||||
|
updated_at DATETIME NOT NULL
|
||||||
|
UNIQUE(item_id, vpk_name)
|
||||||
|
```
|
||||||
|
|
||||||
|
This extra file table handles archives that contain more than one `.vpk` without overloading the item row.
|
||||||
|
|
||||||
|
## Filesystem Layout
|
||||||
|
|
||||||
|
Use a cache separate from Steam Workshop:
|
||||||
|
|
||||||
|
```text
|
||||||
|
${LEFT4ME_ROOT}/
|
||||||
|
global_overlay_cache/
|
||||||
|
l4d2center-maps/
|
||||||
|
archives/
|
||||||
|
vpks/
|
||||||
|
cedapug-maps/
|
||||||
|
archives/
|
||||||
|
vpks/
|
||||||
|
overlays/
|
||||||
|
{overlay_id}/
|
||||||
|
left4dead2/addons/
|
||||||
|
*.vpk -> absolute symlink to global_overlay_cache/.../vpks/*.vpk
|
||||||
|
```
|
||||||
|
|
||||||
|
Cache file writes are atomic: download to `*.partial`, extract to a temporary directory, verify, then `os.replace()` final VPK files.
|
||||||
|
|
||||||
|
Symlink targets are absolute, matching the existing Workshop overlay design.
|
||||||
|
|
||||||
|
## Source Parsing
|
||||||
|
|
||||||
|
### L4D2Center
|
||||||
|
|
||||||
|
Fetch `https://l4d2center.com/maps/servers/index.csv` with a normal HTTP timeout.
|
||||||
|
|
||||||
|
The CSV is semicolon-delimited and contains:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Name;Size;md5;Download link
|
||||||
|
```
|
||||||
|
|
||||||
|
Each item produces:
|
||||||
|
|
||||||
|
- `item_key = Name`
|
||||||
|
- `expected_vpk_name = Name`
|
||||||
|
- `expected_size = Size`
|
||||||
|
- `expected_md5 = md5`
|
||||||
|
- `download_url = Download link`
|
||||||
|
|
||||||
|
Downloads are `.7z` archives. Extraction uses a Python 7z implementation such as `py7zr` so tests do not depend on a system `7z` binary. After extraction, the expected VPK file must exist and match both size and md5. A mismatch fails that item and leaves the prior cached file in place.
|
||||||
|
|
||||||
|
### CEDAPUG
|
||||||
|
|
||||||
|
Fetch `https://cedapug.com/custom` and parse the embedded `renderCustomMapDownloads([...])` data.
|
||||||
|
|
||||||
|
Only direct download links are managed in v1:
|
||||||
|
|
||||||
|
- Relative links like `/maps/FatalFreight.zip` are converted to absolute `https://cedapug.com/maps/FatalFreight.zip`.
|
||||||
|
- External `http` links are logged and skipped in v1.
|
||||||
|
- Entries without a download link are built-in campaigns and skipped.
|
||||||
|
|
||||||
|
Downloads are `.zip` archives extracted with Python's standard `zipfile`. Every `.vpk` in the archive becomes a managed output file for that item. If no `.vpk` is present, the item fails and the prior cached files remain in place.
|
||||||
|
|
||||||
|
Because CEDAPUG does not publish checksums in the observed page, refresh detects changes using `ETag`, `Last-Modified`, `Content-Length`, and local extracted file metadata when available. A manual refresh can force revalidation by clearing item metadata in a later maintenance path; no force-refresh UI is included in this design.
|
||||||
|
|
||||||
|
## Refresh Job
|
||||||
|
|
||||||
|
`refresh_global_overlays` is a global worker operation.
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
1. Ensure both managed global overlays and source rows exist.
|
||||||
|
2. Fetch both manifests.
|
||||||
|
3. Upsert manifest items.
|
||||||
|
4. Mark items absent from the manifest as no longer desired by deleting their item rows; cascading deletes remove their file rows.
|
||||||
|
5. Download and extract new or changed items.
|
||||||
|
6. Keep prior cache files when an item download or verification fails, but record `last_error`.
|
||||||
|
7. Rebuild symlinks for changed sources directly through the same builder interface used by `build_overlay`.
|
||||||
|
8. Emit clear job logs: manifest counts, downloads, skips, removals, verification failures, and build summaries.
|
||||||
|
|
||||||
|
`refresh_global_overlays` does not enqueue child `build_overlay` jobs. Direct builder invocation keeps the overlay in sync before the refresh job releases its global mutex, so a server job cannot start against updated cache metadata but stale overlay symlinks.
|
||||||
|
|
||||||
|
Coalescing:
|
||||||
|
|
||||||
|
- If a `refresh_global_overlays` job is queued or running, CLI/admin requests return the existing job instead of inserting a duplicate.
|
||||||
|
|
||||||
|
## Builder Reconciliation
|
||||||
|
|
||||||
|
`GlobalMapOverlayBuilder` reads desired file rows for the overlay's source and reconciles only symlinks it manages.
|
||||||
|
|
||||||
|
Managed symlink rule:
|
||||||
|
|
||||||
|
- A symlink in `left4dead2/addons/` is managed if its resolved target is under `${LEFT4ME_ROOT}/global_overlay_cache/{source_key}/vpks/`.
|
||||||
|
- Managed symlinks absent from desired files are removed.
|
||||||
|
- Desired files missing from cache are skipped and logged as errors.
|
||||||
|
- Non-symlink files and symlinks outside the source cache are left untouched and logged as foreign entries.
|
||||||
|
|
||||||
|
This mirrors `WorkshopBuilder` behavior and keeps manual files safe.
|
||||||
|
|
||||||
|
## Scheduler Rules
|
||||||
|
|
||||||
|
`refresh_global_overlays` joins the existing global mutex group.
|
||||||
|
|
||||||
|
It must not run concurrently with:
|
||||||
|
|
||||||
|
- `install`
|
||||||
|
- `refresh_workshop_items`
|
||||||
|
- any `build_overlay`
|
||||||
|
- any server job (`initialize`, `start`, `stop`, `delete`)
|
||||||
|
|
||||||
|
No server or overlay job may start while `refresh_global_overlays` is running.
|
||||||
|
|
||||||
|
This conservative rule is acceptable because daily map refreshes are rare and large downloads should not race runtime changes.
|
||||||
|
|
||||||
|
## CLI And Systemd Timer
|
||||||
|
|
||||||
|
Add Flask CLI command:
|
||||||
|
|
||||||
|
```text
|
||||||
|
flask refresh-global-overlays
|
||||||
|
```
|
||||||
|
|
||||||
|
The command:
|
||||||
|
|
||||||
|
- Loads app config and DB.
|
||||||
|
- Ensures global overlays exist.
|
||||||
|
- Enqueues or returns the existing `refresh_global_overlays` job.
|
||||||
|
- Prints the job id.
|
||||||
|
- Does not run downloads itself.
|
||||||
|
|
||||||
|
Add deployment units:
|
||||||
|
|
||||||
|
```text
|
||||||
|
left4me-refresh-global-overlays.service
|
||||||
|
left4me-refresh-global-overlays.timer
|
||||||
|
```
|
||||||
|
|
||||||
|
Service command:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/opt/left4me/.venv/bin/flask --app l4d2web.app:create_app refresh-global-overlays
|
||||||
|
```
|
||||||
|
|
||||||
|
Timer policy:
|
||||||
|
|
||||||
|
```text
|
||||||
|
OnCalendar=daily
|
||||||
|
Persistent=true
|
||||||
|
```
|
||||||
|
|
||||||
|
The service runs as the `left4me` user with `/etc/left4me/host.env` and `/etc/left4me/web.env`, matching `left4me-web.service`.
|
||||||
|
|
||||||
|
## Permissions And UI
|
||||||
|
|
||||||
|
Overlay list behavior:
|
||||||
|
|
||||||
|
- Admins see all overlays, including managed global map overlays.
|
||||||
|
- Non-admin users see system overlays and their own private workshop overlays.
|
||||||
|
- Managed global overlays appear in blueprint overlay selection for every user.
|
||||||
|
|
||||||
|
Creation behavior:
|
||||||
|
|
||||||
|
- Non-admin users can create only user-creatable types, currently `workshop`.
|
||||||
|
- Admins can create normal admin-creatable types, currently `external` and `workshop`.
|
||||||
|
- No user-facing create form offers `l4d2center_maps` or `cedapug_maps`.
|
||||||
|
- Auto-seeding is the only creation path for managed global map overlay types.
|
||||||
|
|
||||||
|
Admin controls:
|
||||||
|
|
||||||
|
- Add a manual "Refresh global overlays" action in the admin area.
|
||||||
|
- The action enqueues the same coalesced `refresh_global_overlays` job as the timer.
|
||||||
|
- Managed overlay detail pages show source type, source URL, last refresh time, last error, item count, and latest related jobs.
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
- Manifest fetch failure fails the job if no source can be processed. If one source succeeds and one fails, the job should still finish failed with partial-success logs and preserve prior content for the failed source.
|
||||||
|
- Per-item download failures do not abort sibling items.
|
||||||
|
- Verification failures keep prior cached files and record `last_error` on the item.
|
||||||
|
- Extraction rejects path traversal entries and ignores non-VPK files.
|
||||||
|
- Unsupported CEDAPUG external links are skipped with a warning.
|
||||||
|
- Initialize-time checks fail if desired global map files are missing from cache, naming the overlay and missing VPK names.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
Test coverage should include:
|
||||||
|
|
||||||
|
- Auto-seeding creates exactly one source overlay per source and repairs missing source rows.
|
||||||
|
- `jobs.user_id` nullable behavior, outer joins, and `system` display.
|
||||||
|
- Non-admins cannot access system jobs directly.
|
||||||
|
- CLI coalesces queued/running `refresh_global_overlays` jobs.
|
||||||
|
- Scheduler truth table for the new global operation.
|
||||||
|
- L4D2Center CSV parser with semicolon-delimited fixture data.
|
||||||
|
- CEDAPUG embedded JavaScript parser with fixture HTML.
|
||||||
|
- L4D2Center download/extract verifies VPK size and md5.
|
||||||
|
- CEDAPUG download/extract records every VPK in a zip archive.
|
||||||
|
- Reconcile removes obsolete managed symlinks and leaves foreign files alone.
|
||||||
|
- Overlay create UI rejects managed singleton types.
|
||||||
|
- Blueprint overlay selection includes managed global overlays for all users.
|
||||||
|
- Deployment tests cover the service and timer artifacts.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- User-created global map source overlays.
|
||||||
|
- Arbitrary configurable HTTP manifest sources.
|
||||||
|
- Force-refresh UI for CEDAPUG items.
|
||||||
|
- Cache garbage collection for unreferenced archive files.
|
||||||
|
- Client-side map download UX.
|
||||||
|
- Steam Workshop links discovered on the CEDAPUG page; those are skipped rather than imported into workshop overlays.
|
||||||
|
- Host-library awareness of managed overlay types.
|
||||||
|
|
||||||
|
## Implementation Boundaries
|
||||||
|
|
||||||
|
- `l4d2host` remains unchanged.
|
||||||
|
- The web app continues to call host operations only through `l4d2ctl`.
|
||||||
|
- Existing blueprint semantics remain unchanged: overlays are live-linked, ordered, and first overlay has highest precedence.
|
||||||
|
- Existing workshop overlay behavior remains unchanged except scheduler interactions with the new global operation.
|
||||||
|
|
@ -0,0 +1,226 @@
|
||||||
|
# L4D2 Workshop Overlays Design
|
||||||
|
|
||||||
|
**Goal:** Let users add Steam Workshop content (.vpk addons and maps) to L4D2 servers from the web UI. Workshop downloads run as a new typed overlay that fits the existing `Overlay` + `BlueprintOverlay` model, downloaded via the public Steam Web API and exposed through the existing fuse-overlayfs mount layer.
|
||||||
|
|
||||||
|
**Approval status:** User-approved design direction. Implementation proceeds in lockstep with the companion plan at `docs/superpowers/plans/2026-05-07-l4d2-workshop-overlays.md`.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
`left4me` users today add `.vpk` content to a server only by SFTP-ing files into a manually-prepared overlay directory or by maintaining shell scripts (`competitive_rework`, `workshop_maps`, `tickrate`, etc.) that wrap `curl`/`steamcmd`. The web app exposes overlay rows but offers no way for users to populate them.
|
||||||
|
|
||||||
|
This spec adds **workshop overlays**: a user-private overlay type that downloads `.vpk` files via the public `ISteamRemoteStorage` API and surfaces them through the existing mount layer. Users keep composing blueprints by stacking overlays — workshop overlays become another row alongside today's externally-managed ones.
|
||||||
|
|
||||||
|
This is the first *typed* overlay. The design adds a `type` column and a builder-registry so future overlay types (tarball, inline, manual upload) plug in without schema churn or workflow changes.
|
||||||
|
|
||||||
|
Steam Workshop content for L4D2 (consumer_app_id 550) is downloadable via two anonymous-POST endpoints with no Steam Web API key required: `GetCollectionDetails` resolves a collection ID to its child item IDs, and `GetPublishedFileDetails` returns per-item metadata including a public `file_url` for the `.vpk`. This is the same API the user's existing `steam-workshop-download` script uses.
|
||||||
|
|
||||||
|
L4D2-specific player-side pain points (sv_consistency / RestrictAddons configuration gotchas, the inability to push workshop content via `sv_downloadurl`) are documented in **Out of scope** and tracked as separate follow-ups. This spec stays strictly on workshop content acquisition.
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
1. **Typed overlays.** `Overlay.type` joins `external` (existing rows; admin-managed; no-op builder) and `workshop` (new). Future types — tarball, inline, manual upload — slot in via the same builder registry without schema churn.
|
||||||
|
2. **No JSON `source_config` blob.** Per-type structured data lives in proper relational tables. JSON is reserved for genuinely opaque diagnostic payloads.
|
||||||
|
3. **Central deduplicated `WorkshopItem` registry** keyed on `steam_id`. Cache lives at `/var/lib/left4me/workshop_cache/{steam_id}.vpk`. Multiple overlays referencing the same Steam item share the same cache file.
|
||||||
|
4. **Symlinks, not copies.** Overlay directories contain `left4dead2/addons/{steam_id}.vpk` symlinks pointing into the cache. Both the cache file and the symlink are named by `{steam_id}` only — no Steam filename in any on-disk path, so Steam can rename the upstream `.vpk` without breaking lookup.
|
||||||
|
5. **Many-to-many association is pure** (no `enabled` flag). Toggle a workshop item by removing or re-adding the association. The shared cache makes this cheap.
|
||||||
|
6. **Collections are atomic UI bulk-imports.** Pasting a collection URL/ID resolves member items and creates N item associations. The DB never tracks "this came from a collection." Re-importing a collection is idempotent on existing items and additive for new ones.
|
||||||
|
7. **Single global admin "Refresh all workshop items" button.** One Steam metadata batch call, then re-download items whose `time_updated` advanced. No per-item, per-overlay, or scheduled refresh in v1.
|
||||||
|
8. **No cache GC in v1.** Cache grows monotonically. Reference-counted cleanup is a follow-up.
|
||||||
|
9. **Globality is independent of overlay type.** `Overlay.user_id` is the scope (NULL = system-wide, set = private to that user). v1 defaults newly-created workshop overlays to private and leaves existing external overlays as system-wide. A future "publish/share" button will let owners toggle `user_id` without changing type.
|
||||||
|
10. **One unified "Create overlay" UI button.** Modal has a type radio (External | Workshop). No path field — the web app generates the path for every new overlay.
|
||||||
|
11. **Strict scope.** v1 ships only the workshop type. L4D2 server-config gotchas, client-subscription helpers, other recipe types — all deferred to follow-up specs.
|
||||||
|
12. **`consumer_app_id == 550` validation** at every Steam API response at fetch/add time; non-L4D2 items are rejected and never reach the row. The value is a fixed precondition, not data.
|
||||||
|
13. **Input field accepts numeric ID, full Workshop URL, or a multi-line batch** of either. Pasting `123456` and pasting `steamcommunity.com/sharedfiles/filedetails/?id=123456` produce the same result; pasting many of either at once works too.
|
||||||
|
14. **Web-managed overlay paths.** All new overlays (any type) get `path = str(overlay_id)` at insert time. The user never picks a path. Existing legacy external overlay rows keep their current path values; migrating them to the ID-based scheme is a follow-up. `Overlay.id` uses SQLite `AUTOINCREMENT` so deleted IDs are never reused.
|
||||||
|
15. **Auto-rebuild on item change.** Adding or removing items from a workshop overlay automatically enqueues a `build_overlay` job. The "Rebuild" button on the detail page is for manual recovery only. New build jobs for an overlay coalesce with any pending one for the same overlay (don't queue duplicates).
|
||||||
|
16. **HTTPS** for all Steam Web API calls. The reference downloader uses HTTP; we don't.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Overlay row (type=workshop)
|
||||||
|
└─refs─▶ overlay_workshop_items
|
||||||
|
└─▶ WorkshopItem (global, by steam_id)
|
||||||
|
│
|
||||||
|
▼ download (Steam GetPublishedFileDetails + HTTP GET)
|
||||||
|
workshop_cache/{steam_id}.vpk
|
||||||
|
▲
|
||||||
|
overlay_dir/left4dead2/addons/{steam_id}.vpk ─symlink─┘
|
||||||
|
```
|
||||||
|
|
||||||
|
Build dispatch via a registry:
|
||||||
|
|
||||||
|
```python
|
||||||
|
BUILDERS = {"external": ExternalBuilder(), "workshop": WorkshopBuilder()}
|
||||||
|
|
||||||
|
def build_overlay(overlay_id):
|
||||||
|
overlay = db.get(Overlay, overlay_id)
|
||||||
|
BUILDERS[overlay.type].build(overlay, on_stdout, on_stderr, should_cancel)
|
||||||
|
```
|
||||||
|
|
||||||
|
`ExternalBuilder` is a no-op for legacy admin-managed dirs. `WorkshopBuilder` performs an idempotent diff-apply of `addons/` symlinks against the current associations. Future types add their own builders without changing the dispatcher, the mount layer, or the blueprint editor.
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### `Overlay` (extended)
|
||||||
|
|
||||||
|
```
|
||||||
|
id INTEGER PK AUTOINCREMENT
|
||||||
|
name VARCHAR(255) NOT NULL
|
||||||
|
path VARCHAR(255) NOT NULL -- new overlays: str(id); legacy externals: existing values
|
||||||
|
type VARCHAR(16) NOT NULL -- 'external' | 'workshop' (extensible)
|
||||||
|
user_id INTEGER NULL REFERENCES users(id) -- NULL = system-wide
|
||||||
|
created_at, updated_at
|
||||||
|
|
||||||
|
UNIQUE INDEX on (name) WHERE user_id IS NULL -- system overlays globally unique by name
|
||||||
|
UNIQUE INDEX on (name, user_id) WHERE user_id IS NOT NULL -- per-user namespace
|
||||||
|
INDEX on (type, user_id)
|
||||||
|
```
|
||||||
|
|
||||||
|
Two partial unique indexes are required because a naive composite `UNIQUE(name, user_id)` doesn't constrain externals — SQLite treats NULL as distinct in unique constraints, so two externals could share a name. Partial indexes preserve the prior global-uniqueness invariant for system rows.
|
||||||
|
|
||||||
|
### `WorkshopItem` (new)
|
||||||
|
|
||||||
|
```
|
||||||
|
id INTEGER PK
|
||||||
|
steam_id VARCHAR(20) NOT NULL UNIQUE -- 64-bit, store as text
|
||||||
|
title VARCHAR(255) NOT NULL DEFAULT ''
|
||||||
|
filename VARCHAR(255) NOT NULL DEFAULT '' -- upstream Steam filename, display only
|
||||||
|
file_url TEXT NOT NULL DEFAULT ''
|
||||||
|
file_size BIGINT NOT NULL DEFAULT 0
|
||||||
|
time_updated INTEGER NOT NULL DEFAULT 0 -- Steam epoch
|
||||||
|
preview_url TEXT NOT NULL DEFAULT '' -- thumbnail URL hot-linked from Steam
|
||||||
|
last_downloaded_at DATETIME NULL
|
||||||
|
last_error TEXT NOT NULL DEFAULT ''
|
||||||
|
created_at, updated_at
|
||||||
|
```
|
||||||
|
|
||||||
|
`consumer_app_id` is **not** stored. It's validated at fetch time and the row never exists for non-L4D2 items.
|
||||||
|
|
||||||
|
### `overlay_workshop_items` (new, pure association)
|
||||||
|
|
||||||
|
```
|
||||||
|
id INTEGER PK
|
||||||
|
overlay_id INTEGER NOT NULL REFERENCES overlays(id) ON DELETE CASCADE
|
||||||
|
workshop_item_id INTEGER NOT NULL REFERENCES workshop_items(id) ON DELETE RESTRICT
|
||||||
|
UNIQUE (overlay_id, workshop_item_id)
|
||||||
|
INDEX (workshop_item_id) -- reverse lookup for refresh
|
||||||
|
```
|
||||||
|
|
||||||
|
No `enabled` column — toggle is remove/add, which is cheap because the cache survives.
|
||||||
|
|
||||||
|
### `Job` (extended)
|
||||||
|
|
||||||
|
Add `overlay_id INTEGER NULL REFERENCES overlays(id)` for `build_overlay` jobs.
|
||||||
|
|
||||||
|
## Filesystem Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
/var/lib/left4me/
|
||||||
|
overlays/
|
||||||
|
{overlay_id}/ # flat — same shape for every type
|
||||||
|
left4dead2/addons/
|
||||||
|
{steam_id}.vpk -> /var/lib/left4me/workshop_cache/{steam_id}.vpk
|
||||||
|
workshop_cache/
|
||||||
|
{steam_id}.vpk # one file per Steam item
|
||||||
|
```
|
||||||
|
|
||||||
|
- Every new overlay (workshop, future tarball/inline/manual) lives at `overlays/{overlay_id}/`. Legacy external overlays keep their pre-migration paths (e.g. `overlays/standard/`).
|
||||||
|
- `workshop_cache/` is created during deploy provisioning, not lazily — avoids races between concurrent first downloads.
|
||||||
|
- Web user owns both trees (mode 0755). Host user (`l4d2ctl`) needs read on both. If web and host are different users, they share a group.
|
||||||
|
- Symlink targets are absolute. Relative targets resolve in the merged-mount namespace and break across the host/web boundary.
|
||||||
|
- The builder never creates a dangling symlink. If a `WorkshopItem` lacks a cache file at build time, the builder logs a warning and skips it — fuse-overlayfs surfaces broken links to L4D2 as opaque addon-scan failures.
|
||||||
|
|
||||||
|
## UI
|
||||||
|
|
||||||
|
A single "Create overlay" button on `/overlays` opens a modal with type radio (External | Workshop) and a name field. No path field. The web app generates `path = str(overlay_id)` after insert.
|
||||||
|
|
||||||
|
Workshop overlay detail page (`/overlays/{id}` when `type='workshop'`) shows:
|
||||||
|
|
||||||
|
- A multi-line input plus a radio (Items | Collection). Pasting one or many IDs/URLs adds them in order; pasting a collection ID resolves its members.
|
||||||
|
- An item table with: thumbnail (`preview_url`), `steam_id` linking to Steam, title, filename, last-updated, size, last-error if any, Remove.
|
||||||
|
- A manual "Rebuild" button (for recovery only — every add/remove auto-enqueues a coalesced `build_overlay` job).
|
||||||
|
- Status indicator pulled from the latest related `Job` row.
|
||||||
|
|
||||||
|
External overlay detail page is unchanged in shape: read-only path display, name edit (admin only). The "External" type retains the existing admin-only SFTP-to-disk workflow until a future "manual upload" type replaces it.
|
||||||
|
|
||||||
|
The blueprint editor is unchanged in structure. Workshop overlays appear alongside externals in the user's overlay picker; ordering and stacking semantics are identical.
|
||||||
|
|
||||||
|
Admin section gets one new control: "Refresh all workshop items" button on the admin landing or workshop subsection. Pressing it enqueues a single `refresh_workshop_items` job.
|
||||||
|
|
||||||
|
### Routes
|
||||||
|
|
||||||
|
| Method | Path | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| GET | `/overlays` | List with Type column, filtered by user permissions |
|
||||||
|
| POST | `/overlays` | Create; reads `type` and `name` only |
|
||||||
|
| GET | `/overlays/{id}` | Type-aware detail page |
|
||||||
|
| POST | `/overlays/{id}/items` | Add items or collection; auto-enqueues coalesced `build_overlay` |
|
||||||
|
| POST | `/overlays/{id}/items/{item_id}/delete` | Remove association; auto-enqueues coalesced `build_overlay` |
|
||||||
|
| POST | `/overlays/{id}/build` | Manual rebuild (recovery) |
|
||||||
|
| POST | `/admin/workshop/refresh` | Admin only; enqueue `refresh_workshop_items` |
|
||||||
|
|
||||||
|
HTMX usage stays minimal: only the add-item form and per-row delete swap a fragment. Everything else is full-page POST/redirect/GET.
|
||||||
|
|
||||||
|
## Job Operations
|
||||||
|
|
||||||
|
Two new operations join the existing job worker:
|
||||||
|
|
||||||
|
- **`build_overlay(overlay_id)`** — `Job.overlay_id` is set; `server_id` is NULL. Dispatches to `BUILDERS[overlay.type].build(...)`. Cancellation between filesystem operations.
|
||||||
|
- **`refresh_workshop_items()`** — admin-only. Both `server_id` and `overlay_id` are NULL. Phases: fetch all metadata in one batched call, download items where `time_updated` advanced, enqueue (coalesced) `build_overlay` for affected overlays. v1 doesn't wait on child builds; the admin sees them in the jobs list.
|
||||||
|
|
||||||
|
### Scheduler rules
|
||||||
|
|
||||||
|
- `install` and `refresh_workshop_items` are mutually exclusive with each other, with all `build_overlay`s, and with all server jobs.
|
||||||
|
- `build_overlay(overlay_id=N)` blocks if `install_running`, `refresh_running`, or another build for the same `overlay_id` is running. Builds for *different* overlays may run concurrently.
|
||||||
|
- Server start/init blocks if `refresh_running` or any `build_overlay` for an overlay referenced by the server's blueprint is running.
|
||||||
|
|
||||||
|
Coalescing: a new `build_overlay` for an overlay that already has a queued (not-yet-running) build returns the existing job instead of inserting a new row.
|
||||||
|
|
||||||
|
`initialize_server` synchronously calls each overlay's builder before writing the spec for `l4d2ctl initialize`. If a workshop overlay references uncached items (no file in `workshop_cache/`), `initialize_server` fails fast with a clear error naming the missing IDs and pointing the user at the overlay page. It never silently mounts a partial overlay.
|
||||||
|
|
||||||
|
## Permissions
|
||||||
|
|
||||||
|
- **External overlays**: admin-only create/edit. Visible to all authenticated users (system-wide).
|
||||||
|
- **Workshop overlays**: any logged-in user can create. Owner or admin can edit and delete. Visible to the owner and admins.
|
||||||
|
- **Admin refresh**: admin-only.
|
||||||
|
|
||||||
|
The `Overlay` listing query for non-admins becomes: `type='external' OR user_id=current_user.id`.
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
- **Broken symlinks across host/web boundary** — mitigated by absolute targets, build-time pre-check skipping uncached items, and `deploy/` documenting permission requirements.
|
||||||
|
- **Initialize against uncached items** — would silently mount overlays missing maps. Mitigated by `initialize_server`'s fail-fast check; tested.
|
||||||
|
- **Steam API rate limits** — refresh of 100 items is one metadata POST plus 100 downloads at 8-way parallelism. No retry/backoff in v1; 429s surface verbatim in the job log.
|
||||||
|
- **Partial failure during refresh** — each item is independent; per-item errors land on the row. Re-running refresh retries failures.
|
||||||
|
- **Concurrent same-ID adds** — `WorkshopItem.steam_id` unique handles cache dedup. `(overlay_id, workshop_item_id)` unique catches double-association; the route returns "already in overlay" rather than 500.
|
||||||
|
- **Build coalescing missed** — would enqueue dozens of redundant builds during multi-item adds. Mitigated by the `enqueue_build_overlay` helper; tested.
|
||||||
|
- **Worker concurrency rule miss** — the truth-table test in `test_job_worker.py` is the only way to trust the new scheduler logic; written before dispatch.
|
||||||
|
- **DB/disk drift** — a stray directory left by a prior failed delete could shadow a fresh overlay. Mitigated by `AUTOINCREMENT` (no ID reuse) and `os.makedirs(exist_ok=False)` (loud failure on collision).
|
||||||
|
- **Partial unique gap on SQLite** — naive composite `UNIQUE(name, user_id)` doesn't constrain externals because NULL is distinct. Mitigated by two partial unique indexes; tested explicitly.
|
||||||
|
- **Cache growth without GC** — accepted v1 trade-off.
|
||||||
|
- **Item removed from Steam** — refresh marks `result != 1`; row keeps last good cache file; UI surfaces error string. Operator decides removal.
|
||||||
|
- **L4D2 containerized run** — symlink absolute targets break if the server runs in a different mount namespace. Re-evaluate when containerization comes up.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
These came up in research and dialog but stay out of v1:
|
||||||
|
|
||||||
|
- **Publish / share button on overlays.** Lets owners flip `Overlay.user_id` between their own ID and NULL without changing type. The schema already supports it; only the UI is deferred.
|
||||||
|
- **Migrate legacy external overlay paths to the ID-based scheme.** Existing external rows keep their pre-migration paths in v1; a follow-up migration moves the directories on disk and updates the rows.
|
||||||
|
- **Switch from fuse-overlayfs to kernel overlayfs via a privileged helper.** Matches the existing systemd / steam-install sudoers helper pattern under `/usr/local/libexec/left4me/`. Workshop overlays would work identically under either mount engine — symlinks resolve through normal VFS in both.
|
||||||
|
- **`sv_consistency` / `addonconfig.cfg RestrictAddons` auto-handling.** When a workshop overlay attaches to a blueprint, surface a banner with a one-click fix. Most-cited L4D2 player pain.
|
||||||
|
- **Shareable Steam Workshop collection link for clients.** Server cannot push workshop content via `sv_downloadurl`; clients must subscribe themselves. A panel-generated collection makes that one click for players. Requires Steam OAuth.
|
||||||
|
- **Other overlay types.** `tarball` (covers the old `competitive_rework` GitHub-tarball recipe), `inline` (covers `tickrate`'s inline `server.cfg`), `manual` (file manager / upload, replaces the admin-SFTP external workflow). All slot in via the builder registry without schema churn.
|
||||||
|
- **Cache GC.** Reference-counted delete or admin "Clear unreferenced" page.
|
||||||
|
- **Per-item / per-overlay / scheduled refresh.** v1 has one global admin button; revisit if users want finer control.
|
||||||
|
- **Update-aware server restart UX.** Notify users when a running server's overlay content has been refreshed underneath it.
|
||||||
|
|
||||||
|
## Implementation Boundaries
|
||||||
|
|
||||||
|
- The host library contract is unchanged. Workshop content arrives in overlay directories the same way externals do today; `l4d2host` doesn't know overlays have types.
|
||||||
|
- The job-execution model is preserved: same workers, same logs, same cancel callbacks. Only the operations table grows.
|
||||||
|
- The blueprint privacy model and desired-vs-actual server state model are unchanged.
|
||||||
|
- No new frontend dependencies. Vendored HTMX + custom CSS + small inline JS.
|
||||||
|
- No new Steam Web API key required; both endpoints used accept anonymous POSTs.
|
||||||
|
- The companion implementation plan governs task ordering and verification commands. Implementation must not start without explicit user approval per that plan's gate.
|
||||||
|
|
@ -0,0 +1,80 @@
|
||||||
|
# Kernel Overlayfs Helper Design
|
||||||
|
|
||||||
|
**Goal:** Replace the per-instance `fuse-overlayfs` mount with kernel-native overlayfs invoked through a privileged sudo helper that mounts in PID 1's mount namespace. Restores host-namespace visibility of the merged overlay so gameserver units (`left4me-server@%i.service`) can `chdir` into it at unshare time.
|
||||||
|
|
||||||
|
**Approval status:** User-approved design direction. Implementation proceeds in lockstep with the companion plan at `docs/superpowers/plans/2026-05-08-kernel-overlayfs-helper.md`.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
**Symptom.** After redeploys, starting a gameserver leaves the systemd unit in `activating (auto-restart)` with `status=200/CHDIR — Changing to the requested working directory failed: No such file or directory`. Investigation showed:
|
||||||
|
|
||||||
|
- `fuse-overlayfs` running as `left4me` user mounts in `left4me-web.service`'s mount namespace.
|
||||||
|
- `ProtectSystem=full` + `ReadWritePaths=/var/lib/left4me` forces `PrivateMounts=yes` on the unit (`systemd-analyze security` confirms).
|
||||||
|
- The unit's bind of `/var/lib/left4me` shows `shared:471 master:1` in `/proc/<pid>/mountinfo` — slave-receive-only — so mounts created beneath it never propagate back to host.
|
||||||
|
- `MountFlags=shared` (added in commit `1968684` to fix this) sets only the unit's *root* propagation; it does not override the slave-direction propagation that `ProtectSystem`/`ReadWritePaths` apply to their bind mounts. The gameserver unit, on unshare, inherits *host* mounts and sees nothing at the merged path → CHDIR fails.
|
||||||
|
|
||||||
|
The system *appeared* to work for ~1d8h before this investigation because the prior fuse daemon happened to land in the host namespace via some transient state. The mechanism documented in `1968684` does not reliably work on systemd 257 with this hardening shape.
|
||||||
|
|
||||||
|
**Out-of-scope item now in scope.** The 2026-05-07 workshop-overlays spec already lists this transition at line 211: *"Switch from fuse-overlayfs to kernel overlayfs via a privileged helper. Matches the existing systemd / steam-install sudoers helper pattern under `/usr/local/libexec/left4me/`."* The mount-propagation bug is the trigger to do it now.
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
1. **Privileged helper does the mount.** New `left4me-overlay` script under `/usr/local/libexec/left4me/`, invoked via `sudo -n`. Mirrors the existing `left4me-systemctl` and `left4me-journalctl` pattern. The helper enters PID 1's mount namespace via `nsenter --mount=/proc/1/ns/mnt` and then calls `/bin/mount -t overlay …` or `/bin/umount`. Result: all overlay mounts live in the host namespace, visible to gameserver units.
|
||||||
|
2. **Kernel-native overlayfs, not fuse.** Once a privileged helper exists, fuse-overlayfs's rootless-mount-via-setuid-`fusermount3` advantage disappears. Kernel overlayfs is faster, has no long-running daemon, simpler unmount, and one fewer runtime dep.
|
||||||
|
3. **Helper is Python, not shell.** Path canonicalization, env-file parsing, and lowerdir prefix-allowlist validation are too brittle in shell. Uses system `/usr/bin/python3` (never the venv) and stdlib only. Owned by root, mode 0755.
|
||||||
|
4. **Verbs are `mount` and `umount`.** Matches the kernel/userspace utility names; reduces cognitive friction over `unmount`.
|
||||||
|
5. **Helper takes only the instance name as input.** It reads `${LEFT4ME_ROOT:-/var/lib/left4me}/instances/<name>/instance.env` for `L4D2_LOWERDIRS=` and computes `upper`/`work`/`merged` from the runtime root. Equivalent in security to taking lowerdirs as args (the user already controls instance.env), and produces a one-line audit trail in `journalctl _COMM=sudo`.
|
||||||
|
6. **Strict path validation in the helper.**
|
||||||
|
- Instance name matches `^[a-z0-9][a-z0-9_-]{0,63}$` (mirrors `validate_instance_name` in `l4d2host/paths.py`).
|
||||||
|
- Each lowerdir from `L4D2_LOWERDIRS` is `os.path.realpath`'d and must resolve under one of an allowlist: `installation/`, `overlays/`, `global_overlay_cache/`, `workshop_cache/`. Empty entries and traversals are rejected.
|
||||||
|
- `upper`/`work`/`merged` must resolve exactly to `runtime/<name>/{upper,work,merged}`.
|
||||||
|
- Lowerdir count ≤ 500 (kernel overlayfs hard cap; was 64 before kernel 5.2).
|
||||||
|
7. **Whiteout-format guard.** `fuse-overlayfs` running as non-root uses `user.fuseoverlayfs.*` xattrs for whiteouts and opaque dirs, which kernel overlayfs ignores entirely. Before mounting, the helper walks `upperdir` once and refuses if any such xattr is present. Defensive; catches a stale fuse-era upperdir that wasn't wiped during migration.
|
||||||
|
8. **One-time migration: wipe existing `upper/` and `work/`.** Deploy script runs a gated migration (sentinel file `/var/lib/left4me/.kernel-overlay-migrated`) that stops gameservers, stops web service, unmounts any stale fuse/overlay mounts, recreates empty `upper`/`work` dirs for every instance. Players' in-place edits to merged content are sacrificed; v1 accepts this for a test deployment.
|
||||||
|
9. **Sudoers verb constraints.** `left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-overlay mount *, /usr/local/libexec/left4me/left4me-overlay umount *`. Defense in depth (real validation lives in the helper); makes `sudo -l` output self-documenting.
|
||||||
|
10. **Wire the existing `OverlayMounter` ABC through.** `start_instance`/`stop_instance`/`delete_instance` today bypass the abstraction at `l4d2host/fs/base.py`. The new `KernelOverlayFSMounter` replaces the unused `FuseOverlayFSMounter` AND becomes the only path through `instances.py`. `FuseOverlayFSMounter` and the `fuse_overlayfs.py` module are deleted.
|
||||||
|
11. **Double-mount guard in `start_instance`.** Kernel mounts persist when the web worker dies (unlike fuse daemons, which die with their cgroup). `start_instance` checks `os.path.ismount(merged)` and refuses with a clear error rather than double-mounting.
|
||||||
|
12. **Hardening cleanup on `left4me-web.service`.** Drop `MountFlags=shared` (no longer the mechanism). Restore `PrivateTmp=true` (was dropped in commit `593611e` for fuse propagation that did not work). Keep `NoNewPrivileges` unset (sudo still requires setuid). Update the comment block to reflect the new model.
|
||||||
|
13. **AGENTS.md contracts unchanged.** The host library's CLI surface (`install`, `initialize`, `start`, `stop`, `delete`, `status`, `logs`) is unchanged. The web app continues to drive operations via `l4d2ctl`. The fuse-overlayfs implementation detail was never part of the public contract.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
left4me-web.service (hardened, private mount namespace)
|
||||||
|
│
|
||||||
|
│ start_instance(name=…)
|
||||||
|
▼
|
||||||
|
l4d2host.instances.start_instance
|
||||||
|
│
|
||||||
|
│ KernelOverlayFSMounter().mount(merged=…)
|
||||||
|
▼
|
||||||
|
sudo -n /usr/local/libexec/left4me/left4me-overlay mount <name>
|
||||||
|
│ • validate name (regex)
|
||||||
|
│ • parse instance.env → L4D2_LOWERDIRS
|
||||||
|
│ • realpath each lowerdir, prefix-allowlist check
|
||||||
|
│ • compute upper/work/merged under runtime/<name>/
|
||||||
|
│ • walk upperdir, refuse if any user.fuseoverlayfs.* xattr
|
||||||
|
▼
|
||||||
|
nsenter --mount=/proc/1/ns/mnt -- \
|
||||||
|
/bin/mount -t overlay overlay \
|
||||||
|
-o "lowerdir=…,upperdir=…,workdir=…" \
|
||||||
|
/var/lib/left4me/runtime/<name>/merged
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
host mount namespace now has the overlay; gameserver unit, on
|
||||||
|
unshare, inherits it and CHDIRs into …/merged/left4dead2 successfully.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Operational Notes
|
||||||
|
|
||||||
|
- **Migration ordering on the test box (test-server, …).** The deploy script must, in order: (1) stop all `left4me-server@*.service`, (2) stop `left4me-web.service` (kills any lingering fuse-overlayfs daemons by reaping their cgroup), (3) `findmnt` + force-unmount any leftover fuse/overlay mounts under `/var/lib/left4me/runtime/`, (4) wipe and recreate `upper`/`work` for every instance, (5) deploy + start the new code. The sentinel file `/var/lib/left4me/.kernel-overlay-migrated` gates reruns.
|
||||||
|
- **Filesystem.** `/var/lib/left4me` is btrfs on the test box. Kernel overlayfs on btrfs is supported on kernel ≥ 5.10; the box is on 6.12 — fine. AppArmor ships enabled on Debian Trixie; verify no overlay-related denials in `journalctl -k` after first start.
|
||||||
|
- **Concurrency.** Two threads racing on `start_instance` for the same name is a latent issue unaffected by this change. The double-mount guard partly mitigates: the loser hits the existing mount and errors cleanly.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- **Replace `sudo` with `AmbientCapabilities=CAP_SYS_ADMIN`** on a dedicated helper unit. Broader blast radius than the wrapper-script approach.
|
||||||
|
- **A `systemd-mount` per-instance mount unit.** Considered as the alternative architectural fix but adds more moving parts than the helper-script approach. The helper matches the established privileged-helper pattern in this codebase.
|
||||||
|
- **Re-enable `NoNewPrivileges` on `left4me-web.service`.** Requires removing sudo; not feasible while the helper invocation pattern stays.
|
||||||
|
- **Multi-process job-worker-claim safety.** The `_claim_lock` in `l4d2host/services/job_worker.py:131-138` is process-local; correctness depends on `--workers 1`. This change doesn't touch it.
|
||||||
|
- **Replicating the migration on production deployments.** v1 covers only the test-server deployment shape.
|
||||||
323
docs/superpowers/specs/2026-05-08-l4d2-script-overlays-design.md
Normal file
323
docs/superpowers/specs/2026-05-08-l4d2-script-overlays-design.md
Normal file
|
|
@ -0,0 +1,323 @@
|
||||||
|
# L4D2 Script Overlays Design
|
||||||
|
|
||||||
|
**Goal:** Add a single new overlay type, `script`, that lets users author arbitrary build recipes as bash and runs them inside a `bubblewrap` + `systemd-run --scope` sandbox. The new type subsumes the existing `l4d2center_maps` and `cedapug_maps` managed-globals overlay types, both of which are removed in the same change. After this work the overlay type list is exactly `workshop` (unchanged) and `script` (new).
|
||||||
|
|
||||||
|
**Approval status:** User-approved design direction. Implementation proceeds in lockstep with the companion plan at `docs/superpowers/plans/2026-05-08-l4d2-script-overlays.md`.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
`left4me` users today have two ways to add content to a server: workshop overlays (rich UI for Steam Workshop items via `WorkshopBuilder`) and a pair of managed global-map overlay types (`l4d2center_maps`, `cedapug_maps`) with bespoke parsers, per-item DB rows, ETag-based change detection, and a daily refresh timer. They cannot author arbitrary build recipes.
|
||||||
|
|
||||||
|
The user's previous setup at `ckn-bw/bundles/left4dead2/files/scripts/overlays/` expressed every recipe as a small bash file: `competitive_rework` (GitHub tarball download), `tickrate` (inline `server.cfg` + addon DLL fetch), `standard` (workshop items + admin-list write), `workshop_maps` (workshop collection import), `l4d2center_maps` (CSV-driven map sync). All five fit naturally into a single "run a sandboxed bash script that populates the overlay dir" model.
|
||||||
|
|
||||||
|
The two managed global-map types in the current codebase are over-engineered for what they do — each is essentially "fetch a manifest, download archives, extract VPKs, place in `addons/`." Folding them into the new `script` type eliminates three database tables, two source-parser modules, the `GlobalMapOverlayBuilder`, the `py7zr` dependency, the global-overlay cache root, and the managed-singleton machinery, while letting an admin paste the equivalent shell code (which the user already wrote years ago) into a normal admin-owned, system-wide script overlay.
|
||||||
|
|
||||||
|
The trust model for the sandbox is "semi-public deployment, registered users." The threat surface is one user reading another user's overlay, the application DB, or arbitrary host secrets, plus runaway scripts exhausting disk/CPU/RAM. Network access is *not* restricted — scripts must be able to download from arbitrary URLs (GitHub, l4d2center, Steam CDN). Sandbox boundaries are namespace-based (mount, PID, IPC, UTS, cgroup), not command-allowlist-based; binary-allowlist sandboxing of bash is theatre because of `eval` and `exec`.
|
||||||
|
|
||||||
|
The test deploy DB is wiped as part of rollout; no data migration is performed. Existing user blueprints that reference `l4d2center_maps` or `cedapug_maps` overlay rows do not survive the change in the test environment.
|
||||||
|
|
||||||
|
A scheduled-refresh feature (the daily timer that today drives the global-map types) is intentionally **out of scope for this iteration**. The two existing systemd units and the `flask refresh-global-overlays` CLI command are deleted with no replacement. Refresh is reintroduced in a later iteration designed against concrete needs.
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
1. **Single new overlay type: `script`.** Replaces both managed-globals types. Final type list: `workshop` + `script`. No `tarball`/`inline`/`manual` types — all of those collapse into `script` (with UI templates as a future ergonomics improvement).
|
||||||
|
2. **`Overlay.script` is a DB `TEXT` column** holding the raw bash. No file storage, no revision history in v1. Empty string for `workshop` rows.
|
||||||
|
3. **Build idempotency contract: script runs against the existing overlay dir.** No automatic wipe between builds. Users write `test -f … || curl …`-style guards if they want bandwidth efficiency. A manual "Wipe overlay" button on the detail page resets the dir to empty.
|
||||||
|
4. **No left4me-aware helpers in the sandbox.** The script sees pure bash plus whatever's in `/usr` (RO bind-mount of the host). Workshop items are not exposed via a helper — users wanting workshop content create a `workshop`-type overlay, which has its own first-class UX (thumbnails, collection paste, dedup cache, refresh).
|
||||||
|
5. **Sandbox engine: `bubblewrap` (`bwrap`) inside `systemd-run --scope --collect`.** `systemd-run` provides cgroup v2 limits + walltime kill via `RuntimeMaxSec`; `bwrap` provides the namespace isolation. Both are stable, well-audited, in-tree on Debian.
|
||||||
|
6. **Resource limits (system-wide, not per-overlay):** 1 hour walltime (`RuntimeMaxSec=3600`), 4 GB RAM (`MemoryMax=4G`, `MemorySwapMax=0`), 512 tasks, 200% CPU quota, post-build 20 GB disk cap on `du -sb` of the overlay dir.
|
||||||
|
7. **Network: host-shared.** No `--unshare-net`. Scripts have full outbound. Egress filtering is not in v1; the sandbox prevents reading internal state but does not prevent talking to internal IPs. Acceptable for the current trust model.
|
||||||
|
8. **No auto-seeding of "default" overlays.** Admin manually creates the equivalents of the old `l4d2center-maps`/`cedapug-maps` post-deploy by pasting the bash. The deploy script does not insert overlay rows.
|
||||||
|
9. **Daily/scheduled refresh: out of scope for this iteration.** No `auto_refresh` flag, no timer, no CLI command. Manual rebuild via the detail-page button is the only build trigger after this change.
|
||||||
|
10. **Permissions mirror workshop overlays.** Any logged-in user can create a private (`user_id = me`) script overlay. Admin can create system-wide (`user_id = NULL`). Owner or admin can edit/delete.
|
||||||
|
11. **Failure semantics via `Overlay.last_build_status`** (`'' | 'ok' | 'failed'`). Drives a "rebuild required" badge on the list and detail pages. Server initialization does **not** auto-block on `failed` (matches workshop's current behavior).
|
||||||
|
12. **Wipe is just another sandbox invocation.** The wipe endpoint runs the literal script `find /overlay -mindepth 1 -delete` through the same `left4me-script-sandbox` helper. No second helper, no privilege/UID puzzle (files are owned by `l4d2-sandbox`, who runs the wipe). After a successful wipe, `last_build_status` is reset to `''`. Wipe does **not** auto-enqueue a rebuild — the user decides.
|
||||||
|
13. **Privileged helper: `/usr/local/libexec/left4me/left4me-script-sandbox`.** Same pattern as the existing `left4me-overlay`, `left4me-systemctl`, `left4me-journalctl` helpers. Bash, owned root, mode 0755. The web user invokes it via `sudo -n` per a sudoers fragment. Root is needed to set up the namespaces; bwrap drops to the unprivileged `l4d2-sandbox` UID immediately.
|
||||||
|
14. **Dedicated sandbox UID `l4d2-sandbox`** (system user, `/usr/sbin/nologin`, no home). Owns nothing on the host outside what bwrap binds in. UID-drop happens inside the bwrap invocation via `--uid`/`--gid`.
|
||||||
|
15. **Strict argument validation in the helper.** Overlay id matches `^[0-9]+$`; overlay dir must exist under `/var/lib/left4me/overlays/`; script path must exist. Defense in depth — the real authorization check lives in the web app.
|
||||||
|
16. **Streaming I/O via the existing `run_with_streamed_output` helper.** Same plumbing `WorkshopBuilder` already uses for `steamcmd`/`curl` invocations. No new SSE/log path.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```text
|
||||||
|
Overlay row (type=script, script=TEXT, last_build_status)
|
||||||
|
│
|
||||||
|
▼ build_overlay(overlay_id) job
|
||||||
|
│
|
||||||
|
▼ BUILDERS["script"].build(overlay, on_stdout, on_stderr, should_cancel)
|
||||||
|
│
|
||||||
|
▼ ScriptBuilder writes overlay.script → tmpfile, then:
|
||||||
|
│ sudo -n /usr/local/libexec/left4me/left4me-script-sandbox <id> <tmpfile>
|
||||||
|
│
|
||||||
|
▼ Helper validates args, then exec()s:
|
||||||
|
│ systemd-run --scope --collect
|
||||||
|
│ -p MemoryMax=4G -p MemorySwapMax=0
|
||||||
|
│ -p TasksMax=512 -p CPUQuota=200%
|
||||||
|
│ -p RuntimeMaxSec=3600
|
||||||
|
│ -- bwrap [namespace flags...] /bin/bash /script.sh
|
||||||
|
│
|
||||||
|
▼ Inside the sandbox the script sees:
|
||||||
|
│ /overlay ← /var/lib/left4me/overlays/{id} RW (the build target)
|
||||||
|
│ /tmp,/run ← fresh tmpfs RW (ephemeral)
|
||||||
|
│ /usr,/lib,/lib64,/etc/{ssl,resolv.conf,nsswitch} RO (host-curated)
|
||||||
|
│ /proc,/dev ← fresh
|
||||||
|
│ network ← shared with host
|
||||||
|
│ UID/GID ← l4d2-sandbox (no_new_privs implicit in bwrap)
|
||||||
|
│
|
||||||
|
▼ stdout/stderr → run_with_streamed_output → existing job-log SSE stream
|
||||||
|
▼ After exit:
|
||||||
|
│ exit 0 ∧ du -sb /overlay ≤ 20 GB → last_build_status='ok'
|
||||||
|
│ any other outcome → last_build_status='failed'
|
||||||
|
```
|
||||||
|
|
||||||
|
The host library (`l4d2host`) is unchanged. The `KernelOverlayFSMounter` already mounts whatever's at `overlays/{id}/` regardless of how it got there. The Job model and worker model are essentially unchanged — `script` is just another overlay type for the same `build_overlay` operation that today supports `workshop`.
|
||||||
|
|
||||||
|
```python
|
||||||
|
BUILDERS = {
|
||||||
|
"workshop": WorkshopBuilder(),
|
||||||
|
"script": ScriptBuilder(),
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### `Overlay` (modified)
|
||||||
|
|
||||||
|
```text
|
||||||
|
id INTEGER PK AUTOINCREMENT
|
||||||
|
name VARCHAR(255) NOT NULL
|
||||||
|
path VARCHAR(255) NOT NULL -- str(id) for new rows
|
||||||
|
type VARCHAR(16) NOT NULL -- 'workshop' | 'script'
|
||||||
|
user_id INTEGER NULL REFERENCES users(id) -- NULL = system-wide
|
||||||
|
|
||||||
|
script TEXT NOT NULL DEFAULT '' -- new; meaningful for type='script'
|
||||||
|
last_build_status VARCHAR(16) NOT NULL DEFAULT '' -- new; '' | 'ok' | 'failed'
|
||||||
|
|
||||||
|
created_at, updated_at
|
||||||
|
|
||||||
|
UNIQUE INDEX on (name) WHERE user_id IS NULL
|
||||||
|
UNIQUE INDEX on (name, user_id) WHERE user_id IS NOT NULL
|
||||||
|
INDEX on (type, user_id)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Tables removed
|
||||||
|
|
||||||
|
- `global_overlay_item_files`
|
||||||
|
- `global_overlay_items`
|
||||||
|
- `global_overlay_sources`
|
||||||
|
|
||||||
|
Drop order matters for the SQLite migration: drop `_item_files` first (FK to `_items`), then `_items` (FK to `_sources`), then `_sources` (FK to `overlays`).
|
||||||
|
|
||||||
|
### Unchanged
|
||||||
|
|
||||||
|
`WorkshopItem`, `overlay_workshop_items`, `Job` (including `Job.overlay_id` and nullable `Job.user_id`), `Server`, `Blueprint`, etc.
|
||||||
|
|
||||||
|
## Filesystem Layout
|
||||||
|
|
||||||
|
```text
|
||||||
|
${LEFT4ME_ROOT}/
|
||||||
|
overlays/
|
||||||
|
{overlay_id}/ # script writes here; mounted by host
|
||||||
|
left4dead2/... # whatever the script produces
|
||||||
|
workshop_cache/{steam_id}.vpk # workshop type only — unchanged
|
||||||
|
|
||||||
|
# removed:
|
||||||
|
# global_overlay_cache/ # was used by managed-globals types
|
||||||
|
```
|
||||||
|
|
||||||
|
Single tree per overlay. No per-overlay scratch cache (the chosen idempotency model is "script runs against existing dir," so any caching the user wants lives inside the overlay dir and is preserved between builds).
|
||||||
|
|
||||||
|
The sandbox bind-mounts `${LEFT4ME_ROOT}/overlays/{id}/` to `/overlay` (RW). Nothing else under `${LEFT4ME_ROOT}` is visible inside the sandbox.
|
||||||
|
|
||||||
|
## Sandbox
|
||||||
|
|
||||||
|
### Helper script
|
||||||
|
|
||||||
|
`deploy/files/usr/local/libexec/left4me/left4me-script-sandbox`, mode 0755, owned root:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# args: <overlay_id> <script_path>
|
||||||
|
set -euo pipefail
|
||||||
|
[[ $# -eq 2 ]] || { echo "usage: $0 <overlay_id> <script>" >&2; exit 64; }
|
||||||
|
OVERLAY_ID=$1; SCRIPT=$2
|
||||||
|
[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]] || { echo "bad overlay id" >&2; exit 64; }
|
||||||
|
OVERLAY_DIR=/var/lib/left4me/overlays/$OVERLAY_ID
|
||||||
|
[[ -d $OVERLAY_DIR ]] || { echo "no overlay dir" >&2; exit 65; }
|
||||||
|
[[ -f $SCRIPT ]] || { echo "no script" >&2; exit 65; }
|
||||||
|
|
||||||
|
SBX_UID=$(id -u l4d2-sandbox); SBX_GID=$(id -g l4d2-sandbox)
|
||||||
|
|
||||||
|
exec systemd-run --quiet --scope --collect \
|
||||||
|
-p MemoryMax=4G -p MemorySwapMax=0 -p TasksMax=512 \
|
||||||
|
-p CPUQuota=200% -p RuntimeMaxSec=3600 \
|
||||||
|
-- bwrap \
|
||||||
|
--die-with-parent --new-session \
|
||||||
|
--unshare-pid --unshare-ipc --unshare-uts --unshare-cgroup \
|
||||||
|
--uid "$SBX_UID" --gid "$SBX_GID" \
|
||||||
|
--proc /proc --dev /dev --tmpfs /tmp --tmpfs /run \
|
||||||
|
--ro-bind /usr /usr --ro-bind /lib /lib --ro-bind /lib64 /lib64 \
|
||||||
|
--symlink usr/bin /bin --symlink usr/sbin /sbin \
|
||||||
|
--ro-bind /etc/resolv.conf /etc/resolv.conf \
|
||||||
|
--ro-bind /etc/ssl /etc/ssl \
|
||||||
|
--ro-bind /etc/ca-certificates /etc/ca-certificates \
|
||||||
|
--ro-bind /etc/nsswitch.conf /etc/nsswitch.conf \
|
||||||
|
--bind "$OVERLAY_DIR" /overlay \
|
||||||
|
--chdir /overlay \
|
||||||
|
--setenv HOME /tmp --setenv PATH /usr/bin:/usr/sbin \
|
||||||
|
--setenv OVERLAY /overlay \
|
||||||
|
--ro-bind "$SCRIPT" /script.sh \
|
||||||
|
/bin/bash /script.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Network is *not* unshared (no `--unshare-net`); the sandbox shares the host network namespace. Every transient unit is visible via `systemctl list-units --type=scope` while running and journaled afterward (`journalctl --user-unit=run-…scope` or system journal depending on invocation).
|
||||||
|
|
||||||
|
### Sudoers fragment
|
||||||
|
|
||||||
|
Append to `deploy/files/etc/sudoers.d/left4me`:
|
||||||
|
|
||||||
|
```
|
||||||
|
left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-script-sandbox
|
||||||
|
```
|
||||||
|
|
||||||
|
### System user
|
||||||
|
|
||||||
|
Provisioned in `deploy/deploy-test-server.sh`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
useradd --system --no-create-home --shell /usr/sbin/nologin l4d2-sandbox
|
||||||
|
apt-get install -y bubblewrap
|
||||||
|
```
|
||||||
|
|
||||||
|
## Build Lifecycle
|
||||||
|
|
||||||
|
`ScriptBuilder` lives in `l4d2web/services/overlay_builders.py` next to `WorkshopBuilder`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class ScriptBuilder:
|
||||||
|
def build(self, overlay, *, on_stdout, on_stderr, should_cancel):
|
||||||
|
with tempfile.NamedTemporaryFile("w", suffix=".sh", delete=False) as f:
|
||||||
|
f.write(overlay.script or "")
|
||||||
|
script_path = f.name
|
||||||
|
try:
|
||||||
|
cmd = [
|
||||||
|
"sudo", "-n",
|
||||||
|
"/usr/local/libexec/left4me/left4me-script-sandbox",
|
||||||
|
str(overlay.id), script_path,
|
||||||
|
]
|
||||||
|
run_with_streamed_output(cmd, on_stdout, on_stderr, should_cancel)
|
||||||
|
self._enforce_disk_budget(overlay.id, on_stderr)
|
||||||
|
finally:
|
||||||
|
os.unlink(script_path)
|
||||||
|
|
||||||
|
def _enforce_disk_budget(self, overlay_id, on_stderr):
|
||||||
|
size = subprocess.check_output(["du", "-sb", overlay_path(overlay_id)])
|
||||||
|
if int(size.split()[0]) > 20 * 1024**3:
|
||||||
|
on_stderr("overlay exceeded 20 GB disk cap")
|
||||||
|
raise BuildError("disk-cap-exceeded")
|
||||||
|
```
|
||||||
|
|
||||||
|
`run_with_streamed_output` is the existing helper used by `WorkshopBuilder` for `steamcmd`/`curl` invocations. The `should_cancel` callback fires `kill -TERM` on the sudo-`systemd-run` process tree; cgroup-collect tears down the whole scope on exit.
|
||||||
|
|
||||||
|
The job worker's existing job-completion path writes `Overlay.last_build_status = 'ok'` on success and `'failed'` on any non-zero exit / `BuildError` / cancel. This is a single column update inside the existing transaction; no new infrastructure.
|
||||||
|
|
||||||
|
## UI
|
||||||
|
|
||||||
|
### Create modal (`templates/overlays.html`)
|
||||||
|
|
||||||
|
The existing modal grows one option in the type radio: `Workshop | Script`. Name field unchanged. After insert, the web app generates `path = str(overlay_id)` for new rows (existing pattern).
|
||||||
|
|
||||||
|
### Detail page when `type='script'` (`templates/overlay_detail.html`)
|
||||||
|
|
||||||
|
- Plain styled `<textarea>` for `overlay.script` with a Save button → `POST /overlays/{id}/script`. No CodeMirror dependency in v1 (out of scope; keep frontend dep-light).
|
||||||
|
- "Rebuild" button → `POST /overlays/{id}/build`. Existing pattern from workshop overlays.
|
||||||
|
- "Wipe overlay" button (red, confirm-modal) → `POST /overlays/{id}/wipe`.
|
||||||
|
- `last_build_status` indicator badge: empty / "ok" / "failed".
|
||||||
|
- Live build log via existing SSE plumbing on the related Job row.
|
||||||
|
|
||||||
|
### Detail page when `type='workshop'`: unchanged.
|
||||||
|
|
||||||
|
### Sections removed
|
||||||
|
|
||||||
|
The global-source detail block (`overlay_detail.html` lines 34–46) is deleted along with the managed-globals subsystem.
|
||||||
|
|
||||||
|
## Routes
|
||||||
|
|
||||||
|
`l4d2web/routes/overlay_routes.py` adds:
|
||||||
|
|
||||||
|
| Method | Path | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| POST | `/overlays/{id}/script` | Update `script` text. Auto-enqueue coalesced `build_overlay` job. |
|
||||||
|
| POST | `/overlays/{id}/wipe` | Invoke `left4me-script-sandbox` with the literal script `find /overlay -mindepth 1 -delete`. Owner/admin only. Refuses if a `build_overlay` for this overlay is running. After success, set `last_build_status=''`. Does not auto-enqueue a rebuild. |
|
||||||
|
| POST | `/overlays/{id}/build` | Manual rebuild — same pattern as today's workshop overlay manual rebuild. |
|
||||||
|
|
||||||
|
Existing `POST /overlays` accepts `type=script` and an optional initial `script` body.
|
||||||
|
|
||||||
|
## Permissions
|
||||||
|
|
||||||
|
| Action | Who |
|
||||||
|
|---|---|
|
||||||
|
| Create script overlay (private, `user_id = me`) | Any authenticated user |
|
||||||
|
| Create script overlay (system-wide, `user_id = NULL`) | Admin |
|
||||||
|
| Edit (script body, name) | Owner or admin |
|
||||||
|
| Wipe / Rebuild | Owner or admin |
|
||||||
|
| Delete | Owner or admin |
|
||||||
|
| View | Owner, admin, or any user when `user_id IS NULL` |
|
||||||
|
|
||||||
|
These match the existing rules for workshop overlays.
|
||||||
|
|
||||||
|
## Job Worker / Scheduler
|
||||||
|
|
||||||
|
`services/job_worker.py` drops `"refresh_global_overlays"` from `GLOBAL_OPERATIONS` and removes the corresponding `refresh_global_overlays_running` and `blocked_servers_by_overlay` plumbing that exists only for the global-maps subsystem. The remaining mutex rules already cover:
|
||||||
|
|
||||||
|
- `build_overlay` per overlay (one running build per overlay).
|
||||||
|
- `install` and `refresh_workshop_items` as global mutexes.
|
||||||
|
- Server start/init blocks if any `build_overlay` for an overlay in the server's blueprint is running.
|
||||||
|
|
||||||
|
No new rules are needed for `script` — its build is mechanically identical to a `workshop` build from the scheduler's perspective.
|
||||||
|
|
||||||
|
## Daily Refresh — Removed
|
||||||
|
|
||||||
|
This iteration deletes the daily-refresh subsystem entirely:
|
||||||
|
|
||||||
|
- `deploy/files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer` and `.service` — deleted.
|
||||||
|
- `flask refresh-global-overlays` CLI command in `l4d2web/cli.py` — deleted.
|
||||||
|
- No replacement timer, no replacement CLI, no `auto_refresh` column on `Overlay`.
|
||||||
|
|
||||||
|
The only build trigger after this change is the user clicking Rebuild on the detail page (or the auto-enqueue when they Save the script body). A scheduled-refresh feature is reintroduced in a future iteration designed against concrete operational needs.
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
- **Sandbox escape via kernel bug.** `bwrap` has a strong track record but is not invulnerable. Mitigated by running as `l4d2-sandbox` (no privileged capabilities), no setuid binaries reachable, `no_new_privs` implicit. A successful escape would land in an unprivileged UID with no host secrets reachable.
|
||||||
|
- **Disk fill via runaway script.** A script that writes a 20 GB+ payload to `/overlay` succeeds inside the sandbox and only fails afterward at the post-build `du` check. The 20 GB lands on disk transiently. Mitigated by the kernel's per-cgroup IO accounting being unaware of file size (no good IO-time limit), accepting this as a v1 trade-off; a future improvement is overlay-dir-on-its-own-filesystem with a quota.
|
||||||
|
- **Network exfiltration.** Script can connect to anything outbound, including internal IPs. Acceptable for the current trust model (semi-public; users have credentials). Egress firewall is out of scope.
|
||||||
|
- **Build-mid-server-running.** The scheduler refuses `build_overlay` for an overlay attached to a starting/running server (existing rule, unchanged). Good. A user can still rebuild while a server using a *different* blueprint runs concurrently.
|
||||||
|
- **Wipe race with running build.** The wipe endpoint refuses if a `build_overlay` for the overlay is running. Without this check, a wipe could blow away files mid-script and produce undefined results.
|
||||||
|
- **Stale `last_build_status`.** A row inserted via direct DB write or restored from backup could carry an `'ok'` status that no longer reflects reality. Treated as cosmetic; users can rebuild to refresh.
|
||||||
|
- **Sudoers misconfig.** A typo in the sudoers fragment could grant `left4me` more than intended. Mitigated by deploy-artifact tests asserting the exact expected lines.
|
||||||
|
- **DB row deletion racing the sandbox.** A user deleting an overlay while its build runs would invalidate the bind-mount target. Mitigated by the existing scheduler rule that tracks running overlays; delete should refuse if a build is running. (Existing pattern for workshop overlays; reuse.)
|
||||||
|
- **Migration drops globals tables.** Acceptable for the test deploy. Production rollout would need a different migration story; this spec explicitly assumes test-deploy DB wipe.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- **Scheduled / daily refresh.** Intentionally removed in this iteration. Reintroduced later, designed against the use cases that emerge.
|
||||||
|
- **Per-overlay resource overrides.** All script overlays share the same 1 h / 4 GB / 20 GB envelope. If a real overlay needs more (l4d2center mirror at peak), revisit.
|
||||||
|
- **CodeMirror or other rich script editor.** Plain `<textarea>` in v1.
|
||||||
|
- **Egress allowlist / proxy.** No network restrictions on the sandbox in v1.
|
||||||
|
- **`$CACHE` scratch dir** persisted across builds. Users cache inside the overlay dir if they want; idempotency model is "script runs against existing dir."
|
||||||
|
- **Multi-tenant cgroup tree per user.** All sandboxes share the same cgroup-quota envelope.
|
||||||
|
- **Revision history on `script` column.** No `overlay_script_revisions` table; whatever's in the row is the current script.
|
||||||
|
- **Auto-seeding of l4d2center / cedapug equivalents.** Admin pastes the script post-deploy.
|
||||||
|
- **Migration that preserves existing global-map overlay rows.** Test deploy DB is wiped.
|
||||||
|
- **Container-per-build (podman / docker).** Heavier than `bwrap`; revisit only if multi-tenant escalates to "fully public sign-up."
|
||||||
|
- **left4me-aware helpers** (`workshop`, `download`, `extract`) inside the sandbox. Pure bash + host `/usr` only.
|
||||||
|
|
||||||
|
## Implementation Boundaries
|
||||||
|
|
||||||
|
- **`l4d2host` is unchanged.** The host library has no concept of overlay types and the mount layer (`KernelOverlayFSMounter`) doesn't care how the overlay dir got populated.
|
||||||
|
- **The `OverlayBuilder` Protocol is unchanged** — same `build(overlay, *, on_stdout, on_stderr, should_cancel)` signature. `ScriptBuilder` plugs into the existing registry.
|
||||||
|
- **The job worker model is unchanged.** Same operations, same logs, same SSE plumbing, same scheduler rules (minus the refresh_global_overlays entry).
|
||||||
|
- **No new application-level dependencies.** Vendored HTMX, no new Python packages. Two new system dependencies: `bubblewrap` apt package and the `l4d2-sandbox` system user.
|
||||||
|
- **No new config keys.** Same env files (`/etc/left4me/host.env`, `/etc/left4me/web.env`).
|
||||||
|
- **DB migration is destructive for global-maps overlay rows.** This is acceptable per the test-deploy assumption; a production-rollout follow-up would need to address it.
|
||||||
|
- The companion implementation plan governs task ordering and verification commands. Implementation must not start without explicit user approval per that plan's gate.
|
||||||
|
|
@ -0,0 +1,138 @@
|
||||||
|
# L4D2 Script Sandbox v2 — Systemd-Only
|
||||||
|
|
||||||
|
**Goal:** Replace the bwrap-based `left4me-script-sandbox` helper with one that uses `systemd-run` in **service-unit mode** alone. Drop `bubblewrap` as a system dependency. Gain capability bounding, seccomp filtering, kernel-tunable / -module / -log protection, address-family restriction, `LockPersonality`, `MemoryDenyWriteExecute`, and `RestrictSUIDSGID` — none of which the bwrap+systemd-run-scope composition could provide. Lose PID-namespace isolation (no `PrivatePID=` directive in systemd) — judged acceptable for the current trust model.
|
||||||
|
|
||||||
|
**Approval status:** User-approved 2026-05-08 after smoke testing on `ckn@10.0.4.128`.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The v1 sandbox (see `2026-05-08-l4d2-script-overlays-design.md`) layers `bubblewrap` for namespacing inside `systemd-run --scope` for cgroup limits. That works, but `--scope` units register an existing process tree and so cannot accept service-only directives like `NoNewPrivileges=`, `ProtectSystem=`, `SystemCallFilter=`, `CapabilityBoundingSet=`, etc. Smoke testing on the deployed host confirmed bwrap covers mount/PID/IPC/UTS namespacing well, but leaves capability bounding, seccomp, and kernel-surface protection unenforced.
|
||||||
|
|
||||||
|
A switch to `systemd-run` in default (transient service) mode unlocks the full hardening surface. Smoke testing of a v2 prototype against the deployed test host confirmed:
|
||||||
|
|
||||||
|
- Every isolation invariant the bwrap version provides (filesystem masking, UID drop, network reachability, `/overlay` RW bind, host-side `l4d2-sandbox` ownership, host secret hiding) is reproducible with systemd directives.
|
||||||
|
- All cgroup limits (`memory.max=4G`, `memory.swap.max=0`, `pids.max=512`, `cpu.max=200%`, `RuntimeMaxSec=3600`) apply identically.
|
||||||
|
- `MemoryError` fires at the 4 GB cap (cgroup-enforced).
|
||||||
|
- The wipe path (`find /overlay -mindepth 1 -delete`) succeeds.
|
||||||
|
- Hardening directives the v1 design couldn't express enforce real syscall blocks: `unshare(CLONE_NEWUSER)`, `mount(2)`, `personality(2)`, `bpf(2)`, `swapoff(2)`, `sysctl -w` are all blocked.
|
||||||
|
|
||||||
|
The single behavioral regression: host process IDs are visible via `/proc` and `ps -ef` because systemd has no `PrivatePID=` directive. Sending signals to those processes is still blocked by the kernel's UID-mismatch check (`l4d2-sandbox` cannot signal `root`-owned processes). Information disclosure is the only leak; signal capability is intact.
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
1. **Replace the helper body wholesale.** No `bwrap` invocation. `systemd-run` in service mode does both isolation and resource limits.
|
||||||
|
2. **Helper path, sudoers rule, ScriptBuilder API, and `l4d2-sandbox` UID are unchanged.** The Python side (`run_sandboxed_script`, route handlers, tests) does not change.
|
||||||
|
3. **`bubblewrap` apt dependency dropped from `deploy-test-server.sh`.**
|
||||||
|
4. **`left4me.db` file mode tightened to 0640 root:left4me at deploy time.** This is a host-hygiene fix that is independent of the sandbox change but was surfaced by smoke testing — without it, *any* host user (and, transitively, the sandbox) could read the application database.
|
||||||
|
5. **`TemporaryFileSystem=/var/lib` is required.** `ProtectSystem=strict` makes `/var/lib/left4me` read-only but visible; the only way to reliably hide its contents from the unit is to mask the parent with a tmpfs. The `BindPaths=…/overlays/{id}:/overlay` mount is unaffected because `/overlay` is at a different path.
|
||||||
|
6. **`PrivatePID=` is not configured.** systemd has no such directive. `ps -ef` from inside the sandbox shows host processes. The kernel's UID-based signal restriction blocks any actual interaction with them. Acceptable for the current trust model.
|
||||||
|
7. **Walltime kill remains `RuntimeMaxSec=3600`.** Same as v1.
|
||||||
|
8. **Network namespace remains shared with the host.** No `PrivateNetwork=`. Scripts must reach Steam / l4d2center / GitHub / etc.
|
||||||
|
9. **`SystemCallFilter=@system-service @network-io`** is the seccomp baseline. systemd's curated `@system-service` group is "everything a normal service does"; adding `@network-io` is explicit even though it overlaps. Build failures revealing missing syscall classes are surfaced via `journalctl` and addressed by widening the filter (`@process`, etc.) on demand.
|
||||||
|
10. **Single helper file replaces v1.** Not adding a `-v2` variant. The v1 implementation is removed in the same change.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```text
|
||||||
|
sudo helper
|
||||||
|
└─ systemd-run --service (default) --pipe --wait
|
||||||
|
(transient .service unit, full hardening directives)
|
||||||
|
└─ /bin/bash /script.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
systemd-run in service mode:
|
||||||
|
- Opens a transient service unit on the system bus.
|
||||||
|
- Applies all `-p` properties as the unit's exec context.
|
||||||
|
- Forks; the child sets up the unit's namespaces (mount, IPC, user), drops privileges to `User=l4d2-sandbox`, applies the seccomp filter, and `execve()`s `/bin/bash /script.sh`.
|
||||||
|
- `--pipe` connects the unit's stdin/stdout/stderr to the calling helper's stdio (so the existing `run_command` harness in `ScriptBuilder` continues to capture line-by-line).
|
||||||
|
- `--wait` blocks until the unit terminates and propagates the exit code.
|
||||||
|
- `--collect` removes the unit on exit even if it failed.
|
||||||
|
- The cgroup carries the resource limits; the systemd timer enforces `RuntimeMaxSec=3600`.
|
||||||
|
|
||||||
|
### Helper
|
||||||
|
|
||||||
|
`deploy/files/usr/local/libexec/left4me/left4me-script-sandbox`, mode 0755, owned root:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
[[ $# -eq 2 ]] || { echo "usage: $0 <overlay_id> <script>" >&2; exit 64; }
|
||||||
|
OVERLAY_ID=$1; SCRIPT=$2
|
||||||
|
[[ "$OVERLAY_ID" =~ ^[0-9]+$ ]] || { echo "bad overlay id" >&2; exit 64; }
|
||||||
|
OVERLAY_DIR=/var/lib/left4me/overlays/$OVERLAY_ID
|
||||||
|
[[ -d $OVERLAY_DIR ]] || { echo "no overlay dir at $OVERLAY_DIR" >&2; exit 65; }
|
||||||
|
[[ -f $SCRIPT ]] || { echo "no script at $SCRIPT" >&2; exit 65; }
|
||||||
|
|
||||||
|
if [[ "${LEFT4ME_SCRIPT_SANDBOX_DRY_RUN:-}" == "1" ]]; then
|
||||||
|
echo "DRY RUN: overlay_id=$OVERLAY_ID script=$SCRIPT overlay_dir=$OVERLAY_DIR"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
chown -R l4d2-sandbox:l4d2-sandbox "$OVERLAY_DIR"
|
||||||
|
chmod 0755 "$OVERLAY_DIR"
|
||||||
|
|
||||||
|
exec systemd-run --quiet --collect --wait --pipe \
|
||||||
|
--unit="left4me-script-${OVERLAY_ID}-$$" \
|
||||||
|
-p User=l4d2-sandbox -p Group=l4d2-sandbox \
|
||||||
|
-p NoNewPrivileges=yes \
|
||||||
|
-p ProtectSystem=strict -p ProtectHome=yes \
|
||||||
|
-p PrivateTmp=yes -p PrivateDevices=yes -p PrivateIPC=yes \
|
||||||
|
-p ProtectKernelTunables=yes -p ProtectKernelModules=yes \
|
||||||
|
-p ProtectKernelLogs=yes -p ProtectControlGroups=yes \
|
||||||
|
-p RestrictNamespaces=yes \
|
||||||
|
-p RestrictAddressFamilies="AF_INET AF_INET6 AF_UNIX" \
|
||||||
|
-p RestrictSUIDSGID=yes -p LockPersonality=yes \
|
||||||
|
-p MemoryDenyWriteExecute=yes \
|
||||||
|
-p SystemCallFilter="@system-service @network-io" \
|
||||||
|
-p SystemCallArchitectures=native \
|
||||||
|
-p CapabilityBoundingSet= -p AmbientCapabilities= \
|
||||||
|
-p TemporaryFileSystem="/etc /var/lib" \
|
||||||
|
-p BindReadOnlyPaths="/etc/resolv.conf /etc/ssl /etc/ca-certificates /etc/nsswitch.conf /etc/alternatives ${SCRIPT}:/script.sh" \
|
||||||
|
-p BindPaths="${OVERLAY_DIR}:/overlay" \
|
||||||
|
-p WorkingDirectory=/overlay \
|
||||||
|
-p Environment="HOME=/tmp PATH=/usr/bin:/usr/sbin OVERLAY=/overlay" \
|
||||||
|
-p MemoryMax=4G -p MemorySwapMax=0 -p TasksMax=512 \
|
||||||
|
-p CPUQuota=200% -p RuntimeMaxSec=3600 \
|
||||||
|
-- /bin/bash /script.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Sudoers fragment
|
||||||
|
|
||||||
|
Unchanged from v1: `left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-script-sandbox`.
|
||||||
|
|
||||||
|
### System user
|
||||||
|
|
||||||
|
Unchanged from v1: `l4d2-sandbox` (`useradd --system --no-create-home --shell /usr/sbin/nologin`).
|
||||||
|
|
||||||
|
### Filesystem expectations
|
||||||
|
|
||||||
|
- `/var/lib/left4me` must be mode 0711 (left4me-owned). Already provisioned by v1 deploy script.
|
||||||
|
- `/var/lib/left4me/left4me.db` mode 0640 root:left4me. **New** — added by this change.
|
||||||
|
- Overlay directory `/var/lib/left4me/overlays/{id}/` chowned to `l4d2-sandbox:l4d2-sandbox` 0755 by the helper before each run. Unchanged from v1.
|
||||||
|
|
||||||
|
## Build Lifecycle (unchanged from v1)
|
||||||
|
|
||||||
|
`ScriptBuilder.build()` writes the script to a 0644 tmpfile, exec's `sudo -n /usr/local/libexec/left4me/left4me-script-sandbox <id> <tmpfile>` via `run_command`, then runs `_enforce_disk_budget`. The helper's internal mechanism changes; the wrapper API is identical. `Overlay.last_build_status` is written by the job worker on completion.
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
- **systemd CVE landing in our directive set.** Single-tool migration removes one isolation layer. Mitigated by uid drop + cgroup limits + `NoNewPrivileges=yes` (kernel-enforced state independent of namespace setup). The escape would be an unprivileged process with no filesystem isolation but still capped on resources; same severity envelope as a hypothetical bwrap CVE in v1. The trust model (registered users) makes a single isolation layer acceptable.
|
||||||
|
- **`SystemCallFilter` rejecting a syscall a user script unexpectedly needs.** Symptom: build fails with SIGSYS. Diagnosis: `journalctl --since "1 min ago" | grep SECCOMP`. Resolution: widen the filter (`+@process`, `+@privileged` if the script genuinely needs more than a normal service). v1 had no syscall filter, so this is a new failure class.
|
||||||
|
- **`ProtectSystem=strict` masking something a script wanted to write to.** Only `/overlay`, `/tmp`, `/run` are writable inside the sandbox. Same as v1.
|
||||||
|
- **Host PID visibility (no `PrivatePID=`).** Information disclosure; not a privilege boundary.
|
||||||
|
- **`MemoryDenyWriteExecute=yes` blocking JITs.** A script that launches `node` / a JIT runtime would fail because W+X mappings are blocked. None of the recipe set the user has historically used (curl + tar + cp) needs a JIT; revisit if a real script trips this.
|
||||||
|
- **`RestrictAddressFamilies` blocking some download tools.** `curl`, `wget`, `git over https` use `AF_INET`/`AF_INET6`; `getent hosts` uses `AF_UNIX` (nss). Smoke-tested as working. A script that wanted raw sockets (`AF_PACKET`) or netlink (`AF_NETLINK`) would fail; neither is plausible for build recipes.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- **Per-overlay UID isolation.** Cross-script-overlay write access is still possible after a hypothetical sandbox bypass (every script overlay's dir is owned by `l4d2-sandbox`). A per-overlay UID pool was discussed as the next-step hardening but is deferred.
|
||||||
|
- **`PrivateNetwork=` / egress filtering.** No change from v1.
|
||||||
|
- **systemd-nspawn or LXC.** Researched; both are heavier than necessary for transient bash builds.
|
||||||
|
- **`PrivatePID=` workaround via `unshare`.** Not pursued — would require re-introducing a wrapper inside the unit, defeating the simplification.
|
||||||
|
|
||||||
|
## Implementation Boundaries
|
||||||
|
|
||||||
|
- **Web app code is unchanged.** `ScriptBuilder`, `run_sandboxed_script`, route handlers, models, migrations — all untouched. The migration is purely in the deployed helper script and adjacent deploy artifacts.
|
||||||
|
- **`bubblewrap` apt package removed.** Already absent from production paths after this change; deploy script updated.
|
||||||
|
- **No new systemd unit files.** Each invocation is a transient unit named `left4me-script-{overlay_id}-{pid}.service`.
|
||||||
|
- **No application-level dependency changes.** No new Python packages, no template changes, no DB migration.
|
||||||
|
|
@ -0,0 +1,113 @@
|
||||||
|
# L4D2 Script Sandbox v3 — Egress Filter (Public Internet Only)
|
||||||
|
|
||||||
|
**Goal:** Restrict the script-overlay sandbox to public-internet egress only. Block reachability to the host's own services (localhost), the LAN, and any private RFC1918 / link-local / multicast / CGNAT / ULA addresses. Public DNS is preserved by bind-mounting a sandbox-only `resolv.conf` pointing at Cloudflare + Google.
|
||||||
|
|
||||||
|
**Approval status:** User-approved 2026-05-08. Implemented and smoke-tested on `ckn@10.0.4.128`.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
After the v2 (systemd-only) migration, the sandbox still shared the host's network namespace. A live probe demonstrated the script could:
|
||||||
|
|
||||||
|
- Reach the web app on `127.0.0.1:8000` (HTTP 200 from `/health`).
|
||||||
|
- Reach the host's SSH daemon on `127.0.0.1:22` (banner returned).
|
||||||
|
- Reach the host on the LAN at `10.0.4.128:22` (banner returned).
|
||||||
|
- Reach the LAN gateway / DNS server at `10.0.0.1`.
|
||||||
|
- See Unix sockets in `/run` (`AF_UNIX` allowed).
|
||||||
|
|
||||||
|
The threat model says the sandbox should reach the public internet to download Workshop / l4d2center / GitHub content, but should **not** be able to talk to the host or LAN. systemd's `IPAddressDeny=` BPF cgroup egress filter is the right tool. It attaches a BPF program (`sd_fw_egress`) to the unit's cgroup; matching packets are silently dropped at send time.
|
||||||
|
|
||||||
|
A complication: the host's `/etc/resolv.conf` typically points at a private-IP DNS server (10.0.0.1 in the test deploy). Naively blocking `10.0.0.0/8` kills DNS, which kills outbound HTTP. The fix is to give the sandbox a static `resolv.conf` with public resolvers; DNS traffic then targets allowed public IPs.
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
1. **`IPAddressDeny=` alone — no `IPAddressAllow=any`.** The systemd documentation claims "more specific rule wins" when both are set, but on systemd 257 + kernel 6.12 (and likely other combos), `IPAddressAllow=any` silently overrides every `IPAddressDeny=` rule. Verified empirically. With only `IPAddressDeny=` set, the kernel's default "allow all" applies to non-listed addresses; the listed CIDRs are dropped at the egress hook. **This must not be regressed** — adding back `IPAddressAllow=any` reopens every blocked range.
|
||||||
|
|
||||||
|
2. **Explicit CIDRs, no shorthand keywords.** systemd's unit-file parser accepts `localhost`, `link-local`, `multicast` shortcuts, but the `systemd-run -p` parser rejects them with `Failed to parse IP address prefix: localhost`. Use the CIDRs directly: `127.0.0.0/8 ::1/128 169.254.0.0/16 fe80::/10 224.0.0.0/4 ff00::/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 fc00::/7`.
|
||||||
|
|
||||||
|
3. **Static `/etc/left4me/sandbox-resolv.conf` with public resolvers** (Cloudflare 1.1.1.1, Google 8.8.8.8). Bind-mounted into the sandbox at `/etc/resolv.conf` via `BindReadOnlyPaths=/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf`. Two nameservers for redundancy. Picking other public resolvers (Quad9, OpenDNS) would also be acceptable; the file is the source of truth, not the helper.
|
||||||
|
|
||||||
|
4. **`AF_UNIX` stays in `RestrictAddressFamilies=`.** Dropping it would risk breaking NSS / syslog / D-Bus introspection paths for marginal gain — the IP-level filter handles the actual threat (reaching host TCP services). The Unix-socket surface (D-Bus system bus, systemd notify) is uid-gated and `l4d2-sandbox` has no special D-Bus permissions.
|
||||||
|
|
||||||
|
5. **No `PrivateNetwork=`.** That would block all networking, including the public internet. The whole point of script overlays is reaching public download sources.
|
||||||
|
|
||||||
|
6. **No DNS-over-HTTPS or DNSSEC.** Plain UDP-53 to public resolvers is sufficient; the threat is "egress targeting", not "DNS hijacking". Revisit if the trust model relaxes.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```text
|
||||||
|
sudo helper (root)
|
||||||
|
└─ chown overlay dir to l4d2-sandbox
|
||||||
|
└─ systemd-run --service [...all v2 directives...]
|
||||||
|
-p IPAddressDeny="<11 CIDRs>"
|
||||||
|
-p BindReadOnlyPaths="/etc/left4me/sandbox-resolv.conf:/etc/resolv.conf [...]"
|
||||||
|
└─ /bin/bash /script.sh
|
||||||
|
(egress to listed CIDRs dropped at sd_fw_egress BPF hook;
|
||||||
|
DNS goes to 1.1.1.1 / 8.8.8.8; everything else
|
||||||
|
reaches the public internet normally)
|
||||||
|
```
|
||||||
|
|
||||||
|
`IPAddressDeny=` blocks egress to:
|
||||||
|
|
||||||
|
| CIDR | Coverage |
|
||||||
|
|---|---|
|
||||||
|
| `127.0.0.0/8` | IPv4 loopback |
|
||||||
|
| `::1/128` | IPv6 loopback |
|
||||||
|
| `169.254.0.0/16` | IPv4 link-local (incl. AWS metadata, DHCP fallback) |
|
||||||
|
| `fe80::/10` | IPv6 link-local |
|
||||||
|
| `224.0.0.0/4` | IPv4 multicast |
|
||||||
|
| `ff00::/8` | IPv6 multicast |
|
||||||
|
| `10.0.0.0/8` | RFC1918 private |
|
||||||
|
| `172.16.0.0/12` | RFC1918 private |
|
||||||
|
| `192.168.0.0/16` | RFC1918 private |
|
||||||
|
| `100.64.0.0/10` | CGNAT (RFC6598) |
|
||||||
|
| `fc00::/7` | IPv6 ULA |
|
||||||
|
|
||||||
|
Public IPv4 / IPv6 destinations are unaffected.
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
- `deploy/files/etc/left4me/sandbox-resolv.conf` *(new)* — `nameserver 1.1.1.1` + `nameserver 8.8.8.8`. Mode 0644 root-owned at deploy time.
|
||||||
|
- `deploy/files/usr/local/libexec/left4me/left4me-script-sandbox` — `IPAddressDeny=` directive added; `BindReadOnlyPaths=` references the sandbox-resolv.conf instead of `/etc/resolv.conf`.
|
||||||
|
- `deploy/deploy-test-server.sh` — `install -m 0644 -o root -g root .../sandbox-resolv.conf /etc/left4me/sandbox-resolv.conf`.
|
||||||
|
- `deploy/tests/test_deploy_artifacts.py` — assert all of the above + the **negative assertion `IPAddressAllow=any not in text`** (regression guard).
|
||||||
|
|
||||||
|
The web app, ScriptBuilder, routes, models, and migrations are all unchanged. Same as v2.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
Smoke battery on the deployed host (probe script invoked through the helper as root):
|
||||||
|
|
||||||
|
| Target | Expected | Actual |
|
||||||
|
|---|---|---|
|
||||||
|
| `1.1.1.1:443` | connected | ✓ CONNECTED |
|
||||||
|
| `https://steamcommunity.com/` (DNS + HTTPS) | 200 | ✓ 200 |
|
||||||
|
| `127.0.0.1:8000` (web app) | blocked | ✓ TimeoutError |
|
||||||
|
| `127.0.0.1:22` (sshd) | blocked | ✓ TimeoutError |
|
||||||
|
| `10.0.4.128:22` (host LAN ssh) | blocked | ✓ TimeoutError |
|
||||||
|
| `10.0.0.1:53` (host's DNS resolver) | blocked | ✓ TimeoutError |
|
||||||
|
| `cat /etc/resolv.conf` inside | shows 1.1.1.1 + 8.8.8.8 | ✓ |
|
||||||
|
|
||||||
|
`bpftool cgroup show` against the unit's cgroup confirms `sd_fw_egress` and `sd_fw_ingress` are attached.
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
- **`IPAddressAllow=` accidentally added back.** Reopens every blocked range silently. Mitigation: explicit negative test in `test_deploy_artifacts.py` plus a comment in the helper.
|
||||||
|
- **Public DNS resolver outage.** 1.1.1.1 and 8.8.8.8 are both down → DNS in sandbox fails → builds fail. Two resolvers from independent operators makes this very unlikely. Operator can change the file in `/etc/left4me/sandbox-resolv.conf` if they prefer different resolvers; the helper picks it up on next invocation.
|
||||||
|
- **Public DNS resolver privacy.** Cloudflare and Google see hostnames the scripts query. Acceptable for the workload (Steam Workshop, GitHub, etc. are public anyway); switch to Quad9 or self-hosted if this is a concern.
|
||||||
|
- **Future kernel/systemd that flips the documented "more specific wins" semantics.** If a future systemd version actually implements the documented behavior, a unit with only `IPAddressDeny=` continues to work; the negative test on `IPAddressAllow=any` keeps the regression-safe configuration locked in. Re-test on each major systemd upgrade.
|
||||||
|
- **Scripts that legitimately need a private IP.** E.g., a self-hosted internal mirror at 10.x. Not a use case today; if it arises, expose specific IPs via a future `IPAddressAllow=10.x.y.z/32` for that one host (not blanket).
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- **Per-overlay UID isolation.** Cross-script-overlay write access via the shared `l4d2-sandbox` UID is still possible after a hypothetical sandbox bypass. Deferred from earlier discussions.
|
||||||
|
- **Egress allowlist by hostname / domain.** Would require a forward proxy (Squid, mitmproxy). Heavier than warranted for the trust model.
|
||||||
|
- **Dropping `AF_UNIX` from `RestrictAddressFamilies=`.** Tangential to IP-level egress; risks breaking NSS / syslog.
|
||||||
|
- **DNSSEC / DoH.** Threat model is egress targeting, not DNS hijacking.
|
||||||
|
- **Network-namespace isolation (`PrivateNetwork=` + custom netns + NAT).** Heavier than `IPAddressDeny=` for equivalent outcome.
|
||||||
|
|
||||||
|
## Implementation Boundaries
|
||||||
|
|
||||||
|
- **No app code change.** Helper-side only.
|
||||||
|
- **No new systemd units.** Same transient `left4me-script-{id}-{pid}.service` pattern.
|
||||||
|
- **No new apt deps.** `bpftool` was used during smoke testing but is not required at runtime.
|
||||||
|
- **One new deploy artifact.** `sandbox-resolv.conf` shipped under `deploy/files/etc/left4me/`.
|
||||||
|
|
@ -0,0 +1,55 @@
|
||||||
|
# Server ID as Host Identifier Design
|
||||||
|
|
||||||
|
**Goal:** Decouple the user-facing server label from the host-side identifier. The systemd unit name and on-disk paths become functions of `Server.id`; `Server.name` becomes a free-form display label.
|
||||||
|
|
||||||
|
**Approval status:** User-approved 2026-05-08.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
`Server.name` was doing two unrelated jobs. It was the human label rendered in the UI *and* the literal string fed to `l4d2ctl`, which became the systemd unit instance (`left4me-server@<name>.service`) and the directories under `/var/lib/left4me/{instances,runtime}/<name>/`. To stay safe as a unit-template parameter and a path component, the name was forced through `[a-z0-9][a-z0-9_-]{0,63}` and held globally unique. The cost was a UI that demanded machine-friendly slugs, no rename support, and an awkward divergence from overlays — which already separate identity (`id`) from label (`name`).
|
||||||
|
|
||||||
|
This change moves servers onto the same model as overlays. Web URLs already key on `id` (`/servers/<int:server_id>`), so the change is mostly local: pick an id-derived host identifier, pass that everywhere `server.name` was passed, and relax the `name` constraints.
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
1. **Host-side identifier = plain numeric id.** `left4me-server@42.service`, `/var/lib/left4me/instances/42/`, `/var/lib/left4me/runtime/42/`. The host CLI's `validate_instance_name` regex (`[a-z0-9][a-z0-9_-]{0,63}`), the systemctl helper's argument check (`[A-Za-z0-9_.-]`), and the unit template (`%i`) all already accept digit-only strings — no host-side change.
|
||||||
|
2. **Name = free-form display label, unique per user, required (≤128 chars).** Whitespace is stripped on save. Two users can both have a server named "Practice"; one user cannot.
|
||||||
|
3. **No data preservation.** Dev-only deploy. Existing servers on the test host are not migrated; their old `left4me-server@<old-name>.service` units and `<old-name>/` directories become orphans and are cleaned up manually.
|
||||||
|
4. **Single source of truth for the id-to-host-name rule.** A one-line helper (`server_unit_name(server_id) -> str(server_id)`) lives in `l4d2web/services/server_identity.py`. Every callsite that used to pass `server.name` to `l4d2ctl` or `journalctl` calls this. Future format tweaks (e.g. `srv-{id}`) are a one-line edit.
|
||||||
|
|
||||||
|
## Schema
|
||||||
|
|
||||||
|
`servers` (Alembic 0006):
|
||||||
|
- Drop the (unnamed) global `UNIQUE (name)` from the original 0001 schema.
|
||||||
|
- Add `UNIQUE (user_id, name)` as `uq_servers_user_name`.
|
||||||
|
- Column stays `name VARCHAR(128) NOT NULL`.
|
||||||
|
|
||||||
|
The migration uses `batch_alter_table(recreate="always")` with a `naming_convention` so the originally-anonymous unique can be referenced as `uq_servers_name` for `drop_constraint`.
|
||||||
|
|
||||||
|
## Code touchpoints
|
||||||
|
|
||||||
|
- `l4d2web/services/server_identity.py` (new)
|
||||||
|
- `l4d2web/models.py` — drop `unique=True` on `Server.name`; add `__table_args__` with the per-user unique.
|
||||||
|
- `l4d2web/alembic/versions/0006_server_name_per_user.py` (new)
|
||||||
|
- `l4d2web/services/l4d2_facade.py` — five `l4d2ctl` invocations switched to `server_unit_name(server.id)`. Parameter renamed to `unit_name` on `server_status` / `stream_server_logs`.
|
||||||
|
- `l4d2web/services/job_worker.py` — status refresh uses `server_unit_name(server.id)`. The `server_name` log-label variable still holds `server.name` (the display label); that's correct now and shows up in job logs as e.g. "starting initialize for My Practice".
|
||||||
|
- `l4d2web/routes/log_routes.py` — SSE log stream feeds `server_unit_name(server.id)` to `journalctl`.
|
||||||
|
- `l4d2web/routes/server_routes.py` — replace `validate_instance_name` with `_validate_display_name` (strip + non-empty + length ≤128). Broaden the `IntegrityError` handler to disambiguate `servers.name` (409 "name already in use") from `servers.port` (409 "port already in use") via the underlying SQLite error string.
|
||||||
|
- `l4d2web/services/security.py` — `validate_instance_name` deleted (no remaining callers).
|
||||||
|
- `l4d2web/templates/servers.html` — name input gains `maxlength="128"`.
|
||||||
|
|
||||||
|
## Failure modes
|
||||||
|
|
||||||
|
- **Name with shell metacharacters reaches a host command.** Cannot happen — the host call now receives only `str(server.id)` (digits). The display name is never passed through `l4d2ctl`.
|
||||||
|
- **Two servers under the same user with the same name.** Blocked at the DB layer (`uq_servers_user_name`); surfaced as a 409 "name already in use" with no row written.
|
||||||
|
- **Migration on a DB with existing servers.** `batch_alter_table(recreate="always")` rebuilds the table preserving rows; the new per-user constraint is satisfied trivially since the old global constraint already enforced strict uniqueness.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
1. `python -m pytest l4d2web l4d2host deploy` from the repo root — green.
|
||||||
|
2. Stepwise migration on a fresh sqlite (upgrade to 0005, insert two users + a server, upgrade to 0006): row preserved, second user can take the same name, same user cannot (UNIQUE constraint failed: servers.user_id, servers.name).
|
||||||
|
3. Post-deploy on the test host: create a server named `"My Practice"` (with the space), confirm the systemd unit is `left4me-server@<id>.service`, confirm `/var/lib/left4me/runtime/<id>/merged` is mounted on start, confirm log streaming still works.
|
||||||
|
|
||||||
|
## Operator note
|
||||||
|
|
||||||
|
After deploy, on the test host: stop and remove any pre-existing `left4me-server@<old-name>.service` units and their `/var/lib/left4me/{instances,runtime}/<old-name>/` directories. The web app no longer references them.
|
||||||
109
examples/script-overlays/cedapug_maps.sh
Normal file
109
examples/script-overlays/cedapug_maps.sh
Normal file
|
|
@ -0,0 +1,109 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Cedapug custom-maps overlay.
|
||||||
|
# Scrapes the JS map list at https://cedapug.com/custom and keeps
|
||||||
|
# $OVERLAY/left4dead2/addons/ in sync. Re-downloads only when the zip's
|
||||||
|
# Last-Modified header changed.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
CEDAPUG_HOST="https://cedapug.com"
|
||||||
|
LIST_URL="$CEDAPUG_HOST/custom"
|
||||||
|
ADDONS_DIR="$OVERLAY/left4dead2/addons"
|
||||||
|
STATE_DIR="$OVERLAY/.cedapug"
|
||||||
|
WORK_DIR="$STATE_DIR/work"
|
||||||
|
MANIFEST="$STATE_DIR/manifest.tsv" # zip_basename \t last_modified \t vpk1,vpk2,...
|
||||||
|
|
||||||
|
mkdir -p "$ADDONS_DIR" "$STATE_DIR" "$WORK_DIR"
|
||||||
|
|
||||||
|
# --- 1. Discover the current zip list -----------------------------------
|
||||||
|
# The page embeds the map list inside renderCustomMapDownloads([...]) with
|
||||||
|
# JSON-escaped slashes (\/maps\/foo.zip). Unescape with sed before grepping.
|
||||||
|
echo ":: fetching $LIST_URL"
|
||||||
|
mapfile -t ZIP_PATHS < <(
|
||||||
|
curl -fsSL --retry 3 --retry-delay 2 "$LIST_URL" \
|
||||||
|
| sed 's@\\/@/@g' \
|
||||||
|
| grep -oE '"/maps/[A-Za-z0-9_.+-]+\.zip"' \
|
||||||
|
| tr -d '"' \
|
||||||
|
| sort -u
|
||||||
|
)
|
||||||
|
if [[ ${#ZIP_PATHS[@]} -eq 0 ]]; then
|
||||||
|
echo "!! no map zips found on $LIST_URL — page format may have changed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo ":: ${#ZIP_PATHS[@]} custom map zip(s) listed"
|
||||||
|
|
||||||
|
# --- 2. Load previous manifest ------------------------------------------
|
||||||
|
declare -A PREV_LM PREV_VPKS
|
||||||
|
if [[ -f "$MANIFEST" ]]; then
|
||||||
|
while IFS=$'\t' read -r zname lm vpks; do
|
||||||
|
[[ -n "$zname" ]] || continue
|
||||||
|
PREV_LM[$zname]=$lm
|
||||||
|
PREV_VPKS[$zname]=$vpks
|
||||||
|
done < "$MANIFEST"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- 3. Download / extract each desired zip -----------------------------
|
||||||
|
declare -A NEW_LM NEW_VPKS
|
||||||
|
for zpath in "${ZIP_PATHS[@]}"; do
|
||||||
|
zname="${zpath##*/}"
|
||||||
|
url="$CEDAPUG_HOST$zpath"
|
||||||
|
|
||||||
|
lm="$(curl -fsSI --retry 3 --retry-delay 2 "$url" \
|
||||||
|
| awk 'BEGIN{IGNORECASE=1} /^last-modified:/ {sub(/^[^:]*:[[:space:]]*/, ""); gsub(/\r/, ""); print; exit}')"
|
||||||
|
NEW_LM[$zname]=$lm
|
||||||
|
|
||||||
|
prev_lm="${PREV_LM[$zname]:-}"
|
||||||
|
prev_vpks="${PREV_VPKS[$zname]:-}"
|
||||||
|
|
||||||
|
# Skip if Last-Modified matches AND every recorded vpk is still on disk.
|
||||||
|
if [[ -n "$lm" && "$lm" == "$prev_lm" && -n "$prev_vpks" ]]; then
|
||||||
|
all_present=1
|
||||||
|
IFS=',' read -r -a v_arr <<<"$prev_vpks"
|
||||||
|
for v in "${v_arr[@]}"; do
|
||||||
|
[[ -f "$ADDONS_DIR/$v" ]] || { all_present=0; break; }
|
||||||
|
done
|
||||||
|
if (( all_present )); then
|
||||||
|
echo ":: $zname unchanged (Last-Modified: $lm) — skipped"
|
||||||
|
NEW_VPKS[$zname]=$prev_vpks
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ":: downloading $zname"
|
||||||
|
zfile="$WORK_DIR/$zname"
|
||||||
|
curl -fSL --retry 3 --retry-delay 2 -o "$zfile" "$url"
|
||||||
|
|
||||||
|
7z x -y -bd -o"$ADDONS_DIR" "$zfile" '*.vpk' >/dev/null
|
||||||
|
|
||||||
|
vpk_list="$(7z l -slt -ba "$zfile" \
|
||||||
|
| awk '/^Path = .*\.vpk$/ {sub(/^Path = /, ""); print}' \
|
||||||
|
| paste -sd, -)"
|
||||||
|
if [[ -z "$vpk_list" ]]; then
|
||||||
|
echo "!! warning: $zname contained no .vpk — leaving manifest entry empty" >&2
|
||||||
|
fi
|
||||||
|
NEW_VPKS[$zname]=$vpk_list
|
||||||
|
|
||||||
|
rm -f "$zfile"
|
||||||
|
done
|
||||||
|
|
||||||
|
# --- 4. Prune vpks from zips that disappeared from the listing ----------
|
||||||
|
for old_zname in "${!PREV_VPKS[@]}"; do
|
||||||
|
if [[ -z "${NEW_LM[$old_zname]:-}" ]]; then
|
||||||
|
IFS=',' read -r -a old_v_arr <<<"${PREV_VPKS[$old_zname]}"
|
||||||
|
for v in "${old_v_arr[@]}"; do
|
||||||
|
if [[ -n "$v" && -f "$ADDONS_DIR/$v" ]]; then
|
||||||
|
echo ":: pruning $v (zip $old_zname no longer listed)"
|
||||||
|
rm -f "$ADDONS_DIR/$v"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# --- 5. Rewrite manifest -------------------------------------------------
|
||||||
|
tmp_manifest="$(mktemp -p "$STATE_DIR")"
|
||||||
|
for zname in "${!NEW_LM[@]}"; do
|
||||||
|
printf '%s\t%s\t%s\n' "$zname" "${NEW_LM[$zname]}" "${NEW_VPKS[$zname]:-}" >>"$tmp_manifest"
|
||||||
|
done
|
||||||
|
mv -f "$tmp_manifest" "$MANIFEST"
|
||||||
|
|
||||||
|
rm -rf "$WORK_DIR"
|
||||||
|
echo ":: done — ${#NEW_LM[@]} zip(s) tracked, addons/ in sync"
|
||||||
15
examples/script-overlays/competitive_rework.sh
Normal file
15
examples/script-overlays/competitive_rework.sh
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# L4D2 Competitive Rework overlay.
|
||||||
|
# Pulls the master branch of SirPlease/L4D2-Competitive-Rework as a tarball
|
||||||
|
# and extracts it into $OVERLAY/left4dead2/. Skipped on subsequent rebuilds
|
||||||
|
# once cfg/cfgogl/ is present — wipe the overlay to force a refresh.
|
||||||
|
# https://github.com/SirPlease/L4D2-Competitive-Rework
|
||||||
|
set -xeuo pipefail
|
||||||
|
|
||||||
|
DEST="$OVERLAY/left4dead2"
|
||||||
|
mkdir -p "$DEST"
|
||||||
|
|
||||||
|
if [[ ! -d "$DEST/cfg/cfgogl" ]]; then
|
||||||
|
curl -fsSL https://github.com/SirPlease/L4D2-Competitive-Rework/archive/refs/heads/master.tar.gz \
|
||||||
|
| tar -xz --strip-components=1 -C "$DEST"
|
||||||
|
fi
|
||||||
48
examples/script-overlays/l4d2center_maps.sh
Normal file
48
examples/script-overlays/l4d2center_maps.sh
Normal file
|
|
@ -0,0 +1,48 @@
|
||||||
|
#!/bin/bash
|
||||||
|
set -xeuo pipefail
|
||||||
|
|
||||||
|
mkdir -p /overlay/left4dead2/addons
|
||||||
|
cd /overlay/left4dead2/addons
|
||||||
|
|
||||||
|
for cmd in curl md5sum 7z; do
|
||||||
|
command -v "$cmd" >/dev/null || { echo "Missing: $cmd" >&2; exit 1; }
|
||||||
|
done
|
||||||
|
|
||||||
|
CSV_URL="https://l4d2center.com/maps/servers/index.csv"
|
||||||
|
TEMP_CSV=$(mktemp)
|
||||||
|
trap 'rm -f "$TEMP_CSV"' EXIT
|
||||||
|
curl -fsSL -o "$TEMP_CSV" "$CSV_URL"
|
||||||
|
|
||||||
|
declare -A map_md5 map_links
|
||||||
|
{
|
||||||
|
IFS= read -r _header
|
||||||
|
while IFS=';' read -r Name _Size MD5 DownloadLink || [[ $Name ]]; do
|
||||||
|
Name=$(echo "$Name" | xargs)
|
||||||
|
MD5=$(echo "$MD5" | xargs)
|
||||||
|
DownloadLink=$(echo "$DownloadLink" | xargs)
|
||||||
|
map_md5["$Name"]="$MD5"
|
||||||
|
map_links["$Name"]="$DownloadLink"
|
||||||
|
done
|
||||||
|
} < "$TEMP_CSV"
|
||||||
|
|
||||||
|
shopt -s nullglob
|
||||||
|
for file in *.vpk; do
|
||||||
|
if [[ -z "${map_md5[$file]:-}" ]]; then
|
||||||
|
rm -f "$file"
|
||||||
|
elif [[ "$(md5sum "$file" | awk '{print $1}')" != "${map_md5[$file]}" ]]; then
|
||||||
|
rm -f "$file"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
for vpk in "${!map_md5[@]}"; do
|
||||||
|
[[ -f "$vpk" ]] && continue
|
||||||
|
url="${map_links[$vpk]}"
|
||||||
|
[[ -n "$url" ]] || continue
|
||||||
|
encoded=$(printf '%s' "$url" | sed 's/ /%20/g')
|
||||||
|
TEMP_7Z=$(mktemp --suffix=.7z)
|
||||||
|
curl -fsSL -o "$TEMP_7Z" "$encoded"
|
||||||
|
7z x -y "$TEMP_7Z"
|
||||||
|
rm -f "$TEMP_7Z"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "Synchronization complete."
|
||||||
25
examples/script-overlays/tickrate.sh
Normal file
25
examples/script-overlays/tickrate.sh
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Tickrate overlay: server.cfg with high-rate settings + the tickrate_enabler
|
||||||
|
# addon (DLL/SO/VDF) from SirPlease/L4D2-Competitive-Rework.
|
||||||
|
# https://github.com/SirPlease/L4D2-Competitive-Rework/blob/7ecc3a32a5e2180d6607a40119ff2f3c072502a9/cfg/server.cfg#L58-L69
|
||||||
|
# https://www.programmersought.com/article/513810199514/
|
||||||
|
set -xeuo pipefail
|
||||||
|
|
||||||
|
CFG_DIR="$OVERLAY/left4dead2/cfg"
|
||||||
|
ADDONS_DIR="$OVERLAY/left4dead2/addons"
|
||||||
|
mkdir -p "$CFG_DIR" "$ADDONS_DIR"
|
||||||
|
|
||||||
|
cat <<'EOF' > "$CFG_DIR/server.cfg"
|
||||||
|
# https://github.com/SirPlease/L4D2-Competitive-Rework/blob/7ecc3a32a5e2180d6607a40119ff2f3c072502a9/cfg/server.cfg#L58-L69
|
||||||
|
sv_minrate 100000
|
||||||
|
sv_maxrate 100000
|
||||||
|
nb_update_frequency 0.014
|
||||||
|
net_splitpacket_maxrate 50000
|
||||||
|
net_maxcleartime 0.0001
|
||||||
|
fps_max 0
|
||||||
|
EOF
|
||||||
|
|
||||||
|
for file in tickrate_enabler.dll tickrate_enabler.so tickrate_enabler.vdf; do
|
||||||
|
curl -fsSL "https://github.com/SirPlease/L4D2-Competitive-Rework/raw/refs/heads/master/addons/${file}" \
|
||||||
|
-o "$ADDONS_DIR/${file}"
|
||||||
|
done
|
||||||
|
|
@ -49,7 +49,7 @@ Validated on Debian 13 during the `ckn@10.0.4.128` smoke test:
|
||||||
- Python 3.12+ with virtualenv/pip tooling for installing `l4d2host`.
|
- Python 3.12+ with virtualenv/pip tooling for installing `l4d2host`.
|
||||||
- `steamcmd` available on `PATH` and able to self-update as the runtime user.
|
- `steamcmd` available on `PATH` and able to self-update as the runtime user.
|
||||||
- 32-bit compatibility libraries for SteamCMD on amd64 Debian: `libc6-i386`, `lib32gcc-s1`, `lib32stdc++6`.
|
- 32-bit compatibility libraries for SteamCMD on amd64 Debian: `libc6-i386`, `lib32gcc-s1`, `lib32stdc++6`.
|
||||||
- `fuse-overlayfs` and `fusermount3` for per-instance overlay mounts.
|
- Kernel overlayfs (`mount -t overlay`); mount/umount go through the `left4me-overlay` privileged helper, which `nsenter`s into PID 1's mount namespace.
|
||||||
- `systemctl --user` and `journalctl --user` available for the runtime user.
|
- `systemctl --user` and `journalctl --user` available for the runtime user.
|
||||||
- User lingering enabled when services must survive SSH sessions: `sudo loginctl enable-linger <user>`.
|
- User lingering enabled when services must survive SSH sessions: `sudo loginctl enable-linger <user>`.
|
||||||
- `/var/lib/left4me` created and writable by the runtime user, unless `LEFT4ME_ROOT` is set to another deployment-managed root.
|
- `/var/lib/left4me` created and writable by the runtime user, unless `LEFT4ME_ROOT` is set to another deployment-managed root.
|
||||||
|
|
@ -61,7 +61,7 @@ sudo apt-get update
|
||||||
sudo apt-get install -y \
|
sudo apt-get install -y \
|
||||||
python3 python3-venv python3-pip \
|
python3 python3-venv python3-pip \
|
||||||
curl ca-certificates tar gzip \
|
curl ca-certificates tar gzip \
|
||||||
fuse-overlayfs fuse3 \
|
util-linux \
|
||||||
libc6-i386 lib32gcc-s1 lib32stdc++6
|
libc6-i386 lib32gcc-s1 lib32stdc++6
|
||||||
|
|
||||||
sudo mkdir -p /opt/steamcmd /var/lib/left4me/{installation,overlays,instances,runtime,tmp}
|
sudo mkdir -p /opt/steamcmd /var/lib/left4me/{installation,overlays,instances,runtime,tmp}
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ import subprocess
|
||||||
|
|
||||||
import typer
|
import typer
|
||||||
|
|
||||||
from l4d2host.instances import delete_instance, initialize_instance, start_instance, stop_instance
|
from l4d2host.instances import delete_instance, initialize_instance, reset_instance, start_instance, stop_instance
|
||||||
from l4d2host.logs import stream_instance_logs
|
from l4d2host.logs import stream_instance_logs
|
||||||
from l4d2host.status import get_instance_status
|
from l4d2host.status import get_instance_status
|
||||||
from l4d2host.steam_install import SteamInstaller
|
from l4d2host.steam_install import SteamInstaller
|
||||||
|
|
@ -59,6 +59,14 @@ def delete(name: str) -> None:
|
||||||
_exit_from_subprocess_error(exc)
|
_exit_from_subprocess_error(exc)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command()
|
||||||
|
def reset(name: str) -> None:
|
||||||
|
try:
|
||||||
|
reset_instance(name, passthrough=True)
|
||||||
|
except subprocess.CalledProcessError as exc:
|
||||||
|
_exit_from_subprocess_error(exc)
|
||||||
|
|
||||||
|
|
||||||
@app.command()
|
@app.command()
|
||||||
def status(name: str, json_output: bool = typer.Option(False, "--json")) -> None:
|
def status(name: str, json_output: bool = typer.Option(False, "--json")) -> None:
|
||||||
instance_status = get_instance_status(name)
|
instance_status = get_instance_status(name)
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,15 @@ from l4d2host.fs.base import OverlayMounter
|
||||||
from l4d2host.process import run_command
|
from l4d2host.process import run_command
|
||||||
|
|
||||||
|
|
||||||
class FuseOverlayFSMounter(OverlayMounter):
|
HELPER_PATH = "/usr/local/libexec/left4me/left4me-overlay"
|
||||||
|
|
||||||
|
|
||||||
|
class KernelOverlayFSMounter(OverlayMounter):
|
||||||
|
# Delegates the actual mount/umount syscalls to the privileged
|
||||||
|
# left4me-overlay helper. The helper takes only the instance name and
|
||||||
|
# rederives lowerdirs/upper/work/merged from disk; the OverlayMounter
|
||||||
|
# ABC accepts those args for compatibility, so we extract the name
|
||||||
|
# from the merged path's parent directory.
|
||||||
def mount(
|
def mount(
|
||||||
self,
|
self,
|
||||||
*,
|
*,
|
||||||
|
|
@ -18,13 +26,9 @@ class FuseOverlayFSMounter(OverlayMounter):
|
||||||
passthrough: bool = False,
|
passthrough: bool = False,
|
||||||
should_cancel: Callable[[], bool] | None = None,
|
should_cancel: Callable[[], bool] | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
|
del lowerdirs, upperdir, workdir
|
||||||
run_command(
|
run_command(
|
||||||
[
|
["sudo", "-n", HELPER_PATH, "mount", merged.parent.name],
|
||||||
"fuse-overlayfs",
|
|
||||||
"-o",
|
|
||||||
f"lowerdir={lowerdirs},upperdir={upperdir},workdir={workdir}",
|
|
||||||
str(merged),
|
|
||||||
],
|
|
||||||
on_stdout=on_stdout,
|
on_stdout=on_stdout,
|
||||||
on_stderr=on_stderr,
|
on_stderr=on_stderr,
|
||||||
passthrough=passthrough,
|
passthrough=passthrough,
|
||||||
|
|
@ -41,7 +45,7 @@ class FuseOverlayFSMounter(OverlayMounter):
|
||||||
should_cancel: Callable[[], bool] | None = None,
|
should_cancel: Callable[[], bool] | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
run_command(
|
run_command(
|
||||||
["fusermount3", "-u", str(merged)],
|
["sudo", "-n", HELPER_PATH, "umount", merged.parent.name],
|
||||||
on_stdout=on_stdout,
|
on_stdout=on_stdout,
|
||||||
on_stderr=on_stderr,
|
on_stderr=on_stderr,
|
||||||
passthrough=passthrough,
|
passthrough=passthrough,
|
||||||
|
|
@ -1,10 +1,11 @@
|
||||||
|
import os
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from typing import Callable
|
from typing import Callable
|
||||||
|
|
||||||
|
from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
|
||||||
from l4d2host.paths import DEFAULT_LEFT4ME_ROOT, get_left4me_root, overlay_path, validate_instance_name
|
from l4d2host.paths import DEFAULT_LEFT4ME_ROOT, get_left4me_root, overlay_path, validate_instance_name
|
||||||
from l4d2host.process import run_command
|
|
||||||
from l4d2host.service_control import start_service, stop_service
|
from l4d2host.service_control import start_service, stop_service
|
||||||
from l4d2host.spec import load_spec
|
from l4d2host.spec import load_spec
|
||||||
|
|
||||||
|
|
@ -12,6 +13,9 @@ from l4d2host.spec import load_spec
|
||||||
from l4d2host.logging import emit_step
|
from l4d2host.logging import emit_step
|
||||||
|
|
||||||
|
|
||||||
|
_mounter = KernelOverlayFSMounter()
|
||||||
|
|
||||||
|
|
||||||
DEFAULT_ROOT = DEFAULT_LEFT4ME_ROOT
|
DEFAULT_ROOT = DEFAULT_LEFT4ME_ROOT
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -82,18 +86,23 @@ def start_instance(
|
||||||
|
|
||||||
env = _load_instance_env(instance_dir / "instance.env")
|
env = _load_instance_env(instance_dir / "instance.env")
|
||||||
|
|
||||||
|
merged = runtime_dir / "merged"
|
||||||
|
if os.path.ismount(merged):
|
||||||
|
# Kernel overlayfs mounts persist when the web worker dies (unlike
|
||||||
|
# fuse daemons, which were reaped with their cgroup). Refuse rather
|
||||||
|
# than double-mount.
|
||||||
|
raise subprocess.CalledProcessError(
|
||||||
|
returncode=1,
|
||||||
|
cmd=["start_instance"],
|
||||||
|
stderr=f"runtime overlay already mounted at {merged}; refusing to double-mount",
|
||||||
|
)
|
||||||
|
|
||||||
emit_step("mounting runtime overlay...", on_stdout, passthrough)
|
emit_step("mounting runtime overlay...", on_stdout, passthrough)
|
||||||
run_command(
|
_mounter.mount(
|
||||||
[
|
lowerdirs=env["L4D2_LOWERDIRS"],
|
||||||
"fuse-overlayfs",
|
upperdir=runtime_dir / "upper",
|
||||||
"-o",
|
workdir=runtime_dir / "work",
|
||||||
(
|
merged=merged,
|
||||||
f"lowerdir={env['L4D2_LOWERDIRS']},"
|
|
||||||
f"upperdir={runtime_dir / 'upper'},"
|
|
||||||
f"workdir={runtime_dir / 'work'}"
|
|
||||||
),
|
|
||||||
str(runtime_dir / "merged"),
|
|
||||||
],
|
|
||||||
on_stdout=on_stdout,
|
on_stdout=on_stdout,
|
||||||
on_stderr=on_stderr,
|
on_stderr=on_stderr,
|
||||||
passthrough=passthrough,
|
passthrough=passthrough,
|
||||||
|
|
@ -135,17 +144,63 @@ def stop_instance(
|
||||||
passthrough=passthrough,
|
passthrough=passthrough,
|
||||||
should_cancel=should_cancel,
|
should_cancel=should_cancel,
|
||||||
)
|
)
|
||||||
emit_step("unmounting runtime overlay...", on_stdout, passthrough)
|
emit_step("unmounting runtime overlay (if mounted)...", on_stdout, passthrough)
|
||||||
run_command(
|
try:
|
||||||
["fusermount3", "-u", str(root / "runtime" / name / "merged")],
|
_mounter.unmount(
|
||||||
on_stdout=on_stdout,
|
merged=root / "runtime" / name / "merged",
|
||||||
on_stderr=on_stderr,
|
on_stdout=on_stdout,
|
||||||
passthrough=passthrough,
|
on_stderr=on_stderr,
|
||||||
should_cancel=should_cancel,
|
passthrough=passthrough,
|
||||||
)
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
except subprocess.CalledProcessError:
|
||||||
|
pass
|
||||||
emit_step("stop complete.", on_stdout, passthrough)
|
emit_step("stop complete.", on_stdout, passthrough)
|
||||||
|
|
||||||
|
|
||||||
|
def _purge_instance(
|
||||||
|
name: str,
|
||||||
|
*,
|
||||||
|
root: Path,
|
||||||
|
on_stdout: Callable[[str], None] | None,
|
||||||
|
on_stderr: Callable[[str], None] | None,
|
||||||
|
passthrough: bool,
|
||||||
|
should_cancel: Callable[[], bool] | None,
|
||||||
|
) -> None:
|
||||||
|
instance_dir = root / "instances" / name
|
||||||
|
runtime_dir = root / "runtime" / name
|
||||||
|
|
||||||
|
emit_step("stopping systemd service (if running)...", on_stdout, passthrough)
|
||||||
|
try:
|
||||||
|
stop_service(
|
||||||
|
name,
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
passthrough=passthrough,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
except subprocess.CalledProcessError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
emit_step("unmounting runtime overlay (if mounted)...", on_stdout, passthrough)
|
||||||
|
try:
|
||||||
|
_mounter.unmount(
|
||||||
|
merged=runtime_dir / "merged",
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
passthrough=passthrough,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
except subprocess.CalledProcessError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
emit_step("removing instance files...", on_stdout, passthrough)
|
||||||
|
if instance_dir.exists():
|
||||||
|
shutil.rmtree(instance_dir)
|
||||||
|
if runtime_dir.exists():
|
||||||
|
shutil.rmtree(runtime_dir)
|
||||||
|
|
||||||
|
|
||||||
def delete_instance(
|
def delete_instance(
|
||||||
name: str,
|
name: str,
|
||||||
*,
|
*,
|
||||||
|
|
@ -163,33 +218,34 @@ def delete_instance(
|
||||||
if not instance_dir.exists() and not runtime_dir.exists():
|
if not instance_dir.exists() and not runtime_dir.exists():
|
||||||
return
|
return
|
||||||
|
|
||||||
emit_step("stopping systemd service (if running)...", on_stdout, passthrough)
|
_purge_instance(
|
||||||
try:
|
name,
|
||||||
stop_service(
|
root=root,
|
||||||
name,
|
on_stdout=on_stdout,
|
||||||
on_stdout=on_stdout,
|
on_stderr=on_stderr,
|
||||||
on_stderr=on_stderr,
|
passthrough=passthrough,
|
||||||
passthrough=passthrough,
|
should_cancel=should_cancel,
|
||||||
should_cancel=should_cancel,
|
)
|
||||||
)
|
|
||||||
except subprocess.CalledProcessError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
emit_step("unmounting runtime overlay (if mounted)...", on_stdout, passthrough)
|
|
||||||
try:
|
|
||||||
run_command(
|
|
||||||
["fusermount3", "-u", str(runtime_dir / "merged")],
|
|
||||||
on_stdout=on_stdout,
|
|
||||||
on_stderr=on_stderr,
|
|
||||||
passthrough=passthrough,
|
|
||||||
should_cancel=should_cancel,
|
|
||||||
)
|
|
||||||
except subprocess.CalledProcessError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
emit_step("removing instance files...", on_stdout, passthrough)
|
|
||||||
if instance_dir.exists():
|
|
||||||
shutil.rmtree(instance_dir)
|
|
||||||
if runtime_dir.exists():
|
|
||||||
shutil.rmtree(runtime_dir)
|
|
||||||
emit_step("delete complete.", on_stdout, passthrough)
|
emit_step("delete complete.", on_stdout, passthrough)
|
||||||
|
|
||||||
|
|
||||||
|
def reset_instance(
|
||||||
|
name: str,
|
||||||
|
*,
|
||||||
|
root: Path | None = None,
|
||||||
|
on_stdout: Callable[[str], None] | None = None,
|
||||||
|
on_stderr: Callable[[str], None] | None = None,
|
||||||
|
passthrough: bool = False,
|
||||||
|
should_cancel: Callable[[], bool] | None = None,
|
||||||
|
) -> None:
|
||||||
|
name = validate_instance_name(name)
|
||||||
|
root = get_left4me_root() if root is None else Path(root)
|
||||||
|
_purge_instance(
|
||||||
|
name,
|
||||||
|
root=root,
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
passthrough=passthrough,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
emit_step("reset complete; next start will reinitialize from blueprint.", on_stdout, passthrough)
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,4 @@
|
||||||
|
import shutil
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Callable
|
from typing import Callable
|
||||||
|
|
||||||
|
|
@ -6,6 +7,57 @@ from l4d2host.process import run_command
|
||||||
from l4d2host.logging import emit_step
|
from l4d2host.logging import emit_step
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_steamcmd_siblings(steamcmd: str) -> tuple[Path, Path] | None:
|
||||||
|
resolved = shutil.which(steamcmd)
|
||||||
|
base = Path(resolved).resolve() if resolved else Path(steamcmd)
|
||||||
|
if base.is_symlink():
|
||||||
|
base = base.resolve()
|
||||||
|
parent = base.parent
|
||||||
|
linux32 = parent / "linux32"
|
||||||
|
linux64 = parent / "linux64"
|
||||||
|
if linux32.is_dir() and linux64.is_dir():
|
||||||
|
return linux32, linux64
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _set_symlink(link: Path, target: Path, on_stderr: Callable[[str], None] | None) -> None:
|
||||||
|
if link.is_symlink():
|
||||||
|
try:
|
||||||
|
current = link.resolve()
|
||||||
|
except OSError:
|
||||||
|
current = None
|
||||||
|
if current == target.resolve():
|
||||||
|
return
|
||||||
|
link.unlink()
|
||||||
|
elif link.exists():
|
||||||
|
if on_stderr is not None:
|
||||||
|
on_stderr(f"refusing to replace non-symlink at {link}")
|
||||||
|
return
|
||||||
|
link.symlink_to(target)
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_steam_sdk_symlinks(
|
||||||
|
install_dir: Path,
|
||||||
|
steamcmd: str,
|
||||||
|
*,
|
||||||
|
on_stdout: Callable[[str], None] | None,
|
||||||
|
on_stderr: Callable[[str], None] | None,
|
||||||
|
passthrough: bool,
|
||||||
|
) -> None:
|
||||||
|
emit_step("ensuring steam sdk symlinks...", on_stdout, passthrough)
|
||||||
|
siblings = _resolve_steamcmd_siblings(steamcmd)
|
||||||
|
if siblings is not None:
|
||||||
|
sdk32_target, sdk64_target = siblings
|
||||||
|
else:
|
||||||
|
fallback = install_dir / "bin"
|
||||||
|
sdk32_target, sdk64_target = fallback, fallback
|
||||||
|
|
||||||
|
steam_dir = Path.home() / ".steam"
|
||||||
|
steam_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
_set_symlink(steam_dir / "sdk32", sdk32_target, on_stderr)
|
||||||
|
_set_symlink(steam_dir / "sdk64", sdk64_target, on_stderr)
|
||||||
|
|
||||||
|
|
||||||
class SteamInstaller:
|
class SteamInstaller:
|
||||||
def __init__(self, install_dir: Path | None = None, steamcmd: str = "steamcmd"):
|
def __init__(self, install_dir: Path | None = None, steamcmd: str = "steamcmd"):
|
||||||
self.install_dir = get_left4me_root() / "installation" if install_dir is None else Path(install_dir)
|
self.install_dir = get_left4me_root() / "installation" if install_dir is None else Path(install_dir)
|
||||||
|
|
@ -40,4 +92,11 @@ class SteamInstaller:
|
||||||
passthrough=passthrough,
|
passthrough=passthrough,
|
||||||
should_cancel=should_cancel,
|
should_cancel=should_cancel,
|
||||||
)
|
)
|
||||||
|
_ensure_steam_sdk_symlinks(
|
||||||
|
self.install_dir,
|
||||||
|
self.steamcmd,
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
passthrough=passthrough,
|
||||||
|
)
|
||||||
emit_step("installation complete.", on_stdout, passthrough)
|
emit_step("installation complete.", on_stdout, passthrough)
|
||||||
|
|
|
||||||
|
|
@ -46,8 +46,8 @@ def test_default_install_dir_uses_left4me_root(tmp_path, monkeypatch):
|
||||||
|
|
||||||
|
|
||||||
def test_steam_installer_emits_steps(tmp_path, monkeypatch) -> None:
|
def test_steam_installer_emits_steps(tmp_path, monkeypatch) -> None:
|
||||||
from pathlib import Path
|
|
||||||
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
monkeypatch.setenv("HOME", str(tmp_path / "home"))
|
||||||
monkeypatch.setattr("l4d2host.steam_install.run_command", lambda cmd, **kwargs: None)
|
monkeypatch.setattr("l4d2host.steam_install.run_command", lambda cmd, **kwargs: None)
|
||||||
|
|
||||||
steps: list[str] = []
|
steps: list[str] = []
|
||||||
|
|
@ -58,5 +58,82 @@ def test_steam_installer_emits_steps(tmp_path, monkeypatch) -> None:
|
||||||
assert steps == [
|
assert steps == [
|
||||||
"Step: downloading windows platform payload...",
|
"Step: downloading windows platform payload...",
|
||||||
"Step: downloading linux platform payload...",
|
"Step: downloading linux platform payload...",
|
||||||
|
"Step: ensuring steam sdk symlinks...",
|
||||||
"Step: installation complete."
|
"Step: installation complete."
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_install_creates_steam_sdk_symlinks_to_steamcmd_siblings(tmp_path, monkeypatch) -> None:
|
||||||
|
home = tmp_path / "home"
|
||||||
|
home.mkdir()
|
||||||
|
monkeypatch.setenv("HOME", str(home))
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
|
||||||
|
steamcmd_dir = tmp_path / "opt" / "steamcmd"
|
||||||
|
(steamcmd_dir / "linux32").mkdir(parents=True)
|
||||||
|
(steamcmd_dir / "linux64").mkdir(parents=True)
|
||||||
|
fake_steamcmd = steamcmd_dir / "steamcmd.sh"
|
||||||
|
fake_steamcmd.write_text("#!/bin/sh\nexit 0\n")
|
||||||
|
fake_steamcmd.chmod(0o755)
|
||||||
|
|
||||||
|
monkeypatch.setattr("l4d2host.steam_install.run_command", lambda cmd, **kwargs: None)
|
||||||
|
|
||||||
|
from l4d2host.steam_install import SteamInstaller
|
||||||
|
SteamInstaller(steamcmd=str(fake_steamcmd)).install_or_update()
|
||||||
|
|
||||||
|
sdk32 = home / ".steam" / "sdk32"
|
||||||
|
sdk64 = home / ".steam" / "sdk64"
|
||||||
|
assert sdk32.is_symlink()
|
||||||
|
assert sdk64.is_symlink()
|
||||||
|
assert sdk32.resolve() == (steamcmd_dir / "linux32").resolve()
|
||||||
|
assert sdk64.resolve() == (steamcmd_dir / "linux64").resolve()
|
||||||
|
|
||||||
|
|
||||||
|
def test_install_creates_steam_sdk_symlinks_falls_back_to_install_bin(tmp_path, monkeypatch) -> None:
|
||||||
|
home = tmp_path / "home"
|
||||||
|
home.mkdir()
|
||||||
|
monkeypatch.setenv("HOME", str(home))
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
|
||||||
|
install_bin = tmp_path / "installation" / "bin"
|
||||||
|
install_bin.mkdir(parents=True)
|
||||||
|
|
||||||
|
isolated_dir = tmp_path / "no-siblings"
|
||||||
|
isolated_dir.mkdir()
|
||||||
|
fake_steamcmd = isolated_dir / "steamcmd.sh"
|
||||||
|
fake_steamcmd.write_text("#!/bin/sh\nexit 0\n")
|
||||||
|
fake_steamcmd.chmod(0o755)
|
||||||
|
|
||||||
|
monkeypatch.setattr("l4d2host.steam_install.run_command", lambda cmd, **kwargs: None)
|
||||||
|
|
||||||
|
from l4d2host.steam_install import SteamInstaller
|
||||||
|
SteamInstaller(steamcmd=str(fake_steamcmd)).install_or_update()
|
||||||
|
|
||||||
|
sdk32 = home / ".steam" / "sdk32"
|
||||||
|
sdk64 = home / ".steam" / "sdk64"
|
||||||
|
assert sdk32.resolve() == install_bin.resolve()
|
||||||
|
assert sdk64.resolve() == install_bin.resolve()
|
||||||
|
|
||||||
|
|
||||||
|
def test_install_steam_sdk_symlinks_is_idempotent(tmp_path, monkeypatch) -> None:
|
||||||
|
home = tmp_path / "home"
|
||||||
|
home.mkdir()
|
||||||
|
monkeypatch.setenv("HOME", str(home))
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
|
||||||
|
steamcmd_dir = tmp_path / "opt" / "steamcmd"
|
||||||
|
(steamcmd_dir / "linux32").mkdir(parents=True)
|
||||||
|
(steamcmd_dir / "linux64").mkdir(parents=True)
|
||||||
|
fake_steamcmd = steamcmd_dir / "steamcmd.sh"
|
||||||
|
fake_steamcmd.write_text("#!/bin/sh\nexit 0\n")
|
||||||
|
fake_steamcmd.chmod(0o755)
|
||||||
|
|
||||||
|
monkeypatch.setattr("l4d2host.steam_install.run_command", lambda cmd, **kwargs: None)
|
||||||
|
|
||||||
|
from l4d2host.steam_install import SteamInstaller
|
||||||
|
installer = SteamInstaller(steamcmd=str(fake_steamcmd))
|
||||||
|
installer.install_or_update()
|
||||||
|
installer.install_or_update()
|
||||||
|
|
||||||
|
sdk32 = home / ".steam" / "sdk32"
|
||||||
|
assert sdk32.resolve() == (steamcmd_dir / "linux32").resolve()
|
||||||
|
|
|
||||||
76
l4d2host/tests/test_kernel_overlayfs.py
Normal file
76
l4d2host/tests/test_kernel_overlayfs.py
Normal file
|
|
@ -0,0 +1,76 @@
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
HELPER_PATH = "/usr/local/libexec/left4me/left4me-overlay"
|
||||||
|
|
||||||
|
|
||||||
|
def test_mount_invokes_helper_with_name_only(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
|
||||||
|
|
||||||
|
calls: list[list[str]] = []
|
||||||
|
|
||||||
|
def fake_run_command(cmd, **kwargs):
|
||||||
|
del kwargs
|
||||||
|
calls.append(list(cmd))
|
||||||
|
|
||||||
|
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
|
||||||
|
|
||||||
|
KernelOverlayFSMounter().mount(
|
||||||
|
lowerdirs="/var/lib/left4me/installation",
|
||||||
|
upperdir=Path("/var/lib/left4me/runtime/alpha/upper"),
|
||||||
|
workdir=Path("/var/lib/left4me/runtime/alpha/work"),
|
||||||
|
merged=Path("/var/lib/left4me/runtime/alpha/merged"),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert calls == [["sudo", "-n", HELPER_PATH, "mount", "alpha"]]
|
||||||
|
|
||||||
|
|
||||||
|
def test_unmount_invokes_helper_with_umount_verb(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
|
||||||
|
|
||||||
|
calls: list[list[str]] = []
|
||||||
|
|
||||||
|
def fake_run_command(cmd, **kwargs):
|
||||||
|
del kwargs
|
||||||
|
calls.append(list(cmd))
|
||||||
|
|
||||||
|
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
|
||||||
|
|
||||||
|
KernelOverlayFSMounter().unmount(merged=Path("/var/lib/left4me/runtime/alpha/merged"))
|
||||||
|
|
||||||
|
assert calls == [["sudo", "-n", HELPER_PATH, "umount", "alpha"]]
|
||||||
|
|
||||||
|
|
||||||
|
def test_mount_propagates_run_command_kwargs(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
|
||||||
|
|
||||||
|
captured: dict = {}
|
||||||
|
|
||||||
|
def fake_run_command(cmd, **kwargs):
|
||||||
|
captured["cmd"] = list(cmd)
|
||||||
|
captured["kwargs"] = kwargs
|
||||||
|
|
||||||
|
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
|
||||||
|
|
||||||
|
out: list[str] = []
|
||||||
|
err: list[str] = []
|
||||||
|
KernelOverlayFSMounter().mount(
|
||||||
|
lowerdirs="/var/lib/left4me/installation",
|
||||||
|
upperdir=Path("/var/lib/left4me/runtime/alpha/upper"),
|
||||||
|
workdir=Path("/var/lib/left4me/runtime/alpha/work"),
|
||||||
|
merged=Path("/var/lib/left4me/runtime/alpha/merged"),
|
||||||
|
on_stdout=out.append,
|
||||||
|
on_stderr=err.append,
|
||||||
|
passthrough=False,
|
||||||
|
should_cancel=lambda: False,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert captured["cmd"][0:3] == ["sudo", "-n", HELPER_PATH]
|
||||||
|
captured["kwargs"]["on_stdout"]("hi")
|
||||||
|
captured["kwargs"]["on_stderr"]("oops")
|
||||||
|
assert out == ["hi"]
|
||||||
|
assert err == ["oops"]
|
||||||
|
assert captured["kwargs"]["passthrough"] is False
|
||||||
|
assert callable(captured["kwargs"]["should_cancel"])
|
||||||
|
|
@ -6,6 +6,7 @@ import pytest
|
||||||
from l4d2host.instances import (
|
from l4d2host.instances import (
|
||||||
delete_instance,
|
delete_instance,
|
||||||
initialize_instance,
|
initialize_instance,
|
||||||
|
reset_instance,
|
||||||
start_instance,
|
start_instance,
|
||||||
stop_instance,
|
stop_instance,
|
||||||
)
|
)
|
||||||
|
|
@ -27,15 +28,51 @@ def test_start_order(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
)
|
)
|
||||||
(instance_dir / "server.cfg").write_text("sv_consistency 1")
|
(instance_dir / "server.cfg").write_text("sv_consistency 1")
|
||||||
|
|
||||||
monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command)
|
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
|
||||||
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
||||||
|
|
||||||
start_instance("alpha", root=tmp_path)
|
start_instance("alpha", root=tmp_path)
|
||||||
|
|
||||||
assert calls[0][0] == "fuse-overlayfs"
|
assert calls[0] == [
|
||||||
|
"sudo",
|
||||||
|
"-n",
|
||||||
|
"/usr/local/libexec/left4me/left4me-overlay",
|
||||||
|
"mount",
|
||||||
|
"alpha",
|
||||||
|
]
|
||||||
assert calls[1] == ["sudo", "-n", "/usr/local/libexec/left4me/left4me-systemctl", "start", "alpha"]
|
assert calls[1] == ["sudo", "-n", "/usr/local/libexec/left4me/left4me-systemctl", "start", "alpha"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_start_refuses_to_double_mount(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
calls: list[list[str]] = []
|
||||||
|
|
||||||
|
def fake_run_command(cmd, **kwargs):
|
||||||
|
del kwargs
|
||||||
|
calls.append(list(cmd))
|
||||||
|
|
||||||
|
instance_dir = tmp_path / "instances" / "alpha"
|
||||||
|
runtime_dir = tmp_path / "runtime" / "alpha"
|
||||||
|
(runtime_dir / "merged").mkdir(parents=True)
|
||||||
|
instance_dir.mkdir(parents=True)
|
||||||
|
(instance_dir / "instance.env").write_text("L4D2_PORT=27015\nL4D2_ARGS=\nL4D2_LOWERDIRS=/x\n")
|
||||||
|
(instance_dir / "server.cfg").write_text("")
|
||||||
|
|
||||||
|
merged = runtime_dir / "merged"
|
||||||
|
|
||||||
|
def fake_ismount(path):
|
||||||
|
return Path(path) == merged
|
||||||
|
|
||||||
|
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
|
||||||
|
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
||||||
|
monkeypatch.setattr("l4d2host.instances.os.path.ismount", fake_ismount)
|
||||||
|
|
||||||
|
with pytest.raises(subprocess.CalledProcessError) as exc_info:
|
||||||
|
start_instance("alpha", root=tmp_path)
|
||||||
|
|
||||||
|
assert "already mounted" in (exc_info.value.stderr or "")
|
||||||
|
assert calls == [], "no mount/start commands must be issued when refusing"
|
||||||
|
|
||||||
|
|
||||||
def test_delete_missing_is_noop(tmp_path: Path) -> None:
|
def test_delete_missing_is_noop(tmp_path: Path) -> None:
|
||||||
delete_instance("missing", root=tmp_path)
|
delete_instance("missing", root=tmp_path)
|
||||||
|
|
||||||
|
|
@ -56,7 +93,7 @@ def test_delete_succeeds_when_stop_service_fails(tmp_path: Path, monkeypatch: py
|
||||||
(tmp_path / "instances" / "alpha").mkdir(parents=True)
|
(tmp_path / "instances" / "alpha").mkdir(parents=True)
|
||||||
(tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
|
(tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
|
||||||
|
|
||||||
monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command)
|
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
|
||||||
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
||||||
|
|
||||||
delete_instance("alpha", root=tmp_path)
|
delete_instance("alpha", root=tmp_path)
|
||||||
|
|
@ -67,7 +104,7 @@ def test_delete_succeeds_when_stop_service_fails(tmp_path: Path, monkeypatch: py
|
||||||
|
|
||||||
@pytest.mark.parametrize("bad_name", ["..", "../escape", "foo/bar", " foo", "Foo"])
|
@pytest.mark.parametrize("bad_name", ["..", "../escape", "foo/bar", " foo", "Foo"])
|
||||||
def test_lifecycle_rejects_unsafe_instance_names(tmp_path: Path, bad_name: str) -> None:
|
def test_lifecycle_rejects_unsafe_instance_names(tmp_path: Path, bad_name: str) -> None:
|
||||||
for func in (start_instance, stop_instance, delete_instance):
|
for func in (start_instance, stop_instance, delete_instance, reset_instance):
|
||||||
with pytest.raises(ValueError):
|
with pytest.raises(ValueError):
|
||||||
func(bad_name, root=tmp_path)
|
func(bad_name, root=tmp_path)
|
||||||
with pytest.raises(ValueError):
|
with pytest.raises(ValueError):
|
||||||
|
|
@ -76,6 +113,49 @@ def test_lifecycle_rejects_unsafe_instance_names(tmp_path: Path, bad_name: str)
|
||||||
assert not (tmp_path / "runtime").exists()
|
assert not (tmp_path / "runtime").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_reset_stops_unmounts_and_removes_dirs(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
calls: list[list[str]] = []
|
||||||
|
|
||||||
|
def fake_run_command(cmd, **kwargs):
|
||||||
|
del kwargs
|
||||||
|
calls.append(list(cmd))
|
||||||
|
|
||||||
|
instance_dir = tmp_path / "instances" / "alpha"
|
||||||
|
runtime_dir = tmp_path / "runtime" / "alpha"
|
||||||
|
instance_dir.mkdir(parents=True)
|
||||||
|
(runtime_dir / "merged").mkdir(parents=True)
|
||||||
|
(instance_dir / "instance.env").write_text("L4D2_PORT=27015\n")
|
||||||
|
(runtime_dir / "upper" / "logs").mkdir(parents=True)
|
||||||
|
(runtime_dir / "upper" / "logs" / "console.log").write_text("noise")
|
||||||
|
|
||||||
|
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
|
||||||
|
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
||||||
|
|
||||||
|
reset_instance("alpha", root=tmp_path)
|
||||||
|
|
||||||
|
assert not instance_dir.exists()
|
||||||
|
assert not runtime_dir.exists()
|
||||||
|
assert any("left4me-systemctl" in arg for cmd in calls for arg in cmd)
|
||||||
|
assert any("stop" in cmd for cmd in calls)
|
||||||
|
|
||||||
|
|
||||||
|
def test_reset_on_never_initialized_is_noop(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""No instance/runtime directories yet — reset should still attempt the
|
||||||
|
stop+unmount (both suppressed on failure) and not raise."""
|
||||||
|
def fake_run_command(cmd, **kwargs):
|
||||||
|
del kwargs
|
||||||
|
if "stop" in cmd:
|
||||||
|
raise subprocess.CalledProcessError(returncode=5, cmd=list(cmd), stderr="not loaded")
|
||||||
|
|
||||||
|
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
|
||||||
|
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
||||||
|
|
||||||
|
reset_instance("alpha", root=tmp_path)
|
||||||
|
|
||||||
|
assert not (tmp_path / "instances" / "alpha").exists()
|
||||||
|
assert not (tmp_path / "runtime" / "alpha").exists()
|
||||||
|
|
||||||
|
|
||||||
def test_delete_stopped_instance_removes_dirs(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_delete_stopped_instance_removes_dirs(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
calls: list[list[str]] = []
|
calls: list[list[str]] = []
|
||||||
|
|
||||||
|
|
@ -86,7 +166,7 @@ def test_delete_stopped_instance_removes_dirs(tmp_path: Path, monkeypatch: pytes
|
||||||
(tmp_path / "instances" / "alpha").mkdir(parents=True)
|
(tmp_path / "instances" / "alpha").mkdir(parents=True)
|
||||||
(tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
|
(tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
|
||||||
|
|
||||||
monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command)
|
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
|
||||||
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
||||||
|
|
||||||
delete_instance("alpha", root=tmp_path)
|
delete_instance("alpha", root=tmp_path)
|
||||||
|
|
@ -96,27 +176,58 @@ def test_delete_stopped_instance_removes_dirs(tmp_path: Path, monkeypatch: pytes
|
||||||
assert ["sudo", "-n", "/usr/local/libexec/left4me/left4me-systemctl", "stop", "alpha"] in calls
|
assert ["sudo", "-n", "/usr/local/libexec/left4me/left4me-systemctl", "stop", "alpha"] in calls
|
||||||
|
|
||||||
|
|
||||||
def test_delete_succeeds_when_unmount_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_stop_succeeds_when_unmount_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
fusermount_calls: list[list[str]] = []
|
umount_calls: list[list[str]] = []
|
||||||
|
|
||||||
def fake_run_command(cmd, **kwargs):
|
def fake_run_command(cmd, **kwargs):
|
||||||
del kwargs
|
del kwargs
|
||||||
if cmd and cmd[0] == "fusermount3":
|
if cmd[:4] == [
|
||||||
fusermount_calls.append(list(cmd))
|
"sudo",
|
||||||
|
"-n",
|
||||||
|
"/usr/local/libexec/left4me/left4me-overlay",
|
||||||
|
"umount",
|
||||||
|
]:
|
||||||
|
umount_calls.append(list(cmd))
|
||||||
raise subprocess.CalledProcessError(
|
raise subprocess.CalledProcessError(
|
||||||
returncode=1,
|
returncode=1,
|
||||||
cmd=list(cmd),
|
cmd=list(cmd),
|
||||||
stderr="fusermount3: entry for merged not found in /etc/mtab",
|
stderr="umount: /var/lib/left4me/runtime/alpha/merged: not mounted",
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
|
||||||
|
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
||||||
|
|
||||||
|
stop_instance("alpha", root=tmp_path)
|
||||||
|
|
||||||
|
assert umount_calls, "stop must always attempt the overlay helper (no preflight)"
|
||||||
|
|
||||||
|
|
||||||
|
def test_delete_succeeds_when_unmount_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
umount_calls: list[list[str]] = []
|
||||||
|
|
||||||
|
def fake_run_command(cmd, **kwargs):
|
||||||
|
del kwargs
|
||||||
|
if cmd[:4] == [
|
||||||
|
"sudo",
|
||||||
|
"-n",
|
||||||
|
"/usr/local/libexec/left4me/left4me-overlay",
|
||||||
|
"umount",
|
||||||
|
]:
|
||||||
|
umount_calls.append(list(cmd))
|
||||||
|
raise subprocess.CalledProcessError(
|
||||||
|
returncode=1,
|
||||||
|
cmd=list(cmd),
|
||||||
|
stderr="umount: /var/lib/left4me/runtime/alpha/merged: not mounted",
|
||||||
)
|
)
|
||||||
|
|
||||||
(tmp_path / "instances" / "alpha").mkdir(parents=True)
|
(tmp_path / "instances" / "alpha").mkdir(parents=True)
|
||||||
(tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
|
(tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
|
||||||
|
|
||||||
monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command)
|
monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
|
||||||
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
|
||||||
|
|
||||||
delete_instance("alpha", root=tmp_path)
|
delete_instance("alpha", root=tmp_path)
|
||||||
|
|
||||||
assert fusermount_calls, "delete must always attempt fusermount3 -u (no preflight)"
|
assert umount_calls, "delete must always attempt the overlay helper (no preflight)"
|
||||||
assert not (tmp_path / "instances" / "alpha").exists()
|
assert not (tmp_path / "instances" / "alpha").exists()
|
||||||
assert not (tmp_path / "runtime" / "alpha").exists()
|
assert not (tmp_path / "runtime" / "alpha").exists()
|
||||||
|
|
|
||||||
168
l4d2host/tests/test_overlay_helper.py
Normal file
168
l4d2host/tests/test_overlay_helper.py
Normal file
|
|
@ -0,0 +1,168 @@
|
||||||
|
import os
|
||||||
|
import shlex
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
HELPER_SOURCE = (
|
||||||
|
Path(__file__).resolve().parents[2]
|
||||||
|
/ "deploy"
|
||||||
|
/ "files"
|
||||||
|
/ "usr"
|
||||||
|
/ "local"
|
||||||
|
/ "libexec"
|
||||||
|
/ "left4me"
|
||||||
|
/ "left4me-overlay"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _setup_instance(root: Path, name: str = "alpha", lowerdirs: list[str] | None = None) -> None:
|
||||||
|
"""Create the on-disk shape the helper expects."""
|
||||||
|
(root / "installation").mkdir(parents=True, exist_ok=True)
|
||||||
|
(root / "overlays" / "workshop").mkdir(parents=True, exist_ok=True)
|
||||||
|
if lowerdirs is None:
|
||||||
|
lowerdirs = [str(root / "overlays" / "workshop"), str(root / "installation")]
|
||||||
|
inst_dir = root / "instances" / name
|
||||||
|
inst_dir.mkdir(parents=True)
|
||||||
|
(inst_dir / "instance.env").write_text(
|
||||||
|
f"L4D2_PORT=27015\nL4D2_ARGS=-tickrate 100\nL4D2_LOWERDIRS={':'.join(lowerdirs)}\n"
|
||||||
|
)
|
||||||
|
runtime = root / "runtime" / name
|
||||||
|
(runtime / "upper").mkdir(parents=True)
|
||||||
|
(runtime / "work").mkdir(parents=True)
|
||||||
|
(runtime / "merged").mkdir(parents=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _run(args: list[str], root: Path, extra_env: dict[str, str] | None = None) -> subprocess.CompletedProcess:
|
||||||
|
env = {
|
||||||
|
**os.environ,
|
||||||
|
"LEFT4ME_ROOT": str(root),
|
||||||
|
"LEFT4ME_OVERLAY_PRINT_ONLY": "1",
|
||||||
|
}
|
||||||
|
if extra_env:
|
||||||
|
env.update(extra_env)
|
||||||
|
return subprocess.run(
|
||||||
|
[sys.executable, str(HELPER_SOURCE), *args],
|
||||||
|
env=env,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_mount_prints_expected_nsenter_command(tmp_path: Path) -> None:
|
||||||
|
_setup_instance(tmp_path)
|
||||||
|
|
||||||
|
result = _run(["mount", "alpha"], tmp_path)
|
||||||
|
|
||||||
|
assert result.returncode == 0, result.stderr
|
||||||
|
parts = shlex.split(result.stdout.strip())
|
||||||
|
assert parts[0] == "/usr/bin/nsenter"
|
||||||
|
assert parts[1] == "--mount=/proc/1/ns/mnt"
|
||||||
|
assert parts[2] == "--"
|
||||||
|
assert parts[3] == "/bin/mount"
|
||||||
|
assert parts[4:6] == ["-t", "overlay"]
|
||||||
|
assert parts[6] == "overlay"
|
||||||
|
assert parts[7] == "-o"
|
||||||
|
options = parts[8]
|
||||||
|
assert f"upperdir={tmp_path}/runtime/alpha/upper" in options
|
||||||
|
assert f"workdir={tmp_path}/runtime/alpha/work" in options
|
||||||
|
assert f"lowerdir={tmp_path}/overlays/workshop:{tmp_path}/installation" in options
|
||||||
|
assert parts[9] == str(tmp_path / "runtime" / "alpha" / "merged")
|
||||||
|
|
||||||
|
|
||||||
|
def test_umount_prints_expected_nsenter_command(tmp_path: Path) -> None:
|
||||||
|
_setup_instance(tmp_path)
|
||||||
|
|
||||||
|
result = _run(["umount", "alpha"], tmp_path)
|
||||||
|
|
||||||
|
assert result.returncode == 0, result.stderr
|
||||||
|
parts = shlex.split(result.stdout.strip())
|
||||||
|
assert parts == [
|
||||||
|
"/usr/bin/nsenter",
|
||||||
|
"--mount=/proc/1/ns/mnt",
|
||||||
|
"--",
|
||||||
|
"/bin/umount",
|
||||||
|
str(tmp_path / "runtime" / "alpha" / "merged"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("bad_name", ["..", "../escape", "FOO", "foo bar", "foo/bar", ""])
|
||||||
|
def test_rejects_bad_instance_name(tmp_path: Path, bad_name: str) -> None:
|
||||||
|
result = _run(["mount", bad_name], tmp_path)
|
||||||
|
assert result.returncode != 0
|
||||||
|
assert "invalid instance name" in result.stderr or "usage:" in result.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_rejects_lowerdir_outside_allowlist(tmp_path: Path) -> None:
|
||||||
|
_setup_instance(tmp_path, lowerdirs=["/etc"])
|
||||||
|
result = _run(["mount", "alpha"], tmp_path)
|
||||||
|
assert result.returncode != 0
|
||||||
|
assert "outside the permitted roots" in result.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_rejects_lowerdir_traversal(tmp_path: Path) -> None:
|
||||||
|
# An overlay subdirectory whose path uses .. to escape the overlays root.
|
||||||
|
_setup_instance(tmp_path, lowerdirs=[str(tmp_path / "overlays" / "..") + "/etc"])
|
||||||
|
result = _run(["mount", "alpha"], tmp_path)
|
||||||
|
assert result.returncode != 0
|
||||||
|
assert "outside the permitted roots" in result.stderr or "path does not exist" in result.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_rejects_lowerdir_symlink_escape(tmp_path: Path) -> None:
|
||||||
|
_setup_instance(tmp_path)
|
||||||
|
sneaky = tmp_path / "overlays" / "sneaky"
|
||||||
|
os.symlink("/etc", sneaky)
|
||||||
|
# rewrite instance.env to point at the symlink
|
||||||
|
inst_env = tmp_path / "instances" / "alpha" / "instance.env"
|
||||||
|
inst_env.write_text(f"L4D2_LOWERDIRS={sneaky}\n")
|
||||||
|
result = _run(["mount", "alpha"], tmp_path)
|
||||||
|
assert result.returncode != 0
|
||||||
|
assert "outside the permitted roots" in result.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_rejects_missing_instance_env(tmp_path: Path) -> None:
|
||||||
|
(tmp_path / "instances" / "alpha").mkdir(parents=True)
|
||||||
|
runtime = tmp_path / "runtime" / "alpha"
|
||||||
|
(runtime / "upper").mkdir(parents=True)
|
||||||
|
(runtime / "work").mkdir(parents=True)
|
||||||
|
(runtime / "merged").mkdir(parents=True)
|
||||||
|
result = _run(["mount", "alpha"], tmp_path)
|
||||||
|
assert result.returncode != 0
|
||||||
|
assert "instance.env not found" in result.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_rejects_lowerdir_count_over_cap(tmp_path: Path) -> None:
|
||||||
|
(tmp_path / "installation").mkdir()
|
||||||
|
many = [str(tmp_path / "installation")] * 501
|
||||||
|
_setup_instance(tmp_path, lowerdirs=many)
|
||||||
|
result = _run(["mount", "alpha"], tmp_path)
|
||||||
|
assert result.returncode != 0
|
||||||
|
assert "501 entries" in result.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_rejects_empty_lowerdir_entry(tmp_path: Path) -> None:
|
||||||
|
(tmp_path / "installation").mkdir()
|
||||||
|
_setup_instance(
|
||||||
|
tmp_path,
|
||||||
|
lowerdirs=[str(tmp_path / "installation"), "", str(tmp_path / "installation")],
|
||||||
|
)
|
||||||
|
result = _run(["mount", "alpha"], tmp_path)
|
||||||
|
assert result.returncode != 0
|
||||||
|
assert "empty entry" in result.stderr
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(sys.platform != "linux", reason="user.* xattrs are Linux-only")
|
||||||
|
def test_rejects_upperdir_with_fuseoverlayfs_xattr(tmp_path: Path) -> None:
|
||||||
|
_setup_instance(tmp_path)
|
||||||
|
tainted = tmp_path / "runtime" / "alpha" / "upper" / "deleted-thing"
|
||||||
|
tainted.write_bytes(b"")
|
||||||
|
try:
|
||||||
|
os.setxattr(tainted, "user.fuseoverlayfs.opaque", b"y")
|
||||||
|
except OSError:
|
||||||
|
pytest.skip("filesystem doesn't support user.* xattrs")
|
||||||
|
result = _run(["mount", "alpha"], tmp_path)
|
||||||
|
assert result.returncode != 0
|
||||||
|
assert "fuse-overlayfs xattr" in result.stderr
|
||||||
174
l4d2web/alembic/versions/0002_workshop_overlays.py
Normal file
174
l4d2web/alembic/versions/0002_workshop_overlays.py
Normal file
|
|
@ -0,0 +1,174 @@
|
||||||
|
"""workshop overlays
|
||||||
|
|
||||||
|
Revision ID: 0002_workshop_overlays
|
||||||
|
Revises: b2c684fddbd3
|
||||||
|
Create Date: 2026-05-07
|
||||||
|
"""
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision: str = "0002_workshop_overlays"
|
||||||
|
down_revision: Union[str, Sequence[str], None] = "b2c684fddbd3"
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def _existing_overlays_table() -> sa.Table:
|
||||||
|
"""Pre-migration shape used as `copy_from` so batch_alter_table rebuilds
|
||||||
|
overlays without the inline UNIQUE on `name` (replaced by partial unique
|
||||||
|
indexes after the recreate)."""
|
||||||
|
metadata = sa.MetaData()
|
||||||
|
return sa.Table(
|
||||||
|
"overlays",
|
||||||
|
metadata,
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column("name", sa.String(length=128), nullable=False),
|
||||||
|
sa.Column("path", sa.String(length=512), nullable=False),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(), nullable=False),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
# 1. Recreate `overlays` with `type`, `user_id`, autoincrement, and no inline UNIQUE on name.
|
||||||
|
with op.batch_alter_table(
|
||||||
|
"overlays",
|
||||||
|
recreate="always",
|
||||||
|
copy_from=_existing_overlays_table(),
|
||||||
|
table_kwargs={"sqlite_autoincrement": True},
|
||||||
|
) as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"type",
|
||||||
|
sa.String(length=16),
|
||||||
|
nullable=False,
|
||||||
|
server_default="external",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"user_id",
|
||||||
|
sa.Integer(),
|
||||||
|
sa.ForeignKey("users.id", name="fk_overlays_user_id_users"),
|
||||||
|
nullable=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.create_index("ix_overlays_type_user_id", ["type", "user_id"])
|
||||||
|
|
||||||
|
# Drop the temporary server_default once existing rows are backfilled.
|
||||||
|
with op.batch_alter_table("overlays") as batch_op:
|
||||||
|
batch_op.alter_column("type", server_default=None)
|
||||||
|
|
||||||
|
# 2. Partial unique indexes for name uniqueness:
|
||||||
|
# - system overlays (user_id IS NULL): globally unique by name
|
||||||
|
# - user overlays (user_id IS NOT NULL): unique per user by name
|
||||||
|
op.create_index(
|
||||||
|
"uq_overlay_name_system",
|
||||||
|
"overlays",
|
||||||
|
["name"],
|
||||||
|
unique=True,
|
||||||
|
sqlite_where=sa.text("user_id IS NULL"),
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"uq_overlay_name_per_user",
|
||||||
|
"overlays",
|
||||||
|
["name", "user_id"],
|
||||||
|
unique=True,
|
||||||
|
sqlite_where=sa.text("user_id IS NOT NULL"),
|
||||||
|
)
|
||||||
|
|
||||||
|
# 3. workshop_items registry (global, deduplicated by steam_id).
|
||||||
|
op.create_table(
|
||||||
|
"workshop_items",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column("steam_id", sa.String(length=20), nullable=False, unique=True),
|
||||||
|
sa.Column("title", sa.String(length=255), nullable=False, server_default=""),
|
||||||
|
sa.Column("filename", sa.String(length=255), nullable=False, server_default=""),
|
||||||
|
sa.Column("file_url", sa.Text(), nullable=False, server_default=""),
|
||||||
|
sa.Column("file_size", sa.BigInteger(), nullable=False, server_default="0"),
|
||||||
|
sa.Column("time_updated", sa.Integer(), nullable=False, server_default="0"),
|
||||||
|
sa.Column("preview_url", sa.Text(), nullable=False, server_default=""),
|
||||||
|
sa.Column("last_downloaded_at", sa.DateTime(), nullable=True),
|
||||||
|
sa.Column("last_error", sa.Text(), nullable=False, server_default=""),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(), nullable=False),
|
||||||
|
)
|
||||||
|
|
||||||
|
# 4. overlay_workshop_items association.
|
||||||
|
op.create_table(
|
||||||
|
"overlay_workshop_items",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column(
|
||||||
|
"overlay_id",
|
||||||
|
sa.Integer(),
|
||||||
|
sa.ForeignKey("overlays.id", ondelete="CASCADE"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column(
|
||||||
|
"workshop_item_id",
|
||||||
|
sa.Integer(),
|
||||||
|
sa.ForeignKey("workshop_items.id", ondelete="RESTRICT"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"overlay_id", "workshop_item_id", name="uq_overlay_workshop_item"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_owi_workshop_item",
|
||||||
|
"overlay_workshop_items",
|
||||||
|
["workshop_item_id"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# 5. Add overlay_id to jobs for build_overlay tracking.
|
||||||
|
with op.batch_alter_table("jobs") as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"overlay_id",
|
||||||
|
sa.Integer(),
|
||||||
|
sa.ForeignKey("overlays.id", name="fk_jobs_overlay_id_overlays"),
|
||||||
|
nullable=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("jobs") as batch_op:
|
||||||
|
batch_op.drop_column("overlay_id")
|
||||||
|
|
||||||
|
op.drop_index("ix_owi_workshop_item", table_name="overlay_workshop_items")
|
||||||
|
op.drop_table("overlay_workshop_items")
|
||||||
|
op.drop_table("workshop_items")
|
||||||
|
|
||||||
|
op.drop_index("uq_overlay_name_per_user", table_name="overlays")
|
||||||
|
op.drop_index("uq_overlay_name_system", table_name="overlays")
|
||||||
|
op.drop_index("ix_overlays_type_user_id", table_name="overlays")
|
||||||
|
|
||||||
|
# Recreate `overlays` to drop type/user_id and restore single-column UNIQUE on name.
|
||||||
|
current_overlays = sa.Table(
|
||||||
|
"overlays",
|
||||||
|
sa.MetaData(),
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column("name", sa.String(length=128), nullable=False),
|
||||||
|
sa.Column("path", sa.String(length=512), nullable=False),
|
||||||
|
sa.Column("type", sa.String(length=16), nullable=False),
|
||||||
|
sa.Column(
|
||||||
|
"user_id",
|
||||||
|
sa.Integer(),
|
||||||
|
sa.ForeignKey("users.id", name="fk_overlays_user_id_users"),
|
||||||
|
nullable=True,
|
||||||
|
),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(), nullable=False),
|
||||||
|
)
|
||||||
|
with op.batch_alter_table(
|
||||||
|
"overlays",
|
||||||
|
recreate="always",
|
||||||
|
copy_from=current_overlays,
|
||||||
|
) as batch_op:
|
||||||
|
batch_op.drop_column("user_id")
|
||||||
|
batch_op.drop_column("type")
|
||||||
|
batch_op.create_unique_constraint("uq_overlays_name", ["name"])
|
||||||
105
l4d2web/alembic/versions/0003_global_map_overlays.py
Normal file
105
l4d2web/alembic/versions/0003_global_map_overlays.py
Normal file
|
|
@ -0,0 +1,105 @@
|
||||||
|
"""global map overlays
|
||||||
|
|
||||||
|
Revision ID: 0003_global_map_overlays
|
||||||
|
Revises: 0002_workshop_overlays
|
||||||
|
Create Date: 2026-05-07
|
||||||
|
"""
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision: str = "0003_global_map_overlays"
|
||||||
|
down_revision: Union[str, Sequence[str], None] = "0002_workshop_overlays"
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
with op.batch_alter_table("jobs") as batch_op:
|
||||||
|
batch_op.alter_column("user_id", existing_type=sa.Integer(), nullable=True)
|
||||||
|
|
||||||
|
op.create_table(
|
||||||
|
"global_overlay_sources",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column(
|
||||||
|
"overlay_id",
|
||||||
|
sa.Integer(),
|
||||||
|
sa.ForeignKey("overlays.id", ondelete="CASCADE"),
|
||||||
|
nullable=False,
|
||||||
|
unique=True,
|
||||||
|
),
|
||||||
|
sa.Column("source_key", sa.String(length=64), nullable=False, unique=True),
|
||||||
|
sa.Column("source_type", sa.String(length=32), nullable=False),
|
||||||
|
sa.Column("source_url", sa.Text(), nullable=False),
|
||||||
|
sa.Column("last_manifest_hash", sa.String(length=64), nullable=False, server_default=""),
|
||||||
|
sa.Column("last_refreshed_at", sa.DateTime(), nullable=True),
|
||||||
|
sa.Column("last_error", sa.Text(), nullable=False, server_default=""),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(), nullable=False),
|
||||||
|
)
|
||||||
|
op.create_index("ix_global_overlay_sources_type", "global_overlay_sources", ["source_type"])
|
||||||
|
|
||||||
|
op.create_table(
|
||||||
|
"global_overlay_items",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column(
|
||||||
|
"source_id",
|
||||||
|
sa.Integer(),
|
||||||
|
sa.ForeignKey("global_overlay_sources.id", ondelete="CASCADE"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column("item_key", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("display_name", sa.String(length=255), nullable=False, server_default=""),
|
||||||
|
sa.Column("download_url", sa.Text(), nullable=False),
|
||||||
|
sa.Column("expected_vpk_name", sa.String(length=255), nullable=False, server_default=""),
|
||||||
|
sa.Column("expected_size", sa.BigInteger(), nullable=True),
|
||||||
|
sa.Column("expected_md5", sa.String(length=32), nullable=False, server_default=""),
|
||||||
|
sa.Column("etag", sa.String(length=255), nullable=False, server_default=""),
|
||||||
|
sa.Column("last_modified", sa.String(length=255), nullable=False, server_default=""),
|
||||||
|
sa.Column("content_length", sa.BigInteger(), nullable=True),
|
||||||
|
sa.Column("last_downloaded_at", sa.DateTime(), nullable=True),
|
||||||
|
sa.Column("last_error", sa.Text(), nullable=False, server_default=""),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(), nullable=False),
|
||||||
|
sa.UniqueConstraint("source_id", "item_key", name="uq_global_overlay_item_source_key"),
|
||||||
|
)
|
||||||
|
op.create_index("ix_global_overlay_items_source", "global_overlay_items", ["source_id"])
|
||||||
|
|
||||||
|
op.create_table(
|
||||||
|
"global_overlay_item_files",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column(
|
||||||
|
"item_id",
|
||||||
|
sa.Integer(),
|
||||||
|
sa.ForeignKey("global_overlay_items.id", ondelete="CASCADE"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column("vpk_name", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("cache_path", sa.Text(), nullable=False),
|
||||||
|
sa.Column("size", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("md5", sa.String(length=32), nullable=False, server_default=""),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False),
|
||||||
|
sa.Column("updated_at", sa.DateTime(), nullable=False),
|
||||||
|
sa.UniqueConstraint("item_id", "vpk_name", name="uq_global_overlay_item_file_name"),
|
||||||
|
)
|
||||||
|
op.create_index("ix_global_overlay_item_files_item", "global_overlay_item_files", ["item_id"])
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_index("ix_global_overlay_item_files_item", table_name="global_overlay_item_files")
|
||||||
|
op.drop_table("global_overlay_item_files")
|
||||||
|
op.drop_index("ix_global_overlay_items_source", table_name="global_overlay_items")
|
||||||
|
op.drop_table("global_overlay_items")
|
||||||
|
op.drop_index("ix_global_overlay_sources_type", table_name="global_overlay_sources")
|
||||||
|
op.drop_table("global_overlay_sources")
|
||||||
|
|
||||||
|
op.execute(
|
||||||
|
"DELETE FROM job_logs WHERE job_id IN "
|
||||||
|
"(SELECT id FROM jobs WHERE user_id IS NULL)"
|
||||||
|
)
|
||||||
|
op.execute("DELETE FROM jobs WHERE user_id IS NULL")
|
||||||
|
|
||||||
|
with op.batch_alter_table("jobs") as batch_op:
|
||||||
|
batch_op.alter_column("user_id", existing_type=sa.Integer(), nullable=False)
|
||||||
|
|
@ -0,0 +1,32 @@
|
||||||
|
"""drop legacy external overlay type
|
||||||
|
|
||||||
|
Revision ID: 0004_drop_legacy_external_overlay_type
|
||||||
|
Revises: 0003_global_map_overlays
|
||||||
|
Create Date: 2026-05-08
|
||||||
|
"""
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
|
||||||
|
revision: str = "0004_drop_legacy_external_overlay_type"
|
||||||
|
down_revision: Union[str, Sequence[str], None] = "0003_global_map_overlays"
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.execute(
|
||||||
|
"DELETE FROM jobs "
|
||||||
|
"WHERE overlay_id IN (SELECT id FROM overlays WHERE type = 'external')"
|
||||||
|
)
|
||||||
|
op.execute(
|
||||||
|
"DELETE FROM blueprint_overlays "
|
||||||
|
"WHERE overlay_id IN (SELECT id FROM overlays WHERE type = 'external')"
|
||||||
|
)
|
||||||
|
op.execute("DELETE FROM overlays WHERE type = 'external'")
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
# data is gone; intentional one-way migration
|
||||||
|
pass
|
||||||
79
l4d2web/alembic/versions/0005_script_overlays.py
Normal file
79
l4d2web/alembic/versions/0005_script_overlays.py
Normal file
|
|
@ -0,0 +1,79 @@
|
||||||
|
"""script overlays
|
||||||
|
|
||||||
|
Revision ID: 0005_script_overlays
|
||||||
|
Revises: 0004_drop_legacy_external_overlay_type
|
||||||
|
Create Date: 2026-05-08
|
||||||
|
"""
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision: str = "0005_script_overlays"
|
||||||
|
down_revision: Union[str, Sequence[str], None] = "0004_drop_legacy_external_overlay_type"
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
# 1. Wipe legacy global-type overlay rows and any references to them.
|
||||||
|
op.execute(
|
||||||
|
"DELETE FROM jobs "
|
||||||
|
"WHERE overlay_id IN (SELECT id FROM overlays "
|
||||||
|
"WHERE type IN ('l4d2center_maps', 'cedapug_maps'))"
|
||||||
|
)
|
||||||
|
op.execute(
|
||||||
|
"DELETE FROM blueprint_overlays "
|
||||||
|
"WHERE overlay_id IN (SELECT id FROM overlays "
|
||||||
|
"WHERE type IN ('l4d2center_maps', 'cedapug_maps'))"
|
||||||
|
)
|
||||||
|
op.execute(
|
||||||
|
"DELETE FROM overlay_workshop_items "
|
||||||
|
"WHERE overlay_id IN (SELECT id FROM overlays "
|
||||||
|
"WHERE type IN ('l4d2center_maps', 'cedapug_maps'))"
|
||||||
|
)
|
||||||
|
op.execute(
|
||||||
|
"DELETE FROM overlays WHERE type IN ('l4d2center_maps', 'cedapug_maps')"
|
||||||
|
)
|
||||||
|
|
||||||
|
# 2. Drop globals tables in FK order: item_files -> items -> sources.
|
||||||
|
op.drop_index(
|
||||||
|
"ix_global_overlay_item_files_item",
|
||||||
|
table_name="global_overlay_item_files",
|
||||||
|
)
|
||||||
|
op.drop_table("global_overlay_item_files")
|
||||||
|
|
||||||
|
op.drop_index(
|
||||||
|
"ix_global_overlay_items_source", table_name="global_overlay_items"
|
||||||
|
)
|
||||||
|
op.drop_table("global_overlay_items")
|
||||||
|
|
||||||
|
op.drop_index(
|
||||||
|
"ix_global_overlay_sources_type", table_name="global_overlay_sources"
|
||||||
|
)
|
||||||
|
op.drop_table("global_overlay_sources")
|
||||||
|
|
||||||
|
# 3. Add new columns on overlays.
|
||||||
|
with op.batch_alter_table("overlays") as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"script",
|
||||||
|
sa.Text(),
|
||||||
|
nullable=False,
|
||||||
|
server_default="",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"last_build_status",
|
||||||
|
sa.String(length=16),
|
||||||
|
nullable=False,
|
||||||
|
server_default="",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
# data is gone; intentional one-way migration
|
||||||
|
pass
|
||||||
44
l4d2web/alembic/versions/0006_server_name_per_user.py
Normal file
44
l4d2web/alembic/versions/0006_server_name_per_user.py
Normal file
|
|
@ -0,0 +1,44 @@
|
||||||
|
"""server name unique per user
|
||||||
|
|
||||||
|
Revision ID: 0006_server_name_per_user
|
||||||
|
Revises: 0005_script_overlays
|
||||||
|
Create Date: 2026-05-08
|
||||||
|
"""
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
|
||||||
|
revision: str = "0006_server_name_per_user"
|
||||||
|
down_revision: Union[str, Sequence[str], None] = "0005_script_overlays"
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
# 0001_initial defined `name` with column-level `unique=True`, which SQLite
|
||||||
|
# stored as an unnamed UNIQUE constraint. The naming_convention here lets
|
||||||
|
# batch_alter_table refer to it as "uq_servers_name" so we can drop it before
|
||||||
|
# recreating the table with the new (user_id, name) composite.
|
||||||
|
_NAMING_CONVENTION = {"uq": "uq_%(table_name)s_%(column_0_name)s"}
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
with op.batch_alter_table(
|
||||||
|
"servers",
|
||||||
|
naming_convention=_NAMING_CONVENTION,
|
||||||
|
recreate="always",
|
||||||
|
) as batch_op:
|
||||||
|
batch_op.drop_constraint("uq_servers_name", type_="unique")
|
||||||
|
batch_op.create_unique_constraint(
|
||||||
|
"uq_servers_user_name", ["user_id", "name"]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table(
|
||||||
|
"servers",
|
||||||
|
naming_convention=_NAMING_CONVENTION,
|
||||||
|
recreate="always",
|
||||||
|
) as batch_op:
|
||||||
|
batch_op.drop_constraint("uq_servers_user_name", type_="unique")
|
||||||
|
batch_op.create_unique_constraint("uq_servers_name", ["name"])
|
||||||
|
|
@ -16,6 +16,7 @@ from l4d2web.routes.log_routes import bp as log_bp
|
||||||
from l4d2web.routes.overlay_routes import bp as overlay_bp
|
from l4d2web.routes.overlay_routes import bp as overlay_bp
|
||||||
from l4d2web.routes.page_routes import bp as page_bp
|
from l4d2web.routes.page_routes import bp as page_bp
|
||||||
from l4d2web.routes.server_routes import bp as server_bp
|
from l4d2web.routes.server_routes import bp as server_bp
|
||||||
|
from l4d2web.routes.workshop_routes import bp as workshop_bp
|
||||||
from l4d2web.services.job_worker import recover_stale_jobs, start_job_workers
|
from l4d2web.services.job_worker import recover_stale_jobs, start_job_workers
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -38,7 +39,11 @@ def create_app(test_config: dict[str, object] | None = None) -> Flask:
|
||||||
if not app.config.get("TESTING") and (not secret_key or secret_key == "dev"):
|
if not app.config.get("TESTING") and (not secret_key or secret_key == "dev"):
|
||||||
raise RuntimeError("SECRET_KEY must be set to a non-default value outside of testing")
|
raise RuntimeError("SECRET_KEY must be set to a non-default value outside of testing")
|
||||||
|
|
||||||
app.config["SESSION_COOKIE_SECURE"] = not app.config.get("TESTING", False)
|
secure_env = os.getenv("SESSION_COOKIE_SECURE")
|
||||||
|
if secure_env is not None:
|
||||||
|
app.config["SESSION_COOKIE_SECURE"] = secure_env.lower() not in {"0", "false", "no"}
|
||||||
|
else:
|
||||||
|
app.config["SESSION_COOKIE_SECURE"] = not app.config.get("TESTING", False)
|
||||||
|
|
||||||
with app.app_context():
|
with app.app_context():
|
||||||
init_db()
|
init_db()
|
||||||
|
|
@ -65,6 +70,7 @@ def create_app(test_config: dict[str, object] | None = None) -> Flask:
|
||||||
app.before_request(load_current_user)
|
app.before_request(load_current_user)
|
||||||
app.register_blueprint(auth_bp)
|
app.register_blueprint(auth_bp)
|
||||||
app.register_blueprint(overlay_bp)
|
app.register_blueprint(overlay_bp)
|
||||||
|
app.register_blueprint(workshop_bp)
|
||||||
app.register_blueprint(blueprint_bp)
|
app.register_blueprint(blueprint_bp)
|
||||||
app.register_blueprint(server_bp)
|
app.register_blueprint(server_bp)
|
||||||
app.register_blueprint(job_bp)
|
app.register_blueprint(job_bp)
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
import os
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
import click
|
import click
|
||||||
from sqlalchemy.exc import IntegrityError
|
from sqlalchemy.exc import IntegrityError
|
||||||
|
|
@ -6,7 +7,11 @@ from sqlalchemy import select
|
||||||
|
|
||||||
from l4d2web.auth import hash_password
|
from l4d2web.auth import hash_password
|
||||||
from l4d2web.db import session_scope
|
from l4d2web.db import session_scope
|
||||||
from l4d2web.models import User
|
from l4d2web.models import Overlay, User
|
||||||
|
from l4d2web.services.overlay_creation import (
|
||||||
|
create_overlay_directory,
|
||||||
|
generate_overlay_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@click.command("promote-admin")
|
@click.command("promote-admin")
|
||||||
|
|
@ -41,6 +46,50 @@ def create_user(username: str, admin: bool) -> None:
|
||||||
click.echo(f"created user {username}")
|
click.echo(f"created user {username}")
|
||||||
|
|
||||||
|
|
||||||
|
@click.command("seed-script-overlays")
|
||||||
|
@click.argument(
|
||||||
|
"directory",
|
||||||
|
type=click.Path(exists=True, file_okay=False, path_type=Path),
|
||||||
|
)
|
||||||
|
def seed_script_overlays(directory: Path) -> None:
|
||||||
|
"""Upsert one system-wide script overlay per *.sh file in DIRECTORY.
|
||||||
|
|
||||||
|
Overlay name = filename stem; user_id stays NULL. Existing rows by name
|
||||||
|
have their script refreshed in place. Hard-errors if a name collides
|
||||||
|
with a non-script overlay.
|
||||||
|
"""
|
||||||
|
sh_files = sorted(p for p in directory.glob("*.sh") if p.stem)
|
||||||
|
if not sh_files:
|
||||||
|
click.echo(f"no *.sh files in {directory}", err=True)
|
||||||
|
return
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
for sh in sh_files:
|
||||||
|
name = sh.stem
|
||||||
|
content = sh.read_text()
|
||||||
|
existing = db.scalar(
|
||||||
|
select(Overlay).where(Overlay.name == name, Overlay.user_id.is_(None))
|
||||||
|
)
|
||||||
|
if existing is not None:
|
||||||
|
if existing.type != "script":
|
||||||
|
raise click.ClickException(
|
||||||
|
f"overlay {name!r} exists but is type={existing.type!r}, not script"
|
||||||
|
)
|
||||||
|
existing.script = content
|
||||||
|
click.echo(f"updated {name} (id={existing.id})")
|
||||||
|
else:
|
||||||
|
overlay = Overlay(
|
||||||
|
name=name, path="", type="script", user_id=None, script=content
|
||||||
|
)
|
||||||
|
db.add(overlay)
|
||||||
|
db.flush()
|
||||||
|
overlay.path = generate_overlay_path(overlay.id)
|
||||||
|
db.flush()
|
||||||
|
create_overlay_directory(overlay)
|
||||||
|
click.echo(f"created {name} (id={overlay.id})")
|
||||||
|
|
||||||
|
|
||||||
def register_cli(app) -> None:
|
def register_cli(app) -> None:
|
||||||
app.cli.add_command(promote_admin)
|
app.cli.add_command(promote_admin)
|
||||||
app.cli.add_command(create_user)
|
app.cli.add_command(create_user)
|
||||||
|
app.cli.add_command(seed_script_overlays)
|
||||||
|
|
|
||||||
|
|
@ -10,6 +10,8 @@ DEFAULT_CONFIG: dict[str, object] = {
|
||||||
"JOB_WORKER_POLL_SECONDS": 1,
|
"JOB_WORKER_POLL_SECONDS": 1,
|
||||||
"JOB_LOG_REPLAY_LIMIT": 2000,
|
"JOB_LOG_REPLAY_LIMIT": 2000,
|
||||||
"JOB_LOG_LINE_MAX_CHARS": 4096,
|
"JOB_LOG_LINE_MAX_CHARS": 4096,
|
||||||
|
"PORT_RANGE_START": 27015,
|
||||||
|
"PORT_RANGE_END": 27115,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -27,4 +29,6 @@ def load_config() -> dict[str, object]:
|
||||||
"JOB_WORKER_POLL_SECONDS": float(os.getenv("JOB_WORKER_POLL_SECONDS", "1")),
|
"JOB_WORKER_POLL_SECONDS": float(os.getenv("JOB_WORKER_POLL_SECONDS", "1")),
|
||||||
"JOB_LOG_REPLAY_LIMIT": int(os.getenv("JOB_LOG_REPLAY_LIMIT", "2000")),
|
"JOB_LOG_REPLAY_LIMIT": int(os.getenv("JOB_LOG_REPLAY_LIMIT", "2000")),
|
||||||
"JOB_LOG_LINE_MAX_CHARS": int(os.getenv("JOB_LOG_LINE_MAX_CHARS", "4096")),
|
"JOB_LOG_LINE_MAX_CHARS": int(os.getenv("JOB_LOG_LINE_MAX_CHARS", "4096")),
|
||||||
|
"PORT_RANGE_START": int(os.getenv("LEFT4ME_PORT_RANGE_START", "27015")),
|
||||||
|
"PORT_RANGE_END": int(os.getenv("LEFT4ME_PORT_RANGE_END", "27115")),
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,17 @@
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, String, Text
|
from sqlalchemy import (
|
||||||
|
BigInteger,
|
||||||
|
Boolean,
|
||||||
|
DateTime,
|
||||||
|
ForeignKey,
|
||||||
|
Index,
|
||||||
|
Integer,
|
||||||
|
String,
|
||||||
|
Text,
|
||||||
|
UniqueConstraint,
|
||||||
|
text,
|
||||||
|
)
|
||||||
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
|
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -25,14 +36,68 @@ class User(Base):
|
||||||
|
|
||||||
class Overlay(Base):
|
class Overlay(Base):
|
||||||
__tablename__ = "overlays"
|
__tablename__ = "overlays"
|
||||||
|
__table_args__ = (
|
||||||
|
Index(
|
||||||
|
"uq_overlay_name_system",
|
||||||
|
"name",
|
||||||
|
unique=True,
|
||||||
|
sqlite_where=text("user_id IS NULL"),
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"uq_overlay_name_per_user",
|
||||||
|
"name",
|
||||||
|
"user_id",
|
||||||
|
unique=True,
|
||||||
|
sqlite_where=text("user_id IS NOT NULL"),
|
||||||
|
),
|
||||||
|
Index("ix_overlays_type_user_id", "type", "user_id"),
|
||||||
|
{"sqlite_autoincrement": True},
|
||||||
|
)
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||||
name: Mapped[str] = mapped_column(String(128), unique=True, nullable=False)
|
name: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||||
path: Mapped[str] = mapped_column(String(512), nullable=False)
|
path: Mapped[str] = mapped_column(String(512), nullable=False)
|
||||||
|
type: Mapped[str] = mapped_column(String(16), nullable=False, default="workshop")
|
||||||
|
user_id: Mapped[int | None] = mapped_column(ForeignKey("users.id"), nullable=True)
|
||||||
|
script: Mapped[str] = mapped_column(Text, default="", nullable=False)
|
||||||
|
last_build_status: Mapped[str] = mapped_column(String(16), default="", nullable=False)
|
||||||
created_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)
|
created_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)
|
||||||
updated_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)
|
updated_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)
|
||||||
|
|
||||||
|
|
||||||
|
class WorkshopItem(Base):
|
||||||
|
__tablename__ = "workshop_items"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
steam_id: Mapped[str] = mapped_column(String(20), unique=True, nullable=False)
|
||||||
|
title: Mapped[str] = mapped_column(String(255), default="", nullable=False)
|
||||||
|
filename: Mapped[str] = mapped_column(String(255), default="", nullable=False)
|
||||||
|
file_url: Mapped[str] = mapped_column(Text, default="", nullable=False)
|
||||||
|
file_size: Mapped[int] = mapped_column(BigInteger, default=0, nullable=False)
|
||||||
|
time_updated: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||||
|
preview_url: Mapped[str] = mapped_column(Text, default="", nullable=False)
|
||||||
|
last_downloaded_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
|
||||||
|
last_error: Mapped[str] = mapped_column(Text, default="", nullable=False)
|
||||||
|
created_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)
|
||||||
|
updated_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)
|
||||||
|
|
||||||
|
|
||||||
|
class OverlayWorkshopItem(Base):
|
||||||
|
__tablename__ = "overlay_workshop_items"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint("overlay_id", "workshop_item_id", name="uq_overlay_workshop_item"),
|
||||||
|
Index("ix_owi_workshop_item", "workshop_item_id"),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
overlay_id: Mapped[int] = mapped_column(
|
||||||
|
ForeignKey("overlays.id", ondelete="CASCADE"), nullable=False
|
||||||
|
)
|
||||||
|
workshop_item_id: Mapped[int] = mapped_column(
|
||||||
|
ForeignKey("workshop_items.id", ondelete="RESTRICT"), nullable=False
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class Blueprint(Base):
|
class Blueprint(Base):
|
||||||
__tablename__ = "blueprints"
|
__tablename__ = "blueprints"
|
||||||
|
|
||||||
|
|
@ -58,11 +123,14 @@ class BlueprintOverlay(Base):
|
||||||
|
|
||||||
class Server(Base):
|
class Server(Base):
|
||||||
__tablename__ = "servers"
|
__tablename__ = "servers"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint("user_id", "name", name="uq_servers_user_name"),
|
||||||
|
)
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
|
user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
|
||||||
blueprint_id: Mapped[int] = mapped_column(ForeignKey("blueprints.id"), nullable=False)
|
blueprint_id: Mapped[int] = mapped_column(ForeignKey("blueprints.id"), nullable=False)
|
||||||
name: Mapped[str] = mapped_column(String(128), unique=True, nullable=False)
|
name: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||||
port: Mapped[int] = mapped_column(Integer, unique=True, nullable=False)
|
port: Mapped[int] = mapped_column(Integer, unique=True, nullable=False)
|
||||||
desired_state: Mapped[str] = mapped_column(String(16), default="stopped", nullable=False)
|
desired_state: Mapped[str] = mapped_column(String(16), default="stopped", nullable=False)
|
||||||
actual_state: Mapped[str] = mapped_column(String(16), default="unknown", nullable=False)
|
actual_state: Mapped[str] = mapped_column(String(16), default="unknown", nullable=False)
|
||||||
|
|
@ -76,8 +144,9 @@ class Job(Base):
|
||||||
__tablename__ = "jobs"
|
__tablename__ = "jobs"
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
|
user_id: Mapped[int | None] = mapped_column(ForeignKey("users.id"), nullable=True)
|
||||||
server_id: Mapped[int | None] = mapped_column(ForeignKey("servers.id"), nullable=True)
|
server_id: Mapped[int | None] = mapped_column(ForeignKey("servers.id"), nullable=True)
|
||||||
|
overlay_id: Mapped[int | None] = mapped_column(ForeignKey("overlays.id"), nullable=True)
|
||||||
operation: Mapped[str] = mapped_column(String(32), nullable=False)
|
operation: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||||
state: Mapped[str] = mapped_column(String(16), default="queued", nullable=False)
|
state: Mapped[str] = mapped_column(String(16), default="queued", nullable=False)
|
||||||
exit_code: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
exit_code: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
|
|
|
||||||
|
|
@ -14,6 +14,7 @@ dependencies = [
|
||||||
"alembic>=1.13",
|
"alembic>=1.13",
|
||||||
"PyYAML>=6.0",
|
"PyYAML>=6.0",
|
||||||
"gunicorn>=22.0",
|
"gunicorn>=22.0",
|
||||||
|
"requests>=2.31",
|
||||||
]
|
]
|
||||||
|
|
||||||
[tool.setuptools]
|
[tool.setuptools]
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@ from sqlalchemy import delete, func, select
|
||||||
from l4d2web.auth import current_user, require_login
|
from l4d2web.auth import current_user, require_login
|
||||||
from l4d2web.db import session_scope
|
from l4d2web.db import session_scope
|
||||||
from l4d2web.models import Blueprint as BlueprintModel
|
from l4d2web.models import Blueprint as BlueprintModel
|
||||||
from l4d2web.models import BlueprintOverlay, Server
|
from l4d2web.models import BlueprintOverlay, Overlay, Server
|
||||||
|
|
||||||
|
|
||||||
bp = Blueprint("blueprint", __name__)
|
bp = Blueprint("blueprint", __name__)
|
||||||
|
|
@ -37,6 +37,19 @@ def replace_blueprint_overlays(db, blueprint_id: int, overlay_ids: list[int]) ->
|
||||||
db.add(BlueprintOverlay(blueprint_id=blueprint_id, overlay_id=overlay_id, position=position))
|
db.add(BlueprintOverlay(blueprint_id=blueprint_id, overlay_id=overlay_id, position=position))
|
||||||
|
|
||||||
|
|
||||||
|
def overlay_ids_authorized(db, overlay_ids: list[int], user_id: int) -> bool:
|
||||||
|
unique_ids = set(overlay_ids)
|
||||||
|
if not unique_ids:
|
||||||
|
return True
|
||||||
|
allowed_count = db.scalar(
|
||||||
|
select(func.count(Overlay.id)).where(
|
||||||
|
Overlay.id.in_(unique_ids),
|
||||||
|
Overlay.user_id.is_(None) | (Overlay.user_id == user_id),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return allowed_count == len(unique_ids)
|
||||||
|
|
||||||
|
|
||||||
@bp.post("/blueprints")
|
@bp.post("/blueprints")
|
||||||
@require_login
|
@require_login
|
||||||
def create_blueprint() -> Response:
|
def create_blueprint() -> Response:
|
||||||
|
|
@ -61,6 +74,8 @@ def create_blueprint() -> Response:
|
||||||
return Response("name is required", status=400)
|
return Response("name is required", status=400)
|
||||||
|
|
||||||
with session_scope() as db:
|
with session_scope() as db:
|
||||||
|
if not overlay_ids_authorized(db, overlay_ids, user.id):
|
||||||
|
return Response("overlay not authorized", status=403)
|
||||||
blueprint = BlueprintModel(user_id=user.id, name=name, arguments=json.dumps(arguments), config=json.dumps(config))
|
blueprint = BlueprintModel(user_id=user.id, name=name, arguments=json.dumps(arguments), config=json.dumps(config))
|
||||||
db.add(blueprint)
|
db.add(blueprint)
|
||||||
db.flush()
|
db.flush()
|
||||||
|
|
@ -87,15 +102,39 @@ def update_blueprint_form(blueprint_id: int) -> Response:
|
||||||
)
|
)
|
||||||
if blueprint is None:
|
if blueprint is None:
|
||||||
return Response(status=404)
|
return Response(status=404)
|
||||||
|
overlay_ids = ordered_overlay_ids_from_form()
|
||||||
|
if not overlay_ids_authorized(db, overlay_ids, user.id):
|
||||||
|
return Response("overlay not authorized", status=403)
|
||||||
|
|
||||||
blueprint.name = name
|
blueprint.name = name
|
||||||
blueprint.arguments = json.dumps(split_textarea_lines(request.form.get("arguments", "")))
|
blueprint.arguments = json.dumps(split_textarea_lines(request.form.get("arguments", "")))
|
||||||
blueprint.config = json.dumps(split_textarea_lines(request.form.get("config", "")))
|
blueprint.config = json.dumps(split_textarea_lines(request.form.get("config", "")))
|
||||||
replace_blueprint_overlays(db, blueprint.id, ordered_overlay_ids_from_form())
|
replace_blueprint_overlays(db, blueprint.id, overlay_ids)
|
||||||
|
|
||||||
return redirect(f"/blueprints/{blueprint_id}")
|
return redirect(f"/blueprints/{blueprint_id}")
|
||||||
|
|
||||||
|
|
||||||
|
def _delete_blueprint(db, user_id: int, blueprint_id: int) -> Response | None:
|
||||||
|
blueprint = db.scalar(
|
||||||
|
select(BlueprintModel).where(
|
||||||
|
BlueprintModel.id == blueprint_id,
|
||||||
|
BlueprintModel.user_id == user_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if blueprint is None:
|
||||||
|
return Response(status=404)
|
||||||
|
|
||||||
|
linked_count = db.scalar(
|
||||||
|
select(func.count(Server.id)).where(Server.blueprint_id == blueprint.id)
|
||||||
|
) or 0
|
||||||
|
if linked_count > 0:
|
||||||
|
return Response("blueprint is in use", status=409)
|
||||||
|
|
||||||
|
db.execute(delete(BlueprintOverlay).where(BlueprintOverlay.blueprint_id == blueprint.id))
|
||||||
|
db.delete(blueprint)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
@bp.delete("/blueprints/<int:blueprint_id>")
|
@bp.delete("/blueprints/<int:blueprint_id>")
|
||||||
@require_login
|
@require_login
|
||||||
def delete_blueprint(blueprint_id: int) -> Response:
|
def delete_blueprint(blueprint_id: int) -> Response:
|
||||||
|
|
@ -103,22 +142,22 @@ def delete_blueprint(blueprint_id: int) -> Response:
|
||||||
assert user is not None
|
assert user is not None
|
||||||
|
|
||||||
with session_scope() as db:
|
with session_scope() as db:
|
||||||
blueprint = db.scalar(
|
error = _delete_blueprint(db, user.id, blueprint_id)
|
||||||
select(BlueprintModel).where(
|
if error is not None:
|
||||||
BlueprintModel.id == blueprint_id,
|
return error
|
||||||
BlueprintModel.user_id == user.id,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
if blueprint is None:
|
|
||||||
return Response(status=404)
|
|
||||||
|
|
||||||
linked_count = db.scalar(
|
|
||||||
select(func.count(Server.id)).where(Server.blueprint_id == blueprint.id)
|
|
||||||
) or 0
|
|
||||||
if linked_count > 0:
|
|
||||||
return Response("blueprint is in use", status=409)
|
|
||||||
|
|
||||||
db.execute(delete(BlueprintOverlay).where(BlueprintOverlay.blueprint_id == blueprint.id))
|
|
||||||
db.delete(blueprint)
|
|
||||||
|
|
||||||
return Response(status=204)
|
return Response(status=204)
|
||||||
|
|
||||||
|
|
||||||
|
@bp.post("/blueprints/<int:blueprint_id>/delete")
|
||||||
|
@require_login
|
||||||
|
def delete_blueprint_form(blueprint_id: int) -> Response:
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
error = _delete_blueprint(db, user.id, blueprint_id)
|
||||||
|
if error is not None:
|
||||||
|
return error
|
||||||
|
|
||||||
|
return redirect("/blueprints")
|
||||||
|
|
|
||||||
|
|
@ -22,7 +22,11 @@ def format_sse_event(seq: int, event: str, data: str) -> str:
|
||||||
|
|
||||||
|
|
||||||
def can_access_job(job: Job, user: User) -> bool:
|
def can_access_job(job: Job, user: User) -> bool:
|
||||||
return user.admin or job.user_id == user.id
|
if user.admin:
|
||||||
|
return True
|
||||||
|
if job.user_id is None:
|
||||||
|
return False
|
||||||
|
return job.user_id == user.id
|
||||||
|
|
||||||
|
|
||||||
@bp.get("/jobs/<int:job_id>")
|
@bp.get("/jobs/<int:job_id>")
|
||||||
|
|
@ -34,7 +38,7 @@ def job_detail(job_id: int) -> str | Response:
|
||||||
with session_scope() as db:
|
with session_scope() as db:
|
||||||
row = db.execute(
|
row = db.execute(
|
||||||
select(Job, User, Server)
|
select(Job, User, Server)
|
||||||
.join(User, User.id == Job.user_id)
|
.outerjoin(User, User.id == Job.user_id)
|
||||||
.outerjoin(Server, Server.id == Job.server_id)
|
.outerjoin(Server, Server.id == Job.server_id)
|
||||||
.where(Job.id == job_id)
|
.where(Job.id == job_id)
|
||||||
).first()
|
).first()
|
||||||
|
|
|
||||||
|
|
@ -5,6 +5,7 @@ from l4d2web.auth import current_user, require_login
|
||||||
from l4d2web.db import session_scope
|
from l4d2web.db import session_scope
|
||||||
from l4d2web.models import Server
|
from l4d2web.models import Server
|
||||||
from l4d2web.services import l4d2_facade as facade
|
from l4d2web.services import l4d2_facade as facade
|
||||||
|
from l4d2web.services.server_identity import server_unit_name
|
||||||
|
|
||||||
|
|
||||||
bp = Blueprint("logs", __name__)
|
bp = Blueprint("logs", __name__)
|
||||||
|
|
@ -27,7 +28,10 @@ def stream_server_logs(server_id: int) -> Response:
|
||||||
return Response(status=404)
|
return Response(status=404)
|
||||||
|
|
||||||
def generate():
|
def generate():
|
||||||
for line in facade.stream_server_logs(server.name, lines=200, follow=True):
|
for line in facade.stream_server_logs(server_unit_name(server.id), lines=200, follow=True):
|
||||||
yield f"data: {line}\n\n"
|
if line == "":
|
||||||
|
yield ": keepalive\n\n"
|
||||||
|
else:
|
||||||
|
yield f"data: {line}\n\n"
|
||||||
|
|
||||||
return Response(generate(), mimetype="text/event-stream")
|
return Response(generate(), mimetype="text/event-stream")
|
||||||
|
|
|
||||||
|
|
@ -1,72 +1,211 @@
|
||||||
|
import shutil
|
||||||
|
|
||||||
from flask import Blueprint, Response, redirect, request
|
from flask import Blueprint, Response, redirect, request
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select
|
||||||
|
|
||||||
from l4d2web.auth import require_admin
|
from l4d2host.paths import get_left4me_root
|
||||||
|
|
||||||
|
from l4d2web.auth import current_user, require_login
|
||||||
from l4d2web.db import session_scope
|
from l4d2web.db import session_scope
|
||||||
from l4d2web.models import BlueprintOverlay, Overlay
|
from l4d2web.models import BlueprintOverlay, Job, Overlay
|
||||||
from l4d2web.services.security import validate_overlay_ref
|
from l4d2web.services import overlay_builders
|
||||||
|
from l4d2web.services.job_worker import enqueue_build_overlay
|
||||||
|
from l4d2web.services.overlay_creation import (
|
||||||
|
create_overlay_directory,
|
||||||
|
generate_overlay_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
CREATABLE_OVERLAY_TYPES = {"workshop", "script"}
|
||||||
|
WIPE_SCRIPT = "find /overlay -mindepth 1 -delete"
|
||||||
|
|
||||||
|
|
||||||
bp = Blueprint("overlay", __name__)
|
bp = Blueprint("overlay", __name__)
|
||||||
|
|
||||||
|
|
||||||
@bp.post("/overlays")
|
def _is_managed_path(overlay: Overlay) -> bool:
|
||||||
@require_admin
|
return overlay.path == str(overlay.id)
|
||||||
def create_overlay() -> Response:
|
|
||||||
name = request.form.get("name", "").strip()
|
|
||||||
raw_path = request.form.get("path", "")
|
|
||||||
if not name or not raw_path:
|
|
||||||
return Response("missing fields", status=400)
|
|
||||||
|
|
||||||
try:
|
|
||||||
overlay_ref = validate_overlay_ref(raw_path)
|
def _can_edit_overlay(overlay: Overlay, user) -> bool:
|
||||||
except ValueError as exc:
|
if user is None:
|
||||||
return Response(str(exc), status=400)
|
return False
|
||||||
|
if user.admin:
|
||||||
|
return True
|
||||||
|
if overlay.type in {"workshop", "script"}:
|
||||||
|
return overlay.user_id == user.id
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _name_already_taken(db, name: str, scope_user_id: int | None, *, except_id: int | None = None) -> bool:
|
||||||
|
query = select(Overlay).where(Overlay.name == name)
|
||||||
|
if scope_user_id is None:
|
||||||
|
query = query.where(Overlay.user_id.is_(None))
|
||||||
|
else:
|
||||||
|
query = query.where(Overlay.user_id == scope_user_id)
|
||||||
|
if except_id is not None:
|
||||||
|
query = query.where(Overlay.id != except_id)
|
||||||
|
return db.scalar(query) is not None
|
||||||
|
|
||||||
|
|
||||||
|
@bp.post("/overlays")
|
||||||
|
@require_login
|
||||||
|
def create_overlay() -> Response:
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
|
|
||||||
|
name = request.form.get("name", "").strip()
|
||||||
|
overlay_type = request.form.get("type", "workshop").strip().lower()
|
||||||
|
system_wide = request.form.get("system_wide") == "1"
|
||||||
|
if not name:
|
||||||
|
return Response("missing fields", status=400)
|
||||||
|
if overlay_type not in CREATABLE_OVERLAY_TYPES:
|
||||||
|
return Response(f"unknown overlay type: {overlay_type}", status=400)
|
||||||
|
|
||||||
|
scope_user_id: int | None = None if (system_wide and user.admin) else user.id
|
||||||
|
|
||||||
with session_scope() as db:
|
with session_scope() as db:
|
||||||
existing = db.scalar(select(Overlay).where(Overlay.name == name))
|
if _name_already_taken(db, name, scope_user_id):
|
||||||
if existing is not None:
|
|
||||||
return Response("overlay already exists", status=409)
|
return Response("overlay already exists", status=409)
|
||||||
db.add(Overlay(name=name, path=overlay_ref))
|
|
||||||
|
|
||||||
return redirect("/overlays")
|
overlay = Overlay(name=name, path="", type=overlay_type, user_id=scope_user_id)
|
||||||
|
db.add(overlay)
|
||||||
|
db.flush()
|
||||||
|
overlay.path = generate_overlay_path(overlay.id)
|
||||||
|
db.flush()
|
||||||
|
create_overlay_directory(overlay)
|
||||||
|
new_id = overlay.id
|
||||||
|
|
||||||
|
return redirect(f"/overlays/{new_id}")
|
||||||
|
|
||||||
|
|
||||||
@bp.post("/overlays/<int:overlay_id>")
|
@bp.post("/overlays/<int:overlay_id>")
|
||||||
@require_admin
|
@require_login
|
||||||
def update_overlay(overlay_id: int) -> Response:
|
def update_overlay(overlay_id: int) -> Response:
|
||||||
name = request.form.get("name", "").strip()
|
user = current_user()
|
||||||
raw_path = request.form.get("path", "")
|
assert user is not None
|
||||||
if not name or not raw_path:
|
|
||||||
return Response("missing fields", status=400)
|
|
||||||
|
|
||||||
try:
|
name = request.form.get("name", "").strip()
|
||||||
overlay_ref = validate_overlay_ref(raw_path)
|
if not name:
|
||||||
except ValueError as exc:
|
return Response("missing fields", status=400)
|
||||||
return Response(str(exc), status=400)
|
|
||||||
|
|
||||||
with session_scope() as db:
|
with session_scope() as db:
|
||||||
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
|
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
|
||||||
if overlay is None:
|
if overlay is None:
|
||||||
return Response(status=404)
|
return Response(status=404)
|
||||||
duplicate = db.scalar(select(Overlay).where(Overlay.name == name, Overlay.id != overlay_id))
|
if not _can_edit_overlay(overlay, user):
|
||||||
if duplicate is not None:
|
return Response(status=403)
|
||||||
|
if _name_already_taken(db, name, overlay.user_id, except_id=overlay_id):
|
||||||
return Response("overlay already exists", status=409)
|
return Response("overlay already exists", status=409)
|
||||||
overlay.name = name
|
overlay.name = name
|
||||||
overlay.path = overlay_ref
|
|
||||||
|
|
||||||
return redirect("/overlays")
|
return redirect(f"/overlays/{overlay_id}")
|
||||||
|
|
||||||
|
|
||||||
@bp.post("/overlays/<int:overlay_id>/delete")
|
@bp.post("/overlays/<int:overlay_id>/delete")
|
||||||
@require_admin
|
@require_login
|
||||||
def delete_overlay(overlay_id: int) -> Response:
|
def delete_overlay(overlay_id: int) -> Response:
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
|
|
||||||
with session_scope() as db:
|
with session_scope() as db:
|
||||||
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
|
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
|
||||||
if overlay is None:
|
if overlay is None:
|
||||||
return Response(status=404)
|
return Response(status=404)
|
||||||
|
if not _can_edit_overlay(overlay, user):
|
||||||
|
return Response(status=403)
|
||||||
in_use = db.scalar(select(BlueprintOverlay).where(BlueprintOverlay.overlay_id == overlay_id))
|
in_use = db.scalar(select(BlueprintOverlay).where(BlueprintOverlay.overlay_id == overlay_id))
|
||||||
if in_use is not None:
|
if in_use is not None:
|
||||||
return Response("overlay is in use", status=409)
|
return Response("overlay is in use", status=409)
|
||||||
|
path_value = overlay.path
|
||||||
|
path_is_managed = _is_managed_path(overlay)
|
||||||
db.delete(overlay)
|
db.delete(overlay)
|
||||||
|
|
||||||
|
if path_is_managed and path_value:
|
||||||
|
target = get_left4me_root() / "overlays" / path_value
|
||||||
|
if target.exists():
|
||||||
|
shutil.rmtree(target)
|
||||||
|
|
||||||
return redirect("/overlays")
|
return redirect("/overlays")
|
||||||
|
|
||||||
|
|
||||||
|
def _load_script_overlay(db, overlay_id: int, user) -> tuple[Overlay | None, Response | None]:
|
||||||
|
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
|
||||||
|
if overlay is None:
|
||||||
|
return None, Response(status=404)
|
||||||
|
if overlay.type != "script":
|
||||||
|
return None, Response("not a script overlay", status=400)
|
||||||
|
if not _can_edit_overlay(overlay, user):
|
||||||
|
return None, Response(status=403)
|
||||||
|
return overlay, None
|
||||||
|
|
||||||
|
|
||||||
|
@bp.post("/overlays/<int:overlay_id>/script")
|
||||||
|
@require_login
|
||||||
|
def update_script(overlay_id: int) -> Response:
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
|
# HTML form submission of <textarea> uses CRLF line endings per spec; bash
|
||||||
|
# treats the trailing \r as part of each argument and breaks every command.
|
||||||
|
# Normalize to LF before storage so the script is well-formed when written
|
||||||
|
# to the sandbox tmpfile.
|
||||||
|
script_text = request.form.get("script", "").replace("\r\n", "\n").replace("\r", "\n")
|
||||||
|
with session_scope() as db:
|
||||||
|
overlay, err = _load_script_overlay(db, overlay_id, user)
|
||||||
|
if err is not None:
|
||||||
|
return err
|
||||||
|
overlay.script = script_text
|
||||||
|
job = enqueue_build_overlay(db, overlay_id=overlay_id, user_id=user.id)
|
||||||
|
job_id = job.id
|
||||||
|
return redirect(f"/jobs/{job_id}")
|
||||||
|
|
||||||
|
|
||||||
|
@bp.post("/overlays/<int:overlay_id>/build")
|
||||||
|
@require_login
|
||||||
|
def manual_build(overlay_id: int) -> Response:
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
|
with session_scope() as db:
|
||||||
|
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
|
||||||
|
if overlay is None:
|
||||||
|
return Response(status=404)
|
||||||
|
if not _can_edit_overlay(overlay, user):
|
||||||
|
return Response(status=403)
|
||||||
|
job = enqueue_build_overlay(db, overlay_id=overlay_id, user_id=user.id)
|
||||||
|
job_id = job.id
|
||||||
|
return redirect(f"/jobs/{job_id}")
|
||||||
|
|
||||||
|
|
||||||
|
@bp.post("/overlays/<int:overlay_id>/wipe")
|
||||||
|
@require_login
|
||||||
|
def wipe_overlay(overlay_id: int) -> Response:
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
|
with session_scope() as db:
|
||||||
|
overlay, err = _load_script_overlay(db, overlay_id, user)
|
||||||
|
if err is not None:
|
||||||
|
return err
|
||||||
|
running = db.scalar(
|
||||||
|
select(Job).where(
|
||||||
|
Job.operation == "build_overlay",
|
||||||
|
Job.overlay_id == overlay_id,
|
||||||
|
Job.state.in_({"running", "cancelling"}),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if running is not None:
|
||||||
|
return Response("build in progress for this overlay", status=409)
|
||||||
|
|
||||||
|
overlay_builders.run_sandboxed_script(
|
||||||
|
overlay_id,
|
||||||
|
WIPE_SCRIPT,
|
||||||
|
on_stdout=lambda _line: None,
|
||||||
|
on_stderr=lambda _line: None,
|
||||||
|
should_cancel=lambda: False,
|
||||||
|
)
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
|
||||||
|
if overlay is not None:
|
||||||
|
overlay.last_build_status = ""
|
||||||
|
|
||||||
|
return redirect(f"/overlays/{overlay_id}")
|
||||||
|
|
|
||||||
|
|
@ -1,12 +1,20 @@
|
||||||
import json
|
import json
|
||||||
|
|
||||||
from flask import Blueprint, Response, redirect, render_template
|
from flask import Blueprint, Response, redirect, render_template, request
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select
|
||||||
|
|
||||||
from l4d2web.auth import current_user, require_admin, require_login
|
from l4d2web.auth import current_user, require_admin, require_login
|
||||||
from l4d2web.db import session_scope
|
from l4d2web.db import session_scope
|
||||||
from l4d2web.models import Blueprint as BlueprintModel
|
from l4d2web.models import Blueprint as BlueprintModel
|
||||||
from l4d2web.models import BlueprintOverlay, Job, Overlay, Server, User
|
from l4d2web.models import (
|
||||||
|
BlueprintOverlay,
|
||||||
|
Job,
|
||||||
|
Overlay,
|
||||||
|
OverlayWorkshopItem,
|
||||||
|
Server,
|
||||||
|
User,
|
||||||
|
WorkshopItem,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
bp = Blueprint("pages", __name__)
|
bp = Blueprint("pages", __name__)
|
||||||
|
|
@ -48,7 +56,7 @@ def admin_jobs() -> str:
|
||||||
with session_scope() as db:
|
with session_scope() as db:
|
||||||
rows = db.execute(
|
rows = db.execute(
|
||||||
select(Job, User, Server)
|
select(Job, User, Server)
|
||||||
.join(User, User.id == Job.user_id)
|
.outerjoin(User, User.id == Job.user_id)
|
||||||
.outerjoin(Server, Server.id == Job.server_id)
|
.outerjoin(Server, Server.id == Job.server_id)
|
||||||
.order_by(Job.created_at.desc())
|
.order_by(Job.created_at.desc())
|
||||||
).all()
|
).all()
|
||||||
|
|
@ -70,7 +78,23 @@ def servers_page() -> str:
|
||||||
blueprints = db.scalars(
|
blueprints = db.scalars(
|
||||||
select(BlueprintModel).where(BlueprintModel.user_id == user.id).order_by(BlueprintModel.name)
|
select(BlueprintModel).where(BlueprintModel.user_id == user.id).order_by(BlueprintModel.name)
|
||||||
).all()
|
).all()
|
||||||
return render_template("servers.html", rows=rows, blueprints=blueprints)
|
|
||||||
|
prefill_blueprint_id: int | None = None
|
||||||
|
raw_prefill = request.args.get("blueprint_id")
|
||||||
|
if raw_prefill:
|
||||||
|
try:
|
||||||
|
candidate = int(raw_prefill)
|
||||||
|
except ValueError:
|
||||||
|
candidate = None
|
||||||
|
if candidate is not None and any(b.id == candidate for b in blueprints):
|
||||||
|
prefill_blueprint_id = candidate
|
||||||
|
|
||||||
|
return render_template(
|
||||||
|
"servers.html",
|
||||||
|
rows=rows,
|
||||||
|
blueprints=blueprints,
|
||||||
|
prefill_blueprint_id=prefill_blueprint_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@bp.get("/servers/<int:server_id>")
|
@bp.get("/servers/<int:server_id>")
|
||||||
|
|
@ -86,7 +110,7 @@ def server_detail(server_id: int):
|
||||||
blueprint = db.scalar(select(BlueprintModel).where(BlueprintModel.id == server.blueprint_id))
|
blueprint = db.scalar(select(BlueprintModel).where(BlueprintModel.id == server.blueprint_id))
|
||||||
recent_job_rows = db.execute(
|
recent_job_rows = db.execute(
|
||||||
select(Job, User, Server)
|
select(Job, User, Server)
|
||||||
.join(User, User.id == Job.user_id)
|
.outerjoin(User, User.id == Job.user_id)
|
||||||
.outerjoin(Server, Server.id == Job.server_id)
|
.outerjoin(Server, Server.id == Job.server_id)
|
||||||
.where(Job.server_id == server.id)
|
.where(Job.server_id == server.id)
|
||||||
.order_by(Job.created_at.desc())
|
.order_by(Job.created_at.desc())
|
||||||
|
|
@ -113,7 +137,7 @@ def server_jobs_page(server_id: int):
|
||||||
return Response(status=404)
|
return Response(status=404)
|
||||||
rows = db.execute(
|
rows = db.execute(
|
||||||
select(Job, User, Server)
|
select(Job, User, Server)
|
||||||
.join(User, User.id == Job.user_id)
|
.outerjoin(User, User.id == Job.user_id)
|
||||||
.outerjoin(Server, Server.id == Job.server_id)
|
.outerjoin(Server, Server.id == Job.server_id)
|
||||||
.where(Job.server_id == server.id)
|
.where(Job.server_id == server.id)
|
||||||
.order_by(Job.created_at.desc())
|
.order_by(Job.created_at.desc())
|
||||||
|
|
@ -125,11 +149,85 @@ def server_jobs_page(server_id: int):
|
||||||
@bp.get("/overlays")
|
@bp.get("/overlays")
|
||||||
@require_login
|
@require_login
|
||||||
def overlays() -> str:
|
def overlays() -> str:
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
with session_scope() as db:
|
with session_scope() as db:
|
||||||
overlays = db.scalars(select(Overlay).order_by(Overlay.name)).all()
|
query = select(Overlay).order_by(Overlay.name)
|
||||||
|
if not user.admin:
|
||||||
|
query = query.where(
|
||||||
|
Overlay.user_id.is_(None) | (Overlay.user_id == user.id)
|
||||||
|
)
|
||||||
|
overlays = db.scalars(query).all()
|
||||||
return render_template("overlays.html", overlays=overlays)
|
return render_template("overlays.html", overlays=overlays)
|
||||||
|
|
||||||
|
|
||||||
|
@bp.get("/overlays/<int:overlay_id>/jobs")
|
||||||
|
@require_login
|
||||||
|
def overlay_jobs_page(overlay_id: int):
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
|
with session_scope() as db:
|
||||||
|
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
|
||||||
|
if overlay is None:
|
||||||
|
return Response(status=404)
|
||||||
|
if not user.admin and overlay.user_id is not None and overlay.user_id != user.id:
|
||||||
|
return Response(status=403)
|
||||||
|
rows = db.execute(
|
||||||
|
select(Job, User, Server)
|
||||||
|
.outerjoin(User, User.id == Job.user_id)
|
||||||
|
.outerjoin(Server, Server.id == Job.server_id)
|
||||||
|
.where(Job.operation == "build_overlay", Job.overlay_id == overlay.id)
|
||||||
|
.order_by(Job.created_at.desc())
|
||||||
|
).all()
|
||||||
|
return render_template("overlay_jobs.html", overlay=overlay, rows=rows)
|
||||||
|
|
||||||
|
|
||||||
|
@bp.get("/overlays/<int:overlay_id>")
|
||||||
|
@require_login
|
||||||
|
def overlay_detail(overlay_id: int):
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
|
with session_scope() as db:
|
||||||
|
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
|
||||||
|
if overlay is None:
|
||||||
|
return Response(status=404)
|
||||||
|
if not user.admin and overlay.user_id is not None and overlay.user_id != user.id:
|
||||||
|
return Response(status=403)
|
||||||
|
using_blueprints_query = (
|
||||||
|
select(BlueprintModel)
|
||||||
|
.join(BlueprintOverlay, BlueprintOverlay.blueprint_id == BlueprintModel.id)
|
||||||
|
.where(BlueprintOverlay.overlay_id == overlay.id)
|
||||||
|
.order_by(BlueprintModel.name)
|
||||||
|
)
|
||||||
|
if not user.admin:
|
||||||
|
using_blueprints_query = using_blueprints_query.where(BlueprintModel.user_id == user.id)
|
||||||
|
using_blueprints = db.scalars(using_blueprints_query).all()
|
||||||
|
workshop_items = []
|
||||||
|
if overlay.type == "workshop":
|
||||||
|
workshop_items = db.scalars(
|
||||||
|
select(WorkshopItem)
|
||||||
|
.join(
|
||||||
|
OverlayWorkshopItem,
|
||||||
|
OverlayWorkshopItem.workshop_item_id == WorkshopItem.id,
|
||||||
|
)
|
||||||
|
.where(OverlayWorkshopItem.overlay_id == overlay.id)
|
||||||
|
.order_by(WorkshopItem.created_at)
|
||||||
|
).all()
|
||||||
|
latest_build_job = db.scalar(
|
||||||
|
select(Job)
|
||||||
|
.where(Job.operation == "build_overlay", Job.overlay_id == overlay.id)
|
||||||
|
.order_by(Job.created_at.desc())
|
||||||
|
.limit(1)
|
||||||
|
)
|
||||||
|
return render_template(
|
||||||
|
"overlay_detail.html",
|
||||||
|
overlay=overlay,
|
||||||
|
using_blueprints=using_blueprints,
|
||||||
|
workshop_items=workshop_items,
|
||||||
|
latest_build_job=latest_build_job,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@bp.get("/blueprints")
|
@bp.get("/blueprints")
|
||||||
@require_login
|
@require_login
|
||||||
def blueprints_page() -> str:
|
def blueprints_page() -> str:
|
||||||
|
|
@ -165,7 +263,11 @@ def blueprint_page(blueprint_id: int):
|
||||||
select(BlueprintOverlay.overlay_id, BlueprintOverlay.position)
|
select(BlueprintOverlay.overlay_id, BlueprintOverlay.position)
|
||||||
.where(BlueprintOverlay.blueprint_id == blueprint.id)
|
.where(BlueprintOverlay.blueprint_id == blueprint.id)
|
||||||
).all()
|
).all()
|
||||||
all_overlays = db.scalars(select(Overlay).order_by(Overlay.name)).all()
|
all_overlays = db.scalars(
|
||||||
|
select(Overlay)
|
||||||
|
.where(Overlay.user_id.is_(None) | (Overlay.user_id == user.id))
|
||||||
|
.order_by(Overlay.name)
|
||||||
|
).all()
|
||||||
|
|
||||||
overlay_positions = {overlay_id: position + 1 for overlay_id, position in position_rows}
|
overlay_positions = {overlay_id: position + 1 for overlay_id, position in position_rows}
|
||||||
return render_template(
|
return render_template(
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
from flask import Blueprint, Response, jsonify, redirect, request
|
from flask import Blueprint, Response, current_app, jsonify, redirect, request
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select
|
||||||
from sqlalchemy.exc import IntegrityError
|
from sqlalchemy.exc import IntegrityError
|
||||||
|
|
||||||
|
|
@ -6,12 +6,55 @@ from l4d2web.auth import current_user, require_login
|
||||||
from l4d2web.db import session_scope
|
from l4d2web.db import session_scope
|
||||||
from l4d2web.models import Blueprint as BlueprintModel
|
from l4d2web.models import Blueprint as BlueprintModel
|
||||||
from l4d2web.models import Job, Server
|
from l4d2web.models import Job, Server
|
||||||
from l4d2web.services.security import validate_instance_name
|
|
||||||
|
|
||||||
|
|
||||||
bp = Blueprint("server", __name__)
|
bp = Blueprint("server", __name__)
|
||||||
|
|
||||||
|
|
||||||
|
_NAME_MAX_LENGTH = 128
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_display_name(raw: object) -> str:
|
||||||
|
if not isinstance(raw, str):
|
||||||
|
raise ValueError("name must be a string")
|
||||||
|
cleaned = raw.strip()
|
||||||
|
if not cleaned:
|
||||||
|
raise ValueError("name must not be empty")
|
||||||
|
if len(cleaned) > _NAME_MAX_LENGTH:
|
||||||
|
raise ValueError("name too long")
|
||||||
|
return cleaned
|
||||||
|
|
||||||
|
|
||||||
|
def _allocate_next_port(db) -> int | None:
|
||||||
|
start = int(current_app.config["PORT_RANGE_START"])
|
||||||
|
end = int(current_app.config["PORT_RANGE_END"])
|
||||||
|
used = set(
|
||||||
|
db.scalars(
|
||||||
|
select(Server.port).where(Server.port >= start, Server.port <= end)
|
||||||
|
).all()
|
||||||
|
)
|
||||||
|
for port in range(start, end + 1):
|
||||||
|
if port not in used:
|
||||||
|
return port
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_port(payload, db) -> tuple[int | None, Response | None]:
|
||||||
|
raw = payload.get("port")
|
||||||
|
if raw is None or (isinstance(raw, str) and raw.strip() == ""):
|
||||||
|
port = _allocate_next_port(db)
|
||||||
|
if port is None:
|
||||||
|
return None, Response("no free port available", status=409)
|
||||||
|
return port, None
|
||||||
|
try:
|
||||||
|
port = int(raw)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None, Response("invalid port", status=400)
|
||||||
|
if not 1 <= port <= 65535:
|
||||||
|
return None, Response("invalid port", status=400)
|
||||||
|
return port, None
|
||||||
|
|
||||||
|
|
||||||
@bp.post("/servers")
|
@bp.post("/servers")
|
||||||
@require_login
|
@require_login
|
||||||
def create_server() -> Response:
|
def create_server() -> Response:
|
||||||
|
|
@ -21,7 +64,7 @@ def create_server() -> Response:
|
||||||
payload = request.get_json(silent=True) if json_response else request.form
|
payload = request.get_json(silent=True) if json_response else request.form
|
||||||
|
|
||||||
try:
|
try:
|
||||||
name = validate_instance_name(str(payload["name"]))
|
name = _validate_display_name(payload["name"])
|
||||||
except (KeyError, TypeError, ValueError):
|
except (KeyError, TypeError, ValueError):
|
||||||
return Response("invalid server name", status=400)
|
return Response("invalid server name", status=400)
|
||||||
|
|
||||||
|
|
@ -35,11 +78,15 @@ def create_server() -> Response:
|
||||||
if blueprint is None:
|
if blueprint is None:
|
||||||
return Response("blueprint not found", status=404)
|
return Response("blueprint not found", status=404)
|
||||||
|
|
||||||
|
port, error = _resolve_port(payload, db)
|
||||||
|
if error is not None:
|
||||||
|
return error
|
||||||
|
|
||||||
server = Server(
|
server = Server(
|
||||||
user_id=user.id,
|
user_id=user.id,
|
||||||
blueprint_id=blueprint.id,
|
blueprint_id=blueprint.id,
|
||||||
name=name,
|
name=name,
|
||||||
port=int(payload["port"]),
|
port=port,
|
||||||
desired_state="stopped",
|
desired_state="stopped",
|
||||||
actual_state="unknown",
|
actual_state="unknown",
|
||||||
last_error="",
|
last_error="",
|
||||||
|
|
@ -48,8 +95,11 @@ def create_server() -> Response:
|
||||||
|
|
||||||
try:
|
try:
|
||||||
db.flush()
|
db.flush()
|
||||||
except IntegrityError:
|
except IntegrityError as exc:
|
||||||
db.rollback()
|
db.rollback()
|
||||||
|
detail = str(exc.orig) if exc.orig is not None else str(exc)
|
||||||
|
if "servers.name" in detail:
|
||||||
|
return Response("name already in use", status=409)
|
||||||
return Response("port already in use", status=409)
|
return Response("port already in use", status=409)
|
||||||
|
|
||||||
server_id = server.id
|
server_id = server.id
|
||||||
|
|
@ -85,7 +135,7 @@ def update_server(server_id: int) -> Response:
|
||||||
return jsonify({"id": server_id}), 200
|
return jsonify({"id": server_id}), 200
|
||||||
|
|
||||||
|
|
||||||
LIFECYCLE_OPERATIONS = {"initialize", "start", "stop", "delete"}
|
LIFECYCLE_OPERATIONS = {"initialize", "start", "stop", "delete", "reset"}
|
||||||
|
|
||||||
|
|
||||||
@bp.post("/servers/<int:server_id>/<operation>")
|
@bp.post("/servers/<int:server_id>/<operation>")
|
||||||
|
|
@ -103,7 +153,9 @@ def enqueue_server_operation(server_id: int, operation: str) -> Response:
|
||||||
db.add(Job(user_id=user.id, server_id=server.id, operation=operation, state="queued"))
|
db.add(Job(user_id=user.id, server_id=server.id, operation=operation, state="queued"))
|
||||||
if operation == "start":
|
if operation == "start":
|
||||||
server.desired_state = "running"
|
server.desired_state = "running"
|
||||||
if operation in {"stop", "delete"}:
|
if operation in {"stop", "delete", "reset"}:
|
||||||
server.desired_state = "stopped"
|
server.desired_state = "stopped"
|
||||||
|
|
||||||
|
if operation == "delete":
|
||||||
|
return redirect("/servers")
|
||||||
return redirect(f"/servers/{server_id}")
|
return redirect(f"/servers/{server_id}")
|
||||||
|
|
|
||||||
137
l4d2web/routes/workshop_routes.py
Normal file
137
l4d2web/routes/workshop_routes.py
Normal file
|
|
@ -0,0 +1,137 @@
|
||||||
|
"""Routes for the workshop overlay type (add/remove items, manual rebuild,
|
||||||
|
admin global refresh)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from flask import Blueprint, Response, redirect, request
|
||||||
|
from sqlalchemy import delete as sa_delete
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from l4d2web.auth import current_user, require_admin, require_login
|
||||||
|
from l4d2web.db import session_scope
|
||||||
|
from l4d2web.models import (
|
||||||
|
Job,
|
||||||
|
Overlay,
|
||||||
|
OverlayWorkshopItem,
|
||||||
|
WorkshopItem,
|
||||||
|
)
|
||||||
|
from l4d2web.services import steam_workshop
|
||||||
|
from l4d2web.services.job_worker import enqueue_build_overlay
|
||||||
|
|
||||||
|
|
||||||
|
bp = Blueprint("workshop", __name__)
|
||||||
|
|
||||||
|
|
||||||
|
def _check_workshop_overlay_access(overlay_id: int, user, db):
|
||||||
|
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
|
||||||
|
if overlay is None:
|
||||||
|
return None, Response(status=404)
|
||||||
|
if overlay.type != "workshop":
|
||||||
|
return None, Response("not a workshop overlay", status=400)
|
||||||
|
if overlay.user_id != user.id and not user.admin:
|
||||||
|
return None, Response(status=403)
|
||||||
|
return overlay, None
|
||||||
|
|
||||||
|
|
||||||
|
@bp.post("/overlays/<int:overlay_id>/items")
|
||||||
|
@require_login
|
||||||
|
def add_items(overlay_id: int) -> Response:
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
|
|
||||||
|
raw_input = request.form.get("input", "").strip()
|
||||||
|
mode = request.form.get("input_mode", "items")
|
||||||
|
if not raw_input:
|
||||||
|
return Response("missing input", status=400)
|
||||||
|
|
||||||
|
try:
|
||||||
|
ids = steam_workshop.parse_workshop_input(raw_input)
|
||||||
|
except ValueError as exc:
|
||||||
|
return Response(str(exc), status=400)
|
||||||
|
|
||||||
|
if mode == "collection":
|
||||||
|
if len(ids) != 1:
|
||||||
|
return Response("collection mode expects exactly one id or url", status=400)
|
||||||
|
try:
|
||||||
|
ids = steam_workshop.resolve_collection(ids[0])
|
||||||
|
except Exception as exc:
|
||||||
|
return Response(f"failed to resolve collection: {exc}", status=502)
|
||||||
|
if not ids:
|
||||||
|
return Response("collection has no items", status=400)
|
||||||
|
|
||||||
|
try:
|
||||||
|
metas = steam_workshop.fetch_metadata_batch(ids, mode="add")
|
||||||
|
except steam_workshop.WorkshopValidationError as exc:
|
||||||
|
return Response(str(exc), status=400)
|
||||||
|
except Exception as exc:
|
||||||
|
return Response(f"steam api error: {exc}", status=502)
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
overlay, err = _check_workshop_overlay_access(overlay_id, user, db)
|
||||||
|
if err is not None:
|
||||||
|
return err
|
||||||
|
for meta in metas:
|
||||||
|
wi = db.scalar(select(WorkshopItem).where(WorkshopItem.steam_id == meta.steam_id))
|
||||||
|
if wi is None:
|
||||||
|
wi = WorkshopItem(steam_id=meta.steam_id)
|
||||||
|
db.add(wi)
|
||||||
|
wi.title = meta.title
|
||||||
|
wi.filename = meta.filename
|
||||||
|
wi.file_url = meta.file_url
|
||||||
|
wi.file_size = meta.file_size
|
||||||
|
wi.time_updated = meta.time_updated
|
||||||
|
wi.preview_url = meta.preview_url
|
||||||
|
wi.last_error = "" if meta.result == 1 else f"steam result {meta.result}"
|
||||||
|
db.flush()
|
||||||
|
|
||||||
|
existing = db.scalar(
|
||||||
|
select(OverlayWorkshopItem).where(
|
||||||
|
OverlayWorkshopItem.overlay_id == overlay_id,
|
||||||
|
OverlayWorkshopItem.workshop_item_id == wi.id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if existing is None:
|
||||||
|
db.add(OverlayWorkshopItem(overlay_id=overlay_id, workshop_item_id=wi.id))
|
||||||
|
|
||||||
|
job = enqueue_build_overlay(db, overlay_id=overlay_id, user_id=user.id)
|
||||||
|
job_id = job.id
|
||||||
|
|
||||||
|
return redirect(f"/jobs/{job_id}")
|
||||||
|
|
||||||
|
|
||||||
|
@bp.post("/overlays/<int:overlay_id>/items/<int:item_id>/delete")
|
||||||
|
@require_login
|
||||||
|
def remove_item(overlay_id: int, item_id: int) -> Response:
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
|
with session_scope() as db:
|
||||||
|
overlay, err = _check_workshop_overlay_access(overlay_id, user, db)
|
||||||
|
if err is not None:
|
||||||
|
return err
|
||||||
|
result = db.execute(
|
||||||
|
sa_delete(OverlayWorkshopItem).where(
|
||||||
|
OverlayWorkshopItem.overlay_id == overlay_id,
|
||||||
|
OverlayWorkshopItem.workshop_item_id == item_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if result.rowcount == 0:
|
||||||
|
return Response(status=404)
|
||||||
|
job = enqueue_build_overlay(db, overlay_id=overlay_id, user_id=user.id)
|
||||||
|
job_id = job.id
|
||||||
|
return redirect(f"/jobs/{job_id}")
|
||||||
|
|
||||||
|
|
||||||
|
@bp.post("/admin/workshop/refresh")
|
||||||
|
@require_admin
|
||||||
|
def admin_refresh() -> Response:
|
||||||
|
user = current_user()
|
||||||
|
assert user is not None
|
||||||
|
with session_scope() as db:
|
||||||
|
db.add(
|
||||||
|
Job(
|
||||||
|
user_id=user.id,
|
||||||
|
server_id=None,
|
||||||
|
operation="refresh_workshop_items",
|
||||||
|
state="queued",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return redirect("/admin/jobs")
|
||||||
|
|
@ -1,5 +1,6 @@
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
import os
|
import os
|
||||||
|
import select
|
||||||
import signal
|
import signal
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
|
@ -143,19 +144,37 @@ def run_command(
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
def stream_command(cmd: Sequence[str]) -> Iterator[str]:
|
def stream_command(cmd: Sequence[str], *, heartbeat_interval: float = 15.0) -> Iterator[str]:
|
||||||
|
# An empty string yielded between real lines is a heartbeat tick: it lets
|
||||||
|
# SSE callers emit a keepalive frame so a closed peer is detected, instead
|
||||||
|
# of blocking forever inside readline() when the child is silent.
|
||||||
proc = subprocess.Popen(
|
proc = subprocess.Popen(
|
||||||
list(cmd),
|
list(cmd),
|
||||||
stdout=subprocess.PIPE,
|
stdout=subprocess.PIPE,
|
||||||
stderr=subprocess.PIPE,
|
stderr=subprocess.PIPE,
|
||||||
text=True,
|
bufsize=0,
|
||||||
bufsize=1,
|
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
if proc.stdout is None:
|
if proc.stdout is None:
|
||||||
return
|
return
|
||||||
for raw in iter(proc.stdout.readline, ""):
|
fd = proc.stdout.fileno()
|
||||||
yield raw.rstrip("\n")
|
buffer = b""
|
||||||
|
while True:
|
||||||
|
ready, _, _ = select.select([fd], [], [], heartbeat_interval)
|
||||||
|
if not ready:
|
||||||
|
if proc.poll() is not None:
|
||||||
|
break
|
||||||
|
yield ""
|
||||||
|
continue
|
||||||
|
chunk = os.read(fd, 4096)
|
||||||
|
if not chunk:
|
||||||
|
break
|
||||||
|
buffer += chunk
|
||||||
|
while b"\n" in buffer:
|
||||||
|
line, buffer = buffer.split(b"\n", 1)
|
||||||
|
yield line.decode("utf-8", errors="replace")
|
||||||
|
if buffer:
|
||||||
|
yield buffer.decode("utf-8", errors="replace")
|
||||||
finally:
|
finally:
|
||||||
if proc.poll() is None:
|
if proc.poll() is None:
|
||||||
proc.terminate()
|
proc.terminate()
|
||||||
|
|
|
||||||
|
|
@ -10,13 +10,26 @@ from sqlalchemy.orm import Session
|
||||||
from typing import Callable
|
from typing import Callable
|
||||||
|
|
||||||
from l4d2web.db import session_scope
|
from l4d2web.db import session_scope
|
||||||
from l4d2web.models import Job, JobLog, Server
|
from l4d2web.models import (
|
||||||
|
Blueprint,
|
||||||
|
BlueprintOverlay,
|
||||||
|
Job,
|
||||||
|
JobLog,
|
||||||
|
Overlay,
|
||||||
|
OverlayWorkshopItem,
|
||||||
|
Server,
|
||||||
|
WorkshopItem,
|
||||||
|
)
|
||||||
from l4d2web.services.host_commands import CommandCancelledError
|
from l4d2web.services.host_commands import CommandCancelledError
|
||||||
|
from l4d2web.services.server_identity import server_unit_name
|
||||||
|
|
||||||
|
|
||||||
TERMINAL_JOB_STATES = {"succeeded", "failed", "cancelled"}
|
TERMINAL_JOB_STATES = {"succeeded", "failed", "cancelled"}
|
||||||
ACTIVE_JOB_STATES = {"running", "cancelling"}
|
ACTIVE_JOB_STATES = {"running", "cancelling"}
|
||||||
SERVER_OPERATIONS = {"initialize", "start", "stop", "delete"}
|
SERVER_OPERATIONS = {"initialize", "start", "stop", "delete", "reset"}
|
||||||
|
OVERLAY_OPERATIONS = {"build_overlay"}
|
||||||
|
GLOBAL_OPERATIONS = {"install", "refresh_workshop_items"}
|
||||||
|
WORKSHOP_REFRESH_DOWNLOAD_WORKERS = 1
|
||||||
|
|
||||||
_claim_lock = threading.Lock()
|
_claim_lock = threading.Lock()
|
||||||
_log_lock = threading.RLock()
|
_log_lock = threading.RLock()
|
||||||
|
|
@ -27,17 +40,55 @@ _workers_started = False
|
||||||
@dataclass
|
@dataclass
|
||||||
class SchedulerState:
|
class SchedulerState:
|
||||||
install_running: bool = False
|
install_running: bool = False
|
||||||
|
refresh_running: bool = False
|
||||||
running_servers: set[int] = field(default_factory=set)
|
running_servers: set[int] = field(default_factory=set)
|
||||||
|
running_overlays: set[int] = field(default_factory=set)
|
||||||
|
blocked_servers_by_overlay: set[int] = field(default_factory=set)
|
||||||
|
|
||||||
|
|
||||||
def can_start(job, state: SchedulerState) -> bool:
|
def can_start(job, state: SchedulerState) -> bool:
|
||||||
|
"""Truth table for the worker's claim policy.
|
||||||
|
|
||||||
|
install / refresh_workshop_items are global mutexes — they block each
|
||||||
|
other, all build_overlay jobs, and all server jobs.
|
||||||
|
|
||||||
|
build_overlay(overlay_id=N) is per-overlay: blocks on install/refresh, and
|
||||||
|
on another build for the same overlay. Different overlays may build
|
||||||
|
concurrently.
|
||||||
|
|
||||||
|
Server start/init blocks on install/refresh and on a build_overlay for any
|
||||||
|
overlay referenced by the server's blueprint.
|
||||||
|
"""
|
||||||
if job.operation == "install":
|
if job.operation == "install":
|
||||||
return (not state.install_running) and (len(state.running_servers) == 0)
|
return (
|
||||||
if state.install_running:
|
not state.install_running
|
||||||
|
and not state.refresh_running
|
||||||
|
and len(state.running_servers) == 0
|
||||||
|
and len(state.running_overlays) == 0
|
||||||
|
)
|
||||||
|
if job.operation == "refresh_workshop_items":
|
||||||
|
return (
|
||||||
|
not state.install_running
|
||||||
|
and not state.refresh_running
|
||||||
|
and len(state.running_servers) == 0
|
||||||
|
and len(state.running_overlays) == 0
|
||||||
|
)
|
||||||
|
if job.operation == "build_overlay":
|
||||||
|
if state.install_running or state.refresh_running:
|
||||||
|
return False
|
||||||
|
if job.overlay_id is None:
|
||||||
|
return False
|
||||||
|
return job.overlay_id not in state.running_overlays
|
||||||
|
# Server operations from here on.
|
||||||
|
if state.install_running or state.refresh_running:
|
||||||
return False
|
return False
|
||||||
if job.server_id is None:
|
if job.server_id is None:
|
||||||
return False
|
return False
|
||||||
return job.server_id not in state.running_servers
|
if job.server_id in state.running_servers:
|
||||||
|
return False
|
||||||
|
if job.server_id in state.blocked_servers_by_overlay:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
def build_scheduler_state(session: Session) -> SchedulerState:
|
def build_scheduler_state(session: Session) -> SchedulerState:
|
||||||
|
|
@ -46,8 +97,22 @@ def build_scheduler_state(session: Session) -> SchedulerState:
|
||||||
for job in running_jobs:
|
for job in running_jobs:
|
||||||
if job.operation == "install":
|
if job.operation == "install":
|
||||||
state.install_running = True
|
state.install_running = True
|
||||||
|
elif job.operation == "refresh_workshop_items":
|
||||||
|
state.refresh_running = True
|
||||||
|
elif job.operation == "build_overlay" and job.overlay_id is not None:
|
||||||
|
state.running_overlays.add(job.overlay_id)
|
||||||
elif job.server_id is not None:
|
elif job.server_id is not None:
|
||||||
state.running_servers.add(job.server_id)
|
state.running_servers.add(job.server_id)
|
||||||
|
|
||||||
|
if state.running_overlays:
|
||||||
|
rows = session.execute(
|
||||||
|
select(Server.id)
|
||||||
|
.join(Blueprint, Blueprint.id == Server.blueprint_id)
|
||||||
|
.join(BlueprintOverlay, BlueprintOverlay.blueprint_id == Blueprint.id)
|
||||||
|
.where(BlueprintOverlay.overlay_id.in_(state.running_overlays))
|
||||||
|
).all()
|
||||||
|
state.blocked_servers_by_overlay = {row[0] for row in rows}
|
||||||
|
|
||||||
return state
|
return state
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -65,8 +130,22 @@ def claim_next_job() -> int | None:
|
||||||
jobs = db.scalars(select(Job).where(Job.state == "queued").order_by(Job.created_at, Job.id)).all()
|
jobs = db.scalars(select(Job).where(Job.state == "queued").order_by(Job.created_at, Job.id)).all()
|
||||||
now = datetime.now(UTC)
|
now = datetime.now(UTC)
|
||||||
for job in jobs:
|
for job in jobs:
|
||||||
malformed_server_job = job.operation != "install" and job.server_id is None
|
malformed_server_job = (
|
||||||
if not malformed_server_job and not can_start(job, state):
|
job.operation in SERVER_OPERATIONS and job.server_id is None
|
||||||
|
)
|
||||||
|
malformed_overlay_job = (
|
||||||
|
job.operation in OVERLAY_OPERATIONS and job.overlay_id is None
|
||||||
|
)
|
||||||
|
if malformed_server_job or malformed_overlay_job:
|
||||||
|
# Mark malformed jobs failed immediately so the scheduler can move on.
|
||||||
|
job.state = "failed"
|
||||||
|
job.exit_code = 1
|
||||||
|
job.started_at = now
|
||||||
|
job.finished_at = now
|
||||||
|
job.updated_at = now
|
||||||
|
db.flush()
|
||||||
|
continue
|
||||||
|
if not can_start(job, state):
|
||||||
continue
|
continue
|
||||||
|
|
||||||
job.state = "running"
|
job.state = "running"
|
||||||
|
|
@ -78,6 +157,31 @@ def claim_next_job() -> int | None:
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def enqueue_build_overlay(session: Session, *, overlay_id: int, user_id: int) -> Job:
|
||||||
|
"""Insert a `build_overlay` job, coalescing against any already-queued
|
||||||
|
(not-yet-running) build for the same overlay. Running jobs are NOT
|
||||||
|
coalesced — a fresh add after a build started gets its own job."""
|
||||||
|
existing = session.scalar(
|
||||||
|
select(Job).where(
|
||||||
|
Job.operation == "build_overlay",
|
||||||
|
Job.overlay_id == overlay_id,
|
||||||
|
Job.state == "queued",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if existing is not None:
|
||||||
|
return existing
|
||||||
|
job = Job(
|
||||||
|
user_id=user_id,
|
||||||
|
server_id=None,
|
||||||
|
overlay_id=overlay_id,
|
||||||
|
operation="build_overlay",
|
||||||
|
state="queued",
|
||||||
|
)
|
||||||
|
session.add(job)
|
||||||
|
session.flush()
|
||||||
|
return job
|
||||||
|
|
||||||
|
|
||||||
def run_worker_once() -> bool:
|
def run_worker_once() -> bool:
|
||||||
job_id = claim_next_job()
|
job_id = claim_next_job()
|
||||||
if job_id is None:
|
if job_id is None:
|
||||||
|
|
@ -90,12 +194,14 @@ def run_job(job_id: int) -> None:
|
||||||
from l4d2web.services import l4d2_facade
|
from l4d2web.services import l4d2_facade
|
||||||
|
|
||||||
server_name = "unknown"
|
server_name = "unknown"
|
||||||
|
overlay_id_for_job: int | None = None
|
||||||
with session_scope() as db:
|
with session_scope() as db:
|
||||||
job = db.scalar(select(Job).where(Job.id == job_id))
|
job = db.scalar(select(Job).where(Job.id == job_id))
|
||||||
if job is None:
|
if job is None:
|
||||||
return
|
return
|
||||||
operation = job.operation
|
operation = job.operation
|
||||||
server_id = job.server_id
|
server_id = job.server_id
|
||||||
|
overlay_id_for_job = job.overlay_id
|
||||||
if server_id is not None:
|
if server_id is not None:
|
||||||
server = db.scalar(select(Server).where(Server.id == server_id))
|
server = db.scalar(select(Server).where(Server.id == server_id))
|
||||||
if server is not None:
|
if server is not None:
|
||||||
|
|
@ -133,6 +239,27 @@ def run_job(job_id: int) -> None:
|
||||||
on_stderr=on_stderr,
|
on_stderr=on_stderr,
|
||||||
should_cancel=should_cancel,
|
should_cancel=should_cancel,
|
||||||
)
|
)
|
||||||
|
elif operation == "refresh_workshop_items":
|
||||||
|
_run_with_boundaries(
|
||||||
|
"refresh",
|
||||||
|
"workshop items",
|
||||||
|
_run_refresh_workshop_items,
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
elif operation == "build_overlay":
|
||||||
|
if overlay_id_for_job is None:
|
||||||
|
raise ValueError("build_overlay job has no overlay_id")
|
||||||
|
_run_with_boundaries(
|
||||||
|
"build",
|
||||||
|
f"overlay {overlay_id_for_job}",
|
||||||
|
_run_build_overlay,
|
||||||
|
overlay_id_for_job,
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
elif operation in SERVER_OPERATIONS and server_id is None:
|
elif operation in SERVER_OPERATIONS and server_id is None:
|
||||||
raise ValueError(f"{operation} job has no server_id")
|
raise ValueError(f"{operation} job has no server_id")
|
||||||
elif operation == "initialize":
|
elif operation == "initialize":
|
||||||
|
|
@ -185,6 +312,25 @@ def run_job(job_id: int) -> None:
|
||||||
on_stderr=on_stderr,
|
on_stderr=on_stderr,
|
||||||
should_cancel=should_cancel,
|
should_cancel=should_cancel,
|
||||||
)
|
)
|
||||||
|
# Host-side cleanup succeeded; remove the DB row so the server
|
||||||
|
# disappears from /servers. Status refresh is skipped — the
|
||||||
|
# systemd unit is gone and querying it would just log noise.
|
||||||
|
with session_scope() as db:
|
||||||
|
server = db.scalar(select(Server).where(Server.id == server_id))
|
||||||
|
if server is not None:
|
||||||
|
db.delete(server)
|
||||||
|
finish_job(job_id, "succeeded", 0)
|
||||||
|
return
|
||||||
|
elif operation == "reset":
|
||||||
|
_run_with_boundaries(
|
||||||
|
"reset",
|
||||||
|
server_name,
|
||||||
|
l4d2_facade.reset_server,
|
||||||
|
server_id,
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
raise ValueError(f"unknown job operation: {operation}")
|
raise ValueError(f"unknown job operation: {operation}")
|
||||||
|
|
||||||
|
|
@ -213,6 +359,155 @@ def run_job(job_id: int) -> None:
|
||||||
finish_job(job_id, "failed", 1, error=error)
|
finish_job(job_id, "failed", 1, error=error)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_build_overlay(
|
||||||
|
overlay_id: int,
|
||||||
|
*,
|
||||||
|
on_stdout: Callable[[str], None],
|
||||||
|
on_stderr: Callable[[str], None],
|
||||||
|
should_cancel: Callable[[], bool],
|
||||||
|
) -> None:
|
||||||
|
"""Dispatch a build_overlay job through the builder registry."""
|
||||||
|
from l4d2web.services.overlay_builders import BUILDERS
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
overlay = db.scalar(select(Overlay).where(Overlay.id == overlay_id))
|
||||||
|
if overlay is None:
|
||||||
|
raise ValueError(f"overlay {overlay_id} not found")
|
||||||
|
builder = BUILDERS.get(overlay.type)
|
||||||
|
if builder is None:
|
||||||
|
raise ValueError(f"no builder registered for overlay type {overlay.type!r}")
|
||||||
|
# Detach overlay before leaving the session so the builder can read its
|
||||||
|
# attributes without a stale connection.
|
||||||
|
db.expunge(overlay)
|
||||||
|
|
||||||
|
builder.build(
|
||||||
|
overlay,
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_refresh_workshop_items(
|
||||||
|
*,
|
||||||
|
on_stdout: Callable[[str], None],
|
||||||
|
on_stderr: Callable[[str], None],
|
||||||
|
should_cancel: Callable[[], bool],
|
||||||
|
) -> list[int]:
|
||||||
|
"""Refresh metadata for every WorkshopItem, redownload changed items, and
|
||||||
|
enqueue (coalesced) build_overlay jobs for any overlay whose items had
|
||||||
|
`time_updated` advance or `filename` change. Returns the affected
|
||||||
|
overlay_ids for testability."""
|
||||||
|
from l4d2web.services import steam_workshop
|
||||||
|
from l4d2web.services.workshop_paths import workshop_cache_root
|
||||||
|
|
||||||
|
# Snapshot all WorkshopItems for the metadata batch.
|
||||||
|
with session_scope() as db:
|
||||||
|
items = db.scalars(select(WorkshopItem)).all()
|
||||||
|
snapshot = [
|
||||||
|
(it.id, it.steam_id, it.time_updated, it.filename) for it in items
|
||||||
|
]
|
||||||
|
|
||||||
|
if not snapshot:
|
||||||
|
on_stdout("no workshop items registered; nothing to refresh")
|
||||||
|
return []
|
||||||
|
|
||||||
|
steam_ids = [s for _, s, _, _ in snapshot]
|
||||||
|
on_stdout(f"fetching metadata for {len(steam_ids)} items")
|
||||||
|
metas = steam_workshop.fetch_metadata_batch(steam_ids, mode="refresh")
|
||||||
|
metas_by_id = {m.steam_id: m for m in metas}
|
||||||
|
on_stdout(f"metadata phase complete (received {len(metas)} entries)")
|
||||||
|
|
||||||
|
if should_cancel():
|
||||||
|
on_stderr("refresh cancelled after metadata phase")
|
||||||
|
return []
|
||||||
|
|
||||||
|
# Update DB rows + collect items that need (re)download.
|
||||||
|
affected_workshop_item_ids: set[int] = set()
|
||||||
|
download_metas: list[steam_workshop.WorkshopMetadata] = []
|
||||||
|
with session_scope() as db:
|
||||||
|
for item_id, steam_id, prior_time_updated, prior_filename in snapshot:
|
||||||
|
meta = metas_by_id.get(steam_id)
|
||||||
|
wi = db.scalar(select(WorkshopItem).where(WorkshopItem.id == item_id))
|
||||||
|
if wi is None:
|
||||||
|
continue
|
||||||
|
if meta is None:
|
||||||
|
# Steam dropped this item from the response.
|
||||||
|
wi.last_error = "steam returned no entry for this item"
|
||||||
|
continue
|
||||||
|
wi.title = meta.title
|
||||||
|
wi.filename = meta.filename
|
||||||
|
wi.file_url = meta.file_url
|
||||||
|
wi.file_size = meta.file_size
|
||||||
|
wi.preview_url = meta.preview_url
|
||||||
|
wi.last_error = "" if meta.result == 1 else f"steam result {meta.result}"
|
||||||
|
if meta.result != 1 or not meta.file_url:
|
||||||
|
continue
|
||||||
|
if (
|
||||||
|
meta.time_updated > prior_time_updated
|
||||||
|
or meta.filename != prior_filename
|
||||||
|
or wi.last_downloaded_at is None
|
||||||
|
):
|
||||||
|
wi.time_updated = meta.time_updated
|
||||||
|
affected_workshop_item_ids.add(item_id)
|
||||||
|
download_metas.append(meta)
|
||||||
|
|
||||||
|
on_stdout(f"downloading {len(download_metas)} items")
|
||||||
|
if download_metas:
|
||||||
|
report = steam_workshop.refresh_all(
|
||||||
|
download_metas,
|
||||||
|
workshop_cache_root(),
|
||||||
|
executor_workers=WORKSHOP_REFRESH_DOWNLOAD_WORKERS,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
on_stdout(
|
||||||
|
f"download phase complete (downloaded={report.downloaded} errors={report.errors})"
|
||||||
|
)
|
||||||
|
if report.errors:
|
||||||
|
for steam_id, err in report.per_item_errors.items():
|
||||||
|
on_stderr(f"download {steam_id}: {err}")
|
||||||
|
# Mark successfully downloaded items.
|
||||||
|
with session_scope() as db:
|
||||||
|
for meta in download_metas:
|
||||||
|
if meta.steam_id in report.per_item_errors:
|
||||||
|
continue
|
||||||
|
wi = db.scalar(select(WorkshopItem).where(WorkshopItem.steam_id == meta.steam_id))
|
||||||
|
if wi is not None:
|
||||||
|
wi.last_downloaded_at = datetime.now(UTC)
|
||||||
|
|
||||||
|
# Enqueue (coalesced) build_overlay for affected overlays.
|
||||||
|
if not affected_workshop_item_ids:
|
||||||
|
on_stdout("no overlays needed rebuilding")
|
||||||
|
return []
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
overlay_rows = db.execute(
|
||||||
|
select(OverlayWorkshopItem.overlay_id)
|
||||||
|
.where(OverlayWorkshopItem.workshop_item_id.in_(affected_workshop_item_ids))
|
||||||
|
.distinct()
|
||||||
|
).all()
|
||||||
|
affected_overlay_ids = [row[0] for row in overlay_rows]
|
||||||
|
for ov_id in affected_overlay_ids:
|
||||||
|
# Find a sensible owner for the auto-enqueued job: the overlay's
|
||||||
|
# user_id if private, else any admin (best effort) — fall back to
|
||||||
|
# the most recent existing job's user_id.
|
||||||
|
overlay = db.scalar(select(Overlay).where(Overlay.id == ov_id))
|
||||||
|
if overlay is None:
|
||||||
|
continue
|
||||||
|
user_id = overlay.user_id
|
||||||
|
if user_id is None:
|
||||||
|
# System overlay — pick any admin user; fall back to first user.
|
||||||
|
user_id = db.scalar(
|
||||||
|
select(Job.user_id).order_by(Job.created_at.desc()).limit(1)
|
||||||
|
)
|
||||||
|
if user_id is None:
|
||||||
|
continue
|
||||||
|
enqueue_build_overlay(db, overlay_id=ov_id, user_id=user_id)
|
||||||
|
|
||||||
|
on_stdout(f"enqueued build_overlay for {len(affected_overlay_ids)} overlay(s)")
|
||||||
|
return list(affected_overlay_ids)
|
||||||
|
|
||||||
|
|
||||||
def finish_job(job_id: int, state: str, exit_code: int | None, error: str = "") -> None:
|
def finish_job(job_id: int, state: str, exit_code: int | None, error: str = "") -> None:
|
||||||
now = datetime.now(UTC)
|
now = datetime.now(UTC)
|
||||||
with session_scope() as db:
|
with session_scope() as db:
|
||||||
|
|
@ -228,6 +523,11 @@ def finish_job(job_id: int, state: str, exit_code: int | None, error: str = "")
|
||||||
if server is not None:
|
if server is not None:
|
||||||
server.last_error = "" if state == "succeeded" else error
|
server.last_error = "" if state == "succeeded" else error
|
||||||
server.updated_at = now
|
server.updated_at = now
|
||||||
|
if job.operation == "build_overlay" and job.overlay_id is not None:
|
||||||
|
overlay = db.scalar(select(Overlay).where(Overlay.id == job.overlay_id))
|
||||||
|
if overlay is not None:
|
||||||
|
overlay.last_build_status = "ok" if state == "succeeded" else "failed"
|
||||||
|
overlay.updated_at = now
|
||||||
|
|
||||||
|
|
||||||
def append_job_log_line(job_id: int, stream: str, line: str, max_chars: int = 4096) -> int:
|
def append_job_log_line(job_id: int, stream: str, line: str, max_chars: int = 4096) -> int:
|
||||||
|
|
@ -271,7 +571,7 @@ def refresh_server_actual_state(server_id: int) -> str:
|
||||||
server = db.scalar(select(Server).where(Server.id == server_id))
|
server = db.scalar(select(Server).where(Server.id == server_id))
|
||||||
if server is None:
|
if server is None:
|
||||||
return "unknown"
|
return "unknown"
|
||||||
status = l4d2_facade.server_status(server.name)
|
status = l4d2_facade.server_status(server_unit_name(server.id))
|
||||||
server.actual_state = status.state
|
server.actual_state = status.state
|
||||||
server.actual_state_updated_at = now
|
server.actual_state_updated_at = now
|
||||||
server.updated_at = now
|
server.updated_at = now
|
||||||
|
|
|
||||||
|
|
@ -5,9 +5,18 @@ from pathlib import Path
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select
|
||||||
|
|
||||||
from l4d2web.db import session_scope
|
from l4d2web.db import session_scope
|
||||||
from l4d2web.models import Blueprint, BlueprintOverlay, Overlay, Server
|
from l4d2web.models import (
|
||||||
|
Blueprint,
|
||||||
|
BlueprintOverlay,
|
||||||
|
Overlay,
|
||||||
|
OverlayWorkshopItem,
|
||||||
|
Server,
|
||||||
|
WorkshopItem,
|
||||||
|
)
|
||||||
from l4d2web.services import host_commands
|
from l4d2web.services import host_commands
|
||||||
|
from l4d2web.services.server_identity import server_unit_name
|
||||||
from l4d2web.services.spec_yaml import write_temp_spec
|
from l4d2web.services.spec_yaml import write_temp_spec
|
||||||
|
from l4d2web.services.workshop_paths import cache_path
|
||||||
|
|
||||||
|
|
||||||
@dataclass(slots=True)
|
@dataclass(slots=True)
|
||||||
|
|
@ -57,10 +66,25 @@ def install_runtime(on_stdout=None, on_stderr=None, should_cancel=None) -> None:
|
||||||
|
|
||||||
def initialize_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
|
def initialize_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
|
||||||
server, blueprint, overlay_refs = load_server_blueprint_bundle(server_id)
|
server, blueprint, overlay_refs = load_server_blueprint_bundle(server_id)
|
||||||
|
|
||||||
|
# Run each overlay's builder synchronously so symlinks/dirs are present
|
||||||
|
# before l4d2ctl initialize composes the lowerdirs.
|
||||||
|
_run_blueprint_builders(
|
||||||
|
blueprint_id=blueprint.id,
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Workshop overlays may have items not yet downloaded. The builders skip
|
||||||
|
# them, but we don't want to mount a partial overlay silently — fail
|
||||||
|
# loudly with the missing IDs.
|
||||||
|
_check_workshop_overlay_caches(blueprint_id=blueprint.id)
|
||||||
|
|
||||||
spec_path = write_temp_spec(build_server_spec_payload(server, blueprint, overlay_refs))
|
spec_path = write_temp_spec(build_server_spec_payload(server, blueprint, overlay_refs))
|
||||||
try:
|
try:
|
||||||
host_commands.run_command(
|
host_commands.run_command(
|
||||||
["l4d2ctl", "initialize", server.name, "-f", str(spec_path)],
|
["l4d2ctl", "initialize", server_unit_name(server.id), "-f", str(spec_path)],
|
||||||
on_stdout=on_stdout,
|
on_stdout=on_stdout,
|
||||||
on_stderr=on_stderr,
|
on_stderr=on_stderr,
|
||||||
should_cancel=should_cancel,
|
should_cancel=should_cancel,
|
||||||
|
|
@ -69,10 +93,91 @@ def initialize_server(server_id: int, on_stdout=None, on_stderr=None, should_can
|
||||||
spec_path.unlink(missing_ok=True)
|
spec_path.unlink(missing_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_blueprint_builders(
|
||||||
|
*,
|
||||||
|
blueprint_id: int,
|
||||||
|
on_stdout=None,
|
||||||
|
on_stderr=None,
|
||||||
|
should_cancel=None,
|
||||||
|
) -> None:
|
||||||
|
"""Synchronously invoke each overlay's builder for the given blueprint."""
|
||||||
|
from l4d2web.services.overlay_builders import BUILDERS
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
overlays = db.scalars(
|
||||||
|
select(Overlay)
|
||||||
|
.join(BlueprintOverlay, BlueprintOverlay.overlay_id == Overlay.id)
|
||||||
|
.where(BlueprintOverlay.blueprint_id == blueprint_id)
|
||||||
|
.order_by(BlueprintOverlay.position)
|
||||||
|
).all()
|
||||||
|
for overlay in overlays:
|
||||||
|
db.expunge(overlay)
|
||||||
|
|
||||||
|
log_stdout = on_stdout if on_stdout is not None else (lambda _line: None)
|
||||||
|
log_stderr = on_stderr if on_stderr is not None else (lambda _line: None)
|
||||||
|
cancel = should_cancel if should_cancel is not None else (lambda: False)
|
||||||
|
|
||||||
|
for overlay in overlays:
|
||||||
|
builder = BUILDERS.get(overlay.type)
|
||||||
|
if builder is None:
|
||||||
|
raise ValueError(f"no builder registered for overlay type {overlay.type!r}")
|
||||||
|
builder.build(
|
||||||
|
overlay,
|
||||||
|
on_stdout=log_stdout,
|
||||||
|
on_stderr=log_stderr,
|
||||||
|
should_cancel=cancel,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _check_workshop_overlay_caches(*, blueprint_id: int) -> None:
|
||||||
|
"""Raise if any workshop overlay attached to this blueprint has items
|
||||||
|
that aren't yet in the workshop_cache. Mounting a partial overlay would
|
||||||
|
leave maps mysteriously missing in-game; surface the issue here instead.
|
||||||
|
"""
|
||||||
|
with session_scope() as db:
|
||||||
|
rows = db.execute(
|
||||||
|
select(Overlay.id, Overlay.name, WorkshopItem.steam_id)
|
||||||
|
.join(BlueprintOverlay, BlueprintOverlay.overlay_id == Overlay.id)
|
||||||
|
.join(
|
||||||
|
OverlayWorkshopItem,
|
||||||
|
OverlayWorkshopItem.overlay_id == Overlay.id,
|
||||||
|
)
|
||||||
|
.join(
|
||||||
|
WorkshopItem,
|
||||||
|
WorkshopItem.id == OverlayWorkshopItem.workshop_item_id,
|
||||||
|
)
|
||||||
|
.where(
|
||||||
|
BlueprintOverlay.blueprint_id == blueprint_id,
|
||||||
|
Overlay.type == "workshop",
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
|
||||||
|
missing: dict[tuple[int, str], list[str]] = {}
|
||||||
|
for overlay_id, overlay_name, steam_id in rows:
|
||||||
|
if not cache_path(steam_id).exists():
|
||||||
|
missing.setdefault((overlay_id, overlay_name), []).append(steam_id)
|
||||||
|
|
||||||
|
if not missing:
|
||||||
|
return
|
||||||
|
|
||||||
|
parts = []
|
||||||
|
for (overlay_id, overlay_name), steam_ids in missing.items():
|
||||||
|
ids = ", ".join(steam_ids)
|
||||||
|
parts.append(
|
||||||
|
f"overlay {overlay_name!r} (id={overlay_id}): items {ids} not yet downloaded"
|
||||||
|
)
|
||||||
|
detail = "; ".join(parts)
|
||||||
|
raise RuntimeError(
|
||||||
|
f"workshop content missing — {detail}. "
|
||||||
|
f"Open the overlay page and click Build (or wait for the auto-rebuild job), "
|
||||||
|
f"then retry."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def start_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
|
def start_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
|
||||||
server, _, _ = load_server_blueprint_bundle(server_id)
|
server, _, _ = load_server_blueprint_bundle(server_id)
|
||||||
host_commands.run_command(
|
host_commands.run_command(
|
||||||
["l4d2ctl", "start", server.name],
|
["l4d2ctl", "start", server_unit_name(server.id)],
|
||||||
on_stdout=on_stdout,
|
on_stdout=on_stdout,
|
||||||
on_stderr=on_stderr,
|
on_stderr=on_stderr,
|
||||||
should_cancel=should_cancel,
|
should_cancel=should_cancel,
|
||||||
|
|
@ -82,7 +187,7 @@ def start_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=N
|
||||||
def stop_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
|
def stop_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
|
||||||
server, _, _ = load_server_blueprint_bundle(server_id)
|
server, _, _ = load_server_blueprint_bundle(server_id)
|
||||||
host_commands.run_command(
|
host_commands.run_command(
|
||||||
["l4d2ctl", "stop", server.name],
|
["l4d2ctl", "stop", server_unit_name(server.id)],
|
||||||
on_stdout=on_stdout,
|
on_stdout=on_stdout,
|
||||||
on_stderr=on_stderr,
|
on_stderr=on_stderr,
|
||||||
should_cancel=should_cancel,
|
should_cancel=should_cancel,
|
||||||
|
|
@ -92,15 +197,25 @@ def stop_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=No
|
||||||
def delete_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
|
def delete_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
|
||||||
server, _, _ = load_server_blueprint_bundle(server_id)
|
server, _, _ = load_server_blueprint_bundle(server_id)
|
||||||
host_commands.run_command(
|
host_commands.run_command(
|
||||||
["l4d2ctl", "delete", server.name],
|
["l4d2ctl", "delete", server_unit_name(server.id)],
|
||||||
on_stdout=on_stdout,
|
on_stdout=on_stdout,
|
||||||
on_stderr=on_stderr,
|
on_stderr=on_stderr,
|
||||||
should_cancel=should_cancel,
|
should_cancel=should_cancel,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def server_status(server_name: str) -> ServerStatus:
|
def reset_server(server_id: int, on_stdout=None, on_stderr=None, should_cancel=None) -> None:
|
||||||
result = host_commands.run_command(["l4d2ctl", "status", server_name, "--json"])
|
server, _, _ = load_server_blueprint_bundle(server_id)
|
||||||
|
host_commands.run_command(
|
||||||
|
["l4d2ctl", "reset", server_unit_name(server.id)],
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def server_status(unit_name: str) -> ServerStatus:
|
||||||
|
result = host_commands.run_command(["l4d2ctl", "status", unit_name, "--json"])
|
||||||
payload = json.loads(result.stdout or "{}")
|
payload = json.loads(result.stdout or "{}")
|
||||||
return ServerStatus(
|
return ServerStatus(
|
||||||
state=str(payload.get("state", "unknown")),
|
state=str(payload.get("state", "unknown")),
|
||||||
|
|
@ -109,7 +224,7 @@ def server_status(server_name: str) -> ServerStatus:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def stream_server_logs(server_name: str, *, lines: int = 200, follow: bool = True):
|
def stream_server_logs(unit_name: str, *, lines: int = 200, follow: bool = True):
|
||||||
command = ["l4d2ctl", "logs", server_name, "--lines", str(lines)]
|
command = ["l4d2ctl", "logs", unit_name, "--lines", str(lines)]
|
||||||
command.append("--follow" if follow else "--no-follow")
|
command.append("--follow" if follow else "--no-follow")
|
||||||
return host_commands.stream_command(command)
|
return host_commands.stream_command(command)
|
||||||
|
|
|
||||||
286
l4d2web/services/overlay_builders.py
Normal file
286
l4d2web/services/overlay_builders.py
Normal file
|
|
@ -0,0 +1,286 @@
|
||||||
|
"""Overlay builder registry.
|
||||||
|
|
||||||
|
Each `Overlay.type` maps to a builder. `build_overlay(overlay_id)` jobs (and
|
||||||
|
the synchronous `initialize_server` hook) dispatch through `BUILDERS`. Adding
|
||||||
|
a new overlay type means writing a new builder and registering it here — no
|
||||||
|
changes to the worker, the mount layer, or the blueprint editor.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Callable, Protocol
|
||||||
|
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from l4d2host.paths import get_left4me_root
|
||||||
|
|
||||||
|
from l4d2web.db import session_scope
|
||||||
|
from l4d2web.models import Overlay, OverlayWorkshopItem, WorkshopItem
|
||||||
|
from l4d2web.services.host_commands import run_command
|
||||||
|
from l4d2web.services.workshop_paths import cache_path, workshop_cache_root
|
||||||
|
|
||||||
|
|
||||||
|
CancelCheck = Callable[[], bool]
|
||||||
|
LogSink = Callable[[str], None]
|
||||||
|
|
||||||
|
|
||||||
|
SCRIPT_SANDBOX_HELPER = "/usr/local/libexec/left4me/left4me-script-sandbox"
|
||||||
|
DISK_BUDGET_BYTES = 20 * 1024**3
|
||||||
|
|
||||||
|
|
||||||
|
def _sandbox_script_dir() -> Path:
|
||||||
|
"""Where script tmpfiles live before being bind-mounted into the sandbox.
|
||||||
|
|
||||||
|
Cannot live in /tmp because the web service unit has PrivateTmp=yes:
|
||||||
|
its /tmp is a per-instance namespace that PID 1 (which actually performs
|
||||||
|
the BindReadOnlyPaths during sandbox setup) cannot resolve. /var/lib is
|
||||||
|
not affected by PrivateTmp and is visible to PID 1, so the bind-mount
|
||||||
|
succeeds.
|
||||||
|
"""
|
||||||
|
return get_left4me_root() / "sandbox-scripts"
|
||||||
|
|
||||||
|
|
||||||
|
class BuildError(RuntimeError):
|
||||||
|
"""Raised by builders when a build fails for a builder-specific reason
|
||||||
|
(e.g. disk-budget exceeded). Distinct from subprocess-level
|
||||||
|
HostCommandError / CommandCancelledError."""
|
||||||
|
|
||||||
|
|
||||||
|
class OverlayBuilder(Protocol):
|
||||||
|
def build(
|
||||||
|
self,
|
||||||
|
overlay: Overlay,
|
||||||
|
*,
|
||||||
|
on_stdout: LogSink,
|
||||||
|
on_stderr: LogSink,
|
||||||
|
should_cancel: CancelCheck,
|
||||||
|
) -> None: ...
|
||||||
|
|
||||||
|
|
||||||
|
def _overlay_root(overlay: Overlay) -> Path:
|
||||||
|
return get_left4me_root() / "overlays" / overlay.path
|
||||||
|
|
||||||
|
|
||||||
|
def overlay_path_for_id(overlay_id: int) -> Path:
|
||||||
|
return get_left4me_root() / "overlays" / str(overlay_id)
|
||||||
|
|
||||||
|
|
||||||
|
class WorkshopBuilder:
|
||||||
|
"""Diff-apply symlinks under `left4dead2/addons/` against the overlay's
|
||||||
|
current `WorkshopItem` associations. Cached items get an absolute symlink
|
||||||
|
into `workshop_cache/{steam_id}.vpk`. Items missing from cache are
|
||||||
|
skipped with a warning rather than turned into broken symlinks."""
|
||||||
|
|
||||||
|
def build(
|
||||||
|
self,
|
||||||
|
overlay: Overlay,
|
||||||
|
*,
|
||||||
|
on_stdout: LogSink,
|
||||||
|
on_stderr: LogSink,
|
||||||
|
should_cancel: CancelCheck,
|
||||||
|
) -> None:
|
||||||
|
addons_dir = _overlay_root(overlay) / "left4dead2" / "addons"
|
||||||
|
addons_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
items = db.scalars(
|
||||||
|
select(WorkshopItem)
|
||||||
|
.join(
|
||||||
|
OverlayWorkshopItem,
|
||||||
|
OverlayWorkshopItem.workshop_item_id == WorkshopItem.id,
|
||||||
|
)
|
||||||
|
.where(OverlayWorkshopItem.overlay_id == overlay.id)
|
||||||
|
).all()
|
||||||
|
# Detach items so we can use them outside the session.
|
||||||
|
items_data = [
|
||||||
|
(it.steam_id, it.last_downloaded_at) for it in items
|
||||||
|
]
|
||||||
|
|
||||||
|
cache_root = workshop_cache_root()
|
||||||
|
# desired: symlink-name -> absolute target path (only for cached items)
|
||||||
|
desired: dict[str, Path] = {}
|
||||||
|
skipped: list[str] = []
|
||||||
|
for steam_id, last_downloaded_at in items_data:
|
||||||
|
target = cache_path(steam_id)
|
||||||
|
if last_downloaded_at is None or not target.exists():
|
||||||
|
skipped.append(steam_id)
|
||||||
|
continue
|
||||||
|
desired[f"{steam_id}.vpk"] = target.resolve()
|
||||||
|
|
||||||
|
if should_cancel():
|
||||||
|
on_stderr("workshop build cancelled before applying symlinks")
|
||||||
|
return
|
||||||
|
|
||||||
|
# existing: symlink-name -> link target (only for symlinks pointing at our cache)
|
||||||
|
existing: dict[str, Path] = {}
|
||||||
|
for entry in os.scandir(addons_dir):
|
||||||
|
if not entry.is_symlink():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
target = Path(os.readlink(entry.path))
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
resolved = target.resolve(strict=False)
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
if not _is_under(resolved, cache_root):
|
||||||
|
continue
|
||||||
|
existing[entry.name] = resolved
|
||||||
|
|
||||||
|
created = 0
|
||||||
|
removed = 0
|
||||||
|
unchanged = 0
|
||||||
|
|
||||||
|
# Remove obsolete or stale symlinks first.
|
||||||
|
for name, current_target in existing.items():
|
||||||
|
if should_cancel():
|
||||||
|
on_stderr("workshop build cancelled mid-removal")
|
||||||
|
return
|
||||||
|
desired_target = desired.get(name)
|
||||||
|
if desired_target is None:
|
||||||
|
os.unlink(addons_dir / name)
|
||||||
|
removed += 1
|
||||||
|
elif current_target != desired_target:
|
||||||
|
os.unlink(addons_dir / name)
|
||||||
|
# will be recreated below
|
||||||
|
else:
|
||||||
|
unchanged += 1
|
||||||
|
|
||||||
|
# Recompute existing post-removal so the create loop knows what's left.
|
||||||
|
post_removal_existing = {
|
||||||
|
name for name in existing if name in desired and existing[name] == desired[name]
|
||||||
|
}
|
||||||
|
|
||||||
|
# Create new symlinks.
|
||||||
|
for name, target in desired.items():
|
||||||
|
if should_cancel():
|
||||||
|
on_stderr("workshop build cancelled mid-creation")
|
||||||
|
return
|
||||||
|
if name in post_removal_existing:
|
||||||
|
continue
|
||||||
|
link_path = addons_dir / name
|
||||||
|
# Defensive: if a non-symlink file collides with our name, leave it.
|
||||||
|
if link_path.exists() and not link_path.is_symlink():
|
||||||
|
on_stderr(
|
||||||
|
f"refusing to overwrite non-symlink at {link_path}; manual intervention required"
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
if link_path.is_symlink():
|
||||||
|
# An obsolete symlink not in `existing` (target outside cache).
|
||||||
|
# We don't manage these — leave alone.
|
||||||
|
on_stderr(
|
||||||
|
f"refusing to overwrite foreign symlink at {link_path}"
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
os.symlink(str(target), str(link_path))
|
||||||
|
created += 1
|
||||||
|
|
||||||
|
on_stdout(
|
||||||
|
f"workshop overlay {overlay.name!r}: created={created} "
|
||||||
|
f"removed={removed} unchanged={unchanged} "
|
||||||
|
f"skipped(uncached)={len(skipped)}"
|
||||||
|
)
|
||||||
|
for steam_id in skipped:
|
||||||
|
on_stderr(
|
||||||
|
f"workshop item {steam_id} skipped: not yet downloaded "
|
||||||
|
f"(refresh required before this overlay can mount it)"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def run_sandboxed_script(
|
||||||
|
overlay_id: int,
|
||||||
|
script_text: str,
|
||||||
|
*,
|
||||||
|
on_stdout: LogSink,
|
||||||
|
on_stderr: LogSink,
|
||||||
|
should_cancel: CancelCheck,
|
||||||
|
) -> None:
|
||||||
|
"""Write `script_text` to a tmpfile and exec it inside the privileged
|
||||||
|
sandbox helper. Used by ScriptBuilder.build and by the wipe route."""
|
||||||
|
script_dir = _sandbox_script_dir()
|
||||||
|
script_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
with tempfile.NamedTemporaryFile(
|
||||||
|
"w", suffix=".sh", delete=False, dir=str(script_dir)
|
||||||
|
) as f:
|
||||||
|
f.write(script_text or "")
|
||||||
|
script_path = f.name
|
||||||
|
# NamedTemporaryFile creates 0600 owned by the web user; the sandbox runs
|
||||||
|
# as l4d2-sandbox and needs to read it (bind-mounted at /script.sh inside
|
||||||
|
# the sandbox). Script content is not a secret — it's plain bash stored
|
||||||
|
# in the DB and editable by the user — so 0644 is appropriate.
|
||||||
|
os.chmod(script_path, 0o644)
|
||||||
|
try:
|
||||||
|
cmd = [
|
||||||
|
"sudo",
|
||||||
|
"-n",
|
||||||
|
SCRIPT_SANDBOX_HELPER,
|
||||||
|
str(overlay_id),
|
||||||
|
script_path,
|
||||||
|
]
|
||||||
|
run_command(
|
||||||
|
cmd,
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
os.unlink(script_path)
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class ScriptBuilder:
|
||||||
|
"""Run an arbitrary user-authored bash script against the overlay dir
|
||||||
|
inside a bubblewrap + systemd-run sandbox. The script sees the overlay
|
||||||
|
dir as RW `/overlay` and a curated host RO mount; everything else is
|
||||||
|
isolated. After exit, enforce a 20 GB cap on `du -sb /overlay`."""
|
||||||
|
|
||||||
|
def build(
|
||||||
|
self,
|
||||||
|
overlay: Overlay,
|
||||||
|
*,
|
||||||
|
on_stdout: LogSink,
|
||||||
|
on_stderr: LogSink,
|
||||||
|
should_cancel: CancelCheck,
|
||||||
|
) -> None:
|
||||||
|
# Ensure target dir exists so the helper's bind-mount validation passes.
|
||||||
|
overlay_path_for_id(overlay.id).mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
run_sandboxed_script(
|
||||||
|
overlay.id,
|
||||||
|
overlay.script or "",
|
||||||
|
on_stdout=on_stdout,
|
||||||
|
on_stderr=on_stderr,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
self._enforce_disk_budget(overlay.id, on_stderr)
|
||||||
|
|
||||||
|
def _enforce_disk_budget(self, overlay_id: int, on_stderr: LogSink) -> None:
|
||||||
|
target = overlay_path_for_id(overlay_id)
|
||||||
|
size_output = subprocess.check_output(["du", "-sb", str(target)])
|
||||||
|
size_bytes = int(size_output.split()[0])
|
||||||
|
if size_bytes > DISK_BUDGET_BYTES:
|
||||||
|
on_stderr(
|
||||||
|
f"overlay exceeded 20 GB disk cap: {size_bytes} bytes > "
|
||||||
|
f"{DISK_BUDGET_BYTES} bytes"
|
||||||
|
)
|
||||||
|
raise BuildError("disk-cap-exceeded")
|
||||||
|
|
||||||
|
|
||||||
|
def _is_under(path: Path, root: Path) -> bool:
|
||||||
|
try:
|
||||||
|
path.relative_to(root)
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
BUILDERS: dict[str, OverlayBuilder] = {
|
||||||
|
"workshop": WorkshopBuilder(),
|
||||||
|
"script": ScriptBuilder(),
|
||||||
|
}
|
||||||
35
l4d2web/services/overlay_creation.py
Normal file
35
l4d2web/services/overlay_creation.py
Normal file
|
|
@ -0,0 +1,35 @@
|
||||||
|
"""Overlay path generation and on-disk directory bootstrap.
|
||||||
|
|
||||||
|
All new overlays (any type) get `path = str(overlay_id)`. The directory is
|
||||||
|
created with `exist_ok=False` so a stray folder from a prior failed delete
|
||||||
|
surfaces loudly instead of silently shadowing fresh content. Combined with
|
||||||
|
SQLite AUTOINCREMENT on `overlays.id`, that catches DB/disk drift.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
from l4d2host.paths import get_left4me_root, validate_overlay_ref
|
||||||
|
|
||||||
|
from l4d2web.models import Overlay
|
||||||
|
|
||||||
|
|
||||||
|
def generate_overlay_path(overlay_id: int) -> str:
|
||||||
|
"""Return the canonical relative path for an overlay row.
|
||||||
|
|
||||||
|
Validates the result through l4d2host's overlay-ref guard. Pure numeric IDs
|
||||||
|
always pass — this is just a belt-and-suspenders check that surfaces
|
||||||
|
immediately if someone changes the scheme.
|
||||||
|
"""
|
||||||
|
candidate = str(overlay_id)
|
||||||
|
return validate_overlay_ref(candidate)
|
||||||
|
|
||||||
|
|
||||||
|
def create_overlay_directory(overlay: Overlay) -> None:
|
||||||
|
"""Create `LEFT4ME_ROOT/overlays/{overlay.path}/` with `exist_ok=False`.
|
||||||
|
|
||||||
|
Raises `FileExistsError` if the directory already exists, surfacing the
|
||||||
|
rare DB/disk-drift state where a stray directory matches a fresh ID.
|
||||||
|
"""
|
||||||
|
target = get_left4me_root() / "overlays" / overlay.path
|
||||||
|
os.makedirs(target, exist_ok=False)
|
||||||
|
|
@ -1,17 +1,3 @@
|
||||||
import re
|
|
||||||
|
|
||||||
|
|
||||||
_INSTANCE_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{0,63}$")
|
|
||||||
|
|
||||||
|
|
||||||
def validate_instance_name(raw: str) -> str:
|
|
||||||
if not _INSTANCE_NAME_RE.fullmatch(raw):
|
|
||||||
raise ValueError(
|
|
||||||
"instance name must match [a-z0-9][a-z0-9_-]{0,63}"
|
|
||||||
)
|
|
||||||
return raw
|
|
||||||
|
|
||||||
|
|
||||||
def validate_overlay_ref(raw: str) -> str:
|
def validate_overlay_ref(raw: str) -> str:
|
||||||
if raw != raw.strip():
|
if raw != raw.strip():
|
||||||
raise ValueError("overlay ref must not have leading or trailing whitespace")
|
raise ValueError("overlay ref must not have leading or trailing whitespace")
|
||||||
|
|
|
||||||
5
l4d2web/services/server_identity.py
Normal file
5
l4d2web/services/server_identity.py
Normal file
|
|
@ -0,0 +1,5 @@
|
||||||
|
def server_unit_name(server_id: int) -> str:
|
||||||
|
"""Host-side identifier for a server (systemd unit suffix and on-disk
|
||||||
|
directory name). Lives in one place so a future format change is a
|
||||||
|
single-line edit."""
|
||||||
|
return str(server_id)
|
||||||
295
l4d2web/services/steam_workshop.py
Normal file
295
l4d2web/services/steam_workshop.py
Normal file
|
|
@ -0,0 +1,295 @@
|
||||||
|
"""Steam Workshop API client + downloader.
|
||||||
|
|
||||||
|
Pure HTTP/file logic — no DB writes, no Flask, no job-worker integration.
|
||||||
|
Used by the workshop overlay builder and the admin refresh job.
|
||||||
|
|
||||||
|
Endpoints:
|
||||||
|
- GetCollectionDetails: resolve a collection ID to its child item IDs.
|
||||||
|
- GetPublishedFileDetails: batch-fetch metadata for items, including a public
|
||||||
|
file_url for the .vpk.
|
||||||
|
|
||||||
|
Both endpoints accept anonymous POSTs; no Steam Web API key required.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import threading
|
||||||
|
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Callable, Iterable, Literal
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
|
|
||||||
|
# HTTPS only (decision 16). The reference downloader uses HTTP — we don't.
|
||||||
|
GET_PUBLISHED_FILE_DETAILS_URL = (
|
||||||
|
"https://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1/"
|
||||||
|
)
|
||||||
|
GET_COLLECTION_DETAILS_URL = (
|
||||||
|
"https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/"
|
||||||
|
)
|
||||||
|
|
||||||
|
L4D2_APP_ID = 550
|
||||||
|
|
||||||
|
REQUEST_TIMEOUT_SECONDS = 30
|
||||||
|
DOWNLOAD_CHUNK_BYTES = 1_048_576
|
||||||
|
|
||||||
|
_NUMERIC_ID_RE = re.compile(r"^\d+$")
|
||||||
|
_URL_ID_RE = re.compile(r"^https?://([a-z0-9.-]*\.)?steamcommunity\.com/.*[?&]id=(\d+)", re.IGNORECASE)
|
||||||
|
_BARE_URL_ID_RE = re.compile(r"^([a-z0-9.-]*\.)?steamcommunity\.com/.*[?&]id=(\d+)", re.IGNORECASE)
|
||||||
|
|
||||||
|
_session_local = threading.local()
|
||||||
|
|
||||||
|
|
||||||
|
def _session() -> requests.Session:
|
||||||
|
"""Per-thread session for connection reuse without cross-thread leakage."""
|
||||||
|
sess = getattr(_session_local, "session", None)
|
||||||
|
if sess is None:
|
||||||
|
sess = requests.Session()
|
||||||
|
_session_local.session = sess
|
||||||
|
return sess
|
||||||
|
|
||||||
|
|
||||||
|
class WorkshopValidationError(ValueError):
|
||||||
|
"""Raised during user-add when an item fails a fixed precondition
|
||||||
|
(e.g. consumer_app_id != 550)."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class WorkshopMetadata:
|
||||||
|
steam_id: str
|
||||||
|
title: str
|
||||||
|
filename: str
|
||||||
|
file_url: str
|
||||||
|
file_size: int
|
||||||
|
time_updated: int
|
||||||
|
preview_url: str
|
||||||
|
consumer_app_id: int
|
||||||
|
result: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class RefreshReport:
|
||||||
|
downloaded: int = 0
|
||||||
|
skipped: int = 0
|
||||||
|
errors: int = 0
|
||||||
|
per_item_errors: dict[str, str] = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_workshop_input(raw: str) -> list[str]:
|
||||||
|
"""Parse a single ID, a single workshop URL, or a multi-line / whitespace-
|
||||||
|
separated batch of either. Returns deduplicated digit-only IDs in order.
|
||||||
|
Raises ValueError on garbage."""
|
||||||
|
if not raw or not raw.strip():
|
||||||
|
raise ValueError("input is empty")
|
||||||
|
|
||||||
|
tokens: list[str] = []
|
||||||
|
for token in re.split(r"\s+", raw.strip()):
|
||||||
|
if not token:
|
||||||
|
continue
|
||||||
|
tokens.append(_extract_id(token))
|
||||||
|
|
||||||
|
seen: set[str] = set()
|
||||||
|
deduped: list[str] = []
|
||||||
|
for tok in tokens:
|
||||||
|
if tok not in seen:
|
||||||
|
seen.add(tok)
|
||||||
|
deduped.append(tok)
|
||||||
|
return deduped
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_id(token: str) -> str:
|
||||||
|
if _NUMERIC_ID_RE.fullmatch(token):
|
||||||
|
return token
|
||||||
|
m = _URL_ID_RE.match(token)
|
||||||
|
if m:
|
||||||
|
return m.group(2)
|
||||||
|
m = _BARE_URL_ID_RE.match(token)
|
||||||
|
if m:
|
||||||
|
return m.group(2)
|
||||||
|
raise ValueError(f"could not parse a Steam workshop id from: {token!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_collection(collection_id: str) -> list[str]:
|
||||||
|
"""POST GetCollectionDetails for one collection; return its non-collection
|
||||||
|
child publishedfileids in order. Nested collections (filetype != 0) are
|
||||||
|
skipped."""
|
||||||
|
if not _NUMERIC_ID_RE.fullmatch(collection_id):
|
||||||
|
raise ValueError("collection_id must be digits only")
|
||||||
|
|
||||||
|
response = _session().post(
|
||||||
|
GET_COLLECTION_DETAILS_URL,
|
||||||
|
data={
|
||||||
|
"collectioncount": 1,
|
||||||
|
"publishedfileids[0]": collection_id,
|
||||||
|
},
|
||||||
|
timeout=REQUEST_TIMEOUT_SECONDS,
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
payload = response.json()
|
||||||
|
children: list[str] = []
|
||||||
|
for collection in payload.get("response", {}).get("collectiondetails", []):
|
||||||
|
for child in collection.get("children", []):
|
||||||
|
if child.get("filetype", 0) != 0:
|
||||||
|
continue # nested collection, skip
|
||||||
|
child_id = child.get("publishedfileid")
|
||||||
|
if child_id is not None:
|
||||||
|
children.append(str(child_id))
|
||||||
|
return children
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_metadata_batch(
|
||||||
|
steam_ids: list[str], *, mode: Literal["add", "refresh"]
|
||||||
|
) -> list[WorkshopMetadata]:
|
||||||
|
"""One POST to GetPublishedFileDetails covering all ids.
|
||||||
|
|
||||||
|
In `mode="add"`, any non-L4D2 (`consumer_app_id != 550`) raises
|
||||||
|
WorkshopValidationError so the user-add request fails cleanly.
|
||||||
|
|
||||||
|
In `mode="refresh"`, non-L4D2 entries are skipped from the result.
|
||||||
|
|
||||||
|
Items with `result != 1` are returned as-is (the caller persists the result
|
||||||
|
code into `WorkshopItem.last_error`).
|
||||||
|
"""
|
||||||
|
if not steam_ids:
|
||||||
|
return []
|
||||||
|
for sid in steam_ids:
|
||||||
|
if not _NUMERIC_ID_RE.fullmatch(sid):
|
||||||
|
raise ValueError(f"steam id must be digits only: {sid!r}")
|
||||||
|
|
||||||
|
payload: dict[str, str | int] = {"itemcount": len(steam_ids)}
|
||||||
|
for index, sid in enumerate(steam_ids):
|
||||||
|
payload[f"publishedfileids[{index}]"] = sid
|
||||||
|
|
||||||
|
response = _session().post(
|
||||||
|
GET_PUBLISHED_FILE_DETAILS_URL,
|
||||||
|
data=payload,
|
||||||
|
timeout=REQUEST_TIMEOUT_SECONDS,
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
body = response.json()
|
||||||
|
|
||||||
|
metas: list[WorkshopMetadata] = []
|
||||||
|
for entry in body.get("response", {}).get("publishedfiledetails", []):
|
||||||
|
meta = WorkshopMetadata(
|
||||||
|
steam_id=str(entry.get("publishedfileid", "")),
|
||||||
|
title=str(entry.get("title", "") or ""),
|
||||||
|
filename=str(entry.get("filename", "") or ""),
|
||||||
|
file_url=str(entry.get("file_url", "") or ""),
|
||||||
|
file_size=int(entry.get("file_size") or 0),
|
||||||
|
time_updated=int(entry.get("time_updated") or 0),
|
||||||
|
preview_url=str(entry.get("preview_url", "") or ""),
|
||||||
|
consumer_app_id=int(entry.get("consumer_app_id") or 0),
|
||||||
|
result=int(entry.get("result") or 0),
|
||||||
|
)
|
||||||
|
|
||||||
|
# consumer_app_id is only meaningful when the lookup itself succeeded.
|
||||||
|
if meta.result == 1 and meta.consumer_app_id != L4D2_APP_ID:
|
||||||
|
if mode == "add":
|
||||||
|
raise WorkshopValidationError(
|
||||||
|
f"item {meta.steam_id} is not a Left 4 Dead 2 workshop "
|
||||||
|
f"item (consumer_app_id={meta.consumer_app_id})"
|
||||||
|
)
|
||||||
|
# refresh mode: drop the entry silently from the batch
|
||||||
|
continue
|
||||||
|
|
||||||
|
metas.append(meta)
|
||||||
|
return metas
|
||||||
|
|
||||||
|
|
||||||
|
def download_to_cache(
|
||||||
|
meta: WorkshopMetadata,
|
||||||
|
cache_root: Path,
|
||||||
|
*,
|
||||||
|
on_progress: Callable[[int, int], None] | None = None,
|
||||||
|
should_cancel: Callable[[], bool] | None = None,
|
||||||
|
) -> Path:
|
||||||
|
"""Download `meta.file_url` to `cache_root/{steam_id}.vpk`.
|
||||||
|
|
||||||
|
Atomic via `*.partial` + `os.replace`. Idempotent: a no-op when the
|
||||||
|
existing file's `(mtime, size)` already matches `(time_updated, file_size)`.
|
||||||
|
Sets `os.utime(target, (time_updated, time_updated))` so the next run
|
||||||
|
short-circuits.
|
||||||
|
"""
|
||||||
|
if not _NUMERIC_ID_RE.fullmatch(meta.steam_id):
|
||||||
|
raise ValueError("meta.steam_id must be digits only")
|
||||||
|
cache_root.mkdir(parents=True, exist_ok=True)
|
||||||
|
target = cache_root / f"{meta.steam_id}.vpk"
|
||||||
|
|
||||||
|
if (
|
||||||
|
target.exists()
|
||||||
|
and int(target.stat().st_mtime) == int(meta.time_updated)
|
||||||
|
and int(target.stat().st_size) == int(meta.file_size)
|
||||||
|
):
|
||||||
|
return target
|
||||||
|
|
||||||
|
if not meta.file_url:
|
||||||
|
raise ValueError(f"item {meta.steam_id} has no file_url; cannot download")
|
||||||
|
|
||||||
|
partial = target.with_suffix(target.suffix + ".partial")
|
||||||
|
response = _session().get(meta.file_url, stream=True, timeout=REQUEST_TIMEOUT_SECONDS)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
written = 0
|
||||||
|
try:
|
||||||
|
with open(partial, "wb") as f:
|
||||||
|
for chunk in response.iter_content(chunk_size=DOWNLOAD_CHUNK_BYTES):
|
||||||
|
if should_cancel is not None and should_cancel():
|
||||||
|
raise InterruptedError("download cancelled")
|
||||||
|
if not chunk:
|
||||||
|
continue
|
||||||
|
f.write(chunk)
|
||||||
|
written += len(chunk)
|
||||||
|
if on_progress is not None:
|
||||||
|
on_progress(written, int(meta.file_size))
|
||||||
|
os.replace(partial, target)
|
||||||
|
except BaseException:
|
||||||
|
partial.unlink(missing_ok=True)
|
||||||
|
raise
|
||||||
|
|
||||||
|
os.utime(target, (meta.time_updated, meta.time_updated))
|
||||||
|
return target
|
||||||
|
|
||||||
|
|
||||||
|
def refresh_all(
|
||||||
|
metas: Iterable[WorkshopMetadata],
|
||||||
|
cache_root: Path,
|
||||||
|
*,
|
||||||
|
executor_workers: int = 8,
|
||||||
|
should_cancel: Callable[[], bool] | None = None,
|
||||||
|
) -> RefreshReport:
|
||||||
|
"""Download (or skip-as-cached) every metadata item using a thread pool.
|
||||||
|
Per-item errors are collected; sibling items continue."""
|
||||||
|
metas_list = list(metas)
|
||||||
|
report = RefreshReport()
|
||||||
|
if not metas_list:
|
||||||
|
return report
|
||||||
|
|
||||||
|
cache_root.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
with ThreadPoolExecutor(max_workers=executor_workers) as executor:
|
||||||
|
futures = {}
|
||||||
|
for meta in metas_list:
|
||||||
|
if should_cancel is not None and should_cancel():
|
||||||
|
break
|
||||||
|
future = executor.submit(
|
||||||
|
download_to_cache,
|
||||||
|
meta,
|
||||||
|
cache_root,
|
||||||
|
should_cancel=should_cancel,
|
||||||
|
)
|
||||||
|
futures[future] = meta
|
||||||
|
|
||||||
|
for future in as_completed(futures):
|
||||||
|
meta = futures[future]
|
||||||
|
try:
|
||||||
|
future.result()
|
||||||
|
except Exception as exc:
|
||||||
|
report.errors += 1
|
||||||
|
report.per_item_errors[meta.steam_id] = str(exc)
|
||||||
|
continue
|
||||||
|
report.downloaded += 1
|
||||||
|
|
||||||
|
return report
|
||||||
24
l4d2web/services/workshop_paths.py
Normal file
24
l4d2web/services/workshop_paths.py
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
"""Cache-path helpers for workshop content.
|
||||||
|
|
||||||
|
The cache lives at `$LEFT4ME_ROOT/workshop_cache/{steam_id}.vpk`. Steam IDs
|
||||||
|
are validated digit-only here so callers don't need to guard separately.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from l4d2host.paths import get_left4me_root
|
||||||
|
|
||||||
|
|
||||||
|
_NUMERIC_ID_RE = re.compile(r"^\d+$")
|
||||||
|
|
||||||
|
|
||||||
|
def workshop_cache_root() -> Path:
|
||||||
|
return get_left4me_root() / "workshop_cache"
|
||||||
|
|
||||||
|
|
||||||
|
def cache_path(steam_id: str) -> Path:
|
||||||
|
if not isinstance(steam_id, str) or not _NUMERIC_ID_RE.fullmatch(steam_id):
|
||||||
|
raise ValueError(f"steam_id must be digits only: {steam_id!r}")
|
||||||
|
return workshop_cache_root() / f"{steam_id}.vpk"
|
||||||
|
|
@ -59,16 +59,20 @@ a:focus-visible {
|
||||||
outline-offset: 2px;
|
outline-offset: 2px;
|
||||||
}
|
}
|
||||||
|
|
||||||
button {
|
button,
|
||||||
|
a.button {
|
||||||
background: var(--color-primary);
|
background: var(--color-primary);
|
||||||
border: none;
|
border: none;
|
||||||
border-radius: var(--radius-s);
|
border-radius: var(--radius-s);
|
||||||
color: #fff;
|
color: #fff;
|
||||||
padding: var(--space-s) var(--space-l);
|
padding: var(--space-s) var(--space-l);
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
|
display: inline-block;
|
||||||
|
text-decoration: none;
|
||||||
}
|
}
|
||||||
|
|
||||||
button.danger {
|
button.danger,
|
||||||
|
a.button.danger {
|
||||||
background: var(--color-danger);
|
background: var(--color-danger);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -88,3 +92,99 @@ button.danger {
|
||||||
.auth-panel {
|
.auth-panel {
|
||||||
max-width: 28rem;
|
max-width: 28rem;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
dialog.modal {
|
||||||
|
background: var(--color-surface);
|
||||||
|
color: var(--color-text);
|
||||||
|
border: var(--line);
|
||||||
|
border-radius: var(--radius-m);
|
||||||
|
padding: 0;
|
||||||
|
width: min(32rem, 90vw);
|
||||||
|
max-height: 90vh;
|
||||||
|
}
|
||||||
|
|
||||||
|
dialog.modal::backdrop {
|
||||||
|
background: rgba(0, 0, 0, 0.45);
|
||||||
|
}
|
||||||
|
|
||||||
|
.modal-header,
|
||||||
|
.modal-body,
|
||||||
|
.modal-footer {
|
||||||
|
padding: var(--space-l);
|
||||||
|
}
|
||||||
|
|
||||||
|
.modal-header {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
align-items: center;
|
||||||
|
border-bottom: var(--line);
|
||||||
|
}
|
||||||
|
|
||||||
|
.modal-header h2 {
|
||||||
|
margin: 0;
|
||||||
|
font-size: 1.1rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.modal-body {
|
||||||
|
display: grid;
|
||||||
|
gap: var(--space-m);
|
||||||
|
overflow-y: auto;
|
||||||
|
}
|
||||||
|
|
||||||
|
.modal-footer {
|
||||||
|
display: flex;
|
||||||
|
justify-content: flex-end;
|
||||||
|
gap: var(--space-m);
|
||||||
|
border-top: var(--line);
|
||||||
|
}
|
||||||
|
|
||||||
|
.modal-close {
|
||||||
|
background: none;
|
||||||
|
color: var(--color-muted);
|
||||||
|
padding: var(--space-xs) var(--space-s);
|
||||||
|
font-size: 1.25rem;
|
||||||
|
line-height: 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
.button-secondary {
|
||||||
|
background: var(--color-surface);
|
||||||
|
color: var(--color-text);
|
||||||
|
border: var(--line);
|
||||||
|
}
|
||||||
|
|
||||||
|
.field-hint {
|
||||||
|
color: var(--color-muted);
|
||||||
|
font-size: 0.875rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.page-heading {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
align-items: center;
|
||||||
|
gap: var(--space-m);
|
||||||
|
margin-bottom: var(--space-l);
|
||||||
|
}
|
||||||
|
|
||||||
|
.page-heading h1,
|
||||||
|
.page-heading h2 {
|
||||||
|
margin: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.button-row {
|
||||||
|
display: flex;
|
||||||
|
gap: var(--space-s);
|
||||||
|
flex-wrap: wrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.detail-grid {
|
||||||
|
display: grid;
|
||||||
|
gap: var(--space-l);
|
||||||
|
}
|
||||||
|
|
||||||
|
.used-by-list {
|
||||||
|
list-style: none;
|
||||||
|
padding: 0;
|
||||||
|
margin: 0;
|
||||||
|
display: grid;
|
||||||
|
gap: var(--space-s);
|
||||||
|
}
|
||||||
|
|
|
||||||
27
l4d2web/static/js/modal.js
Normal file
27
l4d2web/static/js/modal.js
Normal file
|
|
@ -0,0 +1,27 @@
|
||||||
|
document.addEventListener("DOMContentLoaded", () => {
|
||||||
|
document.querySelectorAll("[data-modal-open]").forEach((trigger) => {
|
||||||
|
trigger.addEventListener("click", (event) => {
|
||||||
|
const targetId = trigger.getAttribute("data-modal-open");
|
||||||
|
const dialog = document.getElementById(targetId);
|
||||||
|
if (dialog && typeof dialog.showModal === "function") {
|
||||||
|
event.preventDefault();
|
||||||
|
dialog.showModal();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
document.querySelectorAll("dialog.modal").forEach((dialog) => {
|
||||||
|
dialog.querySelectorAll("[data-modal-close]").forEach((closer) => {
|
||||||
|
closer.addEventListener("click", (event) => {
|
||||||
|
event.preventDefault();
|
||||||
|
dialog.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
dialog.addEventListener("click", (event) => {
|
||||||
|
if (event.target === dialog) {
|
||||||
|
dialog.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
@ -17,7 +17,7 @@
|
||||||
<td><a href="/jobs/{{ job.id }}">#{{ job.id }}</a></td>
|
<td><a href="/jobs/{{ job.id }}">#{{ job.id }}</a></td>
|
||||||
<td>{{ job.operation }}</td>
|
<td>{{ job.operation }}</td>
|
||||||
<td>{{ job.state }}</td>
|
<td>{{ job.state }}</td>
|
||||||
{% if show_user %}<td>{{ user.username }}</td>{% endif %}
|
{% if show_user %}<td>{{ user.username if user else "system" }}</td>{% endif %}
|
||||||
{% if show_server %}<td>{% if server %}<a href="/servers/{{ server.id }}">{{ server.name }}</a>{% else %}-{% endif %}</td>{% endif %}
|
{% if show_server %}<td>{% if server %}<a href="/servers/{{ server.id }}">{{ server.name }}</a>{% else %}-{% endif %}</td>{% endif %}
|
||||||
<td>{{ job.created_at }}</td>
|
<td>{{ job.created_at }}</td>
|
||||||
<td>{{ job.finished_at or "-" }}</td>
|
<td>{{ job.finished_at or "-" }}</td>
|
||||||
|
|
|
||||||
50
l4d2web/templates/_overlay_item_table.html
Normal file
50
l4d2web/templates/_overlay_item_table.html
Normal file
|
|
@ -0,0 +1,50 @@
|
||||||
|
{% set can_edit = g.user.admin or (overlay and overlay.type == 'workshop' and overlay.user_id == g.user.id) %}
|
||||||
|
<table class="table">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th></th>
|
||||||
|
<th>Steam ID</th>
|
||||||
|
<th>Title</th>
|
||||||
|
<th>Filename</th>
|
||||||
|
<th>Size</th>
|
||||||
|
<th>Updated</th>
|
||||||
|
<th>Status</th>
|
||||||
|
{% if can_edit %}<th></th>{% endif %}
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{% for item in workshop_items %}
|
||||||
|
<tr>
|
||||||
|
<td>
|
||||||
|
{% if item.preview_url %}
|
||||||
|
<img src="{{ item.preview_url }}" alt="" width="48" height="48" loading="lazy">
|
||||||
|
{% endif %}
|
||||||
|
</td>
|
||||||
|
<td><a href="https://steamcommunity.com/sharedfiles/filedetails/?id={{ item.steam_id }}" target="_blank" rel="noopener">{{ item.steam_id }}</a></td>
|
||||||
|
<td>{{ item.title }}</td>
|
||||||
|
<td class="muted">{{ item.filename }}</td>
|
||||||
|
<td class="muted">{{ item.file_size }}</td>
|
||||||
|
<td class="muted">{{ item.time_updated }}</td>
|
||||||
|
<td>
|
||||||
|
{% if item.last_error %}
|
||||||
|
<span class="warning">{{ item.last_error }}</span>
|
||||||
|
{% elif item.last_downloaded_at %}
|
||||||
|
cached
|
||||||
|
{% else %}
|
||||||
|
<span class="muted">pending</span>
|
||||||
|
{% endif %}
|
||||||
|
</td>
|
||||||
|
{% if can_edit %}
|
||||||
|
<td>
|
||||||
|
<form method="post" action="/overlays/{{ overlay.id }}/items/{{ item.id }}/delete" class="inline-form">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<button type="submit" class="button-secondary">Remove</button>
|
||||||
|
</form>
|
||||||
|
</td>
|
||||||
|
{% endif %}
|
||||||
|
</tr>
|
||||||
|
{% else %}
|
||||||
|
<tr><td colspan="{% if can_edit %}8{% else %}7{% endif %}" class="muted">No workshop items yet.</td></tr>
|
||||||
|
{% endfor %}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
@ -19,4 +19,13 @@
|
||||||
<button type="submit">Install or update runtime</button>
|
<button type="submit">Install or update runtime</button>
|
||||||
</form>
|
</form>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
<section class="panel">
|
||||||
|
<h2>Workshop</h2>
|
||||||
|
<p class="muted">Re-fetch metadata for every workshop item; re-download those with newer versions on Steam, then enqueue rebuilds for the affected overlays.</p>
|
||||||
|
<form method="post" action="/admin/workshop/refresh">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<button type="submit">Refresh all workshop items</button>
|
||||||
|
</form>
|
||||||
|
</section>
|
||||||
{% endblock %}
|
{% endblock %}
|
||||||
|
|
|
||||||
|
|
@ -39,5 +39,6 @@
|
||||||
<script src="{{ url_for('static', filename='vendor/htmx.min.js') }}"></script>
|
<script src="{{ url_for('static', filename='vendor/htmx.min.js') }}"></script>
|
||||||
<script src="{{ url_for('static', filename='js/csrf.js') }}"></script>
|
<script src="{{ url_for('static', filename='js/csrf.js') }}"></script>
|
||||||
<script src="{{ url_for('static', filename='js/sse.js') }}"></script>
|
<script src="{{ url_for('static', filename='js/sse.js') }}"></script>
|
||||||
|
<script src="{{ url_for('static', filename='js/modal.js') }}"></script>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,10 @@
|
||||||
|
|
||||||
{% block content %}
|
{% block content %}
|
||||||
<section class="panel">
|
<section class="panel">
|
||||||
<h1>Blueprint: {{ blueprint.name }}</h1>
|
<div class="page-heading">
|
||||||
|
<h1>Blueprint: {{ blueprint.name }}</h1>
|
||||||
|
<button type="button" class="danger" data-modal-open="delete-blueprint-modal">Delete</button>
|
||||||
|
</div>
|
||||||
<form method="post" action="/blueprints/{{ blueprint.id }}" class="stack">
|
<form method="post" action="/blueprints/{{ blueprint.id }}" class="stack">
|
||||||
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
<label>Name <input name="name" value="{{ blueprint.name }}" required></label>
|
<label>Name <input name="name" value="{{ blueprint.name }}" required></label>
|
||||||
|
|
@ -28,4 +31,21 @@
|
||||||
<button type="submit">Save blueprint</button>
|
<button type="submit">Save blueprint</button>
|
||||||
</form>
|
</form>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
<dialog id="delete-blueprint-modal" class="modal" aria-labelledby="delete-blueprint-title">
|
||||||
|
<div class="modal-header">
|
||||||
|
<h2 id="delete-blueprint-title">Delete blueprint "{{ blueprint.name }}"?</h2>
|
||||||
|
<button type="button" class="modal-close" data-modal-close aria-label="Close">×</button>
|
||||||
|
</div>
|
||||||
|
<div class="modal-body">
|
||||||
|
<p>This cannot be undone. Blueprints in use by a server cannot be deleted.</p>
|
||||||
|
</div>
|
||||||
|
<div class="modal-footer">
|
||||||
|
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
|
||||||
|
<form method="post" action="/blueprints/{{ blueprint.id }}/delete" class="inline-form">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<button class="danger" type="submit">Delete</button>
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
|
</dialog>
|
||||||
{% endblock %}
|
{% endblock %}
|
||||||
|
|
|
||||||
|
|
@ -4,14 +4,10 @@
|
||||||
|
|
||||||
{% block content %}
|
{% block content %}
|
||||||
<section class="panel">
|
<section class="panel">
|
||||||
<h1>Blueprints</h1>
|
<div class="page-heading">
|
||||||
<form method="post" action="/blueprints" class="stack form-panel">
|
<h1>Blueprints</h1>
|
||||||
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
<button type="button" data-modal-open="create-blueprint-modal">+ Create</button>
|
||||||
<label>Name <input name="name" required></label>
|
</div>
|
||||||
<label>Arguments <textarea name="arguments"></textarea></label>
|
|
||||||
<label>Config <textarea name="config"></textarea></label>
|
|
||||||
<button type="submit">Create blueprint</button>
|
|
||||||
</form>
|
|
||||||
<table class="table">
|
<table class="table">
|
||||||
<thead><tr><th>Name</th><th>Created</th><th>Updated</th><th>Actions</th></tr></thead>
|
<thead><tr><th>Name</th><th>Created</th><th>Updated</th><th>Actions</th></tr></thead>
|
||||||
<tbody>
|
<tbody>
|
||||||
|
|
@ -20,12 +16,7 @@
|
||||||
<td><a href="/blueprints/{{ blueprint.id }}">{{ blueprint.name }}</a></td>
|
<td><a href="/blueprints/{{ blueprint.id }}">{{ blueprint.name }}</a></td>
|
||||||
<td>{{ blueprint.created_at }}</td>
|
<td>{{ blueprint.created_at }}</td>
|
||||||
<td>{{ blueprint.updated_at }}</td>
|
<td>{{ blueprint.updated_at }}</td>
|
||||||
<td>
|
<td><a href="/servers?blueprint_id={{ blueprint.id }}">Create server</a></td>
|
||||||
<form method="post" action="/blueprints/{{ blueprint.id }}/delete" class="inline-form">
|
|
||||||
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
|
||||||
<button class="danger" type="submit">Delete</button>
|
|
||||||
</form>
|
|
||||||
</td>
|
|
||||||
</tr>
|
</tr>
|
||||||
{% else %}
|
{% else %}
|
||||||
<tr><td colspan="4" class="muted">No blueprints configured.</td></tr>
|
<tr><td colspan="4" class="muted">No blueprints configured.</td></tr>
|
||||||
|
|
@ -33,4 +24,23 @@
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
<dialog id="create-blueprint-modal" class="modal" aria-labelledby="create-blueprint-title">
|
||||||
|
<form method="post" action="/blueprints" class="stack">
|
||||||
|
<div class="modal-header">
|
||||||
|
<h2 id="create-blueprint-title">Create blueprint</h2>
|
||||||
|
<button type="button" class="modal-close" data-modal-close aria-label="Close">×</button>
|
||||||
|
</div>
|
||||||
|
<div class="modal-body">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<label>Name <input name="name" required></label>
|
||||||
|
<label>Arguments <textarea name="arguments"></textarea></label>
|
||||||
|
<label>Config <textarea name="config"></textarea></label>
|
||||||
|
</div>
|
||||||
|
<div class="modal-footer">
|
||||||
|
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
|
||||||
|
<button type="submit">Create blueprint</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</dialog>
|
||||||
{% endblock %}
|
{% endblock %}
|
||||||
|
|
|
||||||
|
|
@ -19,7 +19,7 @@
|
||||||
<tbody>
|
<tbody>
|
||||||
<tr><th>Operation</th><td>{{ job.operation }}</td></tr>
|
<tr><th>Operation</th><td>{{ job.operation }}</td></tr>
|
||||||
<tr><th>State</th><td>{{ job.state }}</td></tr>
|
<tr><th>State</th><td>{{ job.state }}</td></tr>
|
||||||
<tr><th>User</th><td>{{ owner.username }}</td></tr>
|
<tr><th>User</th><td>{{ owner.username if owner else "system" }}</td></tr>
|
||||||
<tr><th>Server</th><td>{% if server %}<a href="/servers/{{ server.id }}">{{ server.name }}</a>{% else %}-{% endif %}</td></tr>
|
<tr><th>Server</th><td>{% if server %}<a href="/servers/{{ server.id }}">{{ server.name }}</a>{% else %}-{% endif %}</td></tr>
|
||||||
<tr><th>Created</th><td>{{ job.created_at }}</td></tr>
|
<tr><th>Created</th><td>{{ job.created_at }}</td></tr>
|
||||||
<tr><th>Started</th><td>{{ job.started_at or "-" }}</td></tr>
|
<tr><th>Started</th><td>{{ job.started_at or "-" }}</td></tr>
|
||||||
|
|
|
||||||
182
l4d2web/templates/overlay_detail.html
Normal file
182
l4d2web/templates/overlay_detail.html
Normal file
|
|
@ -0,0 +1,182 @@
|
||||||
|
{% extends "base.html" %}
|
||||||
|
|
||||||
|
{% block title %}Overlay {{ overlay.name }} | left4me{% endblock %}
|
||||||
|
|
||||||
|
{% block content %}
|
||||||
|
<section class="panel">
|
||||||
|
<div class="page-heading">
|
||||||
|
<h1>Overlay: {{ overlay.name }}</h1>
|
||||||
|
{% set can_edit = g.user.admin or (overlay.type in ['workshop', 'script'] and overlay.user_id == g.user.id) %}
|
||||||
|
{% if can_edit %}
|
||||||
|
<button type="button" class="danger" data-modal-open="delete-overlay-modal">Delete</button>
|
||||||
|
{% endif %}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{% if can_edit %}
|
||||||
|
<form method="post" action="/overlays/{{ overlay.id }}" class="stack">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<label>Name <input name="name" value="{{ overlay.name }}" required></label>
|
||||||
|
<div>
|
||||||
|
<button type="submit">Save</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
<table class="definition-table">
|
||||||
|
<tbody>
|
||||||
|
<tr><th>Type</th><td>{{ overlay.type }}</td></tr>
|
||||||
|
<tr><th>Scope</th><td>{% if overlay.user_id %}private{% else %}system{% endif %}</td></tr>
|
||||||
|
<tr><th>Path</th><td class="muted">{{ overlay.path }}</td></tr>
|
||||||
|
<tr>
|
||||||
|
<th>Last build</th>
|
||||||
|
<td>
|
||||||
|
{% if overlay.last_build_status == 'ok' %}
|
||||||
|
<span class="badge badge-ok">ok</span>
|
||||||
|
{% elif overlay.last_build_status == 'failed' %}
|
||||||
|
<span class="badge badge-error">failed</span>
|
||||||
|
{% else %}
|
||||||
|
<span class="badge badge-muted">never</span>
|
||||||
|
{% endif %}
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
{% if overlay.type == 'script' %}
|
||||||
|
<section class="panel">
|
||||||
|
<div class="page-heading">
|
||||||
|
<h2>Script</h2>
|
||||||
|
{% if can_edit %}
|
||||||
|
<div class="inline-form-group">
|
||||||
|
<form method="post" action="/overlays/{{ overlay.id }}/build" class="inline-form">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<button type="submit" class="button-secondary">Rebuild</button>
|
||||||
|
</form>
|
||||||
|
<button type="button" class="danger" data-modal-open="wipe-overlay-modal">Wipe</button>
|
||||||
|
</div>
|
||||||
|
{% endif %}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{% if can_edit %}
|
||||||
|
<form method="post" action="/overlays/{{ overlay.id }}/script" class="stack">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<label>Bash script
|
||||||
|
<textarea name="script" rows="20" spellcheck="false">{{ overlay.script or "" }}</textarea>
|
||||||
|
</label>
|
||||||
|
<p class="muted">Runs sandboxed against the overlay directory mounted at <code>/overlay</code>.</p>
|
||||||
|
<div>
|
||||||
|
<button type="submit">Save and build</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
{% else %}
|
||||||
|
<pre class="script-preview">{{ overlay.script or "" }}</pre>
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
<p>
|
||||||
|
{% if latest_build_job %}
|
||||||
|
Latest build: <a href="/jobs/{{ latest_build_job.id }}">job #{{ latest_build_job.id }}</a>
|
||||||
|
— state: <strong>{{ latest_build_job.state }}</strong>
|
||||||
|
·
|
||||||
|
{% endif %}
|
||||||
|
<a href="/overlays/{{ overlay.id }}/jobs">all builds →</a>
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
{% if overlay.type == 'workshop' %}
|
||||||
|
<section class="panel">
|
||||||
|
<div class="page-heading">
|
||||||
|
<h2>Workshop items</h2>
|
||||||
|
{% if can_edit %}
|
||||||
|
<form method="post" action="/overlays/{{ overlay.id }}/build" class="inline-form">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<button type="submit" class="button-secondary">Rebuild</button>
|
||||||
|
</form>
|
||||||
|
{% endif %}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{% if can_edit %}
|
||||||
|
<form method="post" action="/overlays/{{ overlay.id }}/items" class="stack">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<fieldset class="workshop-input-mode">
|
||||||
|
<legend>Input mode</legend>
|
||||||
|
<label><input type="radio" name="input_mode" value="items" checked> Items (paste IDs or URLs; one or many)</label>
|
||||||
|
<label><input type="radio" name="input_mode" value="collection"> Collection (one ID or URL)</label>
|
||||||
|
</fieldset>
|
||||||
|
<label>Workshop input <textarea name="input" rows="3" placeholder="123456789 https://steamcommunity.com/sharedfiles/filedetails/?id=987654321"></textarea></label>
|
||||||
|
<div>
|
||||||
|
<button type="submit">Add</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
<div id="overlay-item-table">
|
||||||
|
{% include "_overlay_item_table.html" with context %}
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="panel">
|
||||||
|
<h2>Builds</h2>
|
||||||
|
<p>
|
||||||
|
{% if latest_build_job %}
|
||||||
|
Latest build: <a href="/jobs/{{ latest_build_job.id }}">job #{{ latest_build_job.id }}</a>
|
||||||
|
— state: <strong>{{ latest_build_job.state }}</strong>
|
||||||
|
·
|
||||||
|
{% endif %}
|
||||||
|
<a href="/overlays/{{ overlay.id }}/jobs">all builds →</a>
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
<section class="panel">
|
||||||
|
<h2>Used by</h2>
|
||||||
|
{% if using_blueprints %}
|
||||||
|
<ul class="used-by-list">
|
||||||
|
{% for blueprint in using_blueprints %}
|
||||||
|
<li><a href="/blueprints/{{ blueprint.id }}">{{ blueprint.name }}</a></li>
|
||||||
|
{% endfor %}
|
||||||
|
</ul>
|
||||||
|
{% else %}
|
||||||
|
<p class="muted">Not used by any blueprint.</p>
|
||||||
|
{% endif %}
|
||||||
|
</section>
|
||||||
|
|
||||||
|
{% if can_edit %}
|
||||||
|
<dialog id="delete-overlay-modal" class="modal" aria-labelledby="delete-overlay-title">
|
||||||
|
<div class="modal-header">
|
||||||
|
<h2 id="delete-overlay-title">Delete overlay "{{ overlay.name }}"?</h2>
|
||||||
|
<button type="button" class="modal-close" data-modal-close aria-label="Close">×</button>
|
||||||
|
</div>
|
||||||
|
<div class="modal-body">
|
||||||
|
<p>This cannot be undone. Overlays in use by a blueprint cannot be deleted.</p>
|
||||||
|
</div>
|
||||||
|
<div class="modal-footer">
|
||||||
|
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
|
||||||
|
<form method="post" action="/overlays/{{ overlay.id }}/delete" class="inline-form">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<button class="danger" type="submit">Delete</button>
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
|
</dialog>
|
||||||
|
|
||||||
|
{% if overlay.type == 'script' %}
|
||||||
|
<dialog id="wipe-overlay-modal" class="modal" aria-labelledby="wipe-overlay-title">
|
||||||
|
<div class="modal-header">
|
||||||
|
<h2 id="wipe-overlay-title">Wipe overlay "{{ overlay.name }}"?</h2>
|
||||||
|
<button type="button" class="modal-close" data-modal-close aria-label="Close">×</button>
|
||||||
|
</div>
|
||||||
|
<div class="modal-body">
|
||||||
|
<p>Empties the overlay directory. Use Rebuild afterward to repopulate.</p>
|
||||||
|
</div>
|
||||||
|
<div class="modal-footer">
|
||||||
|
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
|
||||||
|
<form method="post" action="/overlays/{{ overlay.id }}/wipe" class="inline-form">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<button class="danger" type="submit">Wipe</button>
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
|
</dialog>
|
||||||
|
{% endif %}
|
||||||
|
{% endif %}
|
||||||
|
{% endblock %}
|
||||||
17
l4d2web/templates/overlay_jobs.html
Normal file
17
l4d2web/templates/overlay_jobs.html
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
{% extends "base.html" %}
|
||||||
|
|
||||||
|
{% block title %}Build jobs for {{ overlay.name }} | left4me{% endblock %}
|
||||||
|
|
||||||
|
{% block content %}
|
||||||
|
<section class="panel">
|
||||||
|
<div class="page-heading">
|
||||||
|
<h1>Build jobs for {{ overlay.name }}</h1>
|
||||||
|
<a href="/overlays/{{ overlay.id }}">Back to overlay</a>
|
||||||
|
</div>
|
||||||
|
{% set show_user = true %}
|
||||||
|
{% set show_server = false %}
|
||||||
|
{% set show_cancel = true %}
|
||||||
|
{% set cancel_next = "/overlays/" ~ overlay.id ~ "/jobs" %}
|
||||||
|
{% include "_job_table.html" %}
|
||||||
|
</section>
|
||||||
|
{% endblock %}
|
||||||
|
|
@ -6,43 +6,49 @@
|
||||||
<section class="panel">
|
<section class="panel">
|
||||||
<div class="page-heading">
|
<div class="page-heading">
|
||||||
<h1>Overlays</h1>
|
<h1>Overlays</h1>
|
||||||
|
<button type="button" data-modal-open="create-overlay-modal">+ Create</button>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{% if g.user.admin %}
|
|
||||||
<form method="post" action="/overlays" class="stack form-panel">
|
|
||||||
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
|
||||||
<label>Name <input name="name" required></label>
|
|
||||||
<label>Path <input name="path" required placeholder="/opt/l4d2/overlays/example"></label>
|
|
||||||
<button type="submit">Add overlay</button>
|
|
||||||
</form>
|
|
||||||
{% endif %}
|
|
||||||
|
|
||||||
<table class="table">
|
<table class="table">
|
||||||
<thead><tr><th>Name</th><th>Path</th>{% if g.user.admin %}<th>Actions</th>{% endif %}</tr></thead>
|
<thead><tr><th>Name</th><th>Type</th><th>Scope</th><th>Path</th></tr></thead>
|
||||||
<tbody>
|
<tbody>
|
||||||
{% for overlay in overlays %}
|
{% for overlay in overlays %}
|
||||||
<tr>
|
<tr>
|
||||||
<td>{{ overlay.name }}</td>
|
<td><a href="/overlays/{{ overlay.id }}">{{ overlay.name }}</a></td>
|
||||||
|
<td>{{ overlay.type }}</td>
|
||||||
|
<td class="muted">{% if overlay.user_id %}private{% else %}system{% endif %}</td>
|
||||||
<td class="muted">{{ overlay.path }}</td>
|
<td class="muted">{{ overlay.path }}</td>
|
||||||
{% if g.user.admin %}
|
|
||||||
<td>
|
|
||||||
<form method="post" action="/overlays/{{ overlay.id }}" class="inline-form">
|
|
||||||
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
|
||||||
<input name="name" value="{{ overlay.name }}" required>
|
|
||||||
<input name="path" value="{{ overlay.path }}" required>
|
|
||||||
<button type="submit">Save</button>
|
|
||||||
</form>
|
|
||||||
<form method="post" action="/overlays/{{ overlay.id }}/delete" class="inline-form">
|
|
||||||
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
|
||||||
<button class="danger" type="submit">Delete</button>
|
|
||||||
</form>
|
|
||||||
</td>
|
|
||||||
{% endif %}
|
|
||||||
</tr>
|
</tr>
|
||||||
{% else %}
|
{% else %}
|
||||||
<tr><td colspan="{% if g.user.admin %}3{% else %}2{% endif %}" class="muted">No overlays configured.</td></tr>
|
<tr><td colspan="4" class="muted">No overlays yet.</td></tr>
|
||||||
{% endfor %}
|
{% endfor %}
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
<dialog id="create-overlay-modal" class="modal" aria-labelledby="create-overlay-title">
|
||||||
|
<form method="post" action="/overlays" class="stack">
|
||||||
|
<div class="modal-header">
|
||||||
|
<h2 id="create-overlay-title">Create overlay</h2>
|
||||||
|
<button type="button" class="modal-close" data-modal-close aria-label="Close">×</button>
|
||||||
|
</div>
|
||||||
|
<div class="modal-body">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<fieldset class="overlay-type-radio">
|
||||||
|
<legend>Type</legend>
|
||||||
|
<label><input type="radio" name="type" value="workshop" checked> Workshop (downloads from Steam)</label>
|
||||||
|
<label><input type="radio" name="type" value="script"> Script (runs sandboxed bash)</label>
|
||||||
|
</fieldset>
|
||||||
|
<label>Name <input name="name" required></label>
|
||||||
|
{% if g.user and g.user.admin %}
|
||||||
|
<label><input type="checkbox" name="system_wide" value="1"> System-wide (visible to all users)</label>
|
||||||
|
{% endif %}
|
||||||
|
<p class="muted">The path is generated automatically.</p>
|
||||||
|
</div>
|
||||||
|
<div class="modal-footer">
|
||||||
|
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
|
||||||
|
<button type="submit">Create</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</dialog>
|
||||||
{% endblock %}
|
{% endblock %}
|
||||||
|
|
|
||||||
|
|
@ -13,10 +13,8 @@
|
||||||
<button type="submit">{{ operation }}</button>
|
<button type="submit">{{ operation }}</button>
|
||||||
</form>
|
</form>
|
||||||
{% endfor %}
|
{% endfor %}
|
||||||
<form method="post" action="/servers/{{ server.id }}/delete" class="inline-form">
|
<button type="button" class="danger" data-modal-open="reset-server-modal">reset</button>
|
||||||
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
<button type="button" class="danger" data-modal-open="delete-server-modal">delete</button>
|
||||||
<button class="danger" type="submit">delete</button>
|
|
||||||
</form>
|
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
|
@ -49,4 +47,38 @@
|
||||||
<h2>Server Log</h2>
|
<h2>Server Log</h2>
|
||||||
<pre class="log-stream" data-sse-url="/servers/{{ server.id }}/logs/stream"></pre>
|
<pre class="log-stream" data-sse-url="/servers/{{ server.id }}/logs/stream"></pre>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
<dialog id="reset-server-modal" class="modal" aria-labelledby="reset-server-title">
|
||||||
|
<div class="modal-header">
|
||||||
|
<h2 id="reset-server-title">Reset server "{{ server.name }}"?</h2>
|
||||||
|
<button type="button" class="modal-close" data-modal-close aria-label="Close">×</button>
|
||||||
|
</div>
|
||||||
|
<div class="modal-body">
|
||||||
|
<p>This stops the server and wipes its runtime state (logs, caches, accumulated game state). The blueprint association is preserved; the next start rebuilds from the current blueprint.</p>
|
||||||
|
</div>
|
||||||
|
<div class="modal-footer">
|
||||||
|
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
|
||||||
|
<form method="post" action="/servers/{{ server.id }}/reset" class="inline-form">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<button class="danger" type="submit">Reset</button>
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
|
</dialog>
|
||||||
|
|
||||||
|
<dialog id="delete-server-modal" class="modal" aria-labelledby="delete-server-title">
|
||||||
|
<div class="modal-header">
|
||||||
|
<h2 id="delete-server-title">Delete server "{{ server.name }}"?</h2>
|
||||||
|
<button type="button" class="modal-close" data-modal-close aria-label="Close">×</button>
|
||||||
|
</div>
|
||||||
|
<div class="modal-body">
|
||||||
|
<p>This stops the server and tears down its runtime files. This cannot be undone.</p>
|
||||||
|
</div>
|
||||||
|
<div class="modal-footer">
|
||||||
|
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
|
||||||
|
<form method="post" action="/servers/{{ server.id }}/delete" class="inline-form">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<button class="danger" type="submit">Delete</button>
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
|
</dialog>
|
||||||
{% endblock %}
|
{% endblock %}
|
||||||
|
|
|
||||||
|
|
@ -4,24 +4,14 @@
|
||||||
|
|
||||||
{% block content %}
|
{% block content %}
|
||||||
<section class="panel">
|
<section class="panel">
|
||||||
<h1>Servers</h1>
|
<div class="page-heading">
|
||||||
{% if blueprints %}
|
<h1>Servers</h1>
|
||||||
<form method="post" action="/servers" class="stack form-panel">
|
{% if blueprints %}
|
||||||
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
<button type="button" data-modal-open="create-server-modal">+ Create</button>
|
||||||
<label>Name <input name="name" required></label>
|
{% else %}
|
||||||
<label>Port <input name="port" type="number" min="1" max="65535" value="27015" required></label>
|
<a class="button" href="/blueprints">Create a blueprint first →</a>
|
||||||
<label>Blueprint
|
{% endif %}
|
||||||
<select name="blueprint_id" required>
|
</div>
|
||||||
{% for blueprint in blueprints %}
|
|
||||||
<option value="{{ blueprint.id }}">{{ blueprint.name }}</option>
|
|
||||||
{% endfor %}
|
|
||||||
</select>
|
|
||||||
</label>
|
|
||||||
<button type="submit">Create server</button>
|
|
||||||
</form>
|
|
||||||
{% else %}
|
|
||||||
<p class="muted">Create a blueprint before adding servers.</p>
|
|
||||||
{% endif %}
|
|
||||||
<table class="table">
|
<table class="table">
|
||||||
<thead><tr><th>Name</th><th>Port</th><th>Blueprint</th><th>Desired</th><th>Actual</th></tr></thead>
|
<thead><tr><th>Name</th><th>Port</th><th>Blueprint</th><th>Desired</th><th>Actual</th></tr></thead>
|
||||||
<tbody>
|
<tbody>
|
||||||
|
|
@ -39,4 +29,45 @@
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
{% if blueprints %}
|
||||||
|
<dialog id="create-server-modal" class="modal" aria-labelledby="create-server-title">
|
||||||
|
<form method="post" action="/servers" class="stack">
|
||||||
|
<div class="modal-header">
|
||||||
|
<h2 id="create-server-title">Create server</h2>
|
||||||
|
<button type="button" class="modal-close" data-modal-close aria-label="Close">×</button>
|
||||||
|
</div>
|
||||||
|
<div class="modal-body">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ session.get('csrf_token', '') }}">
|
||||||
|
<label>Name <input name="name" required maxlength="128"></label>
|
||||||
|
<label>Port
|
||||||
|
<input name="port" type="number" min="1" max="65535" placeholder="auto">
|
||||||
|
<span class="field-hint">Leave empty for the next available port.</span>
|
||||||
|
</label>
|
||||||
|
<label>Blueprint
|
||||||
|
<select name="blueprint_id" required>
|
||||||
|
{% for blueprint in blueprints %}
|
||||||
|
<option value="{{ blueprint.id }}"{% if blueprint.id == prefill_blueprint_id %} selected{% endif %}>{{ blueprint.name }}</option>
|
||||||
|
{% endfor %}
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
<div class="modal-footer">
|
||||||
|
<button type="button" class="button-secondary" data-modal-close>Cancel</button>
|
||||||
|
<button type="submit">Create server</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</dialog>
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
{% if prefill_blueprint_id %}
|
||||||
|
<script>
|
||||||
|
document.addEventListener("DOMContentLoaded", () => {
|
||||||
|
const dialog = document.getElementById("create-server-modal");
|
||||||
|
if (dialog && typeof dialog.showModal === "function") {
|
||||||
|
dialog.showModal();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
</script>
|
||||||
|
{% endif %}
|
||||||
{% endblock %}
|
{% endblock %}
|
||||||
|
|
|
||||||
96
l4d2web/tests/test_alembic_migrations.py
Normal file
96
l4d2web/tests/test_alembic_migrations.py
Normal file
|
|
@ -0,0 +1,96 @@
|
||||||
|
"""Tests for the alembic migration history.
|
||||||
|
|
||||||
|
The 0005 migration adds `script` and `last_build_status` columns to `overlays`,
|
||||||
|
drops the global_overlay_* tables, and wipes legacy l4d2center_maps/cedapug_maps
|
||||||
|
overlay rows. This module pins those behaviors.
|
||||||
|
"""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from alembic import command
|
||||||
|
from alembic.config import Config
|
||||||
|
from sqlalchemy import create_engine, inspect, text
|
||||||
|
|
||||||
|
|
||||||
|
_ALEMBIC_DIR = Path(__file__).resolve().parents[1] / "alembic"
|
||||||
|
|
||||||
|
|
||||||
|
def _alembic_config(db_url: str) -> Config:
|
||||||
|
cfg = Config()
|
||||||
|
cfg.set_main_option("script_location", str(_ALEMBIC_DIR))
|
||||||
|
cfg.set_main_option("sqlalchemy.url", db_url)
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def db_url(tmp_path, monkeypatch):
|
||||||
|
path = tmp_path / "alembic.db"
|
||||||
|
url = f"sqlite:///{path}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", url)
|
||||||
|
yield url
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_0005_adds_script_columns(db_url) -> None:
|
||||||
|
cfg = _alembic_config(db_url)
|
||||||
|
|
||||||
|
command.upgrade(cfg, "0004_drop_legacy_external_overlay_type")
|
||||||
|
|
||||||
|
engine = create_engine(db_url)
|
||||||
|
with engine.begin() as conn:
|
||||||
|
# Seed legacy global-type overlay rows that the migration must wipe.
|
||||||
|
conn.execute(
|
||||||
|
text(
|
||||||
|
"INSERT INTO overlays (name, path, type, created_at, updated_at) "
|
||||||
|
"VALUES ('legacy-l4d2center', '1', 'l4d2center_maps', "
|
||||||
|
"'2026-01-01', '2026-01-01')"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
conn.execute(
|
||||||
|
text(
|
||||||
|
"INSERT INTO overlays (name, path, type, created_at, updated_at) "
|
||||||
|
"VALUES ('legacy-cedapug', '2', 'cedapug_maps', "
|
||||||
|
"'2026-01-01', '2026-01-01')"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
conn.execute(
|
||||||
|
text(
|
||||||
|
"INSERT INTO overlays (name, path, type, created_at, updated_at) "
|
||||||
|
"VALUES ('keep-workshop', '3', 'workshop', "
|
||||||
|
"'2026-01-01', '2026-01-01')"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
command.upgrade(cfg, "0005_script_overlays")
|
||||||
|
|
||||||
|
inspector = inspect(engine)
|
||||||
|
|
||||||
|
overlay_cols = {c["name"]: c for c in inspector.get_columns("overlays")}
|
||||||
|
assert "script" in overlay_cols
|
||||||
|
assert "last_build_status" in overlay_cols
|
||||||
|
assert overlay_cols["script"]["nullable"] is False
|
||||||
|
assert overlay_cols["last_build_status"]["nullable"] is False
|
||||||
|
|
||||||
|
table_names = set(inspector.get_table_names())
|
||||||
|
assert "global_overlay_sources" not in table_names
|
||||||
|
assert "global_overlay_items" not in table_names
|
||||||
|
assert "global_overlay_item_files" not in table_names
|
||||||
|
|
||||||
|
with engine.connect() as conn:
|
||||||
|
rows = conn.execute(
|
||||||
|
text("SELECT name, type FROM overlays ORDER BY name")
|
||||||
|
).all()
|
||||||
|
assert rows == [("keep-workshop", "workshop")]
|
||||||
|
|
||||||
|
defaults = conn.execute(
|
||||||
|
text(
|
||||||
|
"SELECT script, last_build_status FROM overlays "
|
||||||
|
"WHERE name = 'keep-workshop'"
|
||||||
|
)
|
||||||
|
).one()
|
||||||
|
assert defaults == ("", "")
|
||||||
|
|
||||||
|
|
||||||
|
def test_downgrade_0005_skipped() -> None:
|
||||||
|
"""Per the project convention (see 0004) destructive migrations are
|
||||||
|
intentionally one-way; do not test or maintain a downgrade."""
|
||||||
|
pytest.skip("0005 is one-way: globals data is gone after upgrade")
|
||||||
|
|
@ -79,12 +79,144 @@ def test_user_can_create_private_blueprint(user_client) -> None:
|
||||||
assert response.status_code == 201
|
assert response.status_code == 201
|
||||||
|
|
||||||
|
|
||||||
|
def _create_other_users_private_overlay() -> int:
|
||||||
|
with session_scope() as session:
|
||||||
|
other = User(username="mallory", password_digest=hash_password("secret"), admin=False)
|
||||||
|
session.add(other)
|
||||||
|
session.flush()
|
||||||
|
overlay = Overlay(
|
||||||
|
name="mallory-private",
|
||||||
|
path="mallory-private",
|
||||||
|
type="workshop",
|
||||||
|
user_id=other.id,
|
||||||
|
)
|
||||||
|
session.add(overlay)
|
||||||
|
session.flush()
|
||||||
|
return overlay.id
|
||||||
|
|
||||||
|
|
||||||
|
def test_user_cannot_create_blueprint_with_other_users_private_overlay(user_client) -> None:
|
||||||
|
foreign_overlay_id = _create_other_users_private_overlay()
|
||||||
|
payload = {
|
||||||
|
"name": "bad",
|
||||||
|
"arguments": [],
|
||||||
|
"config": [],
|
||||||
|
"overlay_ids": [foreign_overlay_id],
|
||||||
|
}
|
||||||
|
|
||||||
|
response = user_client.post(
|
||||||
|
"/blueprints",
|
||||||
|
data=json.dumps(payload),
|
||||||
|
content_type="application/json",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status_code == 403
|
||||||
|
|
||||||
|
|
||||||
|
def test_user_cannot_update_blueprint_with_other_users_private_overlay(user_client) -> None:
|
||||||
|
foreign_overlay_id = _create_other_users_private_overlay()
|
||||||
|
create = user_client.post(
|
||||||
|
"/blueprints",
|
||||||
|
data={"name": "comp", "arguments": "", "config": "", "overlay_ids": ["1"]},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert create.status_code == 302
|
||||||
|
|
||||||
|
response = user_client.post(
|
||||||
|
"/blueprints/1",
|
||||||
|
data={
|
||||||
|
"name": "edited",
|
||||||
|
"arguments": "",
|
||||||
|
"config": "",
|
||||||
|
"overlay_ids": [str(foreign_overlay_id)],
|
||||||
|
},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status_code == 403
|
||||||
|
|
||||||
|
|
||||||
|
def test_user_can_create_blueprint_with_system_overlay(user_client) -> None:
|
||||||
|
payload = {
|
||||||
|
"name": "system-ok",
|
||||||
|
"arguments": [],
|
||||||
|
"config": [],
|
||||||
|
"overlay_ids": [1],
|
||||||
|
}
|
||||||
|
|
||||||
|
response = user_client.post(
|
||||||
|
"/blueprints",
|
||||||
|
data=json.dumps(payload),
|
||||||
|
content_type="application/json",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status_code == 201
|
||||||
|
blueprint_id = response.get_json()["id"]
|
||||||
|
with session_scope() as session:
|
||||||
|
link = session.query(BlueprintOverlay).filter_by(blueprint_id=blueprint_id, overlay_id=1).one()
|
||||||
|
assert link.position == 0
|
||||||
|
|
||||||
|
|
||||||
def test_delete_blueprint_blocked_when_in_use(linked_blueprint) -> None:
|
def test_delete_blueprint_blocked_when_in_use(linked_blueprint) -> None:
|
||||||
client, blueprint_id = linked_blueprint
|
client, blueprint_id = linked_blueprint
|
||||||
response = client.delete(f"/blueprints/{blueprint_id}", headers={"X-CSRF-Token": "test-token"})
|
response = client.delete(f"/blueprints/{blueprint_id}", headers={"X-CSRF-Token": "test-token"})
|
||||||
assert response.status_code == 409
|
assert response.status_code == 409
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_delete_blueprint_redirects_to_index(user_client) -> None:
|
||||||
|
create = user_client.post(
|
||||||
|
"/blueprints",
|
||||||
|
data={"name": "doomed", "arguments": "", "config": ""},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert create.status_code == 302
|
||||||
|
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from l4d2web.models import Blueprint as BlueprintModel
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
blueprint_id = session.scalars(select(BlueprintModel.id)).one()
|
||||||
|
|
||||||
|
response = user_client.post(
|
||||||
|
f"/blueprints/{blueprint_id}/delete",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
assert response.headers["Location"] == "/blueprints"
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
assert session.scalars(select(BlueprintModel)).all() == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_delete_blueprint_blocked_when_in_use(linked_blueprint) -> None:
|
||||||
|
client, blueprint_id = linked_blueprint
|
||||||
|
response = client.post(
|
||||||
|
f"/blueprints/{blueprint_id}/delete",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 409
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_delete_blueprint_returns_404_for_other_user(user_client, tmp_path) -> None:
|
||||||
|
with session_scope() as session:
|
||||||
|
other = User(username="mallory", password_digest=hash_password("secret"), admin=False)
|
||||||
|
session.add(other)
|
||||||
|
session.flush()
|
||||||
|
foreign = Blueprint(user_id=other.id, name="foreign", arguments="[]", config="[]")
|
||||||
|
session.add(foreign)
|
||||||
|
session.flush()
|
||||||
|
foreign_id = foreign.id
|
||||||
|
|
||||||
|
response = user_client.post(
|
||||||
|
f"/blueprints/{foreign_id}/delete",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
def test_form_update_preserves_ordered_overlays_and_multiline_fields(user_client) -> None:
|
def test_form_update_preserves_ordered_overlays_and_multiline_fields(user_client) -> None:
|
||||||
create = user_client.post(
|
create = user_client.post(
|
||||||
"/blueprints",
|
"/blueprints",
|
||||||
|
|
|
||||||
|
|
@ -53,6 +53,18 @@ def test_session_cookie_secure_in_production(tmp_path, monkeypatch) -> None:
|
||||||
assert app.config["SESSION_COOKIE_SECURE"] is True
|
assert app.config["SESSION_COOKIE_SECURE"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_session_cookie_secure_env_override(tmp_path, monkeypatch) -> None:
|
||||||
|
db_url = f"sqlite:///{tmp_path/'cookie-env.db'}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
monkeypatch.setenv("SESSION_COOKIE_SECURE", "false")
|
||||||
|
monkeypatch.setattr("l4d2web.app.recover_stale_jobs", lambda: None)
|
||||||
|
monkeypatch.setattr("l4d2web.app.start_job_workers", lambda app: None)
|
||||||
|
|
||||||
|
app = create_app({"TESTING": False, "DATABASE_URL": db_url, "SECRET_KEY": "real"})
|
||||||
|
|
||||||
|
assert app.config["SESSION_COOKIE_SECURE"] is False
|
||||||
|
|
||||||
|
|
||||||
def test_session_cookie_secure_disabled_in_testing(tmp_path, monkeypatch) -> None:
|
def test_session_cookie_secure_disabled_in_testing(tmp_path, monkeypatch) -> None:
|
||||||
db_url = f"sqlite:///{tmp_path/'cookie-test.db'}"
|
db_url = f"sqlite:///{tmp_path/'cookie-test.db'}"
|
||||||
monkeypatch.setenv("DATABASE_URL", db_url)
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
|
|
||||||
|
|
@ -72,3 +72,46 @@ def test_stream_command_yields_stdout_lines() -> None:
|
||||||
lines = list(stream_command(["python3", "-c", "print('one'); print('two')"]))
|
lines = list(stream_command(["python3", "-c", "print('one'); print('two')"]))
|
||||||
|
|
||||||
assert lines == ["one", "two"]
|
assert lines == ["one", "two"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_stream_command_emits_heartbeat_when_subprocess_silent() -> None:
|
||||||
|
import time
|
||||||
|
|
||||||
|
from l4d2web.services.host_commands import stream_command
|
||||||
|
|
||||||
|
cmd = [
|
||||||
|
"python3",
|
||||||
|
"-c",
|
||||||
|
"import time; time.sleep(0.4); print('done')",
|
||||||
|
]
|
||||||
|
|
||||||
|
started = time.monotonic()
|
||||||
|
items: list[str] = []
|
||||||
|
for item in stream_command(cmd, heartbeat_interval=0.05):
|
||||||
|
items.append(item)
|
||||||
|
if time.monotonic() - started > 2.0:
|
||||||
|
break
|
||||||
|
|
||||||
|
assert "done" in items, items
|
||||||
|
heartbeats = [i for i in items if i == ""]
|
||||||
|
assert len(heartbeats) >= 2, f"expected ≥2 heartbeat ticks during the silent 0.4s window, got items={items!r}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_stream_command_close_releases_subprocess_promptly() -> None:
|
||||||
|
import time
|
||||||
|
|
||||||
|
from l4d2web.services.host_commands import stream_command
|
||||||
|
|
||||||
|
cmd = [
|
||||||
|
"python3",
|
||||||
|
"-c",
|
||||||
|
"import time;\nwhile True:\n time.sleep(60)",
|
||||||
|
]
|
||||||
|
|
||||||
|
gen = stream_command(cmd, heartbeat_interval=0.05)
|
||||||
|
assert next(gen) == ""
|
||||||
|
|
||||||
|
started = time.monotonic()
|
||||||
|
gen.close()
|
||||||
|
elapsed = time.monotonic() - started
|
||||||
|
assert elapsed < 1.0, f"gen.close() took {elapsed:.2f}s; subprocess cleanup must not block"
|
||||||
|
|
|
||||||
|
|
@ -104,3 +104,28 @@ def test_sse_js_handles_job_log_custom_events() -> None:
|
||||||
|
|
||||||
assert 'addEventListener("stdout"' in js
|
assert 'addEventListener("stdout"' in js
|
||||||
assert 'addEventListener("stderr"' in js
|
assert 'addEventListener("stderr"' in js
|
||||||
|
|
||||||
|
|
||||||
|
def test_system_job_logs_persist(tmp_path, monkeypatch):
|
||||||
|
from l4d2web.models import Job, JobLog
|
||||||
|
from l4d2web.services.job_worker import append_job_log
|
||||||
|
|
||||||
|
monkeypatch.setenv("DATABASE_URL", f"sqlite:///{tmp_path/'system-job-logs.db'}")
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
job = Job(
|
||||||
|
user_id=None,
|
||||||
|
server_id=None,
|
||||||
|
operation="refresh_workshop_items",
|
||||||
|
state="queued",
|
||||||
|
)
|
||||||
|
db.add(job)
|
||||||
|
db.flush()
|
||||||
|
|
||||||
|
seq = append_job_log(db, job.id, "stdout", "queued by system timer")
|
||||||
|
db.flush()
|
||||||
|
|
||||||
|
row = db.query(JobLog).filter_by(job_id=job.id).one()
|
||||||
|
assert seq == 1
|
||||||
|
assert row.line == "queued by system timer"
|
||||||
|
|
|
||||||
|
|
@ -8,16 +8,33 @@ from sqlalchemy import select
|
||||||
|
|
||||||
from l4d2web.auth import hash_password
|
from l4d2web.auth import hash_password
|
||||||
from l4d2web.db import init_db, session_scope
|
from l4d2web.db import init_db, session_scope
|
||||||
from l4d2web.models import Blueprint, Job, Server, User
|
from l4d2web.models import (
|
||||||
|
Blueprint,
|
||||||
|
BlueprintOverlay,
|
||||||
|
Job,
|
||||||
|
Overlay,
|
||||||
|
OverlayWorkshopItem,
|
||||||
|
Server,
|
||||||
|
User,
|
||||||
|
WorkshopItem,
|
||||||
|
)
|
||||||
from l4d2web.services import l4d2_facade
|
from l4d2web.services import l4d2_facade
|
||||||
from l4d2web.services.host_commands import CommandCancelledError
|
from l4d2web.services.host_commands import CommandCancelledError
|
||||||
from l4d2web.services.job_worker import SchedulerState, can_start, recover_stale_jobs, run_worker_once
|
from l4d2web.services.job_worker import (
|
||||||
|
SchedulerState,
|
||||||
|
build_scheduler_state,
|
||||||
|
can_start,
|
||||||
|
enqueue_build_overlay,
|
||||||
|
recover_stale_jobs,
|
||||||
|
run_worker_once,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class DummyJob:
|
class DummyJob:
|
||||||
operation: str
|
operation: str
|
||||||
server_id: int | None = None
|
server_id: int | None = None
|
||||||
|
overlay_id: int | None = None
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
|
|
@ -65,12 +82,14 @@ def add_job(
|
||||||
server_id: int | None,
|
server_id: int | None,
|
||||||
state: str = "queued",
|
state: str = "queued",
|
||||||
created_at: datetime | None = None,
|
created_at: datetime | None = None,
|
||||||
|
overlay_id: int | None = None,
|
||||||
) -> int:
|
) -> int:
|
||||||
now = datetime.now(UTC)
|
now = datetime.now(UTC)
|
||||||
with session_scope() as session:
|
with session_scope() as session:
|
||||||
job = Job(
|
job = Job(
|
||||||
user_id=user_id,
|
user_id=user_id,
|
||||||
server_id=server_id,
|
server_id=server_id,
|
||||||
|
overlay_id=overlay_id,
|
||||||
operation=operation,
|
operation=operation,
|
||||||
state=state,
|
state=state,
|
||||||
created_at=created_at or now,
|
created_at=created_at or now,
|
||||||
|
|
@ -241,6 +260,71 @@ def test_unexpected_exception_fails_job_with_exit_code_one(seeded_worker, monkey
|
||||||
job = load_job(job_id)
|
job = load_job(job_id)
|
||||||
assert job.state == "failed"
|
assert job.state == "failed"
|
||||||
assert job.exit_code == 1
|
assert job.exit_code == 1
|
||||||
|
# Failed delete must keep the Server row.
|
||||||
|
with session_scope() as session:
|
||||||
|
assert session.scalar(select(Server).where(Server.id == ids.server_one)) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_reset_keeps_server_row_and_refreshes_state(seeded_worker, monkeypatch) -> None:
|
||||||
|
app, ids = seeded_worker
|
||||||
|
job_id = add_job(ids.user, "reset", server_id=ids.server_one)
|
||||||
|
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def fake_reset(server_id, *, on_stdout=None, on_stderr=None, should_cancel=None):
|
||||||
|
del should_cancel
|
||||||
|
calls.append(server_id)
|
||||||
|
on_stdout("removing instance files")
|
||||||
|
|
||||||
|
monkeypatch.setattr(l4d2_facade, "reset_server", fake_reset)
|
||||||
|
monkeypatch.setattr(l4d2_facade, "server_status", lambda name: SimpleNamespace(state="stopped"))
|
||||||
|
|
||||||
|
with app.app_context():
|
||||||
|
assert run_worker_once() is True
|
||||||
|
|
||||||
|
assert calls == [ids.server_one]
|
||||||
|
job = load_job(job_id)
|
||||||
|
assert job.state == "succeeded"
|
||||||
|
assert job.exit_code == 0
|
||||||
|
with session_scope() as session:
|
||||||
|
server = session.scalar(select(Server).where(Server.id == ids.server_one))
|
||||||
|
assert server is not None
|
||||||
|
assert server.actual_state == "stopped"
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_delete_removes_server_row(seeded_worker, monkeypatch) -> None:
|
||||||
|
app, ids = seeded_worker
|
||||||
|
job_id = add_job(ids.user, "delete", server_id=ids.server_one)
|
||||||
|
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def fake_delete(server_id, *, on_stdout=None, on_stderr=None, should_cancel=None):
|
||||||
|
del should_cancel
|
||||||
|
calls.append(server_id)
|
||||||
|
on_stdout("removing systemd unit")
|
||||||
|
|
||||||
|
def fake_status(name): # pragma: no cover — would mean delete didn't skip refresh
|
||||||
|
pytest.fail(f"server_status must not be called after a successful delete (got {name!r})")
|
||||||
|
|
||||||
|
monkeypatch.setattr(l4d2_facade, "delete_server", fake_delete)
|
||||||
|
monkeypatch.setattr(l4d2_facade, "server_status", fake_status)
|
||||||
|
|
||||||
|
with app.app_context():
|
||||||
|
assert run_worker_once() is True
|
||||||
|
|
||||||
|
assert calls == [ids.server_one]
|
||||||
|
job = load_job(job_id)
|
||||||
|
assert job.state == "succeeded"
|
||||||
|
assert job.exit_code == 0
|
||||||
|
with session_scope() as session:
|
||||||
|
assert session.scalar(select(Server).where(Server.id == ids.server_one)) is None
|
||||||
|
# Sibling server is untouched.
|
||||||
|
assert session.scalar(select(Server).where(Server.id == ids.server_two)) is not None
|
||||||
|
# The delete job itself stays in the job log; outerjoin in views shows
|
||||||
|
# "-" for its (now-orphaned) server_id pointer.
|
||||||
|
finished_job = session.scalar(select(Job).where(Job.id == job_id))
|
||||||
|
assert finished_job is not None
|
||||||
|
assert finished_job.server_id == ids.server_one
|
||||||
|
|
||||||
|
|
||||||
def test_same_server_jobs_do_not_overlap(seeded_worker, monkeypatch) -> None:
|
def test_same_server_jobs_do_not_overlap(seeded_worker, monkeypatch) -> None:
|
||||||
|
|
@ -379,3 +463,383 @@ def test_worker_startup_when_enabled_outside_testing(monkeypatch, tmp_path) -> N
|
||||||
app = app_module.create_app({"TESTING": False, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
app = app_module.create_app({"TESTING": False, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
|
|
||||||
assert called == [app]
|
assert called == [app]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Scheduler truth table for the new operations (build_overlay,
|
||||||
|
# refresh_workshop_items) and their interaction with existing ops.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_install_blocks_build_overlay_and_refresh() -> None:
|
||||||
|
state = SchedulerState(install_running=True)
|
||||||
|
assert can_start(DummyJob(operation="build_overlay", overlay_id=1), state) is False
|
||||||
|
assert can_start(DummyJob(operation="refresh_workshop_items"), state) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_refresh_blocks_install_build_overlay_and_servers() -> None:
|
||||||
|
state = SchedulerState(refresh_running=True)
|
||||||
|
assert can_start(DummyJob(operation="install"), state) is False
|
||||||
|
assert can_start(DummyJob(operation="build_overlay", overlay_id=1), state) is False
|
||||||
|
assert can_start(DummyJob(operation="start", server_id=1), state) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_overlay_blocks_same_overlay_only() -> None:
|
||||||
|
state = SchedulerState()
|
||||||
|
state.running_overlays.add(7)
|
||||||
|
assert can_start(DummyJob(operation="build_overlay", overlay_id=7), state) is False
|
||||||
|
assert can_start(DummyJob(operation="build_overlay", overlay_id=8), state) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_install_blocked_by_active_build_overlay() -> None:
|
||||||
|
state = SchedulerState()
|
||||||
|
state.running_overlays.add(7)
|
||||||
|
assert can_start(DummyJob(operation="install"), state) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_refresh_blocked_by_active_build_overlay() -> None:
|
||||||
|
state = SchedulerState()
|
||||||
|
state.running_overlays.add(7)
|
||||||
|
assert can_start(DummyJob(operation="refresh_workshop_items"), state) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_server_job_blocked_when_blueprint_overlay_is_building() -> None:
|
||||||
|
state = SchedulerState()
|
||||||
|
state.running_overlays.add(7)
|
||||||
|
state.blocked_servers_by_overlay.add(42)
|
||||||
|
assert can_start(DummyJob(operation="start", server_id=42), state) is False
|
||||||
|
# Other servers (whose blueprints don't reference overlay 7) are NOT blocked.
|
||||||
|
assert can_start(DummyJob(operation="start", server_id=43), state) is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def overlay_seeded_worker(seeded_worker):
|
||||||
|
app, ids = seeded_worker
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = Overlay(name="ws", path="9", type="workshop", user_id=ids.user)
|
||||||
|
s.add(overlay)
|
||||||
|
s.flush()
|
||||||
|
# Move server_two onto a different blueprint with NO workshop overlay,
|
||||||
|
# so the test can distinguish "blocked by overlay build" from "any
|
||||||
|
# server is blocked".
|
||||||
|
bp_with_overlay = s.scalar(select(Blueprint).where(Blueprint.user_id == ids.user))
|
||||||
|
s.add(BlueprintOverlay(blueprint_id=bp_with_overlay.id, overlay_id=overlay.id, position=0))
|
||||||
|
bp_without = Blueprint(user_id=ids.user, name="no-overlay", arguments="[]", config="[]")
|
||||||
|
s.add(bp_without)
|
||||||
|
s.flush()
|
||||||
|
server_two = s.scalar(select(Server).where(Server.id == ids.server_two))
|
||||||
|
server_two.blueprint_id = bp_without.id
|
||||||
|
ids.overlay = overlay.id
|
||||||
|
return app, ids
|
||||||
|
|
||||||
|
|
||||||
|
def test_scheduler_state_finds_servers_blocked_by_running_build(overlay_seeded_worker) -> None:
|
||||||
|
app, ids = overlay_seeded_worker
|
||||||
|
add_job(ids.user, "build_overlay", server_id=None, state="running", overlay_id=ids.overlay)
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
state = build_scheduler_state(s)
|
||||||
|
|
||||||
|
assert ids.overlay in state.running_overlays
|
||||||
|
assert ids.server_one in state.blocked_servers_by_overlay
|
||||||
|
assert ids.server_two not in state.blocked_servers_by_overlay
|
||||||
|
|
||||||
|
|
||||||
|
def test_enqueue_build_overlay_creates_new_job_when_none_pending(overlay_seeded_worker) -> None:
|
||||||
|
app, ids = overlay_seeded_worker
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
job = enqueue_build_overlay(s, overlay_id=ids.overlay, user_id=ids.user)
|
||||||
|
assert job.operation == "build_overlay"
|
||||||
|
assert job.overlay_id == ids.overlay
|
||||||
|
assert job.server_id is None
|
||||||
|
assert job.state == "queued"
|
||||||
|
|
||||||
|
|
||||||
|
def test_enqueue_build_overlay_coalesces_against_pending(overlay_seeded_worker) -> None:
|
||||||
|
app, ids = overlay_seeded_worker
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
first = enqueue_build_overlay(s, overlay_id=ids.overlay, user_id=ids.user)
|
||||||
|
first_id = first.id
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
second = enqueue_build_overlay(s, overlay_id=ids.overlay, user_id=ids.user)
|
||||||
|
assert second.id == first_id, "should coalesce against the pending job"
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
n = s.query(Job).filter_by(operation="build_overlay", overlay_id=ids.overlay).count()
|
||||||
|
assert n == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_enqueue_build_overlay_does_not_coalesce_against_running(overlay_seeded_worker) -> None:
|
||||||
|
app, ids = overlay_seeded_worker
|
||||||
|
add_job(ids.user, "build_overlay", server_id=None, state="running", overlay_id=ids.overlay)
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
new_job = enqueue_build_overlay(s, overlay_id=ids.overlay, user_id=ids.user)
|
||||||
|
assert new_job.state == "queued"
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
running = s.scalars(
|
||||||
|
select(Job).where(
|
||||||
|
Job.operation == "build_overlay",
|
||||||
|
Job.overlay_id == ids.overlay,
|
||||||
|
Job.state == "running",
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
queued = s.scalars(
|
||||||
|
select(Job).where(
|
||||||
|
Job.operation == "build_overlay",
|
||||||
|
Job.overlay_id == ids.overlay,
|
||||||
|
Job.state == "queued",
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
assert len(running) == 1
|
||||||
|
assert len(queued) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_worker_once_dispatches_build_overlay(overlay_seeded_worker, monkeypatch, tmp_path) -> None:
|
||||||
|
app, ids = overlay_seeded_worker
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
wi = WorkshopItem(steam_id="1001", title="A", filename="a.vpk", file_url="u", file_size=3, time_updated=1)
|
||||||
|
s.add(wi)
|
||||||
|
s.flush()
|
||||||
|
s.add(OverlayWorkshopItem(overlay_id=ids.overlay, workshop_item_id=wi.id))
|
||||||
|
cache = tmp_path / "workshop_cache"
|
||||||
|
cache.mkdir()
|
||||||
|
(cache / "1001.vpk").write_bytes(b"abc")
|
||||||
|
import os
|
||||||
|
os.utime(cache / "1001.vpk", (1, 1))
|
||||||
|
# Mark item as downloaded.
|
||||||
|
with session_scope() as s:
|
||||||
|
wi = s.query(WorkshopItem).filter_by(steam_id="1001").one()
|
||||||
|
wi.last_downloaded_at = datetime.now(UTC)
|
||||||
|
|
||||||
|
job_id = add_job(ids.user, "build_overlay", server_id=None, overlay_id=ids.overlay)
|
||||||
|
|
||||||
|
with app.app_context():
|
||||||
|
assert run_worker_once() is True
|
||||||
|
|
||||||
|
job = load_job(job_id)
|
||||||
|
assert job.state == "succeeded", job.exit_code
|
||||||
|
addons = tmp_path / "overlays" / "9" / "left4dead2" / "addons"
|
||||||
|
assert (addons / "1001.vpk").is_symlink()
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_overlay_writes_last_build_status_ok(
|
||||||
|
overlay_seeded_worker, monkeypatch, tmp_path
|
||||||
|
) -> None:
|
||||||
|
app, ids = overlay_seeded_worker
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
|
||||||
|
from l4d2web.services import overlay_builders
|
||||||
|
|
||||||
|
class _StubBuilder:
|
||||||
|
def build(self, overlay, *, on_stdout, on_stderr, should_cancel):
|
||||||
|
on_stdout("stub build ok")
|
||||||
|
|
||||||
|
monkeypatch.setitem(overlay_builders.BUILDERS, "workshop", _StubBuilder())
|
||||||
|
|
||||||
|
job_id = add_job(ids.user, "build_overlay", server_id=None, overlay_id=ids.overlay)
|
||||||
|
|
||||||
|
with app.app_context():
|
||||||
|
assert run_worker_once() is True
|
||||||
|
|
||||||
|
assert load_job(job_id).state == "succeeded"
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=ids.overlay).one()
|
||||||
|
assert overlay.last_build_status == "ok"
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_overlay_writes_last_build_status_failed(
|
||||||
|
overlay_seeded_worker, monkeypatch, tmp_path
|
||||||
|
) -> None:
|
||||||
|
app, ids = overlay_seeded_worker
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
|
||||||
|
from l4d2web.services import overlay_builders
|
||||||
|
|
||||||
|
class _FailingBuilder:
|
||||||
|
def build(self, overlay, *, on_stdout, on_stderr, should_cancel):
|
||||||
|
raise RuntimeError("synthetic build failure")
|
||||||
|
|
||||||
|
monkeypatch.setitem(overlay_builders.BUILDERS, "workshop", _FailingBuilder())
|
||||||
|
|
||||||
|
job_id = add_job(ids.user, "build_overlay", server_id=None, overlay_id=ids.overlay)
|
||||||
|
|
||||||
|
with app.app_context():
|
||||||
|
assert run_worker_once() is True
|
||||||
|
|
||||||
|
assert load_job(job_id).state == "failed"
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=ids.overlay).one()
|
||||||
|
assert overlay.last_build_status == "failed"
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_worker_once_dispatches_refresh(overlay_seeded_worker, monkeypatch, tmp_path) -> None:
|
||||||
|
app, ids = overlay_seeded_worker
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
|
||||||
|
from l4d2web.services import steam_workshop, job_worker
|
||||||
|
|
||||||
|
refresh_calls = []
|
||||||
|
|
||||||
|
def fake_refresh_workshop_items(*, on_stdout, on_stderr, should_cancel):
|
||||||
|
refresh_calls.append(True)
|
||||||
|
on_stdout("refresh phase complete (downloaded=0 errors=0)")
|
||||||
|
return [] # no overlays affected
|
||||||
|
|
||||||
|
monkeypatch.setattr(job_worker, "_run_refresh_workshop_items", fake_refresh_workshop_items)
|
||||||
|
|
||||||
|
job_id = add_job(ids.user, "refresh_workshop_items", server_id=None)
|
||||||
|
|
||||||
|
with app.app_context():
|
||||||
|
assert run_worker_once() is True
|
||||||
|
|
||||||
|
assert refresh_calls == [True]
|
||||||
|
job = load_job(job_id)
|
||||||
|
assert job.state == "succeeded"
|
||||||
|
|
||||||
|
|
||||||
|
def test_refresh_downloads_serially_to_keep_web_worker_responsive(
|
||||||
|
overlay_seeded_worker, monkeypatch, tmp_path
|
||||||
|
) -> None:
|
||||||
|
app, ids = overlay_seeded_worker
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
|
||||||
|
from l4d2web.services import job_worker, steam_workshop
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
wi = WorkshopItem(
|
||||||
|
steam_id="1001",
|
||||||
|
title="A",
|
||||||
|
filename="old.vpk",
|
||||||
|
file_url="https://example.com/old.vpk",
|
||||||
|
file_size=1,
|
||||||
|
time_updated=1,
|
||||||
|
)
|
||||||
|
s.add(wi)
|
||||||
|
s.flush()
|
||||||
|
s.add(OverlayWorkshopItem(overlay_id=ids.overlay, workshop_item_id=wi.id))
|
||||||
|
|
||||||
|
meta = steam_workshop.WorkshopMetadata(
|
||||||
|
steam_id="1001",
|
||||||
|
title="A",
|
||||||
|
filename="new.vpk",
|
||||||
|
file_url="https://example.com/new.vpk",
|
||||||
|
file_size=2,
|
||||||
|
time_updated=2,
|
||||||
|
preview_url="",
|
||||||
|
consumer_app_id=550,
|
||||||
|
result=1,
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(steam_workshop, "fetch_metadata_batch", lambda steam_ids, *, mode: [meta])
|
||||||
|
refresh_calls = []
|
||||||
|
|
||||||
|
def fake_refresh_all(metas, cache_root, *, executor_workers=8, should_cancel=None):
|
||||||
|
refresh_calls.append((list(metas), executor_workers, should_cancel is not None))
|
||||||
|
return steam_workshop.RefreshReport(downloaded=1, errors=0)
|
||||||
|
|
||||||
|
monkeypatch.setattr(steam_workshop, "refresh_all", fake_refresh_all)
|
||||||
|
|
||||||
|
with app.app_context():
|
||||||
|
affected = job_worker._run_refresh_workshop_items(
|
||||||
|
on_stdout=lambda _line: None,
|
||||||
|
on_stderr=lambda _line: None,
|
||||||
|
should_cancel=lambda: False,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert affected == [ids.overlay]
|
||||||
|
assert refresh_calls == [([meta], 1, True)]
|
||||||
|
|
||||||
|
|
||||||
|
def test_refresh_job_enqueues_build_overlay_without_locking_its_final_log(
|
||||||
|
overlay_seeded_worker, monkeypatch, tmp_path
|
||||||
|
) -> None:
|
||||||
|
app, ids = overlay_seeded_worker
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
|
||||||
|
from l4d2web.services import steam_workshop
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
wi = WorkshopItem(
|
||||||
|
steam_id="1001",
|
||||||
|
title="A",
|
||||||
|
filename="old.vpk",
|
||||||
|
file_url="https://example.com/old.vpk",
|
||||||
|
file_size=1,
|
||||||
|
time_updated=1,
|
||||||
|
)
|
||||||
|
s.add(wi)
|
||||||
|
s.flush()
|
||||||
|
s.add(OverlayWorkshopItem(overlay_id=ids.overlay, workshop_item_id=wi.id))
|
||||||
|
|
||||||
|
meta = steam_workshop.WorkshopMetadata(
|
||||||
|
steam_id="1001",
|
||||||
|
title="A",
|
||||||
|
filename="new.vpk",
|
||||||
|
file_url="https://example.com/new.vpk",
|
||||||
|
file_size=2,
|
||||||
|
time_updated=2,
|
||||||
|
preview_url="",
|
||||||
|
consumer_app_id=550,
|
||||||
|
result=1,
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(steam_workshop, "fetch_metadata_batch", lambda steam_ids, *, mode: [meta])
|
||||||
|
monkeypatch.setattr(
|
||||||
|
steam_workshop,
|
||||||
|
"refresh_all",
|
||||||
|
lambda metas, cache_root, **kwargs: steam_workshop.RefreshReport(downloaded=1, errors=0),
|
||||||
|
)
|
||||||
|
|
||||||
|
job_id = add_job(ids.user, "refresh_workshop_items", server_id=None)
|
||||||
|
|
||||||
|
with app.app_context():
|
||||||
|
assert run_worker_once() is True
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
job = s.scalar(select(Job).where(Job.id == job_id))
|
||||||
|
build_job = s.scalar(
|
||||||
|
select(Job).where(
|
||||||
|
Job.operation == "build_overlay",
|
||||||
|
Job.overlay_id == ids.overlay,
|
||||||
|
Job.state == "queued",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
lines = [row.line for row in job_logs_for(s, job_id)]
|
||||||
|
|
||||||
|
assert job is not None
|
||||||
|
assert job.state == "succeeded"
|
||||||
|
assert build_job is not None
|
||||||
|
assert "enqueued build_overlay for 1 overlay(s)" in lines
|
||||||
|
|
||||||
|
|
||||||
|
def test_global_operations_set() -> None:
|
||||||
|
from l4d2web.services.job_worker import GLOBAL_OPERATIONS
|
||||||
|
|
||||||
|
assert GLOBAL_OPERATIONS == {"install", "refresh_workshop_items"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_overlay_script_type_blocks_per_overlay(overlay_seeded_worker) -> None:
|
||||||
|
"""Mechanically identical to workshop builds, but pinned for script type."""
|
||||||
|
app, ids = overlay_seeded_worker
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=ids.overlay).one()
|
||||||
|
overlay.type = "script"
|
||||||
|
overlay.script = "echo hi"
|
||||||
|
|
||||||
|
add_job(ids.user, "build_overlay", server_id=None, state="running", overlay_id=ids.overlay)
|
||||||
|
|
||||||
|
state = SchedulerState()
|
||||||
|
state.running_overlays.add(ids.overlay)
|
||||||
|
assert (
|
||||||
|
can_start(DummyJob(operation="build_overlay", overlay_id=ids.overlay), state)
|
||||||
|
is False
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
can_start(DummyJob(operation="build_overlay", overlay_id=ids.overlay + 1), state)
|
||||||
|
is True
|
||||||
|
)
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,4 @@
|
||||||
|
from datetime import UTC, datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
@ -5,7 +6,15 @@ import pytest
|
||||||
from l4d2web.app import create_app
|
from l4d2web.app import create_app
|
||||||
from l4d2web.auth import hash_password
|
from l4d2web.auth import hash_password
|
||||||
from l4d2web.db import init_db, session_scope
|
from l4d2web.db import init_db, session_scope
|
||||||
from l4d2web.models import Blueprint, BlueprintOverlay, Overlay, Server, User
|
from l4d2web.models import (
|
||||||
|
Blueprint,
|
||||||
|
BlueprintOverlay,
|
||||||
|
Overlay,
|
||||||
|
OverlayWorkshopItem,
|
||||||
|
Server,
|
||||||
|
User,
|
||||||
|
WorkshopItem,
|
||||||
|
)
|
||||||
from l4d2web.services.host_commands import CommandResult
|
from l4d2web.services.host_commands import CommandResult
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -13,6 +22,7 @@ from l4d2web.services.host_commands import CommandResult
|
||||||
def server_with_blueprint(tmp_path, monkeypatch):
|
def server_with_blueprint(tmp_path, monkeypatch):
|
||||||
db_url = f"sqlite:///{tmp_path/'facade.db'}"
|
db_url = f"sqlite:///{tmp_path/'facade.db'}"
|
||||||
monkeypatch.setenv("DATABASE_URL", db_url)
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
init_db()
|
init_db()
|
||||||
|
|
||||||
|
|
@ -22,7 +32,7 @@ def server_with_blueprint(tmp_path, monkeypatch):
|
||||||
session.add(user)
|
session.add(user)
|
||||||
session.flush()
|
session.flush()
|
||||||
|
|
||||||
overlay = Overlay(name="Standard Overlay", path="standard")
|
overlay = Overlay(name="Standard Overlay", path="standard", type="workshop", user_id=user.id)
|
||||||
session.add(overlay)
|
session.add(overlay)
|
||||||
session.flush()
|
session.flush()
|
||||||
|
|
||||||
|
|
@ -41,8 +51,9 @@ def server_with_blueprint(tmp_path, monkeypatch):
|
||||||
session.add(server)
|
session.add(server)
|
||||||
session.flush()
|
session.flush()
|
||||||
server_id = server.id
|
server_id = server.id
|
||||||
|
user_id = user.id
|
||||||
|
|
||||||
return server_id
|
return server_id, user_id
|
||||||
|
|
||||||
|
|
||||||
def test_initialize_uses_l4d2ctl_with_latest_blueprint_data(
|
def test_initialize_uses_l4d2ctl_with_latest_blueprint_data(
|
||||||
|
|
@ -70,9 +81,10 @@ def test_initialize_uses_l4d2ctl_with_latest_blueprint_data(
|
||||||
|
|
||||||
from l4d2web.services.l4d2_facade import initialize_server
|
from l4d2web.services.l4d2_facade import initialize_server
|
||||||
|
|
||||||
initialize_server(server_with_blueprint)
|
server_id, _ = server_with_blueprint
|
||||||
|
initialize_server(server_id)
|
||||||
|
|
||||||
assert calls[0][:3] == ["l4d2ctl", "initialize", "alpha"]
|
assert calls[0][:3] == ["l4d2ctl", "initialize", str(server_id)]
|
||||||
assert calls[0][3] == "-f"
|
assert calls[0][3] == "-f"
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -88,25 +100,35 @@ def test_install_and_lifecycle_commands_use_l4d2ctl(
|
||||||
return CommandResult(returncode=0, stdout="", stderr="")
|
return CommandResult(returncode=0, stdout="", stderr="")
|
||||||
|
|
||||||
monkeypatch.setattr("l4d2web.services.host_commands.run_command", fake_run_command)
|
monkeypatch.setattr("l4d2web.services.host_commands.run_command", fake_run_command)
|
||||||
for name in ["SteamInstaller", "start_instance", "stop_instance", "delete_instance"]:
|
for name in ["SteamInstaller", "start_instance", "stop_instance", "delete_instance", "reset_instance"]:
|
||||||
monkeypatch.setattr(
|
monkeypatch.setattr(
|
||||||
f"l4d2web.services.l4d2_facade.{name}",
|
f"l4d2web.services.l4d2_facade.{name}",
|
||||||
lambda *args, **kwargs: pytest.fail(f"facade must not call l4d2host {name} directly"),
|
lambda *args, **kwargs: pytest.fail(f"facade must not call l4d2host {name} directly"),
|
||||||
raising=False,
|
raising=False,
|
||||||
)
|
)
|
||||||
|
|
||||||
from l4d2web.services.l4d2_facade import delete_server, install_runtime, start_server, stop_server
|
from l4d2web.services.l4d2_facade import (
|
||||||
|
delete_server,
|
||||||
|
install_runtime,
|
||||||
|
reset_server,
|
||||||
|
start_server,
|
||||||
|
stop_server,
|
||||||
|
)
|
||||||
|
|
||||||
|
server_id, _ = server_with_blueprint
|
||||||
install_runtime()
|
install_runtime()
|
||||||
start_server(server_with_blueprint)
|
start_server(server_id)
|
||||||
stop_server(server_with_blueprint)
|
stop_server(server_id)
|
||||||
delete_server(server_with_blueprint)
|
reset_server(server_id)
|
||||||
|
delete_server(server_id)
|
||||||
|
|
||||||
|
unit = str(server_id)
|
||||||
assert calls == [
|
assert calls == [
|
||||||
["l4d2ctl", "install"],
|
["l4d2ctl", "install"],
|
||||||
["l4d2ctl", "start", "alpha"],
|
["l4d2ctl", "start", unit],
|
||||||
["l4d2ctl", "stop", "alpha"],
|
["l4d2ctl", "stop", unit],
|
||||||
["l4d2ctl", "delete", "alpha"],
|
["l4d2ctl", "reset", unit],
|
||||||
|
["l4d2ctl", "delete", unit],
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -159,3 +181,91 @@ def test_server_logs_stream_l4d2ctl_logs(monkeypatch: pytest.MonkeyPatch) -> Non
|
||||||
|
|
||||||
assert calls == [["l4d2ctl", "logs", "alpha", "--lines", "10", "--no-follow"]]
|
assert calls == [["l4d2ctl", "logs", "alpha", "--lines", "10", "--no-follow"]]
|
||||||
assert lines == ["one", "two"]
|
assert lines == ["one", "two"]
|
||||||
|
|
||||||
|
|
||||||
|
def _attach_workshop_overlay_to_blueprint(
|
||||||
|
server_id: int, user_id: int, *, item_cached: bool, tmp_path: Path
|
||||||
|
) -> tuple[int, str]:
|
||||||
|
"""Add a workshop overlay with a single workshop item to the server's
|
||||||
|
blueprint. Returns (overlay_id, steam_id)."""
|
||||||
|
with session_scope() as session:
|
||||||
|
server = session.query(Server).filter_by(id=server_id).one()
|
||||||
|
overlay = Overlay(name="ws", path="placeholder", type="workshop", user_id=user_id)
|
||||||
|
session.add(overlay)
|
||||||
|
session.flush()
|
||||||
|
# Path matches id, like the production create_overlay flow does.
|
||||||
|
overlay.path = str(overlay.id)
|
||||||
|
wi = WorkshopItem(
|
||||||
|
steam_id="1001",
|
||||||
|
title="A",
|
||||||
|
filename="a.vpk",
|
||||||
|
file_url="https://example.com/a.vpk",
|
||||||
|
file_size=3,
|
||||||
|
time_updated=1700000000,
|
||||||
|
last_downloaded_at=datetime.now(UTC) if item_cached else None,
|
||||||
|
)
|
||||||
|
session.add(wi)
|
||||||
|
session.flush()
|
||||||
|
session.add(
|
||||||
|
BlueprintOverlay(blueprint_id=server.blueprint_id, overlay_id=overlay.id, position=1)
|
||||||
|
)
|
||||||
|
session.add(OverlayWorkshopItem(overlay_id=overlay.id, workshop_item_id=wi.id))
|
||||||
|
overlay_id = overlay.id
|
||||||
|
|
||||||
|
if item_cached:
|
||||||
|
cache_root = tmp_path / "workshop_cache"
|
||||||
|
cache_root.mkdir(exist_ok=True)
|
||||||
|
(cache_root / "1001.vpk").write_bytes(b"abc")
|
||||||
|
|
||||||
|
return overlay_id, "1001"
|
||||||
|
|
||||||
|
|
||||||
|
def test_initialize_runs_overlay_builders_synchronously(
|
||||||
|
monkeypatch: pytest.MonkeyPatch, server_with_blueprint, tmp_path
|
||||||
|
) -> None:
|
||||||
|
server_id, user_id = server_with_blueprint
|
||||||
|
overlay_id, _steam_id = _attach_workshop_overlay_to_blueprint(
|
||||||
|
server_id, user_id, item_cached=True, tmp_path=tmp_path
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"l4d2web.services.host_commands.run_command",
|
||||||
|
lambda *args, **kwargs: CommandResult(returncode=0, stdout="", stderr=""),
|
||||||
|
)
|
||||||
|
|
||||||
|
from l4d2web.services.l4d2_facade import initialize_server
|
||||||
|
|
||||||
|
initialize_server(server_id)
|
||||||
|
|
||||||
|
addons = tmp_path / "overlays" / str(overlay_id) / "left4dead2" / "addons"
|
||||||
|
assert (addons / "1001.vpk").is_symlink(), "workshop builder must run before l4d2ctl initialize"
|
||||||
|
|
||||||
|
|
||||||
|
def test_initialize_fails_fast_on_uncached_workshop_items(
|
||||||
|
monkeypatch: pytest.MonkeyPatch, server_with_blueprint, tmp_path
|
||||||
|
) -> None:
|
||||||
|
server_id, user_id = server_with_blueprint
|
||||||
|
overlay_id, steam_id = _attach_workshop_overlay_to_blueprint(
|
||||||
|
server_id, user_id, item_cached=False, tmp_path=tmp_path
|
||||||
|
)
|
||||||
|
|
||||||
|
invocations: list[list[str]] = []
|
||||||
|
|
||||||
|
def fake_run_command(cmd, **kwargs):
|
||||||
|
invocations.append(list(cmd))
|
||||||
|
return CommandResult(returncode=0, stdout="", stderr="")
|
||||||
|
|
||||||
|
monkeypatch.setattr("l4d2web.services.host_commands.run_command", fake_run_command)
|
||||||
|
|
||||||
|
from l4d2web.services.l4d2_facade import initialize_server
|
||||||
|
|
||||||
|
with pytest.raises(Exception) as excinfo:
|
||||||
|
initialize_server(server_id)
|
||||||
|
|
||||||
|
msg = str(excinfo.value)
|
||||||
|
assert steam_id in msg
|
||||||
|
assert str(overlay_id) in msg or "ws" in msg
|
||||||
|
# l4d2ctl initialize MUST NOT run when uncached items are present.
|
||||||
|
assert all("initialize" not in cmd for cmd in invocations), invocations
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
359
l4d2web/tests/test_overlay_builders.py
Normal file
359
l4d2web/tests/test_overlay_builders.py
Normal file
|
|
@ -0,0 +1,359 @@
|
||||||
|
"""Tests for overlay builders (registry, WorkshopBuilder, ScriptBuilder)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from l4d2web.db import init_db, session_scope
|
||||||
|
from l4d2web.models import Overlay, OverlayWorkshopItem, User, WorkshopItem
|
||||||
|
from l4d2web.services import overlay_builders
|
||||||
|
from l4d2web.services.host_commands import CommandCancelledError, CommandResult
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def env(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("DATABASE_URL", f"sqlite:///{tmp_path/'b.db'}")
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
init_db()
|
||||||
|
yield tmp_path
|
||||||
|
|
||||||
|
|
||||||
|
def _create_user_and_overlay(name: str, type_: str) -> tuple[int, int]:
|
||||||
|
with session_scope() as s:
|
||||||
|
user = User(username="alice", password_digest="x")
|
||||||
|
s.add(user)
|
||||||
|
s.flush()
|
||||||
|
overlay = Overlay(name=name, path=str(7), type=type_, user_id=user.id)
|
||||||
|
s.add(overlay)
|
||||||
|
s.flush()
|
||||||
|
return user.id, overlay.id
|
||||||
|
|
||||||
|
|
||||||
|
def _add_workshop_item(steam_id: str, *, downloaded: bool, cache_root: Path, content: bytes = b"x") -> int:
|
||||||
|
if downloaded:
|
||||||
|
cache_root.mkdir(parents=True, exist_ok=True)
|
||||||
|
(cache_root / f"{steam_id}.vpk").write_bytes(content)
|
||||||
|
with session_scope() as s:
|
||||||
|
wi = WorkshopItem(
|
||||||
|
steam_id=steam_id,
|
||||||
|
title=f"item-{steam_id}",
|
||||||
|
filename=f"orig-{steam_id}.vpk",
|
||||||
|
file_url=f"https://example.com/{steam_id}.vpk",
|
||||||
|
file_size=len(content) if downloaded else 0,
|
||||||
|
time_updated=1700000000 if downloaded else 0,
|
||||||
|
last_downloaded_at=datetime.now(UTC) if downloaded else None,
|
||||||
|
)
|
||||||
|
s.add(wi)
|
||||||
|
s.flush()
|
||||||
|
return wi.id
|
||||||
|
|
||||||
|
|
||||||
|
def _associate(overlay_id: int, item_id: int) -> None:
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(OverlayWorkshopItem(overlay_id=overlay_id, workshop_item_id=item_id))
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_logs():
|
||||||
|
out: list[str] = []
|
||||||
|
err: list[str] = []
|
||||||
|
return out, err, out.append, err.append
|
||||||
|
|
||||||
|
|
||||||
|
def test_builders_registry() -> None:
|
||||||
|
assert set(overlay_builders.BUILDERS) == {"workshop", "script"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_registry_excludes_legacy_types() -> None:
|
||||||
|
for legacy in ("external", "l4d2center_maps", "cedapug_maps"):
|
||||||
|
assert legacy not in overlay_builders.BUILDERS
|
||||||
|
|
||||||
|
|
||||||
|
def test_registry_unknown_type_raises_keyerror() -> None:
|
||||||
|
with pytest.raises(KeyError):
|
||||||
|
overlay_builders.BUILDERS["nope"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_workshop_builder_unchanged() -> None:
|
||||||
|
"""Regression guard against accidental removal during refactor."""
|
||||||
|
builder = overlay_builders.BUILDERS["workshop"]
|
||||||
|
assert isinstance(builder, overlay_builders.WorkshopBuilder)
|
||||||
|
assert hasattr(builder, "build")
|
||||||
|
|
||||||
|
|
||||||
|
def test_workshop_builder_creates_absolute_symlinks(env: Path) -> None:
|
||||||
|
_, overlay_id = _create_user_and_overlay("ws", "workshop")
|
||||||
|
cache_root = env / "workshop_cache"
|
||||||
|
item_a = _add_workshop_item("1001", downloaded=True, cache_root=cache_root, content=b"AAA")
|
||||||
|
item_b = _add_workshop_item("1002", downloaded=True, cache_root=cache_root, content=b"BBBB")
|
||||||
|
_associate(overlay_id, item_a)
|
||||||
|
_associate(overlay_id, item_b)
|
||||||
|
|
||||||
|
out, err, on_stdout, on_stderr = _capture_logs()
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
overlay_builders.BUILDERS["workshop"].build(
|
||||||
|
overlay, on_stdout=on_stdout, on_stderr=on_stderr, should_cancel=lambda: False
|
||||||
|
)
|
||||||
|
|
||||||
|
addons = env / "overlays" / "7" / "left4dead2" / "addons"
|
||||||
|
link_a = addons / "1001.vpk"
|
||||||
|
link_b = addons / "1002.vpk"
|
||||||
|
assert link_a.is_symlink()
|
||||||
|
assert link_b.is_symlink()
|
||||||
|
# Targets must be ABSOLUTE so they resolve in the host's namespace.
|
||||||
|
assert os.path.isabs(os.readlink(link_a))
|
||||||
|
assert os.path.isabs(os.readlink(link_b))
|
||||||
|
# And they must resolve to the cache files.
|
||||||
|
assert link_a.resolve() == (cache_root / "1001.vpk").resolve()
|
||||||
|
assert link_b.resolve() == (cache_root / "1002.vpk").resolve()
|
||||||
|
|
||||||
|
|
||||||
|
def test_workshop_builder_skips_uncached_items_with_warning(env: Path) -> None:
|
||||||
|
_, overlay_id = _create_user_and_overlay("ws", "workshop")
|
||||||
|
cache_root = env / "workshop_cache"
|
||||||
|
|
||||||
|
cached = _add_workshop_item("1001", downloaded=True, cache_root=cache_root)
|
||||||
|
uncached = _add_workshop_item("9999", downloaded=False, cache_root=cache_root)
|
||||||
|
_associate(overlay_id, cached)
|
||||||
|
_associate(overlay_id, uncached)
|
||||||
|
|
||||||
|
out, err, on_stdout, on_stderr = _capture_logs()
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
overlay_builders.BUILDERS["workshop"].build(
|
||||||
|
overlay, on_stdout=on_stdout, on_stderr=on_stderr, should_cancel=lambda: False
|
||||||
|
)
|
||||||
|
|
||||||
|
addons = env / "overlays" / "7" / "left4dead2" / "addons"
|
||||||
|
assert (addons / "1001.vpk").is_symlink()
|
||||||
|
assert not (addons / "9999.vpk").exists(), "must NOT create dangling symlink"
|
||||||
|
assert any("9999" in line and ("skip" in line.lower() or "uncached" in line.lower()) for line in err + out), err + out
|
||||||
|
|
||||||
|
|
||||||
|
def test_workshop_builder_rerun_is_idempotent(env: Path) -> None:
|
||||||
|
_, overlay_id = _create_user_and_overlay("ws", "workshop")
|
||||||
|
cache_root = env / "workshop_cache"
|
||||||
|
item = _add_workshop_item("1001", downloaded=True, cache_root=cache_root)
|
||||||
|
_associate(overlay_id, item)
|
||||||
|
|
||||||
|
addons = env / "overlays" / "7" / "left4dead2" / "addons"
|
||||||
|
|
||||||
|
# First run.
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
overlay_builders.BUILDERS["workshop"].build(overlay, on_stdout=lambda _x: None, on_stderr=lambda _x: None, should_cancel=lambda: False)
|
||||||
|
first_inode = (addons / "1001.vpk").lstat().st_ino
|
||||||
|
|
||||||
|
# Second run — no-op.
|
||||||
|
out, err, on_stdout, on_stderr = _capture_logs()
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
overlay_builders.BUILDERS["workshop"].build(overlay, on_stdout=on_stdout, on_stderr=on_stderr, should_cancel=lambda: False)
|
||||||
|
|
||||||
|
second_inode = (addons / "1001.vpk").lstat().st_ino
|
||||||
|
assert first_inode == second_inode, "symlink should be untouched on idempotent rebuild"
|
||||||
|
assert any("unchanged" in line.lower() for line in out), out
|
||||||
|
|
||||||
|
|
||||||
|
def test_workshop_builder_removes_obsolete_symlinks(env: Path) -> None:
|
||||||
|
_, overlay_id = _create_user_and_overlay("ws", "workshop")
|
||||||
|
cache_root = env / "workshop_cache"
|
||||||
|
item_a = _add_workshop_item("1001", downloaded=True, cache_root=cache_root)
|
||||||
|
item_b = _add_workshop_item("1002", downloaded=True, cache_root=cache_root)
|
||||||
|
_associate(overlay_id, item_a)
|
||||||
|
_associate(overlay_id, item_b)
|
||||||
|
|
||||||
|
addons = env / "overlays" / "7" / "left4dead2" / "addons"
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
overlay_builders.BUILDERS["workshop"].build(overlay, on_stdout=lambda _x: None, on_stderr=lambda _x: None, should_cancel=lambda: False)
|
||||||
|
assert (addons / "1002.vpk").is_symlink()
|
||||||
|
|
||||||
|
# Remove the association for 1002.
|
||||||
|
with session_scope() as s:
|
||||||
|
s.query(OverlayWorkshopItem).filter_by(workshop_item_id=item_b).delete()
|
||||||
|
|
||||||
|
out, err, on_stdout, on_stderr = _capture_logs()
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
overlay_builders.BUILDERS["workshop"].build(overlay, on_stdout=on_stdout, on_stderr=on_stderr, should_cancel=lambda: False)
|
||||||
|
|
||||||
|
assert (addons / "1001.vpk").is_symlink()
|
||||||
|
assert not (addons / "1002.vpk").exists()
|
||||||
|
# Cache file must remain — overlays are diff-applied, cache is shared.
|
||||||
|
assert (cache_root / "1002.vpk").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_workshop_builder_leaves_unrelated_files_alone(env: Path) -> None:
|
||||||
|
_, overlay_id = _create_user_and_overlay("ws", "workshop")
|
||||||
|
cache_root = env / "workshop_cache"
|
||||||
|
item = _add_workshop_item("1001", downloaded=True, cache_root=cache_root)
|
||||||
|
_associate(overlay_id, item)
|
||||||
|
|
||||||
|
addons = env / "overlays" / "7" / "left4dead2" / "addons"
|
||||||
|
addons.mkdir(parents=True, exist_ok=True)
|
||||||
|
(addons / "manual_addon.vpk").write_bytes(b"hand-placed")
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
overlay_builders.BUILDERS["workshop"].build(overlay, on_stdout=lambda _x: None, on_stderr=lambda _x: None, should_cancel=lambda: False)
|
||||||
|
|
||||||
|
# Manual file is preserved.
|
||||||
|
assert (addons / "manual_addon.vpk").read_bytes() == b"hand-placed"
|
||||||
|
# Workshop symlink is created alongside.
|
||||||
|
assert (addons / "1001.vpk").is_symlink()
|
||||||
|
|
||||||
|
|
||||||
|
def test_workshop_builder_honors_should_cancel(env: Path) -> None:
|
||||||
|
_, overlay_id = _create_user_and_overlay("ws", "workshop")
|
||||||
|
cache_root = env / "workshop_cache"
|
||||||
|
items = [_add_workshop_item(f"100{i}", downloaded=True, cache_root=cache_root) for i in range(3)]
|
||||||
|
for it in items:
|
||||||
|
_associate(overlay_id, it)
|
||||||
|
|
||||||
|
cancel_calls = {"n": 0}
|
||||||
|
|
||||||
|
def cancel():
|
||||||
|
cancel_calls["n"] += 1
|
||||||
|
return cancel_calls["n"] > 0 # cancel immediately
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
# Should not crash; partial state is consistent (re-run heals).
|
||||||
|
overlay_builders.BUILDERS["workshop"].build(
|
||||||
|
overlay, on_stdout=lambda _x: None, on_stderr=lambda _x: None, should_cancel=cancel
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- ScriptBuilder ---------------------------------------------------------
|
||||||
|
|
||||||
|
def _script_overlay(*, id_: int = 42, script: str = "echo hi") -> SimpleNamespace:
|
||||||
|
return SimpleNamespace(id=id_, type="script", path=str(id_), script=script)
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_builder_invokes_helper(env, monkeypatch) -> None:
|
||||||
|
captured: dict = {}
|
||||||
|
|
||||||
|
def fake_run(cmd, *, on_stdout, on_stderr, should_cancel):
|
||||||
|
captured["cmd"] = list(cmd)
|
||||||
|
captured["script_text"] = open(cmd[-1]).read()
|
||||||
|
captured["script_path_existed"] = os.path.exists(cmd[-1])
|
||||||
|
return CommandResult(returncode=0, stdout="", stderr="")
|
||||||
|
|
||||||
|
monkeypatch.setattr(overlay_builders, "run_command", fake_run)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
overlay_builders.ScriptBuilder, "_enforce_disk_budget", lambda *a, **kw: None
|
||||||
|
)
|
||||||
|
|
||||||
|
overlay = _script_overlay()
|
||||||
|
overlay_builders.ScriptBuilder().build(
|
||||||
|
overlay,
|
||||||
|
on_stdout=lambda _x: None,
|
||||||
|
on_stderr=lambda _x: None,
|
||||||
|
should_cancel=lambda: False,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert captured["cmd"][:4] == [
|
||||||
|
"sudo",
|
||||||
|
"-n",
|
||||||
|
"/usr/local/libexec/left4me/left4me-script-sandbox",
|
||||||
|
"42",
|
||||||
|
]
|
||||||
|
assert captured["script_text"] == "echo hi"
|
||||||
|
assert captured["script_path_existed"] is True
|
||||||
|
# Tmpfile is unlinked after build.
|
||||||
|
assert not os.path.exists(captured["cmd"][-1])
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_builder_disk_cap(env, monkeypatch) -> None:
|
||||||
|
monkeypatch.setattr(
|
||||||
|
overlay_builders,
|
||||||
|
"run_command",
|
||||||
|
lambda *a, **kw: CommandResult(returncode=0, stdout="", stderr=""),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
overlay_builders.subprocess,
|
||||||
|
"check_output",
|
||||||
|
lambda *a, **kw: b"25000000000\t/some/path\n",
|
||||||
|
)
|
||||||
|
|
||||||
|
err: list[str] = []
|
||||||
|
overlay = _script_overlay(script="")
|
||||||
|
|
||||||
|
with pytest.raises(overlay_builders.BuildError):
|
||||||
|
overlay_builders.ScriptBuilder().build(
|
||||||
|
overlay,
|
||||||
|
on_stdout=lambda _x: None,
|
||||||
|
on_stderr=err.append,
|
||||||
|
should_cancel=lambda: False,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert any("20" in line and "GB" in line for line in err), err
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_builder_streams_output(env, monkeypatch) -> None:
|
||||||
|
def fake_run(cmd, *, on_stdout, on_stderr, should_cancel):
|
||||||
|
on_stdout("hello")
|
||||||
|
on_stderr("warn")
|
||||||
|
return CommandResult(returncode=0, stdout="hello", stderr="warn")
|
||||||
|
|
||||||
|
monkeypatch.setattr(overlay_builders, "run_command", fake_run)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
overlay_builders.ScriptBuilder, "_enforce_disk_budget", lambda *a, **kw: None
|
||||||
|
)
|
||||||
|
|
||||||
|
out: list[str] = []
|
||||||
|
err: list[str] = []
|
||||||
|
overlay = _script_overlay(script="")
|
||||||
|
overlay_builders.ScriptBuilder().build(
|
||||||
|
overlay, on_stdout=out.append, on_stderr=err.append, should_cancel=lambda: False
|
||||||
|
)
|
||||||
|
assert out == ["hello"]
|
||||||
|
assert err == ["warn"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_builder_passes_should_cancel_through(env, monkeypatch) -> None:
|
||||||
|
captured: dict = {}
|
||||||
|
|
||||||
|
def fake_run(cmd, *, on_stdout, on_stderr, should_cancel):
|
||||||
|
captured["should_cancel"] = should_cancel
|
||||||
|
raise CommandCancelledError(returncode=1, cmd=cmd, output="", stderr="")
|
||||||
|
|
||||||
|
monkeypatch.setattr(overlay_builders, "run_command", fake_run)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
overlay_builders.ScriptBuilder, "_enforce_disk_budget", lambda *a, **kw: None
|
||||||
|
)
|
||||||
|
|
||||||
|
overlay = _script_overlay(script="")
|
||||||
|
with pytest.raises(CommandCancelledError):
|
||||||
|
overlay_builders.ScriptBuilder().build(
|
||||||
|
overlay,
|
||||||
|
on_stdout=lambda _x: None,
|
||||||
|
on_stderr=lambda _x: None,
|
||||||
|
should_cancel=lambda: True,
|
||||||
|
)
|
||||||
|
assert captured["should_cancel"]() is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_builder_cleans_up_tmpfile_on_failure(env, monkeypatch) -> None:
|
||||||
|
captured: dict = {}
|
||||||
|
|
||||||
|
def fake_run(cmd, *, on_stdout, on_stderr, should_cancel):
|
||||||
|
captured["script_path"] = cmd[-1]
|
||||||
|
raise CommandCancelledError(returncode=1, cmd=cmd, output="", stderr="")
|
||||||
|
|
||||||
|
monkeypatch.setattr(overlay_builders, "run_command", fake_run)
|
||||||
|
|
||||||
|
overlay = _script_overlay(script="")
|
||||||
|
with pytest.raises(CommandCancelledError):
|
||||||
|
overlay_builders.ScriptBuilder().build(
|
||||||
|
overlay,
|
||||||
|
on_stdout=lambda _x: None,
|
||||||
|
on_stderr=lambda _x: None,
|
||||||
|
should_cancel=lambda: False,
|
||||||
|
)
|
||||||
|
assert not os.path.exists(captured["script_path"])
|
||||||
35
l4d2web/tests/test_overlay_creation.py
Normal file
35
l4d2web/tests/test_overlay_creation.py
Normal file
|
|
@ -0,0 +1,35 @@
|
||||||
|
"""Tests for overlay path generation and directory creation."""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from l4d2web.models import Overlay
|
||||||
|
from l4d2web.services import overlay_creation
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_overlay_path_returns_str_id() -> None:
|
||||||
|
assert overlay_creation.generate_overlay_path(42) == "42"
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_overlay_path_validates_through_overlay_ref(monkeypatch) -> None:
|
||||||
|
# Sanity: numeric paths pass validate_overlay_ref. Anything bizarre would raise.
|
||||||
|
assert overlay_creation.generate_overlay_path(1) == "1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_overlay_directory_makes_subtree(monkeypatch, tmp_path: Path) -> None:
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
overlay = Overlay(id=7, name="test", path="7", type="workshop", user_id=None)
|
||||||
|
overlay_creation.create_overlay_directory(overlay)
|
||||||
|
expected = tmp_path / "overlays" / "7"
|
||||||
|
assert expected.is_dir()
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_overlay_directory_raises_if_already_exists(
|
||||||
|
monkeypatch, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
overlay = Overlay(id=7, name="test", path="7", type="workshop", user_id=None)
|
||||||
|
(tmp_path / "overlays" / "7").mkdir(parents=True)
|
||||||
|
# exist_ok=False guards against a stray directory shadowing fresh content.
|
||||||
|
with pytest.raises(FileExistsError):
|
||||||
|
overlay_creation.create_overlay_directory(overlay)
|
||||||
|
|
@ -10,6 +10,7 @@ from l4d2web.services.security import validate_overlay_ref
|
||||||
def admin_client(tmp_path, monkeypatch):
|
def admin_client(tmp_path, monkeypatch):
|
||||||
db_url = f"sqlite:///{tmp_path/'admin_overlay.db'}"
|
db_url = f"sqlite:///{tmp_path/'admin_overlay.db'}"
|
||||||
monkeypatch.setenv("DATABASE_URL", db_url)
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
init_db()
|
init_db()
|
||||||
|
|
||||||
|
|
@ -30,13 +31,17 @@ def admin_client(tmp_path, monkeypatch):
|
||||||
def user_client_with_overlay(tmp_path, monkeypatch):
|
def user_client_with_overlay(tmp_path, monkeypatch):
|
||||||
db_url = f"sqlite:///{tmp_path/'user_overlay.db'}"
|
db_url = f"sqlite:///{tmp_path/'user_overlay.db'}"
|
||||||
monkeypatch.setenv("DATABASE_URL", db_url)
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
init_db()
|
init_db()
|
||||||
|
|
||||||
with session_scope() as session:
|
with session_scope() as session:
|
||||||
user = User(username="alice", password_digest=hash_password("secret"), admin=False)
|
user = User(username="alice", password_digest=hash_password("secret"), admin=False)
|
||||||
session.add(user)
|
session.add(user)
|
||||||
session.add(Overlay(name="standard", path="standard"))
|
# System overlay (workshop, no user_id), pre-existing.
|
||||||
|
session.add(
|
||||||
|
Overlay(name="standard", path="standard", type="workshop", user_id=None)
|
||||||
|
)
|
||||||
session.flush()
|
session.flush()
|
||||||
user_id = user.id
|
user_id = user.id
|
||||||
|
|
||||||
|
|
@ -53,7 +58,8 @@ def test_user_can_view_overlay_catalog(user_client_with_overlay) -> None:
|
||||||
|
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
assert "standard" in text
|
assert "standard" in text
|
||||||
assert "Add overlay" not in text
|
# Non-admin users can create workshop overlays, so the Create button shows.
|
||||||
|
assert "Create overlay" in text
|
||||||
|
|
||||||
|
|
||||||
def test_admin_can_view_overlay_edit_controls(admin_client) -> None:
|
def test_admin_can_view_overlay_edit_controls(admin_client) -> None:
|
||||||
|
|
@ -61,27 +67,29 @@ def test_admin_can_view_overlay_edit_controls(admin_client) -> None:
|
||||||
text = response.get_data(as_text=True)
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
assert "Add overlay" in text
|
assert "Create overlay" in text
|
||||||
assert 'action="/overlays"' in text
|
assert 'action="/overlays"' in text
|
||||||
|
|
||||||
|
|
||||||
def test_admin_can_create_overlay(admin_client) -> None:
|
def test_admin_can_create_workshop_overlay_via_route(admin_client) -> None:
|
||||||
response = admin_client.post(
|
response = admin_client.post(
|
||||||
"/overlays",
|
"/overlays",
|
||||||
data={"name": "standard", "path": "standard"},
|
data={"name": "standard", "type": "workshop"},
|
||||||
headers={"X-CSRF-Token": "test-token"},
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
)
|
)
|
||||||
assert response.status_code == 302
|
assert response.status_code == 302
|
||||||
assert response.headers["Location"] == "/overlays"
|
# Redirect to the new detail page now that paths are auto-generated.
|
||||||
|
assert response.headers["Location"].startswith("/overlays/")
|
||||||
|
|
||||||
|
|
||||||
def test_overlay_ref_must_be_relative(admin_client) -> None:
|
def test_admin_cannot_create_managed_global_overlay_type(admin_client) -> None:
|
||||||
response = admin_client.post(
|
response = admin_client.post(
|
||||||
"/overlays",
|
"/overlays",
|
||||||
data={"name": "bad", "path": "/tmp/bad"},
|
data={"name": "managed", "type": "l4d2center_maps"},
|
||||||
headers={"X-CSRF-Token": "test-token"},
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
)
|
)
|
||||||
assert response.status_code == 400
|
assert response.status_code == 400
|
||||||
|
assert "unknown overlay type" in response.get_data(as_text=True)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("overlay_ref", [" standard", "standard ", "a//b", "a/", "./a", "a/.", "."])
|
@pytest.mark.parametrize("overlay_ref", [" standard", "standard ", "a//b", "a/", "./a", "a/.", "."])
|
||||||
|
|
@ -90,81 +98,241 @@ def test_overlay_ref_rejects_unsafe_components(overlay_ref: str) -> None:
|
||||||
validate_overlay_ref(overlay_ref)
|
validate_overlay_ref(overlay_ref)
|
||||||
|
|
||||||
|
|
||||||
def test_overlay_route_rejects_whitespace_padded_ref(admin_client) -> None:
|
def test_user_can_create_workshop_overlay(user_client_with_overlay) -> None:
|
||||||
response = admin_client.post(
|
|
||||||
"/overlays",
|
|
||||||
data={"name": "bad", "path": " standard"},
|
|
||||||
headers={"X-CSRF-Token": "test-token"},
|
|
||||||
)
|
|
||||||
assert response.status_code == 400
|
|
||||||
|
|
||||||
|
|
||||||
def test_non_admin_cannot_create_overlay(user_client_with_overlay) -> None:
|
|
||||||
response = user_client_with_overlay.post(
|
response = user_client_with_overlay.post(
|
||||||
"/overlays",
|
"/overlays",
|
||||||
data={"name": "bad", "path": "bad"},
|
data={"name": "my-maps", "type": "workshop"},
|
||||||
headers={"X-CSRF-Token": "test-token"},
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
)
|
)
|
||||||
assert response.status_code == 403
|
assert response.status_code == 302
|
||||||
|
with session_scope() as session:
|
||||||
|
overlay = session.query(Overlay).filter_by(name="my-maps").one()
|
||||||
|
assert overlay.type == "workshop"
|
||||||
|
assert overlay.user_id is not None
|
||||||
|
assert overlay.path == str(overlay.id)
|
||||||
|
|
||||||
|
|
||||||
|
def test_workshop_overlay_directory_is_created_on_disk(user_client_with_overlay, tmp_path) -> None:
|
||||||
|
response = user_client_with_overlay.post(
|
||||||
|
"/overlays",
|
||||||
|
data={"name": "my-maps", "type": "workshop"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
with session_scope() as session:
|
||||||
|
overlay = session.query(Overlay).filter_by(name="my-maps").one()
|
||||||
|
overlay_id = overlay.id
|
||||||
|
assert (tmp_path / "overlays" / str(overlay_id)).is_dir()
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_users_can_have_workshop_overlay_with_same_name(tmp_path, monkeypatch) -> None:
|
||||||
|
# Set up a fresh app with two users.
|
||||||
|
db_url = f"sqlite:///{tmp_path/'shared.db'}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
for username in ("alice", "bob"):
|
||||||
|
session.add(User(username=username, password_digest=hash_password("x"), admin=False))
|
||||||
|
session.flush()
|
||||||
|
alice_id, bob_id = (
|
||||||
|
session.query(User).filter_by(username="alice").one().id,
|
||||||
|
session.query(User).filter_by(username="bob").one().id,
|
||||||
|
)
|
||||||
|
|
||||||
|
def client_for(uid):
|
||||||
|
c = app.test_client()
|
||||||
|
with c.session_transaction() as sess:
|
||||||
|
sess["user_id"] = uid
|
||||||
|
sess["csrf_token"] = "test-token"
|
||||||
|
return c
|
||||||
|
|
||||||
|
for uid in (alice_id, bob_id):
|
||||||
|
r = client_for(uid).post(
|
||||||
|
"/overlays",
|
||||||
|
data={"name": "my-maps", "type": "workshop"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 302
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
rows = session.query(Overlay).filter_by(name="my-maps").all()
|
||||||
|
assert {r.user_id for r in rows} == {alice_id, bob_id}
|
||||||
|
|
||||||
|
|
||||||
def test_admin_can_update_and_delete_overlay(admin_client) -> None:
|
def test_admin_can_update_and_delete_overlay(admin_client) -> None:
|
||||||
create = admin_client.post(
|
create = admin_client.post(
|
||||||
"/overlays",
|
"/overlays",
|
||||||
data={"name": "standard", "path": "standard"},
|
data={"name": "standard", "type": "workshop"},
|
||||||
headers={"X-CSRF-Token": "test-token"},
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
)
|
)
|
||||||
assert create.status_code == 302
|
assert create.status_code == 302
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
overlay_id = session.query(Overlay).filter_by(name="standard").one().id
|
||||||
|
|
||||||
update = admin_client.post(
|
update = admin_client.post(
|
||||||
"/overlays/1",
|
f"/overlays/{overlay_id}",
|
||||||
data={"name": "edited", "path": "edited"},
|
data={"name": "edited"},
|
||||||
headers={"X-CSRF-Token": "test-token"},
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
)
|
)
|
||||||
assert update.status_code == 302
|
assert update.status_code == 302
|
||||||
|
|
||||||
delete = admin_client.post(
|
delete = admin_client.post(
|
||||||
"/overlays/1/delete",
|
f"/overlays/{overlay_id}/delete",
|
||||||
headers={"X-CSRF-Token": "test-token"},
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
)
|
)
|
||||||
assert delete.status_code == 302
|
assert delete.status_code == 302
|
||||||
|
|
||||||
|
|
||||||
def test_update_overlay_rejects_duplicate_name(admin_client) -> None:
|
def test_update_overlay_rejects_duplicate_name(admin_client) -> None:
|
||||||
for name in ["standard", "competitive"]:
|
ids: list[int] = []
|
||||||
|
for name in ("standard", "competitive"):
|
||||||
response = admin_client.post(
|
response = admin_client.post(
|
||||||
"/overlays",
|
"/overlays",
|
||||||
data={"name": name, "path": name},
|
data={"name": name, "type": "workshop"},
|
||||||
headers={"X-CSRF-Token": "test-token"},
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
)
|
)
|
||||||
assert response.status_code == 302
|
assert response.status_code == 302
|
||||||
|
with session_scope() as session:
|
||||||
|
ids = [
|
||||||
|
session.query(Overlay).filter_by(name="standard").one().id,
|
||||||
|
session.query(Overlay).filter_by(name="competitive").one().id,
|
||||||
|
]
|
||||||
|
|
||||||
response = admin_client.post(
|
response = admin_client.post(
|
||||||
"/overlays/2",
|
f"/overlays/{ids[1]}",
|
||||||
data={"name": "standard", "path": "competitive"},
|
data={"name": "standard"},
|
||||||
headers={"X-CSRF-Token": "test-token"},
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
)
|
)
|
||||||
|
|
||||||
assert response.status_code == 409
|
assert response.status_code == 409
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_detail_page_lists_using_blueprints(admin_client) -> None:
|
||||||
|
create = admin_client.post(
|
||||||
|
"/overlays",
|
||||||
|
data={"name": "shared", "type": "workshop"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert create.status_code == 302
|
||||||
|
with session_scope() as session:
|
||||||
|
overlay_id = session.query(Overlay).filter_by(name="shared").one().id
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
admin = session.query(User).filter_by(username="admin").one()
|
||||||
|
bp_one = Blueprint(user_id=admin.id, name="alpha-bp", arguments="[]", config="[]")
|
||||||
|
bp_two = Blueprint(user_id=admin.id, name="beta-bp", arguments="[]", config="[]")
|
||||||
|
session.add_all([bp_one, bp_two])
|
||||||
|
session.flush()
|
||||||
|
session.add(BlueprintOverlay(blueprint_id=bp_one.id, overlay_id=overlay_id, position=0))
|
||||||
|
session.add(BlueprintOverlay(blueprint_id=bp_two.id, overlay_id=overlay_id, position=0))
|
||||||
|
|
||||||
|
response = admin_client.get(f"/overlays/{overlay_id}")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "alpha-bp" in text
|
||||||
|
assert "beta-bp" in text
|
||||||
|
assert "Used by" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_admin_overlay_detail_only_lists_own_using_blueprints(user_client_with_overlay) -> None:
|
||||||
|
with session_scope() as session:
|
||||||
|
alice = session.query(User).filter_by(username="alice").one()
|
||||||
|
other = User(username="mallory", password_digest=hash_password("secret"), admin=False)
|
||||||
|
session.add(other)
|
||||||
|
session.flush()
|
||||||
|
|
||||||
|
# Use the seeded system "standard" overlay (id=1).
|
||||||
|
overlay_id = session.query(Overlay).filter_by(name="standard").one().id
|
||||||
|
|
||||||
|
own_bp = Blueprint(user_id=alice.id, name="own-bp", arguments="[]", config="[]")
|
||||||
|
other_bp = Blueprint(user_id=other.id, name="other-private-bp", arguments="[]", config="[]")
|
||||||
|
session.add_all([own_bp, other_bp])
|
||||||
|
session.flush()
|
||||||
|
session.add(BlueprintOverlay(blueprint_id=own_bp.id, overlay_id=overlay_id, position=0))
|
||||||
|
session.add(BlueprintOverlay(blueprint_id=other_bp.id, overlay_id=overlay_id, position=0))
|
||||||
|
|
||||||
|
response = user_client_with_overlay.get(f"/overlays/{overlay_id}")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "own-bp" in text
|
||||||
|
assert "other-private-bp" not in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_blueprint_edit_lists_system_and_owned_overlays_only(user_client_with_overlay) -> None:
|
||||||
|
with session_scope() as session:
|
||||||
|
alice = session.query(User).filter_by(username="alice").one()
|
||||||
|
other = User(username="mallory", password_digest=hash_password("secret"), admin=False)
|
||||||
|
session.add(other)
|
||||||
|
session.flush()
|
||||||
|
system_overlay_id = session.query(Overlay).filter_by(name="standard").one().id
|
||||||
|
foreign_overlay = Overlay(
|
||||||
|
name="other-private-workshop",
|
||||||
|
path="other-private-workshop",
|
||||||
|
type="workshop",
|
||||||
|
user_id=other.id,
|
||||||
|
)
|
||||||
|
blueprint = Blueprint(user_id=alice.id, name="alice-bp", arguments="[]", config="[]")
|
||||||
|
session.add_all([foreign_overlay, blueprint])
|
||||||
|
session.flush()
|
||||||
|
blueprint_id = blueprint.id
|
||||||
|
|
||||||
|
response = user_client_with_overlay.get(f"/blueprints/{blueprint_id}")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "standard" in text
|
||||||
|
assert f'value="{system_overlay_id}"' in text
|
||||||
|
assert "other-private-workshop" not in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_detail_page_404_when_missing(admin_client) -> None:
|
||||||
|
response = admin_client.get("/overlays/999")
|
||||||
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_update_redirects_to_detail(admin_client) -> None:
|
||||||
|
create = admin_client.post(
|
||||||
|
"/overlays",
|
||||||
|
data={"name": "standard", "type": "workshop"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert create.status_code == 302
|
||||||
|
with session_scope() as session:
|
||||||
|
overlay_id = session.query(Overlay).filter_by(name="standard").one().id
|
||||||
|
|
||||||
|
response = admin_client.post(
|
||||||
|
f"/overlays/{overlay_id}",
|
||||||
|
data={"name": "renamed"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
assert response.headers["Location"] == f"/overlays/{overlay_id}"
|
||||||
|
|
||||||
|
|
||||||
def test_delete_overlay_rejects_in_use_overlay(admin_client) -> None:
|
def test_delete_overlay_rejects_in_use_overlay(admin_client) -> None:
|
||||||
create = admin_client.post(
|
create = admin_client.post(
|
||||||
"/overlays",
|
"/overlays",
|
||||||
data={"name": "standard", "path": "standard"},
|
data={"name": "standard", "type": "workshop"},
|
||||||
headers={"X-CSRF-Token": "test-token"},
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
)
|
)
|
||||||
assert create.status_code == 302
|
assert create.status_code == 302
|
||||||
|
with session_scope() as session:
|
||||||
|
overlay_id = session.query(Overlay).filter_by(name="standard").one().id
|
||||||
|
|
||||||
with session_scope() as session:
|
with session_scope() as session:
|
||||||
user = session.query(User).filter_by(username="admin").one()
|
user = session.query(User).filter_by(username="admin").one()
|
||||||
blueprint = Blueprint(user_id=user.id, name="bp", arguments="[]", config="[]")
|
blueprint = Blueprint(user_id=user.id, name="bp", arguments="[]", config="[]")
|
||||||
session.add(blueprint)
|
session.add(blueprint)
|
||||||
session.flush()
|
session.flush()
|
||||||
session.add(BlueprintOverlay(blueprint_id=blueprint.id, overlay_id=1, position=0))
|
session.add(BlueprintOverlay(blueprint_id=blueprint.id, overlay_id=overlay_id, position=0))
|
||||||
|
|
||||||
response = admin_client.post(
|
response = admin_client.post(
|
||||||
"/overlays/1/delete",
|
f"/overlays/{overlay_id}/delete",
|
||||||
headers={"X-CSRF-Token": "test-token"},
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -312,7 +312,10 @@ def test_admin_can_use_admin_pages(tmp_path, monkeypatch) -> None:
|
||||||
|
|
||||||
admin_page = client.get("/admin")
|
admin_page = client.get("/admin")
|
||||||
assert admin_page.status_code == 200
|
assert admin_page.status_code == 200
|
||||||
assert 'action="/admin/install"' in admin_page.get_data(as_text=True)
|
admin_html = admin_page.get_data(as_text=True)
|
||||||
|
assert 'action="/admin/install"' in admin_html
|
||||||
|
assert "/admin/global-overlays/refresh" not in admin_html
|
||||||
|
assert "Global map overlays" not in admin_html
|
||||||
assert client.get("/admin/users").status_code == 200
|
assert client.get("/admin/users").status_code == 200
|
||||||
jobs_response = client.get("/admin/jobs")
|
jobs_response = client.get("/admin/jobs")
|
||||||
assert jobs_response.status_code == 200
|
assert jobs_response.status_code == 200
|
||||||
|
|
@ -457,3 +460,177 @@ def test_blueprint_detail_has_ordered_overlay_form(auth_client_with_server) -> N
|
||||||
assert 'name="config"' in text
|
assert 'name="config"' in text
|
||||||
assert 'name="overlay_ids"' in text
|
assert 'name="overlay_ids"' in text
|
||||||
assert 'name="overlay_position_1"' in text
|
assert 'name="overlay_position_1"' in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_admin_jobs_page_renders_system_job(tmp_path, monkeypatch) -> None:
|
||||||
|
db_url = f"sqlite:///{tmp_path/'admin-system-job.db'}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
admin = User(username="admin", password_digest=hash_password("secret"), admin=True)
|
||||||
|
session.add(admin)
|
||||||
|
session.flush()
|
||||||
|
admin_id = admin.id
|
||||||
|
|
||||||
|
admin_client = app.test_client()
|
||||||
|
with admin_client.session_transaction() as sess:
|
||||||
|
sess["user_id"] = admin_id
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
db.add(Job(user_id=None, server_id=None, operation="refresh_workshop_items", state="queued"))
|
||||||
|
|
||||||
|
response = admin_client.get("/admin/jobs")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "refresh_workshop_items" in text
|
||||||
|
assert "system" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_admin_cannot_view_system_job(tmp_path, monkeypatch) -> None:
|
||||||
|
db_url = f"sqlite:///{tmp_path/'non-admin-system-job.db'}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
user = User(username="alice", password_digest=hash_password("secret"), admin=False)
|
||||||
|
session.add(user)
|
||||||
|
session.flush()
|
||||||
|
user_id = user.id
|
||||||
|
|
||||||
|
user_client = app.test_client()
|
||||||
|
with user_client.session_transaction() as sess:
|
||||||
|
sess["user_id"] = user_id
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
job = Job(user_id=None, server_id=None, operation="refresh_workshop_items", state="queued")
|
||||||
|
db.add(job)
|
||||||
|
db.flush()
|
||||||
|
job_id = job.id
|
||||||
|
|
||||||
|
response = user_client.get(f"/jobs/{job_id}")
|
||||||
|
assert response.status_code == 403
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_create_modal_offers_script_type(auth_client_with_server) -> None:
|
||||||
|
response = auth_client_with_server.get("/overlays")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert 'value="workshop"' in text
|
||||||
|
assert 'value="script"' in text
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_overlay(name: str, type_: str, user_id: int) -> int:
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = Overlay(name=name, path="", type=type_, user_id=user_id)
|
||||||
|
s.add(overlay)
|
||||||
|
s.flush()
|
||||||
|
overlay.path = str(overlay.id)
|
||||||
|
s.flush()
|
||||||
|
return overlay.id
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_detail_script_section(auth_client_with_server) -> None:
|
||||||
|
with session_scope() as s:
|
||||||
|
user_id = s.query(User).filter_by(username="alice").one().id
|
||||||
|
overlay_id = _seed_overlay("build", "script", user_id)
|
||||||
|
|
||||||
|
response = auth_client_with_server.get(f"/overlays/{overlay_id}")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert '<textarea name="script"' in text
|
||||||
|
assert "Rebuild" in text
|
||||||
|
assert "Wipe" in text
|
||||||
|
assert "Last build" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_detail_workshop_section_unchanged(auth_client_with_server) -> None:
|
||||||
|
with session_scope() as s:
|
||||||
|
user_id = s.query(User).filter_by(username="alice").one().id
|
||||||
|
overlay_id = _seed_overlay("ws", "workshop", user_id)
|
||||||
|
|
||||||
|
response = auth_client_with_server.get(f"/overlays/{overlay_id}")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "Workshop items" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_detail_links_to_overlay_jobs_page(auth_client_with_server) -> None:
|
||||||
|
with session_scope() as s:
|
||||||
|
user_id = s.query(User).filter_by(username="alice").one().id
|
||||||
|
overlay_id = _seed_overlay("scripted", "script", user_id)
|
||||||
|
|
||||||
|
response = auth_client_with_server.get(f"/overlays/{overlay_id}")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert f'href="/overlays/{overlay_id}/jobs"' in text
|
||||||
|
assert "Save and build" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_jobs_page_lists_overlay_builds(auth_client_with_server) -> None:
|
||||||
|
with session_scope() as s:
|
||||||
|
user_id = s.query(User).filter_by(username="alice").one().id
|
||||||
|
overlay_id = _seed_overlay("scripted", "script", user_id)
|
||||||
|
other_overlay_id = _seed_overlay("other", "script", user_id)
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
session.add(Job(user_id=user_id, overlay_id=overlay_id, operation="build_overlay", state="succeeded"))
|
||||||
|
session.add(Job(user_id=user_id, overlay_id=overlay_id, operation="build_overlay", state="queued"))
|
||||||
|
# Job for a different overlay must not appear.
|
||||||
|
session.add(Job(user_id=user_id, overlay_id=other_overlay_id, operation="build_overlay", state="succeeded"))
|
||||||
|
|
||||||
|
response = auth_client_with_server.get(f"/overlays/{overlay_id}/jobs")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "Build jobs for scripted" in text
|
||||||
|
assert text.count("build_overlay") == 2
|
||||||
|
assert f'href="/overlays/{overlay_id}"' in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_jobs_page_404_for_unknown_overlay(auth_client_with_server) -> None:
|
||||||
|
response = auth_client_with_server.get("/overlays/9999/jobs")
|
||||||
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_jobs_page_403_for_other_users_private_overlay(tmp_path, monkeypatch) -> None:
|
||||||
|
db_url = f"sqlite:///{tmp_path/'overlay-jobs-403.db'}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
owner = User(username="owner", password_digest=hash_password("x"), admin=False)
|
||||||
|
other = User(username="other", password_digest=hash_password("x"), admin=False)
|
||||||
|
session.add_all([owner, other])
|
||||||
|
session.flush()
|
||||||
|
owner_id = owner.id
|
||||||
|
other_id = other.id
|
||||||
|
|
||||||
|
overlay_id = _seed_overlay("private", "script", owner_id)
|
||||||
|
|
||||||
|
client = app.test_client()
|
||||||
|
with client.session_transaction() as sess:
|
||||||
|
sess["user_id"] = other_id
|
||||||
|
|
||||||
|
response = client.get(f"/overlays/{overlay_id}/jobs")
|
||||||
|
assert response.status_code == 403
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_detail_no_global_source_block(auth_client_with_server) -> None:
|
||||||
|
with session_scope() as s:
|
||||||
|
user_id = s.query(User).filter_by(username="alice").one().id
|
||||||
|
overlay_id = _seed_overlay("ws", "workshop", user_id)
|
||||||
|
|
||||||
|
response = auth_client_with_server.get(f"/overlays/{overlay_id}")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert "Global source" not in text
|
||||||
|
assert "source_url" not in text
|
||||||
|
|
|
||||||
276
l4d2web/tests/test_script_overlay_routes.py
Normal file
276
l4d2web/tests/test_script_overlay_routes.py
Normal file
|
|
@ -0,0 +1,276 @@
|
||||||
|
"""Routes for type='script' overlays: create, /script (update body),
|
||||||
|
/wipe, /build. Permissions mirror workshop overlays (owner or admin)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from l4d2web.app import create_app
|
||||||
|
from l4d2web.auth import hash_password
|
||||||
|
from l4d2web.db import init_db, session_scope
|
||||||
|
from l4d2web.models import Job, Overlay, User
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def app(tmp_path, monkeypatch):
|
||||||
|
db_url = f"sqlite:///{tmp_path/'script-routes.db'}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
flask_app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
|
init_db()
|
||||||
|
return flask_app
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def alice_id(app) -> int:
|
||||||
|
with session_scope() as s:
|
||||||
|
user = User(username="alice", password_digest=hash_password("x"), admin=False)
|
||||||
|
s.add(user)
|
||||||
|
s.flush()
|
||||||
|
return user.id
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def bob_id(app) -> int:
|
||||||
|
with session_scope() as s:
|
||||||
|
user = User(username="bob", password_digest=hash_password("x"), admin=False)
|
||||||
|
s.add(user)
|
||||||
|
s.flush()
|
||||||
|
return user.id
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def admin_id(app) -> int:
|
||||||
|
with session_scope() as s:
|
||||||
|
user = User(username="admin", password_digest=hash_password("x"), admin=True)
|
||||||
|
s.add(user)
|
||||||
|
s.flush()
|
||||||
|
return user.id
|
||||||
|
|
||||||
|
|
||||||
|
def _client_for(app, user_id: int):
|
||||||
|
client = app.test_client()
|
||||||
|
with client.session_transaction() as sess:
|
||||||
|
sess["user_id"] = user_id
|
||||||
|
sess["csrf_token"] = "test-token"
|
||||||
|
return client
|
||||||
|
|
||||||
|
|
||||||
|
def _create_script_overlay(app, user_id: int, *, name: str = "x") -> int:
|
||||||
|
client = _client_for(app, user_id)
|
||||||
|
response = client.post(
|
||||||
|
"/overlays",
|
||||||
|
data={"name": name, "type": "script"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302, response.get_data(as_text=True)
|
||||||
|
with session_scope() as s:
|
||||||
|
return s.scalar(select(Overlay.id).where(Overlay.name == name))
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_script_overlay(app, alice_id) -> None:
|
||||||
|
client = _client_for(app, alice_id)
|
||||||
|
response = client.post(
|
||||||
|
"/overlays",
|
||||||
|
data={"name": "first", "type": "script"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(name="first").one()
|
||||||
|
assert overlay.type == "script"
|
||||||
|
assert overlay.script == ""
|
||||||
|
assert overlay.last_build_status == ""
|
||||||
|
assert overlay.user_id == alice_id
|
||||||
|
assert overlay.path == str(overlay.id)
|
||||||
|
|
||||||
|
|
||||||
|
def test_admin_creates_system_wide_script_overlay(app, admin_id) -> None:
|
||||||
|
client = _client_for(app, admin_id)
|
||||||
|
response = client.post(
|
||||||
|
"/overlays",
|
||||||
|
data={"name": "system", "type": "script", "system_wide": "1"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(name="system").one()
|
||||||
|
assert overlay.user_id is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_admin_system_wide_flag_is_ignored(app, alice_id) -> None:
|
||||||
|
client = _client_for(app, alice_id)
|
||||||
|
response = client.post(
|
||||||
|
"/overlays",
|
||||||
|
data={"name": "evil", "type": "script", "system_wide": "1"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(name="evil").one()
|
||||||
|
assert overlay.user_id == alice_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_update_script_body_enqueues_build(app, alice_id) -> None:
|
||||||
|
overlay_id = _create_script_overlay(app, alice_id)
|
||||||
|
client = _client_for(app, alice_id)
|
||||||
|
|
||||||
|
r1 = client.post(
|
||||||
|
f"/overlays/{overlay_id}/script",
|
||||||
|
data={"script": "echo new"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert r1.status_code == 302
|
||||||
|
assert r1.headers["Location"].startswith("/jobs/")
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
assert overlay.script == "echo new"
|
||||||
|
jobs = s.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
|
||||||
|
assert len(jobs) == 1
|
||||||
|
assert r1.headers["Location"] == f"/jobs/{jobs[0].id}"
|
||||||
|
|
||||||
|
# Coalesce against pending.
|
||||||
|
r2 = client.post(
|
||||||
|
f"/overlays/{overlay_id}/script",
|
||||||
|
data={"script": "echo newer"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert r2.status_code == 302
|
||||||
|
assert r2.headers["Location"] == r1.headers["Location"]
|
||||||
|
with session_scope() as s:
|
||||||
|
jobs = s.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
|
||||||
|
assert len(jobs) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_update_script_normalizes_crlf_to_lf(app, alice_id) -> None:
|
||||||
|
"""HTML <textarea> submits CRLF line endings; bash chokes on trailing \\r
|
||||||
|
in every command. Storage must be LF-only so the sandbox tmpfile is
|
||||||
|
well-formed."""
|
||||||
|
overlay_id = _create_script_overlay(app, alice_id)
|
||||||
|
client = _client_for(app, alice_id)
|
||||||
|
client.post(
|
||||||
|
f"/overlays/{overlay_id}/script",
|
||||||
|
data={"script": "ls /\r\necho hello\r\n"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
assert overlay.script == "ls /\necho hello\n"
|
||||||
|
assert "\r" not in overlay.script
|
||||||
|
|
||||||
|
|
||||||
|
def test_manual_rebuild(app, alice_id) -> None:
|
||||||
|
overlay_id = _create_script_overlay(app, alice_id)
|
||||||
|
client = _client_for(app, alice_id)
|
||||||
|
|
||||||
|
r1 = client.post(
|
||||||
|
f"/overlays/{overlay_id}/build",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert r1.status_code == 302
|
||||||
|
with session_scope() as s:
|
||||||
|
jobs = s.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
|
||||||
|
assert len(jobs) == 1
|
||||||
|
|
||||||
|
# Coalesce.
|
||||||
|
r2 = client.post(
|
||||||
|
f"/overlays/{overlay_id}/build",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert r2.status_code == 302
|
||||||
|
with session_scope() as s:
|
||||||
|
jobs = s.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
|
||||||
|
assert len(jobs) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_wipe_runs_find_delete(app, alice_id, monkeypatch) -> None:
|
||||||
|
overlay_id = _create_script_overlay(app, alice_id)
|
||||||
|
|
||||||
|
# Pre-set a "successful" status so we can verify wipe resets it.
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
overlay.last_build_status = "ok"
|
||||||
|
|
||||||
|
captured: dict = {}
|
||||||
|
|
||||||
|
def fake_run(overlay_id_arg, script_text, *, on_stdout, on_stderr, should_cancel):
|
||||||
|
captured["overlay_id"] = overlay_id_arg
|
||||||
|
captured["script"] = script_text
|
||||||
|
|
||||||
|
from l4d2web.services import overlay_builders
|
||||||
|
monkeypatch.setattr(overlay_builders, "run_sandboxed_script", fake_run)
|
||||||
|
|
||||||
|
client = _client_for(app, alice_id)
|
||||||
|
response = client.post(
|
||||||
|
f"/overlays/{overlay_id}/wipe",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
assert captured["overlay_id"] == overlay_id
|
||||||
|
assert captured["script"] == "find /overlay -mindepth 1 -delete"
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
assert overlay.last_build_status == ""
|
||||||
|
# Wipe does NOT auto-enqueue a rebuild.
|
||||||
|
jobs = s.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
|
||||||
|
assert len(jobs) == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_wipe_refuses_during_running_build(app, alice_id, monkeypatch) -> None:
|
||||||
|
overlay_id = _create_script_overlay(app, alice_id)
|
||||||
|
|
||||||
|
# Mark a build as running for this overlay.
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(
|
||||||
|
Job(
|
||||||
|
user_id=alice_id,
|
||||||
|
server_id=None,
|
||||||
|
overlay_id=overlay_id,
|
||||||
|
operation="build_overlay",
|
||||||
|
state="running",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
invocations: list = []
|
||||||
|
|
||||||
|
def fake_run(*args, **kwargs):
|
||||||
|
invocations.append((args, kwargs))
|
||||||
|
|
||||||
|
from l4d2web.services import overlay_builders
|
||||||
|
monkeypatch.setattr(overlay_builders, "run_sandboxed_script", fake_run)
|
||||||
|
|
||||||
|
client = _client_for(app, alice_id)
|
||||||
|
response = client.post(
|
||||||
|
f"/overlays/{overlay_id}/wipe",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 409
|
||||||
|
assert invocations == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_permissions_non_owner_denied(app, alice_id, bob_id) -> None:
|
||||||
|
overlay_id = _create_script_overlay(app, alice_id, name="alice-private")
|
||||||
|
bob = _client_for(app, bob_id)
|
||||||
|
|
||||||
|
r1 = bob.post(
|
||||||
|
f"/overlays/{overlay_id}/script",
|
||||||
|
data={"script": "boom"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert r1.status_code == 403
|
||||||
|
|
||||||
|
|
||||||
|
def test_permissions_admin_can_edit_any(app, alice_id, admin_id) -> None:
|
||||||
|
overlay_id = _create_script_overlay(app, alice_id, name="alice-private")
|
||||||
|
admin = _client_for(app, admin_id)
|
||||||
|
|
||||||
|
r1 = admin.post(
|
||||||
|
f"/overlays/{overlay_id}/script",
|
||||||
|
data={"script": "echo admin"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert r1.status_code == 302
|
||||||
|
with session_scope() as s:
|
||||||
|
overlay = s.query(Overlay).filter_by(id=overlay_id).one()
|
||||||
|
assert overlay.script == "echo admin"
|
||||||
79
l4d2web/tests/test_seed_script_overlays.py
Normal file
79
l4d2web/tests/test_seed_script_overlays.py
Normal file
|
|
@ -0,0 +1,79 @@
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from l4d2web.app import create_app
|
||||||
|
from l4d2web.db import init_db, session_scope
|
||||||
|
from l4d2web.models import Overlay
|
||||||
|
|
||||||
|
|
||||||
|
def _make_app(tmp_path: Path, monkeypatch):
|
||||||
|
db_url = f"sqlite:///{tmp_path/'seed.db'}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
|
init_db()
|
||||||
|
return app
|
||||||
|
|
||||||
|
|
||||||
|
def test_seed_creates_system_wide_script_overlays(tmp_path: Path, monkeypatch) -> None:
|
||||||
|
app = _make_app(tmp_path, monkeypatch)
|
||||||
|
examples = tmp_path / "examples"
|
||||||
|
examples.mkdir()
|
||||||
|
(examples / "alpha.sh").write_text("#!/bin/bash\necho alpha\n")
|
||||||
|
(examples / "beta.sh").write_text("#!/bin/bash\necho beta\n")
|
||||||
|
|
||||||
|
result = app.test_cli_runner().invoke(args=["seed-script-overlays", str(examples)])
|
||||||
|
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
with session_scope() as db:
|
||||||
|
rows = db.query(Overlay).order_by(Overlay.name).all()
|
||||||
|
assert [(o.name, o.type, o.user_id, o.script.strip()) for o in rows] == [
|
||||||
|
("alpha", "script", None, "#!/bin/bash\necho alpha"),
|
||||||
|
("beta", "script", None, "#!/bin/bash\necho beta"),
|
||||||
|
]
|
||||||
|
for o in rows:
|
||||||
|
assert (tmp_path / "overlays" / o.path).is_dir()
|
||||||
|
|
||||||
|
|
||||||
|
def test_seed_refreshes_existing_script_overlay(tmp_path: Path, monkeypatch) -> None:
|
||||||
|
app = _make_app(tmp_path, monkeypatch)
|
||||||
|
examples = tmp_path / "examples"
|
||||||
|
examples.mkdir()
|
||||||
|
(examples / "alpha.sh").write_text("v1\n")
|
||||||
|
|
||||||
|
runner = app.test_cli_runner()
|
||||||
|
runner.invoke(args=["seed-script-overlays", str(examples)])
|
||||||
|
(examples / "alpha.sh").write_text("v2\n")
|
||||||
|
result = runner.invoke(args=["seed-script-overlays", str(examples)])
|
||||||
|
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
with session_scope() as db:
|
||||||
|
rows = db.query(Overlay).filter(Overlay.name == "alpha").all()
|
||||||
|
assert len(rows) == 1
|
||||||
|
assert rows[0].script.strip() == "v2"
|
||||||
|
|
||||||
|
|
||||||
|
def test_seed_errors_on_type_collision(tmp_path: Path, monkeypatch) -> None:
|
||||||
|
app = _make_app(tmp_path, monkeypatch)
|
||||||
|
examples = tmp_path / "examples"
|
||||||
|
examples.mkdir()
|
||||||
|
(examples / "shared.sh").write_text("#!/bin/bash\n")
|
||||||
|
|
||||||
|
with session_scope() as db:
|
||||||
|
db.add(Overlay(name="shared", path="shared", type="workshop", user_id=None))
|
||||||
|
|
||||||
|
result = app.test_cli_runner().invoke(args=["seed-script-overlays", str(examples)])
|
||||||
|
|
||||||
|
assert result.exit_code != 0
|
||||||
|
assert "not script" in result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_seed_no_files_is_noop(tmp_path: Path, monkeypatch) -> None:
|
||||||
|
app = _make_app(tmp_path, monkeypatch)
|
||||||
|
examples = tmp_path / "examples"
|
||||||
|
examples.mkdir()
|
||||||
|
|
||||||
|
result = app.test_cli_runner().invoke(args=["seed-script-overlays", str(examples)])
|
||||||
|
|
||||||
|
assert result.exit_code == 0
|
||||||
|
with session_scope() as db:
|
||||||
|
assert db.query(Overlay).count() == 0
|
||||||
|
|
@ -37,6 +37,32 @@ def user_client_with_blueprints(tmp_path, monkeypatch):
|
||||||
return client, payload
|
return client, payload
|
||||||
|
|
||||||
|
|
||||||
|
def test_servers_page_without_blueprints_shows_create_blueprint_cta(tmp_path, monkeypatch) -> None:
|
||||||
|
db_url = f"sqlite:///{tmp_path/'no_blueprints.db'}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
user = User(username="solo", password_digest=hash_password("secret"), admin=False)
|
||||||
|
session.add(user)
|
||||||
|
session.flush()
|
||||||
|
user_id = user.id
|
||||||
|
|
||||||
|
client = app.test_client()
|
||||||
|
with client.session_transaction() as sess:
|
||||||
|
sess["user_id"] = user_id
|
||||||
|
sess["csrf_token"] = "test-token"
|
||||||
|
|
||||||
|
response = client.get("/servers")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert 'href="/blueprints"' in text
|
||||||
|
assert "Create a blueprint first" in text
|
||||||
|
assert "disabled" not in text
|
||||||
|
|
||||||
|
|
||||||
def test_create_server_from_blueprint(user_client_with_blueprints) -> None:
|
def test_create_server_from_blueprint(user_client_with_blueprints) -> None:
|
||||||
client, data = user_client_with_blueprints
|
client, data = user_client_with_blueprints
|
||||||
payload = {"name": "alpha", "port": 27015, "blueprint_id": data["blueprint_id"]}
|
payload = {"name": "alpha", "port": 27015, "blueprint_id": data["blueprint_id"]}
|
||||||
|
|
@ -114,8 +140,8 @@ def test_create_server_duplicate_port(user_client_with_blueprints) -> None:
|
||||||
assert servers[0].name == "server-1"
|
assert servers[0].name == "server-1"
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("bad_name", ["..", "../escape", "foo/bar", " foo", "Foo"])
|
@pytest.mark.parametrize("bad_name", ["", " ", "x" * 129])
|
||||||
def test_create_server_rejects_unsafe_names(user_client_with_blueprints, bad_name: str) -> None:
|
def test_create_server_rejects_invalid_display_names(user_client_with_blueprints, bad_name: str) -> None:
|
||||||
client, data = user_client_with_blueprints
|
client, data = user_client_with_blueprints
|
||||||
response = client.post(
|
response = client.post(
|
||||||
"/servers",
|
"/servers",
|
||||||
|
|
@ -132,6 +158,250 @@ def test_create_server_rejects_unsafe_names(user_client_with_blueprints, bad_nam
|
||||||
assert session.scalars(select(Server)).all() == []
|
assert session.scalars(select(Server)).all() == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("name", ["My Practice", "räumlich", "alpha/beta", "..", "Foo"])
|
||||||
|
def test_create_server_accepts_free_form_display_names(user_client_with_blueprints, name: str) -> None:
|
||||||
|
client, data = user_client_with_blueprints
|
||||||
|
response = client.post(
|
||||||
|
"/servers",
|
||||||
|
data={"name": name, "blueprint_id": str(data["blueprint_id"])},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from l4d2web.models import Server
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
server = session.scalars(select(Server)).one()
|
||||||
|
assert server.name == name
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_server_strips_surrounding_whitespace_in_name(user_client_with_blueprints) -> None:
|
||||||
|
client, data = user_client_with_blueprints
|
||||||
|
response = client.post(
|
||||||
|
"/servers",
|
||||||
|
data={"name": " spaced ", "blueprint_id": str(data["blueprint_id"])},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from l4d2web.models import Server
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
server = session.scalars(select(Server)).one()
|
||||||
|
assert server.name == "spaced"
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_server_rejects_duplicate_name_for_same_user(user_client_with_blueprints) -> None:
|
||||||
|
client, data = user_client_with_blueprints
|
||||||
|
first = client.post(
|
||||||
|
"/servers",
|
||||||
|
data={"name": "practice", "port": "27015", "blueprint_id": str(data["blueprint_id"])},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert first.status_code == 302
|
||||||
|
|
||||||
|
second = client.post(
|
||||||
|
"/servers",
|
||||||
|
data={"name": "practice", "port": "27016", "blueprint_id": str(data["blueprint_id"])},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert second.status_code == 409
|
||||||
|
assert b"name already in use" in second.data
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_server_allows_same_name_for_different_users(tmp_path, monkeypatch) -> None:
|
||||||
|
db_url = f"sqlite:///{tmp_path/'two_users.db'}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
alice = User(username="alice", password_digest=hash_password("secret"), admin=False)
|
||||||
|
bob = User(username="bob", password_digest=hash_password("secret"), admin=False)
|
||||||
|
session.add_all([alice, bob])
|
||||||
|
session.flush()
|
||||||
|
alice_bp = Blueprint(user_id=alice.id, name="bp", arguments="[]", config="[]")
|
||||||
|
bob_bp = Blueprint(user_id=bob.id, name="bp", arguments="[]", config="[]")
|
||||||
|
session.add_all([alice_bp, bob_bp])
|
||||||
|
session.flush()
|
||||||
|
alice_id, bob_id = alice.id, bob.id
|
||||||
|
alice_bp_id, bob_bp_id = alice_bp.id, bob_bp.id
|
||||||
|
|
||||||
|
client = app.test_client()
|
||||||
|
|
||||||
|
with client.session_transaction() as sess:
|
||||||
|
sess["user_id"] = alice_id
|
||||||
|
sess["csrf_token"] = "test-token"
|
||||||
|
alice_resp = client.post(
|
||||||
|
"/servers",
|
||||||
|
data={"name": "practice", "port": "27015", "blueprint_id": str(alice_bp_id)},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert alice_resp.status_code == 302
|
||||||
|
|
||||||
|
with client.session_transaction() as sess:
|
||||||
|
sess["user_id"] = bob_id
|
||||||
|
sess["csrf_token"] = "test-token"
|
||||||
|
bob_resp = client.post(
|
||||||
|
"/servers",
|
||||||
|
data={"name": "practice", "port": "27016", "blueprint_id": str(bob_bp_id)},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert bob_resp.status_code == 302
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_server_with_empty_port_auto_assigns(user_client_with_blueprints) -> None:
|
||||||
|
client, data = user_client_with_blueprints
|
||||||
|
response = client.post(
|
||||||
|
"/servers",
|
||||||
|
data={"name": "alpha", "port": "", "blueprint_id": str(data["blueprint_id"])},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from l4d2web.models import Server
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
server = session.scalars(select(Server)).one()
|
||||||
|
assert server.port == 27015
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_server_auto_assign_skips_taken_ports(user_client_with_blueprints) -> None:
|
||||||
|
client, data = user_client_with_blueprints
|
||||||
|
first = client.post(
|
||||||
|
"/servers",
|
||||||
|
data={"name": "alpha", "port": "27015", "blueprint_id": str(data["blueprint_id"])},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert first.status_code == 302
|
||||||
|
|
||||||
|
second = client.post(
|
||||||
|
"/servers",
|
||||||
|
data={"name": "beta", "blueprint_id": str(data["blueprint_id"])},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert second.status_code == 302
|
||||||
|
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from l4d2web.models import Server
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
ports = sorted(session.scalars(select(Server.port)).all())
|
||||||
|
assert ports == [27015, 27016]
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_server_returns_409_when_port_range_exhausted(tmp_path, monkeypatch) -> None:
|
||||||
|
db_url = f"sqlite:///{tmp_path/'exhausted.db'}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
monkeypatch.setenv("LEFT4ME_PORT_RANGE_START", "30000")
|
||||||
|
monkeypatch.setenv("LEFT4ME_PORT_RANGE_END", "30000")
|
||||||
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
user = User(username="alice", password_digest=hash_password("secret"), admin=False)
|
||||||
|
session.add(user)
|
||||||
|
session.flush()
|
||||||
|
blueprint = Blueprint(user_id=user.id, name="bp", arguments="[]", config="[]")
|
||||||
|
session.add(blueprint)
|
||||||
|
session.flush()
|
||||||
|
user_id = user.id
|
||||||
|
blueprint_id = blueprint.id
|
||||||
|
|
||||||
|
client = app.test_client()
|
||||||
|
with client.session_transaction() as sess:
|
||||||
|
sess["user_id"] = user_id
|
||||||
|
sess["csrf_token"] = "test-token"
|
||||||
|
|
||||||
|
first = client.post(
|
||||||
|
"/servers",
|
||||||
|
data={"name": "alpha", "blueprint_id": str(blueprint_id)},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert first.status_code == 302
|
||||||
|
|
||||||
|
second = client.post(
|
||||||
|
"/servers",
|
||||||
|
data={"name": "beta", "blueprint_id": str(blueprint_id)},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert second.status_code == 409
|
||||||
|
|
||||||
|
|
||||||
|
def test_delete_operation_redirects_to_index(user_client_with_blueprints) -> None:
|
||||||
|
client, data = user_client_with_blueprints
|
||||||
|
create = client.post(
|
||||||
|
"/servers",
|
||||||
|
data=json.dumps({"name": "alpha", "port": 27015, "blueprint_id": data["blueprint_id"]}),
|
||||||
|
content_type="application/json",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
server_id = create.get_json()["id"]
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
f"/servers/{server_id}/delete",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status_code == 302
|
||||||
|
assert response.headers["Location"] == "/servers"
|
||||||
|
|
||||||
|
|
||||||
|
def test_servers_page_prefills_blueprint_when_owned(user_client_with_blueprints) -> None:
|
||||||
|
client, data = user_client_with_blueprints
|
||||||
|
response = client.get(f"/servers?blueprint_id={data['other_blueprint_id']}")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert f'<option value="{data["other_blueprint_id"]}" selected>' in text
|
||||||
|
assert "showModal" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_servers_page_ignores_foreign_blueprint_id(user_client_with_blueprints) -> None:
|
||||||
|
client, _ = user_client_with_blueprints
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
other = User(username="bob", password_digest=hash_password("secret"), admin=False)
|
||||||
|
session.add(other)
|
||||||
|
session.flush()
|
||||||
|
foreign = Blueprint(user_id=other.id, name="foreign", arguments="[]", config="[]")
|
||||||
|
session.add(foreign)
|
||||||
|
session.flush()
|
||||||
|
foreign_id = foreign.id
|
||||||
|
|
||||||
|
response = client.get(f"/servers?blueprint_id={foreign_id}")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "selected" not in text
|
||||||
|
assert "showModal" not in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_servers_page_ignores_non_integer_blueprint_id(user_client_with_blueprints) -> None:
|
||||||
|
client, _ = user_client_with_blueprints
|
||||||
|
response = client.get("/servers?blueprint_id=abc")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "showModal" not in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_servers_page_without_param_does_not_auto_open(user_client_with_blueprints) -> None:
|
||||||
|
client, _ = user_client_with_blueprints
|
||||||
|
response = client.get("/servers")
|
||||||
|
text = response.get_data(as_text=True)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "showModal" not in text
|
||||||
|
|
||||||
|
|
||||||
def test_lifecycle_form_creates_queued_job(user_client_with_blueprints) -> None:
|
def test_lifecycle_form_creates_queued_job(user_client_with_blueprints) -> None:
|
||||||
client, data = user_client_with_blueprints
|
client, data = user_client_with_blueprints
|
||||||
create_response = client.post(
|
create_response = client.post(
|
||||||
|
|
@ -149,3 +419,42 @@ def test_lifecycle_form_creates_queued_job(user_client_with_blueprints) -> None:
|
||||||
|
|
||||||
assert response.status_code == 302
|
assert response.status_code == 302
|
||||||
assert response.headers["Location"] == f"/servers/{server_id}"
|
assert response.headers["Location"] == f"/servers/{server_id}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_reset_operation_enqueues_job_and_stops_desired_state(user_client_with_blueprints) -> None:
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from l4d2web.models import Job, Server
|
||||||
|
|
||||||
|
client, data = user_client_with_blueprints
|
||||||
|
create_response = client.post(
|
||||||
|
"/servers",
|
||||||
|
data=json.dumps({"name": "alpha", "port": 27015, "blueprint_id": data["blueprint_id"]}),
|
||||||
|
content_type="application/json",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
server_id = create_response.get_json()["id"]
|
||||||
|
|
||||||
|
# Pretend the user already started it.
|
||||||
|
with session_scope() as session:
|
||||||
|
server = session.scalar(select(Server).where(Server.id == server_id))
|
||||||
|
assert server is not None
|
||||||
|
server.desired_state = "running"
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
f"/servers/{server_id}/reset",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status_code == 302
|
||||||
|
assert response.headers["Location"] == f"/servers/{server_id}"
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
server = session.scalar(select(Server).where(Server.id == server_id))
|
||||||
|
assert server is not None
|
||||||
|
assert server.desired_state == "stopped"
|
||||||
|
jobs = session.scalars(
|
||||||
|
select(Job).where(Job.server_id == server_id, Job.operation == "reset")
|
||||||
|
).all()
|
||||||
|
assert len(jobs) == 1
|
||||||
|
assert jobs[0].state == "queued"
|
||||||
|
|
|
||||||
|
|
@ -48,6 +48,23 @@ def test_owner_can_stream_server_logs(owner_client_with_server, monkeypatch) ->
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
|
||||||
|
def test_log_stream_translates_heartbeat_to_sse_keepalive(owner_client_with_server, monkeypatch) -> None:
|
||||||
|
client, server_id = owner_client_with_server
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"l4d2web.services.l4d2_facade.stream_server_logs",
|
||||||
|
lambda name, lines=200, follow=True: iter(["first", "", "second"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
response = client.get(f"/servers/{server_id}/logs/stream")
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.get_data(as_text=True)
|
||||||
|
assert "data: first\n\n" in body
|
||||||
|
assert "data: second\n\n" in body
|
||||||
|
assert ": keepalive\n\n" in body
|
||||||
|
assert "data: \n\n" not in body
|
||||||
|
|
||||||
|
|
||||||
def test_status_precedence() -> None:
|
def test_status_precedence() -> None:
|
||||||
from l4d2web.services.status import compute_display_state
|
from l4d2web.services.status import compute_display_state
|
||||||
|
|
||||||
|
|
|
||||||
312
l4d2web/tests/test_steam_workshop.py
Normal file
312
l4d2web/tests/test_steam_workshop.py
Normal file
|
|
@ -0,0 +1,312 @@
|
||||||
|
"""Tests for the Steam Workshop API client and downloader."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from l4d2web.services import steam_workshop
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_workshop_input_single_numeric() -> None:
|
||||||
|
assert steam_workshop.parse_workshop_input("12345") == ["12345"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_workshop_input_single_url() -> None:
|
||||||
|
url = "https://steamcommunity.com/sharedfiles/filedetails/?id=98765"
|
||||||
|
assert steam_workshop.parse_workshop_input(url) == ["98765"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_workshop_input_workshop_url_variant() -> None:
|
||||||
|
url = "steamcommunity.com/workshop/filedetails/?id=42"
|
||||||
|
assert steam_workshop.parse_workshop_input(url) == ["42"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_workshop_input_multiline_batch() -> None:
|
||||||
|
raw = """
|
||||||
|
12345
|
||||||
|
https://steamcommunity.com/sharedfiles/filedetails/?id=67890
|
||||||
|
99999
|
||||||
|
"""
|
||||||
|
assert steam_workshop.parse_workshop_input(raw) == ["12345", "67890", "99999"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_workshop_input_deduplicates_preserving_order() -> None:
|
||||||
|
raw = "100\n200\n100\n300"
|
||||||
|
assert steam_workshop.parse_workshop_input(raw) == ["100", "200", "300"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_workshop_input_rejects_garbage() -> None:
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
steam_workshop.parse_workshop_input("not-a-number")
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_workshop_input_rejects_empty() -> None:
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
steam_workshop.parse_workshop_input("")
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_workshop_input_rejects_non_steam_url() -> None:
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
steam_workshop.parse_workshop_input("https://example.com/?id=12345")
|
||||||
|
|
||||||
|
|
||||||
|
def test_endpoints_are_https() -> None:
|
||||||
|
assert steam_workshop.GET_PUBLISHED_FILE_DETAILS_URL.startswith("https://")
|
||||||
|
assert steam_workshop.GET_COLLECTION_DETAILS_URL.startswith("https://")
|
||||||
|
assert "api.steampowered.com" in steam_workshop.GET_PUBLISHED_FILE_DETAILS_URL
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_collection_returns_child_ids() -> None:
|
||||||
|
fake_response = MagicMock(status_code=200)
|
||||||
|
fake_response.raise_for_status = MagicMock()
|
||||||
|
fake_response.json.return_value = {
|
||||||
|
"response": {
|
||||||
|
"collectiondetails": [
|
||||||
|
{
|
||||||
|
"publishedfileid": "555",
|
||||||
|
"result": 1,
|
||||||
|
"children": [
|
||||||
|
{"publishedfileid": "1001", "filetype": 0},
|
||||||
|
{"publishedfileid": "1002", "filetype": 0},
|
||||||
|
{"publishedfileid": "9999", "filetype": 1}, # nested collection — skip
|
||||||
|
],
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
with patch.object(steam_workshop, "_session", return_value=MagicMock(post=MagicMock(return_value=fake_response))):
|
||||||
|
ids = steam_workshop.resolve_collection("555")
|
||||||
|
assert ids == ["1001", "1002"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_metadata_batch_parses_published_file_details() -> None:
|
||||||
|
fake_response = MagicMock(status_code=200)
|
||||||
|
fake_response.raise_for_status = MagicMock()
|
||||||
|
fake_response.json.return_value = {
|
||||||
|
"response": {
|
||||||
|
"publishedfiledetails": [
|
||||||
|
{
|
||||||
|
"publishedfileid": "1001",
|
||||||
|
"result": 1,
|
||||||
|
"consumer_app_id": 550,
|
||||||
|
"title": "Map A",
|
||||||
|
"filename": "map_a.vpk",
|
||||||
|
"file_url": "https://steamusercontent.com/abc/map_a.vpk",
|
||||||
|
"file_size": "1024",
|
||||||
|
"time_updated": 1700000000,
|
||||||
|
"preview_url": "https://steamuserimages.com/preview_a.jpg",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
with patch.object(steam_workshop, "_session", return_value=MagicMock(post=MagicMock(return_value=fake_response))):
|
||||||
|
metas = steam_workshop.fetch_metadata_batch(["1001"], mode="add")
|
||||||
|
assert len(metas) == 1
|
||||||
|
m = metas[0]
|
||||||
|
assert m.steam_id == "1001"
|
||||||
|
assert m.title == "Map A"
|
||||||
|
assert m.filename == "map_a.vpk"
|
||||||
|
assert m.file_url == "https://steamusercontent.com/abc/map_a.vpk"
|
||||||
|
assert m.file_size == 1024
|
||||||
|
assert m.time_updated == 1700000000
|
||||||
|
assert m.preview_url == "https://steamuserimages.com/preview_a.jpg"
|
||||||
|
assert m.consumer_app_id == 550
|
||||||
|
assert m.result == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_metadata_batch_rejects_non_l4d2_in_add_mode() -> None:
|
||||||
|
fake_response = MagicMock(status_code=200)
|
||||||
|
fake_response.raise_for_status = MagicMock()
|
||||||
|
fake_response.json.return_value = {
|
||||||
|
"response": {
|
||||||
|
"publishedfiledetails": [
|
||||||
|
{
|
||||||
|
"publishedfileid": "1001",
|
||||||
|
"result": 1,
|
||||||
|
"consumer_app_id": 440, # TF2
|
||||||
|
"title": "Other",
|
||||||
|
"filename": "x.vpk",
|
||||||
|
"file_url": "https://example.com/x.vpk",
|
||||||
|
"file_size": "0",
|
||||||
|
"time_updated": 0,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
with patch.object(steam_workshop, "_session", return_value=MagicMock(post=MagicMock(return_value=fake_response))):
|
||||||
|
with pytest.raises(steam_workshop.WorkshopValidationError):
|
||||||
|
steam_workshop.fetch_metadata_batch(["1001"], mode="add")
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_metadata_batch_skips_non_l4d2_in_refresh_mode() -> None:
|
||||||
|
fake_response = MagicMock(status_code=200)
|
||||||
|
fake_response.raise_for_status = MagicMock()
|
||||||
|
fake_response.json.return_value = {
|
||||||
|
"response": {
|
||||||
|
"publishedfiledetails": [
|
||||||
|
{
|
||||||
|
"publishedfileid": "1001",
|
||||||
|
"result": 1,
|
||||||
|
"consumer_app_id": 440,
|
||||||
|
"title": "Other",
|
||||||
|
"filename": "x.vpk",
|
||||||
|
"file_url": "https://example.com/x.vpk",
|
||||||
|
"file_size": "0",
|
||||||
|
"time_updated": 0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"publishedfileid": "1002",
|
||||||
|
"result": 1,
|
||||||
|
"consumer_app_id": 550,
|
||||||
|
"title": "Good",
|
||||||
|
"filename": "g.vpk",
|
||||||
|
"file_url": "https://example.com/g.vpk",
|
||||||
|
"file_size": "100",
|
||||||
|
"time_updated": 1,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
with patch.object(steam_workshop, "_session", return_value=MagicMock(post=MagicMock(return_value=fake_response))):
|
||||||
|
metas = steam_workshop.fetch_metadata_batch(["1001", "1002"], mode="refresh")
|
||||||
|
# The non-L4D2 item is dropped; the L4D2 item is kept.
|
||||||
|
assert [m.steam_id for m in metas] == ["1002"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_metadata_batch_captures_result_failure() -> None:
|
||||||
|
fake_response = MagicMock(status_code=200)
|
||||||
|
fake_response.raise_for_status = MagicMock()
|
||||||
|
fake_response.json.return_value = {
|
||||||
|
"response": {
|
||||||
|
"publishedfiledetails": [
|
||||||
|
{
|
||||||
|
"publishedfileid": "999",
|
||||||
|
"result": 9, # not found / hidden / etc.
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
with patch.object(steam_workshop, "_session", return_value=MagicMock(post=MagicMock(return_value=fake_response))):
|
||||||
|
metas = steam_workshop.fetch_metadata_batch(["999"], mode="refresh")
|
||||||
|
# Item is kept but marked with the failing result; consumer app id never validated.
|
||||||
|
assert len(metas) == 1
|
||||||
|
assert metas[0].result == 9
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_to_cache_writes_atomically_and_sets_mtime(tmp_path: Path) -> None:
|
||||||
|
cache_root = tmp_path / "workshop_cache"
|
||||||
|
cache_root.mkdir()
|
||||||
|
meta = steam_workshop.WorkshopMetadata(
|
||||||
|
steam_id="1001",
|
||||||
|
title="A",
|
||||||
|
filename="a.vpk",
|
||||||
|
file_url="https://example.com/a.vpk",
|
||||||
|
file_size=11,
|
||||||
|
time_updated=1700000000,
|
||||||
|
preview_url="",
|
||||||
|
consumer_app_id=550,
|
||||||
|
result=1,
|
||||||
|
)
|
||||||
|
fake_response = MagicMock(status_code=200)
|
||||||
|
fake_response.raise_for_status = MagicMock()
|
||||||
|
fake_response.iter_content.return_value = [b"hello world"]
|
||||||
|
|
||||||
|
with patch.object(steam_workshop, "_session", return_value=MagicMock(get=MagicMock(return_value=fake_response))):
|
||||||
|
path = steam_workshop.download_to_cache(meta, cache_root)
|
||||||
|
|
||||||
|
assert path == cache_root / "1001.vpk"
|
||||||
|
assert path.read_bytes() == b"hello world"
|
||||||
|
assert int(path.stat().st_mtime) == 1700000000
|
||||||
|
# No leftover .partial file.
|
||||||
|
assert not (cache_root / "1001.vpk.partial").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_to_cache_is_idempotent(tmp_path: Path) -> None:
|
||||||
|
cache_root = tmp_path / "workshop_cache"
|
||||||
|
cache_root.mkdir()
|
||||||
|
target = cache_root / "1001.vpk"
|
||||||
|
target.write_bytes(b"existing")
|
||||||
|
os.utime(target, (1700000000, 1700000000))
|
||||||
|
|
||||||
|
meta = steam_workshop.WorkshopMetadata(
|
||||||
|
steam_id="1001",
|
||||||
|
title="A",
|
||||||
|
filename="a.vpk",
|
||||||
|
file_url="https://example.com/a.vpk",
|
||||||
|
file_size=8, # matches existing
|
||||||
|
time_updated=1700000000, # matches existing mtime
|
||||||
|
preview_url="",
|
||||||
|
consumer_app_id=550,
|
||||||
|
result=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
fake_session = MagicMock()
|
||||||
|
with patch.object(steam_workshop, "_session", return_value=fake_session):
|
||||||
|
steam_workshop.download_to_cache(meta, cache_root)
|
||||||
|
|
||||||
|
fake_session.get.assert_not_called()
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_to_cache_redownloads_when_mtime_or_size_differ(tmp_path: Path) -> None:
|
||||||
|
cache_root = tmp_path / "workshop_cache"
|
||||||
|
cache_root.mkdir()
|
||||||
|
target = cache_root / "1001.vpk"
|
||||||
|
target.write_bytes(b"old")
|
||||||
|
os.utime(target, (1500000000, 1500000000))
|
||||||
|
|
||||||
|
meta = steam_workshop.WorkshopMetadata(
|
||||||
|
steam_id="1001",
|
||||||
|
title="A",
|
||||||
|
filename="a.vpk",
|
||||||
|
file_url="https://example.com/a.vpk",
|
||||||
|
file_size=11,
|
||||||
|
time_updated=1700000000,
|
||||||
|
preview_url="",
|
||||||
|
consumer_app_id=550,
|
||||||
|
result=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
fake_response = MagicMock(status_code=200)
|
||||||
|
fake_response.raise_for_status = MagicMock()
|
||||||
|
fake_response.iter_content.return_value = [b"hello world"]
|
||||||
|
|
||||||
|
with patch.object(steam_workshop, "_session", return_value=MagicMock(get=MagicMock(return_value=fake_response))):
|
||||||
|
steam_workshop.download_to_cache(meta, cache_root)
|
||||||
|
|
||||||
|
assert target.read_bytes() == b"hello world"
|
||||||
|
assert int(target.stat().st_mtime) == 1700000000
|
||||||
|
|
||||||
|
|
||||||
|
def test_refresh_all_uses_thread_pool_and_collects_errors(tmp_path: Path) -> None:
|
||||||
|
cache_root = tmp_path / "workshop_cache"
|
||||||
|
cache_root.mkdir()
|
||||||
|
|
||||||
|
metas = [
|
||||||
|
steam_workshop.WorkshopMetadata(
|
||||||
|
steam_id=str(i),
|
||||||
|
title=f"M{i}",
|
||||||
|
filename=f"m{i}.vpk",
|
||||||
|
file_url=f"https://example.com/m{i}.vpk",
|
||||||
|
file_size=5,
|
||||||
|
time_updated=1700000000,
|
||||||
|
preview_url="",
|
||||||
|
consumer_app_id=550,
|
||||||
|
result=1,
|
||||||
|
)
|
||||||
|
for i in (1, 2, 3)
|
||||||
|
]
|
||||||
|
|
||||||
|
def fake_download(meta, cache_root_arg, **kwargs):
|
||||||
|
if meta.steam_id == "2":
|
||||||
|
raise RuntimeError("simulated download failure")
|
||||||
|
return cache_root_arg / f"{meta.steam_id}.vpk"
|
||||||
|
|
||||||
|
with patch.object(steam_workshop, "download_to_cache", side_effect=fake_download):
|
||||||
|
report = steam_workshop.refresh_all(metas, cache_root, executor_workers=4)
|
||||||
|
|
||||||
|
assert report.downloaded == 2
|
||||||
|
assert report.errors == 1
|
||||||
|
assert "2" in report.per_item_errors
|
||||||
193
l4d2web/tests/test_workshop_overlay_models.py
Normal file
193
l4d2web/tests/test_workshop_overlay_models.py
Normal file
|
|
@ -0,0 +1,193 @@
|
||||||
|
"""Tests for the workshop-overlay schema additions: typed Overlay, partial
|
||||||
|
unique indexes, WorkshopItem registry, and overlay_workshop_items association.
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy.exc import IntegrityError
|
||||||
|
|
||||||
|
from l4d2web.db import init_db, session_scope
|
||||||
|
from l4d2web.models import (
|
||||||
|
Job,
|
||||||
|
Overlay,
|
||||||
|
OverlayWorkshopItem,
|
||||||
|
User,
|
||||||
|
WorkshopItem,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def db(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("DATABASE_URL", f"sqlite:///{tmp_path/'workshop.db'}")
|
||||||
|
init_db()
|
||||||
|
yield
|
||||||
|
|
||||||
|
|
||||||
|
def _make_user(username: str) -> int:
|
||||||
|
with session_scope() as s:
|
||||||
|
user = User(username=username, password_digest="x")
|
||||||
|
s.add(user)
|
||||||
|
s.flush()
|
||||||
|
return user.id
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_has_type_and_user_id(db) -> None:
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(Overlay(name="standard", path="standard"))
|
||||||
|
s.flush()
|
||||||
|
row = s.query(Overlay).filter_by(name="standard").one()
|
||||||
|
assert row.type == "workshop"
|
||||||
|
assert row.user_id is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_has_script_columns(db) -> None:
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(Overlay(name="defaulted", path="1"))
|
||||||
|
s.flush()
|
||||||
|
row = s.query(Overlay).filter_by(name="defaulted").one()
|
||||||
|
assert row.script == ""
|
||||||
|
assert row.last_build_status == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_system_overlays_with_same_name_are_rejected(db) -> None:
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(Overlay(name="shared", path="shared", type="l4d2center_maps", user_id=None))
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
with pytest.raises(IntegrityError):
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(Overlay(name="shared", path="other", type="cedapug_maps", user_id=None))
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_users_can_share_workshop_overlay_name(db) -> None:
|
||||||
|
alice_id = _make_user("alice")
|
||||||
|
bob_id = _make_user("bob")
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(Overlay(name="my-maps", path="1", type="workshop", user_id=alice_id))
|
||||||
|
s.add(Overlay(name="my-maps", path="2", type="workshop", user_id=bob_id))
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
rows = s.query(Overlay).filter_by(name="my-maps").all()
|
||||||
|
assert {r.user_id for r in rows} == {alice_id, bob_id}
|
||||||
|
|
||||||
|
|
||||||
|
def test_same_user_cannot_have_duplicate_workshop_name(db) -> None:
|
||||||
|
user_id = _make_user("alice")
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(Overlay(name="dupe", path="1", type="workshop", user_id=user_id))
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
with pytest.raises(IntegrityError):
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(Overlay(name="dupe", path="2", type="workshop", user_id=user_id))
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
|
||||||
|
def test_workshop_item_steam_id_is_unique(db) -> None:
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(WorkshopItem(steam_id="123", title="Map A"))
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
with pytest.raises(IntegrityError):
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(WorkshopItem(steam_id="123", title="Map A duplicate"))
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_workshop_item_unique_per_overlay(db) -> None:
|
||||||
|
user_id = _make_user("alice")
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
ov = Overlay(name="my-maps", path="1", type="workshop", user_id=user_id)
|
||||||
|
wi = WorkshopItem(steam_id="555", title="A")
|
||||||
|
s.add_all([ov, wi])
|
||||||
|
s.flush()
|
||||||
|
s.add(OverlayWorkshopItem(overlay_id=ov.id, workshop_item_id=wi.id))
|
||||||
|
s.flush()
|
||||||
|
overlay_id = ov.id
|
||||||
|
workshop_item_id = wi.id
|
||||||
|
|
||||||
|
with pytest.raises(IntegrityError):
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(
|
||||||
|
OverlayWorkshopItem(
|
||||||
|
overlay_id=overlay_id, workshop_item_id=workshop_item_id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
|
||||||
|
def test_deleting_overlay_cascades_associations_but_not_workshop_items(db) -> None:
|
||||||
|
user_id = _make_user("alice")
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
ov = Overlay(name="my-maps", path="1", type="workshop", user_id=user_id)
|
||||||
|
wi = WorkshopItem(steam_id="777", title="A")
|
||||||
|
s.add_all([ov, wi])
|
||||||
|
s.flush()
|
||||||
|
s.add(OverlayWorkshopItem(overlay_id=ov.id, workshop_item_id=wi.id))
|
||||||
|
s.flush()
|
||||||
|
overlay_id = ov.id
|
||||||
|
|
||||||
|
# Delete via raw connection to actually exercise ON DELETE CASCADE / RESTRICT.
|
||||||
|
from l4d2web.db import get_engine
|
||||||
|
|
||||||
|
engine = get_engine()
|
||||||
|
with engine.begin() as conn:
|
||||||
|
conn.exec_driver_sql("PRAGMA foreign_keys=ON")
|
||||||
|
conn.exec_driver_sql(f"DELETE FROM overlays WHERE id = {overlay_id}")
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
assert s.query(OverlayWorkshopItem).count() == 0
|
||||||
|
assert s.query(WorkshopItem).filter_by(steam_id="777").count() == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_job_has_overlay_id_column(db) -> None:
|
||||||
|
user_id = _make_user("alice")
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
ov = Overlay(name="my-maps", path="1", type="workshop", user_id=user_id)
|
||||||
|
s.add(ov)
|
||||||
|
s.flush()
|
||||||
|
s.add(
|
||||||
|
Job(
|
||||||
|
user_id=user_id,
|
||||||
|
server_id=None,
|
||||||
|
overlay_id=ov.id,
|
||||||
|
operation="build_overlay",
|
||||||
|
state="queued",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
job = s.query(Job).filter_by(operation="build_overlay").one()
|
||||||
|
assert job.overlay_id is not None
|
||||||
|
assert job.server_id is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_id_does_not_reuse_after_delete(db) -> None:
|
||||||
|
"""SQLite AUTOINCREMENT must guarantee deleted IDs are never reused."""
|
||||||
|
user_id = _make_user("alice")
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(Overlay(name="first", path="1", type="workshop", user_id=user_id))
|
||||||
|
s.add(Overlay(name="second", path="2", type="workshop", user_id=user_id))
|
||||||
|
s.flush()
|
||||||
|
ids_before = sorted(o.id for o in s.query(Overlay).all())
|
||||||
|
last_id = ids_before[-1]
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
last = s.query(Overlay).filter_by(id=last_id).one()
|
||||||
|
s.delete(last)
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
with session_scope() as s:
|
||||||
|
s.add(Overlay(name="third", path="3", type="workshop", user_id=user_id))
|
||||||
|
s.flush()
|
||||||
|
new_id = s.query(Overlay).filter_by(name="third").one().id
|
||||||
|
|
||||||
|
assert new_id > last_id, (
|
||||||
|
f"AUTOINCREMENT should never reuse IDs, but got {new_id} after deleting {last_id}"
|
||||||
|
)
|
||||||
23
l4d2web/tests/test_workshop_paths.py
Normal file
23
l4d2web/tests/test_workshop_paths.py
Normal file
|
|
@ -0,0 +1,23 @@
|
||||||
|
"""Tests for workshop_paths cache-resolution helpers."""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from l4d2web.services import workshop_paths
|
||||||
|
|
||||||
|
|
||||||
|
def test_workshop_cache_root_uses_left4me_root(monkeypatch, tmp_path: Path) -> None:
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
assert workshop_paths.workshop_cache_root() == tmp_path / "workshop_cache"
|
||||||
|
|
||||||
|
|
||||||
|
def test_cache_path_returns_id_only_filename(monkeypatch, tmp_path: Path) -> None:
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
assert workshop_paths.cache_path("12345") == tmp_path / "workshop_cache" / "12345.vpk"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("bad", ["abc", "", "12/34", "..", "../etc", "1 2", " 1"])
|
||||||
|
def test_cache_path_rejects_non_digit_steam_id(monkeypatch, tmp_path: Path, bad: str) -> None:
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
workshop_paths.cache_path(bad)
|
||||||
283
l4d2web/tests/test_workshop_routes.py
Normal file
283
l4d2web/tests/test_workshop_routes.py
Normal file
|
|
@ -0,0 +1,283 @@
|
||||||
|
"""Tests for the workshop overlay routes (add items, remove items, build,
|
||||||
|
admin refresh)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Iterable
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from l4d2web.app import create_app
|
||||||
|
from l4d2web.auth import hash_password
|
||||||
|
from l4d2web.db import init_db, session_scope
|
||||||
|
from l4d2web.models import (
|
||||||
|
Job,
|
||||||
|
Overlay,
|
||||||
|
OverlayWorkshopItem,
|
||||||
|
User,
|
||||||
|
WorkshopItem,
|
||||||
|
)
|
||||||
|
from l4d2web.services import steam_workshop
|
||||||
|
|
||||||
|
|
||||||
|
def _meta(steam_id: str, *, app_id: int = 550, result: int = 1) -> steam_workshop.WorkshopMetadata:
|
||||||
|
return steam_workshop.WorkshopMetadata(
|
||||||
|
steam_id=steam_id,
|
||||||
|
title=f"Item {steam_id}",
|
||||||
|
filename=f"{steam_id}.vpk",
|
||||||
|
file_url=f"https://example.com/{steam_id}.vpk",
|
||||||
|
file_size=42,
|
||||||
|
time_updated=1700000000,
|
||||||
|
preview_url=f"https://example.com/preview-{steam_id}.jpg",
|
||||||
|
consumer_app_id=app_id,
|
||||||
|
result=result,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def env_user(tmp_path, monkeypatch):
|
||||||
|
db_url = f"sqlite:///{tmp_path/'wr.db'}"
|
||||||
|
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||||
|
monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))
|
||||||
|
app = create_app({"TESTING": True, "DATABASE_URL": db_url, "SECRET_KEY": "test"})
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
user = User(username="alice", password_digest=hash_password("x"), admin=False)
|
||||||
|
admin = User(username="admin", password_digest=hash_password("x"), admin=True)
|
||||||
|
session.add_all([user, admin])
|
||||||
|
session.flush()
|
||||||
|
user_id = user.id
|
||||||
|
admin_id = admin.id
|
||||||
|
|
||||||
|
def login(uid):
|
||||||
|
c = app.test_client()
|
||||||
|
with c.session_transaction() as sess:
|
||||||
|
sess["user_id"] = uid
|
||||||
|
sess["csrf_token"] = "test-token"
|
||||||
|
return c
|
||||||
|
|
||||||
|
return app, login, user_id, admin_id
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def overlay_for(env_user):
|
||||||
|
app, login, user_id, admin_id = env_user
|
||||||
|
|
||||||
|
user_client = login(user_id)
|
||||||
|
response = user_client.post(
|
||||||
|
"/overlays",
|
||||||
|
data={"name": "my-maps", "type": "workshop"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302, response.get_data(as_text=True)
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
overlay = session.query(Overlay).filter_by(name="my-maps").one()
|
||||||
|
overlay_id = overlay.id
|
||||||
|
|
||||||
|
return app, login, user_id, admin_id, overlay_id
|
||||||
|
|
||||||
|
|
||||||
|
def _patch_steam(metas: Iterable[steam_workshop.WorkshopMetadata]):
|
||||||
|
return patch.object(steam_workshop, "fetch_metadata_batch", return_value=list(metas))
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_single_item_creates_association_and_enqueues_build(overlay_for):
|
||||||
|
app, login, user_id, _admin_id, overlay_id = overlay_for
|
||||||
|
user_client = login(user_id)
|
||||||
|
|
||||||
|
with _patch_steam([_meta("1001")]):
|
||||||
|
response = user_client.post(
|
||||||
|
f"/overlays/{overlay_id}/items",
|
||||||
|
data={"input": "1001", "input_mode": "items"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
assert response.headers["Location"].startswith("/jobs/")
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
n_assoc = session.query(OverlayWorkshopItem).count()
|
||||||
|
assert n_assoc == 1
|
||||||
|
wi = session.query(WorkshopItem).filter_by(steam_id="1001").one()
|
||||||
|
assert wi.title == "Item 1001"
|
||||||
|
assert wi.preview_url.endswith("preview-1001.jpg")
|
||||||
|
# Auto-enqueued build_overlay job.
|
||||||
|
jobs = session.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
|
||||||
|
assert len(jobs) == 1
|
||||||
|
assert jobs[0].state == "queued"
|
||||||
|
assert response.headers["Location"] == f"/jobs/{jobs[0].id}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_multiline_batch_coalesces_into_one_build_job(overlay_for):
|
||||||
|
app, login, user_id, _admin_id, overlay_id = overlay_for
|
||||||
|
user_client = login(user_id)
|
||||||
|
|
||||||
|
with _patch_steam([_meta(s) for s in ("1001", "1002", "1003")]):
|
||||||
|
response = user_client.post(
|
||||||
|
f"/overlays/{overlay_id}/items",
|
||||||
|
data={"input": "1001\n1002\n1003", "input_mode": "items"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
assert response.headers["Location"].startswith("/jobs/")
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
assert session.query(OverlayWorkshopItem).count() == 3
|
||||||
|
jobs = session.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
|
||||||
|
assert len(jobs) == 1, "multi-item add should coalesce into a single build job"
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_collection_resolves_members(overlay_for):
|
||||||
|
app, login, user_id, _admin_id, overlay_id = overlay_for
|
||||||
|
user_client = login(user_id)
|
||||||
|
|
||||||
|
with patch.object(steam_workshop, "resolve_collection", return_value=["1001", "1002"]) as resolve:
|
||||||
|
with _patch_steam([_meta("1001"), _meta("1002")]):
|
||||||
|
response = user_client.post(
|
||||||
|
f"/overlays/{overlay_id}/items",
|
||||||
|
data={"input": "555", "input_mode": "collection"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
assert response.headers["Location"].startswith("/jobs/")
|
||||||
|
resolve.assert_called_once_with("555")
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
assert session.query(OverlayWorkshopItem).count() == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_non_l4d2_item_returns_400(overlay_for):
|
||||||
|
app, login, user_id, _admin_id, overlay_id = overlay_for
|
||||||
|
user_client = login(user_id)
|
||||||
|
|
||||||
|
def raise_validation(*args, **kwargs):
|
||||||
|
raise steam_workshop.WorkshopValidationError("not L4D2")
|
||||||
|
|
||||||
|
with patch.object(steam_workshop, "fetch_metadata_batch", side_effect=raise_validation):
|
||||||
|
response = user_client.post(
|
||||||
|
f"/overlays/{overlay_id}/items",
|
||||||
|
data={"input": "9999", "input_mode": "items"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert b"not L4D2" in response.data
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
assert session.query(WorkshopItem).count() == 0
|
||||||
|
assert session.query(OverlayWorkshopItem).count() == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_duplicate_item_does_not_500(overlay_for):
|
||||||
|
app, login, user_id, _admin_id, overlay_id = overlay_for
|
||||||
|
user_client = login(user_id)
|
||||||
|
|
||||||
|
with _patch_steam([_meta("1001")]):
|
||||||
|
first = user_client.post(
|
||||||
|
f"/overlays/{overlay_id}/items",
|
||||||
|
data={"input": "1001", "input_mode": "items"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert first.status_code == 302
|
||||||
|
|
||||||
|
with _patch_steam([_meta("1001")]):
|
||||||
|
second = user_client.post(
|
||||||
|
f"/overlays/{overlay_id}/items",
|
||||||
|
data={"input": "1001", "input_mode": "items"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert second.status_code == 302
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
assert session.query(OverlayWorkshopItem).count() == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_remove_item_drops_association_and_enqueues_rebuild(overlay_for):
|
||||||
|
app, login, user_id, _admin_id, overlay_id = overlay_for
|
||||||
|
user_client = login(user_id)
|
||||||
|
|
||||||
|
with _patch_steam([_meta("1001")]):
|
||||||
|
user_client.post(
|
||||||
|
f"/overlays/{overlay_id}/items",
|
||||||
|
data={"input": "1001", "input_mode": "items"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
wi = session.query(WorkshopItem).filter_by(steam_id="1001").one()
|
||||||
|
item_id = wi.id
|
||||||
|
|
||||||
|
response = user_client.post(
|
||||||
|
f"/overlays/{overlay_id}/items/{item_id}/delete",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
assert response.headers["Location"].startswith("/jobs/")
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
assert session.query(OverlayWorkshopItem).count() == 0
|
||||||
|
# WorkshopItem itself remains (cache survives the association removal).
|
||||||
|
assert session.query(WorkshopItem).filter_by(steam_id="1001").one() is not None
|
||||||
|
# Coalesced into the same queued build_overlay job.
|
||||||
|
jobs = session.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
|
||||||
|
assert len(jobs) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_manual_build_button_enqueues_job(overlay_for):
|
||||||
|
app, login, user_id, _admin_id, overlay_id = overlay_for
|
||||||
|
user_client = login(user_id)
|
||||||
|
|
||||||
|
response = user_client.post(
|
||||||
|
f"/overlays/{overlay_id}/build",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
assert response.headers["Location"].startswith("/jobs/")
|
||||||
|
with session_scope() as session:
|
||||||
|
jobs = session.query(Job).filter_by(operation="build_overlay", overlay_id=overlay_id).all()
|
||||||
|
assert len(jobs) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_admin_refresh_enqueues_global_job(env_user):
|
||||||
|
app, login, user_id, admin_id = env_user
|
||||||
|
admin_client = login(admin_id)
|
||||||
|
|
||||||
|
response = admin_client.post(
|
||||||
|
"/admin/workshop/refresh",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 302
|
||||||
|
assert response.headers["Location"] == "/admin/jobs"
|
||||||
|
with session_scope() as session:
|
||||||
|
jobs = session.query(Job).filter_by(operation="refresh_workshop_items").all()
|
||||||
|
assert len(jobs) == 1
|
||||||
|
assert jobs[0].state == "queued"
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_admin_cannot_refresh(env_user):
|
||||||
|
app, login, user_id, _admin_id = env_user
|
||||||
|
user_client = login(user_id)
|
||||||
|
|
||||||
|
response = user_client.post(
|
||||||
|
"/admin/workshop/refresh",
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 403
|
||||||
|
|
||||||
|
|
||||||
|
def test_other_user_cannot_modify_workshop_overlay(overlay_for):
|
||||||
|
app, login, user_id, _admin_id, overlay_id = overlay_for
|
||||||
|
|
||||||
|
with session_scope() as session:
|
||||||
|
intruder = User(username="bob", password_digest=hash_password("x"), admin=False)
|
||||||
|
session.add(intruder)
|
||||||
|
session.flush()
|
||||||
|
intruder_id = intruder.id
|
||||||
|
|
||||||
|
intruder_client = login(intruder_id)
|
||||||
|
response = intruder_client.post(
|
||||||
|
f"/overlays/{overlay_id}/items",
|
||||||
|
data={"input": "1001", "input_mode": "items"},
|
||||||
|
headers={"X-CSRF-Token": "test-token"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 403
|
||||||
Loading…
Reference in a new issue