chore(deploy): cleanup left4me-web hardening + docs for kernel overlayfs

Drop MountFlags=shared (the assumption that it propagated fuse mounts
to host was incorrect on systemd 257 with ProtectSystem+ReadWritePaths).
Restore PrivateTmp=true (was dropped in 593611e for fuse propagation
that did not work). Rewrite the comment block to describe the new
model: mounts go through the left4me-overlay helper which nsenters
into PID 1's mount namespace, so the unit's mount-ns layout is no
longer load-bearing.

Update the three user-facing READMEs (root, l4d2host, deploy) to drop
fuse-overlayfs / fusermount3 prereqs and call out the kernel overlayfs
mount path through the privileged helper.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

AGENTS.md

@@ -41,7 +41,7 @@ Do not invent architecture outside these plans unless explicitly requested.
   - `logs <name> --lines <n> --follow/--no-follow`
 - Runtime paths are rooted at `LEFT4ME_ROOT`, defaulting to `/var/lib/left4me`.
 - Deployment/config management owns global units under `/usr/local/lib/systemd/system` and privileged helpers under `/usr/local/libexec/left4me`.
-- Overlays are external directories (no overlay content management here).
+- Overlay directories are populated by the web app (workshop downloads, managed-global refresh). The host library only mounts them.
 - Fail-fast subprocess behavior; pass raw stderr; propagate return code.
 - No lock manager, no rollback, no preflight runtime checks.
 - Delete missing instance/runtime dirs must succeed (no-op).

README.md

@@ -27,7 +27,7 @@ Implementation plans remain the source of truth for architecture and task sequen
   - `logs <name> --lines <n> --follow/--no-follow`
 - The web app calls host operations through `l4d2ctl`, not direct `l4d2host` imports.
 - Deployment uses `/var/lib/left4me` for runtime state, `/opt/left4me` for repository contents and the virtualenv, `/etc/left4me` for environment files, and global units under `/usr/local/lib/systemd/system`.
-- Overlay handling is directory-based and externally populated.
+- Overlay handling is directory-based; the web app populates each overlay (workshop downloads, managed-global refresh).
 - No lock manager, no rollback, no preflight checks in host library.
 - CLI propagates subprocess failures via stderr and return code.
 - `delete` on missing instance is no-op success.
@@ -56,7 +56,7 @@ See `deploy/README.md` for the Linux test deployment contract, including the run
 - Typer, PyYAML, pytest
 - Flask, SQLAlchemy, Alembic
 - HTMX (vendored locally), custom CSS, SSE
-- systemd user units, fuse-overlayfs, steamcmd
+- systemd units, kernel overlayfs (mounted via the `left4me-overlay` privileged helper), steamcmd
 
 ## Recommended Implementation Order
 

deploy/README.md

@@ -12,14 +12,14 @@ The deployment uses these paths:
 - `/opt/left4me`: deployed repository contents.
 - `/var/lib/left4me/left4me.db`: SQLite database used by the web app.
 - `/var/lib/left4me/installation`: shared L4D2 installation.
-- `/var/lib/left4me/overlays`: overlay directories. External (admin-managed) overlays still live at any relative path under here; new overlays created through the web UI use `${overlay_id}` as their path.
+- `/var/lib/left4me/overlays`: overlay directories. Each overlay lives at `${overlay_id}` under here.
 - `/var/lib/left4me/workshop_cache`: deduplicated cache of `.vpk` files downloaded for workshop overlays. One file per Steam item, named `{steam_id}.vpk`. Workshop overlays symlink into this tree.
 - `/var/lib/left4me/global_overlay_cache`: cache of non-Steam map archives and extracted `.vpk` files used by managed global map overlays.
 - `/var/lib/left4me/instances`: rendered instance specifications and per-instance state.
 - `/var/lib/left4me/runtime`: per-instance runtime mount directories.
 - `/var/lib/left4me/tmp`: temporary files used by deployment/runtime operations.
 - `/usr/local/lib/systemd/system`: global systemd unit files, including `left4me-server@.service`.
-- `/usr/local/libexec/left4me`: privileged helper commands, including `left4me-systemctl` and `left4me-journalctl`.
+- `/usr/local/libexec/left4me`: privileged helper commands, including `left4me-systemctl`, `left4me-journalctl`, and `left4me-overlay` (the latter mounts the per-instance kernel overlay in PID 1's mount namespace via `nsenter`).
 - `/etc/sudoers.d/left4me`: sudoers rules allowing the web/runtime commands to call the helpers non-interactively.
 
 Static units are generated for `/var/lib/left4me`. If `LEFT4ME_ROOT` changes, regenerate and reinstall the unit files instead of reusing the existing static units.
@@ -60,13 +60,7 @@ Use a strong one-time password and rotate it after first login if needed.
 
 ## Overlay References
 
-Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`. With the default deployment root, they resolve under `/var/lib/left4me/overlays`.
-
-Valid examples:
-
-- `standard`
-- `competitive/base`
-- `users/42/custom`
+Overlay references are relative paths below `${LEFT4ME_ROOT}/overlays`. With the default deployment root, they resolve under `/var/lib/left4me/overlays`. New overlays use `${overlay_id}` as their path; the digit-only form is the only one created by the web app.
 
 Invalid references are rejected:
 
@@ -75,6 +69,9 @@ Invalid references are rejected:
 - Empty path components such as `competitive//base`.
 - Symlink escapes that resolve outside `${LEFT4ME_ROOT}/overlays`.
 
-Overlay content for `external` (admin-managed) overlays is populated outside the host library — typically via SFTP. The web app does not write into them.
+The web app currently supports two overlay surfaces:
 
-`workshop` overlays are populated by the web app: it downloads `.vpk` files from the public Steam Web API into `${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creates absolute symlinks under `${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`. Both the cache and the overlay directory are owned by the `left4me` runtime user; if the web service ever runs as a different uid, ensure it shares a group with the host process and that both trees are group-readable.
+- `workshop` overlays (user-owned) — populated by downloading `.vpk` files from the public Steam Web API into `${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk` and creating absolute symlinks under `${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk`.
+- Managed global overlays (`l4d2center_maps`, `cedapug_maps`, system-wide) — populated by the daily `left4me-refresh-global-overlays` job, which downloads archives into `${LEFT4ME_ROOT}/global_overlay_cache/` and symlinks them into the overlay directory.
+
+Both the caches and the overlay directories are owned by the `left4me` runtime user; if the web service ever runs as a different uid, ensure it shares a group with the host process and that both trees are group-readable.

deploy/deploy-test-server.sh

@@ -79,9 +79,9 @@ fi
 
 if command -v apt-get >/dev/null 2>&1; then
     $sudo_cmd apt-get update
-    $sudo_cmd apt-get install -y python3 python3-venv python3-pip curl ca-certificates tar gzip fuse-overlayfs fuse3 sudo
+    $sudo_cmd apt-get install -y python3 python3-venv python3-pip curl ca-certificates tar gzip util-linux sudo
 elif command -v dnf >/dev/null 2>&1; then
-    $sudo_cmd dnf install -y python3 python3-pip curl ca-certificates tar gzip fuse-overlayfs fuse3 sudo
+    $sudo_cmd dnf install -y python3 python3-pip curl ca-certificates tar gzip util-linux sudo
 else
     printf 'Unsupported package manager: expected apt-get or dnf\n' >&2
     exit 1
@@ -130,7 +130,8 @@ $sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-refr
 $sudo_cmd cp /opt/left4me/deploy/files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer /usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer
 $sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-systemctl
 $sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-journalctl
-$sudo_cmd chmod 0755 /usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-journalctl
+$sudo_cmd cp /opt/left4me/deploy/files/usr/local/libexec/left4me/left4me-overlay /usr/local/libexec/left4me/left4me-overlay
+$sudo_cmd chmod 0755 /usr/local/libexec/left4me/left4me-systemctl /usr/local/libexec/left4me/left4me-journalctl /usr/local/libexec/left4me/left4me-overlay
 $sudo_cmd cp /opt/left4me/deploy/files/etc/sudoers.d/left4me /etc/sudoers.d/left4me
 $sudo_cmd chmod 0440 /etc/sudoers.d/left4me
 $sudo_cmd visudo -cf /etc/sudoers.d/left4me
@@ -177,6 +178,22 @@ if [ -f "$remote_tmp/admin_username" ] && [ -f "$remote_tmp/admin_password" ]; t
     fi
 fi
 
+# One-shot migration: fuse-overlayfs running as the left4me user used
+# user.fuseoverlayfs.* xattrs for whiteouts and opaque-dir markers; kernel
+# overlayfs ignores those entirely, so a pre-existing upper/ from the fuse
+# era would resurrect "deleted" files. Wipe upper/ and work/ for every
+# instance once, gated by a sentinel file so reruns are no-ops.
+overlay_sentinel=/var/lib/left4me/.kernel-overlay-migrated
+if [ ! -e "$overlay_sentinel" ]; then
+    $sudo_cmd sh -c "systemctl stop 'left4me-server@*.service' 2>/dev/null || true"
+    $sudo_cmd systemctl stop left4me-web.service 2>/dev/null || true
+    $sudo_cmd sh -c "findmnt -t fuse.fuse-overlayfs -o TARGET --noheadings 2>/dev/null | xargs -r -n1 umount -l 2>/dev/null || true"
+    $sudo_cmd sh -c "findmnt -t overlay -o TARGET --noheadings 2>/dev/null | grep '/var/lib/left4me/runtime/' | xargs -r -n1 umount -l 2>/dev/null || true"
+    $sudo_cmd sh -c 'for d in /var/lib/left4me/runtime/*/; do [ -d "$d" ] || continue; rm -rf "$d/upper" "$d/work"; mkdir -p "$d/upper" "$d/work"; chown left4me:left4me "$d/upper" "$d/work"; done'
+    $sudo_cmd touch "$overlay_sentinel"
+    $sudo_cmd chown left4me:left4me "$overlay_sentinel"
+fi
+
 $sudo_cmd systemctl daemon-reload
 $sudo_cmd systemctl enable --now left4me-web.service
 $sudo_cmd systemctl restart left4me-web.service

deploy/files/etc/sudoers.d/left4me

@@ -1,3 +1,4 @@
 Defaults:left4me !requiretty
 left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-systemctl *
 left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-journalctl *
+left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-overlay mount *, /usr/local/libexec/left4me/left4me-overlay umount *

deploy/files/usr/local/lib/systemd/system/left4me-web.service

@@ -12,20 +12,20 @@ Environment=HOME=/var/lib/left4me
 Environment=PATH=/opt/left4me/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 EnvironmentFile=/etc/left4me/host.env
 EnvironmentFile=/etc/left4me/web.env
-ExecStart=/opt/left4me/.venv/bin/gunicorn --workers 1 --threads 8 --bind 0.0.0.0:8000 'l4d2web.app:create_app()'
+ExecStart=/opt/left4me/.venv/bin/gunicorn --workers 1 --threads 32 --bind 0.0.0.0:8000 'l4d2web.app:create_app()'
 Restart=on-failure
 RestartSec=3
-# NoNewPrivileges intentionally not set: the worker invokes fusermount3
-# (setuid-root) and sudo to run the systemctl wrapper.
-# ProtectSystem=full + ReadWritePaths implicitly give this unit a
-# private mount namespace. MountFlags=shared makes its mount events
-# propagate back to the host so per-instance fuse-overlayfs mounts are
-# visible to the gameserver units (which inherit host mounts at their
-# own unshare time). Without it, the per-instance mount only exists
-# inside the worker's namespace and the gameserver units fail CHDIR.
+# NoNewPrivileges intentionally not set: the worker invokes sudo to run
+# the left4me-systemctl, left4me-journalctl, and left4me-overlay
+# privileged helpers, all setuid via sudo.
+# ProtectSystem=full + ReadWritePaths implicitly give this unit a private
+# mount namespace, but mount visibility no longer depends on it: overlay
+# mounts are performed by the left4me-overlay helper, which nsenters into
+# PID 1's mount namespace, so the resulting mount lives in the host
+# namespace where the per-instance gameserver units can see it.
 ProtectSystem=full
 ReadWritePaths=/var/lib/left4me
-MountFlags=shared
+PrivateTmp=true
 
 [Install]
 WantedBy=multi-user.target

deploy/files/usr/local/libexec/left4me/left4me-overlay

@@ -0,0 +1,188 @@
+#!/usr/bin/python3
+"""Privileged overlay mount helper for left4me.
+
+Invoked via sudo by the left4me runtime user. Validates inputs strictly,
+then enters PID 1's mount namespace via nsenter to perform the actual
+mount/umount syscall, so the resulting mount lives in the host namespace
+and is visible to the systemd-managed gameserver units.
+
+Verbs:
+    mount <name>    Reads ${LEFT4ME_ROOT}/instances/<name>/instance.env
+                    for L4D2_LOWERDIRS, validates every lowerdir is
+                    under one of installation/overlays/workshop_cache/
+                    global_overlay_cache, then mounts the kernel
+                    overlay at runtime/<name>/merged.
+    umount <name>   Unmounts runtime/<name>/merged.
+
+Set LEFT4ME_OVERLAY_PRINT_ONLY=1 to print the would-be argv (one line,
+shell-quoted) and exit 0 instead of execv. Used by tests.
+"""
+
+import os
+import re
+import shlex
+import sys
+from pathlib import Path
+
+NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{0,63}$")
+DEFAULT_ROOT = "/var/lib/left4me"
+LOWERDIR_ALLOWLIST = (
+    "installation",
+    "overlays",
+    "global_overlay_cache",
+    "workshop_cache",
+)
+MAX_LOWERDIRS = 500
+NSENTER = "/usr/bin/nsenter"
+MOUNT_BIN = "/bin/mount"
+UMOUNT_BIN = "/bin/umount"
+
+
+def die(msg: str) -> None:
+    sys.stderr.write(f"left4me-overlay: {msg}\n")
+    sys.exit(1)
+
+
+def root() -> Path:
+    return Path(os.environ.get("LEFT4ME_ROOT") or DEFAULT_ROOT)
+
+
+def validate_name(name: str) -> str:
+    if not NAME_RE.fullmatch(name):
+        die(f"invalid instance name: {name!r}")
+    return name
+
+
+def parse_lowerdirs(env_path: Path) -> list[str]:
+    if not env_path.is_file():
+        die(f"instance.env not found: {env_path}")
+    raw = None
+    for line in env_path.read_text().splitlines():
+        if "=" not in line:
+            continue
+        key, value = line.split("=", 1)
+        if key.strip() == "L4D2_LOWERDIRS":
+            raw = value
+            break
+    if raw is None:
+        die(f"L4D2_LOWERDIRS not set in {env_path}")
+    if raw == "":
+        die(f"L4D2_LOWERDIRS is empty in {env_path}")
+    parts = raw.split(":")
+    if any(p == "" for p in parts):
+        die(f"L4D2_LOWERDIRS contains an empty entry: {raw!r}")
+    if len(parts) > MAX_LOWERDIRS:
+        die(f"L4D2_LOWERDIRS has {len(parts)} entries (cap {MAX_LOWERDIRS})")
+    return parts
+
+
+def canonical_under(allowed_roots: list[Path], path: Path) -> Path:
+    try:
+        canonical = path.resolve(strict=True)
+    except (FileNotFoundError, RuntimeError):
+        die(f"path does not exist or has a symlink loop: {path}")
+    for r in allowed_roots:
+        if canonical == r or r in canonical.parents:
+            return canonical
+    die(f"path is outside the permitted roots: {path} (resolved: {canonical})")
+
+
+_LISTXATTR = getattr(os, "listxattr", None)
+
+
+def _entry_has_fuse_xattr(path: str) -> str | None:
+    if _LISTXATTR is None:
+        return None
+    try:
+        attrs = _LISTXATTR(path, follow_symlinks=False)
+    except OSError:
+        return None
+    for a in attrs:
+        if a.startswith("user.fuseoverlayfs."):
+            return a
+    return None
+
+
+def assert_no_fuse_xattrs(upper: Path) -> None:
+    if not upper.exists() or _LISTXATTR is None:
+        return
+    for dirpath, dirnames, filenames in os.walk(upper):
+        for entry in (dirpath, *(os.path.join(dirpath, n) for n in dirnames),
+                      *(os.path.join(dirpath, n) for n in filenames)):
+            tainted = _entry_has_fuse_xattr(entry)
+            if tainted:
+                die(
+                    f"upperdir contains fuse-overlayfs xattr {tainted!r} on {entry}; "
+                    "wipe upper/ and work/ before mounting"
+                )
+
+
+def exec_or_print(argv: list[str]) -> None:
+    if os.environ.get("LEFT4ME_OVERLAY_PRINT_ONLY") == "1":
+        print(" ".join(shlex.quote(a) for a in argv))
+        sys.exit(0)
+    os.execv(argv[0], argv)
+
+
+def cmd_mount(name: str) -> None:
+    name = validate_name(name)
+    r = root()
+    instance_env = r / "instances" / name / "instance.env"
+    raw_lowerdirs = parse_lowerdirs(instance_env)
+
+    allowed_roots = [(r / sub).resolve() for sub in LOWERDIR_ALLOWLIST]
+    canonical_lowerdirs = [str(canonical_under(allowed_roots, Path(p))) for p in raw_lowerdirs]
+
+    runtime_name_dir = (r / "runtime" / name).resolve(strict=True)
+    upper = (runtime_name_dir / "upper").resolve(strict=True)
+    work = (runtime_name_dir / "work").resolve(strict=True)
+    merged = (runtime_name_dir / "merged").resolve(strict=True)
+    for label, path in (("upper", upper), ("work", work), ("merged", merged)):
+        if path.parent != runtime_name_dir:
+            die(f"{label} resolved outside runtime/{name}: {path}")
+
+    assert_no_fuse_xattrs(upper)
+
+    options = f"lowerdir={':'.join(canonical_lowerdirs)},upperdir={upper},workdir={work}"
+    argv = [
+        NSENTER,
+        "--mount=/proc/1/ns/mnt",
+        "--",
+        MOUNT_BIN,
+        "-t", "overlay",
+        "overlay",
+        "-o", options,
+        str(merged),
+    ]
+    exec_or_print(argv)
+
+
+def cmd_umount(name: str) -> None:
+    name = validate_name(name)
+    r = root()
+    runtime_name_dir = (r / "runtime" / name).resolve(strict=True)
+    merged = (runtime_name_dir / "merged").resolve(strict=True)
+    if merged.parent != runtime_name_dir:
+        die(f"merged resolved outside runtime/{name}: {merged}")
+    argv = [
+        NSENTER,
+        "--mount=/proc/1/ns/mnt",
+        "--",
+        UMOUNT_BIN,
+        str(merged),
+    ]
+    exec_or_print(argv)
+
+
+def main(argv: list[str]) -> None:
+    if len(argv) != 3 or argv[1] not in ("mount", "umount"):
+        sys.stderr.write("usage: left4me-overlay mount|umount <name>\n")
+        sys.exit(2)
+    if argv[1] == "mount":
+        cmd_mount(argv[2])
+    else:
+        cmd_umount(argv[2])
+
+
+if __name__ == "__main__":
+    main(sys.argv)

deploy/tests/test_deploy_artifacts.py

@@ -13,6 +13,7 @@ GLOBAL_REFRESH_SERVICE = DEPLOY / "files/usr/local/lib/systemd/system/left4me-re
 GLOBAL_REFRESH_TIMER = DEPLOY / "files/usr/local/lib/systemd/system/left4me-refresh-global-overlays.timer"
 SYSTEMCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-systemctl"
 JOURNALCTL_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-journalctl"
+OVERLAY_HELPER = DEPLOY / "files/usr/local/libexec/left4me/left4me-overlay"
 SUDOERS = DEPLOY / "files/etc/sudoers.d/left4me"
 HOST_ENV = DEPLOY / "templates/etc/left4me/host.env"
 WEB_ENV_TEMPLATE = DEPLOY / "templates/etc/left4me/web.env.template"
@@ -35,11 +36,18 @@ def test_web_unit_contains_required_runtime_contract():
     assert "EnvironmentFile=/etc/left4me/web.env" in unit
     assert "ExecStart=/opt/left4me/.venv/bin/gunicorn" in unit
     assert "--workers 1" in unit
+    assert "--threads 32" in unit
+    # NoNewPrivileges must remain unset because sudo (used by the overlay,
+    # systemctl and journalctl helpers) is setuid.
     assert "NoNewPrivileges=true" not in unit
-    assert "PrivateTmp=true" not in unit
+    # Restored now that fuse-overlayfs propagation is no longer the mechanism.
+    assert "PrivateTmp=true" in unit
     assert "ProtectSystem=full" in unit
     assert "ReadWritePaths=/var/lib/left4me" in unit
-    assert "MountFlags=shared" in unit
+    # Mounts now happen in PID 1's namespace via the left4me-overlay helper,
+    # so MountFlags propagation is irrelevant — and the previous assumption
+    # that MountFlags=shared made it work was incorrect.
+    assert "MountFlags=" not in unit
 
 
 def test_server_unit_contains_required_runtime_contract():
@@ -145,10 +153,60 @@ def test_sudoers_allows_only_left4me_helpers_not_raw_system_tools():
         "left4me ALL=(root) NOPASSWD: "
         "/usr/local/libexec/left4me/left4me-journalctl *"
     ) in sudoers
+    assert "/usr/local/libexec/left4me/left4me-overlay mount *" in sudoers
+    assert "/usr/local/libexec/left4me/left4me-overlay umount *" in sudoers
     assert "/bin/systemctl" not in sudoers
     assert "/usr/bin/systemctl" not in sudoers
     assert "/bin/journalctl" not in sudoers
     assert "/usr/bin/journalctl" not in sudoers
+    assert "/bin/mount" not in sudoers
+    assert "/bin/umount" not in sudoers
+
+
+def test_overlay_helper_is_python_with_strict_validation():
+    text = OVERLAY_HELPER.read_text()
+    assert text.startswith("#!/usr/bin/python3")
+    # Validation surface
+    assert "NAME_RE = re.compile" in text
+    assert "LOWERDIR_ALLOWLIST" in text
+    assert "user.fuseoverlayfs." in text
+    assert "MAX_LOWERDIRS = 500" in text
+    # Mounts via PID 1's mount namespace
+    assert "/proc/1/ns/mnt" in text
+    assert "nsenter" in text
+    # Verbs are mount and umount (not unmount)
+    assert '"mount"' in text and '"umount"' in text
+    assert '"unmount"' not in text
+
+
+def test_deploy_script_installs_overlay_helper_with_executable_mode():
+    script = DEPLOY_SCRIPT.read_text()
+    assert "/usr/local/libexec/left4me/left4me-overlay" in script
+    assert "chmod 0755" in script and "left4me-overlay" in script
+
+
+def test_deploy_script_does_not_install_fuse_overlayfs_apt_dep():
+    # fuse-overlayfs / fuse3 were the previous mount engine; kernel overlayfs
+    # replaces them. Comments in the migration block may legitimately mention
+    # the names, so scope this to the actual apt-get / dnf install lines.
+    install_lines = [
+        line for line in DEPLOY_SCRIPT.read_text().splitlines()
+        if ("apt-get install" in line or "dnf install" in line)
+    ]
+    assert install_lines, "expected at least one apt/dnf install line"
+    for line in install_lines:
+        assert "fuse-overlayfs" not in line, line
+        assert "fuse3" not in line, line
+
+
+def test_deploy_script_runs_one_shot_kernel_overlay_migration():
+    script = DEPLOY_SCRIPT.read_text()
+    assert "/var/lib/left4me/.kernel-overlay-migrated" in script
+    # Migration should stop services + force-unmount stale mounts + wipe upper/work
+    assert "systemctl stop 'left4me-server@" in script
+    assert "systemctl stop left4me-web.service" in script
+    assert "findmnt -t overlay" in script
+    assert "/runtime/" in script and "rm -rf" in script and 'upper"' in script and 'work"' in script
 
 
 def test_env_templates_contain_required_defaults():

docs/superpowers/plans/2026-05-08-kernel-overlayfs-helper.md

@@ -0,0 +1,229 @@
+# Kernel Overlayfs Helper Implementation Plan
+
+> **Approval status:** User-approved 2026-05-08. Implementation proceeds.
+
+**Goal:** Implement the kernel-overlayfs migration per `docs/superpowers/specs/2026-05-08-kernel-overlayfs-helper-design.md`. Add a Python `left4me-overlay` privileged helper, a `KernelOverlayFSMounter` Python class, wire the existing `OverlayMounter` ABC through `l4d2host/instances.py`, drop `fuse-overlayfs` from the deploy stack, and migrate existing on-disk upper/work directories.
+
+**Architecture:** The web app continues to call `l4d2ctl start|stop|delete <name>`; `l4d2host` continues to expose the same CLI verbs. Internally, `start_instance`/`stop_instance`/`delete_instance` move from a hardcoded subprocess call to `fuse-overlayfs`/`fusermount3` to using `KernelOverlayFSMounter`, which invokes the new sudo helper that mounts in PID 1's namespace via `nsenter`.
+
+---
+
+## Locked Decisions
+
+See `docs/superpowers/specs/2026-05-08-kernel-overlayfs-helper-design.md` for the design rationale. Implementation-relevant summary:
+
+- `left4me-overlay` Python helper in `/usr/local/libexec/left4me/`, owned root, mode 0755, system `/usr/bin/python3`, stdlib only.
+- Verbs: `mount <name>`, `umount <name>`.
+- Validation in helper: name regex; realpath + allowlist for each lowerdir; exact-prefix check for upper/work/merged; reject upperdir with `user.fuseoverlayfs.*` xattrs; lowerdir count ≤ 500.
+- Sudoers verb-constrained: `mount *`, `umount *`.
+- `KernelOverlayFSMounter` in `l4d2host/fs/kernel_overlayfs.py` — implements `OverlayMounter`. Derives `name` from the merged path's parent.
+- `start_instance` adds `os.path.ismount(merged)` guard before mounting.
+- Deploy migration: gated on sentinel file `/var/lib/left4me/.kernel-overlay-migrated`; stops gameservers + web, force-unmounts stale mounts, wipes upper/work, recreates empty.
+- Web unit cleanup: drop `MountFlags=shared`, restore `PrivateTmp=true`, rewrite comment block. Keep `NoNewPrivileges` unset.
+- Delete `l4d2host/fs/fuse_overlayfs.py` (currently unused — `start_instance` bypasses it).
+- AGENTS.md contracts unchanged.
+
+---
+
+## Current Gap
+
+- `l4d2host/instances.py` `start_instance` calls `fuse-overlayfs` directly (lines 85-101); `stop_instance`/`delete_instance` call `fusermount3 -u` directly. The `OverlayMounter` ABC at `l4d2host/fs/base.py` and the `FuseOverlayFSMounter` impl at `l4d2host/fs/fuse_overlayfs.py` exist but are unused.
+- Mounts land in the web service's private mount namespace, invisible to host and to gameserver units. `MountFlags=shared` does not fix it.
+- No privileged mount helper exists; only `left4me-systemctl` and `left4me-journalctl`.
+- Deploy script installs `fuse-overlayfs` apt package and assumes it as a runtime tool.
+- Existing `runtime/<name>/upper` directories may carry `user.fuseoverlayfs.*` xattrs that kernel overlayfs would silently ignore (resurrecting "deleted" files).
+
+---
+
+## Task 1: Helper Script + Sudoers + Mounter Class (RED-first)
+
+**Files:**
+- Create: `deploy/files/usr/local/libexec/left4me/left4me-overlay` (Python, mode 0755 after deploy)
+- Modify: `deploy/files/etc/sudoers.d/left4me`
+- Create: `l4d2host/fs/kernel_overlayfs.py`
+- Create: `l4d2host/tests/test_kernel_overlayfs.py`
+- Create: `l4d2host/tests/test_overlay_helper.py`
+- Modify: `deploy/tests/test_deploy_artifacts.py` (assert helper deployed + sudoers entry)
+
+Test plan (RED first):
+
+1. `test_kernel_overlayfs.py::test_mount_invokes_helper_with_name` — mock `run_command`, call `KernelOverlayFSMounter().mount(lowerdirs="/x:/y", upperdir=Path("/var/lib/left4me/runtime/alpha/upper"), workdir=Path("/var/lib/left4me/runtime/alpha/work"), merged=Path("/var/lib/left4me/runtime/alpha/merged"))`, assert argv `["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", "mount", "alpha"]`.
+2. `test_kernel_overlayfs.py::test_unmount_invokes_helper_with_umount_verb` — mock + call + assert argv with `umount`.
+3. `test_overlay_helper.py` — drives the helper script as a subprocess with `LEFT4ME_OVERLAY_PRINT_ONLY=1` env var (helper prints the would-be `nsenter …` command line and exits 0 instead of execve), and with isolated `LEFT4ME_ROOT=tmp_path`. Cases:
+   - Valid mount: prints expected `nsenter --mount=/proc/1/ns/mnt -- /bin/mount -t overlay …` line.
+   - Valid umount: prints expected umount line.
+   - Bad name (`../escape`, uppercase, empty): exit non-zero, stderr matches.
+   - Lowerdir traversal (`/etc`, `/var/lib/left4me/../etc`, symlink escape): exit non-zero.
+   - Missing `instance.env`: exit non-zero.
+   - Tainted upperdir (with `user.fuseoverlayfs.opaque` xattr): exit non-zero with clear message. (Optional: skip if `setfattr` is unavailable on dev machine; keep test on Linux only via `pytest.mark.skipif`.)
+   - Lowerdir count > 500: exit non-zero.
+4. `test_deploy_artifacts.py` — assert `/usr/local/libexec/left4me/left4me-overlay` is present in deployed files; sudoers includes the new lines.
+
+Implementation:
+
+- Helper script structure: `argparse` for the verb, then path-validation funcs, then `os.execv("/usr/bin/nsenter", [...])` (or printing it under `LEFT4ME_OVERLAY_PRINT_ONLY`).
+- `KernelOverlayFSMounter`: `name = merged.parent.name` (with a one-line comment), then `run_command(["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", verb, name], on_stdout=…, on_stderr=…, passthrough=…, should_cancel=…)`.
+
+**Verification:**
+
+```
+python3 -m pytest l4d2host/tests/test_kernel_overlayfs.py l4d2host/tests/test_overlay_helper.py deploy/tests/test_deploy_artifacts.py -q
+```
+
+Expected before implementation: FAIL on missing class/script. After: all green.
+
+**Commit:** `feat(l4d2-host): KernelOverlayFSMounter + left4me-overlay helper`
+
+---
+
+## Task 2: Wire OverlayMounter Through Lifecycle + Drop Fuse Module
+
+**Files:**
+- Modify: `l4d2host/instances.py` (start/stop/delete)
+- Modify: `l4d2host/tests/test_lifecycle.py` (update argv assertions, add double-mount guard test)
+- Delete: `l4d2host/fs/fuse_overlayfs.py`
+- Verify: `l4d2host/fs/__init__.py` does not re-export `FuseOverlayFSMounter`
+
+Test plan (update RED, then GREEN):
+
+1. `test_lifecycle.py::test_start_order` — change assertion: `calls[0]` is now `["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", "mount", "alpha"]`. Adjust setup so the test still creates the merged directory.
+2. `test_lifecycle.py::test_stop_succeeds_when_unmount_fails` — `cmd[0:5] == ["sudo", "-n", "/usr/local/libexec/left4me/left4me-overlay", "umount", "alpha"]`.
+3. `test_lifecycle.py::test_delete_succeeds_when_unmount_fails` — same.
+4. NEW `test_lifecycle.py::test_start_refuses_double_mount` — monkeypatch `os.path.ismount` to return True; expect `start_instance` to raise `subprocess.CalledProcessError`; assert NO mount command was issued.
+5. `test_lifecycle.py::test_lifecycle_rejects_unsafe_instance_names` — unchanged.
+6. `test_lifecycle.py::test_delete_missing_is_noop` — unchanged.
+
+Implementation:
+
+- `instances.py` imports `KernelOverlayFSMounter`. Module-level singleton instance (`_mounter = KernelOverlayFSMounter()`). Replace direct `run_command([...fuse-overlayfs...])` with `_mounter.mount(...)`. Replace direct `run_command([...fusermount3...])` with `_mounter.unmount(...)` (still inside the existing try/except for stop/delete).
+- Add the ismount guard at the top of `start_instance` after `runtime_dir` is computed, before `emit_step("mounting runtime overlay...")`. Raise `subprocess.CalledProcessError(returncode=1, cmd=["mount-guard"], stderr="runtime overlay already mounted at <path>; refusing to double-mount")`.
+- Delete `l4d2host/fs/fuse_overlayfs.py`.
+- Confirm `l4d2host/fs/__init__.py` is empty (already verified to be 1 line).
+
+**Verification:**
+
+```
+python3 -m pytest l4d2host/tests -q
+python3 -m pytest l4d2web/tests -q
+```
+
+Both green. Web tests: the `"Step: mounting runtime overlay..."` log line is preserved in `start_instance`.
+
+**Commit:** `refactor(l4d2-host): start/stop/delete go through OverlayMounter; drop FuseOverlayFSMounter`
+
+---
+
+## Task 3: Deploy Script Migration (Apt Deps + Wipe Upper/Work)
+
+**Files:**
+- Modify: `deploy/deploy-test-server.sh`
+- Modify: `deploy/tests/test_deploy_artifacts.py` (assert deploy script contains migration lines; assert `fuse-overlayfs` no longer in apt-get install)
+
+Test plan:
+
+1. `test_deploy_artifacts.py::test_deploy_script_drops_fuse_overlayfs_apt_dep` — `assert "fuse-overlayfs" not in deploy_script` and `assert "kernel-overlay-migrated" in deploy_script`.
+2. `test_deploy_artifacts.py::test_deploy_script_migration_block_uses_sentinel` — `assert ".kernel-overlay-migrated" in deploy_script`.
+
+Implementation:
+
+In `deploy/deploy-test-server.sh`, drop `fuse-overlayfs` from the apt-get and dnf lines (lines 82, 84). Insert before the existing `systemctl restart left4me-web.service` (line 182):
+
+```sh
+# One-time migration: fuse-overlayfs upperdir → kernel overlayfs upperdir.
+# fuse-overlayfs running as the left4me user uses user.fuseoverlayfs.* xattrs
+# for whiteouts and opaque dirs; kernel overlayfs ignores those, so any
+# pre-existing upper/ from the fuse era would resurrect "deleted" files.
+sentinel=/var/lib/left4me/.kernel-overlay-migrated
+if [ ! -e "$sentinel" ]; then
+    $sudo_cmd systemctl stop 'left4me-server@*.service' 2>/dev/null || true
+    $sudo_cmd systemctl stop left4me-web.service 2>/dev/null || true
+    $sudo_cmd sh -c 'findmnt -t fuse.fuse-overlayfs -o TARGET --noheadings | xargs -r -n1 fusermount3 -u 2>/dev/null || true'
+    $sudo_cmd sh -c "findmnt -t overlay -o TARGET --noheadings | grep '/var/lib/left4me/runtime/' | xargs -r -n1 umount 2>/dev/null || true"
+    $sudo_cmd sh -c 'for d in /var/lib/left4me/runtime/*/; do [ -d "$d" ] || continue; rm -rf "$d/upper" "$d/work"; mkdir -p "$d/upper" "$d/work"; chown left4me:left4me "$d/upper" "$d/work"; done'
+    $sudo_cmd touch "$sentinel"
+    $sudo_cmd chown left4me:left4me "$sentinel"
+fi
+```
+
+**Verification:**
+
+```
+python3 -m pytest deploy/tests -q
+```
+
+Green.
+
+**Commit:** `chore(deploy): drop fuse-overlayfs apt dep + one-shot migrate upper/work`
+
+---
+
+## Task 4: Web Unit Hardening Cleanup + Docs
+
+**Files:**
+- Modify: `deploy/files/usr/local/lib/systemd/system/left4me-web.service`
+- Modify: `deploy/tests/test_deploy_artifacts.py`
+- Modify: `README.md`
+- Modify: `l4d2host/README.md`
+- Modify: `deploy/README.md`
+
+Test plan:
+
+1. `test_deploy_artifacts.py::test_web_unit_contains_required_runtime_contract` — drop `assert "MountFlags=shared" in unit` (or rather: replace with `assert "MountFlags=" not in unit`); add `assert "PrivateTmp=true" in unit`; add `assert "left4me-overlay" not in unit` (just to be precise — the unit shouldn't reference the helper directly, only via Python code).
+
+Implementation:
+
+Edit `left4me-web.service`:
+
+- Drop `MountFlags=shared`.
+- Restore `PrivateTmp=true`.
+- Rewrite the comment block above hardening lines to explain: mounts now go through the `left4me-overlay` helper which `nsenter`s into PID 1's mount namespace, so this unit's namespace is irrelevant to gameserver visibility. `NoNewPrivileges` stays unset because sudo is setuid.
+
+README updates:
+
+- `README.md` (line ~59): drop fuse-overlayfs from tech-stack list; replace with "kernel overlayfs via privileged helper".
+- `l4d2host/README.md`: lines 29, 52, 64 reference fuse — update to "kernel overlayfs (mount via the `left4me-overlay` helper deployed to `/usr/local/libexec/left4me/`)".
+- `deploy/README.md`: add `/usr/local/libexec/left4me/left4me-overlay` to the privileged-helpers inventory.
+
+**Verification:**
+
+```
+python3 -m pytest deploy/tests -q
+```
+
+Green. Manual readthrough of the three READMEs confirms no stale fuse references.
+
+**Commit:** `chore(deploy): cleanup left4me-web hardening + docs for kernel overlayfs`
+
+---
+
+## Task 5: End-to-End Verification on `ckn@10.0.4.128`
+
+**Pre-deploy:** branch is clean, all four prior commits land, all tests green locally.
+
+**Deploy:**
+
+```
+deploy/deploy-test-server.sh ckn@10.0.4.128
+```
+
+**Verification commands on the box:**
+
+1. `test -e /var/lib/left4me/.kernel-overlay-migrated && echo migrated` — sentinel created.
+2. `systemctl status left4me-web.service --no-pager` — `active (running)`, recent invocation timestamp.
+3. From the UI or via `sudo -u left4me /opt/left4me/.venv/bin/l4d2ctl start test-server` — exit 0.
+4. `findmnt /var/lib/left4me/runtime/test-server/merged` — shows fstype `overlay` in the host namespace.
+5. `systemctl status left4me-server@test-server --no-pager` — `active (running)` after the start; **not** in `activating (auto-restart)`. No `status=200/CHDIR` errors in `journalctl -u left4me-server@test-server`.
+6. `sudo journalctl -k --since "5 minutes ago" | grep -i apparmor | tail` — no overlay-related denials.
+7. Negative test: `sudo -u left4me sudo -n /usr/local/libexec/left4me/left4me-overlay mount '../escape'` — exits non-zero with validation error.
+8. Idempotency: `l4d2ctl stop test-server && l4d2ctl stop test-server` — both succeed (per the prior `fix(l4d2-host): make stop_instance idempotent` commit, still holds).
+9. Re-start: `l4d2ctl start test-server` — succeeds, `findmnt` shows the mount again.
+10. Double-mount guard: while the server is running, attempting another start (not via UI; via Python REPL or a second job) — `start_instance` raises `CalledProcessError` with the "refusing to double-mount" message. Optional, can be left to the unit test.
+
+**On failure of any step:** stop and report. Do NOT push. The deploy script is rerunnable; the migration sentinel stays so wipe doesn't repeat.
+
+---
+
+## Out Of Scope
+
+- See spec's "Out Of Scope" section.
+- This plan does not push commits; pushing is a separate user decision after end-to-end verification passes.

docs/superpowers/specs/2026-05-08-kernel-overlayfs-helper-design.md

@@ -0,0 +1,80 @@
+# Kernel Overlayfs Helper Design
+
+**Goal:** Replace the per-instance `fuse-overlayfs` mount with kernel-native overlayfs invoked through a privileged sudo helper that mounts in PID 1's mount namespace. Restores host-namespace visibility of the merged overlay so gameserver units (`left4me-server@%i.service`) can `chdir` into it at unshare time.
+
+**Approval status:** User-approved design direction. Implementation proceeds in lockstep with the companion plan at `docs/superpowers/plans/2026-05-08-kernel-overlayfs-helper.md`.
+
+## Context
+
+**Symptom.** After redeploys, starting a gameserver leaves the systemd unit in `activating (auto-restart)` with `status=200/CHDIR — Changing to the requested working directory failed: No such file or directory`. Investigation showed:
+
+- `fuse-overlayfs` running as `left4me` user mounts in `left4me-web.service`'s mount namespace.
+- `ProtectSystem=full` + `ReadWritePaths=/var/lib/left4me` forces `PrivateMounts=yes` on the unit (`systemd-analyze security` confirms).
+- The unit's bind of `/var/lib/left4me` shows `shared:471 master:1` in `/proc/<pid>/mountinfo` — slave-receive-only — so mounts created beneath it never propagate back to host.
+- `MountFlags=shared` (added in commit `1968684` to fix this) sets only the unit's *root* propagation; it does not override the slave-direction propagation that `ProtectSystem`/`ReadWritePaths` apply to their bind mounts. The gameserver unit, on unshare, inherits *host* mounts and sees nothing at the merged path → CHDIR fails.
+
+The system *appeared* to work for ~1d8h before this investigation because the prior fuse daemon happened to land in the host namespace via some transient state. The mechanism documented in `1968684` does not reliably work on systemd 257 with this hardening shape.
+
+**Out-of-scope item now in scope.** The 2026-05-07 workshop-overlays spec already lists this transition at line 211: *"Switch from fuse-overlayfs to kernel overlayfs via a privileged helper. Matches the existing systemd / steam-install sudoers helper pattern under `/usr/local/libexec/left4me/`."* The mount-propagation bug is the trigger to do it now.
+
+## Locked Decisions
+
+1. **Privileged helper does the mount.** New `left4me-overlay` script under `/usr/local/libexec/left4me/`, invoked via `sudo -n`. Mirrors the existing `left4me-systemctl` and `left4me-journalctl` pattern. The helper enters PID 1's mount namespace via `nsenter --mount=/proc/1/ns/mnt` and then calls `/bin/mount -t overlay …` or `/bin/umount`. Result: all overlay mounts live in the host namespace, visible to gameserver units.
+2. **Kernel-native overlayfs, not fuse.** Once a privileged helper exists, fuse-overlayfs's rootless-mount-via-setuid-`fusermount3` advantage disappears. Kernel overlayfs is faster, has no long-running daemon, simpler unmount, and one fewer runtime dep.
+3. **Helper is Python, not shell.** Path canonicalization, env-file parsing, and lowerdir prefix-allowlist validation are too brittle in shell. Uses system `/usr/bin/python3` (never the venv) and stdlib only. Owned by root, mode 0755.
+4. **Verbs are `mount` and `umount`.** Matches the kernel/userspace utility names; reduces cognitive friction over `unmount`.
+5. **Helper takes only the instance name as input.** It reads `${LEFT4ME_ROOT:-/var/lib/left4me}/instances/<name>/instance.env` for `L4D2_LOWERDIRS=` and computes `upper`/`work`/`merged` from the runtime root. Equivalent in security to taking lowerdirs as args (the user already controls instance.env), and produces a one-line audit trail in `journalctl _COMM=sudo`.
+6. **Strict path validation in the helper.**
+   - Instance name matches `^[a-z0-9][a-z0-9_-]{0,63}$` (mirrors `validate_instance_name` in `l4d2host/paths.py`).
+   - Each lowerdir from `L4D2_LOWERDIRS` is `os.path.realpath`'d and must resolve under one of an allowlist: `installation/`, `overlays/`, `global_overlay_cache/`, `workshop_cache/`. Empty entries and traversals are rejected.
+   - `upper`/`work`/`merged` must resolve exactly to `runtime/<name>/{upper,work,merged}`.
+   - Lowerdir count ≤ 500 (kernel overlayfs hard cap; was 64 before kernel 5.2).
+7. **Whiteout-format guard.** `fuse-overlayfs` running as non-root uses `user.fuseoverlayfs.*` xattrs for whiteouts and opaque dirs, which kernel overlayfs ignores entirely. Before mounting, the helper walks `upperdir` once and refuses if any such xattr is present. Defensive; catches a stale fuse-era upperdir that wasn't wiped during migration.
+8. **One-time migration: wipe existing `upper/` and `work/`.** Deploy script runs a gated migration (sentinel file `/var/lib/left4me/.kernel-overlay-migrated`) that stops gameservers, stops web service, unmounts any stale fuse/overlay mounts, recreates empty `upper`/`work` dirs for every instance. Players' in-place edits to merged content are sacrificed; v1 accepts this for a test deployment.
+9. **Sudoers verb constraints.** `left4me ALL=(root) NOPASSWD: /usr/local/libexec/left4me/left4me-overlay mount *, /usr/local/libexec/left4me/left4me-overlay umount *`. Defense in depth (real validation lives in the helper); makes `sudo -l` output self-documenting.
+10. **Wire the existing `OverlayMounter` ABC through.** `start_instance`/`stop_instance`/`delete_instance` today bypass the abstraction at `l4d2host/fs/base.py`. The new `KernelOverlayFSMounter` replaces the unused `FuseOverlayFSMounter` AND becomes the only path through `instances.py`. `FuseOverlayFSMounter` and the `fuse_overlayfs.py` module are deleted.
+11. **Double-mount guard in `start_instance`.** Kernel mounts persist when the web worker dies (unlike fuse daemons, which die with their cgroup). `start_instance` checks `os.path.ismount(merged)` and refuses with a clear error rather than double-mounting.
+12. **Hardening cleanup on `left4me-web.service`.** Drop `MountFlags=shared` (no longer the mechanism). Restore `PrivateTmp=true` (was dropped in commit `593611e` for fuse propagation that did not work). Keep `NoNewPrivileges` unset (sudo still requires setuid). Update the comment block to reflect the new model.
+13. **AGENTS.md contracts unchanged.** The host library's CLI surface (`install`, `initialize`, `start`, `stop`, `delete`, `status`, `logs`) is unchanged. The web app continues to drive operations via `l4d2ctl`. The fuse-overlayfs implementation detail was never part of the public contract.
+
+## Architecture
+
+```
+left4me-web.service  (hardened, private mount namespace)
+    │
+    │  start_instance(name=…)
+    ▼
+l4d2host.instances.start_instance
+    │
+    │  KernelOverlayFSMounter().mount(merged=…)
+    ▼
+sudo -n /usr/local/libexec/left4me/left4me-overlay mount <name>
+    │   • validate name (regex)
+    │   • parse instance.env → L4D2_LOWERDIRS
+    │   • realpath each lowerdir, prefix-allowlist check
+    │   • compute upper/work/merged under runtime/<name>/
+    │   • walk upperdir, refuse if any user.fuseoverlayfs.* xattr
+    ▼
+nsenter --mount=/proc/1/ns/mnt -- \
+    /bin/mount -t overlay overlay \
+        -o "lowerdir=…,upperdir=…,workdir=…" \
+        /var/lib/left4me/runtime/<name>/merged
+    │
+    ▼
+host mount namespace now has the overlay; gameserver unit, on
+unshare, inherits it and CHDIRs into …/merged/left4dead2 successfully.
+```
+
+## Operational Notes
+
+- **Migration ordering on the test box (test-server, …).** The deploy script must, in order: (1) stop all `left4me-server@*.service`, (2) stop `left4me-web.service` (kills any lingering fuse-overlayfs daemons by reaping their cgroup), (3) `findmnt` + force-unmount any leftover fuse/overlay mounts under `/var/lib/left4me/runtime/`, (4) wipe and recreate `upper`/`work` for every instance, (5) deploy + start the new code. The sentinel file `/var/lib/left4me/.kernel-overlay-migrated` gates reruns.
+- **Filesystem.** `/var/lib/left4me` is btrfs on the test box. Kernel overlayfs on btrfs is supported on kernel ≥ 5.10; the box is on 6.12 — fine. AppArmor ships enabled on Debian Trixie; verify no overlay-related denials in `journalctl -k` after first start.
+- **Concurrency.** Two threads racing on `start_instance` for the same name is a latent issue unaffected by this change. The double-mount guard partly mitigates: the loser hits the existing mount and errors cleanly.
+
+## Out Of Scope
+
+- **Replace `sudo` with `AmbientCapabilities=CAP_SYS_ADMIN`** on a dedicated helper unit. Broader blast radius than the wrapper-script approach.
+- **A `systemd-mount` per-instance mount unit.** Considered as the alternative architectural fix but adds more moving parts than the helper-script approach. The helper matches the established privileged-helper pattern in this codebase.
+- **Re-enable `NoNewPrivileges` on `left4me-web.service`.** Requires removing sudo; not feasible while the helper invocation pattern stays.
+- **Multi-process job-worker-claim safety.** The `_claim_lock` in `l4d2host/services/job_worker.py:131-138` is process-local; correctness depends on `--workers 1`. This change doesn't touch it.
+- **Replicating the migration on production deployments.** v1 covers only the test-server deployment shape.

l4d2host/README.md

@@ -49,7 +49,7 @@ Validated on Debian 13 during the `ckn@10.0.4.128` smoke test:
 - Python 3.12+ with virtualenv/pip tooling for installing `l4d2host`.
 - `steamcmd` available on `PATH` and able to self-update as the runtime user.
 - 32-bit compatibility libraries for SteamCMD on amd64 Debian: `libc6-i386`, `lib32gcc-s1`, `lib32stdc++6`.
-- `fuse-overlayfs` and `fusermount3` for per-instance overlay mounts.
+- Kernel overlayfs (`mount -t overlay`); mount/umount go through the `left4me-overlay` privileged helper, which `nsenter`s into PID 1's mount namespace.
 - `systemctl --user` and `journalctl --user` available for the runtime user.
 - User lingering enabled when services must survive SSH sessions: `sudo loginctl enable-linger <user>`.
 - `/var/lib/left4me` created and writable by the runtime user, unless `LEFT4ME_ROOT` is set to another deployment-managed root.
@@ -61,7 +61,7 @@ sudo apt-get update
 sudo apt-get install -y \
   python3 python3-venv python3-pip \
   curl ca-certificates tar gzip \
-  fuse-overlayfs fuse3 \
+  util-linux \
   libc6-i386 lib32gcc-s1 lib32stdc++6
 
 sudo mkdir -p /opt/steamcmd /var/lib/left4me/{installation,overlays,instances,runtime,tmp}

l4d2host/fs/kernel_overlayfs.py

@@ -5,7 +5,15 @@ from l4d2host.fs.base import OverlayMounter
 from l4d2host.process import run_command
 
 
-class FuseOverlayFSMounter(OverlayMounter):
+HELPER_PATH = "/usr/local/libexec/left4me/left4me-overlay"
+
+
+class KernelOverlayFSMounter(OverlayMounter):
+    # Delegates the actual mount/umount syscalls to the privileged
+    # left4me-overlay helper. The helper takes only the instance name and
+    # rederives lowerdirs/upper/work/merged from disk; the OverlayMounter
+    # ABC accepts those args for compatibility, so we extract the name
+    # from the merged path's parent directory.
     def mount(
         self,
         *,
@@ -18,13 +26,9 @@ class FuseOverlayFSMounter(OverlayMounter):
         passthrough: bool = False,
         should_cancel: Callable[[], bool] | None = None,
     ) -> None:
+        del lowerdirs, upperdir, workdir
         run_command(
-            [
-                "fuse-overlayfs",
-                "-o",
-                f"lowerdir={lowerdirs},upperdir={upperdir},workdir={workdir}",
-                str(merged),
-            ],
+            ["sudo", "-n", HELPER_PATH, "mount", merged.parent.name],
             on_stdout=on_stdout,
             on_stderr=on_stderr,
             passthrough=passthrough,
@@ -41,7 +45,7 @@ class FuseOverlayFSMounter(OverlayMounter):
         should_cancel: Callable[[], bool] | None = None,
     ) -> None:
         run_command(
-            ["fusermount3", "-u", str(merged)],
+            ["sudo", "-n", HELPER_PATH, "umount", merged.parent.name],
             on_stdout=on_stdout,
             on_stderr=on_stderr,
             passthrough=passthrough,

l4d2host/instances.py

@@ -1,10 +1,11 @@
+import os
 from pathlib import Path
 import shutil
 import subprocess
 from typing import Callable
 
+from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
 from l4d2host.paths import DEFAULT_LEFT4ME_ROOT, get_left4me_root, overlay_path, validate_instance_name
-from l4d2host.process import run_command
 from l4d2host.service_control import start_service, stop_service
 from l4d2host.spec import load_spec
 
@@ -12,6 +13,9 @@ from l4d2host.spec import load_spec
 from l4d2host.logging import emit_step
 
 
+_mounter = KernelOverlayFSMounter()
+
+
 DEFAULT_ROOT = DEFAULT_LEFT4ME_ROOT
 
 
@@ -82,18 +86,23 @@ def start_instance(
 
     env = _load_instance_env(instance_dir / "instance.env")
 
+    merged = runtime_dir / "merged"
+    if os.path.ismount(merged):
+        # Kernel overlayfs mounts persist when the web worker dies (unlike
+        # fuse daemons, which were reaped with their cgroup). Refuse rather
+        # than double-mount.
+        raise subprocess.CalledProcessError(
+            returncode=1,
+            cmd=["start_instance"],
+            stderr=f"runtime overlay already mounted at {merged}; refusing to double-mount",
+        )
+
     emit_step("mounting runtime overlay...", on_stdout, passthrough)
-    run_command(
-        [
-            "fuse-overlayfs",
-            "-o",
-            (
-                f"lowerdir={env['L4D2_LOWERDIRS']},"
-                f"upperdir={runtime_dir / 'upper'},"
-                f"workdir={runtime_dir / 'work'}"
-            ),
-            str(runtime_dir / "merged"),
-        ],
+    _mounter.mount(
+        lowerdirs=env["L4D2_LOWERDIRS"],
+        upperdir=runtime_dir / "upper",
+        workdir=runtime_dir / "work",
+        merged=merged,
         on_stdout=on_stdout,
         on_stderr=on_stderr,
         passthrough=passthrough,
@@ -135,14 +144,17 @@ def stop_instance(
         passthrough=passthrough,
         should_cancel=should_cancel,
     )
-    emit_step("unmounting runtime overlay...", on_stdout, passthrough)
-    run_command(
-        ["fusermount3", "-u", str(root / "runtime" / name / "merged")],
-        on_stdout=on_stdout,
-        on_stderr=on_stderr,
-        passthrough=passthrough,
-        should_cancel=should_cancel,
-    )
+    emit_step("unmounting runtime overlay (if mounted)...", on_stdout, passthrough)
+    try:
+        _mounter.unmount(
+            merged=root / "runtime" / name / "merged",
+            on_stdout=on_stdout,
+            on_stderr=on_stderr,
+            passthrough=passthrough,
+            should_cancel=should_cancel,
+        )
+    except subprocess.CalledProcessError:
+        pass
     emit_step("stop complete.", on_stdout, passthrough)
 
 
@@ -177,8 +189,8 @@ def delete_instance(
 
     emit_step("unmounting runtime overlay (if mounted)...", on_stdout, passthrough)
     try:
-        run_command(
-            ["fusermount3", "-u", str(runtime_dir / "merged")],
+        _mounter.unmount(
+            merged=runtime_dir / "merged",
             on_stdout=on_stdout,
             on_stderr=on_stderr,
             passthrough=passthrough,

l4d2host/tests/test_kernel_overlayfs.py

@@ -0,0 +1,76 @@
+from pathlib import Path
+
+import pytest
+
+
+HELPER_PATH = "/usr/local/libexec/left4me/left4me-overlay"
+
+
+def test_mount_invokes_helper_with_name_only(monkeypatch: pytest.MonkeyPatch) -> None:
+    from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
+
+    calls: list[list[str]] = []
+
+    def fake_run_command(cmd, **kwargs):
+        del kwargs
+        calls.append(list(cmd))
+
+    monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
+
+    KernelOverlayFSMounter().mount(
+        lowerdirs="/var/lib/left4me/installation",
+        upperdir=Path("/var/lib/left4me/runtime/alpha/upper"),
+        workdir=Path("/var/lib/left4me/runtime/alpha/work"),
+        merged=Path("/var/lib/left4me/runtime/alpha/merged"),
+    )
+
+    assert calls == [["sudo", "-n", HELPER_PATH, "mount", "alpha"]]
+
+
+def test_unmount_invokes_helper_with_umount_verb(monkeypatch: pytest.MonkeyPatch) -> None:
+    from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
+
+    calls: list[list[str]] = []
+
+    def fake_run_command(cmd, **kwargs):
+        del kwargs
+        calls.append(list(cmd))
+
+    monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
+
+    KernelOverlayFSMounter().unmount(merged=Path("/var/lib/left4me/runtime/alpha/merged"))
+
+    assert calls == [["sudo", "-n", HELPER_PATH, "umount", "alpha"]]
+
+
+def test_mount_propagates_run_command_kwargs(monkeypatch: pytest.MonkeyPatch) -> None:
+    from l4d2host.fs.kernel_overlayfs import KernelOverlayFSMounter
+
+    captured: dict = {}
+
+    def fake_run_command(cmd, **kwargs):
+        captured["cmd"] = list(cmd)
+        captured["kwargs"] = kwargs
+
+    monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
+
+    out: list[str] = []
+    err: list[str] = []
+    KernelOverlayFSMounter().mount(
+        lowerdirs="/var/lib/left4me/installation",
+        upperdir=Path("/var/lib/left4me/runtime/alpha/upper"),
+        workdir=Path("/var/lib/left4me/runtime/alpha/work"),
+        merged=Path("/var/lib/left4me/runtime/alpha/merged"),
+        on_stdout=out.append,
+        on_stderr=err.append,
+        passthrough=False,
+        should_cancel=lambda: False,
+    )
+
+    assert captured["cmd"][0:3] == ["sudo", "-n", HELPER_PATH]
+    captured["kwargs"]["on_stdout"]("hi")
+    captured["kwargs"]["on_stderr"]("oops")
+    assert out == ["hi"]
+    assert err == ["oops"]
+    assert captured["kwargs"]["passthrough"] is False
+    assert callable(captured["kwargs"]["should_cancel"])

l4d2host/tests/test_lifecycle.py

@@ -27,15 +27,51 @@ def test_start_order(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
     )
     (instance_dir / "server.cfg").write_text("sv_consistency 1")
 
-    monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command)
+    monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
     monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
 
     start_instance("alpha", root=tmp_path)
 
-    assert calls[0][0] == "fuse-overlayfs"
+    assert calls[0] == [
+        "sudo",
+        "-n",
+        "/usr/local/libexec/left4me/left4me-overlay",
+        "mount",
+        "alpha",
+    ]
     assert calls[1] == ["sudo", "-n", "/usr/local/libexec/left4me/left4me-systemctl", "start", "alpha"]
 
 
+def test_start_refuses_to_double_mount(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    calls: list[list[str]] = []
+
+    def fake_run_command(cmd, **kwargs):
+        del kwargs
+        calls.append(list(cmd))
+
+    instance_dir = tmp_path / "instances" / "alpha"
+    runtime_dir = tmp_path / "runtime" / "alpha"
+    (runtime_dir / "merged").mkdir(parents=True)
+    instance_dir.mkdir(parents=True)
+    (instance_dir / "instance.env").write_text("L4D2_PORT=27015\nL4D2_ARGS=\nL4D2_LOWERDIRS=/x\n")
+    (instance_dir / "server.cfg").write_text("")
+
+    merged = runtime_dir / "merged"
+
+    def fake_ismount(path):
+        return Path(path) == merged
+
+    monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
+    monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
+    monkeypatch.setattr("l4d2host.instances.os.path.ismount", fake_ismount)
+
+    with pytest.raises(subprocess.CalledProcessError) as exc_info:
+        start_instance("alpha", root=tmp_path)
+
+    assert "already mounted" in (exc_info.value.stderr or "")
+    assert calls == [], "no mount/start commands must be issued when refusing"
+
+
 def test_delete_missing_is_noop(tmp_path: Path) -> None:
     delete_instance("missing", root=tmp_path)
 
@@ -56,7 +92,7 @@ def test_delete_succeeds_when_stop_service_fails(tmp_path: Path, monkeypatch: py
     (tmp_path / "instances" / "alpha").mkdir(parents=True)
     (tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
 
-    monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command)
+    monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
     monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
 
     delete_instance("alpha", root=tmp_path)
@@ -86,7 +122,7 @@ def test_delete_stopped_instance_removes_dirs(tmp_path: Path, monkeypatch: pytes
     (tmp_path / "instances" / "alpha").mkdir(parents=True)
     (tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
 
-    monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command)
+    monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
     monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
 
     delete_instance("alpha", root=tmp_path)
@@ -96,27 +132,58 @@ def test_delete_stopped_instance_removes_dirs(tmp_path: Path, monkeypatch: pytes
     assert ["sudo", "-n", "/usr/local/libexec/left4me/left4me-systemctl", "stop", "alpha"] in calls
 
 
-def test_delete_succeeds_when_unmount_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
-    fusermount_calls: list[list[str]] = []
+def test_stop_succeeds_when_unmount_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    umount_calls: list[list[str]] = []
 
     def fake_run_command(cmd, **kwargs):
         del kwargs
-        if cmd and cmd[0] == "fusermount3":
-            fusermount_calls.append(list(cmd))
+        if cmd[:4] == [
+            "sudo",
+            "-n",
+            "/usr/local/libexec/left4me/left4me-overlay",
+            "umount",
+        ]:
+            umount_calls.append(list(cmd))
             raise subprocess.CalledProcessError(
                 returncode=1,
                 cmd=list(cmd),
-                stderr="fusermount3: entry for merged not found in /etc/mtab",
+                stderr="umount: /var/lib/left4me/runtime/alpha/merged: not mounted",
+            )
+
+    monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
+    monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
+
+    stop_instance("alpha", root=tmp_path)
+
+    assert umount_calls, "stop must always attempt the overlay helper (no preflight)"
+
+
+def test_delete_succeeds_when_unmount_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    umount_calls: list[list[str]] = []
+
+    def fake_run_command(cmd, **kwargs):
+        del kwargs
+        if cmd[:4] == [
+            "sudo",
+            "-n",
+            "/usr/local/libexec/left4me/left4me-overlay",
+            "umount",
+        ]:
+            umount_calls.append(list(cmd))
+            raise subprocess.CalledProcessError(
+                returncode=1,
+                cmd=list(cmd),
+                stderr="umount: /var/lib/left4me/runtime/alpha/merged: not mounted",
             )
 
     (tmp_path / "instances" / "alpha").mkdir(parents=True)
     (tmp_path / "runtime" / "alpha" / "merged").mkdir(parents=True)
 
-    monkeypatch.setattr("l4d2host.instances.run_command", fake_run_command)
+    monkeypatch.setattr("l4d2host.fs.kernel_overlayfs.run_command", fake_run_command)
     monkeypatch.setattr("l4d2host.service_control.run_command", fake_run_command)
 
     delete_instance("alpha", root=tmp_path)
 
-    assert fusermount_calls, "delete must always attempt fusermount3 -u (no preflight)"
+    assert umount_calls, "delete must always attempt the overlay helper (no preflight)"
     assert not (tmp_path / "instances" / "alpha").exists()
     assert not (tmp_path / "runtime" / "alpha").exists()

l4d2host/tests/test_overlay_helper.py

@@ -0,0 +1,168 @@
+import os
+import shlex
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+
+HELPER_SOURCE = (
+    Path(__file__).resolve().parents[2]
+    / "deploy"
+    / "files"
+    / "usr"
+    / "local"
+    / "libexec"
+    / "left4me"
+    / "left4me-overlay"
+)
+
+
+def _setup_instance(root: Path, name: str = "alpha", lowerdirs: list[str] | None = None) -> None:
+    """Create the on-disk shape the helper expects."""
+    (root / "installation").mkdir(parents=True, exist_ok=True)
+    (root / "overlays" / "workshop").mkdir(parents=True, exist_ok=True)
+    if lowerdirs is None:
+        lowerdirs = [str(root / "overlays" / "workshop"), str(root / "installation")]
+    inst_dir = root / "instances" / name
+    inst_dir.mkdir(parents=True)
+    (inst_dir / "instance.env").write_text(
+        f"L4D2_PORT=27015\nL4D2_ARGS=-tickrate 100\nL4D2_LOWERDIRS={':'.join(lowerdirs)}\n"
+    )
+    runtime = root / "runtime" / name
+    (runtime / "upper").mkdir(parents=True)
+    (runtime / "work").mkdir(parents=True)
+    (runtime / "merged").mkdir(parents=True)
+
+
+def _run(args: list[str], root: Path, extra_env: dict[str, str] | None = None) -> subprocess.CompletedProcess:
+    env = {
+        **os.environ,
+        "LEFT4ME_ROOT": str(root),
+        "LEFT4ME_OVERLAY_PRINT_ONLY": "1",
+    }
+    if extra_env:
+        env.update(extra_env)
+    return subprocess.run(
+        [sys.executable, str(HELPER_SOURCE), *args],
+        env=env,
+        capture_output=True,
+        text=True,
+    )
+
+
+def test_mount_prints_expected_nsenter_command(tmp_path: Path) -> None:
+    _setup_instance(tmp_path)
+
+    result = _run(["mount", "alpha"], tmp_path)
+
+    assert result.returncode == 0, result.stderr
+    parts = shlex.split(result.stdout.strip())
+    assert parts[0] == "/usr/bin/nsenter"
+    assert parts[1] == "--mount=/proc/1/ns/mnt"
+    assert parts[2] == "--"
+    assert parts[3] == "/bin/mount"
+    assert parts[4:6] == ["-t", "overlay"]
+    assert parts[6] == "overlay"
+    assert parts[7] == "-o"
+    options = parts[8]
+    assert f"upperdir={tmp_path}/runtime/alpha/upper" in options
+    assert f"workdir={tmp_path}/runtime/alpha/work" in options
+    assert f"lowerdir={tmp_path}/overlays/workshop:{tmp_path}/installation" in options
+    assert parts[9] == str(tmp_path / "runtime" / "alpha" / "merged")
+
+
+def test_umount_prints_expected_nsenter_command(tmp_path: Path) -> None:
+    _setup_instance(tmp_path)
+
+    result = _run(["umount", "alpha"], tmp_path)
+
+    assert result.returncode == 0, result.stderr
+    parts = shlex.split(result.stdout.strip())
+    assert parts == [
+        "/usr/bin/nsenter",
+        "--mount=/proc/1/ns/mnt",
+        "--",
+        "/bin/umount",
+        str(tmp_path / "runtime" / "alpha" / "merged"),
+    ]
+
+
+@pytest.mark.parametrize("bad_name", ["..", "../escape", "FOO", "foo bar", "foo/bar", ""])
+def test_rejects_bad_instance_name(tmp_path: Path, bad_name: str) -> None:
+    result = _run(["mount", bad_name], tmp_path)
+    assert result.returncode != 0
+    assert "invalid instance name" in result.stderr or "usage:" in result.stderr
+
+
+def test_rejects_lowerdir_outside_allowlist(tmp_path: Path) -> None:
+    _setup_instance(tmp_path, lowerdirs=["/etc"])
+    result = _run(["mount", "alpha"], tmp_path)
+    assert result.returncode != 0
+    assert "outside the permitted roots" in result.stderr
+
+
+def test_rejects_lowerdir_traversal(tmp_path: Path) -> None:
+    # An overlay subdirectory whose path uses .. to escape the overlays root.
+    _setup_instance(tmp_path, lowerdirs=[str(tmp_path / "overlays" / "..") + "/etc"])
+    result = _run(["mount", "alpha"], tmp_path)
+    assert result.returncode != 0
+    assert "outside the permitted roots" in result.stderr or "path does not exist" in result.stderr
+
+
+def test_rejects_lowerdir_symlink_escape(tmp_path: Path) -> None:
+    _setup_instance(tmp_path)
+    sneaky = tmp_path / "overlays" / "sneaky"
+    os.symlink("/etc", sneaky)
+    # rewrite instance.env to point at the symlink
+    inst_env = tmp_path / "instances" / "alpha" / "instance.env"
+    inst_env.write_text(f"L4D2_LOWERDIRS={sneaky}\n")
+    result = _run(["mount", "alpha"], tmp_path)
+    assert result.returncode != 0
+    assert "outside the permitted roots" in result.stderr
+
+
+def test_rejects_missing_instance_env(tmp_path: Path) -> None:
+    (tmp_path / "instances" / "alpha").mkdir(parents=True)
+    runtime = tmp_path / "runtime" / "alpha"
+    (runtime / "upper").mkdir(parents=True)
+    (runtime / "work").mkdir(parents=True)
+    (runtime / "merged").mkdir(parents=True)
+    result = _run(["mount", "alpha"], tmp_path)
+    assert result.returncode != 0
+    assert "instance.env not found" in result.stderr
+
+
+def test_rejects_lowerdir_count_over_cap(tmp_path: Path) -> None:
+    (tmp_path / "installation").mkdir()
+    many = [str(tmp_path / "installation")] * 501
+    _setup_instance(tmp_path, lowerdirs=many)
+    result = _run(["mount", "alpha"], tmp_path)
+    assert result.returncode != 0
+    assert "501 entries" in result.stderr
+
+
+def test_rejects_empty_lowerdir_entry(tmp_path: Path) -> None:
+    (tmp_path / "installation").mkdir()
+    _setup_instance(
+        tmp_path,
+        lowerdirs=[str(tmp_path / "installation"), "", str(tmp_path / "installation")],
+    )
+    result = _run(["mount", "alpha"], tmp_path)
+    assert result.returncode != 0
+    assert "empty entry" in result.stderr
+
+
+@pytest.mark.skipif(sys.platform != "linux", reason="user.* xattrs are Linux-only")
+def test_rejects_upperdir_with_fuseoverlayfs_xattr(tmp_path: Path) -> None:
+    _setup_instance(tmp_path)
+    tainted = tmp_path / "runtime" / "alpha" / "upper" / "deleted-thing"
+    tainted.write_bytes(b"")
+    try:
+        os.setxattr(tainted, "user.fuseoverlayfs.opaque", b"y")
+    except OSError:
+        pytest.skip("filesystem doesn't support user.* xattrs")
+    result = _run(["mount", "alpha"], tmp_path)
+    assert result.returncode != 0
+    assert "fuse-overlayfs xattr" in result.stderr

l4d2web/alembic/versions/0004_drop_legacy_external_overlay_type.py

@@ -0,0 +1,32 @@
+"""drop legacy external overlay type
+
+Revision ID: 0004_drop_legacy_external_overlay_type
+Revises: 0003_global_map_overlays
+Create Date: 2026-05-08
+"""
+from typing import Sequence, Union
+
+from alembic import op
+
+
+revision: str = "0004_drop_legacy_external_overlay_type"
+down_revision: Union[str, Sequence[str], None] = "0003_global_map_overlays"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.execute(
+        "DELETE FROM jobs "
+        "WHERE overlay_id IN (SELECT id FROM overlays WHERE type = 'external')"
+    )
+    op.execute(
+        "DELETE FROM blueprint_overlays "
+        "WHERE overlay_id IN (SELECT id FROM overlays WHERE type = 'external')"
+    )
+    op.execute("DELETE FROM overlays WHERE type = 'external'")
+
+
+def downgrade() -> None:
+    # data is gone; intentional one-way migration
+    pass

l4d2web/models.py

@@ -57,7 +57,7 @@ class Overlay(Base):
     id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
     name: Mapped[str] = mapped_column(String(128), nullable=False)
     path: Mapped[str] = mapped_column(String(512), nullable=False)
-    type: Mapped[str] = mapped_column(String(16), nullable=False, default="external")
+    type: Mapped[str] = mapped_column(String(16), nullable=False, default="workshop")
     user_id: Mapped[int | None] = mapped_column(ForeignKey("users.id"), nullable=True)
     created_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)
     updated_at: Mapped[datetime] = mapped_column(DateTime, default=now_utc, nullable=False)

l4d2web/routes/log_routes.py

@@ -28,6 +28,9 @@ def stream_server_logs(server_id: int) -> Response:
 
     def generate():
         for line in facade.stream_server_logs(server.name, lines=200, follow=True):
-            yield f"data: {line}\n\n"
+            if line == "":
+                yield ": keepalive\n\n"
+            else:
+                yield f"data: {line}\n\n"
 
     return Response(generate(), mimetype="text/event-stream")

l4d2web/routes/overlay_routes.py

@@ -29,8 +29,6 @@ def _can_edit_overlay(overlay: Overlay, user) -> bool:
         return False
     if user.admin:
         return True
-    if overlay.type == "external":
-        return False
     if overlay.type == "workshop":
         return overlay.user_id == user.id
     return False
@@ -54,18 +52,13 @@ def create_overlay() -> Response:
     assert user is not None
 
     name = request.form.get("name", "").strip()
-    overlay_type = request.form.get("type", "external").strip().lower()
+    overlay_type = request.form.get("type", "workshop").strip().lower()
     if not name:
         return Response("missing fields", status=400)
     if not is_creatable_overlay_type(overlay_type, admin=user.admin):
         return Response(f"unknown overlay type: {overlay_type}", status=400)
 
-    if overlay_type == "external":
-        if not user.admin:
-            return Response("admin only", status=403)
-        scope_user_id: int | None = None
-    else:  # workshop
-        scope_user_id = user.id
+    scope_user_id: int | None = user.id
 
     with session_scope() as db:
         if _name_already_taken(db, name, scope_user_id):

l4d2web/services/global_overlays.py

@@ -37,7 +37,7 @@ GLOBAL_OVERLAYS = (
 
 MANAGED_GLOBAL_OVERLAY_TYPES = {overlay.overlay_type for overlay in GLOBAL_OVERLAYS}
 USER_CREATABLE_TYPES = {"workshop"}
-ADMIN_CREATABLE_TYPES = {"external", "workshop"}
+ADMIN_CREATABLE_TYPES = {"workshop"}
 
 
 def is_creatable_overlay_type(overlay_type: str, *, admin: bool) -> bool:

l4d2web/services/host_commands.py

@@ -1,5 +1,6 @@
 from dataclasses import dataclass
 import os
+import select
 import signal
 import subprocess
 import sys
@@ -143,19 +144,37 @@ def run_command(
     return result
 
 
-def stream_command(cmd: Sequence[str]) -> Iterator[str]:
+def stream_command(cmd: Sequence[str], *, heartbeat_interval: float = 15.0) -> Iterator[str]:
+    # An empty string yielded between real lines is a heartbeat tick: it lets
+    # SSE callers emit a keepalive frame so a closed peer is detected, instead
+    # of blocking forever inside readline() when the child is silent.
     proc = subprocess.Popen(
         list(cmd),
         stdout=subprocess.PIPE,
         stderr=subprocess.PIPE,
-        text=True,
-        bufsize=1,
+        bufsize=0,
     )
     try:
         if proc.stdout is None:
             return
-        for raw in iter(proc.stdout.readline, ""):
-            yield raw.rstrip("\n")
+        fd = proc.stdout.fileno()
+        buffer = b""
+        while True:
+            ready, _, _ = select.select([fd], [], [], heartbeat_interval)
+            if not ready:
+                if proc.poll() is not None:
+                    break
+                yield ""
+                continue
+            chunk = os.read(fd, 4096)
+            if not chunk:
+                break
+            buffer += chunk
+            while b"\n" in buffer:
+                line, buffer = buffer.split(b"\n", 1)
+                yield line.decode("utf-8", errors="replace")
+        if buffer:
+            yield buffer.decode("utf-8", errors="replace")
     finally:
         if proc.poll() is None:
             proc.terminate()

l4d2web/services/overlay_builders.py

@@ -40,23 +40,6 @@ def _overlay_root(overlay: Overlay) -> Path:
     return get_left4me_root() / "overlays" / overlay.path
 
 
-class ExternalBuilder:
-    """No-op builder for admin-managed overlays. Ensures the overlay directory
-    exists; everything inside it is the admin's responsibility (SFTP, etc.)."""
-
-    def build(
-        self,
-        overlay: Overlay,
-        *,
-        on_stdout: LogSink,
-        on_stderr: LogSink,
-        should_cancel: CancelCheck,
-    ) -> None:
-        root = _overlay_root(overlay)
-        root.mkdir(parents=True, exist_ok=True)
-        on_stdout(f"external overlay {overlay.name!r} ready at {root}")
-
-
 class WorkshopBuilder:
     """Diff-apply symlinks under `left4dead2/addons/` against the overlay's
     current `WorkshopItem` associations. Cached items get an absolute symlink
@@ -280,7 +263,6 @@ def _is_under(path: Path, root: Path) -> bool:
 
 
 BUILDERS: dict[str, OverlayBuilder] = {
-    "external": ExternalBuilder(),
     "workshop": WorkshopBuilder(),
     "l4d2center_maps": GlobalMapOverlayBuilder(),
     "cedapug_maps": GlobalMapOverlayBuilder(),

l4d2web/templates/overlays.html

@@ -37,9 +37,6 @@
       <fieldset class="overlay-type-radio">
         <legend>Type</legend>
         <label><input type="radio" name="type" value="workshop" checked> Workshop (downloads from Steam)</label>
-        {% if g.user.admin %}
-        <label><input type="radio" name="type" value="external"> External (admin-managed; populated via filesystem)</label>
-        {% endif %}
       </fieldset>
       <label>Name <input name="name" required></label>
       <p class="muted">The path is generated automatically.</p>

l4d2web/tests/test_global_overlays.py

@@ -43,7 +43,7 @@ def test_ensure_global_overlays_repairs_existing_rows(tmp_path, monkeypatch):
     init_db()
 
     with session_scope() as session:
-        overlay = Overlay(name="cedapug-maps", path="legacy", type="external", user_id=None)
+        overlay = Overlay(name="cedapug-maps", path="legacy", type="cedapug_maps", user_id=None)
         session.add(overlay)
         session.flush()
         session.add(
@@ -160,8 +160,8 @@ def test_enqueue_refresh_global_overlays_creates_system_job(tmp_path, monkeypatc
 
 def test_is_creatable_overlay_type_policy():
     assert is_creatable_overlay_type("workshop", admin=False) is True
-    assert is_creatable_overlay_type("external", admin=False) is False
-    assert is_creatable_overlay_type("external", admin=True) is True
     assert is_creatable_overlay_type("workshop", admin=True) is True
+    assert is_creatable_overlay_type("external", admin=False) is False
+    assert is_creatable_overlay_type("external", admin=True) is False
     assert is_creatable_overlay_type("l4d2center_maps", admin=True) is False
     assert is_creatable_overlay_type("cedapug_maps", admin=True) is False

l4d2web/tests/test_host_commands.py

@@ -72,3 +72,46 @@ def test_stream_command_yields_stdout_lines() -> None:
     lines = list(stream_command(["python3", "-c", "print('one'); print('two')"]))
 
     assert lines == ["one", "two"]
+
+
+def test_stream_command_emits_heartbeat_when_subprocess_silent() -> None:
+    import time
+
+    from l4d2web.services.host_commands import stream_command
+
+    cmd = [
+        "python3",
+        "-c",
+        "import time; time.sleep(0.4); print('done')",
+    ]
+
+    started = time.monotonic()
+    items: list[str] = []
+    for item in stream_command(cmd, heartbeat_interval=0.05):
+        items.append(item)
+        if time.monotonic() - started > 2.0:
+            break
+
+    assert "done" in items, items
+    heartbeats = [i for i in items if i == ""]
+    assert len(heartbeats) >= 2, f"expected ≥2 heartbeat ticks during the silent 0.4s window, got items={items!r}"
+
+
+def test_stream_command_close_releases_subprocess_promptly() -> None:
+    import time
+
+    from l4d2web.services.host_commands import stream_command
+
+    cmd = [
+        "python3",
+        "-c",
+        "import time;\nwhile True:\n    time.sleep(60)",
+    ]
+
+    gen = stream_command(cmd, heartbeat_interval=0.05)
+    assert next(gen) == ""
+
+    started = time.monotonic()
+    gen.close()
+    elapsed = time.monotonic() - started
+    assert elapsed < 1.0, f"gen.close() took {elapsed:.2f}s; subprocess cleanup must not block"

l4d2web/tests/test_l4d2_facade.py

@@ -32,7 +32,7 @@ def server_with_blueprint(tmp_path, monkeypatch):
             session.add(user)
             session.flush()
 
-            overlay = Overlay(name="Standard Overlay", path="standard", type="external", user_id=None)
+            overlay = Overlay(name="Standard Overlay", path="standard", type="workshop", user_id=user.id)
             session.add(overlay)
             session.flush()
 

l4d2web/tests/test_overlay_builders.py

@@ -1,4 +1,4 @@
-"""Tests for overlay builders (registry, ExternalBuilder, WorkshopBuilder)."""
+"""Tests for overlay builders (registry, WorkshopBuilder)."""
 from __future__ import annotations
 
 import os
@@ -61,9 +61,9 @@ def _capture_logs():
     return out, err, out.append, err.append
 
 
-def test_registry_has_external_and_workshop() -> None:
-    assert "external" in overlay_builders.BUILDERS
+def test_registry_has_workshop() -> None:
     assert "workshop" in overlay_builders.BUILDERS
+    assert "external" not in overlay_builders.BUILDERS
 
 
 def test_registry_unknown_type_raises_keyerror() -> None:
@@ -71,28 +71,6 @@ def test_registry_unknown_type_raises_keyerror() -> None:
         overlay_builders.BUILDERS["nope"]
 
 
-def test_external_builder_is_idempotent_noop_with_log(env: Path) -> None:
-    _, overlay_id = _create_user_and_overlay("ext", "external")
-    out, err, on_stdout, on_stderr = _capture_logs()
-
-    with session_scope() as s:
-        overlay = s.query(Overlay).filter_by(id=overlay_id).one()
-        overlay_builders.BUILDERS["external"].build(
-            overlay, on_stdout=on_stdout, on_stderr=on_stderr, should_cancel=lambda: False
-        )
-
-    assert (env / "overlays" / "7").is_dir()
-    assert any("external overlay" in line for line in out), out
-    # Existing files in the overlay dir are not touched on subsequent build.
-    (env / "overlays" / "7" / "untouched.txt").write_text("data")
-    with session_scope() as s:
-        overlay = s.query(Overlay).filter_by(id=overlay_id).one()
-        overlay_builders.BUILDERS["external"].build(
-            overlay, on_stdout=on_stdout, on_stderr=on_stderr, should_cancel=lambda: False
-        )
-    assert (env / "overlays" / "7" / "untouched.txt").read_text() == "data"
-
-
 def test_workshop_builder_creates_absolute_symlinks(env: Path) -> None:
     _, overlay_id = _create_user_and_overlay("ws", "workshop")
     cache_root = env / "workshop_cache"

l4d2web/tests/test_overlays.py

@@ -38,8 +38,10 @@ def user_client_with_overlay(tmp_path, monkeypatch):
     with session_scope() as session:
         user = User(username="alice", password_digest=hash_password("secret"), admin=False)
         session.add(user)
-        # System external overlay (no user_id), pre-existing.
-        session.add(Overlay(name="standard", path="standard", type="external", user_id=None))
+        # System overlay (managed-global, no user_id), pre-existing.
+        session.add(
+            Overlay(name="standard", path="standard", type="l4d2center_maps", user_id=None)
+        )
         session.flush()
         user_id = user.id
 
@@ -79,10 +81,10 @@ def test_admin_can_view_overlay_edit_controls(admin_client) -> None:
     assert 'action="/overlays"' in text
 
 
-def test_admin_can_create_external_overlay(admin_client) -> None:
+def test_admin_can_create_workshop_overlay_via_route(admin_client) -> None:
     response = admin_client.post(
         "/overlays",
-        data={"name": "standard", "type": "external"},
+        data={"name": "standard", "type": "workshop"},
         headers={"X-CSRF-Token": "test-token"},
     )
     assert response.status_code == 302
@@ -106,15 +108,6 @@ def test_overlay_ref_rejects_unsafe_components(overlay_ref: str) -> None:
         validate_overlay_ref(overlay_ref)
 
 
-def test_non_admin_cannot_create_external_overlay(user_client_with_overlay) -> None:
-    response = user_client_with_overlay.post(
-        "/overlays",
-        data={"name": "bad", "type": "external"},
-        headers={"X-CSRF-Token": "test-token"},
-    )
-    assert response.status_code == 400
-
-
 def test_user_can_create_workshop_overlay(user_client_with_overlay) -> None:
     response = user_client_with_overlay.post(
         "/overlays",
@@ -182,7 +175,7 @@ def test_two_users_can_have_workshop_overlay_with_same_name(tmp_path, monkeypatc
 def test_admin_can_update_and_delete_overlay(admin_client) -> None:
     create = admin_client.post(
         "/overlays",
-        data={"name": "standard", "type": "external"},
+        data={"name": "standard", "type": "workshop"},
         headers={"X-CSRF-Token": "test-token"},
     )
     assert create.status_code == 302
@@ -265,7 +258,7 @@ def test_update_overlay_rejects_duplicate_name(admin_client) -> None:
     for name in ("standard", "competitive"):
         response = admin_client.post(
             "/overlays",
-            data={"name": name, "type": "external"},
+            data={"name": name, "type": "workshop"},
             headers={"X-CSRF-Token": "test-token"},
         )
         assert response.status_code == 302
@@ -286,7 +279,7 @@ def test_update_overlay_rejects_duplicate_name(admin_client) -> None:
 def test_overlay_detail_page_lists_using_blueprints(admin_client) -> None:
     create = admin_client.post(
         "/overlays",
-        data={"name": "shared", "type": "external"},
+        data={"name": "shared", "type": "workshop"},
         headers={"X-CSRF-Token": "test-token"},
     )
     assert create.status_code == 302
@@ -366,8 +359,8 @@ def test_overlay_detail_page_404_when_missing(admin_client) -> None:
     assert response.status_code == 404
 
 
-def test_overlay_detail_hides_edit_for_non_admin_external(user_client_with_overlay) -> None:
-    # The seeded "standard" external overlay (id=1, user_id=NULL) is admin-only edit.
+def test_overlay_detail_hides_edit_for_non_admin_managed_global(user_client_with_overlay) -> None:
+    # The seeded "standard" managed-global overlay (id=1, user_id=NULL) is read-only for non-admins.
     response = user_client_with_overlay.get("/overlays/1")
     text = response.get_data(as_text=True)
 
@@ -377,26 +370,6 @@ def test_overlay_detail_hides_edit_for_non_admin_external(user_client_with_overl
     assert "delete-overlay-modal" not in text
 
 
-def test_non_admin_cannot_view_other_users_private_non_workshop_overlay(user_client_with_overlay) -> None:
-    with session_scope() as session:
-        other = User(username="mallory", password_digest=hash_password("secret"), admin=False)
-        session.add(other)
-        session.flush()
-        overlay = Overlay(
-            name="private-external",
-            path="private-external",
-            type="external",
-            user_id=other.id,
-        )
-        session.add(overlay)
-        session.flush()
-        overlay_id = overlay.id
-
-    response = user_client_with_overlay.get(f"/overlays/{overlay_id}")
-
-    assert response.status_code == 403
-
-
 def test_managed_global_overlay_detail_shows_source_url(admin_client) -> None:
     overlay_id = _create_managed_global_overlay()
 
@@ -410,7 +383,7 @@ def test_managed_global_overlay_detail_shows_source_url(admin_client) -> None:
 def test_overlay_update_redirects_to_detail(admin_client) -> None:
     create = admin_client.post(
         "/overlays",
-        data={"name": "standard", "type": "external"},
+        data={"name": "standard", "type": "workshop"},
         headers={"X-CSRF-Token": "test-token"},
     )
     assert create.status_code == 302
@@ -429,7 +402,7 @@ def test_overlay_update_redirects_to_detail(admin_client) -> None:
 def test_delete_overlay_rejects_in_use_overlay(admin_client) -> None:
     create = admin_client.post(
         "/overlays",
-        data={"name": "standard", "type": "external"},
+        data={"name": "standard", "type": "workshop"},
         headers={"X-CSRF-Token": "test-token"},
     )
     assert create.status_code == 302

l4d2web/tests/test_status_and_server_logs.py

@@ -48,6 +48,23 @@ def test_owner_can_stream_server_logs(owner_client_with_server, monkeypatch) ->
     assert response.status_code == 200
 
 
+def test_log_stream_translates_heartbeat_to_sse_keepalive(owner_client_with_server, monkeypatch) -> None:
+    client, server_id = owner_client_with_server
+
+    monkeypatch.setattr(
+        "l4d2web.services.l4d2_facade.stream_server_logs",
+        lambda name, lines=200, follow=True: iter(["first", "", "second"]),
+    )
+
+    response = client.get(f"/servers/{server_id}/logs/stream")
+    assert response.status_code == 200
+    body = response.get_data(as_text=True)
+    assert "data: first\n\n" in body
+    assert "data: second\n\n" in body
+    assert ": keepalive\n\n" in body
+    assert "data: \n\n" not in body
+
+
 def test_status_precedence() -> None:
     from l4d2web.services.status import compute_display_state
 

l4d2web/tests/test_workshop_overlay_models.py

@@ -34,18 +34,18 @@ def test_overlay_has_type_and_user_id(db) -> None:
         s.add(Overlay(name="standard", path="standard"))
         s.flush()
         row = s.query(Overlay).filter_by(name="standard").one()
-        assert row.type == "external"
+        assert row.type == "workshop"
         assert row.user_id is None
 
 
-def test_two_externals_with_same_name_are_rejected(db) -> None:
+def test_two_system_overlays_with_same_name_are_rejected(db) -> None:
     with session_scope() as s:
-        s.add(Overlay(name="shared", path="shared", type="external", user_id=None))
+        s.add(Overlay(name="shared", path="shared", type="l4d2center_maps", user_id=None))
         s.flush()
 
     with pytest.raises(IntegrityError):
         with session_scope() as s:
-            s.add(Overlay(name="shared", path="other", type="external", user_id=None))
+            s.add(Overlay(name="shared", path="other", type="cedapug_maps", user_id=None))
             s.flush()
 
 
@@ -161,9 +161,10 @@ def test_job_has_overlay_id_column(db) -> None:
 
 def test_overlay_id_does_not_reuse_after_delete(db) -> None:
     """SQLite AUTOINCREMENT must guarantee deleted IDs are never reused."""
+    user_id = _make_user("alice")
     with session_scope() as s:
-        s.add(Overlay(name="first", path="1", type="external", user_id=None))
-        s.add(Overlay(name="second", path="2", type="external", user_id=None))
+        s.add(Overlay(name="first", path="1", type="workshop", user_id=user_id))
+        s.add(Overlay(name="second", path="2", type="workshop", user_id=user_id))
         s.flush()
         ids_before = sorted(o.id for o in s.query(Overlay).all())
         last_id = ids_before[-1]
@@ -174,7 +175,7 @@ def test_overlay_id_does_not_reuse_after_delete(db) -> None:
         s.flush()
 
     with session_scope() as s:
-        s.add(Overlay(name="third", path="3", type="external", user_id=None))
+        s.add(Overlay(name="third", path="3", type="workshop", user_id=user_id))
         s.flush()
         new_id = s.query(Overlay).filter_by(name="third").one().id