refactor(left4me): non-editable install + relocate runtime state to /var/lib/left4me

Two related changes landed together:

1. /opt/left4me/ becomes a root-owned deploy-artifact root. Only
   /opt/left4me/src lives there. Source tree is no longer mutated
   by pip at runtime; left4me only needs read access.
2. Runtime mutable state moved to /var/lib/left4me/: the venv (was
   /opt/left4me/.venv) and steamcmd (was /opt/left4me/steam).

The non-editable install copies the (root-owned) source to a
left4me-owned tempdir before `pip install --force-reinstall`.
setuptools.build_meta writes <pkg>.egg-info/ into the source dir
during get_requires_for_build_wheel, so a direct `pip install
/opt/left4me/src/l4d2*` fails on a root-owned source. The
temp-copy is what `python -m build` ought to do but doesn't (its
build isolation only sandboxes deps).

Prereq for the deployment-responsibility reshape: target-side
symlinks from /etc/... into /opt/left4me/src/deploy/files/... are
now safe by construction (left4me cannot rewrite its own hardening
profile).

Design + verification record: left4me/docs/superpowers/specs/
2026-05-15-runtime-state-relocation-design.md

Verified on ovh.left4me: bw apply idempotent on second pass (0
fixed, 0 failed); pip show reports site-packages location, no
Editable project location; web + gameserver units run clean;
alembic current returns head.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

bundles/left4me/README.md

@@ -44,10 +44,19 @@ from defaults. None of these need to be declared per-node.
   (`left4me-systemctl`, `left4me-journalctl`, `left4me-overlay`,
   `left4me-script-sandbox`) plus a tight sudoers file (validated with
   `visudo -cf` before install).
-- `git_deploy`s the left4me repo to `/opt/left4me/src`, builds a venv at
-  `/opt/left4me/.venv`, `pip install -e`s both `l4d2host` and `l4d2web`,
-  runs `alembic upgrade head` and `flask seed-script-overlays`, then
-  enables `left4me-web.service`.
+- `git_deploy`s the left4me repo to `/opt/left4me/src` (root-owned —
+  the source tree is read-only at runtime so left4me cannot rewrite
+  its own future hardening drop-ins / unit files under
+  `/opt/left4me/src/deploy/`). Builds a venv at `/var/lib/left4me/.venv`
+  and installs `l4d2host` + `l4d2web` non-editably (`pip install` copies
+  source to a left4me-writable tempdir first, since setuptools writes
+  egg-info into the source dir during the wheel build). Then runs
+  `alembic upgrade head` and `flask seed-script-overlays` and enables
+  `left4me-web.service`. Developer machines keep `pip install -e` via
+  direnv for fast iteration; only the production install model differs.
+  Runtime mutable state lives under `/var/lib/left4me/` (venv, steamcmd,
+  game installations, overlays); `/opt/left4me/` stays as a root-owned
+  deploy-artifact root.
 - Emits four systemd units via `systemd/units` metadata (consumed by
   `bundles/systemd/`):
   - `left4me-web.service` — gunicorn on `127.0.0.1:8000` (TLS terminates upstream).

bundles/left4me/files/etc/left4me/host.env.mako

@@ -3,4 +3,4 @@
 LEFT4ME_ROOT=/var/lib/left4me
 # l4d2host invokes steamcmd by absolute path — bypasses PATH lookup so the
 # script's `cd "$(dirname "$0")"` resolves next to the real install dir.
-LEFT4ME_STEAMCMD=/opt/left4me/steam/steamcmd.sh
+LEFT4ME_STEAMCMD=/var/lib/left4me/steam/steamcmd.sh

bundles/left4me/items.py

@@ -6,12 +6,26 @@
 
 directories = {
     '/opt/left4me': {
-        'owner': 'left4me',
-        'group': 'left4me',
+        # Deploy-artifact root. Only /opt/left4me/src lives here; runtime
+        # state (.venv, steamcmd) lives under /var/lib/left4me/. Root-owned
+        # so left4me cannot drop new files alongside src/ (e.g. an attacker
+        # with web-compromise can't plant a 'scripts.d/' loaded by future
+        # deploy logic).
+        'owner': 'root',
+        'group': 'root',
+        'mode': '0755',
     },
     '/opt/left4me/src': {
-        'owner': 'left4me',
-        'group': 'left4me',
+        # Source checkout. Root-owned because the production install model
+        # is non-editable: pip_install copies the source to a left4me-owned
+        # tempdir before building, so the source tree on disk is never
+        # mutated at runtime and left4me only needs read access (which
+        # world-readable bits provide). Keeps left4me from being able to
+        # rewrite its own future hardening drop-ins / unit files under
+        # /opt/left4me/src/deploy/ (target-side symlink model in the
+        # deployment-responsibility reshape).
+        'owner': 'root',
+        'group': 'root',
     },
     '/etc/left4me': {
         'owner': 'root',
@@ -33,7 +47,10 @@ directories = {
     '/var/lib/left4me/runtime':        {'owner': 'left4me', 'group': 'left4me'},
     '/var/lib/left4me/workshop_cache': {'owner': 'left4me', 'group': 'left4me'},
     '/var/lib/left4me/tmp':            {'owner': 'left4me', 'group': 'left4me'},
-    '/opt/left4me/steam':              {'owner': 'left4me', 'group': 'left4me'},
+    '/var/lib/left4me/steam':          {'owner': 'left4me', 'group': 'left4me'},
+    # Note: the venv (/var/lib/left4me/.venv) is created by the
+    # left4me_create_venv action; declaring it here too would race with
+    # `python -m venv` which expects to create the directory itself.
     '/usr/local/libexec/left4me': {
         'owner': 'root',
         'group': 'root',
@@ -133,15 +150,15 @@ actions = {
         # tarball version from bw isn't useful.
         'command': (
             'sudo -u left4me sh -c "'
-            'cd /opt/left4me/steam && '
+            'cd /var/lib/left4me/steam && '
             'curl -fsSL https://media.steampowered.com/installer/steamcmd_linux.tar.gz | '
             'tar -xz'
             '"'
         ),
-        'unless': 'test -x /opt/left4me/steam/steamcmd.sh',
+        'unless': 'test -x /var/lib/left4me/steam/steamcmd.sh',
         'cascade_skip': False,
         'needs': [
-            'directory:/opt/left4me/steam',
+            'directory:/var/lib/left4me/steam',
             'pkg_apt:curl',
             'pkg_apt:libc6_i386',  # bw pkg_apt convention: _ → :
             'pkg_apt:lib32z1',
@@ -159,26 +176,21 @@ git_deploy = {
         'repo': node.metadata.get('left4me/git_url'),
         'rev': node.metadata.get('left4me/git_branch'),
         'triggers': [
-            # On a code-update apply, refresh the DB schema. pip_install
-            # would have triggered alembic in the create_venv path, but on
-            # a normal apply pip_install's `unless` skips (packages still
-            # importable from the previous editable install), and that
-            # would leave alembic_upgrade dormant. Wiring git_deploy →
-            # alembic directly ensures new migrations land whenever new
-            # code lands. alembic upgrade head is idempotent (no-op when
-            # already at head), so this is safe to fire on every code
-            # update; the seed_overlays + service:restart cascade off
-            # alembic also covers picking up the new code in gunicorn.
+            # Rebuild + reinstall the packages whenever the checkout
+            # changes. pip_install does its own out-of-tree build (copies
+            # source to a left4me-owned tempdir before invoking pip), so
+            # the source tree itself stays root-owned and untouched.
+            # pip_install cascades into alembic_upgrade → web restart.
+            'action:left4me_pip_install',
+            # alembic upgrade head is idempotent — keeping it as a direct
+            # trigger off git_deploy is belt-and-braces in case the
+            # pip_install cascade is ever short-circuited.
             'action:left4me_alembic_upgrade',
             # Privileged-helper scripts: reinstall from the new checkout
             # into /usr/local/{libexec,sbin}/ as root-owned. No-op when
             # the checkout didn't actually change (action is triggered).
             'action:install_left4me_scripts',
         ],
-        # chown_src and pip_install are NOT in triggers — they run every
-        # apply gated by their own `unless` guards, which makes the chain
-        # self-healing after a partial failure. (Items in a triggers list
-        # must be triggered:True, which would lose that property.)
     },
 }
 
@@ -204,26 +216,12 @@ actions['install_left4me_scripts'] = {
     ],
 }
 
-actions['left4me_chown_src'] = {
-    # Runs every apply (cheap — chown -R on a small tree). Self-heals
-    # whenever git_deploy extracts a new tarball as root-owned files.
-    # Not in any triggers list so doesn't need triggered:True.
-    'command': 'chown -R left4me:left4me /opt/left4me/src',
-    'unless': 'test -z "$(find /opt/left4me/src \\! -user left4me -print -quit 2>/dev/null)"',
-    'cascade_skip': False,
-    'needs': [
-        'git_deploy:/opt/left4me/src',
-        'user:left4me',
-        'group:left4me',
-    ],
-}
-
 actions['left4me_create_venv'] = {
-    'command': 'sudo -u left4me /usr/bin/python3 -m venv /opt/left4me/.venv',
-    'unless':  'test -x /opt/left4me/.venv/bin/python',
+    'command': 'sudo -u left4me /usr/bin/python3 -m venv /var/lib/left4me/.venv',
+    'unless':  'test -x /var/lib/left4me/.venv/bin/python',
     'cascade_skip': False,
     'needs': [
-        'directory:/opt/left4me',
+        'directory:/var/lib/left4me',
         'pkg_apt:python3-venv',
         'user:left4me',
     ],
@@ -233,29 +231,41 @@ actions['left4me_create_venv'] = {
 }
 
 actions['left4me_pip_upgrade'] = {
-    'command': 'sudo -u left4me /opt/left4me/.venv/bin/python -m pip install --upgrade pip',
+    'command': 'sudo -u left4me /var/lib/left4me/.venv/bin/python -m pip install --upgrade pip',
     'triggered': True,
     'cascade_skip': False,
     'needs': [
         'pkg_apt:python3-pip',
     ],
-    # No triggers — pip_install runs on every apply (gated by `unless`)
-    # rather than being chained from here. Keeps pip_upgrade scoped to
-    # exactly its purpose.
+    # No triggers — pip_install is driven by git_deploy on actual code
+    # updates, not by venv setup. Keeps pip_upgrade scoped to exactly
+    # its purpose.
 }
 
 actions['left4me_pip_install'] = {
-    # Single pip invocation installs both editable packages from the same
-    # checkout. Runs on every apply: pip install -e is fast on no-op, and
-    # any gate weaker than "egg-info matches pyproject.toml" can mask
-    # script regeneration — e.g. adding [project.scripts] later wouldn't
-    # be picked up if `unless` only checks importability.
-    'command': 'sudo -u left4me /opt/left4me/.venv/bin/pip install -e /opt/left4me/src/l4d2host -e /opt/left4me/src/l4d2web',
+    # Non-editable install of l4d2host + l4d2web into the venv. We have
+    # to copy the source to a left4me-writable tempdir first because
+    # setuptools.build_meta writes <pkg>.egg-info/ into the source dir
+    # during `get_requires_for_build_wheel`, and the source tree is
+    # root-owned. cp -r is fast (small tree, world-readable), the build
+    # itself happens in $tmpdir, and pip installs the resulting wheel
+    # into /var/lib/left4me/.venv/site-packages. --force-reinstall
+    # because the version string in pyproject.toml (0.1.0) doesn't
+    # change commit-to-commit; without it pip would skip on no-op.
+    # triggered:True so this only fires on actual git_deploy changes
+    # (the cp + build is too heavy to run on every apply).
+    'command': """sudo -u left4me sh -c '
+set -e
+tmpdir=$(mktemp -d -t left4me-build-XXXXXX)
+trap "rm -rf \\"$tmpdir\\"" EXIT
+cp -r /opt/left4me/src/l4d2host /opt/left4me/src/l4d2web "$tmpdir/"
+/var/lib/left4me/.venv/bin/pip install --force-reinstall "$tmpdir/l4d2host" "$tmpdir/l4d2web"
+'""",
+    'triggered': True,
     'cascade_skip': False,
     'needs': [
         'git_deploy:/opt/left4me/src',
         'action:left4me_create_venv',
-        'action:left4me_chown_src',
     ],
     'triggers': [
         'action:left4me_alembic_upgrade',
@@ -271,7 +281,7 @@ actions['left4me_alembic_upgrade'] = {
         'cd /opt/left4me/src/l4d2web && '
         'set -a && . /etc/left4me/host.env && . /etc/left4me/web.env && set +a && '
         'env JOB_WORKER_ENABLED=false PYTHONPATH=/opt/left4me/src '
-        '/opt/left4me/.venv/bin/alembic -c /opt/left4me/src/l4d2web/alembic.ini upgrade head'
+        '/var/lib/left4me/.venv/bin/alembic -c /opt/left4me/src/l4d2web/alembic.ini upgrade head'
         '"'
     ),
     'triggered': True,
@@ -293,7 +303,7 @@ actions['left4me_seed_overlays'] = {
         'sudo -u left4me sh -c "'
         'set -a && . /etc/left4me/host.env && . /etc/left4me/web.env && set +a && '
         'env JOB_WORKER_ENABLED=false PYTHONPATH=/opt/left4me/src '
-        '/opt/left4me/.venv/bin/flask --app l4d2web.app:create_app '
+        '/var/lib/left4me/.venv/bin/flask --app l4d2web.app:create_app '
         'seed-script-overlays /opt/left4me/src/examples/script-overlays'
         '"'
     ),

bundles/left4me/metadata.py

@@ -101,7 +101,7 @@ defaults = {
         # Idempotent — a re-fire while a refresh is already queued/running
         # is a no-op (see l4d2web/cli.py:workshop_refresh).
         'left4me-workshop-refresh': {
-            'command': '/opt/left4me/.venv/bin/flask --app l4d2web.app:create_app workshop-refresh',
+            'command': '/var/lib/left4me/.venv/bin/flask --app l4d2web.app:create_app workshop-refresh',
             'when': '*-*-* 04:00:00',
             'persistent': True,
             'user': 'left4me',
@@ -199,10 +199,10 @@ HARDENING_SERVER = {
         # fails and the server falls back to LAN-only mode regardless
         # of sv_lan=0 — clients then get "LAN servers are restricted
         # to local clients (class C)". .steam holds symlinks into
-        # /opt/left4me/steam, so both paths need to be bound back
+        # /var/lib/left4me/steam, so both paths need to be bound back
         # through TemporaryFileSystem.
         '/var/lib/left4me/.steam',
-        '/opt/left4me/steam',
+        '/var/lib/left4me/steam',
         '/etc/left4me/host.env',
         '/etc/ssl',
         '/etc/ca-certificates',
@@ -330,14 +330,14 @@ def systemd_units(metadata):
                         'WorkingDirectory': '/opt/left4me/src',
                         'Environment': {
                             'HOME=/var/lib/left4me',
-                            'PATH=/opt/left4me/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
+                            'PATH=/var/lib/left4me/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
                         },
                         'EnvironmentFile': (
                             '/etc/left4me/host.env',
                             '/etc/left4me/web.env',
                         ),
                         'ExecStart': (
-                            '/opt/left4me/.venv/bin/gunicorn '
+                            '/var/lib/left4me/.venv/bin/gunicorn '
                             f'--workers {workers} --threads {threads} '
                             "--bind 127.0.0.1:8000 'l4d2web.app:create_app()'"
                         ),