left4me/docs/superpowers/plans/2026-05-14-overlay-idmap.md

Idmapped lowerdirs for left4me kernel-overlayfs

Context

Kernel-overlayfs copy-up preserves the lower-layer file's owner and mode in the upperdir. Script overlays today are built by left4me-script-sandbox running as uid l4d2-sandbox, and the helper finalizes them as l4d2-sandbox:l4d2-sandbox 0755. When the L4D2 server (uid left4me) tries to write into a directory that exists only in the lower layer — e.g. SourceMod's addons/sourcemod/logs/ for log rotation — copy-up succeeds but the result is l4d2-sandbox-owned, so the write fails EACCES. Workshop overlays are unaffected because the web app (uid left4me) builds them as left4me-owned with symlinks into a left4me- owned cache.

We considered four fixes (chown-flip on every rebuild, shared group, collapse sandbox uid into left4me, idmapped lowerdir bind mounts). The user chose the idmap path: disk state stays untouched, the mount stack remaps l4d2-sandbox → left4me at mount time, kernel-overlayfs sees a left4me-owned lower layer, and copy-up creates upperdir entries owned by left4me naturally. No ownership flipping, no shared group, no security regression.

Outcome: sm_cvar writes succeed, SM logs land in runtime/<n>/upper/..., and any future "writes into a sandbox-built lower layer" works without left4me-specific plumbing.

Environment confirmed

Test server: Debian Trixie, kernel 6.12.86+deb13-amd64, util-linux supports mount --map-users <on_disk>:<in_mount>:<count>. Idmapped lowerdirs for overlayfs landed mainline in 6.6, so 6.12 is fine. Verified end-to-end on /var/lib/left4me/ (ext4) in a temp dir on 2026-05-14:

1. Source dir owned l4d2-sandbox:l4d2-sandbox (uid 981).
2. mount --bind --map-users=981:980:1 --map-groups=981:980:1 src dst — dst view shows uid 980 (left4me).
3. Overlay mount with the idmapped path as lowerdir= — merged view also shows uid 980.
4. sudo -u left4me touch merged/addons/sourcemod/logs/L_test.log — succeeds. sudo -u left4me bash -c "echo x >> merged/file-from-sandbox.txt" — succeeds (copy-up of existing file).
5. upper/ after writes is entirely left4me-owned (uid 980).

Caveat surfaced during testing: --map-users direction is on-disk uid first, not "inner-namespace uid". The util-linux man page calls it <inner>:<outer>:<count> but <inner> means "the filesystem's native view" (on disk) and <outer> means "what the mount exposes outward". Easy to get wrong; do not trust the man page wording.

Approach

The privileged mount helper grows one step before the overlay mount: for each lowerdir whose owning uid is l4d2-sandbox, create an idmapped bind mount at runtime/<n>/idmap/<basename> that remaps that uid to left4me. Use the idmapped paths (instead of raw paths) in the lowerdir= string passed to mount -t overlay. On umount, tear the idmap binds down after the overlay itself is unmounted.

Lowerdirs already owned by left4me (workshop builds, installation/, caches) bypass the idmap step and are used as-is, so workshop overlays keep working without behavior change.

Execution shape

Implementation will be driven by superpowers:subagent-driven-development: fresh implementer subagent per task, followed by a spec-compliance reviewer then a code-quality reviewer. The project's AGENTS.md forbids git worktrees, so all work happens in the live tree. Commits land directly on master per the user-confirmed project pattern.

Tasks ordered for review-friendly progression. Each task is independently committable; the deploy/verify step at the end exercises the whole chain on the real test server.

Task 1 — Idmap bind mounts in left4me-overlay

Edit deploy/files/usr/local/libexec/left4me/left4me-overlay and l4d2host/tests/test_overlay_helper.py together (TDD: write failing PRINT_ONLY-mode tests first, then make them pass).

Behavior to add:

• Resolve l4d2_sandbox_uid and left4me_uid (and gids) via pwd.getpwnam / grp.getgrnam. Hard fail with a clear message if either is missing.
• On mount <name>: before constructing the lowerdir= string, for each resolved lowerdir, stat it; if the top-level dir's st_uid equals l4d2_sandbox_uid, create runtime/<n>/idmap/<basename> (mode 0700, root-owned), and if it's not already a mountpoint, exec mount --bind --map-users=<l4d2_sandbox_uid>:<left4me_uid>:1 --map-groups=<l4d2_sandbox_gid>:<left4me_gid>:1 <src> <target>. Use numeric uids/gids in the argv. Substitute the idmap path into the lowerdir= colon string in place of the original path.
• On umount <name>: after the existing umount of merged, iterate runtime/<n>/idmap/*, umount each that is a mountpoint, then shutil.rmtree(runtime/<n>/idmap, ignore_errors=True). Idempotent.
• PRINT_ONLY mode emits the bind-mount argv (one line per bind) before the overlay-mount argv, same shell-quoting style.

Tests to add to test_overlay_helper.py (reuse PRINT_ONLY harness):

• test_mount_idmaps_sandbox_owned_lowerdir — tmp lower owned by faked-sandbox uid, assert helper emits mount --bind --map-users=... argv and the overlay lowerdir= references the idmap path.
• test_mount_skips_idmap_for_left4me_owned_lowerdir — assert no bind argv, raw path in lowerdir=.
• test_umount_unwinds_idmap_binds — pre-seed an idmap subdir as a mountpoint sentinel; assert the umount sequence in PRINT_ONLY includes the bind teardown after the overlay umount.

Uid lookup in tests: monkeypatch pwd.getpwnam to return synthetic uids matching what the test's chown set up. (No root required.)

Task 2 — Deploy-artifact regression test

Edit deploy/tests/test_deploy_artifacts.py. Add a single test that opens deploy/files/usr/local/libexec/left4me/left4me-overlay and asserts the strings --map-users and runtime/ followed by idmap (or similar identifying marker) are present. Cheap guard against silent regression of the deploy artifact.

Task 3 — Deploy README mirror note

Edit deploy/README.md. Add one line under the existing ckn-bw mirror notes flagging that the helper file change must be picked up by bundles/left4me/ in ckn-bw (no new group, user, or unit needed).

Task 4 — Persist the plan in-repo

Per AGENTS.md: "the persisted artifact must end up under docs/superpowers/ and be committed." Copy this scratch plan to docs/superpowers/plans/2026-05-14-overlay-idmap.md and commit it. Do this as a separate commit from Task 1 so the plan lands before the implementation.

Task 5 — Deploy and verify on left4.me

Out-of-band, after the code tasks land:

1. ckn-bw apply (or scp the helper into place on the test server) to get the new helper deployed.
2. Stop server 2: sudo systemctl stop left4me-server@2.
3. Clear the stale l4d2-sandbox-owned upperdir SM dirs: sudo rm -rf /var/lib/left4me/runtime/2/upper/left4dead2/addons/sourcemod/{logs,data}.
4. Start server 2: sudo systemctl start left4me-server@2.
5. Confirm journalctl -u left4me-server@2 -o cat -n 50 shows the new mount --bind --map-users=... line.
6. RCON sm_cvar nb_update_frequency 0.0333. Expect no Platform returned error: "Permission denied" log line.
7. sudo ls -ln /var/lib/left4me/runtime/2/upper/left4dead2/addons/sourcemod/logs/. Expect uid 980 (left4me).
8. Restart again to confirm idempotency: idmap binds set up fresh, no leftover mounts from prior start.

Files to modify

1. deploy/files/usr/local/libexec/left4me/left4me-overlay

Single privileged code path; everything else flows from here.

Add a helper function to decide whether a path needs idmapping. Stat the directory; if its st_uid matches the resolved l4d2-sandbox uid, return the idmapped path under runtime/<n>/idmap/; otherwise return the input path unchanged.

In cmd_mount, before constructing lowerdir=:

1. os.makedirs(runtime_dir / "idmap", exist_ok=True) (root-owned, mode 0o700; only the helper writes here).
2. Resolve l4d2_sandbox_uid = pwd.getpwnam("l4d2-sandbox").pw_uid and left4me_uid = pwd.getpwnam("left4me").pw_uid. Cache. Fail fast with a clear message if either user is missing.
3. For each lowerdir in the resolved list, compute idmapped_path(lowerdir). If remapping is required:
   • Create the target directory under runtime/<n>/idmap/<basename> if missing.
   • Skip the bind if it's already mounted there (os.path.ismount).
   • Otherwise exec mount --bind --map-users=<l4d2_sandbox_uid>:<left4me_uid>:1 --map-groups=<l4d2_sandbox_gid>:<left4me_gid>:1 <src> <target>. Direction note: first arg is the on-disk uid, second arg is the uid the mount exposes. We verified empirically that this is what the kernel honors despite ambiguous man-page wording.
4. Pass the (possibly idmapped) paths into the lowerdir= colon string in the same order.

In cmd_umount, after the existing overlay umount:

• For each subdirectory under runtime/<n>/idmap/, if it's a mountpoint, umount it.
• shutil.rmtree(runtime_dir / "idmap", ignore_errors=True) after all binds are gone.
• Idempotent: re-running after the dir is gone is a no-op.

PRINT_ONLY mode: emit the bind-mount argv before the overlay-mount argv, on separate lines, so test assertions can match. Same shell-quoting.

Allowlist: no change needed — the idmap binds land under runtime/, which is already write-permitted for the helper.

2. l4d2host/tests/test_overlay_helper.py

Add tests using the existing PRINT_ONLY harness:

• test_mount_idmaps_sandbox_owned_lowerdir: create a tmp lowerdir, chown it to a fake-l4d2-sandbox (use monkeypatch on the uid lookup if running unprivileged), run helper in PRINT_ONLY, assert a mount --bind --map-users=... line appears and the lowerdir= string references the idmap path.
• test_mount_skips_idmap_for_left4me_owned_lowerdir: tmp lowerdir owned by the test user, assert no bind-mount argv emitted and lowerdir= references the raw path.
• test_umount_unwinds_idmap_binds: pre-create runtime/<n>/idmap/foo as a mountpoint sentinel, assert the PRINT_ONLY umount sequence includes the bind-mount teardown before the overlay umount? Actually overlay-first, then binds — match the helper order.

Reuse the existing LEFT4ME_OVERLAY_PRINT_ONLY=1 plumbing rather than inventing a new mode.

3. deploy/tests/test_deploy_artifacts.py

Add a grep-style assertion that the helper file contains the strings --map-users and idmap so the deploy artifact can't silently regress.

4. deploy/README.md

One-line mirror note: ckn-bw's bundles/left4me/ ships the helper verbatim, so no new bundle-side change is needed beyond updating the file. No new group, no new user, no new systemd unit. Flag this explicitly so the next deploy-to-prod step is "rebuild the helper file in ckn-bw, bw apply ovh.left4me".

Migration

No on-disk schema change. Existing overlays keep their current ownership (l4d2-sandbox-owned for script builds, left4me-owned for workshop builds). The mount helper picks the right path per lowerdir at next systemctl start <instance>.

Already-running instances on the test server pick up the change after a service restart. Live SourceMod sessions whose addons/sourcemod/logs/ copy-up is already broken in upperdir need a fix too: the upperdir entries are l4d2-sandbox:l4d2-sandbox 0755 from the previous broken copy-up. The helper doesn't touch upper/ on mount, so those stale entries persist.

Two safe migration options:

1. Manual: on the test server, stop server 2, rm -rf runtime/2/upper/left4dead2/addons/sourcemod/logs runtime/2/upper/left4dead2/addons/sourcemod/data, start it again. Copy-up will redo with idmapped lower → left4me-owned upper.
2. Automatic: have start_instance proactively delete known-SM writable dirs from upper/ if their uid is l4d2-sandbox. Out of scope for this change unless we hit it again.

Recommend option 1 — one-shot, no code change.

Verification

End-to-end test on left4.me:

1. bw apply ovh.left4me (or scp the updated helper into place).
2. Stop server 2: sudo systemctl stop left4me-server@2.
3. Clean the stale broken SM upperdir: sudo rm -rf /var/lib/left4me/runtime/2/upper/left4dead2/addons/sourcemod/{logs,data}.
4. Start server 2: sudo systemctl start left4me-server@2.
5. From inside left4me-overlay mount argv (check journalctl -u left4me-server@2 -o cat -n 50), confirm mount --bind --map-users=... was executed.
6. RCON to server 2, sm_cvar nb_update_frequency 0.0333. Expect no Platform returned error: "Permission denied" log line.
7. ls -ln /var/lib/left4me/runtime/2/upper/left4dead2/addons/sourcemod/logs/. Expect files owned by left4me's numeric uid.
8. sudo umount /var/lib/left4me/runtime/2/merged should still work; then verify runtime/2/idmap/ is cleaned up by ExecStopPost. Restart and confirm idempotent (no leftover binds).

Local tests:

• pytest l4d2host/tests/test_overlay_helper.py -q
• pytest deploy/tests/test_deploy_artifacts.py -q

Risks and edge cases

• Workshop overlay misidentification: a workshop overlay with a l4d2-sandbox-owned subdir somehow (e.g. partial migration) would get idmapped despite containing left4me-owned files. Files with other uids through an idmap appear as the overflow uid (nobody/65534), which would break reads. Mitigation: check ownership of the top-level overlay directory as the trigger, not file-by-file. If the top is sandbox-owned, trust the whole tree; if the top is left4me-owned, no idmap. This matches what each builder actually produces.
• installation/ and caches: always left4me-owned, never idmapped.
• Symlinks inside script overlays: idmap operates at the mount level, not per-inode. Symlink ownership translates the same as files. Targets inside the overlay resolve through the same mount. Targets outside (none in script overlays today; workshop ones don't take this code path) would not be affected.
• Mount namespace: the helper runs in PID 1's mount namespace via the unit's ExecStartPre=+nsenter .... Bind mounts created there persist until the matching ExecStopPost umount, exactly like the overlay mount itself.
• Crash mid-build: idmap binds are created only at mount time, not at build time. A crashed build leaves no orphan mounts.
• Crash mid-start (ExecStartPre fails between bind and overlay mount): systemd's Restart=on-failure re-invokes ExecStartPre. The helper checks os.path.ismount on each idmap target and skips already-mounted ones. Idempotent.
• runtime/<n>/idmap/ cleanup on _purge_instance: existing shutil.rmtree(runtime_dir) after disable_service already triggers the helper's umount sequence, which removes the idmap dir. No new code.
• util-linux flag form: prefer --map-users <inner>:<outer>:<count> and --map-groups (numeric uids/gids resolved by the helper) over the X-mount.idmap= mount-option syntax — clearer and easier to log.

Out of scope

• Web app uid split (l4d2-web separate from left4me) — orthogonal, rejected for this change.
• Gameserver uid split (separating the gameserver-runtime uid from left4me) — planned for a later session. One forward-compat coupling: the helper looks up pwd.getpwnam("left4me") as the in-mount target uid. When the gameserver moves to its own user (e.g. l4d2-game), change that one string. Everything else (script-sandbox uid, workshop builder uid, file-tree endpoint, idmap cleanup) is uid-agnostic.
• Replacing l4d2-sandbox with a different uid scheme — kept as defense in depth.
• Spec doc updates (including the bubblewrap → systemd-run wording correction in the existing script-overlay spec): dropped per user decision; this change ships plan-only.