left4me/docs/superpowers/plans/2026-05-15-deploy-dir-rethink.md

Deploy-dir architecture rethink — implementation plan

Context

Resolves the open questions in docs/superpowers/specs/2026-05-15-deploy-dir-rethink-design.md. After the 2026-05-15 script-consolidation work, deploy/ ended up half-canonical / half-historical: the privileged scripts were treated as load-bearing source-of-truth there, while sudoers/sysctl/env-templates stayed duplicated against ckn-bw, and the obsolete deploy-test-server.sh plus a pile of dead static unit files lingered. The shape worked but couldn't be described in two sentences.

This plan commits to the framing the user picked: deploy/ is a reference exemplar — readable enough that a fresh consumer (ckn-bw today, hypothetical docker/ansible/manual tomorrow) could build a deployment from it, but not the live source of truth for installed binaries. The privileged scripts are application-inherent code and move out of deploy/ to top-level scripts/{libexec,sbin}/. Dead code is deleted in the same pass. ckn-bw is updated to read scripts from the new location. The intended outcome: deploy/ shrinks to README + example configs + a couple of curated example units, the rules for "what goes here" fit in two sentences, and the cross-repo install path becomes self-explanatory.

End state

left4me/
  scripts/
    libexec/
      left4me-overlay              # 244-line Python helper (mount/umount)
      left4me-script-sandbox       # 109-line bash (systemd-run sandbox)
      left4me-systemctl            # 44-line sh wrapper
      left4me-journalctl           # 53-line sh wrapper
    sbin/
      left4me                      # 17-line admin CLI wrapper
    tests/
      test_overlay.py
      test_script_sandbox.py
      test_systemctl_helper.py
      test_journalctl_helper.py
      test_sudoers_grants.py       # tests the contract between scripts and sudoers
  deploy/                          # REFERENCE ONLY — see deploy/README.md
    README.md                      # rewritten: explains target layout, points at scripts/
    files/
      etc/
        sudoers.d/left4me                # example, ckn-bw ships its own verbatim copy
        sysctl.d/99-left4me.conf         # example
        left4me/sandbox-resolv.conf      # example
      usr/local/lib/systemd/system/
        left4me-server@.service          # curated example of what ckn-bw's reactor emits
        left4me-web.service              # curated example
        left4me-workshop-refresh.service # curated example
        left4me-workshop-refresh.timer   # curated example
        l4d2-game.slice                  # curated example
        l4d2-build.slice                 # curated example
    templates/etc/left4me/
      host.env                           # example, ckn-bw renders its own mako version
      web.env.template
    tests/
      test_example_units.py              # slimmed: just locks down the curated examples
  l4d2host/                        # unchanged
  l4d2web/                         # unchanged
  docs/

Step-by-step

1. Create scripts/ and move helpers

• mkdir -p scripts/libexec scripts/sbin scripts/tests
• git mv the four live helpers and the admin CLI:
  • deploy/files/usr/local/libexec/left4me/left4me-overlay → scripts/libexec/left4me-overlay
  • deploy/files/usr/local/libexec/left4me/left4me-script-sandbox → scripts/libexec/left4me-script-sandbox
  • deploy/files/usr/local/libexec/left4me/left4me-systemctl → scripts/libexec/left4me-systemctl
  • deploy/files/usr/local/libexec/left4me/left4me-journalctl → scripts/libexec/left4me-journalctl
  • deploy/files/usr/local/sbin/left4me → scripts/sbin/left4me
• The scripts' contents are unchanged. Every install-target path inside them (/usr/local/libexec/left4me/..., /etc/left4me/..., /var/lib/left4me/...) stays exactly as is — those are runtime paths, not source-tree paths.

2. Delete dead code

• git rm (truly obsolete; replacements live elsewhere or feature was retired):
  • deploy/files/usr/local/libexec/left4me/left4me-apply-cake — CAKE migrated to systemd-networkd via network/<iface>/cake node metadata in ckn-bw.
  • deploy/files/usr/local/lib/systemd/system/left4me-cake.service — same reason.
  • deploy/files/etc/left4me/cake.env — bandwidth lives in node metadata, not an env file.
  • deploy/files/usr/local/lib/systemd/system/left4me-nft-mark.service — central bundles/nftables/ consumes the rules now.
  • deploy/files/usr/local/lib/left4me/nft/left4me-mark.nft — same. After the delete, the now-empty deploy/files/usr/local/lib/left4me/ and its nft/ child disappear (git doesn't track empty dirs).
  • deploy/deploy-test-server.sh — superseded by bw apply; content survives in git history.
• Do NOT delete deploy/files/usr/local/lib/systemd/system/left4me-workshop-refresh.{service,timer}. The workshop-refresh job is live (invokes flask workshop-refresh, defined in l4d2web/cli.py); ckn-bw's reactor emits these on production. They stay as curated examples, same category as left4me-server@.service / left4me-web.service / the slices. (This corrects the framing in docs/superpowers/specs/2026-05-15-deploy-dir-rethink-design.md and item 2 of docs/superpowers/specs/2026-05-15-janitorial-cleanup.md, both of which lumped workshop-refresh together with truly-dead units.)
• Stale __pycache__ dirs under deploy/files/usr/local/libexec/left4me/ are deleted by the moves in step 1.

3. Split and relocate deploy/tests/test_deploy_artifacts.py

The current file (~880 lines) is doing four jobs. Split as follows; do not duplicate tests across files.

Concrete sequence to preserve git history where it counts:

1. git mv deploy/tests/test_deploy_artifacts.py deploy/tests/test_example_units.py — single rename, history follows via git log --follow.
2. In the renamed file, delete every test except the "Keep in deploy/tests/test_example_units.py" list below. The kept tests track the unit/sysctl/env-template examples, which is what deploy/tests/ will mean afterwards.
3. Create new scripts/tests/*.py files (and conftest.py) by writing them fresh — pasting the relevant test functions across. The extracted tests lose direct rename history, but blame against the new files still resolves to the originals one git ref back; acceptable tradeoff.

Move to scripts/tests/ (tests of script behavior + the sudoers contract that gates the scripts):

• scripts/tests/test_overlay.py — test_overlay_helper_is_python_with_strict_validation, test_overlay_helper_mount_is_idempotent_when_already_mounted
• scripts/tests/test_script_sandbox.py — test_script_sandbox_helper_present, test_script_sandbox_helper_passes_shell_syntax_check, test_script_sandbox_helper_invokes_systemd_run_with_hardening, test_script_sandbox_uses_idmap_staging, test_script_sandbox_in_build_slice_with_oom_adjust, test_script_sandbox_helper_validates_overlay_id, test_script_sandbox_helper_dry_run_mode
• scripts/tests/test_systemctl_helper.py — test_systemctl_helper_passes_shell_syntax_check_and_rejects_bad_args
• scripts/tests/test_journalctl_helper.py — test_journalctl_helper_passes_shell_syntax_check_and_rejects_bad_args
• scripts/tests/test_helpers_use_fixed_paths.py — test_helpers_use_fixed_system_tool_paths_not_sudo_path
• scripts/tests/test_sudoers_grants.py — test_sudoers_allows_only_left4me_helpers_not_raw_system_tools (still reads deploy/files/etc/sudoers.d/left4me as the canonical example; comment why)

The ROOT/DEPLOY path-prefix constants in each file get rewritten so SCRIPTS = Path(__file__).resolve().parents[2] / "scripts" and helpers resolve to SCRIPTS / "libexec/left4me-overlay" etc. Shared helpers (_fake_command, _env_with_fake_commands) move into scripts/tests/conftest.py.

Keep in deploy/tests/test_example_units.py (locks down the curated examples; renamed from the current file):

• test_global_unit_files_exist_at_product_level_paths
• test_web_unit_contains_required_runtime_contract
• test_server_unit_contains_required_runtime_contract
• test_server_unit_mounts_overlay_via_exec_start_pre
• test_server_unit_unmounts_overlay_via_exec_stop_post
• test_server_unit_contains_perf_baseline_directives
• test_l4d2_game_slice_exists_with_high_weights
• test_l4d2_build_slice_exists_with_low_weights
• test_sysctl_conf_present_with_perf_settings
• test_env_templates_contain_required_defaults
• test_sandbox_resolv_conf_exists

Add a top-of-file docstring: "These tests lock down the curated examples kept in deploy/files/ for reference. The production units are emitted by ckn-bw's reactor in bundles/left4me/metadata.py; when reactor output drifts intentionally, update the examples here too."

Delete entirely (target removed or no longer load-bearing):

• All test_deploy_script_* tests (12 tests; deploy-test-server.sh is gone)
• test_globals_refresh_units_removed — file already deleted; nothing to lock down
• test_nft_mark_file_marks_left4me_udp_with_dscp_ef_and_priority, test_nft_mark_unit_loads_and_clears_left4me_table — nft-mark moved to central nftables bundle
• test_cake_env_template_documents_required_knobs, test_apply_cake_helper_supports_apply_and_clear_modes, test_apply_cake_helper_passes_shell_syntax_check, test_cake_unit_runs_helper_in_apply_and_clear_modes — CAKE moved to systemd-networkd
• test_deploy_script_installs_overlay_helper_with_executable_mode, test_deploy_script_installs_script_sandbox_helper — install responsibility now lives in ckn-bw's bundle, not in any left4me-side script

Final file count: scripts/tests/ gets 6 files, deploy/tests/test_example_units.py is one file, deploy/tests/test_deploy_artifacts.py is gone (renamed).

4. Rewrite deploy/README.md

Reframe the top of the file as: "This directory is a reference exemplar. The canonical deploy is ckn-bw's bundles/left4me/ (run bw apply ovh.left4me). Files under deploy/files/ and deploy/templates/ are readable examples — not the binaries / configs ckn-bw actually installs. Read them to understand the target layout if you're building a fresh deployment by other means."

Update the file/status table:

• Drop rows for files that no longer exist (apply-cake, cake.service, cake.env, nft-mark., workshop-refresh.).
• Drop the deploy-test-server.sh row.
• For the privileged-scripts rows, change files/usr/local/libexec/left4me/... → (moved to scripts/libexec/, installed by ckn-bw's install_left4me_scripts action); same for the sbin row.
• Mark the remaining files/etc/... and files/usr/local/lib/systemd/system/... entries explicitly as example: ckn-bw ships its own verbatim copies of the configs, its reactor emits the units.

Keep the "Target Layout" / "Runtime User" / "Overlay References" / "Performance Tuning" sections — they're useful reference prose. Strip the "Running A Test Deployment" / "Admin Bootstrap" sections that refer to the deleted shell installer; replace with a one-paragraph pointer to ckn-bw.

5. ckn-bw cross-repo update

The install_left4me_scripts action in bundles/left4me/items.py currently reads from /opt/left4me/src/deploy/files/usr/local/{libexec,sbin}/. Update it to read from /opt/left4me/src/scripts/{libexec,sbin}/. The install target is unchanged (/usr/local/libexec/left4me/, /usr/local/sbin/left4me), so nothing on the deployed host moves.

This is a separate PR in the ckn-bw repo. It must land at the same time as the left4me move — the install action depends on the source paths existing. Coordination:

1. Open both PRs simultaneously.
2. Merge order: left4me first (scripts exist at the new path in /opt/left4me/src/ only after a fresh git_deploy), then ckn-bw, then bw apply ovh.left4me.
3. Alternative: have the ckn-bw PR fall back to the old path if the new path doesn't exist (one extra glob); decide during ckn-bw review whether the complexity is worth the looser coupling. Default: no fallback, coordinate the merges.

Verification on the deploy target: after bw apply, the files under /usr/local/libexec/left4me/ and /usr/local/sbin/left4me should be byte-identical to before. Sudoers, services, the web app: all unchanged.

6. Mark adjacent specs / docs as resolved

• docs/superpowers/specs/2026-05-15-deploy-dir-rethink-design.md: prepend a **Resolved 2026-05-15 by docs/superpowers/plans/…</plan-name>.md.** line at the top. Leave the body intact for archaeology.
• docs/superpowers/specs/2026-05-15-janitorial-cleanup.md: cross out items 1, 5, 6 (now handled here). Item 2 needs a rewrite — the framing "all static unit files are obsolete drift" was wrong; the live reactor-emitted set (server@, web, workshop-refresh.{service,timer}, l4d2-{game,build}.slice) stays in deploy/files/ as curated examples. The truly-dead two (left4me-cake.service, left4me-nft-mark.service) are already deleted by this plan, so item 2 collapses to "no remaining work."
• No memory file changes needed; the project state captured here is structural and re-derivable from deploy/README.md after the rewrite lands.

7. Rollback notes

If bw apply ovh.left4me against the test server breaks something after the cross-repo merge:

1. Revert the ckn-bw install_left4me_scripts action change to the old source path (/opt/left4me/src/deploy/files/usr/local/{libexec,sbin}/). Re-apply.
2. The left4me side never needs reverting in isolation — the scripts at the new path are byte-identical to the old ones, so a stale ckn-bw install action against a new left4me checkout would fail at install -t (source path missing). That failure is loud and safe: nothing on the deployed system gets modified.
3. The only foot-gun is partial rollout: ckn-bw updated but left4me not yet checked out at the right revision. The git_deploy step pins the revision, so as long as the two PRs reference compatible commits, the deployed /opt/left4me/src/ always matches the action's expectation.

What does NOT change

• Runtime install-target paths (/usr/local/libexec/left4me/..., /usr/local/sbin/left4me) — every reference inside l4d2host/service_control.py:7-8, l4d2web/services/overlay_builders.py:34, the sudoers file, and the systemd units stays the same.
• The Python packages l4d2host/ and l4d2web/.
• ckn-bw's bundles for sudoers / sysctl / sandbox-resolv.conf — those keep their own verbatim copies (the user picked "deploy/ keeps configs as examples; duplication-with-ckn-bw is OK because deploy/ is explicitly reference"). Janitoring the duplication is not in scope for this plan.
• The Mako env templates in ckn-bw — they stay where they are, since they need bw's metadata access for rendering.
• The recent overlay-idmap / script-sandbox idmap-staging work — untouched.

Critical files (jump points for the implementor)

• deploy/tests/test_deploy_artifacts.py — the source for the test split (lines 20-32 are the path constants; tests grouped roughly by helper from line 138 onward)
• deploy/README.md — full rewrite of the top section, partial rewrite of the table
• l4d2host/service_control.py:7-8 — verify install-target paths unchanged (sanity)
• l4d2web/services/overlay_builders.py:34 — same
• deploy/files/etc/sudoers.d/left4me — sanity-check that no path inside changed
• deploy/files/usr/local/lib/systemd/system/{left4me-server@.service,left4me-web.service,l4d2-{game,build}.slice} — survive as curated examples
• ckn-bw repo: bundles/left4me/items.py — the install_left4me_scripts action (separate PR)

Verification

End-to-end:

1. Source-tree consistency. find scripts deploy -type f | sort matches the layout in "End state" above (modulo __pycache__).
2. All tests pass locally. From the repo root: pytest scripts/tests/ deploy/tests/ l4d2host/tests/ l4d2web/tests/ — every test passes. Specifically verify scripts/tests/test_sudoers_grants.py still reads deploy/files/etc/sudoers.d/left4me correctly (path constant points across the dir boundary).
3. Shell syntax checks. The split tests should still run sh -n / bash -n against the moved scripts; no script edits means no syntax regressions, but the test paths must resolve.
4. No accidental application breakage. grep -rn '/usr/local/libexec/left4me\|/usr/local/sbin/left4me' l4d2host l4d2web returns the same hits as before (paths are install-target, source moves don't affect them).
5. ckn-bw dry-run. Once the ckn-bw PR is up, bw apply --dry-run ovh.left4me from the ckn-bw repo: the diff should show no changes to files under /usr/local/libexec/left4me/ or /usr/local/sbin/left4me (byte-identical content via the new path).
6. Production apply. bw apply ovh.left4me against the real test server. After apply: systemctl status left4me-web.service is green, starting a game server via the web UI still works (overlay mount → srcds_run → unmount on stop), running an overlay build script through the sandbox still works.

Out of scope (handled elsewhere or deferred)

• The Mako template duplication in ckn-bw — separate cleanup; the templates legitimately need bw's metadata access.
• The 1/2/3-user uid-split decision — docs/superpowers/specs/2026-05-15-user-uid-split-design.md.
• The script-sandbox → systemd template unit refactor — docs/superpowers/specs/2026-05-15-build-overlay-unit-design.md.
• Remaining janitorial items: item 3 (bubblewrap→systemd-run doc drift), item 4 (stale gameserver-side idmap binds), calendar reminder for SM 1.13 stable. Items 1, 2 (partial — see step 6), 5, 6 are subsumed here.
• Rewriting the shell helpers in Python / packaging them as console_scripts — explicitly rejected in the recent script-consolidation plan (egg-info + TOCTOU privilege concerns).
• Historical references inside docs/superpowers/plans/* and docs/superpowers/specs/* to deploy/files/... or deploy-test-server.sh paths. Those are time-stamped snapshots of past sessions; they don't get rewritten when the underlying tree moves.