left4me/docs/superpowers/specs/2026-05-15-deployment-responsibility-design.md

Deployment responsibility — design

Status

Shipped 2026-05-15. All five migration steps landed and verified on ovh.left4me. Implementation plan: docs/superpowers/plans/2026-05-15-deployment-responsibility.md.

Context

Trace: 2026-05-06-left4me-deployment-design.md established the original model — left4me's deploy/files/ mirrors target filesystem paths; ckn-bw integrates. The hardening refactor (2026-05-15-hardening-refactor-design.md) landed inline-in-reactor as an explicit tradeoff and queued the responsibility question for this brainstorm (handoff: 2026-05-15-handoff-deployment-responsibility.md). The runtime-state relocation (2026-05-15-runtime-state-relocation-design.md) made /opt/left4me/src root-owned, which is the prerequisite that makes target-side symlinks into the checkout safe — left4me cannot rewrite its own deployment artifacts at runtime.

This design picks a narrow, conservative line. Application-shape artifacts that are static across hosts move to left4me's deploy/ tree and are delivered to the target via target-side symlinks. Per-host shape (CPU pinning, gunicorn workers, env file values) stays bw-managed. The base systemd unit bodies stay bw-managed too — they encode per-host values (workers, threads, CPU set) that are awkward to parameterize cleanly, and ckn-bw is already the right place for that computation.

The wedge between "moves" and "stays" is threat model knowledge vs. host shape. The hardening profile is the security knowledge of the application; the base unit body is the operational shape of the host. Different repos.

Scope

Moves to left4me/deploy/, delivered via target-side symlinks

Artifact	Source path	Symlink target
Hardening drop-in for left4me-web	deploy/files/etc/systemd/system/left4me-web.service.d/10-hardening.conf (NEW)	/etc/systemd/system/left4me-web.service.d/10-hardening.conf
Hardening drop-in for left4me-server@	deploy/files/etc/systemd/system/left4me-server@.service.d/10-hardening.conf (NEW)	same pattern
Sudoers	deploy/files/etc/sudoers.d/left4me (exists)	/etc/sudoers.d/left4me
Sysctl drop-in (absorbs ptrace_scope)	deploy/files/etc/sysctl.d/99-left4me.conf (exists; one line added)	/etc/sysctl.d/99-left4me.conf
Privileged helpers (libexec/)	deploy/scripts/libexec/* (relocated from scripts/libexec/)	/usr/local/libexec/left4me/<name>
Privileged helpers (sbin/)	deploy/scripts/sbin/* (relocated from scripts/sbin/)	/usr/local/sbin/<name>

All symlinks are created by bw symlinks{} items in bundles/left4me/items.py. git_deploy:/opt/left4me/src triggers systemctl daemon-reload (for unit drop-ins) and sysctl --system (for sysctl) so changes to the symlink-target content propagate even though the symlink path itself doesn't change.

Stays bw-managed

• Base unit bodies (left4me-web.service, left4me-server@.service): emitted by the systemd/units reactor in bundles/left4me/metadata.py. These encode per-host values (gunicorn workers/threads, CPU pinning, instance bind paths). Pulling them into left4me would require either templating or env-var parameterization that doesn't cleanly cover everything (systemd doesn't substitute env vars in non-Exec directives like SocketBindAllow=).
• Slice units (l4d2-game.slice, l4d2-build.slice) and cpuset drop-ins (system.slice.d/99-left4me-cpuset.conf, user.slice.d/99-left4me-cpuset.conf): all encode per-host CPU pinning. Reactor stays.
• host.env.mako, web.env.mako: per-host secret + scalar templating. Stays.
• nginx/vhosts, nftables/input, nftables/output: bundle abstractions (letsencrypt auto-population, set-merge) add real value over raw files.
• systemd-timers/left4me-workshop-refresh: same — bundle synthesizes the .timer + .service from the metadata dict.
• Action chains: git_deploy, pip_install, alembic_upgrade, seed_overlays, create_venv, pip_upgrade, install_steamcmd. Stays.
• directory, user, group items: must exist before git_deploy runs.
• apt/packages, backup/paths defaults. Stays.

Stays in left4me as reference fixtures (no change)

deploy/files/usr/local/lib/systemd/system/*.{service,slice} — reference units matched against the live form by deploy/tests/test_deploy_artifacts.py. Base units stay bw-emitted, so reference-vs-live assertion stays valid. Reference units should not include hardening directives once the drop-in extraction lands; the live form's hardening lives in the drop-in, not the base unit.

Repo layout (left4me)

deploy/
  files/
    etc/sudoers.d/left4me
    etc/sysctl.d/99-left4me.conf
    etc/systemd/system/left4me-web.service.d/10-hardening.conf       # NEW
    etc/systemd/system/left4me-server@.service.d/10-hardening.conf   # NEW
    etc/left4me/sandbox-resolv.conf                                  # unchanged
    usr/local/lib/systemd/system/*.{service,slice}                   # reference (unchanged shape)
  scripts/                                                           # moves in from scripts/
    libexec/{left4me-overlay,left4me-systemctl,left4me-journalctl,left4me-script-sandbox}
    sbin/<wrappers>
  tests/

Mechanism: target-side symlinks

bw symlinks{} item type. One entry per artifact:

symlinks = {
    '/etc/sudoers.d/left4me': {
        'target': '/opt/left4me/src/deploy/files/etc/sudoers.d/left4me',
        'owner': 'root',
        'group': 'root',
        'needs': ['git_deploy:/opt/left4me/src'],
    },
    '/etc/sysctl.d/99-left4me.conf': {
        'target': '/opt/left4me/src/deploy/files/etc/sysctl.d/99-left4me.conf',
        'owner': 'root',
        'group': 'root',
        'needs': ['git_deploy:/opt/left4me/src'],
        'triggers': ['action:left4me_sysctl_reload'],
    },
    '/etc/systemd/system/left4me-web.service.d/10-hardening.conf': {
        'target': '/opt/left4me/src/deploy/files/etc/systemd/system/left4me-web.service.d/10-hardening.conf',
        'needs': [
            'directory:/etc/systemd/system/left4me-web.service.d',
            'git_deploy:/opt/left4me/src',
        ],
        'triggers': ['action:systemd_daemon_reload'],
    },
    # …same for left4me-server@.service.d/10-hardening.conf
    # …same for each script in /usr/local/{libexec/left4me,sbin}/
}

Drop-in directories (*.service.d/) need explicit directory: items in items.py (the bw systemd bundle does not create them automatically for symlink-only drop-ins). Mode 0755, owner root:root.

bw fires the symlink's triggers: when the symlink itself changes (path/target update). It does not fire when the symlink's target content changes — that's still a git_deploy: event. So both wirings are needed: every symlink declares needs: ['git_deploy:/opt/left4me/src'], and ckn-bw declares triggered_by: [git_deploy:/opt/left4me/src] actions for the global reloads (daemon-reload, sysctl --system).

Per-artifact details

Hardening drop-ins

Extract from HARDENING_COMMON, HARDENING_SERVER, HARDENING_WEB Python dicts in bundles/left4me/metadata.py into static .conf files:

# deploy/files/etc/systemd/system/left4me-web.service.d/10-hardening.conf
[Service]
ProtectProc=invisible
ProcSubset=pid
ProtectKernelTunables=true
…
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete

Per-directive comments documenting why each directive is set the way it is (sudo-incompatibility carve-outs for web; i386 amendment and PrivatePIDs rationale for server@) should live inline as # comments. Today those rationale comments live in the Python source; they need to come along.

After extraction, the reactor's emitted unit bodies drop the **HARDENING_WEB / **HARDENING_SERVER splat. The reactor still emits the base unit and is responsible for everything except the hardening profile.

Sudoers

Today: identical content in left4me/deploy/files/etc/sudoers.d/left4me and ckn-bw/bundles/left4me/files/etc/sudoers.d/left4me. The bw item sources from the ckn-bw copy.

After: bw symlinks{} item; delete the ckn-bw copy and the bw files{} entry. The test_with: 'visudo -cf {}' semantics don't apply to symlinks; tested instead on commit in left4me CI (a test_sudoers.py that runs visudo -cf against the live file).

Sysctl drop-in + ptrace_scope absorption

Today: same dual-copy story as sudoers, plus kernel.yama.ptrace_scope exists as a metadata default (sysctl/kernel/yama/ptrace_scope: '2') that gets deployed via bundles/sysctl/ into a separate file.

After: append kernel.yama.ptrace_scope = 2 to deploy/files/etc/sysctl.d/99-left4me.conf. Delete the metadata entry. Delete the bw files{} entry + ckn-bw mirror; replace with symlink. bundles/sysctl/ no longer renders anything for left4me; all left4me sysctl tuning lives in the one drop-in.

Privileged scripts

Done (Task 4): deploy/scripts/libexec/, deploy/scripts/sbin/ under deploy/ for layout consistency. install_left4me_scripts copy-action replaced by target-side symlinks from /usr/local/libexec/left4me/ and /usr/local/sbin/ into the checkout at /opt/left4me/src/deploy/scripts/{libexec,sbin}/.

Sudo follows symlinks. With /opt/left4me/src root-owned, the symlink target is root-owned, and sudo's Cmnd_Alias path matching sees the original /usr/local/{libexec,sbin}/<name> path.

Reference units in deploy/files/

No structural change. Remove hardening directives from the reference files in lockstep with extracting them into the drop-ins (otherwise test_deploy_artifacts.py sees the reference unit with hardening inline but the live unit without). The reference file then represents "the base unit ckn-bw emits"; the drop-in represents "the hardening profile left4me ships".

Migration order

Each step is an independent landable PR.

1. Canary — sysctl consolidation.

   • Add kernel.yama.ptrace_scope = 2 to deploy/files/etc/sysctl.d/99-left4me.conf.
   • Delete defaults['sysctl']['kernel']['yama']['ptrace_scope'] from bundles/left4me/metadata.py.
   • Delete bundles/left4me/files/etc/sysctl.d/99-left4me.conf (the verbatim mirror).
   • Replace the bw files{} entry with a symlinks{} entry pointing at the checkout.
   • Verify: sysctl kernel.yama.ptrace_scope reads 2; bw apply idempotent.

2. Hardening drop-ins.

   • Create deploy/files/etc/systemd/system/left4me-web.service.d/10-hardening.conf and …/left4me-server@.service.d/10-hardening.conf from the HARDENING_* dicts.
   • Remove **HARDENING_WEB / **HARDENING_SERVER splats from the reactor; delete the three constants.
   • Remove hardening directives from the reference units in deploy/files/usr/local/lib/systemd/system/.
   • Add directory:/etc/systemd/system/left4me-{web,server@}.service.d items + symlinks for the drop-ins.
   • Wire systemctl daemon-reload to fire on git_deploy:/opt/left4me/src.
   • Verify: systemctl show -p ProtectSystem,ProtectKernelTunables,PrivateUsers,… left4me-web.service left4me-server@1.service matches the pre-extraction values (full hardening test plan rerun is the gold standard).

3. Sudoers.

   • Replace bw files{} entry with symlinks{}.
   • Delete bundles/left4me/files/etc/sudoers.d/left4me.
   • Add left4me CI test running visudo -cf on the file.
   • Verify: sudo -l -U left4me lists the expected commands; gameserver start via the web app still works.

4. Privileged scripts.

   • git mv left4me/scripts left4me/deploy/scripts.
   • Update any references (commit hooks, docs).
   • Replace actions['install_left4me_scripts'] with symlinks{} items, one per script. Drop the action.
   • Update git_deploy: triggers: to remove action:install_left4me_scripts.
   • Verify: sudo /usr/local/libexec/left4me/left4me-overlay status 1 still works; gameserver lifecycle (start/stop) still works.

5. Cleanup.

   • Prune gunicorn_workers / gunicorn_threads metadata defaults if they end up referenced only by web.env.mako (they do today; keep the metadata, they're real per-host values).
   • Update deploy/README.md to describe the new layout (deploy/files = symlink source-of-truth; deploy/scripts = same for helpers).
   • Update bundles/left4me/README.md to describe the new symlink-based delivery model.

Sequence vs. build-overlay-unit refactor

This design lands before the build-overlay-unit refactor (2026-05-15-build-overlay-unit-design.md). Reasons:

• build-overlay-unit introduces a dispatcher unit template; its hardening profile should live as a drop-in alongside the dispatcher from the start, using the pattern this design establishes.
• The reactor surgery in step 2 (removing HARDENING_* splats) is cleaner against today's reactor than against a reactor that's also being reshaped for the build-overlay-unit work.

Verification (end-to-end)

After all five steps land and bw apply is idempotent on ovh.left4me:

1. systemctl show -p ProtectSystem,PrivateUsers,SystemCallFilter,… left4me-web.service left4me-server@1.service matches the hardening test plan's reference values (run the relevant tests from docs/superpowers/specs/2026-05-15-hardening-test-plan.md).
2. sysctl kernel.yama.ptrace_scope net.core.rmem_max net.ipv4.tcp_congestion_control returns expected values.
3. sudo -l -U left4me reports the same allowed commands as before.
4. ls -la /etc/sudoers.d/left4me /etc/sysctl.d/99-left4me.conf /etc/systemd/system/left4me-*.service.d/10-hardening.conf /usr/local/libexec/left4me/* /usr/local/sbin/<wrappers> shows symlinks into /opt/left4me/src/deploy/....
5. A gameserver round-trip (start via web app → cvar inspect → stop) succeeds.
6. bw verify ovh.left4me reports no drift.

Out of scope

• Moving base unit bodies into left4me. Per-host shape stays reactor-emitted.
• AppArmor profiles (deferred from the defenses survey).
• Reshaping the bw files{} items for host.env.mako / web.env.mako — they need mako templating with metadata context, which ckn-bw is the right place for.
• The build-overlay-unit refactor itself. Lands separately on top of this.

Pointers

• Handoff (this brainstorm's framing): docs/superpowers/specs/2026-05-15-handoff-deployment-responsibility.md
• Prereq (runtime state relocation + non-editable install, shipped): docs/superpowers/specs/2026-05-15-runtime-state-relocation-design.md
• Original deployment design (the model being reaffirmed for application-shape artifacts): docs/superpowers/specs/2026-05-06-left4me-deployment-design.md
• Hardening refactor design (the inline-in-reactor approach this design supersedes for hardening): docs/superpowers/specs/2026-05-15-hardening-refactor-design.md
• Hardening test plan (reference for step-2 verification): docs/superpowers/specs/2026-05-15-hardening-test-plan.md
• ckn-bw left4me bundle: ~/Projekte/ckn-bw/bundles/left4me/