left4me/docs/superpowers/specs/2026-05-15-deployment-responsibility-design.md

# Deployment responsibility — design

## Status

**Shipped 2026-05-15.** All five migration steps landed and verified on
ovh.left4me. Implementation plan:
`docs/superpowers/plans/2026-05-15-deployment-responsibility.md`.

## Context

Trace: `2026-05-06-left4me-deployment-design.md` established the original
model — left4me's `deploy/files/` mirrors target filesystem paths;
ckn-bw integrates. The hardening refactor
(`2026-05-15-hardening-refactor-design.md`) landed *inline-in-reactor*
as an explicit tradeoff and queued the responsibility question for this
brainstorm (handoff: `2026-05-15-handoff-deployment-responsibility.md`).
The runtime-state relocation
(`2026-05-15-runtime-state-relocation-design.md`) made
`/opt/left4me/src` root-owned, which is the prerequisite that makes
target-side symlinks into the checkout safe — left4me cannot rewrite
its own deployment artifacts at runtime.

This design picks a narrow, conservative line. Application-shape
artifacts that are static across hosts move to left4me's `deploy/`
tree and are delivered to the target via **target-side symlinks**.
Per-host shape (CPU pinning, gunicorn workers, env file values) stays
bw-managed. The base systemd unit bodies stay bw-managed too — they
encode per-host values (workers, threads, CPU set) that are awkward to
parameterize cleanly, and ckn-bw is already the right place for that
computation.

The wedge between "moves" and "stays" is **threat model knowledge vs.
host shape**. The hardening profile is the security knowledge of the
application; the base unit body is the operational shape of the host.
Different repos.

## Scope

### Moves to left4me/deploy/, delivered via target-side symlinks

| Artifact | Source path | Symlink target |
|---|---|---|
| Hardening drop-in for `left4me-web` | `deploy/files/etc/systemd/system/left4me-web.service.d/10-hardening.conf` (NEW) | `/etc/systemd/system/left4me-web.service.d/10-hardening.conf` |
| Hardening drop-in for `left4me-server@` | `deploy/files/etc/systemd/system/left4me-server@.service.d/10-hardening.conf` (NEW) | same pattern |
| Sudoers | `deploy/files/etc/sudoers.d/left4me` (exists) | `/etc/sudoers.d/left4me` |
| Sysctl drop-in (absorbs `ptrace_scope`) | `deploy/files/etc/sysctl.d/99-left4me.conf` (exists; one line added) | `/etc/sysctl.d/99-left4me.conf` |
| Privileged helpers (`libexec/`) | `deploy/scripts/libexec/*` (relocated from `scripts/libexec/`) | `/usr/local/libexec/left4me/<name>` |
| Privileged helpers (`sbin/`) | `deploy/scripts/sbin/*` (relocated from `scripts/sbin/`) | `/usr/local/sbin/<name>` |

All symlinks are created by bw `symlinks{}` items in
`bundles/left4me/items.py`. `git_deploy:/opt/left4me/src` triggers
`systemctl daemon-reload` (for unit drop-ins) and `sysctl --system`
(for sysctl) so changes to the symlink-target content propagate even
though the symlink path itself doesn't change.

### Stays bw-managed

- **Base unit bodies** (`left4me-web.service`, `left4me-server@.service`):
  emitted by the `systemd/units` reactor in
  `bundles/left4me/metadata.py`. These encode per-host values
  (gunicorn workers/threads, CPU pinning, instance bind paths). Pulling
  them into left4me would require either templating or env-var
  parameterization that doesn't cleanly cover everything (systemd
  doesn't substitute env vars in non-Exec directives like
  `SocketBindAllow=`).
- **Slice units** (`l4d2-game.slice`, `l4d2-build.slice`) and cpuset
  drop-ins (`system.slice.d/99-left4me-cpuset.conf`,
  `user.slice.d/99-left4me-cpuset.conf`): all encode per-host CPU
  pinning. Reactor stays.
- **`host.env.mako`, `web.env.mako`**: per-host secret + scalar
  templating. Stays.
- **`nginx/vhosts`, `nftables/input`, `nftables/output`**: bundle
  abstractions (letsencrypt auto-population, set-merge) add real value
  over raw files.
- **`systemd-timers/left4me-workshop-refresh`**: same — bundle
  synthesizes the `.timer` + `.service` from the metadata dict.
- **Action chains**: `git_deploy`, `pip_install`, `alembic_upgrade`,
  `seed_overlays`, `create_venv`, `pip_upgrade`, `install_steamcmd`.
  Stays.
- **`directory`, `user`, `group`** items: must exist before
  `git_deploy` runs.
- **`apt/packages`, `backup/paths`** defaults. Stays.

### Stays in left4me as reference fixtures (no change)

`deploy/files/usr/local/lib/systemd/system/*.{service,slice}` —
reference units matched against the live form by
`deploy/tests/test_deploy_artifacts.py`. Base units stay bw-emitted,
so reference-vs-live assertion stays valid. Reference units should
**not** include hardening directives once the drop-in extraction
lands; the live form's hardening lives in the drop-in, not the base
unit.

## Repo layout (left4me)

```
deploy/
  files/
    etc/sudoers.d/left4me
    etc/sysctl.d/99-left4me.conf
    etc/systemd/system/left4me-web.service.d/10-hardening.conf       # NEW
    etc/systemd/system/left4me-server@.service.d/10-hardening.conf   # NEW
    etc/left4me/sandbox-resolv.conf                                  # unchanged
    usr/local/lib/systemd/system/*.{service,slice}                   # reference (unchanged shape)
  scripts/                                                           # moves in from scripts/
    libexec/{left4me-overlay,left4me-systemctl,left4me-journalctl,left4me-script-sandbox}
    sbin/<wrappers>
  tests/
```

## Mechanism: target-side symlinks

bw `symlinks{}` item type. One entry per artifact:

```python
symlinks = {
    '/etc/sudoers.d/left4me': {
        'target': '/opt/left4me/src/deploy/files/etc/sudoers.d/left4me',
        'owner': 'root',
        'group': 'root',
        'needs': ['git_deploy:/opt/left4me/src'],
    },
    '/etc/sysctl.d/99-left4me.conf': {
        'target': '/opt/left4me/src/deploy/files/etc/sysctl.d/99-left4me.conf',
        'owner': 'root',
        'group': 'root',
        'needs': ['git_deploy:/opt/left4me/src'],
        'triggers': ['action:left4me_sysctl_reload'],
    },
    '/etc/systemd/system/left4me-web.service.d/10-hardening.conf': {
        'target': '/opt/left4me/src/deploy/files/etc/systemd/system/left4me-web.service.d/10-hardening.conf',
        'needs': [
            'directory:/etc/systemd/system/left4me-web.service.d',
            'git_deploy:/opt/left4me/src',
        ],
        'triggers': ['action:systemd_daemon_reload'],
    },
    # …same for left4me-server@.service.d/10-hardening.conf
    # …same for each script in /usr/local/{libexec/left4me,sbin}/
}
```

Drop-in directories (`*.service.d/`) need explicit `directory:` items
in `items.py` (the bw systemd bundle does not create them
automatically for symlink-only drop-ins). Mode `0755`, owner
`root:root`.

bw fires the symlink's `triggers:` when **the symlink itself
changes** (path/target update). It does *not* fire when the symlink's
*target content* changes — that's still a `git_deploy:` event. So
both wirings are needed: every symlink declares
`needs: ['git_deploy:/opt/left4me/src']`, and ckn-bw declares
`triggered_by: [git_deploy:/opt/left4me/src]` actions for the global
reloads (`daemon-reload`, `sysctl --system`).

## Per-artifact details

### Hardening drop-ins

Extract from `HARDENING_COMMON`, `HARDENING_SERVER`, `HARDENING_WEB`
Python dicts in `bundles/left4me/metadata.py` into static `.conf`
files:

```ini
# deploy/files/etc/systemd/system/left4me-web.service.d/10-hardening.conf
[Service]
ProtectProc=invisible
ProcSubset=pid
ProtectKernelTunables=true
…
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@debug @mount @raw-io @reboot @swap @cpu-emulation @obsolete
```

Per-directive comments documenting *why* each directive is set the
way it is (sudo-incompatibility carve-outs for web; i386 amendment
and `PrivatePIDs` rationale for server@) should live inline as `#`
comments. Today those rationale comments live in the Python source;
they need to come along.

After extraction, the reactor's emitted unit bodies drop the
`**HARDENING_WEB` / `**HARDENING_SERVER` splat. The reactor still
emits the base unit and is responsible for everything except the
hardening profile.

### Sudoers

Today: identical content in
`left4me/deploy/files/etc/sudoers.d/left4me` and
`ckn-bw/bundles/left4me/files/etc/sudoers.d/left4me`. The bw item
sources from the ckn-bw copy.

After: bw `symlinks{}` item; delete the ckn-bw copy and the bw
`files{}` entry. The `test_with: 'visudo -cf {}'` semantics don't
apply to symlinks; tested instead on commit in left4me CI (a
`test_sudoers.py` that runs `visudo -cf` against the live file).

### Sysctl drop-in + ptrace_scope absorption

Today: same dual-copy story as sudoers, plus `kernel.yama.ptrace_scope`
exists as a metadata default (`sysctl/kernel/yama/ptrace_scope: '2'`)
that gets deployed via `bundles/sysctl/` into a separate file.

After: append `kernel.yama.ptrace_scope = 2` to
`deploy/files/etc/sysctl.d/99-left4me.conf`. Delete the metadata
entry. Delete the bw `files{}` entry + ckn-bw mirror; replace with
symlink. `bundles/sysctl/` no longer renders anything for left4me;
all left4me sysctl tuning lives in the one drop-in.

### Privileged scripts

Done (Task 4): `deploy/scripts/libexec/`, `deploy/scripts/sbin/` under
`deploy/` for layout consistency.
`install_left4me_scripts` copy-action replaced by target-side symlinks
from `/usr/local/libexec/left4me/` and `/usr/local/sbin/` into the
checkout at `/opt/left4me/src/deploy/scripts/{libexec,sbin}/`.

Sudo follows symlinks. With `/opt/left4me/src` root-owned, the
symlink target is root-owned, and sudo's `Cmnd_Alias` path matching
sees the original `/usr/local/{libexec,sbin}/<name>` path.

### Reference units in `deploy/files/`

No structural change. Remove hardening directives from the reference
files in lockstep with extracting them into the drop-ins (otherwise
`test_deploy_artifacts.py` sees the reference unit with hardening
inline but the live unit without). The reference file then represents
"the base unit ckn-bw emits"; the drop-in represents "the hardening
profile left4me ships".

## Migration order

Each step is an independent landable PR.

1. **Canary — sysctl consolidation.**
   - Add `kernel.yama.ptrace_scope = 2` to
     `deploy/files/etc/sysctl.d/99-left4me.conf`.
   - Delete `defaults['sysctl']['kernel']['yama']['ptrace_scope']`
     from `bundles/left4me/metadata.py`.
   - Delete `bundles/left4me/files/etc/sysctl.d/99-left4me.conf`
     (the verbatim mirror).
   - Replace the bw `files{}` entry with a `symlinks{}` entry
     pointing at the checkout.
   - Verify: `sysctl kernel.yama.ptrace_scope` reads `2`;
     `bw apply` idempotent.

2. **Hardening drop-ins.**
   - Create `deploy/files/etc/systemd/system/left4me-web.service.d/10-hardening.conf`
     and `…/left4me-server@.service.d/10-hardening.conf` from the
     `HARDENING_*` dicts.
   - Remove `**HARDENING_WEB` / `**HARDENING_SERVER` splats from the
     reactor; delete the three constants.
   - Remove hardening directives from the reference units in
     `deploy/files/usr/local/lib/systemd/system/`.
   - Add `directory:/etc/systemd/system/left4me-{web,server@}.service.d`
     items + symlinks for the drop-ins.
   - Wire `systemctl daemon-reload` to fire on
     `git_deploy:/opt/left4me/src`.
   - Verify: `systemctl show -p ProtectSystem,ProtectKernelTunables,PrivateUsers,…
     left4me-web.service left4me-server@1.service` matches the
     pre-extraction values (full hardening test plan rerun is the
     gold standard).

3. **Sudoers.**
   - Replace bw `files{}` entry with `symlinks{}`.
   - Delete `bundles/left4me/files/etc/sudoers.d/left4me`.
   - Add left4me CI test running `visudo -cf` on the file.
   - Verify: `sudo -l -U left4me` lists the expected commands;
     gameserver start via the web app still works.

4. **Privileged scripts.**
   - `git mv left4me/scripts left4me/deploy/scripts`.
   - Update any references (commit hooks, docs).
   - Replace `actions['install_left4me_scripts']` with `symlinks{}`
     items, one per script. Drop the action.
   - Update `git_deploy:` `triggers:` to remove
     `action:install_left4me_scripts`.
   - Verify: `sudo /usr/local/libexec/left4me/left4me-overlay status 1`
     still works; gameserver lifecycle (start/stop) still works.

5. **Cleanup.**
   - Prune `gunicorn_workers` / `gunicorn_threads` metadata defaults
     if they end up referenced only by `web.env.mako` (they do today;
     keep the metadata, they're real per-host values).
   - Update `deploy/README.md` to describe the new layout
     (deploy/files = symlink source-of-truth; deploy/scripts = same
     for helpers).
   - Update `bundles/left4me/README.md` to describe the new
     symlink-based delivery model.

## Sequence vs. build-overlay-unit refactor

This design lands **before** the build-overlay-unit refactor
(`2026-05-15-build-overlay-unit-design.md`). Reasons:

- build-overlay-unit introduces a dispatcher unit template; its
  hardening profile should live as a drop-in alongside the dispatcher
  from the start, using the pattern this design establishes.
- The reactor surgery in step 2 (removing `HARDENING_*` splats) is
  cleaner against today's reactor than against a reactor that's also
  being reshaped for the build-overlay-unit work.

## Verification (end-to-end)

After all five steps land and `bw apply` is idempotent on
`ovh.left4me`:

1. `systemctl show -p ProtectSystem,PrivateUsers,SystemCallFilter,…
   left4me-web.service left4me-server@1.service` matches the
   hardening test plan's reference values (run the relevant tests
   from `docs/superpowers/specs/2026-05-15-hardening-test-plan.md`).
2. `sysctl kernel.yama.ptrace_scope net.core.rmem_max
   net.ipv4.tcp_congestion_control` returns expected values.
3. `sudo -l -U left4me` reports the same allowed commands as before.
4. `ls -la /etc/sudoers.d/left4me /etc/sysctl.d/99-left4me.conf
   /etc/systemd/system/left4me-*.service.d/10-hardening.conf
   /usr/local/libexec/left4me/* /usr/local/sbin/<wrappers>` shows
   symlinks into `/opt/left4me/src/deploy/...`.
5. A gameserver round-trip (start via web app → cvar inspect → stop)
   succeeds.
6. `bw verify ovh.left4me` reports no drift.

## Out of scope

- Moving base unit bodies into left4me. Per-host shape stays
  reactor-emitted.
- AppArmor profiles (deferred from the defenses survey).
- Reshaping the bw `files{}` items for `host.env.mako` /
  `web.env.mako` — they need mako templating with metadata context,
  which ckn-bw is the right place for.
- The build-overlay-unit refactor itself. Lands separately on top of
  this.

## Pointers

- Handoff (this brainstorm's framing):
  `docs/superpowers/specs/2026-05-15-handoff-deployment-responsibility.md`
- Prereq (runtime state relocation + non-editable install, shipped):
  `docs/superpowers/specs/2026-05-15-runtime-state-relocation-design.md`
- Original deployment design (the model being reaffirmed for
  application-shape artifacts):
  `docs/superpowers/specs/2026-05-06-left4me-deployment-design.md`
- Hardening refactor design (the inline-in-reactor approach this
  design supersedes for hardening):
  `docs/superpowers/specs/2026-05-15-hardening-refactor-design.md`
- Hardening test plan (reference for step-2 verification):
  `docs/superpowers/specs/2026-05-15-hardening-test-plan.md`
- ckn-bw left4me bundle: `~/Projekte/ckn-bw/bundles/left4me/`