# L4D2 Workshop Overlays Design

**Goal:** Let users add Steam Workshop content (.vpk addons and maps) to L4D2 servers from the web UI. Workshop downloads run as a new typed overlay that fits the existing `Overlay` + `BlueprintOverlay` model, downloaded via the public Steam Web API and exposed through the existing fuse-overlayfs mount layer.

**Approval status:** User-approved design direction. Implementation proceeds in lockstep with the companion plan at `docs/superpowers/plans/2026-05-07-l4d2-workshop-overlays.md`.

## Context

`left4me` users today add `.vpk` content to a server only by SFTP-ing files into a manually-prepared overlay directory or by maintaining shell scripts (`competitive_rework`, `workshop_maps`, `tickrate`, etc.) that wrap `curl`/`steamcmd`. The web app exposes overlay rows but offers no way for users to populate them.

This spec adds **workshop overlays**: a user-private overlay type that downloads `.vpk` files via the public `ISteamRemoteStorage` API and surfaces them through the existing mount layer. Users keep composing blueprints by stacking overlays — workshop overlays become another row alongside today's externally-managed ones.

This is the first *typed* overlay. The design adds a `type` column and a builder-registry so future overlay types (tarball, inline, manual upload) plug in without schema churn or workflow changes.

Steam Workshop content for L4D2 (consumer_app_id 550) is downloadable via two anonymous-POST endpoints with no Steam Web API key required: `GetCollectionDetails` resolves a collection ID to its child item IDs, and `GetPublishedFileDetails` returns per-item metadata including a public `file_url` for the `.vpk`. This is the same API the user's existing `steam-workshop-download` script uses.

L4D2-specific player-side pain points (sv_consistency / RestrictAddons configuration gotchas, the inability to push workshop content via `sv_downloadurl`) are documented in **Out of scope** and tracked as separate follow-ups. This spec stays strictly on workshop content acquisition.

## Locked Decisions

1. **Typed overlays.** `Overlay.type` joins `external` (existing rows; admin-managed; no-op builder) and `workshop` (new). Future types — tarball, inline, manual upload — slot in via the same builder registry without schema churn.
2. **No JSON `source_config` blob.** Per-type structured data lives in proper relational tables. JSON is reserved for genuinely opaque diagnostic payloads.
3. **Central deduplicated `WorkshopItem` registry** keyed on `steam_id`. Cache lives at `/var/lib/left4me/workshop_cache/{steam_id}.vpk`. Multiple overlays referencing the same Steam item share the same cache file.
4. **Symlinks, not copies.** Overlay directories contain `left4dead2/addons/{steam_id}.vpk` symlinks pointing into the cache. Both the cache file and the symlink are named by `{steam_id}` only — no Steam filename in any on-disk path, so Steam can rename the upstream `.vpk` without breaking lookup.
5. **Many-to-many association is pure** (no `enabled` flag). Toggle a workshop item by removing or re-adding the association. The shared cache makes this cheap.
6. **Collections are atomic UI bulk-imports.** Pasting a collection URL/ID resolves member items and creates N item associations. The DB never tracks "this came from a collection." Re-importing a collection is idempotent on existing items and additive for new ones.
7. **Single global admin "Refresh all workshop items" button.** One Steam metadata batch call, then re-download items whose `time_updated` advanced. No per-item, per-overlay, or scheduled refresh in v1.
8. **No cache GC in v1.** Cache grows monotonically. Reference-counted cleanup is a follow-up.
9. **Globality is independent of overlay type.** `Overlay.user_id` is the scope (NULL = system-wide, set = private to that user). v1 defaults newly-created workshop overlays to private and leaves existing external overlays as system-wide. A future "publish/share" button will let owners toggle `user_id` without changing type.
10. **One unified "Create overlay" UI button.** Modal has a type radio (External | Workshop). No path field — the web app generates the path for every new overlay.
11. **Strict scope.** v1 ships only the workshop type. L4D2 server-config gotchas, client-subscription helpers, other recipe types — all deferred to follow-up specs.
12. **`consumer_app_id == 550` validation** at every Steam API response at fetch/add time; non-L4D2 items are rejected and never reach the row. The value is a fixed precondition, not data.
13. **Input field accepts numeric ID, full Workshop URL, or a multi-line batch** of either. Pasting `123456` and pasting `steamcommunity.com/sharedfiles/filedetails/?id=123456` produce the same result; pasting many of either at once works too.
14. **Web-managed overlay paths.** All new overlays (any type) get `path = str(overlay_id)` at insert time. The user never picks a path. Existing legacy external overlay rows keep their current path values; migrating them to the ID-based scheme is a follow-up. `Overlay.id` uses SQLite `AUTOINCREMENT` so deleted IDs are never reused.
15. **Auto-rebuild on item change.** Adding or removing items from a workshop overlay automatically enqueues a `build_overlay` job. The "Rebuild" button on the detail page is for manual recovery only. New build jobs for an overlay coalesce with any pending one for the same overlay (don't queue duplicates).
16. **HTTPS** for all Steam Web API calls. The reference downloader uses HTTP; we don't.

## Architecture

```
Overlay row (type=workshop)
   └─refs─▶ overlay_workshop_items
                  └─▶ WorkshopItem (global, by steam_id)
                          │
                          ▼ download (Steam GetPublishedFileDetails + HTTP GET)
                     workshop_cache/{steam_id}.vpk
                          ▲
overlay_dir/left4dead2/addons/{steam_id}.vpk ─symlink─┘
```

Build dispatch via a registry:

```python
BUILDERS = {"external": ExternalBuilder(), "workshop": WorkshopBuilder()}

def build_overlay(overlay_id):
    overlay = db.get(Overlay, overlay_id)
    BUILDERS[overlay.type].build(overlay, on_stdout, on_stderr, should_cancel)
```

`ExternalBuilder` is a no-op for legacy admin-managed dirs. `WorkshopBuilder` performs an idempotent diff-apply of `addons/` symlinks against the current associations. Future types add their own builders without changing the dispatcher, the mount layer, or the blueprint editor.

## Data Model

### `Overlay` (extended)

```
id INTEGER PK AUTOINCREMENT
name VARCHAR(255) NOT NULL
path VARCHAR(255) NOT NULL                 -- new overlays: str(id); legacy externals: existing values
type VARCHAR(16) NOT NULL                  -- 'external' | 'workshop' (extensible)
user_id INTEGER NULL REFERENCES users(id)  -- NULL = system-wide
created_at, updated_at

UNIQUE INDEX on (name) WHERE user_id IS NULL    -- system overlays globally unique by name
UNIQUE INDEX on (name, user_id) WHERE user_id IS NOT NULL  -- per-user namespace
INDEX on (type, user_id)
```

Two partial unique indexes are required because a naive composite `UNIQUE(name, user_id)` doesn't constrain externals — SQLite treats NULL as distinct in unique constraints, so two externals could share a name. Partial indexes preserve the prior global-uniqueness invariant for system rows.

### `WorkshopItem` (new)

```
id INTEGER PK
steam_id VARCHAR(20) NOT NULL UNIQUE        -- 64-bit, store as text
title VARCHAR(255) NOT NULL DEFAULT ''
filename VARCHAR(255) NOT NULL DEFAULT ''   -- upstream Steam filename, display only
file_url TEXT NOT NULL DEFAULT ''
file_size BIGINT NOT NULL DEFAULT 0
time_updated INTEGER NOT NULL DEFAULT 0     -- Steam epoch
preview_url TEXT NOT NULL DEFAULT ''        -- thumbnail URL hot-linked from Steam
last_downloaded_at DATETIME NULL
last_error TEXT NOT NULL DEFAULT ''
created_at, updated_at
```

`consumer_app_id` is **not** stored. It's validated at fetch time and the row never exists for non-L4D2 items.

### `overlay_workshop_items` (new, pure association)

```
id INTEGER PK
overlay_id INTEGER NOT NULL REFERENCES overlays(id) ON DELETE CASCADE
workshop_item_id INTEGER NOT NULL REFERENCES workshop_items(id) ON DELETE RESTRICT
UNIQUE (overlay_id, workshop_item_id)
INDEX (workshop_item_id)                    -- reverse lookup for refresh
```

No `enabled` column — toggle is remove/add, which is cheap because the cache survives.

### `Job` (extended)

Add `overlay_id INTEGER NULL REFERENCES overlays(id)` for `build_overlay` jobs.

## Filesystem Layout

```
/var/lib/left4me/
  overlays/
    {overlay_id}/                                   # flat — same shape for every type
      left4dead2/addons/
        {steam_id}.vpk -> /var/lib/left4me/workshop_cache/{steam_id}.vpk
  workshop_cache/
    {steam_id}.vpk                                  # one file per Steam item
```

- Every new overlay (workshop, future tarball/inline/manual) lives at `overlays/{overlay_id}/`. Legacy external overlays keep their pre-migration paths (e.g. `overlays/standard/`).
- `workshop_cache/` is created during deploy provisioning, not lazily — avoids races between concurrent first downloads.
- Web user owns both trees (mode 0755). Host user (`l4d2ctl`) needs read on both. If web and host are different users, they share a group.
- Symlink targets are absolute. Relative targets resolve in the merged-mount namespace and break across the host/web boundary.
- The builder never creates a dangling symlink. If a `WorkshopItem` lacks a cache file at build time, the builder logs a warning and skips it — fuse-overlayfs surfaces broken links to L4D2 as opaque addon-scan failures.

## UI

A single "Create overlay" button on `/overlays` opens a modal with type radio (External | Workshop) and a name field. No path field. The web app generates `path = str(overlay_id)` after insert.

Workshop overlay detail page (`/overlays/{id}` when `type='workshop'`) shows:

- A multi-line input plus a radio (Items | Collection). Pasting one or many IDs/URLs adds them in order; pasting a collection ID resolves its members.
- An item table with: thumbnail (`preview_url`), `steam_id` linking to Steam, title, filename, last-updated, size, last-error if any, Remove.
- A manual "Rebuild" button (for recovery only — every add/remove auto-enqueues a coalesced `build_overlay` job).
- Status indicator pulled from the latest related `Job` row.

External overlay detail page is unchanged in shape: read-only path display, name edit (admin only). The "External" type retains the existing admin-only SFTP-to-disk workflow until a future "manual upload" type replaces it.

The blueprint editor is unchanged in structure. Workshop overlays appear alongside externals in the user's overlay picker; ordering and stacking semantics are identical.

Admin section gets one new control: "Refresh all workshop items" button on the admin landing or workshop subsection. Pressing it enqueues a single `refresh_workshop_items` job.

### Routes

| Method | Path | Purpose |
|---|---|---|
| GET | `/overlays` | List with Type column, filtered by user permissions |
| POST | `/overlays` | Create; reads `type` and `name` only |
| GET | `/overlays/{id}` | Type-aware detail page |
| POST | `/overlays/{id}/items` | Add items or collection; auto-enqueues coalesced `build_overlay` |
| POST | `/overlays/{id}/items/{item_id}/delete` | Remove association; auto-enqueues coalesced `build_overlay` |
| POST | `/overlays/{id}/build` | Manual rebuild (recovery) |
| POST | `/admin/workshop/refresh` | Admin only; enqueue `refresh_workshop_items` |

HTMX usage stays minimal: only the add-item form and per-row delete swap a fragment. Everything else is full-page POST/redirect/GET.

## Job Operations

Two new operations join the existing job worker:

- **`build_overlay(overlay_id)`** — `Job.overlay_id` is set; `server_id` is NULL. Dispatches to `BUILDERS[overlay.type].build(...)`. Cancellation between filesystem operations.
- **`refresh_workshop_items()`** — admin-only. Both `server_id` and `overlay_id` are NULL. Phases: fetch all metadata in one batched call, download items where `time_updated` advanced, enqueue (coalesced) `build_overlay` for affected overlays. v1 doesn't wait on child builds; the admin sees them in the jobs list.

### Scheduler rules

- `install` and `refresh_workshop_items` are mutually exclusive with each other, with all `build_overlay`s, and with all server jobs.
- `build_overlay(overlay_id=N)` blocks if `install_running`, `refresh_running`, or another build for the same `overlay_id` is running. Builds for *different* overlays may run concurrently.
- Server start/init blocks if `refresh_running` or any `build_overlay` for an overlay referenced by the server's blueprint is running.

Coalescing: a new `build_overlay` for an overlay that already has a queued (not-yet-running) build returns the existing job instead of inserting a new row.

`initialize_server` synchronously calls each overlay's builder before writing the spec for `l4d2ctl initialize`. If a workshop overlay references uncached items (no file in `workshop_cache/`), `initialize_server` fails fast with a clear error naming the missing IDs and pointing the user at the overlay page. It never silently mounts a partial overlay.

## Permissions

- **External overlays**: admin-only create/edit. Visible to all authenticated users (system-wide).
- **Workshop overlays**: any logged-in user can create. Owner or admin can edit and delete. Visible to the owner and admins.
- **Admin refresh**: admin-only.

The `Overlay` listing query for non-admins becomes: `type='external' OR user_id=current_user.id`.

## Risks

- **Broken symlinks across host/web boundary** — mitigated by absolute targets, build-time pre-check skipping uncached items, and `deploy/` documenting permission requirements.
- **Initialize against uncached items** — would silently mount overlays missing maps. Mitigated by `initialize_server`'s fail-fast check; tested.
- **Steam API rate limits** — refresh of 100 items is one metadata POST plus 100 downloads at 8-way parallelism. No retry/backoff in v1; 429s surface verbatim in the job log.
- **Partial failure during refresh** — each item is independent; per-item errors land on the row. Re-running refresh retries failures.
- **Concurrent same-ID adds** — `WorkshopItem.steam_id` unique handles cache dedup. `(overlay_id, workshop_item_id)` unique catches double-association; the route returns "already in overlay" rather than 500.
- **Build coalescing missed** — would enqueue dozens of redundant builds during multi-item adds. Mitigated by the `enqueue_build_overlay` helper; tested.
- **Worker concurrency rule miss** — the truth-table test in `test_job_worker.py` is the only way to trust the new scheduler logic; written before dispatch.
- **DB/disk drift** — a stray directory left by a prior failed delete could shadow a fresh overlay. Mitigated by `AUTOINCREMENT` (no ID reuse) and `os.makedirs(exist_ok=False)` (loud failure on collision).
- **Partial unique gap on SQLite** — naive composite `UNIQUE(name, user_id)` doesn't constrain externals because NULL is distinct. Mitigated by two partial unique indexes; tested explicitly.
- **Cache growth without GC** — accepted v1 trade-off.
- **Item removed from Steam** — refresh marks `result != 1`; row keeps last good cache file; UI surfaces error string. Operator decides removal.
- **L4D2 containerized run** — symlink absolute targets break if the server runs in a different mount namespace. Re-evaluate when containerization comes up.

## Out Of Scope

These came up in research and dialog but stay out of v1:

- **Publish / share button on overlays.** Lets owners flip `Overlay.user_id` between their own ID and NULL without changing type. The schema already supports it; only the UI is deferred.
- **Migrate legacy external overlay paths to the ID-based scheme.** Existing external rows keep their pre-migration paths in v1; a follow-up migration moves the directories on disk and updates the rows.
- **Switch from fuse-overlayfs to kernel overlayfs via a privileged helper.** Matches the existing systemd / steam-install sudoers helper pattern under `/usr/local/libexec/left4me/`. Workshop overlays would work identically under either mount engine — symlinks resolve through normal VFS in both.
- **`sv_consistency` / `addonconfig.cfg RestrictAddons` auto-handling.** When a workshop overlay attaches to a blueprint, surface a banner with a one-click fix. Most-cited L4D2 player pain.
- **Shareable Steam Workshop collection link for clients.** Server cannot push workshop content via `sv_downloadurl`; clients must subscribe themselves. A panel-generated collection makes that one click for players. Requires Steam OAuth.
- **Other overlay types.** `tarball` (covers the old `competitive_rework` GitHub-tarball recipe), `inline` (covers `tickrate`'s inline `server.cfg`), `manual` (file manager / upload, replaces the admin-SFTP external workflow). All slot in via the builder registry without schema churn.
- **Cache GC.** Reference-counted delete or admin "Clear unreferenced" page.
- **Per-item / per-overlay / scheduled refresh.** v1 has one global admin button; revisit if users want finer control.
- **Update-aware server restart UX.** Notify users when a running server's overlay content has been refreshed underneath it.

## Implementation Boundaries

- The host library contract is unchanged. Workshop content arrives in overlay directories the same way externals do today; `l4d2host` doesn't know overlays have types.
- The job-execution model is preserved: same workers, same logs, same cancel callbacks. Only the operations table grows.
- The blueprint privacy model and desired-vs-actual server state model are unchanged.
- No new frontend dependencies. Vendored HTMX + custom CSS + small inline JS.
- No new Steam Web API key required; both endpoints used accept anonymous POSTs.
- The companion implementation plan governs task ordering and verification commands. Implementation must not start without explicit user approval per that plan's gate.