left4me/docs/superpowers/specs/2026-05-07-l4d2-workshop-overlays-design.md

L4D2 Workshop Overlays Design

Goal: Let users add Steam Workshop content (.vpk addons and maps) to L4D2 servers from the web UI. Workshop downloads run as a new typed overlay that fits the existing Overlay + BlueprintOverlay model, downloaded via the public Steam Web API and exposed through the existing fuse-overlayfs mount layer.

Approval status: User-approved design direction. Implementation proceeds in lockstep with the companion plan at docs/superpowers/plans/2026-05-07-l4d2-workshop-overlays.md.

Context

left4me users today add .vpk content to a server only by SFTP-ing files into a manually-prepared overlay directory or by maintaining shell scripts (competitive_rework, workshop_maps, tickrate, etc.) that wrap curl/steamcmd. The web app exposes overlay rows but offers no way for users to populate them.

This spec adds workshop overlays: a user-private overlay type that downloads .vpk files via the public ISteamRemoteStorage API and surfaces them through the existing mount layer. Users keep composing blueprints by stacking overlays — workshop overlays become another row alongside today's externally-managed ones.

This is the first typed overlay. The design adds a type column and a builder-registry so future overlay types (tarball, inline, manual upload) plug in without schema churn or workflow changes.

Steam Workshop content for L4D2 (consumer_app_id 550) is downloadable via two anonymous-POST endpoints with no Steam Web API key required: GetCollectionDetails resolves a collection ID to its child item IDs, and GetPublishedFileDetails returns per-item metadata including a public file_url for the .vpk. This is the same API the user's existing steam-workshop-download script uses.

L4D2-specific player-side pain points (sv_consistency / RestrictAddons configuration gotchas, the inability to push workshop content via sv_downloadurl) are documented in Out of scope and tracked as separate follow-ups. This spec stays strictly on workshop content acquisition.

Locked Decisions

1. Typed overlays. Overlay.type joins external (existing rows; admin-managed; no-op builder) and workshop (new). Future types — tarball, inline, manual upload — slot in via the same builder registry without schema churn.
2. No JSON source_config blob. Per-type structured data lives in proper relational tables. JSON is reserved for genuinely opaque diagnostic payloads.
3. Central deduplicated WorkshopItem registry keyed on steam_id. Cache lives at /var/lib/left4me/workshop_cache/{steam_id}.vpk. Multiple overlays referencing the same Steam item share the same cache file.
4. Symlinks, not copies. Overlay directories contain left4dead2/addons/{steam_id}.vpk symlinks pointing into the cache. Both the cache file and the symlink are named by {steam_id} only — no Steam filename in any on-disk path, so Steam can rename the upstream .vpk without breaking lookup.
5. Many-to-many association is pure (no enabled flag). Toggle a workshop item by removing or re-adding the association. The shared cache makes this cheap.
6. Collections are atomic UI bulk-imports. Pasting a collection URL/ID resolves member items and creates N item associations. The DB never tracks "this came from a collection." Re-importing a collection is idempotent on existing items and additive for new ones.
7. Single global admin "Refresh all workshop items" button. One Steam metadata batch call, then re-download items whose time_updated advanced. No per-item, per-overlay, or scheduled refresh in v1.
8. No cache GC in v1. Cache grows monotonically. Reference-counted cleanup is a follow-up.
9. Globality is independent of overlay type. Overlay.user_id is the scope (NULL = system-wide, set = private to that user). v1 defaults newly-created workshop overlays to private and leaves existing external overlays as system-wide. A future "publish/share" button will let owners toggle user_id without changing type.
10. One unified "Create overlay" UI button. Modal has a type radio (External | Workshop). No path field — the web app generates the path for every new overlay.
11. Strict scope. v1 ships only the workshop type. L4D2 server-config gotchas, client-subscription helpers, other recipe types — all deferred to follow-up specs.
12. consumer_app_id == 550 validation at every Steam API response at fetch/add time; non-L4D2 items are rejected and never reach the row. The value is a fixed precondition, not data.
13. Input field accepts numeric ID, full Workshop URL, or a multi-line batch of either. Pasting 123456 and pasting steamcommunity.com/sharedfiles/filedetails/?id=123456 produce the same result; pasting many of either at once works too.
14. Web-managed overlay paths. All new overlays (any type) get path = str(overlay_id) at insert time. The user never picks a path. Existing legacy external overlay rows keep their current path values; migrating them to the ID-based scheme is a follow-up. Overlay.id uses SQLite AUTOINCREMENT so deleted IDs are never reused.
15. Auto-rebuild on item change. Adding or removing items from a workshop overlay automatically enqueues a build_overlay job. The "Rebuild" button on the detail page is for manual recovery only. New build jobs for an overlay coalesce with any pending one for the same overlay (don't queue duplicates).
16. HTTPS for all Steam Web API calls. The reference downloader uses HTTP; we don't.

Architecture

Overlay row (type=workshop)
   └─refs─▶ overlay_workshop_items
                  └─▶ WorkshopItem (global, by steam_id)
                          │
                          ▼ download (Steam GetPublishedFileDetails + HTTP GET)
                     workshop_cache/{steam_id}.vpk
                          ▲
overlay_dir/left4dead2/addons/{steam_id}.vpk ─symlink─┘

Build dispatch via a registry:

BUILDERS = {"external": ExternalBuilder(), "workshop": WorkshopBuilder()}

def build_overlay(overlay_id):
    overlay = db.get(Overlay, overlay_id)
    BUILDERS[overlay.type].build(overlay, on_stdout, on_stderr, should_cancel)

ExternalBuilder is a no-op for legacy admin-managed dirs. WorkshopBuilder performs an idempotent diff-apply of addons/ symlinks against the current associations. Future types add their own builders without changing the dispatcher, the mount layer, or the blueprint editor.

Data Model

Overlay (extended)

id INTEGER PK AUTOINCREMENT
name VARCHAR(255) NOT NULL
path VARCHAR(255) NOT NULL                 -- new overlays: str(id); legacy externals: existing values
type VARCHAR(16) NOT NULL                  -- 'external' | 'workshop' (extensible)
user_id INTEGER NULL REFERENCES users(id)  -- NULL = system-wide
created_at, updated_at

UNIQUE INDEX on (name) WHERE user_id IS NULL    -- system overlays globally unique by name
UNIQUE INDEX on (name, user_id) WHERE user_id IS NOT NULL  -- per-user namespace
INDEX on (type, user_id)

Two partial unique indexes are required because a naive composite UNIQUE(name, user_id) doesn't constrain externals — SQLite treats NULL as distinct in unique constraints, so two externals could share a name. Partial indexes preserve the prior global-uniqueness invariant for system rows.

WorkshopItem (new)

id INTEGER PK
steam_id VARCHAR(20) NOT NULL UNIQUE        -- 64-bit, store as text
title VARCHAR(255) NOT NULL DEFAULT ''
filename VARCHAR(255) NOT NULL DEFAULT ''   -- upstream Steam filename, display only
file_url TEXT NOT NULL DEFAULT ''
file_size BIGINT NOT NULL DEFAULT 0
time_updated INTEGER NOT NULL DEFAULT 0     -- Steam epoch
preview_url TEXT NOT NULL DEFAULT ''        -- thumbnail URL hot-linked from Steam
last_downloaded_at DATETIME NULL
last_error TEXT NOT NULL DEFAULT ''
created_at, updated_at

consumer_app_id is not stored. It's validated at fetch time and the row never exists for non-L4D2 items.

overlay_workshop_items (new, pure association)

id INTEGER PK
overlay_id INTEGER NOT NULL REFERENCES overlays(id) ON DELETE CASCADE
workshop_item_id INTEGER NOT NULL REFERENCES workshop_items(id) ON DELETE RESTRICT
UNIQUE (overlay_id, workshop_item_id)
INDEX (workshop_item_id)                    -- reverse lookup for refresh

No enabled column — toggle is remove/add, which is cheap because the cache survives.

Job (extended)

Add overlay_id INTEGER NULL REFERENCES overlays(id) for build_overlay jobs.

Filesystem Layout

/var/lib/left4me/
  overlays/
    {overlay_id}/                                   # flat — same shape for every type
      left4dead2/addons/
        {steam_id}.vpk -> /var/lib/left4me/workshop_cache/{steam_id}.vpk
  workshop_cache/
    {steam_id}.vpk                                  # one file per Steam item

• Every new overlay (workshop, future tarball/inline/manual) lives at overlays/{overlay_id}/. Legacy external overlays keep their pre-migration paths (e.g. overlays/standard/).
• workshop_cache/ is created during deploy provisioning, not lazily — avoids races between concurrent first downloads.
• Web user owns both trees (mode 0755). Host user (l4d2ctl) needs read on both. If web and host are different users, they share a group.
• Symlink targets are absolute. Relative targets resolve in the merged-mount namespace and break across the host/web boundary.
• The builder never creates a dangling symlink. If a WorkshopItem lacks a cache file at build time, the builder logs a warning and skips it — fuse-overlayfs surfaces broken links to L4D2 as opaque addon-scan failures.

UI

A single "Create overlay" button on /overlays opens a modal with type radio (External | Workshop) and a name field. No path field. The web app generates path = str(overlay_id) after insert.

Workshop overlay detail page (/overlays/{id} when type='workshop') shows:

• A multi-line input plus a radio (Items | Collection). Pasting one or many IDs/URLs adds them in order; pasting a collection ID resolves its members.
• An item table with: thumbnail (preview_url), steam_id linking to Steam, title, filename, last-updated, size, last-error if any, Remove.
• A manual "Rebuild" button (for recovery only — every add/remove auto-enqueues a coalesced build_overlay job).
• Status indicator pulled from the latest related Job row.

External overlay detail page is unchanged in shape: read-only path display, name edit (admin only). The "External" type retains the existing admin-only SFTP-to-disk workflow until a future "manual upload" type replaces it.

The blueprint editor is unchanged in structure. Workshop overlays appear alongside externals in the user's overlay picker; ordering and stacking semantics are identical.

Admin section gets one new control: "Refresh all workshop items" button on the admin landing or workshop subsection. Pressing it enqueues a single refresh_workshop_items job.

Routes

Method	Path	Purpose
GET	/overlays	List with Type column, filtered by user permissions
POST	/overlays	Create; reads type and name only
GET	/overlays/{id}	Type-aware detail page
POST	/overlays/{id}/items	Add items or collection; auto-enqueues coalesced build_overlay
POST	/overlays/{id}/items/{item_id}/delete	Remove association; auto-enqueues coalesced build_overlay
POST	/overlays/{id}/build	Manual rebuild (recovery)
POST	/admin/workshop/refresh	Admin only; enqueue refresh_workshop_items

HTMX usage stays minimal: only the add-item form and per-row delete swap a fragment. Everything else is full-page POST/redirect/GET.

Job Operations

Two new operations join the existing job worker:

• build_overlay(overlay_id) — Job.overlay_id is set; server_id is NULL. Dispatches to BUILDERS[overlay.type].build(...). Cancellation between filesystem operations.
• refresh_workshop_items() — admin-only. Both server_id and overlay_id are NULL. Phases: fetch all metadata in one batched call, download items where time_updated advanced, enqueue (coalesced) build_overlay for affected overlays. v1 doesn't wait on child builds; the admin sees them in the jobs list.

Scheduler rules

• install and refresh_workshop_items are mutually exclusive with each other, with all build_overlays, and with all server jobs.
• build_overlay(overlay_id=N) blocks if install_running, refresh_running, or another build for the same overlay_id is running. Builds for different overlays may run concurrently.
• Server start/init blocks if refresh_running or any build_overlay for an overlay referenced by the server's blueprint is running.

Coalescing: a new build_overlay for an overlay that already has a queued (not-yet-running) build returns the existing job instead of inserting a new row.

initialize_server synchronously calls each overlay's builder before writing the spec for l4d2ctl initialize. If a workshop overlay references uncached items (no file in workshop_cache/), initialize_server fails fast with a clear error naming the missing IDs and pointing the user at the overlay page. It never silently mounts a partial overlay.

Permissions

• External overlays: admin-only create/edit. Visible to all authenticated users (system-wide).
• Workshop overlays: any logged-in user can create. Owner or admin can edit and delete. Visible to the owner and admins.
• Admin refresh: admin-only.

The Overlay listing query for non-admins becomes: type='external' OR user_id=current_user.id.

Risks

• Broken symlinks across host/web boundary — mitigated by absolute targets, build-time pre-check skipping uncached items, and deploy/ documenting permission requirements.
• Initialize against uncached items — would silently mount overlays missing maps. Mitigated by initialize_server's fail-fast check; tested.
• Steam API rate limits — refresh of 100 items is one metadata POST plus 100 downloads at 8-way parallelism. No retry/backoff in v1; 429s surface verbatim in the job log.
• Partial failure during refresh — each item is independent; per-item errors land on the row. Re-running refresh retries failures.
• Concurrent same-ID adds — WorkshopItem.steam_id unique handles cache dedup. (overlay_id, workshop_item_id) unique catches double-association; the route returns "already in overlay" rather than 500.
• Build coalescing missed — would enqueue dozens of redundant builds during multi-item adds. Mitigated by the enqueue_build_overlay helper; tested.
• Worker concurrency rule miss — the truth-table test in test_job_worker.py is the only way to trust the new scheduler logic; written before dispatch.
• DB/disk drift — a stray directory left by a prior failed delete could shadow a fresh overlay. Mitigated by AUTOINCREMENT (no ID reuse) and os.makedirs(exist_ok=False) (loud failure on collision).
• Partial unique gap on SQLite — naive composite UNIQUE(name, user_id) doesn't constrain externals because NULL is distinct. Mitigated by two partial unique indexes; tested explicitly.
• Cache growth without GC — accepted v1 trade-off.
• Item removed from Steam — refresh marks result != 1; row keeps last good cache file; UI surfaces error string. Operator decides removal.
• L4D2 containerized run — symlink absolute targets break if the server runs in a different mount namespace. Re-evaluate when containerization comes up.

Out Of Scope

These came up in research and dialog but stay out of v1:

• Publish / share button on overlays. Lets owners flip Overlay.user_id between their own ID and NULL without changing type. The schema already supports it; only the UI is deferred.
• Migrate legacy external overlay paths to the ID-based scheme. Existing external rows keep their pre-migration paths in v1; a follow-up migration moves the directories on disk and updates the rows.
• Switch from fuse-overlayfs to kernel overlayfs via a privileged helper. Matches the existing systemd / steam-install sudoers helper pattern under /usr/local/libexec/left4me/. Workshop overlays would work identically under either mount engine — symlinks resolve through normal VFS in both.
• sv_consistency / addonconfig.cfg RestrictAddons auto-handling. When a workshop overlay attaches to a blueprint, surface a banner with a one-click fix. Most-cited L4D2 player pain.
• Shareable Steam Workshop collection link for clients. Server cannot push workshop content via sv_downloadurl; clients must subscribe themselves. A panel-generated collection makes that one click for players. Requires Steam OAuth.
• Other overlay types. tarball (covers the old competitive_rework GitHub-tarball recipe), inline (covers tickrate's inline server.cfg), manual (file manager / upload, replaces the admin-SFTP external workflow). All slot in via the builder registry without schema churn.
• Cache GC. Reference-counted delete or admin "Clear unreferenced" page.
• Per-item / per-overlay / scheduled refresh. v1 has one global admin button; revisit if users want finer control.
• Update-aware server restart UX. Notify users when a running server's overlay content has been refreshed underneath it.

Implementation Boundaries

• The host library contract is unchanged. Workshop content arrives in overlay directories the same way externals do today; l4d2host doesn't know overlays have types.
• The job-execution model is preserved: same workers, same logs, same cancel callbacks. Only the operations table grows.
• The blueprint privacy model and desired-vs-actual server state model are unchanged.
• No new frontend dependencies. Vendored HTMX + custom CSS + small inline JS.
• No new Steam Web API key required; both endpoints used accept anonymous POSTs.
• The companion implementation plan governs task ordering and verification commands. Implementation must not start without explicit user approval per that plan's gate.