left4me/docs/superpowers/specs/2026-05-07-l4d2-global-map-overlays-design.md

# L4D2 Global Map Overlays Design

**Goal:** Add two managed, system-wide map overlays, `l4d2center-maps` and `cedapug-maps`, populated from upstream map sources and refreshed daily through the existing job system.

**Approval status:** User-approved design direction. Implementation must not start until this spec is reviewed and an implementation plan is written.

## Context

`left4me` already has typed overlays, a builder registry, global overlays through `Overlay.user_id = NULL`, and queued overlay build jobs. Steam Workshop overlays use a cache plus symlinks into `left4dead2/addons/`, and server initialization already runs overlay builders before calling `l4d2ctl initialize`.

Global map sources fit the same model. The host library remains unchanged: it receives overlay refs and mounts directories. The web app owns map-source fetching, cache management, reconciliation, and job logs.

The two upstream sources are:

- `https://l4d2center.com/maps/servers/index.csv`
- `https://cedapug.com/custom`

## Locked Decisions

1. **One general operation.** Use `refresh_global_overlays`, not source-specific cron operations.
2. **Systemd owns time.** A systemd timer runs daily and invokes a Flask CLI command. The CLI only enqueues work; the existing worker performs downloads and writes logs.
3. **System jobs are nullable-owner jobs.** `jobs.user_id` becomes nullable. `NULL` means the job was created by the system. UI displays owner as `system`. Only admins can access system jobs.
4. **Managed global overlays are auto-seeded.** The app creates or repairs exactly one `l4d2center-maps` overlay and exactly one `cedapug-maps` overlay.
5. **Global overlays are normal system overlays for users.** `Overlay.user_id = NULL` makes them visible to every authenticated user and selectable in every user's blueprint editor.
6. **Managed types are not user-creatable.** Normal overlay creation does not offer `l4d2center_maps` or `cedapug_maps`. The seeder is the only code path that creates those types.
7. **Exact reconciliation.** Refresh makes each managed overlay match its upstream manifest. Removed upstream maps are removed from the managed overlay symlink set. Foreign files are left alone and logged.
8. **No initialize-time downloads.** `initialize_server()` may run builders to repair symlinks, but it must not fetch remote manifests or download large archives. Missing cache content fails clearly.
9. **Separate cache from Workshop.** Non-Steam global maps use `${LEFT4ME_ROOT}/global_overlay_cache`, not `${LEFT4ME_ROOT}/workshop_cache`.
10. **Source-specific parsing stays explicit.** Do not introduce a generic arbitrary HTTP source framework in this phase.

## Architecture

The design extends the existing overlay-builder registry:

```python
BUILDERS = {
    "external": ExternalBuilder(),
    "workshop": WorkshopBuilder(),
    "l4d2center_maps": GlobalMapOverlayBuilder(),
    "cedapug_maps": GlobalMapOverlayBuilder(),
}
```

Both global map overlay types share the same filesystem builder. Source-specific code lives in refresh services that know how to fetch and parse upstream manifests.

High-level flow:

```text
systemd timer
  -> flask refresh-global-overlays
     -> ensure_global_overlays()
     -> enqueue refresh_global_overlays job (coalesced)
        -> worker fetches manifests
        -> worker downloads/extracts cache files
        -> worker records desired VPK files
        -> worker rebuilds overlay symlinks directly
```

Auto-seeded overlay rows use fixed names, managed types, `user_id = NULL`, and web-generated paths:

```text
name=l4d2center-maps, type=l4d2center_maps, user_id=NULL, path=str(id)
name=cedapug-maps, type=cedapug_maps, user_id=NULL, path=str(id)
```

## Data Model

### `jobs`

Change `jobs.user_id` from required to nullable.

`NULL` means a system-created job. Authorization rules become:

- Admins can view, stream, and cancel every job, including system jobs.
- Non-admins can access only jobs where `job.user_id == current_user.id`.
- System jobs are not visible to non-admins through direct job URLs.

Job list/detail pages use outer joins to `users` and render missing owners as `system`.

### `global_overlay_sources`

One row per managed global source overlay:

```text
id INTEGER PRIMARY KEY
overlay_id INTEGER NOT NULL UNIQUE REFERENCES overlays(id) ON DELETE CASCADE
source_key VARCHAR(64) NOT NULL UNIQUE       -- l4d2center-maps | cedapug-maps
source_type VARCHAR(32) NOT NULL             -- l4d2center_csv | cedapug_custom_page
source_url TEXT NOT NULL
last_manifest_hash VARCHAR(64) NOT NULL DEFAULT ''
last_refreshed_at DATETIME NULL
last_error TEXT NOT NULL DEFAULT ''
created_at DATETIME NOT NULL
updated_at DATETIME NOT NULL
```

`source_key` is stable and used by the seeder to repair missing rows.

### `global_overlay_items`

One row per manifest item belonging to a global overlay source:

```text
id INTEGER PRIMARY KEY
source_id INTEGER NOT NULL REFERENCES global_overlay_sources(id) ON DELETE CASCADE
item_key VARCHAR(255) NOT NULL               -- stable per source
display_name VARCHAR(255) NOT NULL DEFAULT ''
download_url TEXT NOT NULL
expected_vpk_name VARCHAR(255) NOT NULL DEFAULT ''
expected_size BIGINT NULL
expected_md5 VARCHAR(32) NOT NULL DEFAULT ''
etag VARCHAR(255) NOT NULL DEFAULT ''
last_modified VARCHAR(255) NOT NULL DEFAULT ''
content_length BIGINT NULL
last_downloaded_at DATETIME NULL
last_error TEXT NOT NULL DEFAULT ''
created_at DATETIME NOT NULL
updated_at DATETIME NOT NULL
UNIQUE(source_id, item_key)
```

For `l4d2center`, `item_key` and `expected_vpk_name` come from the CSV `Name` column, and `expected_size` / `expected_md5` come from the CSV.

For `cedapug`, `item_key` is the direct download URL path basename, normalized without query parameters. CEDAPUG does not publish checksums in the observed page, so integrity uses HTTP metadata when available and archive extraction checks.

### `global_overlay_item_files`

One row per extracted VPK file that should appear in an overlay:

```text
id INTEGER PRIMARY KEY
item_id INTEGER NOT NULL REFERENCES global_overlay_items(id) ON DELETE CASCADE
vpk_name VARCHAR(255) NOT NULL
cache_path TEXT NOT NULL                     -- relative path under global_overlay_cache
size BIGINT NOT NULL
md5 VARCHAR(32) NOT NULL DEFAULT ''
created_at DATETIME NOT NULL
updated_at DATETIME NOT NULL
UNIQUE(item_id, vpk_name)
```

This extra file table handles archives that contain more than one `.vpk` without overloading the item row.

## Filesystem Layout

Use a cache separate from Steam Workshop:

```text
${LEFT4ME_ROOT}/
  global_overlay_cache/
    l4d2center-maps/
      archives/
      vpks/
    cedapug-maps/
      archives/
      vpks/
  overlays/
    {overlay_id}/
      left4dead2/addons/
        *.vpk -> absolute symlink to global_overlay_cache/.../vpks/*.vpk
```

Cache file writes are atomic: download to `*.partial`, extract to a temporary directory, verify, then `os.replace()` final VPK files.

Symlink targets are absolute, matching the existing Workshop overlay design.

## Source Parsing

### L4D2Center

Fetch `https://l4d2center.com/maps/servers/index.csv` with a normal HTTP timeout.

The CSV is semicolon-delimited and contains:

```text
Name;Size;md5;Download link
```

Each item produces:

- `item_key = Name`
- `expected_vpk_name = Name`
- `expected_size = Size`
- `expected_md5 = md5`
- `download_url = Download link`

Downloads are `.7z` archives. Extraction uses a Python 7z implementation such as `py7zr` so tests do not depend on a system `7z` binary. After extraction, the expected VPK file must exist and match both size and md5. A mismatch fails that item and leaves the prior cached file in place.

### CEDAPUG

Fetch `https://cedapug.com/custom` and parse the embedded `renderCustomMapDownloads([...])` data.

Only direct download links are managed in v1:

- Relative links like `/maps/FatalFreight.zip` are converted to absolute `https://cedapug.com/maps/FatalFreight.zip`.
- External `http` links are logged and skipped in v1.
- Entries without a download link are built-in campaigns and skipped.

Downloads are `.zip` archives extracted with Python's standard `zipfile`. Every `.vpk` in the archive becomes a managed output file for that item. If no `.vpk` is present, the item fails and the prior cached files remain in place.

Because CEDAPUG does not publish checksums in the observed page, refresh detects changes using `ETag`, `Last-Modified`, `Content-Length`, and local extracted file metadata when available. A manual refresh can force revalidation by clearing item metadata in a later maintenance path; no force-refresh UI is included in this design.

## Refresh Job

`refresh_global_overlays` is a global worker operation.

Behavior:

1. Ensure both managed global overlays and source rows exist.
2. Fetch both manifests.
3. Upsert manifest items.
4. Mark items absent from the manifest as no longer desired by deleting their item rows; cascading deletes remove their file rows.
5. Download and extract new or changed items.
6. Keep prior cache files when an item download or verification fails, but record `last_error`.
7. Rebuild symlinks for changed sources directly through the same builder interface used by `build_overlay`.
8. Emit clear job logs: manifest counts, downloads, skips, removals, verification failures, and build summaries.

`refresh_global_overlays` does not enqueue child `build_overlay` jobs. Direct builder invocation keeps the overlay in sync before the refresh job releases its global mutex, so a server job cannot start against updated cache metadata but stale overlay symlinks.

Coalescing:

- If a `refresh_global_overlays` job is queued or running, CLI/admin requests return the existing job instead of inserting a duplicate.

## Builder Reconciliation

`GlobalMapOverlayBuilder` reads desired file rows for the overlay's source and reconciles only symlinks it manages.

Managed symlink rule:

- A symlink in `left4dead2/addons/` is managed if its resolved target is under `${LEFT4ME_ROOT}/global_overlay_cache/{source_key}/vpks/`.
- Managed symlinks absent from desired files are removed.
- Desired files missing from cache are skipped and logged as errors.
- Non-symlink files and symlinks outside the source cache are left untouched and logged as foreign entries.

This mirrors `WorkshopBuilder` behavior and keeps manual files safe.

## Scheduler Rules

`refresh_global_overlays` joins the existing global mutex group.

It must not run concurrently with:

- `install`
- `refresh_workshop_items`
- any `build_overlay`
- any server job (`initialize`, `start`, `stop`, `delete`)

No server or overlay job may start while `refresh_global_overlays` is running.

This conservative rule is acceptable because daily map refreshes are rare and large downloads should not race runtime changes.

## CLI And Systemd Timer

Add Flask CLI command:

```text
flask refresh-global-overlays
```

The command:

- Loads app config and DB.
- Ensures global overlays exist.
- Enqueues or returns the existing `refresh_global_overlays` job.
- Prints the job id.
- Does not run downloads itself.

Add deployment units:

```text
left4me-refresh-global-overlays.service
left4me-refresh-global-overlays.timer
```

Service command:

```text
/opt/left4me/.venv/bin/flask --app l4d2web.app:create_app refresh-global-overlays
```

Timer policy:

```text
OnCalendar=daily
Persistent=true
```

The service runs as the `left4me` user with `/etc/left4me/host.env` and `/etc/left4me/web.env`, matching `left4me-web.service`.

## Permissions And UI

Overlay list behavior:

- Admins see all overlays, including managed global map overlays.
- Non-admin users see system overlays and their own private workshop overlays.
- Managed global overlays appear in blueprint overlay selection for every user.

Creation behavior:

- Non-admin users can create only user-creatable types, currently `workshop`.
- Admins can create normal admin-creatable types, currently `external` and `workshop`.
- No user-facing create form offers `l4d2center_maps` or `cedapug_maps`.
- Auto-seeding is the only creation path for managed global map overlay types.

Admin controls:

- Add a manual "Refresh global overlays" action in the admin area.
- The action enqueues the same coalesced `refresh_global_overlays` job as the timer.
- Managed overlay detail pages show source type, source URL, last refresh time, last error, item count, and latest related jobs.

## Error Handling

- Manifest fetch failure fails the job if no source can be processed. If one source succeeds and one fails, the job should still finish failed with partial-success logs and preserve prior content for the failed source.
- Per-item download failures do not abort sibling items.
- Verification failures keep prior cached files and record `last_error` on the item.
- Extraction rejects path traversal entries and ignores non-VPK files.
- Unsupported CEDAPUG external links are skipped with a warning.
- Initialize-time checks fail if desired global map files are missing from cache, naming the overlay and missing VPK names.

## Tests

Test coverage should include:

- Auto-seeding creates exactly one source overlay per source and repairs missing source rows.
- `jobs.user_id` nullable behavior, outer joins, and `system` display.
- Non-admins cannot access system jobs directly.
- CLI coalesces queued/running `refresh_global_overlays` jobs.
- Scheduler truth table for the new global operation.
- L4D2Center CSV parser with semicolon-delimited fixture data.
- CEDAPUG embedded JavaScript parser with fixture HTML.
- L4D2Center download/extract verifies VPK size and md5.
- CEDAPUG download/extract records every VPK in a zip archive.
- Reconcile removes obsolete managed symlinks and leaves foreign files alone.
- Overlay create UI rejects managed singleton types.
- Blueprint overlay selection includes managed global overlays for all users.
- Deployment tests cover the service and timer artifacts.

## Out Of Scope

- User-created global map source overlays.
- Arbitrary configurable HTTP manifest sources.
- Force-refresh UI for CEDAPUG items.
- Cache garbage collection for unreferenced archive files.
- Client-side map download UX.
- Steam Workshop links discovered on the CEDAPUG page; those are skipped rather than imported into workshop overlays.
- Host-library awareness of managed overlay types.

## Implementation Boundaries

- `l4d2host` remains unchanged.
- The web app continues to call host operations only through `l4d2ctl`.
- Existing blueprint semantics remain unchanged: overlays are live-linked, ordered, and first overlay has highest precedence.
- Existing workshop overlay behavior remains unchanged except scheduler interactions with the new global operation.