left4me/docs/superpowers/specs/2026-05-07-l4d2-global-map-overlays-design.md

L4D2 Global Map Overlays Design

Goal: Add two managed, system-wide map overlays, l4d2center-maps and cedapug-maps, populated from upstream map sources and refreshed daily through the existing job system.

Approval status: User-approved design direction. Implementation must not start until this spec is reviewed and an implementation plan is written.

Context

left4me already has typed overlays, a builder registry, global overlays through Overlay.user_id = NULL, and queued overlay build jobs. Steam Workshop overlays use a cache plus symlinks into left4dead2/addons/, and server initialization already runs overlay builders before calling l4d2ctl initialize.

Global map sources fit the same model. The host library remains unchanged: it receives overlay refs and mounts directories. The web app owns map-source fetching, cache management, reconciliation, and job logs.

The two upstream sources are:

• https://l4d2center.com/maps/servers/index.csv
• https://cedapug.com/custom

Locked Decisions

1. One general operation. Use refresh_global_overlays, not source-specific cron operations.
2. Systemd owns time. A systemd timer runs daily and invokes a Flask CLI command. The CLI only enqueues work; the existing worker performs downloads and writes logs.
3. System jobs are nullable-owner jobs. jobs.user_id becomes nullable. NULL means the job was created by the system. UI displays owner as system. Only admins can access system jobs.
4. Managed global overlays are auto-seeded. The app creates or repairs exactly one l4d2center-maps overlay and exactly one cedapug-maps overlay.
5. Global overlays are normal system overlays for users. Overlay.user_id = NULL makes them visible to every authenticated user and selectable in every user's blueprint editor.
6. Managed types are not user-creatable. Normal overlay creation does not offer l4d2center_maps or cedapug_maps. The seeder is the only code path that creates those types.
7. Exact reconciliation. Refresh makes each managed overlay match its upstream manifest. Removed upstream maps are removed from the managed overlay symlink set. Foreign files are left alone and logged.
8. No initialize-time downloads. initialize_server() may run builders to repair symlinks, but it must not fetch remote manifests or download large archives. Missing cache content fails clearly.
9. Separate cache from Workshop. Non-Steam global maps use ${LEFT4ME_ROOT}/global_overlay_cache, not ${LEFT4ME_ROOT}/workshop_cache.
10. Source-specific parsing stays explicit. Do not introduce a generic arbitrary HTTP source framework in this phase.

Architecture

The design extends the existing overlay-builder registry:

BUILDERS = {
    "external": ExternalBuilder(),
    "workshop": WorkshopBuilder(),
    "l4d2center_maps": GlobalMapOverlayBuilder(),
    "cedapug_maps": GlobalMapOverlayBuilder(),
}

Both global map overlay types share the same filesystem builder. Source-specific code lives in refresh services that know how to fetch and parse upstream manifests.

High-level flow:

systemd timer
  -> flask refresh-global-overlays
     -> ensure_global_overlays()
     -> enqueue refresh_global_overlays job (coalesced)
        -> worker fetches manifests
        -> worker downloads/extracts cache files
        -> worker records desired VPK files
        -> worker rebuilds overlay symlinks directly

Auto-seeded overlay rows use fixed names, managed types, user_id = NULL, and web-generated paths:

name=l4d2center-maps, type=l4d2center_maps, user_id=NULL, path=str(id)
name=cedapug-maps, type=cedapug_maps, user_id=NULL, path=str(id)

Data Model

jobs

Change jobs.user_id from required to nullable.

NULL means a system-created job. Authorization rules become:

• Admins can view, stream, and cancel every job, including system jobs.
• Non-admins can access only jobs where job.user_id == current_user.id.
• System jobs are not visible to non-admins through direct job URLs.

Job list/detail pages use outer joins to users and render missing owners as system.

global_overlay_sources

One row per managed global source overlay:

id INTEGER PRIMARY KEY
overlay_id INTEGER NOT NULL UNIQUE REFERENCES overlays(id) ON DELETE CASCADE
source_key VARCHAR(64) NOT NULL UNIQUE       -- l4d2center-maps | cedapug-maps
source_type VARCHAR(32) NOT NULL             -- l4d2center_csv | cedapug_custom_page
source_url TEXT NOT NULL
last_manifest_hash VARCHAR(64) NOT NULL DEFAULT ''
last_refreshed_at DATETIME NULL
last_error TEXT NOT NULL DEFAULT ''
created_at DATETIME NOT NULL
updated_at DATETIME NOT NULL

source_key is stable and used by the seeder to repair missing rows.

global_overlay_items

One row per manifest item belonging to a global overlay source:

id INTEGER PRIMARY KEY
source_id INTEGER NOT NULL REFERENCES global_overlay_sources(id) ON DELETE CASCADE
item_key VARCHAR(255) NOT NULL               -- stable per source
display_name VARCHAR(255) NOT NULL DEFAULT ''
download_url TEXT NOT NULL
expected_vpk_name VARCHAR(255) NOT NULL DEFAULT ''
expected_size BIGINT NULL
expected_md5 VARCHAR(32) NOT NULL DEFAULT ''
etag VARCHAR(255) NOT NULL DEFAULT ''
last_modified VARCHAR(255) NOT NULL DEFAULT ''
content_length BIGINT NULL
last_downloaded_at DATETIME NULL
last_error TEXT NOT NULL DEFAULT ''
created_at DATETIME NOT NULL
updated_at DATETIME NOT NULL
UNIQUE(source_id, item_key)

For l4d2center, item_key and expected_vpk_name come from the CSV Name column, and expected_size / expected_md5 come from the CSV.

For cedapug, item_key is the direct download URL path basename, normalized without query parameters. CEDAPUG does not publish checksums in the observed page, so integrity uses HTTP metadata when available and archive extraction checks.

global_overlay_item_files

One row per extracted VPK file that should appear in an overlay:

id INTEGER PRIMARY KEY
item_id INTEGER NOT NULL REFERENCES global_overlay_items(id) ON DELETE CASCADE
vpk_name VARCHAR(255) NOT NULL
cache_path TEXT NOT NULL                     -- relative path under global_overlay_cache
size BIGINT NOT NULL
md5 VARCHAR(32) NOT NULL DEFAULT ''
created_at DATETIME NOT NULL
updated_at DATETIME NOT NULL
UNIQUE(item_id, vpk_name)

This extra file table handles archives that contain more than one .vpk without overloading the item row.

Filesystem Layout

Use a cache separate from Steam Workshop:

${LEFT4ME_ROOT}/
  global_overlay_cache/
    l4d2center-maps/
      archives/
      vpks/
    cedapug-maps/
      archives/
      vpks/
  overlays/
    {overlay_id}/
      left4dead2/addons/
        *.vpk -> absolute symlink to global_overlay_cache/.../vpks/*.vpk

Cache file writes are atomic: download to *.partial, extract to a temporary directory, verify, then os.replace() final VPK files.

Symlink targets are absolute, matching the existing Workshop overlay design.

Source Parsing

L4D2Center

Fetch https://l4d2center.com/maps/servers/index.csv with a normal HTTP timeout.

The CSV is semicolon-delimited and contains:

Name;Size;md5;Download link

Each item produces:

• item_key = Name
• expected_vpk_name = Name
• expected_size = Size
• expected_md5 = md5
• download_url = Download link

Downloads are .7z archives. Extraction uses a Python 7z implementation such as py7zr so tests do not depend on a system 7z binary. After extraction, the expected VPK file must exist and match both size and md5. A mismatch fails that item and leaves the prior cached file in place.

CEDAPUG

Fetch https://cedapug.com/custom and parse the embedded renderCustomMapDownloads([...]) data.

Only direct download links are managed in v1:

• Relative links like /maps/FatalFreight.zip are converted to absolute https://cedapug.com/maps/FatalFreight.zip.
• External http links are logged and skipped in v1.
• Entries without a download link are built-in campaigns and skipped.

Downloads are .zip archives extracted with Python's standard zipfile. Every .vpk in the archive becomes a managed output file for that item. If no .vpk is present, the item fails and the prior cached files remain in place.

Because CEDAPUG does not publish checksums in the observed page, refresh detects changes using ETag, Last-Modified, Content-Length, and local extracted file metadata when available. A manual refresh can force revalidation by clearing item metadata in a later maintenance path; no force-refresh UI is included in this design.

Refresh Job

refresh_global_overlays is a global worker operation.

Behavior:

1. Ensure both managed global overlays and source rows exist.
2. Fetch both manifests.
3. Upsert manifest items.
4. Mark items absent from the manifest as no longer desired by deleting their item rows; cascading deletes remove their file rows.
5. Download and extract new or changed items.
6. Keep prior cache files when an item download or verification fails, but record last_error.
7. Rebuild symlinks for changed sources directly through the same builder interface used by build_overlay.
8. Emit clear job logs: manifest counts, downloads, skips, removals, verification failures, and build summaries.

refresh_global_overlays does not enqueue child build_overlay jobs. Direct builder invocation keeps the overlay in sync before the refresh job releases its global mutex, so a server job cannot start against updated cache metadata but stale overlay symlinks.

Coalescing:

• If a refresh_global_overlays job is queued or running, CLI/admin requests return the existing job instead of inserting a duplicate.

Builder Reconciliation

GlobalMapOverlayBuilder reads desired file rows for the overlay's source and reconciles only symlinks it manages.

Managed symlink rule:

• A symlink in left4dead2/addons/ is managed if its resolved target is under ${LEFT4ME_ROOT}/global_overlay_cache/{source_key}/vpks/.
• Managed symlinks absent from desired files are removed.
• Desired files missing from cache are skipped and logged as errors.
• Non-symlink files and symlinks outside the source cache are left untouched and logged as foreign entries.

This mirrors WorkshopBuilder behavior and keeps manual files safe.

Scheduler Rules

refresh_global_overlays joins the existing global mutex group.

It must not run concurrently with:

• install
• refresh_workshop_items
• any build_overlay
• any server job (initialize, start, stop, delete)

No server or overlay job may start while refresh_global_overlays is running.

This conservative rule is acceptable because daily map refreshes are rare and large downloads should not race runtime changes.

CLI And Systemd Timer

Add Flask CLI command:

flask refresh-global-overlays

The command:

• Loads app config and DB.
• Ensures global overlays exist.
• Enqueues or returns the existing refresh_global_overlays job.
• Prints the job id.
• Does not run downloads itself.

Add deployment units:

left4me-refresh-global-overlays.service
left4me-refresh-global-overlays.timer

Service command:

/opt/left4me/.venv/bin/flask --app l4d2web.app:create_app refresh-global-overlays

Timer policy:

OnCalendar=daily
Persistent=true

The service runs as the left4me user with /etc/left4me/host.env and /etc/left4me/web.env, matching left4me-web.service.

Permissions And UI

Overlay list behavior:

• Admins see all overlays, including managed global map overlays.
• Non-admin users see system overlays and their own private workshop overlays.
• Managed global overlays appear in blueprint overlay selection for every user.

Creation behavior:

• Non-admin users can create only user-creatable types, currently workshop.
• Admins can create normal admin-creatable types, currently external and workshop.
• No user-facing create form offers l4d2center_maps or cedapug_maps.
• Auto-seeding is the only creation path for managed global map overlay types.

Admin controls:

• Add a manual "Refresh global overlays" action in the admin area.
• The action enqueues the same coalesced refresh_global_overlays job as the timer.
• Managed overlay detail pages show source type, source URL, last refresh time, last error, item count, and latest related jobs.

Error Handling

• Manifest fetch failure fails the job if no source can be processed. If one source succeeds and one fails, the job should still finish failed with partial-success logs and preserve prior content for the failed source.
• Per-item download failures do not abort sibling items.
• Verification failures keep prior cached files and record last_error on the item.
• Extraction rejects path traversal entries and ignores non-VPK files.
• Unsupported CEDAPUG external links are skipped with a warning.
• Initialize-time checks fail if desired global map files are missing from cache, naming the overlay and missing VPK names.

Tests

Test coverage should include:

• Auto-seeding creates exactly one source overlay per source and repairs missing source rows.
• jobs.user_id nullable behavior, outer joins, and system display.
• Non-admins cannot access system jobs directly.
• CLI coalesces queued/running refresh_global_overlays jobs.
• Scheduler truth table for the new global operation.
• L4D2Center CSV parser with semicolon-delimited fixture data.
• CEDAPUG embedded JavaScript parser with fixture HTML.
• L4D2Center download/extract verifies VPK size and md5.
• CEDAPUG download/extract records every VPK in a zip archive.
• Reconcile removes obsolete managed symlinks and leaves foreign files alone.
• Overlay create UI rejects managed singleton types.
• Blueprint overlay selection includes managed global overlays for all users.
• Deployment tests cover the service and timer artifacts.

Out Of Scope

• User-created global map source overlays.
• Arbitrary configurable HTTP manifest sources.
• Force-refresh UI for CEDAPUG items.
• Cache garbage collection for unreferenced archive files.
• Client-side map download UX.
• Steam Workshop links discovered on the CEDAPUG page; those are skipped rather than imported into workshop overlays.
• Host-library awareness of managed overlay types.

Implementation Boundaries

• l4d2host remains unchanged.
• The web app continues to call host operations only through l4d2ctl.
• Existing blueprint semantics remain unchanged: overlays are live-linked, ordered, and first overlay has highest precedence.
• Existing workshop overlay behavior remains unchanged except scheduler interactions with the new global operation.