left4me/docs/superpowers/specs/2026-05-08-overlay-file-tree-design.md

Overlay File Tree Section Design

Goal: Add a "Files" section to the overlay detail page (/overlays/<id>) that renders a collapsible tree of the overlay's runtime directory at ${LEFT4ME_ROOT}/overlays/{overlay.id}/, with lazy expansion of folders (one fetch per first-time expand) and click-to-download for individual files. Same access rule as the rest of the overlay detail page (admin or overlay.user_id == g.user.id). Read-only; no rename/delete/upload in v1.

Approval status: User-approved 2026-05-08 (visual companion brainstorm + plan-mode review). Implemented + deployed in the same session. The lazy-load originally targeted HTMX (vendored in base.html), but the post-deploy smoke uncovered that static/vendor/htmx.min.js was a 33-byte placeholder — the real library was never vendored. Rather than vendoring full HTMX for one feature, the lazy-load was switched to plain JS using the same fetch + innerHTML pattern (~30 lines in static/js/file-tree.js). The route + partial contracts are unchanged.

Context

Today, the overlay detail page shows the row's metadata (name, type, scope, path, last build status), a workshop-items table or script editor depending on overlay.type, and links to the build-job stream. It never shows what's actually inside the overlay directory on disk. To verify "did my script actually produce what I expected?" or "did the right VPKs land in addons/?" the user has to SSH into the host and ls /var/lib/left4me/overlays/{id}/.

Click-to-download is a secondary nice-to-have: workshop overlays' addons/*.vpk are absolute symlinks into the shared ${LEFT4ME_ROOT}/workshop_cache/, and pulling a single VPK to a dev box otherwise means scping with the right path translation.

Locked Decisions

1. All overlay types show the section. Script + workshop + system/managed. Consistency over a tighter scope; even workshop's predictable addons/*.vpk layout is worth confirming.
2. Collapsible tree, lazy load on expand. Tree can get large; only the root level is rendered server-side at first paint. Each folder click fires GET /overlays/<id>/files?path=<rel> and innerHTMLs the response into that folder's .file-tree-children div. The no-JS path still shows the root level (the same partial is server-rendered) — folders just won't expand.
3. Single delegated JS handler. static/js/file-tree.js listens for click on document, finds the closest .file-tree-toggle button, toggles aria-expanded + hidden, and on first expand fires a fetch() against the URL in the button's data-files-url. Subsequent toggles never re-fetch (button.dataset.loaded flag, set optimistically before the fetch to dedupe rapid clicks; cleared on error to allow retry).
4. Single-file download in v1. No bulk archive (e.g., "download whole overlay as .tar.gz"). Files are streamed via Flask send_file(..., as_attachment=True). No size cap — VPKs are commonly 100–500 MB and that's the whole point.
5. No auto-refresh. The tree reflects what was on disk at page render. After a build, the user reloads the page. Polling/SSE would duplicate the existing live-log mechanism on the build-job page for negligible benefit.
6. Same access rule as the rest of the page. g.user.admin or overlay.user_id is None or overlay.user_id == g.user.id. GETs need no CSRF (l4d2web/app.py:56).
7. overlay.path not overlay.id. The runtime directory is reached via overlay.path (current creation flow guarantees path == str(id), but legacy/seeded rows may differ). Path resolution happens through the existing l4d2host.paths.overlay_path() helper, which already validates the ref string and resolves+verifies it stays under ${LEFT4ME_ROOT}/overlays/.
8. Empty / unresolvable → empty state. If the overlay's path is unresolvable (legacy absolute-path rows) or the directory doesn't exist (overlay never built), the section renders "No files yet — build this overlay to populate it." rather than crashing.
9. 500-entry cap per folder. Folders with more than 500 children render the alphabetical-first 500 plus a + M more (truncated) footer. Tunable at runtime via l4d2web.services.overlay_files.DEFAULT_MAX_ENTRIES (re-resolved per call so tests can monkeypatch).
10. Hidden files shown. No filtering of .git, .DS_Store, etc. Users want ground truth.
11. One dedicated blueprint, files_bp. Not folded into overlay_routes.py (which is exclusively POST mutations) or page_routes.py (top-level pages, not embedded fragments). files_bp owns both the tree fragment and the download endpoint.

Architecture

GET /overlays/<id>                                 (page_routes.overlay_detail)
   │
   ▼  computes (file_tree_root_entries, truncated_count) via
   │  _root_file_tree(overlay) → safe_resolve_for_listing(overlay.path, "")
   │                          → list_directory(overlay_root, overlay_root)
   │
   ▼  renders overlay_detail.html, which includes _overlay_file_tree.html
       for the root level (or the empty-state <p>).

GET /overlays/<id>/files?path=<rel>                (files_routes.overlay_files_fragment)
   │
   ▼  auth gate (admin or owner)
   │
   ▼  safe_resolve_for_listing(overlay.path, rel) → Path under overlay_root
   │
   ▼  list_directory(target, overlay_root) → entries[], truncated_count
   │
   ▼  renders _overlay_file_tree.html (partial only — no base.html)

GET /overlays/<id>/files/download?path=<rel>       (files_routes.overlay_files_download)
   │
   ▼  auth gate
   │
   ▼  safe_resolve_for_download(overlay.path, rel) → real Path under LEFT4ME_ROOT
   │  (follows symlinks; allows targets anywhere in LEFT4ME_ROOT, e.g. workshop_cache)
   │
   ▼  Flask send_file(real, as_attachment=True, download_name=basename(real))

File tree fragment shape

_overlay_file_tree.html produces a <ul class="file-tree"> containing one _overlay_file_node.html row per entry plus an optional truncated-footer <li>. A folder row is a <button class="file-tree-toggle" data-files-url="…"> followed by an empty <div class="file-tree-children" hidden> that becomes the fetch target. A file row is an <a href=".../files/download?path=…"> (or a plain <span> for broken symlinks) plus optional badges (link, broken link) and the resolved size.

Nesting after expand:

<li class="file-tree-row file-tree-row-dir">
  <button class="file-tree-toggle" aria-expanded="true" data-files-url="…">…</button>
  <div class="file-tree-children">
    <ul class="file-tree" role="group">…</ul>   <!-- inserted by file-tree.js -->
  </div>
</li>

Path Safety

Two resolvers in l4d2web/services/overlay_files.py:

• safe_resolve_for_listing(overlay.path, sub_path) — resolves overlay_root / sub_path, applies Path.resolve(strict=False), requires the result to be the overlay root or a descendant. Used by the tree-fragment route. Refuses to recurse through symlinks that leave the overlay root, including symlinks pointing into workshop_cache/ — listing has no need to follow them, since workshop addons are leaf files, not directories we'd descend into.
• safe_resolve_for_download(overlay.path, sub_path) — resolves the candidate path, applies os.path.realpath(), requires the result to be under ${LEFT4ME_ROOT} (anywhere — overlay dir, workshop_cache/, future siblings). This is the relaxed gate that lets workshop addons stream from the shared cache while still blocking absolute symlinks to /etc/passwd planted by a malicious script overlay.

Both resolvers re-use l4d2host.paths.overlay_path() (which itself calls validate_overlay_ref) for the overlay-root resolution, and l4d2web.services.security.validate_overlay_ref for the sub-path component (rejects empty / . / .. / absolute / whitespace / backslash). Empty sub_path is valid for listing (means "the overlay root") and invalid for download.

Listing: target.is_dir() check after resolution; non-directory → 404.

Download: real.exists() check (404), real.is_dir() rejection (400 — "not a file").

Symlink Behaviour

list_directory uses os.scandir() with explicit follow_symlinks flags:

• is_symlink = entry.is_symlink()
• kind: entry.is_dir(follow_symlinks=True) inside a try block. Raised OSError → broken symlink, treated as kind="file" with broken=True and no <a> download link.
• size: entry.stat(follow_symlinks=True).st_size for files (resolved target's size — what users care about for VPKs); None for dirs and broken symlinks.

Symlinked directories pointing inside the overlay root are rendered as folders and remain expandable; the listing-time safety check rejects expansion if the symlink resolves outside the overlay root.

Concurrent build vs listing race: a build mid-symlink-rewrite can yield a transient broken-symlink view. Acceptable — page is a snapshot; the visible "broken link" badge tells the user to refresh.

Test Strategy

Two test modules, both following existing fixture patterns (tests/test_script_overlay_routes.py style — monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path)), app fixture with TESTING=True).

• tests/test_overlay_files.py — pure-helper unit tests (Flask-free): listing-resolver happy/sad paths (root, sub-path, .., absolute, empty component, symlink-out-of-overlay), download-resolver happy/sad paths (regular file, workshop-cache symlink, outside-LEFT4ME_ROOT symlink, traversal, absolute, empty), list_directory behaviour (empty, dir-first sort, kind detection, rel paths, symlink markers, broken-symlink markers, truncation cap, human-size formatting).
• tests/test_overlay_files_routes.py — HTTP integration tests: tree-fragment 200 / 400 / 403 / 404 across the same axes; download 200 / 400 / 403 / 404 + content-disposition + byte-exact body for both regular files and workshop-cache symlinks; admin-can-view-foreign overlay; truncation-via-route (monkeypatching DEFAULT_MAX_ENTRIES); broken-symlink rendering omits the <a> download link; the page-level overlay_detail integration shows the section with entries when populated and the empty state when the directory is missing.

39 tests total. The full web suite (pytest l4d2web/tests/ -q) must remain green.

Out of Scope

• Bulk download (e.g., "download overlay as tar.gz").
• Inline file preview (text peek, image thumbnail).
• File deletion / rename / upload from the UI.
• Auto-refresh while a build is active.
• Filtering hidden files or applying a .gitignore-style rule.
• Reusable file-tree component for things outside overlays.