left4me/docs/superpowers/specs/2026-05-08-overlay-file-tree-design.md

# Overlay File Tree Section Design

**Goal:** Add a "Files" section to the overlay detail page (`/overlays/<id>`) that renders a collapsible tree of the overlay's runtime directory at `${LEFT4ME_ROOT}/overlays/{overlay.id}/`, with lazy expansion of folders (one fetch per first-time expand) and click-to-download for individual files. Same access rule as the rest of the overlay detail page (admin or `overlay.user_id == g.user.id`). Read-only; no rename/delete/upload in v1.

**Approval status:** User-approved 2026-05-08 (visual companion brainstorm + plan-mode review). Implemented + deployed in the same session. The lazy-load originally targeted HTMX (vendored in `base.html`), but the post-deploy smoke uncovered that `static/vendor/htmx.min.js` was a 33-byte placeholder — the real library was never vendored. Rather than vendoring full HTMX for one feature, the lazy-load was switched to plain JS using the same fetch + innerHTML pattern (~30 lines in `static/js/file-tree.js`). The route + partial contracts are unchanged.

## Context

Today, the overlay detail page shows the row's metadata (name, type, scope, path, last build status), a workshop-items table or script editor depending on `overlay.type`, and links to the build-job stream. It never shows what's actually inside the overlay directory on disk. To verify "did my script actually produce what I expected?" or "did the right VPKs land in `addons/`?" the user has to SSH into the host and `ls /var/lib/left4me/overlays/{id}/`.

Click-to-download is a secondary nice-to-have: workshop overlays' `addons/*.vpk` are absolute symlinks into the shared `${LEFT4ME_ROOT}/workshop_cache/`, and pulling a single VPK to a dev box otherwise means scping with the right path translation.

## Locked Decisions

1. **All overlay types show the section.** Script + workshop + system/managed. Consistency over a tighter scope; even workshop's predictable `addons/*.vpk` layout is worth confirming.
2. **Collapsible tree, lazy load on expand.** Tree can get large; only the root level is rendered server-side at first paint. Each folder click fires `GET /overlays/<id>/files?path=<rel>` and innerHTMLs the response into that folder's `.file-tree-children` div. The no-JS path still shows the root level (the same partial is server-rendered) — folders just won't expand.
3. **Single delegated JS handler.** `static/js/file-tree.js` listens for `click` on `document`, finds the closest `.file-tree-toggle` button, toggles `aria-expanded` + `hidden`, and on first expand fires a `fetch()` against the URL in the button's `data-files-url`. Subsequent toggles never re-fetch (`button.dataset.loaded` flag, set optimistically before the fetch to dedupe rapid clicks; cleared on error to allow retry).
4. **Single-file download in v1.** No bulk archive (e.g., "download whole overlay as `.tar.gz`"). Files are streamed via Flask `send_file(..., as_attachment=True)`. No size cap — VPKs are commonly 100–500 MB and that's the whole point.
5. **No auto-refresh.** The tree reflects what was on disk at page render. After a build, the user reloads the page. Polling/SSE would duplicate the existing live-log mechanism on the build-job page for negligible benefit.
6. **Same access rule as the rest of the page.** `g.user.admin or overlay.user_id is None or overlay.user_id == g.user.id`. GETs need no CSRF (`l4d2web/app.py:56`).
7. **`overlay.path` not `overlay.id`.** The runtime directory is reached via `overlay.path` (current creation flow guarantees `path == str(id)`, but legacy/seeded rows may differ). Path resolution happens through the existing `l4d2host.paths.overlay_path()` helper, which already validates the ref string and resolves+verifies it stays under `${LEFT4ME_ROOT}/overlays/`.
8. **Empty / unresolvable → empty state.** If the overlay's path is unresolvable (legacy absolute-path rows) or the directory doesn't exist (overlay never built), the section renders "No files yet — build this overlay to populate it." rather than crashing.
9. **500-entry cap per folder.** Folders with more than 500 children render the alphabetical-first 500 plus a `+ M more (truncated)` footer. Tunable at runtime via `l4d2web.services.overlay_files.DEFAULT_MAX_ENTRIES` (re-resolved per call so tests can monkeypatch).
10. **Hidden files shown.** No filtering of `.git`, `.DS_Store`, etc. Users want ground truth.
11. **One dedicated blueprint, `files_bp`.** Not folded into `overlay_routes.py` (which is exclusively POST mutations) or `page_routes.py` (top-level pages, not embedded fragments). `files_bp` owns both the tree fragment and the download endpoint.

## Architecture

```text
GET /overlays/<id>                                 (page_routes.overlay_detail)
   │
   ▼  computes (file_tree_root_entries, truncated_count) via
   │  _root_file_tree(overlay) → safe_resolve_for_listing(overlay.path, "")
   │                          → list_directory(overlay_root, overlay_root)
   │
   ▼  renders overlay_detail.html, which includes _overlay_file_tree.html
       for the root level (or the empty-state <p>).

GET /overlays/<id>/files?path=<rel>                (files_routes.overlay_files_fragment)
   │
   ▼  auth gate (admin or owner)
   │
   ▼  safe_resolve_for_listing(overlay.path, rel) → Path under overlay_root
   │
   ▼  list_directory(target, overlay_root) → entries[], truncated_count
   │
   ▼  renders _overlay_file_tree.html (partial only — no base.html)

GET /overlays/<id>/files/download?path=<rel>       (files_routes.overlay_files_download)
   │
   ▼  auth gate
   │
   ▼  safe_resolve_for_download(overlay.path, rel) → real Path under LEFT4ME_ROOT
   │  (follows symlinks; allows targets anywhere in LEFT4ME_ROOT, e.g. workshop_cache)
   │
   ▼  Flask send_file(real, as_attachment=True, download_name=basename(real))
```

### File tree fragment shape

`_overlay_file_tree.html` produces a `<ul class="file-tree">` containing one `_overlay_file_node.html` row per entry plus an optional truncated-footer `<li>`. A folder row is a `<button class="file-tree-toggle" data-files-url="…">` followed by an empty `<div class="file-tree-children" hidden>` that becomes the fetch target. A file row is an `<a href=".../files/download?path=…">` (or a plain `<span>` for broken symlinks) plus optional badges (`link`, `broken link`) and the resolved size.

Nesting after expand:

```html
<li class="file-tree-row file-tree-row-dir">
  <button class="file-tree-toggle" aria-expanded="true" data-files-url="…">…</button>
  <div class="file-tree-children">
    <ul class="file-tree" role="group">…</ul>   <!-- inserted by file-tree.js -->
  </div>
</li>
```

## Path Safety

Two resolvers in `l4d2web/services/overlay_files.py`:

- `safe_resolve_for_listing(overlay.path, sub_path)` — resolves `overlay_root / sub_path`, applies `Path.resolve(strict=False)`, requires the result to be the overlay root or a descendant. Used by the tree-fragment route. **Refuses to recurse through symlinks that leave the overlay root**, including symlinks pointing into `workshop_cache/` — listing has no need to follow them, since workshop addons are leaf files, not directories we'd descend into.
- `safe_resolve_for_download(overlay.path, sub_path)` — resolves the candidate path, applies `os.path.realpath()`, requires the result to be under `${LEFT4ME_ROOT}` (anywhere — overlay dir, `workshop_cache/`, future siblings). This is the relaxed gate that lets workshop addons stream from the shared cache while still blocking absolute symlinks to `/etc/passwd` planted by a malicious script overlay.

Both resolvers re-use `l4d2host.paths.overlay_path()` (which itself calls `validate_overlay_ref`) for the overlay-root resolution, and `l4d2web.services.security.validate_overlay_ref` for the sub-path component (rejects empty / `.` / `..` / absolute / whitespace / backslash). Empty `sub_path` is valid for listing (means "the overlay root") and invalid for download.

Listing: `target.is_dir()` check after resolution; non-directory → 404.

Download: `real.exists()` check (404), `real.is_dir()` rejection (400 — "not a file").

## Symlink Behaviour

`list_directory` uses `os.scandir()` with explicit `follow_symlinks` flags:

- `is_symlink = entry.is_symlink()`
- `kind`: `entry.is_dir(follow_symlinks=True)` inside a try block. Raised `OSError` → broken symlink, treated as `kind="file"` with `broken=True` and no `<a>` download link.
- `size`: `entry.stat(follow_symlinks=True).st_size` for files (resolved target's size — what users care about for VPKs); `None` for dirs and broken symlinks.

Symlinked directories pointing inside the overlay root are rendered as folders and remain expandable; the listing-time safety check rejects expansion if the symlink resolves outside the overlay root.

Concurrent build vs listing race: a build mid-symlink-rewrite can yield a transient broken-symlink view. Acceptable — page is a snapshot; the visible "broken link" badge tells the user to refresh.

## Test Strategy

Two test modules, both following existing fixture patterns (`tests/test_script_overlay_routes.py` style — `monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))`, app fixture with `TESTING=True`).

- `tests/test_overlay_files.py` — pure-helper unit tests (Flask-free): listing-resolver happy/sad paths (root, sub-path, `..`, absolute, empty component, symlink-out-of-overlay), download-resolver happy/sad paths (regular file, workshop-cache symlink, outside-LEFT4ME_ROOT symlink, traversal, absolute, empty), `list_directory` behaviour (empty, dir-first sort, kind detection, rel paths, symlink markers, broken-symlink markers, truncation cap, human-size formatting).
- `tests/test_overlay_files_routes.py` — HTTP integration tests: tree-fragment 200 / 400 / 403 / 404 across the same axes; download 200 / 400 / 403 / 404 + content-disposition + byte-exact body for both regular files and workshop-cache symlinks; admin-can-view-foreign overlay; truncation-via-route (monkeypatching `DEFAULT_MAX_ENTRIES`); broken-symlink rendering omits the `<a>` download link; the page-level `overlay_detail` integration shows the section with entries when populated and the empty state when the directory is missing.

39 tests total. The full web suite (`pytest l4d2web/tests/ -q`) must remain green.

## Out of Scope

- Bulk download (e.g., "download overlay as tar.gz").
- Inline file preview (text peek, image thumbnail).
- File deletion / rename / upload from the UI.
- Auto-refresh while a build is active.
- Filtering hidden files or applying a `.gitignore`-style rule.
- Reusable file-tree component for things outside overlays.