docs(specs): overlay file tree — design + implementation plan

Captures the design rationale for the new overlay-detail Files section (verify build output, click-to-download for individual files via Flask send_file, HTMX-driven lazy folder expansion) and the paired implementation plan that produced it. Adds .superpowers/ to .gitignore so brainstorm session artifacts never sneak into a future commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 20:16:10 +02:00 · 2026-05-08 20:16:10 +02:00 · 76bd6e8d4d
commit 76bd6e8d4d
parent 1166e13e44
3 changed files with 279 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@ -9,3 +9,4 @@ __pycache__/
 l4d2web.db*
 # CocoIndex Code (ccc)
 /.cocoindex_code/
 .superpowers/
--- a/docs/superpowers/plans/2026-05-08-overlay-file-tree.md
+++ b/docs/superpowers/plans/2026-05-08-overlay-file-tree.md
@ -0,0 +1,161 @@
 # Overlay File Tree Implementation Plan
 > **Approval status:** User-approved 2026-05-08; implemented + deployed in the same session. This plan is committed retrospectively to record the work.
 **Goal:** Build the overlay-detail "Files" section per `docs/superpowers/specs/2026-05-08-overlay-file-tree-design.md` — a server-rendered collapsible tree of `${LEFT4ME_ROOT}/overlays/{overlay.id}/` with HTMX lazy expansion and click-to-download for individual files. Read-only; same access rule as the rest of the overlay detail page.
 **Architecture:** A new `files_bp` blueprint exposes two GETs: `/overlays/<id>/files?path=<rel>` returns the listing as an HTML fragment (used both for first paint at the root level via `page_routes.overlay_detail` context, and for HTMX swaps when a folder expands), and `/overlays/<id>/files/download?path=<rel>` streams a single file. Pure helpers live in `l4d2web/services/overlay_files.py`: `safe_resolve_for_listing` (refuses symlink escape from overlay root), `safe_resolve_for_download` (allows symlink targets anywhere under `LEFT4ME_ROOT` — workshop addons stream from the shared cache; absolute symlinks to `/etc/passwd` are still blocked), and `list_directory` (one-level scan, dirs-first sort, 500-entry cap, symlink + broken-symlink markers, resolved size for files). Two Jinja partials (`_overlay_file_tree.html`, `_overlay_file_node.html`) plus a 12-line event-delegated `static/js/file-tree.js` for collapse/re-expand handle the UI; styles append to `static/css/components.css` against existing tokens.
 ---
 ## Locked Decisions
 See the design doc for rationale. Implementation-relevant summary:
 - New blueprint `files_bp` registered in `l4d2web/app.py` next to `overlay_bp`.
 - Path resolution chains through `l4d2host.paths.overlay_path()` (already validates the overlay ref + resolves under `LEFT4ME_ROOT/overlays/`) and `l4d2web.services.security.validate_overlay_ref` (rejects empty/`.`/`..`/absolute/whitespace/backslash for the sub-path component).
 - Listing rule: target must be a descendant of `overlay_root` after `Path.resolve()`. Download rule: real path must be a descendant of `LEFT4ME_ROOT` after `os.path.realpath()`.
 - Tree shape: single recursive partial. `_overlay_file_tree.html` renders `<ul>`; `_overlay_file_node.html` renders one folder or file `<li>`. Folder buttons carry `hx-get`, `hx-target="next .file-tree-children"`, `hx-swap="innerHTML"`, `hx-trigger="click once"`. Subsequent toggles handled by `static/js/file-tree.js`.
 - `DEFAULT_MAX_ENTRIES = 500` in the helper module; re-resolved per call so tests can monkeypatch.
 - No changes to `l4d2host`, builders, or workshop/script edit flows.
 ---
 ## Task 1: Pure helpers — path safety + directory listing
 **Files:**
 - Create: `l4d2web/services/overlay_files.py` — `safe_resolve_for_listing`, `safe_resolve_for_download`, `list_directory`, `_format_size`, `DEFAULT_MAX_ENTRIES`.
 - Create: `l4d2web/tests/test_overlay_files.py` — 20 tests (path safety, listing semantics, symlink + broken-symlink handling, sort order, truncation cap, human-size formatting).
 Test plan (RED first):
 1. Listing returns overlay root for empty sub-path; joins under root for nested sub-path; rejects `..`, absolute path, empty component (`foo//bar`); rejects symlink escaping the overlay root even when target sits in `workshop_cache/`.
 2. Download rejects empty path; returns real path for a regular file; follows a symlink into `workshop_cache/`; rejects a symlink to a path outside `LEFT4ME_ROOT`; rejects `..` and absolute paths.
 3. `list_directory`: empty dir → empty list, truncated 0; dirs-first then files, both case-insensitive alphabetical; `kind ∈ {"dir", "file"}`; `rel` is forward-slash relative to overlay root; symlinks marked with `is_symlink=True` and resolved-target size; broken symlinks marked `broken=True` with `size=None`; truncation at supplied cap returns first N + `truncated_count`; `size_human` formats `5 B` and `3.0 MB` correctly.
 **Implementation:**
 - `safe_resolve_for_listing` calls `l4d2host.paths.overlay_path(overlay_path_value).resolve()` for the overlay root, short-circuits on empty `sub_path`, validates the sub-path via `validate_overlay_ref`, then `(overlay_root / sub_path).resolve(strict=False)` and asserts the result is the overlay root or a descendant.
 - `safe_resolve_for_download` rejects empty `sub_path`, validates, builds `overlay_root / sub_path`, applies `os.path.realpath()`, asserts the result is under `get_left4me_root().resolve()`.
 - `list_directory(target, overlay_root, *, max_entries=None)` uses `os.scandir` (free `stat` cache, `follow_symlinks` toggle). Per entry: `is_symlink = entry.is_symlink()`; `is_dir = entry.is_dir(follow_symlinks=True)` inside a try (OSError → broken=True, kind=file, size=None); regular files use `entry.stat(follow_symlinks=True).st_size`. `rel` is `"/".join(Path(entry.path).relative_to(overlay_root).parts)`. Sort by `(0 if dir else 1, name.casefold())`. Truncate to `max_entries or DEFAULT_MAX_ENTRIES`.
 - `_format_size`: bytes (`N B`, no decimal) up to 1024, then KB/MB/GB/TB at one decimal place.
 **Verification:**
 ```
 pytest l4d2web/tests/test_overlay_files.py -q
 ```
 **Commit:** part of Task 4's bundled `feat` commit.
 ---
 ## Task 2: HTTP routes — files_bp blueprint
 **Files:**
 - Create: `l4d2web/routes/files_routes.py` — `files_bp` with `GET /overlays/<id>/files` (fragment) and `GET /overlays/<id>/files/download` (stream).
 - Modify: `l4d2web/app.py` — `from l4d2web.routes.files_routes import bp as files_bp` and `app.register_blueprint(files_bp)` next to `overlay_bp`.
 - Create: `l4d2web/tests/test_overlay_files_routes.py` — 16 HTTP-level tests at this stage (3 more added in Task 4).
 Test plan (RED first):
 - Fragment: 200 + entries for root listing; 200 + entries for sub-directory; 400 on `..`, absolute path, empty component; 404 on unknown overlay; 404 on missing sub-dir; 403 on foreign user's overlay; 200 for admin viewing foreign overlay; truncation cap exposes "+ N more" footer (monkeypatch `DEFAULT_MAX_ENTRIES`); broken symlink rendered with `broken` badge and no `<a>` link.
 - Download: 200 + `Content-Disposition: attachment` + exact byte match for regular file; 200 + cache content for workshop-cache symlink; 400 for symlink resolving outside `LEFT4ME_ROOT`; 400 for directory target; 404 for missing file; 403 for foreign user's overlay.
 **Implementation:**
 - Decorator stack: `@files_bp.get(...)` + `@require_login`. Auth gate inside the handler mirrors `page_routes.overlay_detail:194` (`g.user.admin or overlay.user_id is None or overlay.user_id == g.user.id`).
 - Shared `_load_overlay_for_user(overlay_id, user)` does the lookup, the auth gate, and `db.expunge(overlay)` so the route can read scalar attributes after the session closes.
 - `ValueError` from either resolver → `Response("invalid path", status=400)`. `target.is_dir()` failure on the listing route → 404. `real.exists()` / `real.is_dir()` failure on the download route → 404 / 400.
 - `send_file(str(real), as_attachment=True, download_name=os.path.basename(real))`.
 - The fragment renders `_overlay_file_tree.html` only — no `base.html` shell — so HTMX swaps inject just the `<ul>` content.
 **Verification:**
 ```
 pytest l4d2web/tests/test_overlay_files_routes.py -q
 ```
 **Commit:** part of Task 4's bundled `feat` commit.
 ---
 ## Task 3: Templates + page-routes integration
 **Files:**
 - Create: `l4d2web/templates/_overlay_file_tree.html` — `<ul class="file-tree" role="group">` + per-entry `_overlay_file_node.html` include + optional truncated-footer `<li>`.
 - Create: `l4d2web/templates/_overlay_file_node.html` — folder row (button + HTMX attrs + empty `<div class="file-tree-children" hidden>`) or file row (`<a>` for regular/symlink files; `<span>` for broken symlinks; `link` / `broken link` badges; `size_human`).
 - Modify: `l4d2web/templates/overlay_detail.html` — add `<section class="panel"><h2>Files</h2>…</section>` between the type-specific sections and the existing "Used by" section. Renders empty-state `<p class="muted">No files yet — build this overlay to populate it.</p>` when `file_tree_root_entries is none`, else includes the partial.
 - Modify: `l4d2web/routes/page_routes.py` — import the helpers, add `_root_file_tree(overlay)` (returns `(entries, truncated_count)` or `(None, 0)` on `ValueError` / missing dir / legacy absolute `overlay.path`), pass `file_tree_root_entries` + `file_tree_truncated` + `file_tree_truncated_count` into `render_template("overlay_detail.html", …)`.
 Test plan (RED first, added to `test_overlay_files_routes.py`):
 - `test_overlay_detail_renders_files_section_with_tree` — page contains "Files" header + entry names.
 - `test_overlay_detail_shows_empty_state_when_overlay_dir_missing` — wipe directory, page shows "No files yet".
 - `test_overlay_detail_files_section_present_for_workshop_overlays` — workshop type also gets the section.
 **Implementation:**
 - Section placement matters: `<section><h2>Files</h2>…</section>` is inserted before the existing "Used by" `<section>`.
 - The partial uses `{% set entries = file_tree_root_entries %}` etc. so the same partial works whether called from the page (with full context) or from the HTMX route (rendering directly with named kwargs).
 - `_root_file_tree` swallows `ValueError` and missing-dir cases into `(None, 0)`, and the template's `{% if file_tree_root_entries is none %}` renders the empty state.
 - Use `overlay.path` (not `str(overlay.id)`) so legacy/seeded rows whose path differs still work correctly when resolvable.
 **Verification:**
 ```
 pytest l4d2web/tests/test_overlay_files_routes.py -q -k overlay_detail
 pytest l4d2web/tests/ -q              # no regressions across the full suite
 ```
 **Commit:** part of Task 4's bundled `feat` commit.
 ---
 ## Task 4: CSS + JS + base.html script wiring
 **Files:**
 - Create: `l4d2web/static/js/file-tree.js` — 12-line event-delegated `click` handler that toggles `aria-expanded` on `.file-tree-toggle` and `hidden` on the next `.file-tree-children` sibling.
 - Modify: `l4d2web/templates/base.html` — `<script src="{{ url_for('static', filename='js/file-tree.js') }}"></script>` next to the existing `csrf.js` / `sse.js` / `modal.js` lines.
 - Modify: `l4d2web/static/css/components.css` — append `~50` lines: `.file-tree`, `.file-tree-row`, `.file-tree-toggle` (transparent button, inherits color), `.file-tree-toggle .chevron` rotation transform on `aria-expanded="true"`, `.file-tree-children[hidden]`, `.file-tree-badge` + `.file-tree-badge-warn`. All against existing tokens (`--space-xs`, `--space-l`, `--color-surface-muted`, `--color-muted`, `--color-danger`, `--radius-s`).
 **Implementation:**
 - `hx-trigger="click once"` makes HTMX fire the GET only on the first expansion; the JS handler fires on every click and toggles aria/hidden state. First click: HTMX populates `.file-tree-children`, JS toggles to expanded. Subsequent clicks: only JS fires, no re-fetch.
 - The CSS chevron is a Unicode `›` inside a `<span class="chevron">`; rotated 90° on expanded via `transform: rotate(90deg)` with a 120ms transition.
 **Verification:**
 ```
 pytest l4d2web/tests/ -q            # 293 passed, 1 skipped
 ```
 Manual smoke (post-deploy on `ckn@10.0.4.128`):
 - Navigate to an overlay detail page with a populated runtime directory.
 - Confirm the "Files" section renders the root level.
 - Click a folder: HTMX request fires once, children appear, chevron rotates.
 - Click again: children hide; no second request in DevTools network tab.
 - Click a file: browser downloads it with the correct filename.
 - Visit another user's overlay as a non-admin: 403.
 **Commit:** `feat(l4d2-web): overlay detail Files section with HTMX file tree + downloads` — covers all four tasks (services helper + routes + templates + CSS/JS), since the feature is small and the tasks share a single set of integration tests.
 ---
 ## End-to-end verification
 After all tasks committed:
 ```
 pytest l4d2web/tests/ -q                 # 293 passed, 1 skipped
 deploy/deploy-test-server.sh ckn@10.0.4.128
 ssh ckn@10.0.4.128 'systemctl status left4me-web --no-pager | head -10'
 curl -s http://10.0.4.128:8000/health    # {"status":"ok"}
 ```
 Then exercise the manual smoke checklist from Task 4 against the deployed instance.
--- a/docs/superpowers/specs/2026-05-08-overlay-file-tree-design.md
+++ b/docs/superpowers/specs/2026-05-08-overlay-file-tree-design.md
@ -0,0 +1,117 @@
 # Overlay File Tree Section Design
 **Goal:** Add a "Files" section to the overlay detail page (`/overlays/<id>`) that renders a collapsible tree of the overlay's runtime directory at `${LEFT4ME_ROOT}/overlays/{overlay.id}/`, with HTMX-driven lazy expansion of folders and click-to-download for individual files. Same access rule as the rest of the overlay detail page (admin or `overlay.user_id == g.user.id`). Read-only; no rename/delete/upload in v1.
 **Approval status:** User-approved 2026-05-08 (visual companion brainstorm + plan-mode review). Implemented + deployed in the same session.
 ## Context
 Today, the overlay detail page shows the row's metadata (name, type, scope, path, last build status), a workshop-items table or script editor depending on `overlay.type`, and links to the build-job stream. It never shows what's actually inside the overlay directory on disk. To verify "did my script actually produce what I expected?" or "did the right VPKs land in `addons/`?" the user has to SSH into the host and `ls /var/lib/left4me/overlays/{id}/`.
 Click-to-download is a secondary nice-to-have: workshop overlays' `addons/*.vpk` are absolute symlinks into the shared `${LEFT4ME_ROOT}/workshop_cache/`, and pulling a single VPK to a dev box otherwise means scping with the right path translation.
 This is also the codebase's **first HTMX-driven feature**: HTMX is vendored in `base.html` but not previously used in any template. The route + partial + tiny event-delegated JS established here become the reusable pattern for future HTMX features.
 ## Locked Decisions
 1. **All overlay types show the section.** Script + workshop + system/managed. Consistency over a tighter scope; even workshop's predictable `addons/*.vpk` layout is worth confirming.
 2. **Collapsible tree, lazy load on expand.** Tree can get large; only the root level is rendered server-side at first paint. Each folder click fires `GET /overlays/<id>/files?path=<rel>` and swaps in that folder's children. The HTMX-disabled / no-JS path still shows the root level (the same partial is server-rendered).
 3. **`hx-trigger="click once"` + JS for re-toggle.** First click does the GET (HTMX disables further fetches via `once`); every subsequent click runs through `static/js/file-tree.js`, an event-delegated handler that toggles `aria-expanded` on the button and `hidden` on the next `.file-tree-children` sibling.
 4. **Single-file download in v1.** No bulk archive (e.g., "download whole overlay as `.tar.gz`"). Files are streamed via Flask `send_file(..., as_attachment=True)`. No size cap — VPKs are commonly 100–500 MB and that's the whole point.
 5. **No auto-refresh.** The tree reflects what was on disk at page render. After a build, the user reloads the page. Polling/SSE would duplicate the existing live-log mechanism on the build-job page for negligible benefit.
 6. **Same access rule as the rest of the page.** `g.user.admin or overlay.user_id is None or overlay.user_id == g.user.id`. GETs need no CSRF (`l4d2web/app.py:56`).
 7. **`overlay.path` not `overlay.id`.** The runtime directory is reached via `overlay.path` (current creation flow guarantees `path == str(id)`, but legacy/seeded rows may differ). Path resolution happens through the existing `l4d2host.paths.overlay_path()` helper, which already validates the ref string and resolves+verifies it stays under `${LEFT4ME_ROOT}/overlays/`.
 8. **Empty / unresolvable → empty state.** If the overlay's path is unresolvable (legacy absolute-path rows) or the directory doesn't exist (overlay never built), the section renders "No files yet — build this overlay to populate it." rather than crashing.
 9. **500-entry cap per folder.** Folders with more than 500 children render the alphabetical-first 500 plus a `+ M more (truncated)` footer. Tunable at runtime via `l4d2web.services.overlay_files.DEFAULT_MAX_ENTRIES` (re-resolved per call so tests can monkeypatch).
 10. **Hidden files shown.** No filtering of `.git`, `.DS_Store`, etc. Users want ground truth.
 11. **One dedicated blueprint, `files_bp`.** Not folded into `overlay_routes.py` (which is exclusively POST mutations) or `page_routes.py` (top-level pages, not embedded fragments). `files_bp` owns both the tree fragment and the download endpoint.
 ## Architecture
 ```text
 GET /overlays/<id>                                 (page_routes.overlay_detail)
   │
   ▼  computes (file_tree_root_entries, truncated_count) via
   │  _root_file_tree(overlay) → safe_resolve_for_listing(overlay.path, "")
   │                          → list_directory(overlay_root, overlay_root)
   │
   ▼  renders overlay_detail.html, which includes _overlay_file_tree.html
       for the root level (or the empty-state <p>).
 GET /overlays/<id>/files?path=<rel>                (files_routes.overlay_files_fragment)
   │
   ▼  auth gate (admin or owner)
   │
   ▼  safe_resolve_for_listing(overlay.path, rel) → Path under overlay_root
   │
   ▼  list_directory(target, overlay_root) → entries[], truncated_count
   │
   ▼  renders _overlay_file_tree.html (partial only — no base.html)
 GET /overlays/<id>/files/download?path=<rel>       (files_routes.overlay_files_download)
   │
   ▼  auth gate
   │
   ▼  safe_resolve_for_download(overlay.path, rel) → real Path under LEFT4ME_ROOT
   │  (follows symlinks; allows targets anywhere in LEFT4ME_ROOT, e.g. workshop_cache)
   │
   ▼  Flask send_file(real, as_attachment=True, download_name=basename(real))
 ```
 ### File tree fragment shape
 `_overlay_file_tree.html` produces a `<ul class="file-tree">` containing one `_overlay_file_node.html` row per entry plus an optional truncated-footer `<li>`. A folder row is a `<button class="file-tree-toggle">` (HTMX attributes for the GET) followed by an empty `<div class="file-tree-children" hidden>` that becomes the swap target. A file row is a `<a href=".../files/download?path=…">` (or a plain `<span>` for broken symlinks) plus optional badges (`link`, `broken link`) and the resolved size.
 Nesting after expand:
 ```html
 <li class="file-tree-row file-tree-row-dir">
  <button class="file-tree-toggle" aria-expanded="true" hx-get="…" …>…</button>
  <div class="file-tree-children">
    <ul class="file-tree" role="group">…</ul>   <!-- inserted by HTMX -->
  </div>
 </li>
 ```
 ## Path Safety
 Two resolvers in `l4d2web/services/overlay_files.py`:
 - `safe_resolve_for_listing(overlay.path, sub_path)` — resolves `overlay_root / sub_path`, applies `Path.resolve(strict=False)`, requires the result to be the overlay root or a descendant. Used by the tree-fragment route. **Refuses to recurse through symlinks that leave the overlay root**, including symlinks pointing into `workshop_cache/` — listing has no need to follow them, since workshop addons are leaf files, not directories we'd descend into.
 - `safe_resolve_for_download(overlay.path, sub_path)` — resolves the candidate path, applies `os.path.realpath()`, requires the result to be under `${LEFT4ME_ROOT}` (anywhere — overlay dir, `workshop_cache/`, future siblings). This is the relaxed gate that lets workshop addons stream from the shared cache while still blocking absolute symlinks to `/etc/passwd` planted by a malicious script overlay.
 Both resolvers re-use `l4d2host.paths.overlay_path()` (which itself calls `validate_overlay_ref`) for the overlay-root resolution, and `l4d2web.services.security.validate_overlay_ref` for the sub-path component (rejects empty / `.` / `..` / absolute / whitespace / backslash). Empty `sub_path` is valid for listing (means "the overlay root") and invalid for download.
 Listing: `target.is_dir()` check after resolution; non-directory → 404.
 Download: `real.exists()` check (404), `real.is_dir()` rejection (400 — "not a file").
 ## Symlink Behaviour
 `list_directory` uses `os.scandir()` with explicit `follow_symlinks` flags:
 - `is_symlink = entry.is_symlink()`
 - `kind`: `entry.is_dir(follow_symlinks=True)` inside a try block. Raised `OSError` → broken symlink, treated as `kind="file"` with `broken=True` and no `<a>` download link.
 - `size`: `entry.stat(follow_symlinks=True).st_size` for files (resolved target's size — what users care about for VPKs); `None` for dirs and broken symlinks.
 Symlinked directories pointing inside the overlay root are rendered as folders and remain expandable; the listing-time safety check rejects expansion if the symlink resolves outside the overlay root.
 Concurrent build vs listing race: a build mid-symlink-rewrite can yield a transient broken-symlink view. Acceptable — page is a snapshot; the visible "broken link" badge tells the user to refresh.
 ## Test Strategy
 Two test modules, both following existing fixture patterns (`tests/test_script_overlay_routes.py` style — `monkeypatch.setenv("LEFT4ME_ROOT", str(tmp_path))`, app fixture with `TESTING=True`).
 - `tests/test_overlay_files.py` — pure-helper unit tests (Flask-free): listing-resolver happy/sad paths (root, sub-path, `..`, absolute, empty component, symlink-out-of-overlay), download-resolver happy/sad paths (regular file, workshop-cache symlink, outside-LEFT4ME_ROOT symlink, traversal, absolute, empty), `list_directory` behaviour (empty, dir-first sort, kind detection, rel paths, symlink markers, broken-symlink markers, truncation cap, human-size formatting).
 - `tests/test_overlay_files_routes.py` — HTTP integration tests: tree-fragment 200 / 400 / 403 / 404 across the same axes; download 200 / 400 / 403 / 404 + content-disposition + byte-exact body for both regular files and workshop-cache symlinks; admin-can-view-foreign overlay; truncation-via-route (monkeypatching `DEFAULT_MAX_ENTRIES`); broken-symlink rendering omits the `<a>` download link; the page-level `overlay_detail` integration shows the section with entries when populated and the empty state when the directory is missing.
 39 tests total. The full web suite (`pytest l4d2web/tests/ -q`) must remain green.
 ## Out of Scope
 - Bulk download (e.g., "download overlay as tar.gz").
 - Inline file preview (text peek, image thumbnail).
 - File deletion / rename / upload from the UI.
 - Auto-refresh while a build is active.
 - Filtering hidden files or applying a `.gitignore`-style rule.
 - Reusable file-tree component for things outside overlays.