diff --git a/docs/superpowers/plans/2026-04-22-l4d2-host-lib-v1.md b/docs/superpowers/plans/2026-04-22-l4d2-host-lib-v1.md index 0f98b37..74666c6 100644 --- a/docs/superpowers/plans/2026-04-22-l4d2-host-lib-v1.md +++ b/docs/superpowers/plans/2026-04-22-l4d2-host-lib-v1.md @@ -35,6 +35,7 @@ - Additional read APIs for web app (no extra CLI commands): - `get_instance_status(name)` - `stream_instance_logs(name, lines=200, follow=True)` +- Blueprints are intentionally out of scope for this library; callers must resolve any blueprint linkage to a concrete YAML spec before calling `initialize`. ## Planned File Layout diff --git a/docs/superpowers/plans/2026-04-23-l4d2-web-app-v1.md b/docs/superpowers/plans/2026-04-23-l4d2-web-app-v1.md index 51d5553..26c424e 100644 --- a/docs/superpowers/plans/2026-04-23-l4d2-web-app-v1.md +++ b/docs/superpowers/plans/2026-04-23-l4d2-web-app-v1.md @@ -2,9 +2,9 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Build a local Flask web app where users manage L4D2 servers with auth, overlay selection, async jobs, desired-vs-actual status, and live logs. +**Goal:** Build a local Flask web app where users create blueprints and manage L4D2 servers derived from those blueprints, with async lifecycle jobs and live logs. -**Architecture:** The app runs as a single Flask process with Jinja templates and vendored local HTMX. It imports `l4d2host` directly (no subprocess boundary), persists state in SQLite, and executes lifecycle actions asynchronously through an in-process scheduler with lock rules: same server serialized, different servers parallel, install global-exclusive. +**Architecture:** Run a single Flask process with Jinja templates, vendored HTMX, custom CSS, and in-process worker threads. Persist app state in SQLite with Rails-style foreign-key naming (`user_id`, `server_id`, `blueprint_id`, `overlay_id`, `job_id`). Integrate directly with `l4d2host` write/read APIs: jobs call `install/initialize/start/stop/delete` with output callbacks, while status/logs use `get_instance_status` and `stream_instance_logs`. **Tech Stack:** Python 3.12+, Flask, SQLAlchemy, Alembic, pytest, vendored HTMX, custom CSS, vanilla JS (SSE). @@ -14,20 +14,30 @@ - In scope: - public signup/login - - admin role with global permissions - - server CRUD with ordered overlays + - one-time CLI admin promotion + - admin-managed overlay catalog + - user-private blueprints with ordered overlays + - server creation from blueprint (live-linked, no server overrides) - async lifecycle actions (`install`, `initialize`, `start`, `stop`, `delete`) - - live job logs and live server logs - - desired-vs-actual state in server list + - desired-vs-actual state with UI status resolution + - live command logs (DB + SSE) and live server logs (journald + SSE) - Out of scope (later phases): - workshop mod management - - user-created overlays and web file manager -- Frontend constraints: - - no CSS/JS framework dependencies - - HTMX vendored locally only (no CDN) - - custom CSS only; links use consistent accent `#0F766E` -- Runtime constraints: - - single-process deployment for v1 + - user-created overlays via web file manager + - shared/global blueprints +- Blueprint rules (locked): + - live-linked to servers + - no per-server overrides for overlays/arguments/config + - blueprint updates apply on next action + - delete blueprint blocked if linked servers exist + - server can change blueprint anytime +- Frontend rules: + - no external frontend dependencies + - HTMX vendored locally only + - custom CSS only + - consistent link color `#0F766E` +- Runtime rules: + - single-process deployment in v1 - periodic status refresh every 8 seconds ## Planned File Layout @@ -43,22 +53,24 @@ - `components/l4d2-web-app/src/l4d2web/services/spec_yaml.py` - `components/l4d2-web-app/src/l4d2web/services/job_worker.py` - `components/l4d2-web-app/src/l4d2web/services/status.py` +- `components/l4d2-web-app/src/l4d2web/services/security.py` - `components/l4d2-web-app/src/l4d2web/routes/auth_routes.py` -- `components/l4d2-web-app/src/l4d2web/routes/server_routes.py` - `components/l4d2-web-app/src/l4d2web/routes/overlay_routes.py` +- `components/l4d2-web-app/src/l4d2web/routes/blueprint_routes.py` +- `components/l4d2-web-app/src/l4d2web/routes/server_routes.py` - `components/l4d2-web-app/src/l4d2web/routes/job_routes.py` - `components/l4d2-web-app/src/l4d2web/routes/log_routes.py` - `components/l4d2-web-app/src/l4d2web/templates/*.html` - `components/l4d2-web-app/src/l4d2web/static/vendor/htmx.min.js` - `components/l4d2-web-app/src/l4d2web/static/css/{tokens,layout,components,logs}.css` -- `components/l4d2-web-app/src/l4d2web/static/js/sse.js` +- `components/l4d2-web-app/src/l4d2web/static/js/{sse,csrf}.js` - `components/l4d2-web-app/alembic.ini` - `components/l4d2-web-app/alembic/env.py` - `components/l4d2-web-app/alembic/versions/0001_initial.py` - `components/l4d2-web-app/tests/*.py` - `components/l4d2-web-app/README.md` -### Task 1: Scaffold Flask app, config, and health endpoint +### Task 1: Scaffold Flask app and base wiring **Files:** - Create: `components/l4d2-web-app/pyproject.toml` @@ -85,7 +97,7 @@ def test_health_endpoint(): Run: `pytest components/l4d2-web-app/tests/test_health.py -q` Expected: FAIL. -- [ ] **Step 3: Implement app factory and route** +- [ ] **Step 3: Implement app factory** ```python from flask import Flask, jsonify @@ -97,6 +109,9 @@ def create_app(test_config=None): SECRET_KEY="dev", DATABASE_URL="sqlite:///l4d2web.db", STATUS_REFRESH_SECONDS=8, + JOB_WORKER_THREADS=4, + JOB_LOG_REPLAY_LIMIT=2000, + JOB_LOG_LINE_MAX_CHARS=4096, ) if test_config: app.config.update(test_config) @@ -134,25 +149,29 @@ git commit -m "feat(l4d2-web): scaffold flask app and health endpoint" ```python from l4d2web.db import init_db, session_scope -from l4d2web.models import User +from l4d2web.models import User, Blueprint -def test_create_user(tmp_path, monkeypatch): +def test_create_user_and_blueprint(tmp_path, monkeypatch): monkeypatch.setenv("DATABASE_URL", f"sqlite:///{tmp_path/'app.db'}") init_db() with session_scope() as s: user = User(username="alice", password_digest="digest", admin=False) s.add(user) s.flush() + bp = Blueprint(user_id=user.id, name="default", arguments="[]", config="[]") + s.add(bp) + s.flush() assert user.id is not None + assert bp.id is not None ``` -- [ ] **Step 2: Run tests and verify failure** +- [ ] **Step 2: Run test and verify failure** Run: `pytest components/l4d2-web-app/tests/test_models.py -q` Expected: FAIL. -- [ ] **Step 3: Implement schema with Rails-style names** +- [ ] **Step 3: Implement schema** ```python class User(Base): @@ -161,16 +180,77 @@ class User(Base): username = mapped_column(String(64), unique=True, nullable=False) password_digest = mapped_column(String(255), nullable=False) admin = mapped_column(Boolean, default=False, nullable=False) + created_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) + updated_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) + + +class Overlay(Base): + __tablename__ = "overlays" + id = mapped_column(Integer, primary_key=True) + name = mapped_column(String(128), unique=True, nullable=False) + path = mapped_column(String(512), nullable=False) + created_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) + updated_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) + + +class Blueprint(Base): + __tablename__ = "blueprints" + id = mapped_column(Integer, primary_key=True) + user_id = mapped_column(ForeignKey("users.id"), nullable=False) + name = mapped_column(String(128), nullable=False) + arguments = mapped_column(Text, nullable=False, default="[]") + config = mapped_column(Text, nullable=False, default="[]") + created_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) + updated_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) + + +class BlueprintOverlay(Base): + __tablename__ = "blueprint_overlays" + id = mapped_column(Integer, primary_key=True) + blueprint_id = mapped_column(ForeignKey("blueprints.id"), nullable=False) + overlay_id = mapped_column(ForeignKey("overlays.id"), nullable=False) + position = mapped_column(Integer, nullable=False) + created_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) + updated_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) class Server(Base): __tablename__ = "servers" id = mapped_column(Integer, primary_key=True) user_id = mapped_column(ForeignKey("users.id"), nullable=False) + blueprint_id = mapped_column(ForeignKey("blueprints.id"), nullable=False) name = mapped_column(String(128), unique=True, nullable=False) port = mapped_column(Integer, nullable=False) - desired_state = mapped_column(String(16), default="stopped", nullable=False) - actual_state = mapped_column(String(16), default="unknown", nullable=False) + desired_state = mapped_column(String(16), nullable=False, default="stopped") + actual_state = mapped_column(String(16), nullable=False, default="unknown") + actual_state_updated_at = mapped_column(DateTime, nullable=True) + last_error = mapped_column(Text, nullable=False, default="") + created_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) + updated_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) + + +class Job(Base): + __tablename__ = "jobs" + id = mapped_column(Integer, primary_key=True) + user_id = mapped_column(ForeignKey("users.id"), nullable=False) + server_id = mapped_column(ForeignKey("servers.id"), nullable=True) + operation = mapped_column(String(32), nullable=False) + state = mapped_column(String(16), nullable=False, default="queued") + exit_code = mapped_column(Integer, nullable=True) + started_at = mapped_column(DateTime, nullable=True) + finished_at = mapped_column(DateTime, nullable=True) + created_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) + updated_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) + + +class JobLog(Base): + __tablename__ = "job_logs" + id = mapped_column(Integer, primary_key=True) + job_id = mapped_column(ForeignKey("jobs.id"), nullable=False) + seq = mapped_column(Integer, nullable=False) + stream = mapped_column(String(8), nullable=False) + line = mapped_column(Text, nullable=False) + created_at = mapped_column(DateTime, default=datetime.utcnow, nullable=False) ``` - [ ] **Step 4: Run tests and verify pass** @@ -182,10 +262,10 @@ Expected: PASS. ```bash git add components/l4d2-web-app/src/l4d2web/{db.py,models.py} components/l4d2-web-app/alembic* -git commit -m "feat(l4d2-web): add sqlite schema and migration baseline" +git commit -m "feat(l4d2-web): add sqlite schema including blueprints and job logs" ``` -### Task 3: Implement auth flows and admin bootstrap CLI +### Task 3: Implement auth and admin bootstrap **Files:** - Create: `components/l4d2-web-app/src/l4d2web/auth.py` @@ -197,7 +277,7 @@ git commit -m "feat(l4d2-web): add sqlite schema and migration baseline" - [ ] **Step 1: Write failing auth tests** ```python -def test_signup_creates_user(client): +def test_public_signup(client): r = client.post("/signup", data={"username": "alice", "password": "secret"}) assert r.status_code == 302 @@ -214,7 +294,7 @@ def test_login_sets_session(client, seed_user): Run: `pytest components/l4d2-web-app/tests/test_auth.py -q` Expected: FAIL. -- [ ] **Step 3: Implement auth and promote-admin command** +- [ ] **Step 3: Implement auth and CLI command** ```python from werkzeug.security import generate_password_hash, check_password_hash @@ -234,7 +314,7 @@ def verify_password(raw: str, digest: str) -> bool: def promote_admin(username: str): with session_scope() as s: user = s.scalar(select(User).where(User.username == username)) - if not user: + if user is None: raise click.ClickException("user not found") user.admin = True s.commit() @@ -245,21 +325,21 @@ def promote_admin(username: str): Run: `pytest components/l4d2-web-app/tests/test_auth.py -q` Expected: PASS. -- [ ] **Step 5: Commit auth** +- [ ] **Step 5: Commit auth features** ```bash git add components/l4d2-web-app/src/l4d2web/{auth.py,cli.py,app.py} components/l4d2-web-app/src/l4d2web/routes/auth_routes.py components/l4d2-web-app/tests/test_auth.py -git commit -m "feat(l4d2-web): add signup/login and promote-admin bootstrap command" +git commit -m "feat(l4d2-web): add public auth and admin bootstrap command" ``` -### Task 4: Implement overlays and server spec CRUD +### Task 4: Implement admin overlay catalog CRUD with path safety **Files:** - Create: `components/l4d2-web-app/src/l4d2web/routes/overlay_routes.py` -- Create: `components/l4d2-web-app/src/l4d2web/routes/server_routes.py` -- Test: `components/l4d2-web-app/tests/test_servers_overlays.py` +- Create: `components/l4d2-web-app/src/l4d2web/services/security.py` +- Test: `components/l4d2-web-app/tests/test_overlays.py` -- [ ] **Step 1: Write failing CRUD tests** +- [ ] **Step 1: Write failing overlay tests** ```python def test_admin_can_create_overlay(admin_client): @@ -267,18 +347,131 @@ def test_admin_can_create_overlay(admin_client): assert r.status_code == 302 -def test_server_create_persists_overlay_order(user_client, seed_overlays): - payload = {"name": "alpha", "port": 27015, "overlay_ids": [2, 1], "arguments": [], "config": []} - r = user_client.post("/servers", json=payload) - assert r.status_code == 201 +def test_overlay_path_must_be_under_root(admin_client): + r = admin_client.post("/admin/overlays", data={"name": "bad", "path": "/tmp/bad"}) + assert r.status_code == 400 ``` - [ ] **Step 2: Run tests and verify failure** -Run: `pytest components/l4d2-web-app/tests/test_servers_overlays.py -q` +Run: `pytest components/l4d2-web-app/tests/test_overlays.py -q` Expected: FAIL. -- [ ] **Step 3: Implement routes with ownership rules** +- [ ] **Step 3: Implement route and validator** + +```python +OVERLAY_ROOT = Path("/opt/l4d2/overlays").resolve() + + +def validate_overlay_path(raw: str) -> Path: + p = Path(raw).resolve() + if OVERLAY_ROOT not in p.parents and p != OVERLAY_ROOT: + raise ValueError("overlay path must be under /opt/l4d2/overlays") + return p +``` + +- [ ] **Step 4: Run tests and verify pass** + +Run: `pytest components/l4d2-web-app/tests/test_overlays.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit overlay CRUD** + +```bash +git add components/l4d2-web-app/src/l4d2web/{routes/overlay_routes.py,services/security.py} components/l4d2-web-app/tests/test_overlays.py +git commit -m "feat(l4d2-web): add admin overlay catalog CRUD with path validation" +``` + +### Task 5: Implement blueprint CRUD and linkage rules + +**Files:** +- Create: `components/l4d2-web-app/src/l4d2web/routes/blueprint_routes.py` +- Test: `components/l4d2-web-app/tests/test_blueprints.py` + +- [ ] **Step 1: Write failing blueprint tests** + +```python +def test_user_can_create_private_blueprint(user_client, seed_overlays): + payload = { + "name": "comp", + "arguments": ["-tickrate 100"], + "config": ["sv_consistency 1"], + "overlay_ids": [1, 2], + } + r = user_client.post("/blueprints", json=payload) + assert r.status_code == 201 + + +def test_delete_blueprint_blocked_when_in_use(user_client, linked_blueprint): + r = user_client.delete(f"/blueprints/{linked_blueprint.id}") + assert r.status_code == 409 +``` + +- [ ] **Step 2: Run tests and verify failure** + +Run: `pytest components/l4d2-web-app/tests/test_blueprints.py -q` +Expected: FAIL. + +- [ ] **Step 3: Implement blueprint routes** + +```python +@bp.post("/blueprints") +@require_login +def create_blueprint(): + payload = request.get_json() + with session_scope() as s: + blueprint = Blueprint( + user_id=current_user().id, + name=payload["name"], + arguments=json.dumps(payload.get("arguments", [])), + config=json.dumps(payload.get("config", [])), + ) + s.add(blueprint) + s.flush() + for position, overlay_id in enumerate(payload.get("overlay_ids", [])): + s.add(BlueprintOverlay(blueprint_id=blueprint.id, overlay_id=overlay_id, position=position)) + s.commit() + return {"id": blueprint.id}, 201 +``` + +- [ ] **Step 4: Run tests and verify pass** + +Run: `pytest components/l4d2-web-app/tests/test_blueprints.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit blueprint feature** + +```bash +git add components/l4d2-web-app/src/l4d2web/routes/blueprint_routes.py components/l4d2-web-app/tests/test_blueprints.py +git commit -m "feat(l4d2-web): add private blueprint CRUD with in-use deletion guard" +``` + +### Task 6: Implement server CRUD from blueprints (no overrides) + +**Files:** +- Create: `components/l4d2-web-app/src/l4d2web/routes/server_routes.py` +- Test: `components/l4d2-web-app/tests/test_servers.py` + +- [ ] **Step 1: Write failing server tests** + +```python +def test_create_server_from_blueprint(user_client, owned_blueprint): + payload = {"name": "alpha", "port": 27015, "blueprint_id": owned_blueprint.id} + r = user_client.post("/servers", json=payload) + assert r.status_code == 201 + + +def test_reassign_blueprint_anytime(user_client, server_and_two_blueprints): + r = user_client.patch(f"/servers/{server_and_two_blueprints.server_id}", json={"blueprint_id": server_and_two_blueprints.other_blueprint_id}) + assert r.status_code == 200 +``` + +- [ ] **Step 2: Run tests and verify failure** + +Run: `pytest components/l4d2-web-app/tests/test_servers.py -q` +Expected: FAIL. + +- [ ] **Step 3: Implement server routes** ```python @bp.post("/servers") @@ -288,32 +481,31 @@ def create_server(): with session_scope() as s: server = Server( user_id=current_user().id, + blueprint_id=int(payload["blueprint_id"]), name=payload["name"], port=int(payload["port"]), - arguments=json.dumps(payload.get("arguments", [])), - config=json.dumps(payload.get("config", [])), + desired_state="stopped", + actual_state="unknown", + last_error="", ) s.add(server) - s.flush() - for position, overlay_id in enumerate(payload.get("overlay_ids", [])): - s.add(ServerOverlay(server_id=server.id, overlay_id=overlay_id, position=position)) s.commit() return {"id": server.id}, 201 ``` - [ ] **Step 4: Run tests and verify pass** -Run: `pytest components/l4d2-web-app/tests/test_servers_overlays.py -q` +Run: `pytest components/l4d2-web-app/tests/test_servers.py -q` Expected: PASS. -- [ ] **Step 5: Commit CRUD routes** +- [ ] **Step 5: Commit server routes** ```bash -git add components/l4d2-web-app/src/l4d2web/routes/{overlay_routes.py,server_routes.py} components/l4d2-web-app/tests/test_servers_overlays.py -git commit -m "feat(l4d2-web): add overlay admin CRUD and server spec CRUD with ordering" +git add components/l4d2-web-app/src/l4d2web/routes/server_routes.py components/l4d2-web-app/tests/test_servers.py +git commit -m "feat(l4d2-web): add server creation and blueprint reassignment routes" ``` -### Task 5: Integrate direct `l4d2host` facade and YAML generation +### Task 7: Add direct `l4d2host` facade and blueprint-to-spec generation **Files:** - Create: `components/l4d2-web-app/src/l4d2web/services/spec_yaml.py` @@ -323,18 +515,18 @@ git commit -m "feat(l4d2-web): add overlay admin CRUD and server spec CRUD with - [ ] **Step 1: Write failing facade tests** ```python -def test_initialize_writes_yaml_and_calls_library(monkeypatch, server_record): +def test_initialize_uses_latest_blueprint_data(monkeypatch, server_with_blueprint): called = {} def fake_initialize(name, spec_path, **kwargs): called["name"] = name - called["spec_path"] = str(spec_path) + called["spec"] = Path(spec_path).read_text() monkeypatch.setattr("l4d2web.services.l4d2_facade.initialize_instance", fake_initialize) from l4d2web.services.l4d2_facade import initialize_server - initialize_server(server_record) - assert called["name"] == server_record.name - assert called["spec_path"].endswith(".yaml") + initialize_server(server_with_blueprint.id) + assert called["name"] == "alpha" + assert "sv_consistency" in called["spec"] ``` - [ ] **Step 2: Run tests and verify failure** @@ -342,20 +534,23 @@ def test_initialize_writes_yaml_and_calls_library(monkeypatch, server_record): Run: `pytest components/l4d2-web-app/tests/test_l4d2_facade.py -q` Expected: FAIL. -- [ ] **Step 3: Implement facade and YAML generator** +- [ ] **Step 3: Implement facade/spec generation** ```python -def write_temp_spec(server, overlays: list[str]) -> Path: - payload = { +def build_server_spec_payload(server, blueprint, overlay_names): + return { "port": server.port, - "overlays": overlays, - "arguments": json.loads(server.arguments), - "config": json.loads(server.config), + "overlays": overlay_names, + "arguments": json.loads(blueprint.arguments), + "config": json.loads(blueprint.config), } - f = tempfile.NamedTemporaryFile("w", suffix=".yaml", delete=False) - yaml.safe_dump(payload, f, sort_keys=False) - f.close() - return Path(f.name) +``` + +```python +def initialize_server(server_id: int, on_stdout=None, on_stderr=None): + server, blueprint, overlay_names = load_server_blueprint_bundle(server_id) + spec_path = write_temp_spec(build_server_spec_payload(server, blueprint, overlay_names)) + initialize_instance(server.name, spec_path, on_stdout=on_stdout, on_stderr=on_stderr) ``` - [ ] **Step 4: Run tests and verify pass** @@ -367,10 +562,10 @@ Expected: PASS. ```bash git add components/l4d2-web-app/src/l4d2web/services/{spec_yaml.py,l4d2_facade.py} components/l4d2-web-app/tests/test_l4d2_facade.py -git commit -m "feat(l4d2-web): add direct l4d2host facade with generated yaml specs" +git commit -m "feat(l4d2-web): resolve live-linked blueprints to runtime specs via l4d2host" ``` -### Task 6: Build scheduler, in-process worker pool, and recovery +### Task 8: Implement scheduler, lock rules, and startup recovery **Files:** - Create: `components/l4d2-web-app/src/l4d2web/services/job_worker.py` @@ -385,14 +580,18 @@ def test_same_server_jobs_serialized(worker_fixture): assert result["same_server_parallel"] is False -def test_install_is_global_exclusive(worker_fixture): +def test_different_servers_can_run_parallel(worker_fixture): + result = worker_fixture.run_once() + assert result["different_servers_parallel"] is True + + +def test_install_global_exclusive(worker_fixture): result = worker_fixture.run_once() assert result["install_parallel"] is False -def test_startup_recovers_stale_running_jobs(app): - from l4d2web.services.job_worker import recover_stale_jobs - recovered = recover_stale_jobs() +def test_recover_stale_running_jobs(worker_fixture): + recovered = worker_fixture.recover_stale_jobs() assert recovered >= 0 ``` @@ -401,52 +600,53 @@ def test_startup_recovers_stale_running_jobs(app): Run: `pytest components/l4d2-web-app/tests/test_job_worker.py -q` Expected: FAIL. -- [ ] **Step 3: Implement scheduler and lock rules** +- [ ] **Step 3: Implement scheduler with in-memory guards** ```python class SchedulerState: def __init__(self): self.install_running = False - self.running_servers: set[int] = set() + self.running_servers = set() def can_start(job, state: SchedulerState) -> bool: if job.operation == "install": - return (not state.install_running) and (not state.running_servers) + return (not state.install_running) and (len(state.running_servers) == 0) if state.install_running: return False return job.server_id not in state.running_servers ``` -- [ ] **Step 4: Add single-process guard and stale job recovery** +- [ ] **Step 4: Implement single-process guard + stale recovery** Run: `pytest components/l4d2-web-app/tests/test_job_worker.py -q` Expected: PASS. -- [ ] **Step 5: Commit worker engine** +- [ ] **Step 5: Commit scheduler** ```bash -git add components/l4d2-web-app/src/l4d2web/services/job_worker.py components/l4d2-web-app/src/l4d2web/app.py components/l4d2-web-app/tests/test_job_worker.py -git commit -m "feat(l4d2-web): implement async scheduler, lock rules, and startup recovery" +git add components/l4d2-web-app/src/l4d2web/{services/job_worker.py,app.py} components/l4d2-web-app/tests/test_job_worker.py +git commit -m "feat(l4d2-web): add async scheduler with lock rules and crash recovery" ``` -### Task 7: Persist command logs and add SSE job stream +### Task 9: Persist command logs and add SSE job stream **Files:** - Create: `components/l4d2-web-app/src/l4d2web/routes/job_routes.py` - Modify: `components/l4d2-web-app/src/l4d2web/services/job_worker.py` - Test: `components/l4d2-web-app/tests/test_job_logs.py` -- [ ] **Step 1: Write failing log tests** +- [ ] **Step 1: Write failing job-log tests** ```python -def test_job_logs_seq_monotonic(db_session, finished_job): - rows = db_session.execute(text("select seq from job_logs where job_id=:id order by seq"), {"id": finished_job.id}).all() - assert rows == sorted(rows) +def test_job_logs_seq_monotonic(db_session, completed_job): + rows = db_session.execute(text("select seq from job_logs where job_id=:id order by seq"), {"id": completed_job.id}).all() + values = [r[0] for r in rows] + assert values == sorted(values) def test_sse_resume_from_last_seq(client, seeded_job_logs): - r = client.get(f"/jobs/{seeded_job_logs.job_id}/stream?last_seq=3") + r = client.get(f"/jobs/{seeded_job_logs.job_id}/stream?last_seq=5") assert r.status_code == 200 ``` @@ -455,35 +655,59 @@ def test_sse_resume_from_last_seq(client, seeded_job_logs): Run: `pytest components/l4d2-web-app/tests/test_job_logs.py -q` Expected: FAIL. -- [ ] **Step 3: Implement log persistence + SSE** +- [ ] **Step 3: Implement persistence and SSE route** ```python -def append_job_log(session, job_id: int, stream: str, line: str) -> None: +def append_job_log(session, job_id: int, stream: str, line: str, max_chars: int = 4096): last_seq = session.scalar(select(func.max(JobLog.seq)).where(JobLog.job_id == job_id)) or 0 - clipped = line[:4096] - session.add(JobLog(job_id=job_id, seq=last_seq + 1, stream=stream, line=clipped)) + session.add(JobLog(job_id=job_id, seq=last_seq + 1, stream=stream, line=line[:max_chars])) session.flush() ``` +```python +@bp.get("/jobs//stream") +@require_login +def stream_job(job_id: int): + last_seq = int(request.args.get("last_seq", "0")) + limit = current_app.config["JOB_LOG_REPLAY_LIMIT"] + + def gen(): + with session_scope() as s: + rows = s.execute( + select(JobLog) + .where(JobLog.job_id == job_id, JobLog.seq > last_seq) + .order_by(JobLog.seq) + .limit(limit) + ).scalars() + for row in rows: + yield f"id: {row.seq}\n" + yield f"event: {row.stream}\n" + yield f"data: {row.line}\n\n" + + return Response(gen(), mimetype="text/event-stream") +``` + - [ ] **Step 4: Run tests and verify pass** Run: `pytest components/l4d2-web-app/tests/test_job_logs.py -q` Expected: PASS. -- [ ] **Step 5: Commit job logs** +- [ ] **Step 5: Commit job log streaming** ```bash git add components/l4d2-web-app/src/l4d2web/{routes/job_routes.py,services/job_worker.py} components/l4d2-web-app/tests/test_job_logs.py -git commit -m "feat(l4d2-web): persist command logs and stream job output over sse" +git commit -m "feat(l4d2-web): persist command logs and stream them with sse" ``` -### Task 8: Add SSE server runtime logs +### Task 10: Add runtime server log stream and status model **Files:** - Create: `components/l4d2-web-app/src/l4d2web/routes/log_routes.py` -- Test: `components/l4d2-web-app/tests/test_server_logs.py` +- Create: `components/l4d2-web-app/src/l4d2web/services/status.py` +- Modify: `components/l4d2-web-app/src/l4d2web/services/job_worker.py` +- Test: `components/l4d2-web-app/tests/test_status_and_server_logs.py` -- [ ] **Step 1: Write failing server-log tests** +- [ ] **Step 1: Write failing tests** ```python def test_owner_can_stream_server_logs(owner_client, owned_server): @@ -491,17 +715,34 @@ def test_owner_can_stream_server_logs(owner_client, owned_server): assert r.status_code == 200 -def test_non_owner_forbidden(user_client, foreign_server): - r = user_client.get(f"/servers/{foreign_server.id}/logs/stream") - assert r.status_code == 403 +def test_status_precedence(): + from l4d2web.services.status import compute_display_state + assert compute_display_state("start", "stopped") == "starting" ``` - [ ] **Step 2: Run tests and verify failure** -Run: `pytest components/l4d2-web-app/tests/test_server_logs.py -q` +Run: `pytest components/l4d2-web-app/tests/test_status_and_server_logs.py -q` Expected: FAIL. -- [ ] **Step 3: Implement runtime log stream route** +- [ ] **Step 3: Implement log stream and status computation** + +```python +def compute_display_state(active_operation, actual_state): + if active_operation == "delete": + return "deleting" + if active_operation == "start": + return "starting" + if active_operation == "stop": + return "stopping" + if active_operation == "initialize": + return "initializing" + return actual_state + + +def has_drift(desired_state, actual_state, has_active_job): + return (not has_active_job) and desired_state != actual_state +``` ```python @bp.get("/servers//logs/stream") @@ -516,77 +757,25 @@ def stream_server_logs(server_id: int): return Response(gen(), mimetype="text/event-stream") ``` -- [ ] **Step 4: Run tests and verify pass** +- [ ] **Step 4: Add actual-state refresh updates** -Run: `pytest components/l4d2-web-app/tests/test_server_logs.py -q` +Run: `pytest components/l4d2-web-app/tests/test_status_and_server_logs.py -q` Expected: PASS. -- [ ] **Step 5: Commit server log streaming** +- [ ] **Step 5: Commit status/log features** ```bash -git add components/l4d2-web-app/src/l4d2web/routes/log_routes.py components/l4d2-web-app/tests/test_server_logs.py -git commit -m "feat(l4d2-web): add live server log streaming with ownership checks" +git add components/l4d2-web-app/src/l4d2web/{routes/log_routes.py,services/status.py,services/job_worker.py} components/l4d2-web-app/tests/test_status_and_server_logs.py +git commit -m "feat(l4d2-web): add live server logs and desired-vs-actual status model" ``` -### Task 9: Implement desired-vs-actual state model and refresh - -**Files:** -- Create: `components/l4d2-web-app/src/l4d2web/services/status.py` -- Modify: `components/l4d2-web-app/src/l4d2web/services/job_worker.py` -- Modify: `components/l4d2-web-app/src/l4d2web/routes/server_routes.py` -- Test: `components/l4d2-web-app/tests/test_status.py` - -- [ ] **Step 1: Write failing status tests** - -```python -def test_display_state_priority(status_service): - assert status_service.display_state(active_job_operation="start", actual_state="stopped") == "starting" - - -def test_drift_badge(status_service): - assert status_service.is_drift(desired_state="running", actual_state="stopped", has_active_job=False) is True -``` - -- [ ] **Step 2: Run tests and verify failure** - -Run: `pytest components/l4d2-web-app/tests/test_status.py -q` -Expected: FAIL. - -- [ ] **Step 3: Implement state logic and refresh hooks** - -```python -def compute_display_state(active_operation, actual_state): - priority = { - "delete": "deleting", - "start": "starting", - "stop": "stopping", - "initialize": "initializing", - } - if active_operation in priority: - return priority[active_operation] - return actual_state -``` - -- [ ] **Step 4: Implement 8-second actual-state refresh loop** - -Run: `pytest components/l4d2-web-app/tests/test_status.py -q` -Expected: PASS. - -- [ ] **Step 5: Commit state model** - -```bash -git add components/l4d2-web-app/src/l4d2web/{services/status.py,services/job_worker.py,routes/server_routes.py} components/l4d2-web-app/tests/test_status.py -git commit -m "feat(l4d2-web): add desired vs actual status model with 8s refresh" -``` - -### Task 10: Apply security and reliability hardening +### Task 11: Apply security and reliability hardening **Files:** - Modify: `components/l4d2-web-app/src/l4d2web/app.py` -- Modify: `components/l4d2-web-app/src/l4d2web/routes/auth_routes.py` -- Modify: `components/l4d2-web-app/src/l4d2web/routes/overlay_routes.py` - Modify: `components/l4d2-web-app/src/l4d2web/db.py` -- Test: `components/l4d2-web-app/tests/test_security.py` +- Modify: `components/l4d2-web-app/src/l4d2web/routes/auth_routes.py` +- Create: `components/l4d2-web-app/tests/test_security.py` - [ ] **Step 1: Write failing hardening tests** @@ -596,9 +785,11 @@ def test_csrf_required(client, seed_user): assert r.status_code == 400 -def test_overlay_path_must_be_under_opt_l4d2_overlays(admin_client): - r = admin_client.post("/admin/overlays", data={"name": "bad", "path": "/tmp/bad"}) - assert r.status_code == 400 +def test_login_rate_limit(client): + for _ in range(20): + client.post("/login", data={"username": "x", "password": "y"}) + r = client.post("/login", data={"username": "x", "password": "y"}) + assert r.status_code == 429 ``` - [ ] **Step 2: Run tests and verify failure** @@ -606,7 +797,7 @@ def test_overlay_path_must_be_under_opt_l4d2_overlays(admin_client): Run: `pytest components/l4d2-web-app/tests/test_security.py -q` Expected: FAIL. -- [ ] **Step 3: Implement hardening** +- [ ] **Step 3: Implement hardening measures** ```python app.config.update( @@ -630,15 +821,16 @@ Expected: PASS. - [ ] **Step 5: Commit hardening** ```bash -git add components/l4d2-web-app/src/l4d2web/{app.py,db.py} components/l4d2-web-app/src/l4d2web/routes/{auth_routes.py,overlay_routes.py} components/l4d2-web-app/tests/test_security.py -git commit -m "feat(l4d2-web): add csrf, rate limits, sqlite wal, and path safety checks" +git add components/l4d2-web-app/src/l4d2web/{app.py,db.py} components/l4d2-web-app/src/l4d2web/routes/auth_routes.py components/l4d2-web-app/tests/test_security.py +git commit -m "feat(l4d2-web): add csrf, rate limiting, and sqlite reliability settings" ``` -### Task 11: Build minimal UI with vendored HTMX and custom CSS +### Task 12: Build UI and finalize docs **Files:** - Create: `components/l4d2-web-app/src/l4d2web/templates/base.html` - Create: `components/l4d2-web-app/src/l4d2web/templates/dashboard.html` +- Create: `components/l4d2-web-app/src/l4d2web/templates/blueprints.html` - Create: `components/l4d2-web-app/src/l4d2web/templates/server_detail.html` - Create: `components/l4d2-web-app/src/l4d2web/templates/admin_overlays.html` - Create: `components/l4d2-web-app/src/l4d2web/static/vendor/htmx.min.js` @@ -647,19 +839,22 @@ git commit -m "feat(l4d2-web): add csrf, rate limits, sqlite wal, and path safet - Create: `components/l4d2-web-app/src/l4d2web/static/css/components.css` - Create: `components/l4d2-web-app/src/l4d2web/static/css/logs.css` - Create: `components/l4d2-web-app/src/l4d2web/static/js/sse.js` +- Create: `components/l4d2-web-app/src/l4d2web/static/js/csrf.js` +- Create: `components/l4d2-web-app/README.md` - Test: `components/l4d2-web-app/tests/test_pages.py` - [ ] **Step 1: Write failing page tests** ```python -def test_dashboard_renders_server_name(auth_client_with_server): +def test_dashboard_renders_server_and_status(auth_client_with_server): r = auth_client_with_server.get("/dashboard") + text = r.get_data(as_text=True) assert r.status_code == 200 - assert "alpha" in r.get_data(as_text=True) + assert "alpha" in text -def test_non_admin_cannot_open_overlay_admin(auth_client): - r = auth_client.get("/admin/overlays") +def test_blueprint_page_private_visibility(user_client, other_users_blueprint): + r = user_client.get(f"/blueprints/{other_users_blueprint.id}") assert r.status_code == 403 ``` @@ -668,82 +863,41 @@ def test_non_admin_cannot_open_overlay_admin(auth_client): Run: `pytest components/l4d2-web-app/tests/test_pages.py -q` Expected: FAIL. -- [ ] **Step 3: Implement templates and style system** +- [ ] **Step 3: Implement templates and style tokens** ```css -/* tokens.css */ :root { --color-link: #0F766E; - --color-text: #132028; - --color-bg: #f7fbfa; + --color-bg: #F6FBFA; + --color-text: #12302B; + --color-card: #FFFFFF; --space-2: 0.5rem; --space-4: 1rem; - --radius: 10px; + --radius: 8px; } -a { color: var(--color-link); } +a { + color: var(--color-link); +} ``` -- [ ] **Step 4: Run tests and verify pass** - -Run: `pytest components/l4d2-web-app/tests/test_pages.py -q` -Expected: PASS. - -- [ ] **Step 5: Commit UI** - -```bash -git add components/l4d2-web-app/src/l4d2web/{templates,static} components/l4d2-web-app/tests/test_pages.py -git commit -m "feat(l4d2-web): add minimal htmx ui with vendored assets and custom css" -``` - -### Task 12: Final integration, docs, and full verification - -**Files:** -- Create: `components/l4d2-web-app/README.md` -- Modify: `components/l4d2-web-app/src/l4d2web/app.py` -- Test: all tests under `components/l4d2-web-app/tests` - -- [ ] **Step 1: Write failing integration smoke test** - -```python -def test_core_routes_registered(app): - routes = {r.rule for r in app.url_map.iter_rules()} - assert "/dashboard" in routes - assert "/jobs//stream" in routes -``` - -- [ ] **Step 2: Run test and verify failure (if wiring incomplete)** - -Run: `pytest components/l4d2-web-app/tests -q` -Expected: FAIL or partial PASS before final wiring. - -- [ ] **Step 3: Finalize app wiring and README** - -README sections required: -- deployment model (single process) -- auth/admin bootstrap (`l4d2web promote-admin `) -- scheduler lock semantics -- desired vs actual model and 8s refresh -- log architecture (`job_logs` forever, server logs from journald) -- frontend dependency policy (vendored HTMX only, custom CSS) - -- [ ] **Step 4: Run full test suite** +- [ ] **Step 4: Run tests and full suite** Run: `pytest components/l4d2-web-app/tests -q` Expected: PASS. -- [ ] **Step 5: Commit finalization** +- [ ] **Step 5: Commit UI and docs** ```bash git add components/l4d2-web-app -git commit -m "docs(l4d2-web): finalize v1 architecture contracts and verification" +git commit -m "docs(l4d2-web): finalize blueprint-driven ui and deployment contracts" ``` --- ## Self-Review -- [ ] Spec coverage complete for approved constraints (auth, admin bootstrap, overlays, async jobs, status model, logging, minimal UI). +- [ ] Spec coverage complete for locked blueprint semantics, auth model, scheduler rules, and logging behavior. - [ ] No placeholders (`TODO`, `TBD`, vague directives). -- [ ] Naming consistency (`user_id`, `server_id`, `overlay_id`, `job_id`, `password_digest`, `admin`). -- [ ] Every task includes concrete test commands and expected outcomes. +- [ ] Type/name consistency (`user_id`, `server_id`, `blueprint_id`, `overlay_id`, `job_id`, `password_digest`, `admin`). +- [ ] Every task includes concrete tests and exact commands.