diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..f1848b0 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,106 @@ +# ckn-bw — agent & contributor guide + +## What this repo is + +A [BundleWrap](https://bundlewrap.org/) configuration-management repo +for ~22 personal/family-infra nodes. Nodes, groups, and bundles are +defined in plain Python; `bw apply` deploys the resulting state to +real machines. + +Note: the root `README.md` is the maintainer's personal scratchpad, +not project documentation. Onboarding lives **here**, in `AGENTS.md`. + +## Quickstart for agents + +Five rules; follow these and you won't break things: + +1. **Read-only by default.** Never run `bw apply`, `bw run`, or + `bw lock` without explicit user request — even with `-i`. Stick + to `bw test`, `bw nodes`, `bw groups`, `bw bundles`, + `bw items`, `bw metadata`, `bw hash`, `bw debug`. See + [`docs/agents/commands.md`](docs/agents/commands.md) and the + fork's [safety envelope](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md). +2. **Never echo decrypted secrets.** Don't print, paste, or log the + value behind a `!password_for:`, `!decrypt:`, or + `!32_random_bytes_as_base64_for:` magic string — not even from + `bw debug` exploration. See + [`conventions.md#secrets`](docs/agents/conventions.md#secrets). +3. **Don't touch the do-not-modify list.** `.secrets.cfg*`, `.venv`, + `.cache`, `.bw_debug_history`, `.envrc`, root `README.md`. Treat + `hooks/` and `items/` (custom item types) with extra care: a + broken hook or item type breaks every `bw` command repo-wide. +4. **Use the fork.** The venv runs editable from + [`github.com/CroneKorkN/bundlewrap`](https://github.com/CroneKorkN/bundlewrap) + (branch `main`). Behavior tracks upstream `main`; the fork's + [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) + is the canonical bundlewrap-language reference. See + [`conventions.md#bundlewrap-version`](docs/agents/conventions.md#bundlewrap-version). +5. **Prefer adding helpers to `libs/`** over duplicating logic across + bundles. Repo-wide helpers go in + [`libs/`](libs/AGENTS.md), reachable as `repo.libs.`. + +## Layout + +| Dir | What's there | +|---|---| +| [`bundles/`](bundles/AGENTS.md) | 103 bundles. One subdir per bundle (`items.py`, `metadata.py`, `files/`). | +| [`nodes/`](nodes/AGENTS.md) | One file per node (~22). `eval()`-loaded; demagified through `repo.vault`. | +| [`groups/`](groups/AGENTS.md) | Group definitions, organized by axis (`applications/`, `locations/`, `machine/`, `os/`). | +| [`libs/`](libs/AGENTS.md) | Shared Python helpers reachable as `repo.libs.`. | +| [`hooks/`](hooks/AGENTS.md) | bw lifecycle hooks (`apply_start`, `test`, `node_apply_start`, …). | +| [`data/`](data/AGENTS.md) | Out-of-bundle data assets (apt keys, grafana dashboards, …). | +| [`items/`](items/AGENTS.md) | Custom item types (currently `download:`). | +| [`bin/`](bin/AGENTS.md) | Operator scripts; not invoked by bundlewrap. | +| [`docs/agents/`](docs/agents/conventions.md) | Repo conventions and command deltas. | + +## How nodes, groups, and bundles fit together + +- A **node** (`nodes/..py`) declares the groups it + belongs to and any node-local bundles + metadata overrides. +- A **group** (`groups//.py`) attaches bundles and shared + metadata to its members. Groups inherit via `supergroups`. +- A **bundle** (`bundles//`) is one chunk of configuration: + `items.py` produces the items (files, services, packages), + `metadata.py` declares `defaults` and `@metadata_reactor` functions + that derive metadata from other metadata. +- The repo-root loaders (`nodes.py`, `groups.py`) walk these dirs and + `eval()` each file. `nodes.py` additionally **demagifies** the + result, resolving `!password_for:` etc. through `repo.vault`. See + [`conventions.md#eval-loaded-node-and-group-files`](docs/agents/conventions.md#eval-loaded-node-and-group-files) + for the constraints this places on editors. +- Metadata merges along: `all → location → os → machine → + applications → node`. + +## Conventions you must know + +| Topic | Where | +|---|---| +| Bundlewrap-language reference (item types, dep keywords, reactors) | Fork's [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) — read first if new to bundlewrap | +| Vault / demagify magic strings | [`conventions.md#secrets`](docs/agents/conventions.md#secrets) | +| Bundlewrap install (editable from the fork) | [`conventions.md#bundlewrap-version`](docs/agents/conventions.md#bundlewrap-version) | +| Group inheritance order, naming patterns | [`conventions.md#group-inheritance-order`](docs/agents/conventions.md#group-inheritance-order), [`#naming-conventions`](docs/agents/conventions.md#naming-conventions) | +| Repo-specific bw command deltas (apt keys, suspended nodes, vault echo) | [`commands.md`](docs/agents/commands.md) | +| Lib helpers | top-of-file docstrings in `libs/*.py` (`head -1 libs/*.py`) | +| Suspension idioms (`*.py_`, `_old/`, "for now") | [`conventions.md#suspension-and-soft-delete-idioms`](docs/agents/conventions.md#suspension-and-soft-delete-idioms) | + +## Where to look for examples + +When writing a new bundle, copy patterns from one that already does +the thing you need: + +| Pattern | Look at | +|---|---| +| Vault calls inside metadata reactors | `bundles/dm-crypt/metadata.py` (compact, focused) | +| Mako-templated files | `bundles/bind/items.py` (DNS zonefile rendering) | +| Cross-bundle reactor writing | `bundles/nextcloud/metadata.py` (writes into `apt.packages`, `archive.paths`) | +| Custom `download:` items | `bundles/minecraft/items.py` | +| Node file (single-purpose) | `nodes/home.server.py` | +| Group with `supergroups` chain | `groups/os/debian-13.py` | + +## Where this doc lives + +- This file: `AGENTS.md` at the repo root. +- `CLAUDE.md` is a symlink to this file — both names point to the same + content so different tools can find it. +- The personal TODO scratchpad (`README.md`) is **separate** and not + project documentation. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 120000 index 0000000..47dc3e3 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/bin/AGENTS.md b/bin/AGENTS.md new file mode 100644 index 0000000..ed596eb --- /dev/null +++ b/bin/AGENTS.md @@ -0,0 +1,62 @@ +# bin/ + +## What's here + +Operator scripts — invoked manually by the maintainer, **not** by +bundlewrap itself. Each is a standalone Python (or shell) script that +opens the repo via `Repository(dirname(dirname(realpath(__file__))))`. + +Discovery is by `ls bin/` plus the `# purpose:` header line at the top +of each script: + +```sh +head -2 bin/* +``` + +## Conventions + +- **`# purpose:` header.** Every script under `bin/` starts with + `#!/usr/bin/env python3` (or appropriate shebang), then a + `# purpose: ` comment. Baseline enforced by + `grep -L '^# purpose' bin/*`. +- **Self-contained.** A script must work when run from anywhere — it + resolves the repo via the script's own path, not `cwd`. +- **Read-only by default.** Most operator scripts query/print state + (`passwords-for`, `wireguard-client-config`). Mutating scripts + (`upgrade_and_restart_all`, `mikrotik-firmware-updater`, + `sync_1password`) are the exception, not the rule, and prompt for + confirmation. + +## How to add a script + +1. Start from [`bin/script_template`](script_template) — it carries + the canonical shebang + `# purpose:` header + `Repository(...)` + bootstrap. +2. Add the `# purpose:` line; lowercase, terse, include a `usage:` + example if the script takes arguments. +3. `chmod +x bin/`. +4. The script can reach helpers via `bw.libs.` exactly like a + bundle does. + +## Pitfalls + +- **`bin/` is not on `$PATH` by default.** Invoke as `bin/` from + the repo root, or via `direnv` if `.envrc` exposes it. +- **Mutating scripts can hit Tier-3 territory** (per the fork's + safety envelope). Don't run `upgrade_and_restart_all`, + `mikrotik-firmware-updater`, or anything that does `node.run(...)` + without explicit user instruction. See the fork's + [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) + for the three-tier model. +- **Vault echo.** Scripts like `passwords-for` print decrypted values + by design; that's allowed for the human at the terminal but *not* + for the agent — never paste output into chat, ticket, or PR + description. + +## See also + +- [`script_template`](script_template) — canonical starter. +- [`docs/agents/conventions.md`](../docs/agents/conventions.md) — + vault rules. +- [`docs/agents/commands.md`](../docs/agents/commands.md) — read-only + bw-command guidance. diff --git a/bundles/AGENTS.md b/bundles/AGENTS.md new file mode 100644 index 0000000..c4d0b73 --- /dev/null +++ b/bundles/AGENTS.md @@ -0,0 +1,124 @@ +# bundles/ + +## Before you start + +Read [`docs/agents/conventions.md`](../docs/agents/conventions.md) first +— it covers vault calls, demagify, the `repo.libs.` helpers, and the +files agents must not modify. Skipping it leads to subtly broken bundles +(vault calls in the wrong place, dict-in-set `TypeError` because of +unhashable nesting, etc.). + +For bundlewrap-language reference (item types, dep keywords, +`metadata_reactor`, `defaults`, item-file template syntax) see the fork's +[`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) +and its [`docs/content/guide/item_file_templates.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/docs/content/guide/item_file_templates.md). + +## What's here + +103 bundles. Each is a directory `bundles//` containing some of: + +``` +bundles// +├── items.py # the items this bundle creates (files, services, packages, …) +├── metadata.py # `defaults` + `@metadata_reactor` functions +├── files/ # static or templated file payloads referenced from items.py +├── AGENTS.md # this bundle's doc (template at AGENTS.template.md) +└── README.md # legacy; being phased out (see "Documentation transition" below) +``` + +## Conventions + +- **Bundle names** are lowercase, hyphen-separated: `backup-server`, + `bind-acme`, `dm-crypt`. No underscores in new bundle names — see + [`conventions.md#naming-conventions`](../docs/agents/conventions.md#naming-conventions). +- **`items.py`** is plain Python; it produces `files = {...}`, + `pkg_apt = {...}`, `svc_systemd = {...}`, etc. dicts at module scope. + Cross-item dependencies use `needs` / `triggers` / `triggered_by` — + see the fork's `AGENTS.md` for the full keyword cheat sheet. +- **`metadata.py`** uses `defaults = {...}` for static seed values and + `@metadata_reactor.provides(...)` for derived values. Reactors are + pure functions of `(metadata,)` — no side effects, no I/O. +- **Helpers go in [`libs/`](../libs/AGENTS.md)** when they're useful to + more than one bundle. Don't duplicate logic across bundles. +- **Custom item types** (e.g. `download:`) live in + [`items/`](../items/AGENTS.md), not per-bundle. + +## How to add a new bundle + +1. `mkdir bundles//` (lowercase, hyphenated). +2. Write `items.py` and (if anything is configurable) `metadata.py`. + Use `repo.libs.hashable.hashable(...)` when you need to nest a dict + or set inside a metadata set; raw dicts/sets aren't hashable. +3. Drop static payloads into `bundles//files/`. For Mako-templated + files, declare `'content_type': 'mako'` on the `file:` item — see + the fork's + [item-file-templates guide](https://github.com/CroneKorkN/bundlewrap/blob/main/docs/content/guide/item_file_templates.md). +4. **Wire to nodes.** Either add an entry to the relevant + [`groups//.py`](../groups/AGENTS.md) (preferred for shared + bundles) or to the node's `bundles` list directly + ([`nodes/AGENTS.md`](../nodes/AGENTS.md)). +5. Verify, in this order: + - `bw test` — sanity (loaders + reactors). + - `bw items ` — confirm new items appear on a node that opts in. + - `bw hash ` — confirm the change is what you expected. See + [`docs/agents/commands.md`](../docs/agents/commands.md) and the + fork's hash-diff workflow. +6. Create `bundles//AGENTS.md` from + [`AGENTS.template.md`](AGENTS.template.md). For a brand-new bundle + without consumers yet, leave `Depends on` and `Produces` empty or + marked TBD; fill them in after the first verify run. + +## How to remove a bundle + +1. `git grep ''` in `nodes/`, `groups/`, and other `bundles/` to + find references. +2. Remove those references. +3. `rm -rf bundles//`. +4. `bw test` and `bw nodes` to confirm clean. + +## Pitfalls + +- **`metadata.py` is evaluated at load time** for *every* node, every + invocation of `bw`. Heavy work or I/O slows the whole repo. Keep + reactors pure and fast; pre-compute in `libs/` if you must. +- **Static files vs templates.** `bundles//files/` is static + unless the matching `file:` item declares `content_type='mako'` + (or a templating extension triggers it). To check, read the matching + `file:` entry in `items.py`. +- **Reactors writing across namespaces.** Some bundles' reactors write + into other bundles' metadata namespaces (e.g. `nextcloud` writes + into `apt.packages`, `archive.paths`). When you change such a bundle, + every consumer's metadata changes too. Per-bundle docs declare these + in an optional `## Writes into` section — read it before assuming the + blast radius is local. +- **`bw hash` doesn't accept selectors.** Use `bw hash ` per + literal name; see the fork's runbook. + +## Documentation transition + +This repo is migrating from bundle `README.md` files to per-bundle +`AGENTS.md` files (one balanced doc per bundle, agents + humans). + +- Where both exist, **`AGENTS.md` is canonical**; the `README.md` is + being phased out. +- ~28 bundle READMEs survive after the seed migration (the seed PR + folds in 5–10 of them; the rest are addressed lazily on the next + material edit — Phase 3 leave-as-you-go). +- Phase-3 rule: any time you (or any agent) materially edits a bundle, + top-up its `AGENTS.md` or create one from + [`AGENTS.template.md`](AGENTS.template.md). If a stale `README.md` + is still around in the bundle, fold it in and remove it in the same + commit. + +## See also + +- [`AGENTS.template.md`](AGENTS.template.md) — per-bundle doc template. +- [`docs/agents/conventions.md`](../docs/agents/conventions.md) — repo + idioms (vault, demagify, naming, do-not-touch list). +- [`docs/agents/commands.md`](../docs/agents/commands.md) — repo-specific + command deltas. +- [`items/AGENTS.md`](../items/AGENTS.md) — custom item types + (`download:`); when to write a new one vs use `file:`. +- [`libs/AGENTS.md`](../libs/AGENTS.md) — shared helpers. +- Fork's [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) + — bundlewrap-language reference + safety envelope. diff --git a/bundles/AGENTS.template.md b/bundles/AGENTS.template.md new file mode 100644 index 0000000..79cc611 --- /dev/null +++ b/bundles/AGENTS.template.md @@ -0,0 +1,61 @@ +# + + + +<1–3 sentences: what this bundle does and when you'd use it.> + +## Usage + + + +## Metadata + +Keys read from `node.metadata`: + +```python +{ + '': { + 'key': 'value', # str, required — short note + 'flag': True, # bool, default True + 'list': [], # list[str], default [] — short note + 'nested': { + 'subkey': 0, # int, default 0 + }, + }, +} +``` + +## Produces + + + +## Depends on + + + +## Writes into + + + +## Gotchas + + diff --git a/data/AGENTS.md b/data/AGENTS.md new file mode 100644 index 0000000..78f75bf --- /dev/null +++ b/data/AGENTS.md @@ -0,0 +1,63 @@ +# data/ + +## What's here + +Out-of-bundle data assets consumed by one or more bundles. Each subdir +maps to a consumer: + +``` +data/ +├── apt/keys/ # binary GPG keys for apt sources +├── grafana/rows/ # Mako-templated dashboard panels (Python) +├── nginx/ # nginx snippets shared across vhosts +├── homeassistant/, mailman/, nextcloud/, ... +└── network.py # repo-wide network metadata (one-off file) +``` + +## Two distinct content models + +Same directory shape, different content kinds. When you add a new +`data//` subdir, declare which model it follows: + +| Model | Example | Consumer | +|---|---|---| +| **Binary / static** | `data/apt/keys/*.{asc,gpg}` | `bundles/apt` reads files at apply time | +| **Python module / template** | `data/grafana/rows/*.py`, `data/routeros-monitoring/*.py` | bundle `import`s and renders | + +If a data asset is read by **exactly one bundle**, prefer +`bundles//files/` instead of `data//`. `data/` is for +shared / multi-consumer artifacts. Single-instance evidence: commit +`78a8abc` moved `mikrotik.mib` *out* of `data/` *into* the bundle for +this reason. + +## Conventions + +- **One subdir per consumer.** Subdir name = consumer bundle name + (`data/apt/`, `data/nextcloud/`). +- **`network.py` exception.** A single file at `data/network.py` holds + repo-wide network metadata; it doesn't belong to one bundle. Treat + it as cross-cutting infrastructure metadata. + +## How to add data + +1. Decide the content model (binary or Python). +2. `mkdir data//`. +3. Drop assets in. +4. The consumer bundle (`bundles//items.py` or + `metadata.py`) reads them via `repo.path` + `os.path.join` or + similar. + +## Pitfalls + +- **Apt keys** trigger an offline-verify rule before they're committed. + See [`commands.md#apt-key-changes-need-offline-verification`](../docs/agents/commands.md#apt-key-changes-need-offline-verification). +- **Mako-templated Python data** evaluates at bundle render time. Side + effects in those modules slow the whole repo (same caveat as + [`libs/`](../libs/AGENTS.md)). + +## See also + +- [`bundles/AGENTS.md`](../bundles/AGENTS.md) — when bundle `files/` + beats `data/`. +- [`docs/agents/commands.md`](../docs/agents/commands.md) — apt-key + verification rule. diff --git a/docs/agents/commands.md b/docs/agents/commands.md new file mode 100644 index 0000000..2e78658 --- /dev/null +++ b/docs/agents/commands.md @@ -0,0 +1,50 @@ +# Commands + +The canonical bw-command runbook — read-only allowlist, three-tier +safety envelope, after-change table, hash-diff workflow, `bw debug` +sketch — lives in the fork's +[`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md). +Read that first. + +This file collects only the deltas specific to `ckn-bw`. + +## Apt-key changes need offline verification + +Editing files under `data/apt/keys/*.{asc,gpg}` rotates a signing key +the whole apt subsystem trusts. Trial-and-error with `bw apply` is the +*failure* path: a wrong key blocks unattended upgrades cluster-wide +until corrected manually. + +Before touching `data/apt/keys/`: + +1. Fetch the new key from its upstream source (project release page, + `keys.openpgp.org`, etc.). +2. `gpg --show-keys ` — print the fingerprint. +3. Diff against the fingerprint published by the upstream source. +4. Only after the fingerprint matches, place the file under + `data/apt/keys/` and let `bundles/apt` consume it on the next + apply. + +## `*.py_` suspended nodes are invisible to `bw nodes` + +The repo loader (`nodes.py`) only matches files ending in `.py`. Files +ending in `.py_` are silently skipped. If `bw nodes` reports a node +missing, check whether its file has been parked: + +```sh +ls nodes/ | grep '\.py_$' +``` + +This is the [suspension idiom](conventions.md#suspension-and-soft-delete-idioms), +not a bug. + +## Vault output never leaves the terminal + +The fork's runbook calls out that `bw debug` resolves vault magic +strings transparently. In `ckn-bw` specifically: never echo, log, or +paste decrypted values, even from a `bw debug -c` one-liner. If you +need to verify a secret resolved correctly, hash or fingerprint it +instead. + +See [`conventions.md#secrets`](conventions.md#secrets) for the +demagify magic-string list and the rule's full rationale. diff --git a/docs/agents/conventions.md b/docs/agents/conventions.md new file mode 100644 index 0000000..9e37479 --- /dev/null +++ b/docs/agents/conventions.md @@ -0,0 +1,146 @@ +# Conventions + +Repo-specific idioms an agent has to know before editing this BundleWrap +config. For bundlewrap-the-language reference (item types, dep keywords, +metadata-reactor semantics, after-change runbook), see the fork's +[`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md). + +## Secrets + +Secrets live in `.secrets.cfg` and are referenced from node files by +"demagify" magic strings. The loader (`nodes.py`) walks node dicts and +resolves any leaf string of the form `!:`. + +| Magic string | Resolves to | +|---|---| +| `!password_for:` | `repo.vault.password_for(id)` | +| `!decrypt:` | `repo.vault.decrypt(ciphertext)` | +| `!decrypt_file:` | `repo.vault.decrypt_file(path)` | +| `!32_random_bytes_as_base64_for:` | `repo.vault.random_bytes_as_base64_for(id, length=32)` | + +Magic strings are only resolved inside **node files** (everything under +`nodes/`). They are *not* resolved in group files, bundle metadata +defaults, or item attributes — call `repo.vault.(...)` directly +there. + +**Never echo decrypted values.** Don't print, log, or paste them, even +in `bw debug` exploration. If you need a sanity check, hash or +fingerprint instead. This applies to AI agents and humans equally. + +## Bundlewrap version + +The venv runs **editable** from the maintainer's fork: + +``` +-e git+https://github.com/CroneKorkN/bundlewrap.git@main#egg=bundlewrap +``` + +Pinned in `requirements.txt`. The fork's `main` tracks upstream +`bundlewrap/bundlewrap` master; the fork's [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) +is the canonical agent-oriented bundlewrap-language reference. + +To pull upstream changes into the venv: + +```sh +(cd .venv/src/bundlewrap && git pull) +``` + +## Eval-loaded node and group files + +`nodes/*.py` and `groups/*.py` are loaded by `eval()` in `nodes.py` / +`groups.py` (top of the repo). Each file must be a single Python +expression — typically a dict literal. + +Consequences: + +- **No top-level imports.** No `from foo import bar`, no `import os`. + Helpers go in `libs/` and are reached via `repo.libs.` from + bundle code, not from node files. +- **No statements.** No `if`/`for`/`def` at the top level. Use + expressions (ternaries, comprehensions) instead. +- **Silent drop on parse error.** Vanilla bundlewrap silently omits a + node/group whose file fails to eval. The maintainer patched + `groups.py` (commit `dc40295`) to print the error instead — but it + still skips the file. Symptom: `bw nodes` lists fewer nodes than you + expect. Cure: re-read the file and check for accidental imports or + syntax errors. + +## Group inheritance order + +Metadata merges along this chain (from the fork's docs): + +``` +all → location → os → machine → applications → node +``` + +`groups/all.py` is the universal base; `groups/{locations,os,machine,applications}/` +hold the per-axis groups; the node's own metadata wins last. + +## Naming conventions + +| Where | Convention | Example | +|---|---|---| +| `nodes/*.py` | `..py` | `home.server.py`, `htz.mails.py` | +| `groups/*.py` | one file per group; subdir = purpose | `groups/applications/nextcloud.py`, `groups/os/debian-13.py` | +| `bundles/` | lowercase, hyphen-separated | `backup-server`, `bind-acme`, `routeros-monitoring` | +| Custom items | `items/.py` | `items/download.py` | + +Underscores in bundle names appear only in `_old`-suffixed leftovers +(see below); don't introduce new ones. + +## Suspension and soft-delete idioms + +These conventions look like dead code; they aren't. Don't clean them up. + +- **`*.py_` parked node.** A node file whose name ends in `.py_` is + silently excluded from `bw nodes` (the loader only matches `*.py`). + Used to keep decommissioned-but-not-deleted node configs in tree. + Example: `nodes/htz.l4d2.py_`. +- **"for now / disable / dummy / offline" markers.** Commits or comments + containing these phrases mark deliberate suspensions, not bugs. Check + `git log -- ` before re-enabling — the maintainer reverses + these manually when the underlying condition resolves. +- **`_old` / `_old2` directories.** Recovery buffers during big + refactors. Don't delete them without asking, even if they look + orphaned. + +## Files agents must not modify + +| Path | Why | +|---|---| +| `.secrets.cfg*` | vault key material | +| `.venv/` | local Python environment | +| `.cache/` | bw runtime cache | +| `.bw_debug_history` | shell history for `bw debug` | +| `.envrc` | direnv local-environment config | +| `/.cocoindex_code/` | local code index, not in git | +| `README.md` (root) | maintainer's personal TODO scratchpad — not project docs | + +Treat `hooks/` and `items/` (custom item types) with extra care: they +affect `bw`'s behavior for the whole repo, not a single bundle. + +## Working style + +- **Iterative commits are normal.** The maintainer commits in small + checkpoints (`+`, `fix`, `whitespace`, terse one-liners). Don't + rebase WIP branches without asking. As an agent, prefer + complete-feeling commits over mimicking the iterative style. +- **Burst-state awareness.** Before writing into a subsystem, run + `git log --since='1 month ago' bundles/` (or `nodes/.py`). + ≥10 recent commits means the subsystem is in flux; read the most + recent diffs first — your assumptions about its metadata shape may + already be stale. +- **Branch names.** PRs go through self-hosted Gitea/Forgejo. Branch + names are lowercase snake_case, descriptive + (`debian-13`, `htz.mails_debian_13_squash`, `l4d2_the_next`). + +## Bundlewrap-version migration recipe + +When the next bw major lands: + +1. Read the upstream migration guide. +2. `git grep` for affected reactor / item patterns. +3. Rewrite each (one commit per pattern is fine — see Working style). +4. Bump `requirements.txt` last. + +Pattern from commit `186d503` (bw 4 → 5). diff --git a/groups/AGENTS.md b/groups/AGENTS.md new file mode 100644 index 0000000..f8a91ea --- /dev/null +++ b/groups/AGENTS.md @@ -0,0 +1,110 @@ +# groups/ + +## What's here + +Groups attach bundles and shared metadata to nodes. One file per group, +organized by axis: + +``` +groups/ +├── all.py # universal base (every node belongs) +├── applications/.py # role-shaped groups (mailserver, monitored, …) +├── locations/.py # physical/network location (home, htz, …) +├── machine/.py # hardware kind (hardware, hetzner-cloud, raspberry-pi) +└── os/.py # OS major/variant (debian-13, debian-13-pve, routeros, …) +``` + +## Loader mechanism + +`groups.py` (top of the repo) walks `groups/` and runs `eval()` on each +`*.py`. Same eval-as-expression rule as +[`nodes/`](../nodes/AGENTS.md#loader-mechanism): one dict literal, no +top-level imports, no statements. Errors print and the group is skipped +— a real foot-gun, since a missing group silently changes node +membership. + +**Group files are *not* demagified.** Magic strings like +`!password_for:` only resolve in `nodes/*.py`. Inside a group file, +call `repo.vault.(...)` directly. + +## Inheritance and merge order + +Metadata merges along this chain: + +``` +all → location → os → machine → applications → node +``` + +Per-axis subdirs are conventional, not enforced — `bw` doesn't read the +subdir. Each group lists its `supergroups`, and `bw` resolves the DAG. +Membership is set-union; metadata merge follows the order above, with +the node's own `metadata` block winning last. + +## Conventions + +- **One group per file.** Filename without `.py` = group name. Subdir + groups them by axis for humans, not for bw. +- **Family files for OS variants.** Common parent + per-variant child. + Example: `debian-13-common.py` is shared by `debian-13.py` and + `debian-13-pve.py`. Use this pattern when introducing + related-but-distinct OS group families. +- **`all.py` is the universal default.** Currently empty (`{}`); kept + for the rare repo-wide opt-in. + +## How to add a group + +1. Pick the right axis subdir (or root `all.py` for universal default). +2. Create `groups//.py` as a single dict expression: + + ```python + { + 'supergroups': [ + # parent groups whose bundles/metadata this one extends + ], + 'bundles': [ + # bundles every member of this group should have + ], + 'metadata': { + # shared metadata for members + }, + } + ``` + +3. Wire the group into the relevant `nodes/*.py` (`'groups': {...}`) + or `groups/*.py` `supergroups` list. + +4. Verify with `bw nodes -a groups` and `bw metadata `. + +## How to add a new OS major (recipe) + +Pattern from prior debian-12 → debian-13 work: + +1. Add `groups/os/debian-N.py` and `groups/os/debian-N-common.py` + parallel to the existing files. Don't edit in place. +2. Add `data/apt/keys/debian-N-*.{asc,gpg}` for the new release's + signing keys. See + [`commands.md#apt-key-changes-need-offline-verification`](../docs/agents/commands.md#apt-key-changes-need-offline-verification) + before pushing keys live. +3. Bump dependent bundles that branch on `os_version` / + `os_codename` (`bundles/bind/items.py`, etc.). +4. Bump affected nodes' `groups` lists one at a time. Apply, watch. +5. Delete the old OS group file once no node references it. + +## Pitfalls + +- **`bw groups -n ` doesn't exist.** Use + `bw nodes -a groups`. +- **Cycles.** A group can't be its own supergroup transitively; + `bw test` catches this but the error message is terse. +- **Silent eval failure.** A group file with a syntax error is skipped + and prints a one-line error. If a node loses bundles unexpectedly, + scan `groups.py` output for the error. + +## See also + +- [`nodes/AGENTS.md`](../nodes/AGENTS.md) — node files; how + `groups: {...}` attaches groups. +- [`docs/agents/conventions.md`](../docs/agents/conventions.md) — + inheritance order, naming conventions, eval-loader constraints. +- Fork's [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) + — group attribute reference, metadata-merge semantics. diff --git a/hooks/AGENTS.md b/hooks/AGENTS.md new file mode 100644 index 0000000..3f58b35 --- /dev/null +++ b/hooks/AGENTS.md @@ -0,0 +1,62 @@ +# hooks/ + +## What's here + +Repo-level lifecycle hooks. Each `*.py` exports one or more functions +named after the lifecycle event they listen to (`apply_start`, +`node_apply_start`, `node_run_start`, `test`, `test_node`, …). Bw +discovers them by importing each module from `hooks/`. + +Discovery is by `ls hooks/` + the one-line docstring at the top of +each file: + +```sh +head -1 hooks/*.py +``` + +## Conventions + +- **One-line module docstring.** Every `hooks/*.py` starts with + `""": ."""`. Add one when introducing a new + hook; baseline is enforced by `grep -L '"""' hooks/*.py`. +- **Function name = event name.** Bw calls + `apply_start(repo, target, nodes, interactive=False, **kwargs)`, + `node_apply_start(repo, node, interactive, **kwargs)`, + `test(repo, **kwargs)`, etc. Always accept `**kwargs` so future bw + arguments don't break the hook. +- **Test gates use `test` / `test_node`.** Anything that should fail + `bw test` (and therefore CI / pre-apply sanity) goes here; avoid + doing test-style assertions in `apply_start`. + +## How to add a hook + +1. Pick the lifecycle event (see fork's `AGENTS.md` for the full list). +2. Create `hooks/.py` with the matching function and a + docstring. +3. Run `bw test` once to confirm the hook loads cleanly. + +## Pitfalls + +- **A hook that errors at import breaks every `bw` invocation** that + fires the hook's lifecycle — including `bw test` itself, which + defeats the obvious diagnostic. Test new hooks in isolation first: + + ```sh + bw debug -c "import sys; sys.path.insert(0, 'hooks'); import " + ``` + + Iterate there until the import is clean, then commit. +- **Hooks have access to the full repo (`repo`, `node`, `nodes`).** + Don't make them block on network unless that's the explicit purpose + (e.g. `test_ptr_records.py` does `dig` against `9.9.9.9`). +- **Order is not guaranteed across hook files.** Two hooks that both + define `apply_start` will both fire; don't assume which runs first. + +## See also + +- [`docs/agents/conventions.md`](../docs/agents/conventions.md) — + files-not-to-touch, vault rules. +- [`docs/agents/commands.md`](../docs/agents/commands.md) — test + workflow. +- Fork's [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) + — full hook lifecycle and signatures. diff --git a/items/AGENTS.md b/items/AGENTS.md new file mode 100644 index 0000000..7c35a6b --- /dev/null +++ b/items/AGENTS.md @@ -0,0 +1,64 @@ +# items/ + +## What's here + +Custom item types — each `*.py` is a `bundlewrap.items.Item` subclass +that adds a new item kind to the repo (e.g. `download:` from +`items/download.py`). + +Currently: + +- `items/download.py` — verifying file downloads with sha256 / GPG. + (Originally from `bundlewrap/plugins/item_download`; vendored in + because bw 4 dropped plugin support and it never came back.) + +## Conventions + +- **Filename = item-type slug.** `items/download.py` defines item + type `download:`. The class's `BUNDLE_ATTRIBUTE_NAME` is the dict + name in `items.py` (`downloads = {...}`). +- **Subclass `bundlewrap.items.Item`.** See the upstream + [bundlewrap docs on writing item types](https://docs.bundlewrap.org/dev/items/) + for the contract: `ITEM_ATTRIBUTES`, `fix`, `sdict`, `cdict`, + `display_dicts`, etc. +- **No metadata-reactor integration unless deliberate.** Item types + get to declare `NEEDS_STATIC` (cross-type ordering hints) — use it + sparingly; broad `NEEDS_STATIC` slows the whole DAG. + +## When to write a new custom item type vs. use `file:` (or +`action:`, etc.) + +| Situation | Use | +|---|---| +| One-off shell command at apply time | `action:` in the bundle | +| File whose source is in `bundles//files/` | `file:` | +| File you have to **fetch + verify** at apply | `download:` (already custom) | +| Behavior bw doesn't model and that recurs across bundles | new custom item | + +If you only need it once, an `action:` is almost always enough. +Custom item types are repo-wide and load on every `bw` invocation — +the cost is paid forever. + +## How to add a custom item type + +1. Pick a slug (lowercase, no underscore). +2. Create `items/.py` with an `Item` subclass. +3. `bw test` — broken item types break the loader for every bundle + that uses them, so test in isolation first. +4. Document the contract in the file's module docstring. + +## Pitfalls + +- **Items affect the whole repo.** A change to `items/.py` runs in + every node's apply. Treat custom items the way you treat `libs/` — + pure, fast, side-effect-free at import. +- **`NEEDS_STATIC` is a coarse hammer.** It enforces ordering across + *all* nodes for *all* items of those types; don't add unless + you've actually hit ordering issues. + +## See also + +- [`bundles/AGENTS.md`](../bundles/AGENTS.md) — items are consumed by + `items.py` in each bundle. +- Fork's [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) + — built-in item-type catalogue. diff --git a/libs/AGENTS.md b/libs/AGENTS.md new file mode 100644 index 0000000..00a44fc --- /dev/null +++ b/libs/AGENTS.md @@ -0,0 +1,68 @@ +# libs/ + +## What's here + +Shared Python helpers reachable from any bundle as +`repo.libs..`. One module per file (no packages). +Discovery is by `ls libs/` plus the one-line module docstring at the +top of each file. + +```sh +head -1 libs/*.py +``` + +## Conventions + +- **One file per module.** Filename (without `.py`) is the + importable name. `libs/wireguard.py` exposes `repo.libs.wireguard`. +- **One-line module docstring.** Every `libs/*.py` starts with + `""": ."""` so `ls + head` is a working index. + Add one when introducing a new lib; the rule is enforced by the + Phase-0 baseline (`grep -L '"""' libs/*.py` should report zero + files). +- **Pure helpers, no I/O at import.** Libs are imported on every `bw` + invocation. Heavy work or filesystem reads at module scope slow the + whole repo. Use functions, not module-level side effects. +- **No magic-string demagification here.** Libs receive already- + resolved values from bundles or nodes. Don't pass `!password_for:` + strings into libs — they won't be resolved. +- **`repo` and `vault` are globals available at runtime.** A few libs + (`wireguard.py`, etc.) reach `repo.vault.(...)` directly. + That's intentional inside libs, since libs run in the same loader + scope as bundles. + +## How to add a helper + +1. Either extract from a bundle that's growing complex, or add a new + `libs/.py` file. +2. Top line: `""": ."""`. +3. Pure functions only; document side effects in the docstring if any + slip through. +4. Use it from `bundles//items.py` or `metadata.py` as + `repo.libs..(...)`. + +## Pitfalls + +- **Lib changes have repo-wide blast radius.** Every bundle that + imports the lib re-evaluates on the next `bw test` / `bw apply`. + Before changing a lib's API, find consumers: + + ```sh + git grep -l 'repo.libs.' + ``` + +- **`@cache` is your friend** for deterministic key derivations + (`wireguard`, `rsa`, `ssh`); without it, `bw` recomputes per call, + which is slow. +- **Hashable wrappers.** `libs/hashable.py` exists because raw dicts + and sets aren't hashable, so you can't put them inside metadata + sets. Wrap with `repo.libs.hashable.hashable(...)` before nesting. + +## See also + +- [`bundles/AGENTS.md`](../bundles/AGENTS.md) — when to extract a + helper into `libs/` instead of duplicating across bundles. +- [`docs/agents/conventions.md`](../docs/agents/conventions.md) — + vault, demagify, naming. +- The fork's [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) + — `repo.libs` mechanism. diff --git a/nodes/AGENTS.md b/nodes/AGENTS.md new file mode 100644 index 0000000..150ed45 --- /dev/null +++ b/nodes/AGENTS.md @@ -0,0 +1,92 @@ +# nodes/ + +## What's here + +One file per node, ~22 in total. Each file is a single Python +expression (a dict literal) describing one machine: hostname, groups, +bundles, and metadata. + +## Loader mechanism + +`nodes.py` (top of the repo) walks `nodes/`, reads each `*.py` file, +runs `eval()` on its content, and then *demagifies* the result — +resolving any `!password_for:`, `!decrypt:`, `!decrypt_file:`, and +`!32_random_bytes_as_base64_for:` magic strings via `repo.vault`. See +[`docs/agents/conventions.md#secrets`](../docs/agents/conventions.md#secrets) +for the magic-string list. + +This loader shape has consequences: + +- **No top-level imports.** The file must be a single expression. No + `import os`, no `def`, no `if`. Use `repo.libs.` from bundle code + if you need a helper. +- **Silent drop on parse failure.** Vanilla bundlewrap omits a node + whose file fails to eval. The maintainer's `groups.py` was patched + in commit `dc40295` to print the error; the node-loader prints + on `nodes.py` errors via the same shape. Symptom either way: + `bw nodes` lists fewer nodes than expected. + +## Conventions + +- **Filename = node name.** `home.server.py` defines the node `home.server`. +- **Naming pattern: `..py`.** Examples: + `home.server.py`, `htz.mails.py`, `ovh.left4me.py`, + `mseibert.freescout.py`. +- **`*.py_` parks a node** without loading it. Used to keep + decommissioned-but-not-deleted configs in tree (e.g. + `htz.l4d2.py_`). The loader only matches `*.py`. +- **Magic strings only resolve here.** `!password_for:` etc. work in + node files; in groups, bundles, or items they don't — call + `repo.vault.(...)` directly there. + +## How to add a new node + +1. Create `nodes/..py` with a single dict expression: + + ```python + { + 'id': 'a-uuid-or-stable-name', + 'hostname': '', + 'groups': { + 'debian-13', + 'monitored', + # … + }, + 'bundles': { + # only bundles not provided by the groups above + }, + 'metadata': { + # node-local overrides and required keys + }, + } + ``` + +2. Add to relevant `groups//.py` if group membership is the + attachment point (preferred over per-node `bundles` lists). + +3. Verify: + - `bw nodes` — your node should appear. + - `bw nodes -a groups` — confirm group membership resolved + as expected (`bw groups -n ` does **not** exist). + - `bw metadata ` — confirm merged metadata. + +## Pitfalls + +- **Renaming a node renames the node.** Vault entries (anything keyed + on `!password_for:`), `bw hash` records, and ssh known_hosts + associations all key on node name. Search-and-replace before + renaming, or vault lookups silently return new (wrong) values. +- **Don't restore `_old` or `*.py_` files** without checking + [`conventions.md#suspension-and-soft-delete-idioms`](../docs/agents/conventions.md#suspension-and-soft-delete-idioms). + These are intentional parks/buffers, not bugs. +- **`id` must be unique.** A pre-apply hook (`hooks/unique_node_ids.py`) + enforces this; duplicate IDs fail `bw test` and `bw apply`. + +## See also + +- [`groups/AGENTS.md`](../groups/AGENTS.md) — group-membership patterns, + how metadata merges along the chain. +- [`docs/agents/conventions.md`](../docs/agents/conventions.md) — + demagify magic-strings, naming, eval-loader constraints. +- Fork's [`AGENTS.md`](https://github.com/CroneKorkN/bundlewrap/blob/main/AGENTS.md) + — node attribute reference (`hostname`, `username`, `dummy`, etc.).