The backup server sleeps most of the time, so the check was UNKNOWN.
check_by_sshmon now runs an optional wake command first; the freshness
check uses the wol-sleeper wake command, runs every 24h and retries
after 6h so a failure does not wake the server every 30s. nagios on
the icinga node may run the waker command as wol.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
The for loop's else branch always ran, so every authorized user with
commands also got its key without restriction; root on every node had
a full shell as wol on the waker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
Since icingaweb2 2.12 the monitoring module is a separate package;
without it the web UI showed no hosts or services at all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
Clients deleted every auto-backup_ snapshot on the server except the
newest, so the backup held one state only. prune-snapshots now keeps
the newest per day, week and month (14/8/24 by default) independent of
the source, and the clients call it after each receive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
Remote clients reach the backup server over wireguard only on its
internal address; waiting for the wakeonlan network address never
ended there, so htz.mails and the mseibert nodes had no backup since
2026-01-07. The wait now gives up after ten minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
It was commented out in 8539f59 while the old binary did not run on
Debian 13; the statically linked build does, so proc_io gets data
again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
Together 2 TB on home.backups; the bootshorn recordings are being
reduced to their findings, the windows backup moves elsewhere.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
OC\Preview\Image is only the base class, so since 2899cd5 no JPEG or
PNG preview could be generated ("No provider successfully handled the
preview generation").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
dig @9.9.9.9 fails in the agent sandbox, which allows no UDP, so bw
test aborted there. The check now asks dns.google over HTTPS and runs
once per bw test instead of once per node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
postfix refuses to start with an empty hash_queue_names ("bad string
length 0 < 1"); only deferred has to stay unhashed for telegraf.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
postfix recreates the hash subdirectories of deferred and defer with
mode 0700, whose mask cancels the inherited telegraf acl, so
inputs.postfix failed with permission denied. Also drop the deprecated
smtpd_use_tls; smtpd_tls_security_level = may already covers it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
procio and pressure_stall are now linked statically. Apply only after
the binaries on dl.sublimity.de have been rebuilt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
build-agent is no longer needed for local static builds; the ci
strategy had no repo configured, and lonercrew, its only user, is
not deployed on any node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
The 2022 binaries link libpcre.so.3, which Debian 13 no longer ships.
crystal now comes from the openSUSE repo of the node's Debian release
and links with --static, so the result runs on any release; the build
runs locally instead of cross-compiling through the build-agent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
VLAN 52 "carina" on sfp-sfpplus7 of the rack switch, routed and
masqueraded by home.router via 192.168.178.1; pppoe and the rolf test
VLAN are commented out for now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
Every linux node gets the user with the ssh-grant agent key and
passwordless sudo; the key only works while a grant is active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RywMtsL8qXG7GDY3MHq9Uh
Derives LOG_LISTENER_ADDR from the node's external IPv4 (stripping the
CIDR suffix). The destination MUST be non-loopback — Source silently
drops logaddress UDP destinations in 127.0.0.0/8 (registration
succeeds but no sendto ever happens). LOG_LISTENER_BIND is pinned
to 0.0.0.0:28000 so the listener accepts on every interface,
including the lo-routed packets the kernel uses for same-host
destinations.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sleeper now exposes the full `/usr/bin/wakeonlan -i <broadcast> <mac>`
invocation as wol-sleeper/waker_command; waker reads that instead of
rebuilding the command line itself. Adds the `-i <subnet-broadcast>`
flag so magic packets reach sleepers on a different L2 segment than
the waker.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops both git_deploy clones (ohmyzsh, zsh-autosuggestions), the chown
action, the bw.zsh-theme, and the plugin directories. /etc/zsh/zprofile
now embeds a compact prompt mirroring the old theme's colors (no git
segment), sources zsh-autosuggestions from the new apt package, and
binds prefix-history search on the arrow keys.
Per-user ~/.zshrc is kept as a one-line marker so zsh-newuser-install
doesn't fire when ckn logs in.
Leftover /etc/zsh/oh-my-zsh/ trees aren't removed automatically — clean
once with `bw run linux 'rm -rf /etc/zsh/oh-my-zsh'` after the next
fleet-wide apply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Direnv has no built-in `use_uv`; the `use uv` line in 219f045 silently
errored out (`use_uv: command not found`) and the shell ended up with
an unactivated `.venv`. The matching upstream proposal is `layout_uv`
in direnv/direnv#1329 (open, maintainer-approved). Install that
function (verbatim, plus a small auto-sync-on-pyproject-change block)
in ~/.config/direnv/direnvrc on the local machine and switch the
.envrc verb here to match. When #1329 ships in a direnv release, the
custom direnvrc function can stay (strict superset) or be removed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces requirements.txt + `python -m venv` + periodic `pip install`
with a single pyproject.toml driven by `uv sync`. .envrc shrinks to
`use uv` (plus the BW_*_WORKERS helper). The CroneKorkN/bundlewrap
fork is now pinned via [tool.uv.sources] (git, non-editable) instead
of `pip install -e git+...`; pinned rev lives in uv.lock and bumps
via `uv sync --upgrade-package bundlewrap`. Mirrors left4me 77b5e01.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace left4me_create_venv + left4me_pip_upgrade + left4me_pip_install
(the tempdir-copy dance) with a single left4me_uv_sync action driven by
left4me's committed uv.lock. Deterministic dep versions, no source-tree
mutation during build (hatchling PEP 660 editable installs don't write
to source), one action instead of three.
uv is not in Trixie's apt archive (experimental/sid only), so a new
left4me_install_uv action downloads a pinned 0.11.8 binary tarball
from astral-sh/uv releases, SHA256-verifies against the published
.sha256 sibling, and installs into /usr/local/bin. Idempotent via
`unless` on the version string — only re-runs when the pin is bumped.
Pattern matches left4me_install_steamcmd elsewhere in this bundle.
apt.packages: drop python3-pip and python3-venv (uv replaces both;
no other consumer in the bundle). Keep python3 and python3-dev — uv
shells out to the system Python interpreter.
PYTHONPATH=/opt/left4me/src removed from left4me_alembic_upgrade and
left4me_seed_overlays — was a workaround for the previous layout's
package-dir indirection; with the new standard layout + editable
install, the venv resolves both members natively.
Per left4me/docs/superpowers/plans/2026-05-15-uv-workspace-execution.md.
Requires the matching commit on left4me's master.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Rewrite "What this bundle does" to reflect the post-migration model:
- Target-side symlinks table for the 6 static artifacts (sudoers, sysctl,
2 hardening drop-ins, 4 libexec helpers, sbin wrapper)
- Reactor-emitted units section (per-host shape: web/server@ units, slices,
cpuset drop-ins)
- bw files{} for the templated env files
- Action chains section covering the full deploy lifecycle
- Reference to the design doc for rationale
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/usr/local/libexec/left4me/* and /usr/local/sbin/left4me are now
target-side symlinks into /opt/left4me/src/deploy/scripts/...
Replaces the install_left4me_scripts copy-action.
Part of 2026-05-15-deployment-responsibility-design.md migration
step 4. Verified on ovh.left4me: helpers run via sudo from the web
app context; gameserver lifecycle helper invocation succeeds.
Sudoers drop-in lives in left4me/deploy/files/etc/sudoers.d/left4me
(single source of truth). Deleted the verbatim mirror in this bundle's
files/ tree. Added an idempotent chmod action so the in-checkout file
is 0440 root:root — required for sudo to accept it through the symlink.
Syntax check on the source file is now a left4me-side pytest
(deploy/tests/test_sudoers.py) running visudo -cf.
Part of 2026-05-15-deployment-responsibility-design.md migration step 3.
Post-daemon-reload self-test that asserts both
/etc/systemd/system/left4me-{web,server@}.service.d/10-hardening.conf
appear in `systemctl show -p DropInPaths` for the unit. Catches drift
where the symlink lands but daemon-reload didn't take, or someone
manually unlinked the drop-in.
For the gameserver template we query `left4me-server@verify.service` —
systemd resolves drop-ins for a template instance against
`name@.service.d/` regardless of whether the instance has ever started.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reactor stops emitting hardening directives in the unit bodies. The
HARDENING_COMMON / HARDENING_SERVER / HARDENING_WEB constants are gone.
Effective hardening on the live units now comes from drop-in files
shipped by left4me at:
/etc/systemd/system/left4me-web.service.d/10-hardening.conf
/etc/systemd/system/left4me-server@.service.d/10-hardening.conf
Both are target-side symlinks into /opt/left4me/src/deploy/files/...
(safe because /opt/left4me/src is root-owned post-relocation refactor).
Verified on ovh.left4me: systemctl show reports the same directives as
the pre-refactor baseline; relevant hardening test-plan checks pass.
Sysctl drop-in lives in left4me/deploy/files/etc/sysctl.d/99-left4me.conf
(absorbed kernel.yama.ptrace_scope from the metadata entry). Deliver
via target-side symlink instead of a verbatim copy.
Canary for the deployment-responsibility reshape (left4me design doc
2026-05-15-deployment-responsibility-design.md, step 1). Validated
end-to-end on ovh.left4me: symlink resolves to the checkout,
sysctl --system fires on apply, kernel target value matches, idempotent.
One-shot cleanup of stale /etc/sysctl.d/99-left4me-ptrace.conf
(orphan from earlier apply; bundles/sysctl does not auto-purge unmanaged
files).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related changes landed together:
1. /opt/left4me/ becomes a root-owned deploy-artifact root. Only
/opt/left4me/src lives there. Source tree is no longer mutated
by pip at runtime; left4me only needs read access.
2. Runtime mutable state moved to /var/lib/left4me/: the venv (was
/opt/left4me/.venv) and steamcmd (was /opt/left4me/steam).
The non-editable install copies the (root-owned) source to a
left4me-owned tempdir before `pip install --force-reinstall`.
setuptools.build_meta writes <pkg>.egg-info/ into the source dir
during get_requires_for_build_wheel, so a direct `pip install
/opt/left4me/src/l4d2*` fails on a root-owned source. The
temp-copy is what `python -m build` ought to do but doesn't (its
build isolation only sandboxes deps).
Prereq for the deployment-responsibility reshape: target-side
symlinks from /etc/... into /opt/left4me/src/deploy/files/... are
now safe by construction (left4me cannot rewrite its own hardening
profile).
Design + verification record: left4me/docs/superpowers/specs/
2026-05-15-runtime-state-relocation-design.md
Verified on ovh.left4me: bw apply idempotent on second pass (0
fixed, 0 failed); pip show reports site-packages location, no
Editable project location; web + gameserver units run clean;
alembic current returns head.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same class of leak as the .steam bind: workshop VPKs in overlays are
symlinks pointing to /var/lib/left4me/workshop_cache/<id>.vpk. With
TemporaryFileSystem=/var/lib in HARDENING_SERVER and workshop_cache
not in BindReadOnlyPaths, the targets are invisible inside the unit's
mount namespace. Source silently fails to load the addons — no log
message, the addon just doesn't appear in-game (saw the ions vocalizer
workshop VPK dangling on server@2).
Add workshop_cache to the bind list. Read-only is fine; srcds reads
the VPKs, doesn't write them (web app populates the cache as left4me).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original LAN-mode rejection turned out to be a downstream cascade
from Steam SDK init failure (.steam dir invisible + cpu.cpp assert
on hidden /proc/cpuinfo) — fixed in subsequent commits by binding
/var/lib/left4me/.steam + /opt/left4me/steam back through
TemporaryFileSystem and dropping ProcSubset=pid. With Steam master-
server registration now succeeding ("Connection to Steam servers
successful. VAC secure mode is activated."), the server defaults to
internet mode (sv_lan=0) on its own.
Drop the explicit override. If LAN-mode rejection ever recurs the
fix is to identify the upstream Steam-registration issue, not to
re-set this cvar.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same pattern as the web-unit fix (commit b3f...): ProcSubset=pid hides
/proc/cpuinfo and /proc/sys/*. Source's tier0/cpu.cpp asserts on
cpuinfo read failure; SteamAPI_Init then fails with "create pipe
failed" as a downstream cascade, and srcds registers as LAN (rejecting
external clients with "LAN servers are restricted to local clients").
PrivatePIDs=true (private PID namespace) remains the load-bearing
peer-process isolation: no foreign PIDs visible to srcds in its own
namespace. ProtectProc=invisible is the foreign-uid /proc hide.
ProcSubset=pid was a defense-in-depth layer hiding kernel-introspection
files (cpuinfo, meminfo, sysctls); losing it only exposes host kernel
info, which is not sensitive in this threat model and is the same
information any user on the host already sees.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Server@.service has TemporaryFileSystem virtualizing /var/lib and /opt;
the .steam home dir (which holds symlinks to /opt/left4me/steam/linux{32,64})
wasn't bound back into the unit's filesystem view. srcds dlopen's
~/.steam/sdk32/steamclient.so for Steam master-server registration —
under the unit it returned ENOENT, SteamAPI_Init failed, and the server
fell back to LAN-only mode regardless of +sv_lan 0. Clients then got
"LAN servers are restricted to local clients (class C)" on connect.
Bind both /var/lib/left4me/.steam (the symlinks) and /opt/left4me/steam
(the symlink targets) read-only into the unit. The Steam SDK file is
written by steamcmd as part of the install flow, so RO is fine — srcds
doesn't write back.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
With +ip 0.0.0.0 (added in previous commit to make TCP RCON reachable
via loopback), Source engine can't auto-determine whether the server
is public-facing and defaults to LAN mode (sv_lan=1). Clients
connecting from public IPs get rejected with "LAN servers are
restricted to local clients (class C)".
Force sv_lan=0 explicitly so public clients can connect.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes to fix web's "No data / Connection refused" symptoms:
1. ExecStart adds +ip 0.0.0.0 so srcds_linux binds both UDP and TCP
to all interfaces (incl. loopback). Default Source auto-IP picks
the primary host IP (141.95.32.8 here), so TCP RCON was only on
the public IP; web's 127.0.0.1 connect got ECONNREFUSED.
2. nftables/input drops the TCP accept on the game port range. UDP
stays open for players. Loopback (127.0.0.1) bypasses the input
chain via the existing 'iifname lo accept' rule in
bundles/nftables/files/nftables.conf, so web→RCON via loopback
still works.
Net effect: web's live-state polling + console RCON work via
loopback; RCON is no longer reachable from the public internet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Discovered post-deploy: ProcSubset=pid hides /proc/sys/kernel/random/boot_id
which journalctl reads at startup. The web app invokes
`sudo -n left4me-journalctl` to stream live server logs into the UI;
journalctl bails with "Failed to get boot ID" before producing any
output. Web log streaming was silently broken.
Server unit keeps ProcSubset=pid (srcds doesn't invoke journalctl);
web unit drops it. ProtectProc=invisible remains in COMMON — that's
the load-bearing D4 defense (foreign-uid /proc hidden).
Reproducer that confirms the diagnosis:
systemd-run --pipe --uid=left4me --gid=left4me \
-p ProcSubset=pid -p ProtectProc=invisible \
-p ProtectSystem=strict -p PrivateTmp=true \
[...rest of web hardening...] \
-- sh -c 'sudo -n left4me-journalctl 2 --lines 3 --follow >/var/lib/left4me/tmp/out 2>&1'
# cat /var/lib/left4me/tmp/out → "Failed to get boot ID: No such file or directory"
# rc → 1
With ProcSubset=all: timeout 124 (helper running), 3 lines streamed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion to the uid-collapse refactor on the left4me side
(docs/superpowers/plans/2026-05-15-uid-collapse.md). The script-
sandbox now runs as left4me too, defended by the hardening profile
that landed earlier today rather than a kernel uid boundary.
users + groups dicts: remove the l4d2-sandbox entry (uid/gid 981).
/var/lib/left4me mode: 0711 → 0755. The 0711 was specifically a
traverse-only loosening for the sandbox uid; with one user, the
natural mode is back.
Belt-and-braces with the gameserver unit's SystemCallFilter=~@debug +
PrivateUsers=true. Currently applied by hand on left4.me (left over
from the hardening test plan's Test 9); landing in the bundle so it
survives bw apply and is reproducible on any future host.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the sudo-compatible hardening subset to the web unit. Tightens
ProtectSystem=full → strict. NoNewPrivileges, PrivateUsers,
RestrictSUIDSGID, empty CapabilityBoundingSet, and ~@privileged in the
syscall filter intentionally absent (sudo-incompatible until a future
refactor replaces the helper sudo with systemctl-managed transient
units).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the inline hardening directives on the gameserver unit with
the shared HARDENING_SERVER dict. Removes legacy ReadOnlyPaths /
ReadWritePaths (superseded by TemporaryFileSystem + BindReadOnlyPaths
+ BindPaths in the dict). Brings the unit to the proven Test 7
composition with the i386 amendment (SystemCallArchitectures=native x86)
and PrivatePIDs=true.
Not deployed until bw apply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three shared dicts capturing the proven hardening composition from the
left4me hardening test plan (left4me commit 461b8d0). HARDENING_COMMON
is the directive set both managed units take verbatim; HARDENING_SERVER
adds the sudo-incompatible flags + filesystem virtualization + i386
amendment + PrivatePIDs + SocketBindAllow; HARDENING_WEB adds the
sudo-compatible syscall filter.
Not yet spread into the unit emission — that's the next commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
