left4me/deploy

left4me deploy — reference exemplar

The canonical deploy of ovh.left4me is driven by ckn-bw's bundles/left4me/ (attached via groups/applications/left4me.py); run bw apply ovh.left4me from the ckn-bw repo to deploy.

deploy/files/ is the canonical source of truth for static deployment artifacts — sudoers, sysctl drop-in, and hardening drop-ins for the systemd service units. ckn-bw delivers these via target-side symlinks from their on-host paths into /opt/left4me/src/deploy/files/... (safe because /opt/left4me/src is root-owned at runtime; the application cannot rewrite its own deployment artifacts).

deploy/scripts/ is the canonical source of truth for privileged helpers. ckn-bw creates target-side symlinks from /usr/local/{libexec/left4me,sbin}/ into /opt/left4me/src/deploy/scripts/{libexec,sbin}/ after git_deploy.

What remains under deploy/files/usr/local/lib/systemd/system/ is a set of reference fixtures — a curated subset of the systemd units ckn-bw's reactor emits at apply time. They exist so a fresh consumer (other than ckn-bw) can read this tree and understand the live unit shape, and so that deploy/tests/test_example_units.py can assert the reference matches the live form. The live base units are emitted by ckn-bw's systemd/units reactor with per-host CPU pinning and worker counts; the reference files must not include hardening directives (those live in the drop-ins, not the base units).

What's here

Path	Role
files/etc/sudoers.d/left4me	Canonical sudoers grants. Symlinked to /etc/sudoers.d/left4me. CI syntax test: tests/test_sudoers.py.
files/etc/sysctl.d/99-left4me.conf	Canonical sysctl drop-in (UDP buffers, fq_codel + BBR, kernel.yama.ptrace_scope=2). Symlinked to /etc/sysctl.d/99-left4me.conf.
files/etc/systemd/system/left4me-web.service.d/10-hardening.conf	Canonical hardening drop-in for left4me-web.service. Symlinked to the same on-host path.
files/etc/systemd/system/left4me-server@.service.d/10-hardening.conf	Canonical hardening drop-in for left4me-server@.service. Symlinked to the same on-host path.
files/etc/left4me/sandbox-resolv.conf	Example /etc/resolv.conf bound into the script-overlay sandbox (delivered as a bw files{} item, not a symlink).
files/usr/local/lib/systemd/system/left4me-web.service	Reference fixture — the web-app unit the reactor emits (per-host worker/thread counts omitted).
files/usr/local/lib/systemd/system/left4me-server@.service	Reference fixture — the per-instance gameserver unit template the reactor emits.
files/usr/local/lib/systemd/system/left4me-workshop-refresh.{service,timer}	Reference fixture — the daily workshop-refresh cron-equivalent.
files/usr/local/lib/systemd/system/l4d2-{game,build}.slice	Reference fixture — slice definitions (CPU/IO weights; reactor fills in AllowedCPUs= from host metadata).
scripts/libexec/{left4me-overlay,left4me-systemctl,left4me-journalctl,left4me-script-sandbox}	Canonical privileged helper commands. Symlinked under /usr/local/libexec/left4me/.
scripts/sbin/left4me	Canonical admin CLI wrapper. Symlinked to /usr/local/sbin/left4me.
templates/etc/left4me/host.env	Example host-library env (deployment-fixed paths).
templates/etc/left4me/web.env.template	Example web-app env. ckn-bw renders the real version via the matching Mako template in bundles/left4me/files/etc/left4me/web.env.mako.
tests/test_example_units.py	Locks down the reference units and env templates above; also asserts hardening drop-in shape.
tests/test_sudoers.py	Runs visudo -cf against the sudoers file in CI.

Target layout

The deployment uses these on-host paths (FHS-aligned):

• /etc/left4me/host.env — host library environment configuration.
• /etc/left4me/web.env — web app environment configuration.
• /etc/left4me/sandbox-resolv.conf — DNS resolv.conf bound into the script-overlay sandbox.
• /etc/sudoers.d/left4me — sudoers rules letting the left4me uid call the privileged helpers non-interactively.
• /etc/sysctl.d/99-left4me.conf — perf-baseline sysctls.
• /opt/left4me/src — deployed repository contents (via ckn-bw git_deploy). Root-owned; read-only at runtime. /opt/left4me/ itself is also root-owned and contains only src/.
• /var/lib/left4me/.venv — Python virtual environment for the web app (non-editable install of l4d2host + l4d2web).
• /var/lib/left4me/steam — steamcmd install (self-updates).
• /var/lib/left4me/left4me.db — SQLite database used by the web app.
• /var/lib/left4me/installation — shared L4D2 installation.
• /var/lib/left4me/overlays — overlay directories. Each overlay lives at ${overlay_id} under here.
• /var/lib/left4me/workshop_cache — deduplicated cache of .vpk files downloaded for workshop overlays. One file per Steam item, named {steam_id}.vpk. Workshop overlays symlink into this tree.
• /var/lib/left4me/instances — rendered instance specifications and per-instance state.
• /var/lib/left4me/runtime — per-instance runtime mount directories.
• /var/lib/left4me/tmp — temporary files used by deployment/runtime operations (incl. idmap staging binds).
• /usr/local/lib/systemd/system/ — global systemd unit files emitted by ckn-bw's systemd_units reactor.
• /usr/local/libexec/left4me/ — privileged helper commands, symlinked from deploy/scripts/libexec/.
• /usr/local/sbin/left4me — admin CLI wrapper, symlinked from deploy/scripts/sbin/left4me.

Runtime users

One system user does everything:

• left4me (home /var/lib/left4me, shell /usr/sbin/nologin): web app, host library, gameserver runtime, and script-overlay sandbox. The sandbox unit drops privileges via systemd-run and runs the user-authored bash inside a fully hardened transient service (see deploy/scripts/libexec/left4me-script-sandbox). Same-uid attack surface — sandbox escape reaching web.env, the SQLite DB, or running gameservers — is closed by that hardening profile plus system-wide kernel.yama.ptrace_scope=2, rather than by a uid boundary.

The user-count decision and its history live in docs/superpowers/specs/2026-05-15-user-uid-split-design.md.

Deployment

Production deploy:

# In the ckn-bw repo:
bw apply ovh.left4me

Admin bootstrap is a manual one-time step after the first apply (ckn-bw deliberately doesn't seed an admin to keep credentials out of the metadata pipeline):

sudo -u left4me LEFT4ME_ADMIN_USERNAME=admin \
  LEFT4ME_ADMIN_PASSWORD='change-me' \
  /var/lib/left4me/.venv/bin/flask --app l4d2web.app:create_app \
  create-user "$LEFT4ME_ADMIN_USERNAME" --admin

Rotate the bootstrap password after first login.

Overlay references

Overlay references are relative paths below ${LEFT4ME_ROOT}/overlays. With the default deployment root, they resolve under /var/lib/left4me/overlays. New overlays use ${overlay_id} as their path; the digit-only form is the only one created by the web app.

Invalid references are rejected:

• Absolute paths such as /srv/overlay.
• Parent traversal such as ../other or competitive/../../base.
• Empty path components such as competitive//base.
• Symlink escapes that resolve outside ${LEFT4ME_ROOT}/overlays.

The web app currently supports two overlay surfaces:

• workshop overlays (user-owned) — populated by downloading .vpk files from the public Steam Web API into ${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk and creating absolute symlinks under ${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk.
• script overlays — populated by an arbitrary user-authored bash script that runs inside systemd-run as left4me (under a fully hardened transient service unit), with the overlay directory bind-mounted RW at /overlay. Resource caps: 1h walltime, 4 GB RAM, 512 tasks, 200% CPU, 20 GB post-build disk cap.

Both caches and overlay directories are owned by left4me. If the web service ever runs as a different uid, ensure it shares a group with the host process and that both trees are group-readable.

Performance tuning

The deployment ships a host-side perf baseline (slices, unit directives, sysctls). See docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md for design rationale.

The knobs below are documented escape hatches — not auto-applied. Apply only after measuring a need and understanding the failure modes.

Network shaping

Three pieces of the baseline affect player-experience network behaviour:

1. Per-flow marking. ckn-bw's central bundles/nftables/ consumes left4me's nftables defaults and marks every UDP packet from uid left4me with DSCP EF and skb->priority 6. srcds doesn't set these itself, so without this rule its UDP is indistinguishable from any other flow.
2. Sysctl baseline. 99-left4me.conf sets udp_rmem_min=16384, udp_wmem_min=16384, default_qdisc=fq_codel, and tcp_congestion_control=bbr. Reduces head-of-line blocking when bulk TCP egress coexists with game UDP.
3. CAKE egress shaping. Configured per-interface via systemd-networkd metadata (network/<iface>/cake in ckn-bw's bundles/network/), which reapplies the CAKE qdisc across iface lifecycle events. Set the declared bandwidth to ≈95% of measured uplink — CAKE only shapes if its declared bandwidth is below the real bottleneck. Idle links with no competing egress see no visible CAKE effect; the win materialises under bulk traffic that would otherwise bufferbloat the link the players share.

CPU governor

The performance governor squeezes a few percent off jitter under bursty load. schedutil is acceptable for sustained UDP workloads.

sudo cpupower frequency-set -g performance

Install via sudo apt install linux-cpupower if the binary isn't present. Persist via your distro's CPU-frequency tooling (e.g. /etc/default/cpufrequtils).

CPU isolation (cores)

The deploy writes four AllowedCPUs= drop-ins so that by default only l4d2-game.slice is allowed to run on cores 1..N-1; system.slice, user.slice, and l4d2-build.slice are pinned to core 0. Game servers get the host minus core 0 exclusively; the build sandbox and the web app stay on core 0; a logged-in admin running CPU-heavy work in their shell can't steal cycles from a live match. Single-core hosts skip the cpuset drop-ins entirely; the rest of the perf baseline (cgroup weights, sysctls, OOM scores) still applies.

Per-instance CPUAffinity= (next subsection) composes on top of this — the per-instance value must be a subset of l4d2-game.slice's AllowedCPUs=, which the kernel enforces.

Per-instance CPU affinity

srcds is single-threaded per instance. On a multi-core host, pinning each instance to its own core can cut jitter under contention. Drop in /etc/systemd/system/left4me-server@<name>.service.d/affinity.conf:

[Service]
CPUAffinity=2

This pins the instance to CPU 2. A reasonable strategy on an N-core host: leave core 0 for the kernel + IRQs + system services, then pin one instance per remaining core.

NIC tuning

Hardware-specific (install via sudo apt install ethtool if not present). On a host with a single primary interface (replace eth0):

sudo ethtool -G eth0 rx 4096 tx 4096
sudo ethtool -K eth0 gro on lro off

If you run a high instance count, also pin the NIC's interrupts off the cores that game servers occupy (see /proc/interrupts and /proc/irq/<n>/smp_affinity).

Real-time scheduling (advanced, opt-in)

Source-engine servers do not need real-time scheduling, and a misbehaving srcds at any RT priority can starve kernel threads — even with the default kernel.sched_rt_runtime_us=950000 throttling 5% of CPU back. Use only if you have a measured jitter problem that the baseline does not solve.

/etc/systemd/system/left4me-server@.service.d/realtime.conf:

[Service]
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=10
LimitRTPRIO=10
AmbientCapabilities=CAP_SYS_NICE

The AmbientCapabilities=CAP_SYS_NICE line is needed because the service runs as User=left4me with NoNewPrivileges=true; without it some kernels/systemd combinations refuse to apply the RT policy.

Additional opt-in network knobs

• Ingress shaping via IFB. Egress CAKE alone does not protect srcds receive against ingress saturation (large workshop downloads, package fetches arriving at line rate). Worth flipping only when measurement shows ingress hurting receive.

  sudo modprobe ifb && sudo ip link set ifb0 up
  sudo tc qdisc add dev <uplink> handle ffff: ingress
  sudo tc filter add dev <uplink> parent ffff: protocol ip u32 \
      match u32 0 0 action mirred egress redirect dev ifb0
  sudo tc qdisc add dev ifb0 root cake bandwidth Xmbit ingress \
      diffserv4 dual-srchost
  

• net.core.busy_poll = 50 / net.core.busy_read = 50. Reduces UDP receive median latency by polling for incoming packets briefly at syscall boundaries. Cost: measurable CPU per syscall under load. Worth flipping if a host is dedicated to game serving and CPU headroom is plentiful.

• ethtool -K <iface> gro off. Some Source-engine ops disable generic receive offload to avoid receive-side coalescing latency. Hardware/driver dependent; document only.

Applying changes to running servers

Unit-file changes do not apply to already-running services. After any change:

sudo systemctl daemon-reload
# Restart each game server via the web UI's stop + start, or:
sudo systemctl restart 'left4me-server@*.service'