left4me/deploy/README.md

left4me deploy — reference exemplar

This directory is reference material, not the source of truth. The canonical deploy of ovh.left4me is driven by ckn-bw's bundles/left4me/ (attached via groups/applications/left4me.py); run bw apply ovh.left4me from the ckn-bw repo to deploy.

The privileged scripts the application installs live at the repo root under scripts/libexec/ and scripts/sbin/ — application code, not deploy artifacts. ckn-bw's install_left4me_scripts action reads them from /opt/left4me/src/scripts/{libexec,sbin}/ after git_deploy and installs them into the standard FHS targets on the host.

What remains under deploy/files/ and deploy/templates/ is a set of readable examples — sudoers, sysctl, sandbox-resolv.conf, env templates, and a curated subset of the systemd units ckn-bw's reactor emits at apply time. They exist so a fresh consumer (other than ckn-bw) could read this tree and assemble an equivalent deployment. They are not the bytes ckn-bw installs; ckn-bw carries its own copies of the verbatim configs in bundles/left4me/files/etc/, and emits the live units from bundles/left4me/metadata.py's systemd_units reactor.

What's here

Path	Role
files/etc/sudoers.d/left4me	Example sudoers grants. Lockdown test: scripts/tests/test_sudoers_grants.py.
files/etc/sysctl.d/99-left4me.conf	Example sysctl perf baseline (UDP buffers, fq_codel + BBR).
files/etc/left4me/sandbox-resolv.conf	Example /etc/resolv.conf bound into the script-overlay sandbox.
files/usr/local/lib/systemd/system/left4me-web.service	Example of the web-app unit the reactor emits.
files/usr/local/lib/systemd/system/left4me-server@.service	Example of the per-instance gameserver unit.
files/usr/local/lib/systemd/system/left4me-workshop-refresh.{service,timer}	Example of the daily workshop-refresh cron-equivalent.
files/usr/local/lib/systemd/system/l4d2-{game,build}.slice	Example slice definitions (CPU/IO weights).
templates/etc/left4me/host.env	Example host-library env (deployment-fixed paths).
templates/etc/left4me/web.env.template	Example web-app env. ckn-bw renders the real version via the matching Mako template in bundles/left4me/files/etc/left4me/web.env.mako.
tests/test_example_units.py	Locks down the example units & env templates above.

The privileged scripts (left4me-overlay, left4me-script-sandbox, left4me-systemctl, left4me-journalctl, sbin/left4me) used to live under this tree at files/usr/local/{libexec,sbin}/; they moved to scripts/{libexec,sbin}/ because they are application code, not deploy artifacts.

Target layout

The deployment uses these on-host paths (FHS-aligned):

• /etc/left4me/host.env — host library environment configuration.
• /etc/left4me/web.env — web app environment configuration.
• /etc/left4me/sandbox-resolv.conf — DNS resolv.conf bound into the script-overlay sandbox.
• /etc/sudoers.d/left4me — sudoers rules letting the left4me uid call the privileged helpers non-interactively.
• /etc/sysctl.d/99-left4me.conf — perf-baseline sysctls.
• /opt/left4me — deployed repository contents (via ckn-bw git_deploy).
• /opt/left4me/.venv — Python virtual environment for the web app.
• /var/lib/left4me/left4me.db — SQLite database used by the web app.
• /var/lib/left4me/installation — shared L4D2 installation.
• /var/lib/left4me/overlays — overlay directories. Each overlay lives at ${overlay_id} under here.
• /var/lib/left4me/workshop_cache — deduplicated cache of .vpk files downloaded for workshop overlays. One file per Steam item, named {steam_id}.vpk. Workshop overlays symlink into this tree.
• /var/lib/left4me/instances — rendered instance specifications and per-instance state.
• /var/lib/left4me/runtime — per-instance runtime mount directories.
• /var/lib/left4me/tmp — temporary files used by deployment/runtime operations (incl. idmap staging binds).
• /usr/local/lib/systemd/system/ — global systemd unit files emitted by ckn-bw's systemd_units reactor.
• /usr/local/libexec/left4me/ — privileged helper commands installed from scripts/libexec/.
• /usr/local/sbin/left4me — admin CLI wrapper installed from scripts/sbin/left4me.

Runtime users

Two system users are involved:

• left4me (home /var/lib/left4me, shell /usr/sbin/nologin): web app, host library, and gameserver runtime.
• l4d2-sandbox (no home, shell /usr/sbin/nologin): unprivileged uid the script-overlay sandbox drops into via systemd-run. The left4me-script-sandbox helper sets up an idmapped bind from the sandbox uid back to left4me on a staging path so overlay writes land on disk as left4me-owned. The split is load-bearing: a sandbox escape would otherwise see web.env, the SQLite DB, and running gameservers.

(Whether the gameserver runtime should be split off into a third uid is an open design question — see docs/superpowers/specs/2026-05-15-user-uid-split-design.md.)

Deployment

Production deploy:

# In the ckn-bw repo:
bw apply ovh.left4me

Admin bootstrap is a manual one-time step after the first apply (ckn-bw deliberately doesn't seed an admin to keep credentials out of the metadata pipeline):

sudo -u left4me LEFT4ME_ADMIN_USERNAME=admin \
  LEFT4ME_ADMIN_PASSWORD='change-me' \
  /opt/left4me/.venv/bin/flask --app l4d2web.app:create_app \
  create-user "$LEFT4ME_ADMIN_USERNAME" --admin

Rotate the bootstrap password after first login.

Overlay references

Overlay references are relative paths below ${LEFT4ME_ROOT}/overlays. With the default deployment root, they resolve under /var/lib/left4me/overlays. New overlays use ${overlay_id} as their path; the digit-only form is the only one created by the web app.

Invalid references are rejected:

• Absolute paths such as /srv/overlay.
• Parent traversal such as ../other or competitive/../../base.
• Empty path components such as competitive//base.
• Symlink escapes that resolve outside ${LEFT4ME_ROOT}/overlays.

The web app currently supports two overlay surfaces:

• workshop overlays (user-owned) — populated by downloading .vpk files from the public Steam Web API into ${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk and creating absolute symlinks under ${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk.
• script overlays — populated by an arbitrary user-authored bash script that runs inside systemd-run as the unprivileged l4d2-sandbox UID, with the overlay directory bind-mounted RW at /overlay. Resource caps: 1h walltime, 4 GB RAM, 512 tasks, 200% CPU, 20 GB post-build disk cap.

Both caches and overlay directories are owned by left4me. If the web service ever runs as a different uid, ensure it shares a group with the host process and that both trees are group-readable.

Performance tuning

The deployment ships a host-side perf baseline (slices, unit directives, sysctls). See docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md for design rationale.

The knobs below are documented escape hatches — not auto-applied. Apply only after measuring a need and understanding the failure modes.

Network shaping

Three pieces of the baseline affect player-experience network behaviour:

1. Per-flow marking. ckn-bw's central bundles/nftables/ consumes left4me's nftables defaults and marks every UDP packet from uid left4me with DSCP EF and skb->priority 6. srcds doesn't set these itself, so without this rule its UDP is indistinguishable from any other flow.
2. Sysctl baseline. 99-left4me.conf sets udp_rmem_min=16384, udp_wmem_min=16384, default_qdisc=fq_codel, and tcp_congestion_control=bbr. Reduces head-of-line blocking when bulk TCP egress coexists with game UDP.
3. CAKE egress shaping. Configured per-interface via systemd-networkd metadata (network/<iface>/cake in ckn-bw's bundles/network/), which reapplies the CAKE qdisc across iface lifecycle events. Set the declared bandwidth to ≈95% of measured uplink — CAKE only shapes if its declared bandwidth is below the real bottleneck. Idle links with no competing egress see no visible CAKE effect; the win materialises under bulk traffic that would otherwise bufferbloat the link the players share.

CPU governor

The performance governor squeezes a few percent off jitter under bursty load. schedutil is acceptable for sustained UDP workloads.

sudo cpupower frequency-set -g performance

Install via sudo apt install linux-cpupower if the binary isn't present. Persist via your distro's CPU-frequency tooling (e.g. /etc/default/cpufrequtils).

CPU isolation (cores)

The deploy writes four AllowedCPUs= drop-ins so that by default only l4d2-game.slice is allowed to run on cores 1..N-1; system.slice, user.slice, and l4d2-build.slice are pinned to core 0. Game servers get the host minus core 0 exclusively; the build sandbox and the web app stay on core 0; a logged-in admin running CPU-heavy work in their shell can't steal cycles from a live match. Single-core hosts skip the cpuset drop-ins entirely; the rest of the perf baseline (cgroup weights, sysctls, OOM scores) still applies.

Per-instance CPUAffinity= (next subsection) composes on top of this — the per-instance value must be a subset of l4d2-game.slice's AllowedCPUs=, which the kernel enforces.

Per-instance CPU affinity

srcds is single-threaded per instance. On a multi-core host, pinning each instance to its own core can cut jitter under contention. Drop in /etc/systemd/system/left4me-server@<name>.service.d/affinity.conf:

[Service]
CPUAffinity=2

This pins the instance to CPU 2. A reasonable strategy on an N-core host: leave core 0 for the kernel + IRQs + system services, then pin one instance per remaining core.

NIC tuning

Hardware-specific (install via sudo apt install ethtool if not present). On a host with a single primary interface (replace eth0):

sudo ethtool -G eth0 rx 4096 tx 4096
sudo ethtool -K eth0 gro on lro off

If you run a high instance count, also pin the NIC's interrupts off the cores that game servers occupy (see /proc/interrupts and /proc/irq/<n>/smp_affinity).

Real-time scheduling (advanced, opt-in)

Source-engine servers do not need real-time scheduling, and a misbehaving srcds at any RT priority can starve kernel threads — even with the default kernel.sched_rt_runtime_us=950000 throttling 5% of CPU back. Use only if you have a measured jitter problem that the baseline does not solve.

/etc/systemd/system/left4me-server@.service.d/realtime.conf:

[Service]
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=10
LimitRTPRIO=10
AmbientCapabilities=CAP_SYS_NICE

The AmbientCapabilities=CAP_SYS_NICE line is needed because the service runs as User=left4me with NoNewPrivileges=true; without it some kernels/systemd combinations refuse to apply the RT policy.

Additional opt-in network knobs

• Ingress shaping via IFB. Egress CAKE alone does not protect srcds receive against ingress saturation (large workshop downloads, package fetches arriving at line rate). Worth flipping only when measurement shows ingress hurting receive.

  sudo modprobe ifb && sudo ip link set ifb0 up
  sudo tc qdisc add dev <uplink> handle ffff: ingress
  sudo tc filter add dev <uplink> parent ffff: protocol ip u32 \
      match u32 0 0 action mirred egress redirect dev ifb0
  sudo tc qdisc add dev ifb0 root cake bandwidth Xmbit ingress \
      diffserv4 dual-srchost
  

• net.core.busy_poll = 50 / net.core.busy_read = 50. Reduces UDP receive median latency by polling for incoming packets briefly at syscall boundaries. Cost: measurable CPU per syscall under load. Worth flipping if a host is dedicated to game serving and CPU headroom is plentiful.

• ethtool -K <iface> gro off. Some Source-engine ops disable generic receive offload to avoid receive-side coalescing latency. Hardware/driver dependent; document only.

Applying changes to running servers

Unit-file changes do not apply to already-running services. After any change:

sudo systemctl daemon-reload
# Restart each game server via the web UI's stop + start, or:
sudo systemctl restart 'left4me-server@*.service'