History

mwiegand 9e0f6f17ef docs(deploy): performance-tuning escape-hatch section in README Documents CPU governor, per-instance CPUAffinity, NIC tuning, and SCHED_FIFO opt-in patterns. None of these are auto-applied; they're ops-side knobs for measured problems the perf baseline doesn't solve.		2026-05-09 10:09:40 +02:00
..
files	feat(deploy): script-sandbox runs in l4d2-build.slice + OOMScoreAdjust=500	2026-05-09 10:01:38 +02:00
templates/etc/left4me	feat(deploy): add production-like test deployment	2026-05-06 19:30:10 +02:00
tests	feat(deploy): install slice + sysctl artifacts and apply via sysctl --system	2026-05-09 10:05:41 +02:00
deploy-test-server.sh	feat(deploy): install slice + sysctl artifacts and apply via sysctl --system	2026-05-09 10:05:41 +02:00
README.md	docs(deploy): performance-tuning escape-hatch section in README	2026-05-09 10:09:40 +02:00

README.md

left4me Deployment

This directory contains the production-like test deployment for a Linux server. It installs the repository into a fixed host layout, configures a dedicated runtime user, installs systemd units, and wires the web app to host operations through privileged helper commands.

Target Layout

The deployment uses these paths:

/etc/left4me/host.env: host library environment configuration.
/etc/left4me/web.env: web app environment configuration.
/opt/left4me/.venv: Python virtual environment for deployed commands.
/opt/left4me: deployed repository contents.
/var/lib/left4me/left4me.db: SQLite database used by the web app.
/var/lib/left4me/installation: shared L4D2 installation.
/var/lib/left4me/overlays: overlay directories. Each overlay lives at ${overlay_id} under here.
/var/lib/left4me/workshop_cache: deduplicated cache of .vpk files downloaded for workshop overlays. One file per Steam item, named {steam_id}.vpk. Workshop overlays symlink into this tree.
/var/lib/left4me/global_overlay_cache: cache of non-Steam map archives and extracted .vpk files used by managed global map overlays.
/var/lib/left4me/instances: rendered instance specifications and per-instance state.
/var/lib/left4me/runtime: per-instance runtime mount directories.
/var/lib/left4me/tmp: temporary files used by deployment/runtime operations.
/usr/local/lib/systemd/system: global systemd unit files, including left4me-server@.service.
/usr/local/libexec/left4me: privileged helper commands, including left4me-systemctl, left4me-journalctl, and left4me-overlay (the latter mounts the per-instance kernel overlay in PID 1's mount namespace via nsenter).
/etc/sudoers.d/left4me: sudoers rules allowing the web/runtime commands to call the helpers non-interactively.

Static units are generated for /var/lib/left4me. If LEFT4ME_ROOT changes, regenerate and reinstall the unit files instead of reusing the existing static units.

Runtime User

The deployment creates and runs host operations as the dedicated runtime user:

Username: left4me
Home: /var/lib/left4me
Shell: /usr/sbin/nologin

Running A Test Deployment

Run the deployment from the repository root:

deploy/deploy-test-server.sh deploy-user@example-host

The SSH user must be able to run sudo on the target host. The deployment configures system packages, directories, environment files, helper scripts, sudoers rules, Python dependencies, and systemd units.

Admin Bootstrap

Set the bootstrap credentials in the environment when creating the first admin user:

LEFT4ME_ADMIN_USERNAME=admin \
LEFT4ME_ADMIN_PASSWORD='change-me' \
flask create-user "$LEFT4ME_ADMIN_USERNAME" --admin

Use a strong one-time password and rotate it after first login if needed.

Overlay References

Overlay references are relative paths below ${LEFT4ME_ROOT}/overlays. With the default deployment root, they resolve under /var/lib/left4me/overlays. New overlays use ${overlay_id} as their path; the digit-only form is the only one created by the web app.

Invalid references are rejected:

Absolute paths such as /srv/overlay.
Parent traversal such as ../other or competitive/../../base.
Empty path components such as competitive//base.
Symlink escapes that resolve outside ${LEFT4ME_ROOT}/overlays.

The web app currently supports two overlay surfaces:

workshop overlays (user-owned) — populated by downloading .vpk files from the public Steam Web API into ${LEFT4ME_ROOT}/workshop_cache/{steam_id}.vpk and creating absolute symlinks under ${LEFT4ME_ROOT}/overlays/{overlay_id}/left4dead2/addons/{steam_id}.vpk.
script overlays — populated by an arbitrary user-authored bash script that runs inside bubblewrap + systemd-run --scope as the unprivileged l4d2-sandbox UID, with the overlay directory bind-mounted RW at /overlay. Resource caps: 1h walltime, 4 GB RAM, 512 tasks, 200% CPU, 20 GB post-build disk cap.

Both the caches and the overlay directories are owned by the left4me runtime user; if the web service ever runs as a different uid, ensure it shares a group with the host process and that both trees are group-readable.

Performance Tuning

The deployment ships a host-side perf baseline (slices, unit directives, sysctls). See docs/superpowers/specs/2026-05-09-l4d2-server-host-perf-baseline-design.md for design rationale.

The following knobs are documented escape hatches — they are not auto-applied. Apply only if you have measured a need and understand the failure modes.

CPU governor

The performance governor squeezes a few percent off jitter under bursty load. schedutil is acceptable for sustained UDP workloads.

sudo cpupower frequency-set -g performance

Persist via your distro's CPU-frequency tooling (e.g. /etc/default/cpufrequtils).

Per-instance CPU affinity

srcds is single-threaded per instance. On a multi-core host, pinning each instance to its own core can cut jitter under contention. Drop in /etc/systemd/system/left4me-server@<name>.service.d/affinity.conf:

[Service]
CPUAffinity=2

A reasonable strategy on an N-core host: leave core 0 for the kernel + IRQs + system services, then pin one instance per remaining core.

NIC tuning

Hardware-specific. On a host with a single primary interface (replace eth0):

sudo ethtool -G eth0 rx 4096 tx 4096
sudo ethtool -K eth0 gro on lro off

If you run a high instance count, also pin the NIC's interrupts off the cores that game servers occupy (see /proc/interrupts and /proc/irq/<n>/smp_affinity).

Real-time scheduling (advanced, opt-in)

Source-engine servers do not need real-time scheduling, and a misbehaving srcds at any RT priority can starve kernel threads — even with the default kernel.sched_rt_runtime_us=950000 throttling 5% of CPU back. Use only if you have a measured jitter problem that the baseline does not solve.

/etc/systemd/system/left4me-server@.service.d/realtime.conf:

[Service]
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=10
LimitRTPRIO=10

Applying changes to running servers

Unit-file changes do not apply to already-running services. After any change:

sudo systemctl daemon-reload
# Restart each game server via the web UI's stop + start, or:
sudo systemctl restart 'left4me-server@*.service'