docs(deploy): document network-shaping defaults + opt-in network knobs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
26f3d270b0
commit
9f0b51b455
1 changed files with 70 additions and 0 deletions
|
|
@ -78,6 +78,51 @@ The deployment ships a host-side perf baseline (slices, unit directives, sysctls
|
||||||
|
|
||||||
The following knobs are documented escape hatches — they are **not** auto-applied. Apply only if you have measured a need and understand the failure modes.
|
The following knobs are documented escape hatches — they are **not** auto-applied. Apply only if you have measured a need and understand the failure modes.
|
||||||
|
|
||||||
|
### Network shaping
|
||||||
|
|
||||||
|
The deploy ships three things that affect player-experience network behaviour:
|
||||||
|
|
||||||
|
1. **Per-flow marking.** `left4me-nft-mark.service` loads a small nftables
|
||||||
|
table (`inet left4me_mark`) that marks every UDP packet from uid `left4me`
|
||||||
|
with DSCP EF and `skb->priority` 6. srcds doesn't set these itself, so
|
||||||
|
without this rule its UDP is indistinguishable from any other flow.
|
||||||
|
2. **Sysctl baseline.** `99-left4me.conf` sets `udp_rmem_min=16384`,
|
||||||
|
`udp_wmem_min=16384`, `default_qdisc=fq_codel`, and
|
||||||
|
`tcp_congestion_control=bbr`. Reduces head-of-line blocking when bulk
|
||||||
|
TCP egress (backups, package fetches, web responses) coexists with
|
||||||
|
game UDP.
|
||||||
|
3. **CAKE egress shaping.** `left4me-cake.service` runs
|
||||||
|
`tc qdisc replace dev <iface> root cake bandwidth Xmbit internet
|
||||||
|
diffserv4 dual-dsthost` from `/etc/left4me/cake.env`. CAKE only shapes
|
||||||
|
if its declared bandwidth is **below** the real bottleneck, so set
|
||||||
|
`LEFT4ME_UPLINK_MBIT` to ≈95% of measured uplink:
|
||||||
|
|
||||||
|
sudoedit /etc/left4me/cake.env
|
||||||
|
# set LEFT4ME_UPLINK_MBIT=480 (or whatever ~95% of your uplink is)
|
||||||
|
sudo systemctl restart left4me-cake.service
|
||||||
|
|
||||||
|
`LEFT4ME_UPLINK_IFACE` is auto-detected from the IPv4 default route;
|
||||||
|
override only on hosts with multi-homed setups.
|
||||||
|
|
||||||
|
At idle 500 Mbit with no competing egress, CAKE shapes nothing — that's
|
||||||
|
expected, not a bug. The win materialises when bulk traffic on the
|
||||||
|
same uplink would otherwise bufferbloat the link the players share.
|
||||||
|
|
||||||
|
**Production hosts running `systemd-networkd`** should NOT use the
|
||||||
|
`left4me-cake.service` oneshot. Instead, configure the equivalent in the
|
||||||
|
matching `.network` file, which systemd-networkd reapplies across iface
|
||||||
|
lifecycle events:
|
||||||
|
|
||||||
|
# /etc/systemd/network/<your-uplink>.network
|
||||||
|
[CAKE]
|
||||||
|
Bandwidth=480M
|
||||||
|
OverheadKeyword=internet
|
||||||
|
PriorityQueueingPreset=diffserv4
|
||||||
|
EgressHostIsolation=yes
|
||||||
|
|
||||||
|
The nftables marking from (1) is qdisc-installer-agnostic and ships
|
||||||
|
unchanged on production.
|
||||||
|
|
||||||
### CPU governor
|
### CPU governor
|
||||||
|
|
||||||
The performance governor squeezes a few percent off jitter under bursty load. `schedutil` is acceptable for sustained UDP workloads.
|
The performance governor squeezes a few percent off jitter under bursty load. `schedutil` is acceptable for sustained UDP workloads.
|
||||||
|
|
@ -144,6 +189,31 @@ AmbientCapabilities=CAP_SYS_NICE
|
||||||
|
|
||||||
The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the service runs as `User=left4me` with `NoNewPrivileges=true`; without it some kernels/systemd combinations refuse to apply the RT policy.
|
The `AmbientCapabilities=CAP_SYS_NICE` line is needed because the service runs as `User=left4me` with `NoNewPrivileges=true`; without it some kernels/systemd combinations refuse to apply the RT policy.
|
||||||
|
|
||||||
|
### Additional opt-in network knobs
|
||||||
|
|
||||||
|
- **Ingress shaping via IFB.** Egress CAKE alone does not protect srcds
|
||||||
|
receive against ingress saturation (large workshop downloads, package
|
||||||
|
fetches arriving at line rate). One-liner:
|
||||||
|
|
||||||
|
sudo modprobe ifb && sudo ip link set ifb0 up
|
||||||
|
sudo tc qdisc add dev <uplink> handle ffff: ingress
|
||||||
|
sudo tc filter add dev <uplink> parent ffff: protocol ip u32 \
|
||||||
|
match u32 0 0 action mirred egress redirect dev ifb0
|
||||||
|
sudo tc qdisc add dev ifb0 root cake bandwidth Xmbit ingress \
|
||||||
|
diffserv4 dual-srchost
|
||||||
|
|
||||||
|
Worth flipping only when measurement shows ingress hurting receive.
|
||||||
|
|
||||||
|
- **`net.core.busy_poll = 50` / `net.core.busy_read = 50`.** Reduces UDP
|
||||||
|
receive median latency by polling for incoming packets briefly at
|
||||||
|
syscall boundaries. Cost: measurable CPU per syscall under load. Worth
|
||||||
|
flipping if a host is dedicated to game serving and CPU headroom is
|
||||||
|
plentiful.
|
||||||
|
|
||||||
|
- **`ethtool -K <iface> gro off`.** Some Source-engine ops disable
|
||||||
|
generic receive offload to avoid receive-side coalescing latency.
|
||||||
|
Hardware/driver dependent; document only.
|
||||||
|
|
||||||
### Applying changes to running servers
|
### Applying changes to running servers
|
||||||
|
|
||||||
Unit-file changes do not apply to already-running services. After any change:
|
Unit-file changes do not apply to already-running services. After any change:
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue