Dissecting TCP BBR and Pacing Queue Dynamics on High-BDP Cross-Border Links

On long-fat networks (LFNs) where the base round-trip time (RTT) exceeds 140–160 ms and shallow policing queues sit at autonomous system (AS) boundaries, loss-based congestion control algorithms such as CUBIC systematically underperform. Even a 0.2% random packet drop triggers a multiplicative window reduction, collapsing throughput to a fraction of the bottleneck link capacity.

While migrating our edge nodes to Linux 6.12 with BBR and Fair Queue (sch_fq) pacing solved raw throughput collapse, interactive workloads—such as multiplexed HTTP/2 streams and token-streaming RPC calls—still suffered from periodic latency spikes. Here is a breakdown of the kernel buffer and pacing interactions responsible, and the exact sysctl profile we use in production.

1. The Hidden Cost of tcp_slow_start_after_idle

Interactive request-response workloads rarely saturate a TCP flow continuously. Instead, a client issues a burst of requests, waits 5–15 seconds while rendering or awaiting user input, and then transmits another burst over the same persistent TLS connection.

With the Linux default net.ipv4.tcp_slow_start_after_idle = 1, any idle gap exceeding one retransmission timeout (RTO) resets the congestion window (cwnd) back toward the initial window (IW10). On a 148 ms RTT link, ramping back up to bandwidth-delay product (BDP) equilibrium costs several extra round trips:

# Inspect live BBR state and pacing rate per socket
ss -tin dst 103.97.201.0/24

# Prevent cwnd collapse between bursty interactive requests
sysctl -w net.ipv4.tcp_slow_start_after_idle=0

2. Taming Socket Send Bufferbloat with tcp_notsent_lowat

When an application writes data to a non-blocking TCP socket faster than the bottleneck pacing rate drains it, the kernel accumulates unsent bytes in the socket write queue behind already-in-flight segments. If the socket buffer auto-tunes up to 4 MB during a bulk transfer, newly enqueued high-priority framing (such as an HTTP/2 PING or a parallel stream request) waits behind megabytes of unsent payload in user-to-kernel queue space.

Setting net.ipv4.tcp_notsent_lowat = 131072 (128 KB) instructs the kernel not to wake EPOLLOUT once unsent data in the write queue exceeds 128 KB, keeping the socket write buffer lean while letting BBR maintain full in-flight BDP on the wire.

3. Complete Production Sysctl Profile

cat > /etc/sysctl.d/99-bbr-lowlatency.conf <<'EOF'
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_notsent_lowat = 131072
net.ipv4.tcp_mtu_probing = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
EOF
sysctl --system

Combined with clean TLS 1.3 session resumption and eliminating unnecessary userspace TLS double-wrapping, this profile keeps p99 time-to-first-byte within 3–5% of the theoretical 1.5×RTT handshake floor.