Systems That
Don’t Lie

Imagine you are running a GPU server that twenty different companies share. Each of them pays for a slice of the GPU’s time, and each sends inference requests for their language model workloads. Now imagine that one of those companies—call them the Flooder—sends requests thirty times faster than anyone else. Under a naive FIFO queue, the Flooder’s requests are almost always at the front. Every other tenant waits. In practice, this creates starvation: a paying customer who sends one carefully crafted prompt every ten seconds finds that prompt sitting for several minutes while the Flooder’s low-quality queries drain the budget.

When I measured this on real hardware during the kvwarden benchmarks—an H100 running Llama-3.1-70B with tensor parallelism across four cards—the starvation gap between the most-served and least-served tenant reached 523×. That is not a rounding error. It means the slowest tenant waited five hundred and twenty-three times longer per token than the fastest. In a real product, that means SLA breaches, chargebacks, and lost trust.

Why FIFO Fails at the Boundary

The failure is structural, not incidental. A FIFO queue has no notion of who owns what portion of shared capacity. It treats every request identically regardless of which tenant sent it, how many tokens they’ve consumed this minute, or what their fair share of the GPU’s KV cache ought to be. This is fine when tenants are statistically similar. It collapses the moment one tenant is even modestly more aggressive than the others.

Weighted Fair Queuing (WFQ) was designed for exactly this problem in networking—give each flow a weight, then drain the queue proportionally. But WFQ assumes cheap dequeue operations. An LLM token generation step is not cheap: it occupies the GPU for tens of milliseconds, and prefill for a long prompt can occupy it for hundreds. If you pick the wrong request to prefill, you’ve committed that capacity for the entire duration of that generation. Cancelation mid-sequence is expensive and often impossible with current inference engines.

Deficit Round Robin: Fairness with Memory

Deficit Round Robin (DRR) is the practical answer. The insight is simple: instead of trying to schedule every request perfectly, maintain a running deficit counter for each tenant. When a tenant’s request is served, subtract its cost (in tokens or GPU time) from their quantum. If they still have budget left, they can go again immediately. If they’ve exceeded their quantum, they go to the back and their deficit carries forward to the next round.

# Pseudocode: DRR admission loop
for tenant in round_robin_order:
    tenant.deficit += quantum
    while queue[tenant] and tenant.deficit >= cost(queue[tenant].head):
        req = dequeue(tenant)
        tenant.deficit -= cost(req)
        admit(req)

The deficit counter is the key. A tenant who was blocked last round because their request was expensive gets credit carried forward. A flooder who submitted twenty cheap requests doesn’t starve others; they simply burn through their quantum faster and wait longer for the next round. Over time, every tenant gets exactly their fair share of GPU capacity, regardless of how aggressively any one of them submits.

Token-Bucket Admission: Smoothing the Bursts

DRR handles fairness at the scheduling layer, but there’s a second problem: admission. A tenant can overwhelm the queue before DRR even gets a chance to enforce fairness, simply by filling the request buffer faster than the GPU drains it. The solution is a per-tenant token-bucket rate limiter at the ingress layer.

Each tenant gets a bucket with a capacity B and a refill rate r tokens per second. Every incoming request costs tokens proportional to its estimated prompt length and desired output tokens. If the bucket has enough tokens, the request is admitted. If not, it is either queued (with backpressure) or rejected with a 429. The bucket refills continuously, so a tenant who sends one large request and then waits accumulates tokens for the next one; a flooder who hammers the endpoint runs dry and gets throttled.

Token buckets are not about punishment. They are about giving every tenant a guaranteed floor of service, regardless of what their neighbors are doing. That guarantee is what makes multi-tenancy viable as a product. — kvwarden design notes

Per-Tenant TTFT: The Metric That Tells the Truth

Time-to-first-token (TTFT) is the single most user-visible latency metric for streaming LLM inference. It measures how long a user waits from sending their request to seeing the first token of the response. Everything that happens in prefill (attending over the prompt, building the KV cache) shows up in TTFT. Queuing delay shows up in TTFT. Admission waiting shows up in TTFT.

The mistake most systems make is reporting TTFT as a global average. That hides the distribution across tenants entirely. If one tenant has P50 TTFT of 120ms and another has P99 TTFT of 8 seconds, the global average might look fine. kvwarden tracks TTFT as a per-tenant histogram—separate P50, P95, P99 for every tenant, streamed to Grafana in real time. This makes fairness violations visible immediately. If DRR is working, all tenants’ TTFT distributions should look roughly similar. If one is an order of magnitude worse, something is wrong.

The inference infrastructure conversation has a gravitational pull toward complexity. Every time someone wants to serve a large language model, the default recommendation is: containerize the model server, deploy it on Kubernetes, add a load balancer, wire up a service mesh, and attach a monitoring stack. This is reasonable at scale. It is completely unreasonable for a team of two running on one H100.

kvwarden was built on a deliberate constraint: one GPU, one machine, no Kubernetes. The entire system runs as a Docker Compose stack. This is not a temporary compromise waiting to be replaced by a “proper” Kubernetes deployment. It is a design choice with specific benefits that Kubernetes cannot match at this scale.

The Case Against Premature Distribution

Distribution solves availability and scale. It introduces latency, complexity, and failure modes that do not exist on a single machine. Network hops between the router and the inference engine add latency. The Kubernetes API server is another system that can fail. Service discovery is another moving part. etcd is another thing to back up and restore. None of these exist on a single machine.

More importantly, the failure surface of a single machine is well-understood and predictable. When something goes wrong, you read the logs, inspect the process, and fix it. You do not trace a request through five hops of network infrastructure, wondering whether the problem is in the sidecar, the ingress controller, the load balancer, or the model server itself.

# kvwarden docker-compose.yml (simplified)
services:
  router:
    image: kvwarden:latest
    ports: ["8080:8080"]
    volumes: ["./config.toml:/etc/kvwarden/config.toml"]
    depends_on: [vllm]

  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - MODEL=meta-llama/Llama-3.1-70B-Instruct
      - TENSOR_PARALLEL_SIZE=4
    deploy:
      resources:
        reservations:
          devices: [{driver: nvidia, count: all, capabilities: [gpu]}]

  prometheus:
    image: prom/prometheus:latest
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml"]

  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]

The Streaming Router

The most technically interesting component in the single-machine stack is the streaming router. Its job is to sit between the client and the inference engine, enforce admission control, assign tenant identities to requests, and pass the response stream back to the client with minimal overhead.

Getting this right requires attention to three subtle invariants:

• Admission slot lifetime. An admission slot is opened when a request passes the token-bucket check and enters the DRR queue. It must be held until the last token of the response is sent, not just until generation starts. If you release the slot on first token, you can admit a new request before the GPU has actually freed its KV cache—causing memory pressure you didn’t account for.
• Real TTFT measurement. TTFT should be measured at the client, or at a proxy that sits close to the client. If you measure it at the model server, you exclude network transit time and router queuing time, which are often the dominant contributors for throttled tenants. kvwarden measures TTFT from request arrival at the router to first byte forwarded to the client.
• Max-stream-duration fence. A runaway generation can hold an admission slot indefinitely. The router enforces a hard maximum stream duration per request, after which it terminates the connection and releases the slot. This is a safety valve, not a latency guarantee.

Why One Command Matters

The goal of the one-command Docker Compose bundle was not cleverness. It was reproducibility. A researcher who wants to benchmark fairness algorithms should not spend a day configuring infrastructure. They should run one command, wait for the stack to come up, and start sending requests. The eval bundle includes pre-configured Grafana dashboards, a Prometheus scrape config, and a bench harness that exercises multi-tenant load patterns out of the box.

Most software engineers encounter latency as an occasional irritant: a slow API call, a database query that takes 200ms instead of 10. In high-frequency trading infrastructure, latency is the fundamental physics of the system. A signal that arrives 2 microseconds late is worth zero. A signal that arrives 1.9 microseconds after a competitor’s signal is worth less than zero—it burns a trade that your competitor is already exiting.

Building at this layer requires unlearning almost everything that makes software tractable in other domains. Dynamic memory allocation is forbidden. Locks are forbidden. Virtual dispatch is forbidden. Exceptions are forbidden. What remains is a carefully constrained subset of C++ where every instruction has a predictable cost, every cache line access is deliberate, and every nanosecond is accounted for.

The Lock-Free SPSC Queue

The workhorse of low-latency inter-thread communication is the single-producer single-consumer (SPSC) lock-free ring buffer. It is the simplest data structure that allows two threads to exchange data without ever contending on a mutex. The producer writes to a head index; the consumer reads from a tail index. As long as only one thread writes and one reads, no atomic operations are needed beyond the head and tail indices themselves.

// Cache-line aligned SPSC ring buffer
template<typename T, size_t N>
struct alignas(64) SPSCQueue {
    static_assert((N & (N-1)) == 0, "N must be power of 2");

    alignas(64) std::atomic<size_t> head_{0};
    alignas(64) std::atomic<size_t> tail_{0};
    T slots_[N];

    bool try_push(const T& val) noexcept {
        auto h = head_.load(std::memory_order_relaxed);
        if (h - tail_.load(std::memory_order_acquire) == N)
            return false;  // full
        slots_[h & (N-1)] = val;
        head_.store(h + 1, std::memory_order_release);
        return true;
    }

    bool try_pop(T& val) noexcept {
        auto t = tail_.load(std::memory_order_relaxed);
        if (head_.load(std::memory_order_acquire) == t)
            return false;  // empty
        val = slots_[t & (N-1)];
        tail_.store(t + 1, std::memory_order_release);
        return true;
    }
};

The alignas(64) annotation is not decorative. A cache line is 64 bytes on x86. If the head and tail indices share a cache line, every update to the head (by the producer) invalidates the cache line for the consumer, and vice versa. This is called false sharing, and it can turn a theoretically lock-free structure into something with the same throughput characteristics as a mutex. Keeping head and tail on separate cache lines eliminates this pathology entirely.

RDMA: Bypassing the Kernel

The kernel network stack is fast for most purposes. For HFT signal dissemination, it is prohibitively slow. The overhead comes from context switches (user-space to kernel-space and back), memory copies (from kernel buffer to user buffer), and interrupt handling. Each of these adds microseconds. In a signal pipeline, you want the data to move from one machine’s memory to another machine’s memory with no kernel involvement at all.

RDMA (Remote Direct Memory Access) provides exactly this. With RDMA, a NIC can read from or write to a remote machine’s memory directly, without the remote CPU being involved and without touching the kernel network stack on either end. Round-trip latency drops from ~10 microseconds (kernel TCP) to ~1–2 microseconds (RDMA).

Paravirtual NVMe: When Disk Is in the Latency Budget

Most HFT systems treat persistent storage as something that happens at the end of the day, during batch settlement. But there are signal types—particularly those derived from order-book state reconstruction or regulatory audit logging—where you need durability in the critical path. This is where paravirtual NVMe becomes relevant.

FastLane_NVMe is a paravirtual NVMe driver with RDMA hooks written in Rust. The key insight is that a modern NVMe SSD can sustain <100 microsecond write latency when accessed directly through a paravirtual device—bypassing the host OS’s block layer entirely. Combined with FPGA-ready hooks for signal preprocessing, this creates a persistent write path that does not blow the latency budget for the overall signal pipeline.

Every abstraction layer has a latency cost. In low-latency systems, the discipline is knowing which abstraction layers you can afford and which you cannot. The kernel network stack is a layer you cannot afford. A carefully designed paravirtual device layer might be. — FastLane_NVMe design notes

We are in the middle of a transition that happens once or twice per generation in computing: a new computational primitive is becoming so common that the existing substrate—the operating system—can no longer represent it well. The last time this happened was with threads. Before threads became a first-class kernel primitive, userspace libraries tried to simulate them. The result was brittle, non-portable, and impossible to reason about under preemption. Once the kernel absorbed threads (POSIX pthreads, then native kernel threads), scheduling became predictable, isolation became real, and the entire ecosystem could build on a reliable foundation.

Agents are in the same position today that threads were in 1990. They exist everywhere. They are simulated with userspace libraries, daemon processes, API calls, and LLM endpoints. The operating system knows nothing about them. From the kernel’s perspective, an agent is just a process making network calls, which is indistinguishable from a web server or a database client. There is no first-class representation of agent identity, capability, or lifecycle.

What POSIX Cannot Express

Consider what you need to safely run an autonomous agent on a real system:

• Capability confinement. The agent should be able to read specific files, make specific network calls, and invoke specific tools—nothing else. POSIX can approximate this with filesystem permissions and seccomp filters, but the granularity is wrong: permissions are file-based, not intent-based, and seccomp operates on syscall numbers rather than on the agent’s declared capabilities.
• Auditable action history. Every action the agent takes—every file read, every network request, every tool invocation—should be recorded in an append-only log that the agent cannot tamper with. There is no kernel mechanism for this today. You can log at the application layer, but an adversarial agent could simply skip the logging call.
• Compositional identity. Agents spawn sub-agents. Sub-agents spawn tools. A tool call in a deep subagent tree should carry the identity of every agent in its lineage so the system can reason about who delegated what capability to whom. POSIX process IDs are flat—a child process has one parent PID and that’s the entire ancestry.

The Coconut OS Thesis

Coconut OS is the thesis that these problems should be solved at the kernel layer, not the application layer. An agent-native Linux distribution where agents are first-class kernel primitives—with their own scheduling class, their own capability model, and kernel-enforced audit logging from the moment of creation.

The practical starting point is a new kernel subsystem that manages agent namespaces. Just as Linux namespaces (pid, mount, network, user) isolate different views of system resources, an agent namespace isolates the agent’s view of its own capabilities. When an agent is instantiated, it is assigned an agent namespace that specifies exactly which tools it can call, which filesystem paths it can access, and which network endpoints it can reach. Attempts to exceed this namespace are blocked at the syscall boundary, not by the agent’s own code.

# Conceptual agent namespace definition
[agent.namespace]
id     = "research-agent-a3f7"
parent = "orchestrator-b2c1"    # lineage chain

[capabilities]
filesystem.read  = ["/data/corpus/**", "/tmp/scratch/**"]
filesystem.write = ["/tmp/scratch/**"]
network.allowed  = ["api.anthropic.com:443"]
tools.invoke     = ["web_search", "read_file", "write_file"]
tools.spawn      = ["sub-agent"]

[audit]
log_destination  = "/kernel/audit/agents/a3f7.log"
tamper_resistant = true   # kernel enforces, agent cannot skip

This is not science fiction. The underlying kernel mechanisms exist: LSM hooks (Linux Security Modules) provide the interception points; eBPF programs can enforce complex policies without kernel modifications; io_uring provides the high-throughput async I/O path that agents need for parallel tool execution. What does not yet exist is the coherent abstraction that ties these together under a unified “agent primitive.”

Observability tools are only as good as the data they are trained on and validated against. If you test your anomaly detection system on synthetic logs that look nothing like production logs, you will tune your model for the wrong distribution and discover this at the worst possible moment: when there is a real incident and your detector is silent.

Most synthetic log generators fail in the same way: they produce independent samples from some distribution (usually a simple Poisson for arrival times, Gaussian for metric values). Real production logs have structure that these generators cannot produce: autocorrelation, diurnal patterns, cascading failures, and service-to-service dependencies. A Poisson generator can produce the right average request rate; it cannot produce the bursty, correlated, heavy-tailed patterns that actually stress observability systems.

Autocorrelation: The Property Most Generators Miss

If your API handles 1,000 requests per second at 2pm, it almost certainly handles close to 1,000 requests per second at 2:00:01pm. This is autocorrelation: the value at time t is predictable from values at nearby times. A Poisson process has zero autocorrelation by definition. Real traffic has very high autocorrelation over short windows and periodic autocorrelation over 24-hour windows (diurnal pattern).

The right model is AR(1): an autoregressive process of order 1. The value at time t is a weighted combination of the value at time t-1 and a noise term:

x_t = phi * x_{t-1} + epsilon_t,  epsilon_t ~ N(0, sigma^2)

# In practice:
phi   = 0.92          # high autocorrelation
sigma = 0.08          # small perturbations
x_0   = baseline_rps  # anchored to real baseline

With phi = 0.92, the series has strong memory: a spike at time t decays slowly over the next several time steps rather than returning immediately to the mean. This matches the “traffic tsunami” pattern common in production: a celebrity tweet or a product launch sends traffic to 10× normal, and it stays elevated for minutes or hours while the event plays out, rather than spiking and immediately subsiding.

Cascading Failures: Where Anomaly Detectors Go to Die

The hardest class of incidents to detect is the cascading failure: a downstream service becomes slow, which causes upstream callers to accumulate open connections, which exhausts their connection pools, which makes them slow, which causes their upstream callers to do the same thing. The signature in the logs is subtle at first: a small latency increase in one service, followed minutes later by latency increases in the services that depend on it, followed by error spikes once queues fill.

Generating realistic cascade signatures requires a dependency graph and a propagation model. The hyper-realistic-synthetic-logs-generator encodes service dependencies explicitly, then simulates failure propagation: when a leaf service fails, add a configurable delay to all callers’ latency distributions, then propagate the increased latency (and eventual errors) up the tree with realistic time offsets.

Calibration: KL Divergence as Your Acceptance Test

A synthetic log generator is only useful if the logs it produces are statistically similar to real production logs. The right validation metric is KL divergence (Kullback-Leibler divergence) between the distribution of each metric in the synthetic data and in real data.

KL divergence measures how much information is lost when you use the synthetic distribution to approximate the real distribution. A value of zero means they are identical. In practice, you want KL divergence below a threshold you set based on your use case—for anomaly detection training, a good target is KL < 0.1 across all metric distributions.

from scipy.stats import entropy
import numpy as np

def kl_divergence(real: np.ndarray, synth: np.ndarray, bins=100):
    real_hist, edges = np.histogram(real, bins=bins, density=True)
    synth_hist, _    = np.histogram(synth, bins=edges, density=True)
    eps = 1e-10
    return entropy(real_hist + eps, synth_hist + eps)

# Acceptance gate
for metric_name, (real, synth) in metrics.items():
    kl = kl_divergence(real, synth)
    assert kl < 0.1, f"{metric_name}: KL={kl:.4f} exceeds threshold"

If calibration fails, the generator’s parameters need tuning: adjust phi for autocorrelation, scale factors for diurnal amplitude, or the failure injection probability. Treat calibration as a continuous feedback loop, not a one-time check. As your production traffic patterns evolve, your synthetic generator parameters need to evolve with them.