Systems That
Don’t Lie

Twenty different companies share one GPU server. Each pays for a slice of the GPU’s time, and each sends inference requests for their own language model workloads. One of them, call them the flooder, sends requests thirty times faster than anyone else. Under a naive FIFO queue the flooder’s requests are almost always at the front, so every other tenant waits. A quiet tenant who sends one carefully written prompt every ten seconds finds it sitting in the queue while the flooder holds the GPU. That is starvation.

I measured this on real hardware in the kvwarden benchmarks: one A100 running Llama-3.1-8B on vLLM 0.19.1, with a flooder and a quiet tenant sharing the engine for 300 seconds. Solo, the quiet tenant’s median time-to-first-token was 53.9 ms. Under FIFO with the flooder active, it was 1,585 ms: a 29× starvation gap at steady state. (An earlier run on vLLM 0.8.5 measured 523×; the v1 continuous batcher in newer engines absorbs cold-start backpressure, so 29× is the honest steady-state figure.) With a per-tenant admission gate in front of the same load, the quiet tenant’s post-warmup TTFT came back to 61.5 ms (n=311), 1.14× the solo baseline and a 26× improvement over FIFO. Read the gap from the quiet tenant’s seat: 1,585 ms of waiting for a prompt that costs 53.9 ms when nobody else is on the box.

Where FIFO breaks

The failure is structural, not incidental. A FIFO queue has no notion of who owns what portion of shared capacity. It treats every request identically regardless of which tenant sent it, how many tokens they’ve consumed this minute, or what their share of the GPU’s KV cache ought to be. That is fine while tenants are statistically similar. It collapses the moment one of them is even modestly more aggressive than the rest.

Weighted Fair Queuing (WFQ) was designed for this problem in networking: give each flow a weight, then drain the queue proportionally. WFQ assumes cheap dequeue operations. An LLM token generation step is not cheap. It occupies the GPU for tens of milliseconds, and prefill for a long prompt can occupy it for hundreds. Pick the wrong request to prefill and you have committed that capacity for the whole generation. Cancellation mid-sequence is expensive and often impossible with current inference engines.

The deficit counter

Deficit Round Robin (DRR) is the practical answer. Instead of scheduling every request perfectly, keep a running deficit counter per tenant. When a tenant’s request is served, subtract its cost (in tokens or GPU time) from their quantum. If budget remains, they go again immediately. If they have overrun the quantum, they go to the back and the deficit carries forward to the next round.

# Pseudocode: DRR admission loop
for tenant in round_robin_order:
    tenant.deficit += quantum
    while queue[tenant] and tenant.deficit >= cost(queue[tenant].head):
        req = dequeue(tenant)
        tenant.deficit -= cost(req)
        admit(req)

The carry is the whole trick. A tenant blocked last round because their request was expensive keeps the credit. A flooder who submits twenty cheap requests does not starve anyone; they burn through their quantum faster and wait longer for the next round. Across rounds the split converges on each tenant’s share, and submitting harder stops buying position in the queue.

Admission comes first

DRR handles fairness at the scheduling layer. Admission is a second problem. A tenant can swamp the queue before DRR gets a chance to enforce anything, by filling the request buffer faster than the GPU drains it. A per-tenant token bucket at ingress holds that line.

Each tenant gets a bucket with a capacity B and a refill rate r tokens per second. Every incoming request costs tokens proportional to its estimated prompt length and desired output tokens. If the bucket has enough tokens, the request is admitted. If not, it is either queued (with backpressure) or rejected with a 429. The bucket refills continuously, so a tenant who sends one large request and then waits accumulates tokens for the next one; a flooder who hammers the endpoint runs dry and gets throttled.

A token bucket is not a punishment. It is a floor of service every tenant keeps regardless of what their neighbours are doing. That floor is what makes multi-tenancy viable as a product. kvwarden design notes

TTFT, per tenant

Time-to-first-token (TTFT) is the wait a user of a streaming endpoint sees: from sending the request to the first token coming back. Everything in prefill (attending over the prompt, building the KV cache) shows up in TTFT. Queuing delay shows up in TTFT. Admission waiting shows up in TTFT.

Most systems report TTFT as a global average, which hides the distribution across tenants. If one tenant has P50 TTFT of 120 ms and another has P99 TTFT of 8 seconds, the average still looks fine. kvwarden tracks TTFT as a per-tenant histogram: separate P50, P95, P99 for every tenant, streamed to Grafana in real time. A fairness violation is then visible while it is happening. If DRR is working, the tenants’ distributions sit close together. If one is an order of magnitude worse, something is wrong.

Ask how to serve a large language model and the answer comes back the same way every time: containerize the model server, deploy it on Kubernetes, add a load balancer, wire up a service mesh, attach a monitoring stack. That answer is right for a fleet. For a team of two on a single GPU, it is wrong.

I built kvwarden on a deliberate constraint: one GPU, one machine, no Kubernetes. The entire system runs as a Docker Compose stack. This is not a temporary compromise waiting to be replaced by a “proper” Kubernetes deployment. It is the design.

What distribution costs

Distribution solves availability and scale. It charges latency, complexity, and failure modes that do not exist on one machine. Network hops between the router and the inference engine add latency. The Kubernetes API server is another system that can fail. Service discovery is another moving part. etcd is another thing to back up and restore. None of that exists on a single machine.

A single machine also fails in ways you can hold in your head. Something breaks, you read the logs, you look at the process, you fix it. You are not tracing a request through five hops of network infrastructure, wondering whether the problem is the sidecar, the ingress controller, the load balancer, or the model server.

# kvwarden docker-compose.yml (simplified)
services:
  router:
    image: kvwarden:latest
    ports: ["8080:8080"]
    volumes: ["./config.toml:/etc/kvwarden/config.toml"]
    depends_on: [vllm]

  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - MODEL=meta-llama/Llama-3.1-70B-Instruct
      - TENSOR_PARALLEL_SIZE=4
    deploy:
      resources:
        reservations:
          devices: [{driver: nvidia, count: all, capabilities: [gpu]}]

  prometheus:
    image: prom/prometheus:latest
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml"]

  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]

The streaming router

The router is the unglamorous piece in the middle. It sits between the client and the inference engine, enforces admission control, assigns tenant identities to requests, and passes the response stream back to the client with minimal overhead.

Three invariants have to hold:

• Admission slot lifetime. An admission slot is opened when a request passes the token-bucket check and enters the DRR queue. It must be held until the last token of the response is sent, not released when generation starts. Release the slot on first token and you can admit a new request before the GPU has freed its KV cache, which is memory pressure nobody budgeted for.
• Real TTFT measurement. TTFT should be measured at the client, or at a proxy that sits close to the client. Measure it at the model server and you drop network transit time and router queuing time, which are often the dominant terms for a throttled tenant. kvwarden measures TTFT from request arrival at the router to first byte forwarded to the client.
• Max-stream-duration fence. A runaway generation can hold an admission slot indefinitely. The router enforces a hard maximum stream duration per request, then terminates the connection and releases the slot. That is a safety valve, not a latency guarantee.

One command, on purpose

The one-command Docker Compose bundle is not cleverness. It is reproducibility. Someone who wants to benchmark fairness algorithms should not spend a day configuring infrastructure first. They should run one command, wait for the stack to come up, and start sending requests. The eval bundle ships pre-configured Grafana dashboards, a Prometheus scrape config, and a bench harness that exercises multi-tenant load patterns.

Most engineers meet latency as an occasional irritant: a slow API call, a database query that takes 200ms instead of 10. In high-frequency trading infrastructure, latency is the budget everything else is spent against. Numbers illustrative: a signal that arrives 2 microseconds late is worth zero. A signal that arrives 1.9 microseconds after a competitor’s signal is worth less than zero, because it burns a trade the competitor is already exiting.

Working here means giving up most of what makes software tractable elsewhere. Dynamic memory allocation is forbidden. Locks are forbidden. Virtual dispatch is forbidden. Exceptions are forbidden. What is left is a subset of C++ where every instruction has a cost you can predict before you run it.

The lock-free SPSC queue

The core structure for low-latency inter-thread communication is the single-producer single-consumer (SPSC) lock-free ring buffer. It is the simplest way for two threads to exchange data without ever contending on a mutex. The producer writes to a head index, the consumer reads from a tail index. As long as only one thread writes and one reads, no atomic operations are needed beyond the head and tail indices themselves.

// Cache-line aligned SPSC ring buffer
template<typename T, size_t N>
struct alignas(64) SPSCQueue {
    static_assert((N & (N-1)) == 0, "N must be power of 2");

    alignas(64) std::atomic<size_t> head_{0};
    alignas(64) std::atomic<size_t> tail_{0};
    T slots_[N];

    bool try_push(const T& val) noexcept {
        auto h = head_.load(std::memory_order_relaxed);
        if (h - tail_.load(std::memory_order_acquire) == N)
            return false;  // full
        slots_[h & (N-1)] = val;
        head_.store(h + 1, std::memory_order_release);
        return true;
    }

    bool try_pop(T& val) noexcept {
        auto t = tail_.load(std::memory_order_relaxed);
        if (head_.load(std::memory_order_acquire) == t)
            return false;  // empty
        val = slots_[t & (N-1)];
        tail_.store(t + 1, std::memory_order_release);
        return true;
    }
};

The alignas(64) annotation is not decorative. A cache line is 64 bytes on x86. If the head and tail indices share a cache line, every update to the head (by the producer) invalidates the line for the consumer, and the reverse. That is false sharing, and it can leave you with a mutex wearing a lock-free costume. Put head and tail on their own cache lines and it goes away.

Bypassing the kernel

The kernel network stack is fast for most purposes. For HFT signal dissemination it is too slow. The overhead comes from context switches (user space to kernel space and back), memory copies (kernel buffer to user buffer), and interrupt handling. Each of those adds microseconds. What a signal pipeline wants is data moving from one machine’s memory to another machine’s memory with no kernel in the path.

RDMA (Remote Direct Memory Access) is that path. A NIC can read from or write to a remote machine’s memory directly, without the remote CPU being involved and without touching the kernel network stack on either end. Round-trip latency drops from ~10 microseconds (kernel TCP) to ~1 to 2 microseconds (RDMA).

When disk is in the budget

Most HFT systems treat persistent storage as an end-of-day concern, handled during batch settlement. Some signals do not allow that, particularly ones derived from order-book state reconstruction or regulatory audit logging, where durability sits in the critical path. That is where paravirtual NVMe comes in.

FastLane_NVMe is a paravirtual NVMe driver with RDMA hooks, written in Rust. A modern NVMe SSD can sustain <100 microsecond write latency when it is reached directly through a paravirtual device, with the host OS block layer out of the path. With FPGA-ready hooks for signal preprocessing on top, the persistent write stays inside the latency budget of the pipeline around it.

Every abstraction layer has a latency cost. The discipline is knowing which layers you can afford and which you cannot. The kernel network stack is one you cannot afford. A carefully designed paravirtual device layer might be one you can. FastLane_NVMe design notes

The last time a computational primitive outgrew the operating system, it was threads. Before threads became a first-class kernel primitive, userspace libraries simulated them. The result was brittle, non-portable, and impossible to reason about under preemption. Then the kernel absorbed them: POSIX pthreads, then native kernel threads. Scheduling became predictable and isolation became real, because the kernel was finally holding the thing everyone was already using.

Agents are in the same position today that threads were in 1990. They exist everywhere. They are simulated with userspace libraries, daemon processes, API calls, and LLM endpoints. The operating system knows nothing about them. From the kernel’s side, an agent is a process making network calls, indistinguishable from a web server or a database client. There is no first-class representation of agent identity, capability, or lifecycle.

What POSIX cannot say

Three things an autonomous agent needs on a real machine:

• Capability confinement. The agent should be able to read specific files, make specific network calls, and invoke specific tools. Nothing else. POSIX approximates this with filesystem permissions and seccomp filters, and the granularity is wrong: permissions are file-based, not intent-based, and seccomp operates on syscall numbers rather than on the agent’s declared capabilities.
• Auditable action history. Every action the agent takes, every file read, every network request, every tool invocation, should land in an append-only log the agent cannot tamper with. No kernel mechanism does this today. You can log at the application layer, and an adversarial agent can skip the logging call.
• Compositional identity. Agents spawn sub-agents. Sub-agents spawn tools. A tool call in a deep subagent tree should carry the identity of every agent in its lineage, so the system can reason about who delegated what capability to whom. POSIX process IDs are flat: a child process has one parent PID and that is the entire ancestry.

Where Coconut OS sits

Coconut OS is our bet that these belong at the kernel layer, not the application layer: an agent-native Linux distribution where agents are first-class kernel primitives, with their own scheduling class, their own capability model, and kernel-enforced audit logging from the moment of creation. Coconut OS today is a set of written specifications and spec-phase prototypes, not a shipping distribution.

The practical starting point is a kernel subsystem that manages agent namespaces. Linux namespaces (pid, mount, network, user) isolate different views of system resources. An agent namespace isolates the agent’s view of its own capabilities. When an agent is instantiated it is assigned one, and that namespace names which tools it can call, which filesystem paths it can reach, and which network endpoints it can open. Attempts to exceed it are blocked at the syscall boundary, not by the agent’s own code.

# Conceptual agent namespace definition
[agent.namespace]
id     = "research-agent-a3f7"
parent = "orchestrator-b2c1"    # lineage chain

[capabilities]
filesystem.read  = ["/data/corpus/**", "/tmp/scratch/**"]
filesystem.write = ["/tmp/scratch/**"]
network.allowed  = ["api.anthropic.com:443"]
tools.invoke     = ["web_search", "read_file", "write_file"]
tools.spawn      = ["sub-agent"]

[audit]
log_destination  = "/kernel/audit/agents/a3f7.log"
tamper_resistant = true   # kernel enforces, agent cannot skip

The kernel mechanisms are already there. LSM hooks (Linux Security Modules) provide the interception points, eBPF programs can enforce complex policies without kernel modifications, and io_uring provides the high-throughput async I/O path agents need for parallel tool execution. What is missing is the abstraction that ties them together under one “agent primitive.”

Somewhere right now an anomaly detector is being tuned against synthetic logs that look nothing like the traffic it will see in production. It will pass every test in front of it. Then it will sit silent through the incident it was bought for, because it was fitted to the wrong distribution.

Most synthetic log generators fail the same way: they produce independent samples from some distribution, usually Poisson for arrival times and Gaussian for metric values. Real production logs carry structure those generators cannot make: autocorrelation, diurnal patterns, cascading failures, service-to-service dependencies. A Poisson generator gets the average request rate right. It cannot produce the bursty, correlated, heavy-tailed patterns that stress an observability system.

What the generators miss

An API that handles 1,000 requests per second at 2pm almost certainly handles close to 1,000 requests per second at 2:00:01pm. That is autocorrelation: the value at time t is predictable from values at nearby times. A Poisson process has zero autocorrelation by definition. Real traffic has high autocorrelation over short windows and periodic autocorrelation over 24-hour windows, the diurnal pattern.

An AR(1) process, autoregression of order 1, puts the memory back. The value at time t is a weighted combination of the value at time t-1 and a noise term:

x_t = phi * x_{t-1} + epsilon_t,  epsilon_t ~ N(0, sigma^2)

# In practice:
phi   = 0.92          # high autocorrelation
sigma = 0.08          # small perturbations
x_0   = baseline_rps  # anchored to real baseline

With phi = 0.92 the series has strong memory: a spike at time t decays over the next several steps instead of snapping back to the mean. That is how real spikes behave. A celebrity tweet or a product launch sends traffic to 10× normal, and it stays elevated for minutes or hours while the event plays out, rather than spiking and immediately subsiding.

The cascade

The hardest class of incident to detect is the cascading failure: a downstream service goes slow, upstream callers accumulate open connections, their connection pools exhaust, they go slow, and their own callers do the same thing. The signature in the logs starts small. A modest latency increase in one service, then minutes later latency increases in the services that depend on it, then error spikes once the queues fill.

Generating a realistic cascade signature takes a dependency graph and a propagation model. The hyper-realistic-synthetic-logs-generator encodes service dependencies explicitly, then simulates failure propagation: when a leaf service fails, add a configurable delay to every caller’s latency distribution, then propagate the increased latency (and eventual errors) up the tree with realistic time offsets.

The calibration gate

A synthetic log generator earns its keep only if the logs it produces are statistically close to real production logs. The check is a KL divergence (Kullback-Leibler divergence) between the distribution of each metric in the synthetic data and in the real data.

KL divergence measures how much information is lost when the synthetic distribution stands in for the real one. A value of zero means the two are identical. The threshold you accept depends on what the data is for. For anomaly detection training, the target is KL < 0.1 across all metric distributions.

from scipy.stats import entropy
import numpy as np

def kl_divergence(real: np.ndarray, synth: np.ndarray, bins=100):
    real_hist, edges = np.histogram(real, bins=bins, density=True)
    synth_hist, _    = np.histogram(synth, bins=edges, density=True)
    eps = 1e-10
    return entropy(real_hist + eps, synth_hist + eps)

# Acceptance gate
for metric_name, (real, synth) in metrics.items():
    kl = kl_divergence(real, synth)
    assert kl < 0.1, f"{metric_name}: KL={kl:.4f} exceeds threshold"

When the gate fails, the generator is what changes: phi for autocorrelation, the scale factor for diurnal amplitude, the failure injection probability. Run it again whenever production drifts, because a calibration from last quarter is a claim about last quarter.