SOZENTA

Trevi — infrastructure that keeps the inference stream alive

Trevi keeps connections, GPU memory, and in-flight request state alive outside the application process — so a crashed, killed, or upgraded process is a brief pause on an open stream, never a dropped one.
50 / 50 ✓
in-flight streams survive SIGKILL, byte-perfect
measured on deployed cloud GPU stacks · n=50 per scenario · 0 failures
0 ✓
client reconnects, across every kill tested
same TCP socket before & after · no retry logic needed in any client
1.45 s
mean pause when the LB container is killed
container restart dominates; lightweight services recover in ~0.2 s
0.534 s
GPU model weights restored after worker death
vs 30–126 s reload from disk — the weights never left the GPU
LLM / SSE clients · WebSocket apps · DB clients long-lived TCP LINUX HOST — EDGE / LB DOCKER — standard containers Trevi Proxy — drop-in load balancer terminates client TCP · streams SSE / WebSocket · pipes upstream deployed like any container, restarted like any container — the difference is what happens when it dies clients need zero changes · no SDK · no retry logic Trevi runtime kernel-anchored state custody · patent pending holds live custody of every connection and every in-flight request, continuously and outside the app process · self-healing (~1 s), supervised connections in-flight requests state custody — both ways Linux kernel connections live here, below any process — Trevi anchors them so no process death can close them LINUX HOST — GPU INFERENCE (separate machine) DOCKER — standard containers Inference worker — Trevi-enabled vLLM serves the OpenAI-compatible API · streams tokens on restart it does NOT reload the model from disk — the weights never left the GPU works with OOM-kills · segfaults · deploys · node maintenance Trevi runtime patent pending custody of connections, requests, and GPU memory — outside the worker process connections requests GPU memory GPU model weights stay resident across worker death ✓ restored in 0.534 s state custody Linux kernel the upstream connection survives the worker the exact same way upstream TCP across the network data plane — client TCP (never closes) Trevi state custody

What survives a kill

  • Every open connection — TCP stays ESTABLISHED; the client never sees a reset
  • Every in-flight request — streams resume exactly where the wire left off, byte-perfect
  • GPU model weights — restored to a new worker in 0.534 s, not re-read from disk
  • Session state — DB pool sessions keep parameters, prepared statements, cancel keys

What you have to change

  • In your clients: nothing. No SDK, no retry logic, no reconnect handling
  • In your deployment: almost nothing. Standard Docker containers, standard restart policies
  • Works for the failures that skip every graceful path: OOM-kill, segfault, node eviction
  • The same mechanism makes deploys zero-downtime — an upgrade is just a restart

Why alternatives can’t do this

  • nginx / HAProxy / Envoy: every drain mechanism needs a running process — SIGKILL skips them all; the client gets a TCP RST (measured: 0/6 streams survive)
  • Retry layers: require idempotency and can’t resume a partial stream — SSE/WebSocket/LLM generation usually can’t re-enter mid-response
  • Managed load balancers: fail over to a different backend — that’s a new connection and a lost response, not a resumed one
the load balancer container is SIGKILL’d mid-stream — the client’s connection survives
Client app SSE stream / WebSocket / DB one TCP connection, no retry logic anywhere in its code GET /v1/completions (stream) ⏱ client-visible gap · 1.45 s mean ✓ 50/50 survived · 0 reconnects DOCKER · TREVI PROXY standard restart policy — nothing special proxy process — running LB: client TCP ⇄ backend TCP Trevi keeps custody of its state continuously, in the background ✕ process gone no shutdown handler ran no drain, no goodbye, nothing ↻ fresh proxy process empty state — about to get everything back from Trevi every flow’s state lives OUTSIDE this box → it survives the box ✕ SIGKILL — OOM-kill / segfault / docker kill -9 skips every handler; drain configs are useless here Backend — GPU worker keeps generating; its half of the pipe is protected too (if IT dies → see tab ③) the one TCP connection — never closes ✓ no reset — still connected resumes exactly where it left off connections + in-flight state return to the new process Trevi runtime — patent pending live custody of every flow, held outside the app process: connections in-flight requests LINUX KERNEL client ⇄ LB ESTABLISHED — the connection never closes, because Trevi never lets go
client’s view ›
 
the GPU worker is SIGKILL’d mid-generation — weights stay on the GPU, the stream resumes byte-perfect
Client app SSE stream — direct, or via the load balancer (tab ②): same story deterministic sampling POST /v1/completions (stream) ⏱ one mid-stream pause · 28.3 s mean ✓ 50/50 byte-identical · 0 reconnects DOCKER · TREVI-ENABLED vLLM · GPU HOST standard restart policy — nothing special inference worker — running scheduler · OpenAI-compatible server Trevi keeps custody of the request and the GPU state, continuously ✕ process gone killed mid-generation vanilla vLLM: request lost + 30–126 s reload ↻ fresh worker booting framework init — the fixed cost (~27 s of the 28; restart itself: ~0.5 s) the request and the weights live OUTSIDE this box → the restart is warm, not cold ✕ SIGKILL — OOM-kill / segfault / node evict mid-generation, ~2 s into an 80-token stream GPU model weights multi-GB, resident held under Trevi custody — worker death does not release them ✓ weights still resident weights restored from resident GPU memory — 0.534 s, no disk the one TCP connection — never closes ✓ no reset — still connected continues at the next token connection + request + GPU state return to the new worker Trevi runtime — patent pending custody of everything the worker needs back: connections request state GPU memory LINUX KERNEL client ⇄ worker ESTABLISHED — the connection never closes, because Trevi never lets go
client’s view ›