Turnstile

Token-level rollout capture for RL training of LLM agents.

Turnstile is a proxy that sits between an agent orchestrator and an inference backend. To the orchestrator it looks like a normal OpenAI Chat Completions API. To the trainer, it exposes token sequences, trainable token segments, and log-probabilities you actually need to update a model. It closes the gap between agent rollout frameworks and RL training frameworks without requiring either side to change.

• Drop-in proxy. Any orchestrator that speaks the OpenAI Chat Completions API works — Strands, OpenHands, or your own harness. No SDK changes or special clients needed.
• Token-level capture. Records exact prompt/output token IDs and per-token logprobs straight from the backend, so training sees the same tokens the model saw.
• Prefix-aware sequence collapsing. Multi-turn agent loops collapse into one contiguous sequence with output/input segment metadata instead of N redundant copies of the history.
• Generic training schema. Exports (tokens, segment_info, logprobs). Framework-specific conversion (SLIME, etc.) lives in client-side adapters, not in the proxy.
• Pluggable frontend and backend. OpenAI Chat Completions on the front, sglang on the back today — both interfaces are designed to grow.
• Embeddable. Run as a standalone server, or drop it into a Python training script via PyO3 bindings and get in-process access to training data.

RL training for LLM agents splits into two phases handled by different systems:

1. Rollout — an agent attempts a task. Orchestrators like Strands, OpenHands, and others drive this through an OpenAI-compatible chat API.
2. Training — a model is updated on what happened. Frameworks like SLIME need token sequences, model-output boundaries, and log-probabilities.

The orchestrator talks to a black-box chat API. The trainer needs the stuff that black box hides. The agent doesn't know — or need to know — the exact tokens its model saw, but the trainer can't update weights without them. Teams end up writing custom adapters, forking orchestrators, or piping brittle log scraping into training — and nobody owns the gap. Turnstile owns it, so neither the orchestrator nor the trainer has to change.

Turnstile sits in the middle. The orchestrator thinks it's talking to a normal OpenAI API. Turnstile forwards requests to the real inference backend (sglang today, vLLM and others planned), streams responses back, and silently records every token in and out along with its logprob.

flowchart LR
    Orchestrator["Orchestrator\n(any OpenAI-compatible)"]
    Turnstile["Turnstile\n(proxy)"]
    Engine["Inference Engine\n(sglang, vllm)"]
    Data["Training-ready\nsequences + segments"]
    Training["RL Training\n(SLIME, etc.)"]

    Orchestrator -- "OpenAI API" --> Turnstile
    Turnstile -- "stream response" --> Orchestrator
    Turnstile -- "Backend API" --> Engine
    Engine -- "stream response" --> Turnstile
    Turnstile -- "episode complete" --> Data
    Data --> Training

When the rollout finishes, the training loop asks Turnstile for the episode's data and gets back training-ready sequences grouped by group_id (an arbitrary user-chosen tag for a rollout or experiment).

Install the Python bindings:

pip install turnstile

Embed Turnstile inside a training script and route an agent through it:

import turnstile
from openai import OpenAI

with turnstile.Proxy(
    backend_url="http://localhost:30000",        # your sglang server
    model="Qwen/Qwen2.5-0.5B-Instruct",
) as proxy:
    group_id = proxy.create_group(max_trajectory_tokens=4096)

    # Point any OpenAI-compatible orchestrator at proxy.addr.
    # group_id is supplied via the URL path: /group/<id>/v1/...
    client = OpenAI(base_url=f"http://{proxy.addr}/group/{group_id}/v1", api_key="-")

    # ... run your agent against `client`; Turnstile captures every turn ...
    client.chat.completions.create(
        model="Qwen/Qwen2.5-0.5B-Instruct",
        messages=[{"role": "user", "content": "Plan a trip to Paris."}],
    )

    # At the end of the episode, pull training data in-process.
    for seq in proxy.get_training_sequences(group_id):
        tokens   = seq.tokens     # list[int]
        segments = seq.segment_info  # list[tuple[bool, int]] — (is_output, length)
        logprobs = seq.logprobs   # list[float] — 0.0 at input positions
        versions = seq.weight_versions  # list[tuple[int, str]]

Any orchestrator with a configurable base URL (Strands, OpenHands, raw OpenAI SDK, etc.) can point at Turnstile with a one-line config change after creating a group for the run. The agent code is unchanged between an unmonitored run and a training run.

Prefer a standalone server? Run turnstile as a service, create a group with POST /v1/create_group, and talk to it over HTTP with the turnstile Python client or any HTTP client. Training data is fetched with GET /group/{id}/v1/training/sequences.

Turnstile handles the two shapes agent rollouts actually take:

Prefix-overlapping turns (the common multi-turn agent shape). Each turn's prompt is the full conversation history plus new environment tokens, so turn N+1's input is a suffix extension of turn N's full sequence. Turnstile detects the overlap and collapses every turn into one contiguous sequence with segment metadata:

tokens: [sys] [usr] [ast] [ast] [ast] [env] [env] [ast] [ast] [ast] [env] [ast] [ast] ...
segment_info: [[false, 2], [true, 3], [false, 2], [true, 3], [false, 1], [true, 2], ...]

true segments are model-generated and trainable; false segments are environment/user input. This eliminates the N redundant copies of the conversation prefix you'd get from recording each turn independently, and produces exactly the shape an RL trainer expects.

Non-overlapping turns. When turns don't share a prefix — independent requests within an episode, or an orchestrator that rewrites context between turns — collapsing isn't possible. Turnstile returns multiple independent sequences per episode, each with its own segment metadata, and the trainer treats them as separate examples.

One schema for every backend and every trainer:

• tokens — the full collapsed token sequence for the episode (or sequence segment).
• segment_info — run-length encoded [is_output, length] pairs, e.g. [[false, 41], [true, 12]].
• logprobs — per-token log-probabilities; 0.0 at input positions, actual backend values at generated positions.
• weight_versions — token offsets where SGLang reported a new backend weight version for the turn, e.g. [[0, "default"], [41, "2"]].
• routed_experts — optional base64-encoded SGLang MoE routing payload for R3 routing replay. Enable it with TURNSTILE_RETURN_ROUTED_EXPERTS=true or turnstile.Proxy(..., return_routed_experts=True). By default, TURNSTILE_ROUTED_EXPERTS_MODE=strict starts a new sequence when routed experts for an overlapping prefix disagree; latest keeps the sequence collapsed and exports the latest full payload. In Python, pass turnstile.RoutedExpertsMode.STRICT or turnstile.RoutedExpertsMode.LATEST for routed_experts_mode; raw strings are rejected by the Python bindings. Downstream training decodes it as int32 expert IDs and reshapes it with the model's (num_layers, top_k).
• response_parser — controls how decoded backend output becomes assistant content and structured tool calls. The default qwen_permissive preserves Turnstile's liberal parser. Set TURNSTILE_RESPONSE_PARSER=qwen_sglang or turnstile.Proxy(..., response_parser="qwen_sglang") for SGLang-compatible Qwen non-stream tool-call parsing.

Turnstile always sends the group ID as the SGLang Model Gateway routing header. SGLang only uses that header for worker affinity when the request goes through a gateway configured for consistent-hashing routing; direct worker requests and other routing modes ignore it. If a deployment also needs separate SGLang radix-cache namespaces per group, enable TURNSTILE_SGLANG_GROUP_CACHE_NAMESPACE=true or turnstile.Proxy(..., sglang_group_cache_namespace=True); this sends the group ID as SGLang's backend-only JSON extra_key field. The cache namespace option is disabled by default and requires SGLang v0.5.11 or newer; older releases accepted extra_key but did not pass it into scheduler prefix-cache matching. See sgl-project/sglang#23300.

Framework-specific conversion (e.g. SLIME's expected layout) lives in client-side adapters, not in the proxy. The proxy stays small and generic.

R3 capture is disabled by default and is not part of CI. See docs/r3-moe-smoke-test.md for the manual smoke test against a real MoE model.

Three components, intentionally small:

• Core proxy (Rust). Async Axum server. Handles request forwarding, response streaming, token recording, sequence collapsing, and group-scoped storage. Loads the tokenizer and HuggingFace chat template directly so that what the trainer sees matches what the model saw.
• Python bindings (PyO3). Wrap the Rust core. Start/stop the proxy from a training script, read training data in-process — no extra HTTP hop when you're co-located.
• Python client library. Standalone HTTP client for remote Turnstile instances, plus adapters that convert the generic schema into framework-specific formats (SLIME first; more to follow).

The API surface is modular on purpose. The OpenAI Chat Completions frontend is what most orchestrators already speak, and the recording and collapsing logic is independent of any one API shape. The inference backend interface operates at the token level (token-in, token-out), so adding vLLM or another token-level engine doesn't touch capture or collapsing.

Designed to compose with what people are already using:

Turnstile is orchestrator-agnostic by design — any system that speaks OpenAI Chat Completions can produce RL-ready rollouts through it without modification.

To keep the scope honest:

• Not a model, dataset, or set of weights.
• Not an RL algorithm or training recipe — it captures rollouts, it doesn't prescribe how you train on them.
• Not a replacement for your orchestrator, inference engine, or training framework.
• Not a managed or production service.

Turnstile has no built-in authentication, authorization, TLS termination, or multi-tenant isolation. Run it only on trusted networks or behind external access controls that match your deployment requirements.

Early development. APIs, schemas, and backend support will move. Feedback and integrations are welcome.