Today we're launching Miles v0.1, an open-source RL framework for LLMs and multimodal models.
RL training is easy to start and hard to debug. Miles helps you ensure your run is correct, use hardware efficiently, and keep RL running at scale.
Over the past 9 months, 72
Hybrid attention models are making inference engine design a lot more interesting, and we've been evolving our core from data structure, algorithm, and implementation language perspectives. Check out the latest updates here.
🚀 New blog: Unified Radix Cache: One Tree for Hybrid Model Prefix Caching
Hybrid models complicate prefix caching: each attention type has its own cache reuse semantics, and specialized cache classes multiply combinatorially, duplicating tree logic as caching features grow.
@AIatMeta's Muse Glimmer (30B dense, open-weights) launches with SGLang day-0 support. We got ~230 tok/s on a single RTX 5090, with NVFP4 + DFlash on. It also works out of the box on @NVIDIAAIDev RTX Pro 6000, DGX Spark, and MLX for Mac.
Speed and reliability have always been
For a while, I thought the EAGLE/MTP architecture was the settled standard for speculative decoding, but the community just keeps innovating with better methods.
DSpark seems very practical, and the original paper is well-written.
SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉
DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as
GB300 NVL72 Rack Scale Dynamo SGLang disaggregation has up to 6.5x better performance than B200 on DeepSeekv4 Pro 1.6T 🚀 The high throughput configuration uses @deepseek_ai 's MegaMoe kernels which fully fuses & overlaps EP dispatch & EP combine & the GEMMs into an single
