🚀 New blog: Advanced CUDA Graph Techniques in Inference and the design story behind them.
CUDA Graph removes kernel-launch overhead by recording GPU work once and replaying it. But bringing it into a real inference engine introduces challenges in ops compatibility, graph
Very excited to see native vision come to the @Zai_org GLM family, and impressed by the architecture work behind it!
With SGLang features like RadixAttention and HiCache, GLM-5.3-Flash is one of the most efficient and capable choices for agentic work across coding and
GLM-5.3-Flash (320B-A18B) from @Zai_org drops today with day-0 support in SGLang. You may know it as ox-alpha from the past few days!
👀 It's the first native multimodal model in the GLM-5 series, able to review its own output visually and fix what's wrong.
🚀 It outperforms
SGLang brings Day-0 support for Qwen 3.8-Flash-Next, an early preview of the Qwen4 architecture!
In collaboration with @Alibaba_Qwen, @NVIDIAAI , and @AIatAMD, we've optimized this 125B MoE (6B active, plus 51B N-gram embeddings) for efficient inference:
1️⃣540 tok/s decode
Congrats to @Alibaba_Qwen on launching Qwen3.8-Flash! SGLang is proud to be a day-0 partner supporting the new architecture preview for Qwen4.
It's a 125B main model with 51B of N-gram embeddings and 6B activated per token.
The 51B N-gram embeddings scale model capacity with
New Blog: Fast Engine Recovery: Sub-Second Engine Restart for SGLang via Weight Cache Daemon.
We built a persistent GPU process that cuts weight loading for Ling-2.6-1T FP8 from ~495s to ~0.63s, by serving post-quantized weights to new engines via CUDA IPC zero-copy.
- ~785×
🚀 New blog: Pushing the Limits of Serving DeepSeek-V4-Pro
DeepSeek-V4-Pro (1.6T MoE) on H20 reaches 271 output tokens/s at batch size 1, just 1.42× off B300 on hardware with no native FP4 Tensor Cores.
Together with @ant_oss, we built a scenario-specific serving stack on
