Linden Li on X: "Important, very much needed update to inference performance benchmarks from @SemiAnalysis_ ! Fixed token-in, token-out is a very different inference workload from what we observe in practice (both for RL training and production inference). The inference optimization story (engine, hardware, etc) is very different for these workloads, and the benchmarking north star should reflect that."

Linden Li

@lindensli

Important, very much needed update to inference performance benchmarks from @SemiAnalysis_ ! Fixed token-in, token-out is a very different inference workload from what we observe in practice (both for RL training and production inference). The inference optimization story (engine, hardware, etc) is very different for these workloads, and the benchmarking north star should reflect that.

SemiAnalysis

@SemiAnalysis_

Aug 25

Introducing AgentX: InferenceXs new agentic inference performance benchmark. (1/2)🧵

00:00

3:24 AM · Aug 25, 20267.6KViews

3

53

18