SEIS: Self-Evolving Inference Systems
Zhen Xu, Jingyu Liu, Zongze Li, Tahseen Rabbani and Ce Zhang (University of Chicago) introduce SEIS, which has coding agents optimize an entire LLM inference engine, mini-sglang, over chained sessions that inherit earlier code changes and notes.
Ask this paper
Setup. Agents start from unmodified mini-sglang, which already has paged KV storage, FlashInfer attention and CUDA graphs. Each session inherits the previous session's code and experience, and every engine must pass a numerical precision gate on perplexity, KL divergence and argmax agreement.
What the agents built. Quantized kernels, fused computation across operator boundaries, lower execution overhead, and prompt-lookup speculative decoding.
Speed. Serving Qwen3-0.6B on an H100, the best engine reaches 3.27x the throughput of the original and beats vLLM, TensorRT-LLM and SGLang on single-request workloads. Mean request latency falls from 295 to 89 ms.
Accuracy. GSM8K accuracy is 58.48% against 57.42% for the original, and long-input retrieval 87.78% against 88.15%. Chained sessions beat independent attempts.
Abstract
Inference systems determine how fast and how cheaply language models can be served, so making them faster has direct practical value. However, prior work focuses mostly on optimizing certain parts such as kernels or memory within the large system. In this work, we take a holistic approach and apply agentic self-evolution to optimize the whole system end-to-end. Our SEIS (Self-Evolving Inference Systems) autonomously optimizes the entire mini-sglang engine without human intervention through iterative sessions with inherited experiences and code changes. Serving Qwen3-0.6B on H100, the resulting engine reaches 3.27X the throughput of the original mini-sglang implementation and beats SOTA engines like vLLM, TensorRT-LLM, and SGLang in the single-request workload. The correctness of the optimized inference engine by SEIS is tested in terms of numerical difference and downstream accuracy on math and long-context retrieval tasks. The code and session histories show that the speedup comes from redesigning the whole engine and that building on earlier sessions beats independent attempts. These results suggest that agentic self-evolution can optimize a complex system end-to-end. The evaluation also has to evolve with the engine, and letting agents evolve it is a natural next step.