Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving
Jae Gon Kim, Donghoon Yoo and colleagues (Xenoscube) measure NVIDIA's Max-Q inference power profile on a disaggregated B200 serving system and replace it with separate, calibrated power settings for the prefill GPUs and the decode GPUs.
Ask this paper
Vendor profile: Max-Q gives only +8.6% tokens per joule, varies by model, adds +5.2% mean end-to-end latency, and applies one setting to prefill and decode GPUs that run in opposite hardware regimes.
Controller: The prefill lane runs under an SM-clock window whose floor guarantees latency. The decode lane runs under a power cap that automatic calibration places just above a measured throughput and latency cliff, with a runtime SLO guard.
Why capping works here: A disaggregated decode lane draws flat, memory-bound power, so the cap binds continuously and avoids the reactive overshoot that led POLCA to reject power capping.
Results: Serving Qwen3-Coder-480B (FP8) under agentic load on 8x B200, the balanced mode gives +20.4% tokens per joule at +3.5% latency, and a three-day run saves 32.3% of a lane pair's electricity. Dense models recover about 5x less, so the claims are limited to MoE serving.
Abstract
Datacenter GPU power is the binding constraint on LLM serving capacity, and production serving has shifted to prefill/decode (PD) disaggregation. Deploying NVIDIA's Max-Q inference profile on a disaggregated B200 system, we found its realized gain modest (+8.6% tokens/J), model-dependent, and carrying a mean end-to-end latency cost (+5.2%) that throughput-only evaluation does not surface; the profile also applies one setting to prefill and decode GPUs that operate in opposite hardware regimes. We hypothesize that the optimal power setting is a property of the deployed (model, quantization, engine, hardware) combination rather than of the GPU class, that each lane warrants its own profile, and that converting SLO headroom into energy safely requires latency-gated calibration under a runtime SLO guard rather than a fixed recipe. We present a phase-decoupled, model-calibrated controller: the prefill lane runs under an SM-clock window whose floor is a latency guarantee by construction, and the decode lane under a power cap placed by automatic calibration just above a measured throughput/latency cliff. Because a disaggregated decode lane draws flat, memory-bound power, the cap binds continuously, the reactive-overshoot weakness that led POLCA to reject capping is absent, and the GPU's own power manager retains throughput under the cap. On an 8x B200 node serving Qwen3-Coder-480B (FP8) under agentic load, our balanced mode delivers +20.4% tokens/J at +3.5% mean e2e versus +8.6% at +5.2% for Max-Q, a Pareto improvement on both axes. On Qwen3-235B-A22B (NVFP4) every operating mode meets the ITL-p99 SLO in every repetition; both vendor profiles miss it. A decode-actuator A/B shows the calibrated cap beats static clock locks, and a three-day sustained run saves 32.3% of a lane pair's electricity. Both models are MoE; a dense model recovers roughly 5x less, so we scope our claims to MoE serving.