dQwen3.5: Hybrid-Attention Diffusion Language Models
Anton Xue and colleagues adapt hybrid attention-plus-RNN Qwen3.5 backbones into diffusion language models at 0.8B, 2B, 4B and 9B, producing the dQwen3.5 family.
Ask this paper
The obstacle is structural causality. Nearly all AR-to-diffusion adaptations start from full-attention transformers, but RNN layers are causal by construction and not straightforward to bidirectionalize.
Hybrid backbones adapt about twice as fast. Against a full-attention control, the hybrid reaches a given training loss in roughly half the tokens.
Any-order decoding behaviour is retained. Across scales dQwen3.5 resembles full-attention diffusion models in how it handles arbitrary decoding orders.
Parallel decoding performance holds up. Which is the practical reason to want a diffusion LM in the first place, so the adaptation does not trade away the payoff.
Abstract
Adapting a pretrained autoregressive (AR) model is a cost-efficient route to a diffusion language model (DLM). While nearly all such adaptations start from a full-attention transformer, AR modeling has shifted toward hybrid architectures that interleave attention and RNN layers. This creates an obstacle for adaptation: unlike attention, RNNs are structurally causal and nontrivial to bidirectionalize. Despite this mismatch, we investigate whether such backbones can become effective DLMs by adapting Qwen3.5 at 0.8B, 2B, 4B, and 9B scales, yielding the dQwen3.5 family. We find that hybrid backbones can be efficient starting points for adaptation: against a full-attention control, the hybrid reaches a given training loss in about half the tokens. Across scales, dQwen3.5 resembles full-attention DLMs in any-order decoding behavior and performs strongly under parallel decoding.