Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning
A large InstaDeep team with AIMS and Stellenbosch extends offline sequence models to variable agent counts and multi-task observation and action spaces, then measures which scaling axis actually produces zero-shot transfer in offline multi-agent RL.
Ask this paper
The headline finding: scaling task diversity, not dataset size, is the dominant factor in robust zero-shot transfer. That reorders the usual data-collection priority for offline MARL.
Architecture work required to run the study: offline sequence-modelling architectures were extended to handle multi-task observation and action spaces and variable numbers of agents across tasks, which is what makes a single held-out evaluation possible.
Scale of the sweep: task diversity, dataset size and network capacity are varied across four environments, Connector, RWARE, SMAX and LBF.
Magnitude: a mean improvement of 3.2x on held-out test tasks over single-task models, and consistent gains over strong behaviour-cloning baselines.
Practical consequence: collection effort should go into varying the number of agents and the task set rather than into more trajectories from the same tasks.
Abstract
Generalising to unseen tasks remains a fundamental challenge in offline multi-agent reinforcement learning (MARL). In this work, we present a principled analysis of zero-shot task generalisation in the offline setting and conduct an extensive empirical investigation into the scaling behaviour governing task diversity, dataset size, and network capacity. To facilitate this study, we extend offline sequence modelling architectures to handle multi-task observation and action spaces alongside variable agent counts across tasks. Our primary finding is that scaling task diversity---rather than sheer dataset size is the dominant factor in achieving robust zero-shot transfer. Through large-scale experiments across four challenging environments (Connector, RWARE, SMAX, and LBF), we demonstrate that our multi-task approach achieves a mean improvement of 3.2x on held-out test tasks compared to single-task models and consistently outperforms strong behaviour cloning baselines. These results suggest that the development of generalisable MARL agents should prioritise the diversity of the training distribution with varying numbers of agents, providing a roadmap for scaling offline MARL effectively.