LaMer
Free while signed in. Answers cite the passages they came from.
LaMer introduces a Meta-RL framework that enables LLM agents to actively explore and learn from environment feedback at test time. Unlike standard RL-trained agents that learn fixed policies and struggle with novel tasks, LaMer agents learn exploration strategies that transfer across environments. - **Cross-episode training:** Instead of optimizing single episodes independently, LaMer trains agents across sequences of episodes on the same task. Early episodes encourage exploration to gather information, while later episodes exploit that knowledge. A cross-episode discount factor controls the exploration-exploitation tradeoff. - **In-context policy adaptation:** The agent uses self-reflection to summarize past experiences and adjust strategy without gradient updates. This leverages LLMs' natural in-context learning abilities - the agent essentially implements an RL algorithm in context during deployment. - **Strong performance gains:** On Qwen3-4B, LaMer achieves 11% improvement on Sokoban, 14% on MineSweeper, and 19% on Webshop over RL baselines. The framework produces more diverse trajectories while achieving higher success rates, reaching a better exploration-exploitation balance. - **Better generalization:** LaMer-trained agents generalize better to harder and out-of-distribution tasks compared to standard RL agents. The learned exploration strategies transfer to novel environments, enabling more robust adaptation at test time.
Cross-episode training: Instead of optimizing single episodes independently, LaMer trains agents across sequences of episodes on the same task. Early episodes encourage exploration to gather information, while later episodes exploit that knowledge. A cross-episode discount factor controls the exploration-exploitation tradeoff.
In-context policy adaptation: The agent uses self-reflection to summarize past experiences and adjust strategy without gradient updates. This leverages LLMs’ natural in-context learning abilities - the agent essentially implements an RL algorithm in context during deployment.
Strong performance gains: On Qwen3-4B, LaMer achieves 11% improvement on Sokoban, 14% on MineSweeper, and 19% on Webshop over RL baselines. The framework produces more diverse trajectories while achieving higher success rates, reaching a better exploration-exploitation balance.
Better generalization: LaMer-trained agents generalize better to harder and out-of-distribution tasks compared to standard RL agents. The learned exploration strategies transfer to novel environments, enabling more robust adaptation at test time.
Abstract
Reinforcement learning (RL) has enabled the training of large language model (LLM) agents to interact with the environment and to solve multi-turn long-horizon tasks. However, the RL-trained agents often struggle in tasks that require active exploration and fail to efficiently adapt from trial-and-error experiences. In this paper, we present LaMer, a general Meta-RL framework that enables LLM agents to actively explore and learn from the environment feedback at test time. LaMer consists of two key components: (i) a cross-episode training framework to encourage exploration and long-term rewards optimization; and (ii) in-context policy adaptation via reflection, allowing the agent to adapt their policy from task feedback signal without gradient update. Experiments across diverse environments show that LaMer significantly improves performance over RL baselines, with 11%, 14%, and 19% performance gains on Sokoban, MineSweeper and Webshop, respectively. Moreover, LaMer also demonstrates better generalization to more challenging or previously unseen tasks compared to the RL-trained agents. Overall, our results demonstrate that Meta-RL provides a principled approach to induce exploration in language agents, enabling more robust adaptation to novel environments through learned exploration strategies.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack