Empirical Study of Agent Developer Practices
Free while signed in. Answers cite the passages they came from.
This paper presents the first large-scale empirical study of LLM-based agent frameworks, analyzing 11,910 developer discussions across ten popular frameworks. The research identifies practical challenges developers face and evaluates how well current frameworks meet their needs. - **Four challenge domains:** Developers encounter issues across logic (25.6% related to task termination and loop prevention), tools (14% from API limitations and permission errors), performance (25% involving context retention and memory management), and version conflicts (23% causing build failures and compatibility issues). - **Framework selection is hard:** More than 80% of developers report difficulty identifying frameworks that best meet their specific requirements. The study recommends prioritizing ecosystem robustness and long-term maintenance over short-term popularity when choosing frameworks. - **Multi-framework combinations dominate:** Combining multiple frameworks with different functions has become the primary approach to agent development. Each framework excels in different areas: LangChain and CrewAI lower barriers for beginners, while AutoGen and LangChain lead in task decomposition and multi-agent collaboration. - **Performance optimization is universally weak:** Across all ten frameworks studied, performance optimization remains a common shortcoming. Despite mature ecosystems, AutoGen and LangChain face the highest maintenance complexity, highlighting tradeoffs between feature richness and long-term maintainability.
Four challenge domains: Developers encounter issues across logic (25.6% related to task termination and loop prevention), tools (14% from API limitations and permission errors), performance (25% involving context retention and memory management), and version conflicts (23% causing build failures and compatibility issues).
Framework selection is hard: More than 80% of developers report difficulty identifying frameworks that best meet their specific requirements. The study recommends prioritizing ecosystem robustness and long-term maintenance over short-term popularity when choosing frameworks.
Multi-framework combinations dominate: Combining multiple frameworks with different functions has become the primary approach to agent development. Each framework excels in different areas: LangChain and CrewAI lower barriers for beginners, while AutoGen and LangChain lead in task decomposition and multi-agent collaboration.
Performance optimization is universally weak: Across all ten frameworks studied, performance optimization remains a common shortcoming. Despite mature ecosystems, AutoGen and LangChain face the highest maintenance complexity, highlighting tradeoffs between feature richness and long-term maintainability.
Abstract
The rise of large language models (LLMs) has sparked a surge of interest in agents, leading to the rapid growth of agent frameworks. Agent frameworks are software toolkits and libraries that provide standardized components, abstractions, and orchestration mechanisms to simplify agent development. Despite widespread use of agent frameworks, their practical applications and how they influence the agent development process remain underexplored. Different agent frameworks encounter similar problems during use, indicating that these recurring issues deserve greater attention and call for further improvements in agent framework design. Meanwhile, as the number of agent frameworks continues to grow and evolve, more than 80% of developers report difficulties in identifying the frameworks that best meet their specific development requirements. In this paper, we conduct the first empirical study of LLM-based agent frameworks, exploring real-world experiences of developers in building AI agents. To compare how well the agent frameworks meet developer needs, we further collect developer discussions for the ten previously identified agent frameworks, resulting in a total of 11,910 discussions. Finally, by analyzing these discussions, we compare the frameworks across five dimensions: development efficiency, functional abstraction, learning cost, performance optimization, and maintainability, which refers to how easily developers can update and extend both the framework itself and the agents built upon it over time. Our comparative analysis reveals significant differences among frameworks in how they meet the needs of agent developers. Overall, we provide a set of findings and implications for the LLM-driven AI agent framework ecosystem and offer insights for the design of future LLM-based agent frameworks and agent developers.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack