How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding
Jeongyeon Kim and John Mitchell (Stanford University) measure how well multi-agent LLM pipelines perform qualitative coding, where agents code independently, debate and reconcile, and identify the dataset and process factors that decide accuracy.
Ask this paper
Pipeline: In a literature-informed baseline, AI agents code the same data independently, debate their disagreements and reconcile to a final label.
Accuracy factors: Coding accuracy depends on codebook length, on how similar the qualitative data items are to each other, and on how much the agents disagree.
Debate: Intense debates between agents that stayed unresolved were associated with higher accuracy.
Limits: The agents reproduce many human discussion behaviors but do not adapt their responses to context. The paper releases an AI discussion dataset and design recommendations for automated coding systems.
Abstract
The utility of AI in multi-coder qualitative coding has been widely discussed, yet little empirical evidence exists to delineate the contexts in which it performs reliably. We address this gap by quantifying the effectiveness of multi-agent LLM coding across varied qualitative datasets, revealing key contextual and structural factors that mediate coding outcomes. We developed a literature-informed baseline pipeline that enables AI agents to independently code, debate, and reconcile disagreements. Results revealed that coding accuracy depends on factors such as codebook length, qualitative data similarity, and agent disagreement. Notably, intense and unresolved debates between agents led to higher accuracy. Our analysis showed that while LLMs emulate many human discussion behaviors, they lack adaptive responsiveness to context. From these findings, we offer design recommendations for building automated coding systems. Our open-source AI discussion dataset and methodological framework lay the groundwork for advancing the design of AI-mediated automated thematic analysis.