How AI Impacts Skill Formation
Free while signed in. Answers cite the passages they came from.
Researchers from Anthropic conducted randomized experiments to study how AI assistance affects the development of software engineering skills. They find that using AI to complete coding tasks with a new Python library significantly impaired conceptual understanding, code reading, and debugging abilities - without delivering significant efficiency gains on average. - **Learning loss from AI assistance:** In a controlled study with 52 developers learning the Python Trio library, participants using AI scored 17% lower (Cohen's d=0.738, p=0.01) on a skills evaluation covering conceptual understanding, debugging, and code reading. The largest gap appeared in debugging questions, likely because control group participants encountered and independently resolved more errors during the task. - **No significant productivity gains:** Contrary to prior work showing AI-assisted coding speedups, AI did not significantly reduce task completion time in this learning context. Several participants spent up to 11 minutes composing queries to the AI assistant, offsetting potential time savings from code generation. - **Six distinct AI interaction patterns:** Qualitative analysis of screen recordings revealed three low-scoring patterns (AI Delegation, Progressive AI Reliance, Iterative AI Debugging) averaging below 40% quiz scores, and three high-scoring patterns (Conceptual Inquiry at 86%, Generation-Then-Comprehension at 68%, Hybrid Code-Explanation at 65%) where participants stayed cognitively engaged. - **Implications for AI-assisted workflows:** The findings suggest that AI-enhanced productivity is not a shortcut to competence. The high-scoring interaction patterns all involved independent thinking and cognitive effort, indicating that how AI is used matters more than whether it is used - particularly in safety-critical domains requiring human oversight of AI-generated code.
Learning loss from AI assistance: In a controlled study with 52 developers learning the Python Trio library, participants using AI scored 17% lower (Cohen’s d=0.738, p=0.01) on a skills evaluation covering conceptual understanding, debugging, and code reading. The largest gap appeared in debugging questions, likely because control group participants encountered and independently resolved more errors during the task.
No significant productivity gains: Contrary to prior work showing AI-assisted coding speedups, AI did not significantly reduce task completion time in this learning context. Several participants spent up to 11 minutes composing queries to the AI assistant, offsetting potential time savings from code generation.
Six distinct AI interaction patterns: Qualitative analysis of screen recordings revealed three low-scoring patterns (AI Delegation, Progressive AI Reliance, Iterative AI Debugging) averaging below 40% quiz scores, and three high-scoring patterns (Conceptual Inquiry at 86%, Generation-Then-Comprehension at 68%, Hybrid Code-Explanation at 65%) where participants stayed cognitively engaged.
Implications for AI-assisted workflows: The findings suggest that AI-enhanced productivity is not a shortcut to competence. The high-scoring interaction patterns all involved independent thinking and cognitive effort, indicating that how AI is used matters more than whether it is used - particularly in safety-critical domains requiring human oversight of AI-generated code.
Abstract
AI assistance produces significant productivity gains across professional domains, particularly for novice workers. Yet how this assistance affects the development of skills required to effectively supervise AI remains unclear. Novice workers who rely heavily on AI to complete unfamiliar tasks may compromise their own skill acquisition in the process. We conduct randomized experiments to study how developers gained mastery of a new asynchronous programming library with and without the assistance of AI. We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library. We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance. Our findings suggest that AI-enhanced productivity is not a shortcut to competence and AI assistance should be carefully adopted into workflows to preserve skill formation -- particularly in safety-critical domains.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack