AutoCrawler
AutoCrawler is a two-stage framework that combines LLMs with the hierarchical structure of HTML to auto-generate reusable web scrapers. Wrapper-based scrapers break on new sites and pure LLM agents don't reuse well across pages; AutoCrawler addresses both limitations.
Ask this paper
Hierarchical DOM understanding: The agent walks the HTML tree with top-down and step-back operations, progressively refining its understanding of a page before emitting a complete and executable scraper.
Similarity across pages: Patterns learned on one page of a site generalize to structurally similar pages, making the generated scraper durable rather than a one-shot extraction.
New executability metric: The paper introduces an executability metric for evaluating scraper-generation systems, filling a gap in prior benchmarks that focused only on extraction accuracy.
EMNLP 2024: Experiments across multiple LLM backends validate the framework on diverse websites; the work was accepted to EMNLP 2024.