LLM-as-a-Judge
Learn to evaluate open-ended AI output by building an LLM judge from scratch.
$ open "Why rules can't grade this"
Open-ended output has no single correct answer, so keyword rules fail. Name the qualities a judge will score instead.
checkpoints run as you work
✓ checkpoint passed, lab 2 unlocked
What you'll build
Curriculum
6 labs, about 1 h
- 1
Why rules can't grade this
Open-ended output has no single correct answer, so keyword rules fail. Name the qualities a judge will score instead.
4 min
Pro - 2
Your first judge call
Call a frontier model from the sandbox and ask it to judge one support reply with a simple yes or no verdict.
12 min
Pro - 3
A rubric and a score
Give the judge a three-part rubric and have it score every reply as parseable JSON instead of a yes or no.
14 min
Pro - 4
Who judges the judge?
Measure whether the judge can be trusted by comparing its scores against human labels and reporting the agreement rate.
14 min
Pro - 5
Bias and reliability
Learn the common LLM-judge biases and harden the judge with reason-first scoring and temperature 0, then document one failure mode.
15 min
Pro - 6
Capstone: ship the judge
Turn the judge's scores and its agreement with humans into a short, honest evaluation report, and confirm the suite runs end to end.
15 min
Pro
Your instructor
Elvis Saravia
Founder, DAIR.AI
Elvis founded DAIR.AI and wrote the Prompt Engineering Guide, one of the most widely used references on working with language models. He has worked at companies like Meta AI and Elastic, and specializes in AI agents, RAG, and harness engineering.
How labs work
Live workspace
You run the real tools in a browser workspace. Nothing to install.
Checkpoints
Each lab checks your files and output, then unlocks the next one.
Your work stays
The workspace persists between labs so you finish with a project.