Evaluating and Testing LLM Applications
Evals, golden sets, LLM-as-judge and regression testing for AI products.
Taught by Rohan Maharjan · ML Engineer
What you'll learn
- Build and maintain a golden evaluation dataset sourced from production traffic and labelled with clear guidelines
- Implement deterministic, reference-based and embedding-based metrics and know when each applies
- Design LLM-as-judge rubrics, control position bias and calibrate judges against human labels
- Evaluate RAG retrieval quality, answer faithfulness and multi-step agent trajectories
- Instrument an LLM application with traces and turn production logs into new eval cases
- Run prompt and model regression tests in CI with pass thresholds and cost budgets
About this course
Evaluating and Testing LLM Applications is an advanced course for engineers and technical product leads who ship features built on large language models and need to know, with numbers, whether a change made things better or worse. Evaluation is the bottleneck skill in production AI: teams that lack it tune prompts by feel, break working behaviour silently and cannot defend model upgrades. You will build an evaluation system from scratch in Python: source and label a golden dataset from real traffic, write deterministic checks and reference-based metrics, implement LLM-as-judge grading with calibration against human labels, evaluate retrieval-augmented generation and agent trajectories, add tracing and observability with tools such as Langfuse and Arize Phoenix, and wire prompt regression tests into CI with promptfoo and GitHub Actions. The final lesson closes the loop with online evaluation and drift monitoring. You should be fluent in Python and have built at least one LLM-backed feature.
Course content
8 lessons- 1Why Evals Are the BottleneckVibes versus measurement, the taxonomy of LLM evaluations, and eval-driven development as a working method.Preview
- 2Building a Golden DatasetSource cases from logs and synthesis, write labelling guidelines, and structure the dataset so it stays useful as the product changes.
- 3Metrics: Deterministic Checks and Reference-Based ScoresExact and structural assertions, when overlap metrics help, and embedding similarity as a middle ground.
- 4LLM-as-Judge Done ProperlyRubric design, pairwise comparison, position bias, judge calibration and the failure modes of model graders.
- 5Evaluating RAG and AgentsRetrieval metrics, faithfulness and answer relevance, and trajectory evals for multi-step tool-using agents.
- 6Observability and Tracing in ProductionInstrument every model call with traces, capture the context needed to reproduce failures, and turn logs into new eval cases.
- 7Regression Testing and CI for PromptsVersion prompts like code, run the eval suite in GitHub Actions, and gate merges on thresholds and cost.
- 8Online Evaluation and Closing the LoopA/B testing model and prompt changes, using implicit signals, monitoring drift and guardrail metrics on live traffic.