AI Evals 101: Evaluating GenAI & Agentic AI Systems the Right Way

Everyone is building and shipping GenAI and Agentic features, but how do you prove they are truly reliable in the real world?

GenAI and Agentic AI systems can retrieve knowledge, plan tasks, call tools, reflect, and even collaborate, but knowing whether they are actually correct is surprisingly hard, and the popular shortcuts quietly mislead you: 

• Off-the-shelf libraries and metrics that return confident but wrong scores 

• LLM-as-judge that behaves like a biased black box 

• Agents that reach the right answer through the wrong process 

• No clear way to know whether a judge can even be trusted 

This session draws from my hands-on experience building & evaluating RAG and Agentic AI systems over the past few years. We'll focus on three pillars:  

• Evaluating GenAI/RAG systems 

• Evaluating Agentic AI systems  

• Building judges you can actually trust 

Overall there will be one constant running theme: surface metrics lie, so you must evaluate what is happening under the hood, not just the final text. 

Agenda 

1. Introduction

• Why evaluation is the hardest (and most skipped) part of shipping GenAI 

• The core trap: surface metrics lie - "relevant ≠ correct" and "right answer ≠ right process" 

• An evaluation taxonomy - deterministic vs LLM-as-judge, component vs trajectory 

2. Evaluating GenAI & RAG Systems

• Deterministic retrieval metrics — Precision@K / Recall@K and where they shine 

• Off-the-shelf LLM-judge libraries — RAGAS, DeepEval etc. why you should not trust them blindly 

• Building your own judges - deterministic, chain-of-thought (CoT) and hybrid (deterministic + LLM-judge) decision-graph (DAG) approaches 

3. Evaluating Agentic AI Systems

• Capturing the full agent trajectory, tool calls (not just final responses) in LangGraph 

• The four evaluation layers - final decision, tool calls, trajectory, and reasoning 

• Deterministic-first - why most agent checks are a simple ==, not a fancy judge 

• Reasoning quality - where an LLM judge (CoT / DAG) genuinely earns its place 

• Newer methods worth knowing 

4. Bonus: Coding Agents for Evals (if timepermits)

• Eval-driven development - core methodology and workflow 

• Packaging evals as reusable skills - an "eval-runner" you and your team can invoke on demand 

• Key limitations and future scope  

Throughout the Session 

• Best practices and caveats for real-world readiness 

• Hands-on code demos on real examples