Loss Went Down, So We Shipped It - A Cautionary Tale of LLM Fine-Tuning

Fine-tuning LLMs has become a common approach for adapting models to domain-specific tasks, but improving training loss does not necessarily mean improving model performance . Models can memorize data, lose general capabilities, or overfit benchmarks , leading to misleading results in real-world applications.

In this session, we will focus on the most overlooked aspect of the fine-tuning pipeline: evaluation . You’ll learn how to move beyond loss of metrics and build rigorous evaluation strategies to truly assess whether your fine-tuned model is better, more reliable, and ready for production.

This is an insight-driven and practical session designed to help practitioners build more robust and trustworthy LLM systems .

Key Takeaways:

• Pitfalls of Fine-Tuning – why lower loss doesn’t guarantee better performance
• Overfitting & Memorization – understanding hidden risks in model training
• Evaluation Strategies – moving beyond loss to meaningful metrics
• Benchmarking Challenges – avoiding misleading performance signals
• Practical Framework – evaluating fine-tuned models for real-world use

Raju Joshi is an AI Architect at AWS with 7.5+ years of experience in AI/ML, Generative AI, and Agentic AI.

He has led end-to-end LLM customization projects, including continued pre-training on large-scale financial datasets, instruction fine-tuning for tool-calling systems, and RAG pipelines for enterprise use cases across multiple industries. He holds an M.Tech from IIT Bombay.