DataHour: Current and Best Practices for LLM Evaluation

Enterprises are eagerly integrating large language models (LLMs) into their products, yet most of the products don't get deployed. This is largely due to the fact that there is no universal framework for effectively evaluating and benchmarking these LLM-based applications. In this DataHour, the speaker will share the current stage of LLM evaluation, and some best practices and will also demonstrate a practical use case.

As we navigate the fast-paced and cluttered field of LLM evaluation, we encounter a spectrum of assessment methodologies. For tasks with supervised datasets, traditional machine learning metrics such as accuracy, and F1-score remain relevant. However, in scenarios lacking a definite target, similarity metrics like BLEU and ROUGE come into play, despite their noted limitations in capturing human-like creativity and diversity in generated text. She will also talk about the case, where we will use LLMs to evaluate LLMs. This session will also cover the evolving space of multimodal LMs and why we need evaluation for multimodal models as well. As the space of LLMs is ever-evolving, our evaluation strategies must evolve in tandem to fully understand their potential and mitigate their risks when using them for specific use cases. 

Sonali is a seasoned data scientist with six years of experience in AI. As the lead data scientist at Progressive, she pioneers innovative data-driven products and crafts AI strategies for clients and stakeholders. Sonali is a dedicated learner and adept problem-solver, constantly acquiring new skills to tackle complex business challenges. Armed with a Bachelor's degree in Geophysics from IIT Kharagpur and a Master's degree in Applied Mathematics from the University of Washington, Seattle, she has also authored six US patents in AI.