Large language models (LLMs) are essential for delivering accurate information across various applications. To enhance their factual performance, understanding specific use cases where they falter is crucial. Introducing the FACTS Benchmark Suite, in collaboration with Kaggle, expands on previous developments by including three new benchmarks. The Parametric Benchmark assesses LLMs’ ability to answer trivia-style questions accurately without external tools. The Search Benchmark evaluates how effectively models utilize search capabilities to retrieve and synthesize information, while the Multimodal Benchmark tests their accuracy in responding to image-based prompts. Alongside the updated Grounding Benchmark v2, which ensures responses are contextually grounded, the suite consists of 3,513 curated examples. The FACTS Score, calculated from public and private set averages across these benchmarks, provides an overall accuracy metric. Kaggle will manage and oversee the benchmark suite, including maintaining private datasets and curating results on a public leaderboard. For more on evaluation methodology, refer to our technical report.
Source link
