Google Unveils LMEval: An Open-Source Tool for Evaluating Cross-Provider LLMs

LMEval is a tool designed to help AI researchers and developers compare the performance of various large language models (LLMs) efficiently and accurately. Given the rapid development of new models, LMEval facilitates quick evaluations for safety and security. Its key features include compatibility with numerous LLM providers, incremental benchmark execution to enhance efficiency, and support for multimodal evaluations involving text, images, and code. Utilizing the LiteLLM framework, LMEval allows for consistent benchmarking across different APIs. Written in Python and available on GitHub, users define benchmarks and tasks, such as identifying eye colors in images, before evaluating selected models. The tool also enables result storage in SQLite databases, with encrypted security measures. LMEval includes a dashboard for performance analysis and has been pivotal in creating the Phare LLM Benchmark, assessing LLM safety and reliability. Other frameworks like Harbor Bench and EleutherAI’s LM Evaluation Harness also exist for similar purposes.

Source link