Evaluating large language models (LLMs) presents significant challenges, as even advanced systems can deliver unreliable judgments. Researchers Hamid Dadkhahi, Firas Trabelsi, and Parker Riley, in collaboration with colleagues from Google and DeepMind, developed a distribution-calibrated aggregation method to enhance evaluation reliability. By optimizing inference-time compute and employing a novel statistical approach, they transformed noisy individual assessments into robust ratings. Their technique, known as BTD (Bias Towards Distribution), calibrates LLM output distributions to align with the true problem context, significantly reducing biases and errors across tasks like machine translation and reasoning.
The introduction of a three-way preference model further refines LLM evaluations, allowing nuanced assessments of positive, negative, and tied votes using a Bradley-Terry-Davidson framework. Their findings indicate that this method achieves results comparable to or exceeding human evaluators, marking a breakthrough in automated assessments. This innovative research has implications across diverse LLM applications, enhancing both robustness and generalization in model evaluations.
Source link
