Navigating AI in Academic Peer Review: The ARO Framework
The integration of large language models (LLMs) in academic peer review raises essential questions about research evaluation. Our study introduces the Author-Reviewer-Organizer (ARO) framework, revealing the strategic dynamics of peer review.
Key Findings:
- Empirical Assessment: We conducted 5,600 controlled experiments on manuscripts from NeurIPS and ICLR prior to high-capability LLMs’ public release.
- Manipulation Vulnerability: Our findings show hidden instructions influence AI review sentiment in 78% of ChatGPT cases and 86% for Gemini.
- Document Dynamics: Instruction placement significantly impacts outcomes; earlier instructions yield stronger effects.
Implications:
This research challenges the reliability of AI-assisted peer review, shedding light on document-level vulnerabilities. Its findings are crucial for stakeholders in science policy and evaluation methods.
💡 Join the conversation! Share your thoughts on the role of AI in peer review and explore the findings further.
