An artificial-intelligence coach has been developed to provide feedback aimed at making peer reviews more specific and useful. The potential for this tool to improve the overall quality of research papers is currently under evaluation.
Scientists involved in peer review are increasingly utilizing AI for various tasks, including literature searches and prose refinement. James Zou, a computer scientist at Stanford University, and his colleagues investigated whether large language models (LLMs) could address common issues in peer review feedback, such as lack of thoroughness or inappropriate tone.
Authors at the 2023 Association for Computational Linguistics annual meeting identified 12.9% of reviews as poor quality.
Zou indicated that vague comments, such as "not novel," were a primary cause of poor quality reviews. He also noted that, although rare, reviews could be unprofessional, contain personal attacks, or include factual errors, such as criticizing work for omitting an analysis that was, in fact, present.
Development of the Review Feedback Agent
Zou's team collected approximately twelve reviews considered vague, unprofessional, or incorrect, along with examples of appropriate feedback. This curated data was used to train an LLM and subsequently develop a Review Feedback Agent.
This agent integrates five LLMs that collaborate and cross-check each other's work.
Implementation in a Major Conference
The AI tool was deployed during the review process for the 2025 International Conference on Learning Representations in Singapore. This significant AI conference consistently receives over 10,000 submissions annually. Each submitted paper is typically reviewed by 3–4 individuals, with around 30% being accepted.
The research group randomly selected about 20,000 previously written reviews. The Review Feedback Agent evaluated these reviews, and its generated feedback was then transmitted to the respective reviewers.
The AI system's suggestions frequently focused on enhancing specificity and constructiveness, often employing the phrase "to make this feedback more actionable …".