For more than three centuries, peer review has remained the cornerstone of scientific publishing. Despite continuous improvements in editorial management systems, plagiarism detection software, and reviewer databases, the fundamental process has changed very little: reviewers evaluate an entire manuscript and provide an overall recommendation based on their expertise.
However, this traditional approach faces growing challenges. The increasing volume of scientific publications, reviewer fatigue, long evaluation times, and concerns about consistency have led researchers and publishers to explore new models for assessing scientific quality.
One of the most ambitious initiatives in this area is OpenEval, a project that proposes a radical shift in how scientific manuscripts are evaluated by combining human expertise with large language models (LLMs).
From Article-Level Review to Claim-Level Assessment
Traditional peer review treats a manuscript as a single entity. Reviewers assess originality, methodology, clarity, novelty, and significance before recommending acceptance, revision, or rejection.
OpenEval proposes a fundamentally different perspective.
Instead of evaluating the paper as a whole, the system automatically decomposes the manuscript into its individual scientific claims. Each claim is then independently analyzed using artificial intelligence and supporting evidence.
For example, rather than simply concluding that a paper is «well written» or «scientifically sound,» the system may identify dozens of individual statements such as:
- The experimental methodology follows recognized standards.
- The statistical analysis supports the reported conclusions.
- The literature review accurately represents previous work.
- The claimed novelty is justified by existing publications.
Each claim can then be examined separately, allowing reviewers to focus their attention where uncertainty is greatest.
This granular approach has the potential to make peer review more transparent, objective, and reproducible.
How OpenEval Works
Although still under development, the OpenEval framework follows several sequential stages.
First, the manuscript is processed using natural language processing techniques to identify factual statements, hypotheses, interpretations, methodological descriptions, and conclusions.
Next, large language models evaluate these claims by comparing them with the evidence presented in the manuscript and, where appropriate, with external scientific knowledge.
Rather than replacing reviewers, AI generates structured assessments that include confidence scores, explanations, and potential weaknesses.
Human reviewers can then validate, modify, or reject these AI-generated evaluations.
The final assessment becomes a transparent combination of machine-assisted analysis and expert judgment.
Improving Transparency
One of the persistent criticisms of traditional peer review is its limited transparency.
Authors often receive brief reviewer comments that provide little insight into how individual aspects of the manuscript influenced the final recommendation.
Two reviewers evaluating the same paper may also reach completely different conclusions without clearly explaining why.
OpenEval addresses this issue by making the evaluation process far more explicit.
Instead of a single recommendation, every scientific claim receives its own assessment, supporting rationale, and confidence level.
This allows authors to understand precisely which parts of their work require revision while enabling editors to make better-informed decisions.
Consistency Across Reviews
Reviewer variability is another long-standing challenge in scholarly publishing.
Different reviewers may place different emphasis on novelty, statistical rigor, methodological quality, or writing style.
Such variability can produce inconsistent editorial decisions.
AI-assisted claim evaluation offers the possibility of introducing greater consistency into the review process.
Because every manuscript is analyzed using the same structured framework, the initial assessment becomes more standardized.
Human reviewers remain essential, but their expertise is directed toward interpreting complex scientific questions rather than performing repetitive screening tasks.
Reducing Reviewer Workload
Finding qualified reviewers has become increasingly difficult.
Many researchers receive dozens of review invitations every year while balancing research, teaching, grant writing, and administrative responsibilities.
OpenEval aims to reduce this burden by automating many routine evaluation tasks.
Rather than reading an entire manuscript without guidance, reviewers receive a structured report highlighting:
- Claims with low confidence.
- Potential inconsistencies.
- Missing evidence.
- Statistical concerns.
- Unsupported conclusions.
This allows reviewers to focus on areas requiring genuine scientific expertise.
Potential Benefits for Editors
Editors often face difficult decisions when reviewer recommendations conflict.
An AI-assisted claim-level assessment could provide an additional layer of objective information.
Instead of relying solely on overall reviewer opinions, editors could examine how individual claims performed during evaluation.
This may improve editorial consistency while helping identify manuscripts that require additional specialist review.
For journals handling thousands of submissions annually, such tools could significantly streamline editorial workflows without compromising quality.
Important Limitations
Despite its promise, OpenEval should not be viewed as a replacement for human peer review.
Large language models remain susceptible to several limitations, including:
- Hallucinated information.
- Overconfidence.
- Difficulty evaluating highly specialized or emerging research.
- Dependence on training data.
- Limited understanding of experimental context.
Scientific evaluation involves far more than checking factual consistency.
Assessing originality, creativity, experimental design, theoretical significance, and future research impact still requires experienced researchers.
For this reason, OpenEval is best understood as a decision-support system rather than an autonomous reviewer.
Ethical and Practical Challenges
The integration of AI into peer review also raises important questions.
Should authors know when AI has contributed to the evaluation?
How should journals validate AI-generated assessments?
Who is responsible if an AI system incorrectly identifies methodological flaws or overlooks critical errors?
Publishers will need clear governance policies addressing transparency, accountability, confidentiality, and human oversight.
Maintaining trust in scientific publishing will require that AI remains a tool assisting editorial decisions rather than replacing expert judgment.
A Glimpse of the Future
Artificial intelligence is already transforming many aspects of scholarly publishing, from plagiarism detection and language editing to reviewer recommendation systems and editorial analytics.
OpenEval represents one of the first serious attempts to rethink peer review itself.
Whether this specific framework becomes widely adopted remains uncertain, but the underlying idea—combining structured AI analysis with human expertise—is likely to influence the future of scientific publishing.
Rather than asking whether AI will participate in peer review, the more relevant question is how journals can integrate these technologies responsibly while preserving scientific integrity, fairness, and transparency.
If implemented carefully, systems like OpenEval could help create a more efficient, consistent, and evidence-based peer review process—one that supports editors, assists reviewers, and ultimately strengthens the reliability of published research.