Don’t get me wrong. I do understand the value of peer review in scholarly publishing, but I still think, we should end the human-dependent peer-review system and move to a completely AI-based one. The main reasons for this proposal are: human-dependent peer review is inequitable, suffers from injustice, and is potentially unsustainable. In last week’s Ask the Chefs piece on peer review, I shared a few examples; let me give some more below.
We, too, are enthusiastic about the potential role of artificial intelligence systems in peer review. The problems and biases of the current system are legion and discussed extensively in the academic publishing ecosystem. But the bloopers, imitations and amplified biases of the systems, like ChatGPT, are sobering, if not alarming. The use of LLMs and other artificial intelligence systems needs to be careful and considered. If not, we risk creating a worse mess than the status quo. This piece published by Scholarly Kitchen in September 2023 provides a roadmap for the transition to 100% AI peer review. We tend to believe a realistic best case scenario is that 100% of peer reviews conducted by humans is supported by AI technology.
2) We are told that another motivation for peer-reviewers is to learn about the latest research before it gets published. It seems that millions of openly available preprints as well as research reports, post-graduate theses, posters, presentations, and conference proceedings are not enough to learn about new unpublished research.
3) Like many of you, I also see peer reviewing as a “good karma” activity — you help strangers (even if you know them, or at least their names) in your discipline to communicate their research better, and some other strangers will do the same favor for you when you submit your article to a journal. It has recently been reckoned that, if scientists would conduct at least one peer review per article they publish, that should ensure sustainable functioning of the peer-review system. But it isn’t so simple in reality. In my experience working on the editorial side of journals, recently-published authors regularly decline the same journals’ requests to review a relevant manuscript — reciprocation or good karma doesn’t happen as much as one might hope.
