Although generative AI applications promise efficiency and can benefit the peer review process, given their shortcomings and our limited knowledge of their innerworkings, Mohammad Hosseini and Serge P.J.M. Horbach argue they should not be used independently nor indiscriminately across all settings. Focusing on recent developments, they suggest the grant peer review process is among contexts that generative AI should be used very carefully, if at all.
In the ever-evolving landscape of academic research and scholarly communication, the advent of generative AI and large language models (LLMs) like OpenAI’s ChatGPT has sparked attention, praise and criticism. The use of generative AI for various academic tasks has been discussed at length, including on this blog (e.g. in education). Among the potential use cases, having Generative AI support reviewers and editors in the peer review process seems like a promising option. The peer review system has long been facing various challenges including biased and/or unconstructive reviews, paucity of expert reviewers and the time-consuming nature of the endeavour.
This fascinating and timely London School of Economics Blog piece takes a sober and helpful look at the question of whether generative AI can play a useful role in peer review. #SpoilerAlert not on its own and not now (at least for a while), like many things regarding AI, it can greatly enhance the work of thoughtful humans. We’re not on the cusp of LLMs replacing humans, but of making good work by humans better.
Assisting reviews, but not an independent reviewer
In their current form, generative AI applications are unable to perform peer review independently (i.e., without human supervision), because they still make too many mistakes and their inner workings are unknown and rapidly changing, resulting in unpredictable outcomes. However, generative AI can assist actors in the peer review process in other ways, for example by helping reviewers improve their initial notes to become more constructive and respectful. In addition, generative AI can enable scholars not writing in their native language to contribute to the review process in other languages (e.g., English), or help editors in writing decision letters based on a set of review reports. These use cases could help to broaden the reviewer pool and make the process more efficient and equitable.