Peer review

The Best Practices handbook was developed by the Association’s Acquisitions Editorial Committee through a consensus-building two-year process to articulate a set of practices that comprise a rigorous process of peer review. The Committee has rightly noted that, “the peer review process is highly complex, involves many individuals, and must be responsive to the norms of the appropriate fields.” Disciplinary expectations, administrative procedures, inter-disciplinary and creative works, and innovative publishing formats may all demand changes in approach. However, well-reasoned differences in practices can only be evaluated against a solid understanding of what constitutes a standard practice of high-quality peer review.

Peer reviewers play a central and critical part in the peer review process, but may come to the role without any guidance and be unaware of their ethical obligations. Journals
have an obligation to provide transparent policies for peer review, and reviewers have an obligation to conduct reviews in an ethical and accountable manner. Clear communication between the journal and the reviewers is essential to facilitate consistent, fair and timely review. COPE has heard cases from its members related to peer review issues and bases these guidelines, in part, on the collective experience and wisdom of the COPE Forum participants.

One of the key findings in this report is that researchers from emerging regions are
under-represented in the peer review process. While there are indications this trend may self-correct in the long run, we believe it is an urgent issue to address. We will struggle to meet the growing demands on the system without the contributions of the full global research community. The reality is that this peer review disparity is harming the development of non-Western researchers — fewer review invitations means fewer opportunities to see the latest research trends, learn what journals are looking for in a great manuscript, make professional connections with journal editors, and develop critical analysis skills.

SSH editors saw anonymized peer review benefits to outweigh those of open peer review. They considered anonymized peer review the “gold standard” that authors and editors are expected to follow to respect institutional policies; moreover, anonymized review was also perceived as ethically superior due to the protection it provides, and more pragmatic due to eased seeking of reviewers. Finally, editors acknowledged their power in the publication process and reported strategies for keeping their work as unbiased as possible.

We found that the total time reviewers globally worked on peer reviews was over 100 million hours in 2020, equivalent to over 15 thousand years. The estimated monetary value of the time US-based reviewers spent on reviews was over 1.5 billion USD in 2020. For China-based reviewers, the estimate is over 600 million USD, and for UK-based, close to 400 million USD.

Additionally, many medical, nursing, and healthcare educators, peer reviewers, and authors may not be completely familiar with the current changes in peer review. Moreover, reviewer education and training have unfortunately remained lacking. This is especially crucial since current initiatives to improve the review process are now influenced by factors other than academic needs. Thus, increasing attention has recently focused on ways of streamlining the peer review process and implementing alternative peer-review methods using new technologies and open access models.

This article aims to give an overview of the innovative strategies for peer review and to consider perspectives that may be helpful in introducing changes to peer review. Critical assessments of peer review innovations and incentives based on past and present experiences are indispensable. A theoretical appraisal must be balanced by a realistic appraisal of the ethical roles of all stakeholders in enhancing the peer review process. As the peer review system is far from being perfect, identifying and developing core competencies among reviewers, continuing education of researchers, reviewer education and training, and professional engagement of the scientific community in various disciplines may help bridge gaps in an imperfect but indispensable peer review system.

A propensity score matching with the radius matching method was performed using 14 confounding variables. The other five common matching methods of propensity score matching, the bias-adjusted matching estimator developed by Abadie and Imbens (Simple and bias-corrected matching estimators for average treatment effects, National Bureau of Economic Research, Cambridge, pp 1–57, 2002), and the data set excluding articles with an extremely high citation count were used to check the robustness of the results. The results of this study showed that articles with open peer review history could be expected to have significantly greater citation counts than articles with closed peer review history. Our results suggest that open peer review can improve citation count, and that the best practice for open peer review might be a compromise open peer review.

The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review.

Stakeholders expected peer review (1) to assess the contributions of a manuscript, (2) to conduct quality control, (3) to improve manuscripts, (4) to assess the suitability of manuscripts for a journal, (5) to provide a decision-making tool for editors, (6) to provide feedback by peers, (7) to curate a community, and (8) to provide a seal of accreditation for published articles. Stakeholders with different roles and tasks in the peer review process differed in the value they attached to the functions of peer review.

By considering 9,220 submissions and 18,525 reviews from 2010 to 2017, we measured changes both before and during the pilot and found that publishing reports did not significantly compromise referees’ willingness to review, recommendations, or turn-around times. Younger and non-academic scholars were more willing to accept to review and provided more positive and objective recommendations. Male referees tended to write more constructive reports during the pilot. Only 8.1% of referees agreed to reveal their identity in the published report. These findings suggest that open peer review does not compromise the process, at least when referees are able to protect their anonymity.

In this article, we describe the role of peer review and how our five principles support that goal. This article is intended to continue the conversation about improving peer review standards and provide guidance to journal teams looking to improve their standards. It is accompanied by a detailed checklist, which could be used by journal teams to assess their current peer review standards.

Journal editors’ understanding of the roles and partly of tasks of peer reviewers are profoundly shaped by each journal’s unique context and characteristics, including financial and human resources and journal reputation or prestige. There was a broad agreement among journal editors on expected technical tasks of peer reviewers related to scientific aspects, but there were different expectations in the level of depth. We also found that most journal editors support the perspective that authorship experience is key to high-quality reviews, while formal training in peer reviewing is not.

Specifically, the aims of the paper are to show: (1) how much experience and knowledge ECRs had of peer review – both as authors and as reviewers; (2) what they felt the benefits were and what suggestions they had for improvement; (3) what they thought of open peer review (OPR); and (4) who they felt should organize peer review. Data were obtained from 116 science and social science ECRs, most of whom had published and were subject to in-depth interviews conducted face-to-face, via Skype, or over the telephone.

We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

This quantifiable data yields rich information on the range and extent of differing definitions over time and by broad subject area. Quantifying definitions in this way allows us to accurately portray exactly how ambiguously the phrase “open peer review” has been used thus far, for the literature offers 22 distinct configurations of seven traits, effectively meaning that there are 22 different definitions of OPR in the literature reviewed. I propose a pragmatic definition of open peer review as an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the aims of Open Science, including making reviewer and author identities open, publishing review reports and enabling greater participation in the peer review process.

Based on an initial literature review, we construct a sample of 82 channels of scientific communication covering all forms of review identified by the survey, and analyze the review mechanisms used by each channel. We identify two major trends: the rapidly expanding role of preprint servers (e.g., ArXiv) that dispense with traditional peer review altogether, and the growth of “non-selective review,” focusing on papers' scientific quality rather than their perceived importance and novelty. Other potentially important developments include forms of “open review,” which remove reviewer anonymity, and interactive review, as well as new mechanisms for post-publication review and out-of-channel reader commentary, especially critical commentary targeting high profile papers. One of the strongest findings of the survey is the persistence of major differences between the peer review processes used by different disciplines. None of these differences is likely to disappear in the foreseeable future. The most likely scenario for the coming years is thus continued diversification, in which different review mechanisms serve different author, reader, and publisher needs. Relatively little is known about the impact of these innovations on the problems they address. These are important questions for future quantitative research.