The peer review process: challenges and progress
Dr. Irene Hames is an independent research-publication and peer-review specialist with over 30 years’ practice in scholarly publishing. She served as specialist advisor to the UK Parliamentary House of Commons Science and Technology Committee, and helped produce the report Peer Review in Scientific Publications. Dr. Hames is also author of the book Peer Review and Manuscript Management in Scientific Journals, published by Wiley-Blackwell in association with ALPSP (the Association of Learned and Professional Society Publishers). She has several papers published on peer review of which the more latest are, The switching face of peer review and Peer review at the beginning of the 21st century. (See her extended biographical summary in part one of the interview series.)
In the first part of our interview series, Dr. Hames discussed ethical issues in scholarly publication and the work of the Committee on Publication Ethics. Here in the 2nd part, she discusses how the journal editorial process has evolved over the years and what the community can expect from peer review.
You have 20 years’ practice as a managing editor, over which time journal processes have evolved remarkably. In your opinion why are some researchers and peer reviewers dissatisfied with the traditional peer review process?
There has been considerable evolution in the way journal editorial work is done, such as the introduction of online working and implements to detect textual duplication or inappropriate pic manipulation, but the basic principles of good practice in peer review remain the same – they are independent of system and access or business model. There have always been criticisms of peer review, but they are now more visible and the subject of much public discussion and debate. Also, the scholarly publishing world, and its relationship with the research community, is switching. However, surveys demonstrate that researchers generally want to improve peer review rather than get rid of it. The Taylor & Francis Open Access Survey (March 2013) found that peer review was rated the most significant service authors expect to receive when paying to have their papers published open access, coming above both rapid peer review and rapid publication. A major problem is the superb variation in the quality of peer review across journals. Also, the ‘fake reviewer’ cases that began emerging in 2012 (details can be found in the ‘faked emails’ category on the Retraction Watch blog) decreased confidence enormously because it became apparent that some journals weren’t carrying out the most basic reviewer checks. Even I, who thought I’d seen it all in my 30-plus years in scholarly publishing, was shocked.
There is also enlargening concern and impatience with the ‘wastage’ in traditional peer review, where manuscripts may be reviewed a number of times, going from journal to journal until accepted for publication. This is opening up a system that is already embarking to creak. I’m hearing from a number of editors that they are finding it more and more difficult to find reviewers. This is, therefore, one of the main areas people are addressing and looking for solutions, for example cascading review, where manuscripts and the associated reviews can be passed on with the authors’ permission to other journals for consideration, and portable review, where reviews are provided to authors, sometimes by independent peer-review services, and the authors can include them with their submissions. Other initiatives to avoid review and reviewer time wastage include review and referral to best-fit target journals by independent companies.
The internet has opened up opportunities for experimentation with fresh systems and models of peer review. Many fresh players are coming in the field, some from outside the traditional journal ecosystem. It is a very arousing time! It’s difficult to predict which initiatives will be successful, which sustainable and scalable. They do, however, represent a real challenge to traditional journals and publishers, who need to evolve and innovate, which they are beginning to, to provide what the research community needs and to meet the expectations and requirements of research funders.
Two other issues have contributed to researcher dissatisfaction with peer review. Very first, the lack of transparency in the process, and efforts are being made to address this, for example by publishing the reviewers’ reports and editorial decisions with articles. 2nd, there has been growing frustration that peer-review activity hasn’t to date been considered as something that ‘counts’ in a researcher’s research and scholarly contribution record. That too is switching and projects are underway looking into how peer-review activity can be appropriately acknowledged and tied to, for example, researchers’ ORCID records (http://orcid.org/blog/2014/04/08/orcid-and-casrai-acknowledging-peer-review-activities). Once peer-reviewing is awarded adequate recognition as a professional activity, this may act as an enlargened incentive to review and provide quality reviews.
In light of the latest cases of unethical or poor research practices, what should the scholarly community realistically expect from a peer review?
Members of the scholarly community have a right to expect that the work they send to journals is treated and assessed appropriately and fairly, also that information about its subjugation and review is kept confidential unless a journal’s model involves openness, and that no one involved in the treating and peer review of their manuscripts uses that privileged information for their own build up. They have a right to expect good constructive reviews from individuals who are experts and that it is the work they’ve submitted that is being assessed, not them personally. If researchers get quality service when they are authors and are treated well by journals when they are reviewers – not all are – it’s more likely that they’ll take part in reviewing for those journals and provide quality reviews.
Researchers have many commitments; they are often overstretched, and their time is valuable. Many, fairly understandably, feel considerable frustration when they have to spend more time than should be necessary attempting to submit manuscripts or reviews because of inadequacies in journals’ system or instructions. All journals should regularly evaluate their processes and make sure they are fit for purpose. All the information and correspondence for authors and reviewers should also be checked regularly and updated as necessary, ensuring consistency across website pages, the online manuscript system, and correspondence. If this isn’t done, researchers may determine to go elsewhere next time to submit or review.
Dr. Irene Hames with Dr. Yan Shuai, President of the Society of China University Journals and Deputy Director of Tsinghua University Press (which publishes the Chinese version of Dr. Hames peer review book, Peer Review and Manuscript Management in Scientific Journals). Photo taken at International Seminar of Medical Journal Editors, Shanghai, China, April 2014, where Dr. Hames gave invited talks on peer review and publication ethics.
You have explained that peer review relies on a “reciprocal process” in which “authors and reviewers are mostly the same community” and everyone does their fair share of reviewing. Is it challenging for journal editors to ensure that there is a good balance inbetween submitting and reviewing manuscripts?
It’s fairly straightforward nowadays for journals to monitor reviewing activity at their own journals – manuscript tracking systems make this very effortless. A potential reviewer’s reviewing geyser, present and past, should always be checked before they are approached to make sure they aren’t overcharged at any one time or haven’t been asked too many times over the past year. But editors can only work towards achieving a good balance inbetween submissions and reviewing at their own journals (and give gentle reminders to those who submit lots of manuscripts but do little reviewing for them). Researchers get review requests not only from a range of other journals, but also from grant funders and, now, the fresh organizations that are coming in the field. All are rivaling for the same pool of reviewers, so building strong relationships and treating reviewers well are becoming even more significant.
Be sure to read the last part of this interview series, Keeping the scholarly record sound — a journal editor’s duty. Dr. Hames explains how journals should treat studies with flaws. She also discusses how she works with over Five,000 scientists, from Nobel prize winners to postdoctoral researchers and PhD students to help promote a public understanding of science.