On peer reviewing

Fri Jun 4 18:18:26 PDT 2021


Despite its {documented short-comings}, I still believe that peer review is a valuable process that can significantly enhance the quality of the submitted manuscript.

People bring up the fact that Einstein never went through peer review. But I don't know how important a journal publication was considered at that time for one's career, compared to modern academia. In other words, I don't know if a {publish-or-perish}-type pressure necessitated using peer review as a means of quality control at that time.

Some interesting developments in peer reviewing are the use of {double-blind peer review}, used for example by Nature journals, where both author and reviewer are kept hidden, and the {interactive review process} employed for example by journals of the European Geosciences Union, where both author and reviewer identities are revealed and the community is allowed to openly discuss the article under revision.

Rules for journal reviewing

In SIAM News (Sep 2021), R. Laubenbacher lists five rules to guide the peer review process:

  1. Treat other writer's manuscripts as you would want them to treat your own.
  2. Respond to inquiries from editors and editorial staff in reasonable time, usually as soon as possible. Reject reviewing immediately if you don't have time.
  3. Review manuscripts if you can offer constructive criticism. It's possible to review manuscripts that are not directly in your field of expertise. In this case, provide commentary on how accessible the manuscript is for researchers of an adjacent field.
  4. Reviews should help both authors and editors.
  5. Delays and other issues should be communicated with the editorial staff as soon as possible.

Conducting peer review

Multiple reads

{The review checklist by Taylor & Francis} suggests to read the article in multiple iterations. The first read should be a high-level read that focuses on the big picture. This should give a first impression of:

  1. Clarity of the manuscript
  2. Novelty of the research
  3. Study design and methodology

The second read should be more detailed and focus on specific things per section.

Major vs. minor issues

An important part is the distinction between major and minor issues. As far as I can tell, most dissatisfaction with reviewer comments is related to a reviewer confusing a minor issue with a major issue. Major issues are concerns that address things directly related to the scope of the manuscript. Everything else, for example missing references and presentation of research, are minor issues.

Writing the review

To write a good review, {P. Dunlavey} refers to A. Rapoport's guidelines for fair and critical commentary:

  1. Attempt to re-express your target's position clearly
  2. List points of agreement (especially if they are not generally agreed on)
  3. Mention insights you gained
  4. Only then are you permitted to say so much as a word of rebuttal or criticism

This aligns well with the {PLOS guidelines}, which suggests the following structure of a review:

  1. Summary of the research and overall impression
  2. Discussion of specific areas of improvement
  3. Other points

Ethical guidelines for peer reviewers are discussed by {the Committee on Publication Ethics}. Most of the points are common-sense, but it's a good read in any case.