Science and Art of Reviewing Papers to Maintain Standards in Academia
Correspondence Address: Source of Support: None, Conflict of Interest: None DOI: 10.4103/0028-3886.333463
Source of Support: None, Conflict of Interest: None
Keywords: Peer review and Neurology, Peer review of articles
“I am afraid this manuscript may contribute not so much toward the field's advancement as much as toward its eventual demise”.
“There is so much good in the worst of us and so much bad in the best of us, that it ill behooves any of us to find fault with the rest of us”.
It all started in 1985. Prof S. Kalyanaraman had taken over as Editor of Neurology India. Having completed a 5-year postgraduate course in Neurosurgery in 1980, having obtained a National Board Certification (MNAMS) in 1981, and having just registered for a Ph.D. in neurosurgery, I was pretending to be a pseudo scientist and a workaholic to boot. Here I was, waiting for an opportunity to prove myself in academics in a private practice environment. As I already had 5 years of experience in correcting hundreds of pages of “proofs” for the NSI CME books from 1980 onwards, I was perhaps a good material for honorary editorial work. Being super smart, Prof. Kalyanaraman gave me the title of Associate Editor, Neurology India and dwelt at length on the tangible and intangible benefits, which would accrue to me, should I accept the position. Within days, I was given an article to “pre review”. With the unbridled enthusiasm of youth and wanting to justify my new position, I went through the article with an electron microscope. In 2021, it would be impossible for a present day reviewer to understand how one could review an article without access to a computer or the Internet. I patted myself on the back for a good job well done! I had pointed out 35 spelling mistakes, errors in grammar and syntax, poor quality photographs, incorrect legends, and wrong references (the young lady librarian of Madras Medical College got me the original references). As I had been reading for my Ph.D. Part I exam, this was an opportunity to display my newly acquired knowledge of Aims and Objectives, Materials and Methods, and so on. My ninth para in the extensive review concluded in no uncertain terms that the article should be summarily rejected - after all, there was incontrovertible evidence! I had even prepared a covering letter for the editor's signature. A few days later, Dr. Kalyanaraman called me and asked me to go through “some minor changes” that he had made in the “reply to the author.” The lengthy communication to the author, whom I had strongly berated, with all the words at my command had been replaced with a 'Thank you' note! The editor had pointed out that survival of Neurology India (in 1985) depended on article submission. For a state government doctor, doing private practice to submit an academic article needs to be extolled. The subsequent paras mentioned that “perhaps you could consider further improving the already good quality of your submission by paying attention to some minor inconsistencies like (a to z!). In 1985, such an endearing personalized letter coming from a renowned neurosurgeon would have made the young author forget that it was very unlikely that his article would ever see the light of day. He would have been delighted with the rejection!!
Peer Review has been defined as “a process of subjecting an author's scholarly work, research, or ideas to the scrutiny of others who are experts in the same field.” The author was amazed at the plethora of literature on the subject of peer review in the form of papers, chapters, standalone books, online courses, and formal training programmes. For more than 200 years, clinicians and researchers have published their work in scientific and medical journals. It was only in 1731 that the Royal Society of Edinburgh published 'Medical Essays and Observations' using what is now considered to be the first peer review process. A publication in 2009 estimated that 1.3 million articles are published annually in peer-reviewed journals.
It is assumed that the reviewer has read the abstract or even quickly glanced through the article and genuinely feels that he is competent to evaluate the submission. He/she must be in a position to spend 1–3 hours within the ensuing 10–14 days to thoroughly review the article. Comments must be submitted to the author and editor separately in the manner prescribed. Any delay in submission is a major disservice to one's colleagues and the journal. The reviewer should express his inability to review the article. Step-by-step guides to reviewing manuscripts have been published. Experienced reviewers go through the article meticulously in the second and even a third reading. Most journals offer three categories of recommendations: (1) accept as is; (2) revision and re-review; and (3) reject. One should focus on presence or absence of methodological flaws and presence of a new or reiteration of an existing important message. Poor writing can always be corrected. By default and as far as possible, a reviewer should suggest ways and means to modify the article to make it suitable for publication. Advising rejection immediately and antagonizing the authors is extremely simple. It is unbecoming unless all the reasons are meticulously documented politely. Rude reviewers offer condescending or outright offensive comments, often urging irrelevant citation of their own work.
Good reviewers have a resolute sense of responsibility realizing that they play a critical role in the progress of science, acknowledging the opportunity for teaching.
A thoughtful, thorough review takes time and effort. Reviewing should be an enjoyable, intellectually stimulating, challenging assignment not viewed as a labor intensive, time-consuming, and non-remunerative imposed task. After all, it is voluntary. If one does not have the time or inclination, invitation to review a paper should be declined. Demeaning, offensive words, and sarcasm reflects reviewer's ethics and behavior. As the reviewer has been chosen by the editor, it gives a wrong impression of the journal. A good reviewer is one who makes the author submit a better manuscript the next time. Currie et al. have pointed out the importance of never hurting an author with overtly candid remarks. The statement “there are no proper references at all” could be replaced with “Although references have been quoted, addition of key recent manuscripts will lend additional weight to the authors' arguments. Some suggestions are -----------(list citations).” Unfortunately, there is evidence that objectivity is not always present. Bias cannot be excluded in the peer-review process. Language, institutional affiliation, nationality, gender, territoriality within a field, personal gripes with authors, scientific dogma, and discontent/distrust of methodological advances are some of the causes. It is the editor's job to read a paper and make a decision also considering reviewer's comments. A different decision is the editor's prerogative. When the topic is highly specialized, it is possible that the reviewer could reasonably guess the authorship. It is critical that there should not be a bias for or against. “Do unto others as you would have others do unto you” should be a reviewer's maxim. Effective peer reviewing presupposes that one is professional, pleasant, helpful, scientific, timely, realistic, and sympathetic. Most importantly, it is essential to see the big picture holistically, contextually, and not get diverted by a few very minor correctable inconsistencies. An experienced reviewer can correctly and easily surmise if the paper was written merely to fulfill one's KPI (Key Performance Indicator) or as a genuine contribution to existing knowledge.
Authors' reaction to reviewers' comments
Dissatisfied authors see reviewers as being hasty, arbitrary, dogmatic, indifferent, superficial, mistaken, judgmental, arrogant, unfair, jealous, and self-serving! Such perceptions are understandable, given the high stakes for authors and the power, which anonymous reviewers wield! Occasionally such perceptions may not be incorrect. The authors can judge the reviewer's competence particularly when acidic comments display an obvious bias. Authors are not disgruntled because the article was not accepted. On the contrary, they are eternally grateful to a kind knowledgeable reviewer who has taken the effort and time to point out why the article needs to be resubmitted or why it is not suitable for that journal. Authors take umbrage when acidic, incorrect, inappropriate, hurting comments are made without evidence. Authors expect reviewers to judge a paper taking into account the context in which the work was done. Concluding in no uncertain terms, that at the height of the pandemic, 67 remote neuro ICU consults is too small a number to even make preliminary observations does not appear reasonable to the authors. This is particularly so when the authors have sought “anticipatory bail” by specifically referring to the lack of complete follow-up, as a limitation of the study. Details of sending one man to the moon is enough to justify a preliminary review article on interplanetary travel. To insist on statistically significant large numbers for such an article would reflect the reviewer's inability to see holistically the big picture. The same applies to relatively new applications in clinical neurosciences. The context and situation needs to be considered. Not all authors may be aware or would take the effort and time to represent to the editor initially and if necessary to the ombudsperson of the journal. The ruling given by the ombudsman is to be honored by the editor as well as the authors.
Hoppin in a publication in 2002 laments that no how-to-review paper had been published. In 2021, this is not true. The challenge to the reviewer is the daunting task of making the authors see and graciously accept what was unseen earlier. This requires not only domain expertise, but also excellent communication skills. Reviewing is as much an art as it is a science. In a study of 522 papers, which were rejected, Garg et al. noted that topic repetition, faulty methodology, ethical, patient, and authorship issues were common causes. Other causes included plagiarism, poor drafting, data inconsistency, fabrication, and failure to follow instructions mentioned on the journal website. Reviewing a manuscript is similar to volunteer work. It takes quality time to compile a detailed and balanced review ranging from 45 minutes to 8 hours with a median of 2.7 hours. Reviewing is a skill that benefits the author and the reviewer and is critical for the reputation of the journal. The reviewers are anonymous, but not journals for which the review is done. Authors have a choice of submitting to different journals and the time taken for reviewing and quality of the review is a major factor. Training of new reviewers has been advised.
The Lancet in one of its correspondence pages had a statement, “How do you know that [your] system of peer review would be any better than no review at all?” The reader had pointed out that a system yet to be scientifically proven to be effective, is being used globally as a differentiator. A Cochrane Collaboration study published a systematic review – investigating role of editorial peer review for improving quality of reports of biomedical studies. The authors concluded that blinding (of referees or authors), training of referees, and electronic communication media had no effect on the quality of the peer-review process. Editorial peer review improved the general quality and readability of the final product. In spite of absence of proof, the quality of review articles are improved with peer review. The Lancet Neurology, for example, sends the articles to two to four referees enclosing guidelines. The editors opine, “We realize that this system is not perfect, but we firmly believe that peer review improves the review articles we publish.”
Journal of Medical Internet Research (JMIR) with an impact factor of 5.03 invented karma credits in 2015. These are used to incentivize and reward reviewers and most productive authors and editors. Karma credits are Karma points collected by users for authoring, reviewing, or editing JMIR papers. These points can be set off for example against Article Processing Fees. Reviewer Credits is a start-up company, providing an independent platform dedicated to certify, measure, and reward the activity of scientists as peer-reviewers and conference speakers. Credits can also be converted into e-learning courses, author publication services, audiobooks, and environmental activities. Credits can even be converted to Gift Cards from Amazon India and Uber. Organizations like Publons offer reviewer recognition services, which may be useful for faculty applying for promotions. Crowd reviewing is the practice of gathering opinion or feedback from a large number of people, typically via the internet or an online community. It is a method of having articles reviewed in real-time online with anonymity preserved. Bias is supposedly removed when 10–30 domain experts give their opinion. Technology makes documentation much easier. The live online experience is highly instructive for all reviewers.
The raison d'etre for this communication was the tenor of the comments communicated to the author by a learned reviewer. It appeared that unlearning and relearning is as essential as learning afresh. In a point-to-point refutation, illustrations were given to demonstrate that every statement made by the honorable reviewer was incorrect. As a septuagenarian, the author had time, access to information, and inclination to submit a point-to-point rebuttal. This may not have been possible for a young upcoming busy clinician. He/she would have been devastated seeing months of hard work going down the drain, for all the wrong reasons. A word of caution to all authors – develop a hide, not just a thick skin, do not be hypersensitive. All reviewers may not be reading this article! Do not be naïve to expect your submissions to be in the “ready to publish” category. Remember life is all about perception. Every author thinks that every article he/she submits is the most erudite and perfect communication of all times and finds it difficult even to accept suggestions to resubmit the article. It is also amazing how standards change when the same individual is an author and when he is a reviewer!
Peer review is the “heart and soul of scientific publishing.” Perhaps it is not out of place to suggest that invitations to review an article submitted to Neurology India may be preferentially extended to those who have attended programmes on peer reviewing. Such refresher courses could be included in the annual CME programme or NSI conference. We need to raise the bar. Hope reviewers of this article will implement recommendations in this submission!!
The author places on record the secretarial assistance given by Ms. Lakshmi.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.