Crafting Scholarship: Writing Great Reviews

by Robert Milardo, Ph.D.
NCFR Report
Content Area
Research

Thumbnail
Robert Milardo, Ph.D.

"Crafting Scholarship" is a regular NCFR Report column by Bob Milardo, former editor of the Journal of Marriage and Family and founding editor of the Journal of Family Theory & Review. He is author of Crafting Scholarship for the Behavioral and Social Sciences: Writing, Reviewing and Editing (Fall 2014, Routledge).

Reviewing

In this new column I hope to examine the core elements of crafting scholarship for the behavioral and social sciences. The column will focus on the essentials of writing, reviewing and editing. I'll focus largely on journal articles, but much of what we explore will apply to writing grants, books, book chapters and similar forms of scholarship. As professionals we are all writers of one sort or another. Many of the columns will have a decidedly instructional focus, but some are simply intended to be playful. I love writing, speaking with writers, and thinking about how we come to the page-crafting ideas. Let me know how you like the column or if you have suggested topics.

The first few columns begin with examining the best practices in writing reviews of journal articles and related issues, like responding to editors and understanding common editorial practices. These are issues I've had experience with as an editor but they also are issues with a rich empirical literature from which we can draw insights. I'll also use the column to visit one of my favorite topics: how successful writers work. We can learn much from visiting with writers, some well-established and some new professionals. They are all a bit quirky but share some core habits that help to account for their productivity.

Writing great reviews

Writing masterful reviews is an important skill and a central part of the peer review process. I approach the topic with the implicit belief that the best reviews are critical and generative, and this applies equally well to manuscripts submitted to journals for review and to reviews of grant applications, books, or other forms of scholarship. Well-executed reviews inform an editor's decision about the deposition of a manuscript and include a more generative purpose to help authors improve their work. We can use these dual purposes to evaluate the quality of reviews and make judgments about best practices. Effective reviews benefit the profession and quality of the published work in addition to informing the reviewer's personal success in writing for publication.

To better understand the components of masterful reviews, and not so effective reviews, I analyzed the content of 111 reviews received over several years by the Journal of Family Theory & Review. At the outset, I stripped any identifying material and compared reviews with high ratings to reviews with low ratings. Reviewers for this journal, as is the case for many if not most journals in the behavioral and social sciences, are typically members of the journal's board and a legion of scholars who have volunteered to review for the journal. (To review for NCFR journals, visit the journal's website and provide the requested information.) For each review, the editorial office maintains records on the time it takes a reviewer to complete a review. I rate the quality of each review, with 1 being substandard quality and 3 being a superior review. Reviewers who consistently receive ratings of 2.0 to 3.0 make important contributions to editors and authors and are highly regarded. Many journals use similar systems to effectively manage the enterprise. The ratings are consistently applied subjective evaluations that permit a means to compare reviews, identify common characteristics, and quantify the elements of effective reviews based upon a systematic analysis.

Masterful reviews differ from the not-so-useful reviews in several ways. The best reviews are detailed and, as a result, fairly long. Quality reviews, and in this instance reviews with ratings of 2.5 to 3.0, average 865 words and some exceed 3,000 words. In comparison, reviews with low ratings (1.0 to 1.5) are substantially shorter, averaging 250 or fewer words. Our current record is a scant 15 words. Brevity may be the soul of wit but it is not the substance of masterful reviewing.

Reviews that fall short are of two varieties. One variety begins with an overall comment about the quality of the manuscript that may be very positive, or very negative, and little else.

From my perspective, this manuscript is very useful and interesting. I think it is a contribution to the field about what we might usefully be doing in incorporating same-sex parenting into our research, our theorizing, and our teaching. For me, among the many strengths of the article is the review of the etymological history of the terms like heterogamy. The review not only makes it easier to disinvest from common current usages of the terms in the field, it also helps the reader to recognize that usage of the terms has been historically diverse and culturally embedded.

I also like the piece as a teaching piece. Not only did I learn from it but I think it is well written and succinct and no doubt will be useful to students as well as more established scholars.

I did not see any technical errors in the paper except that the reference list is not in APA style.

There is nothing inherently wrong with this review. The reviewer states what she or he likes about the manuscript and how it will be useful to readers. The problem is the lack of commentary that would help the author to improve the manuscript. The review lacks depth, and this is certainly the most common feature of reviews evaluated (and rated) as less than useful. By the way, these are not in fact direct quotes of actual reviews, but rather reconstructions that seek to preserve the more important matters of tone and content, while altering the less important issues of specific topics being addressed or precise wording of the author. In this way I preserve all parties' privacy.

Let's take another example. In this instance, the review may be longer and include more direct commentary but the substance focuses on editorial issues, the use of language, and matters of grammar or style.

This was a well-written and comprehensive manuscript. I highly recommend it for publication with some changes:

  1. The first sentence of the abstract misuses the term risk factor and is likely to confuse readers.
  2. Page 3, 1st paragraph: Consider changing the two uses of the word "adolescent" to "youth."
  3. Page 3, 2nd paragraph: Please elaborate more on what specific confounding variables you are referring to.
  4. Page 3, 2nd paragraph: Spell out the Smoker et al. (2009) reference since it is the first time that it is being presented.
  5. Page 3, last paragraph: Spell out the Beatrix et al. (2013) reference since it is the first time that it is being presented.
  6. Page 4, end of page: Please elaborate more on the contextual issues you are referring to.
  7. You tend to use the word "plurality" a lot throughout the manuscript.

Overall, this manuscript will make a wonderful contribution to the field.

Like the previous example, this review begins with a positive startup, a statement that compliments some aspect of the manuscript at the outset--not all do. The reviewer concentrates on relatively minor issues regarding word choice and the style of citations. Suggestions regarding word choices and other comments that increase the clarity of an idea are useful, but such comments should not replace an evaluation of the core conceptual model framing the paper, the core theory or hypothesis, the choice and interpretation of analyses, the integration with pertinent literature, or other critical elements of any theoretical paper, review, case study, or empirical report. Incidentally, there were seven spelling errors in the original review. An occasional error is not an issue. A pattern of errors influences my confidence in the content of the review and the care in which the reviewer approached the assignment. In addition, all of the comments referred to the first seven pages of a 30-page manuscript. Our confidence in the thoroughness of the review is further eroded.

The following example is possibly one of the briefest we have received to date: "This is an excellent piece of work. I look forward to seeing it in print." In fact, there were four reviewers assigned to the manuscript: Two recommended rejection, one suggested a major revision, and one was ready to accept as is. Perhaps this later reviewer presaged the potential of the manuscript because after several revisions it developed substantially and was eventually accepted and published. The purpose of the editorial process is to provide authors feedback that will help to achieve excellence. Kudos are fine, but we need to go further if our reviews are to be helpful.

The best reviews begin with a positive startup and continue with commentary that is detailed, critical, and serves both core purposes: to inform an editor's decision, and guide authors in improving their manuscript.

Best Practice: Setting the Tone with Start-ups

Reviews can begin in several ways. The following are the types of startup phrases reviewers most typically use. As an author which do you prefer?

Harsh Start-up: "I have read better papers written by undergraduates…."

Woeful Start-up: "I found this paper disappointing…."

Neutral Start-up: "This is an interesting topic…."

Descriptive Start-up: "This paper addresses the issue of mothers' influence on fathers' participation in routine child care…."

Descriptive Start with Double Entendre: "This paper addresses the issue of mothers' influence on fathers' participation in routine child care, which is really interesting because I didn't know there was any."

Positive Start-up: "This manuscript has a number of strengths…."

Positive Start-up with Take-back: "This paper is impressive in many regards, but unfortunately reads rather like third-rate journalism."

Overall, I prefer a review that begins with a positive statement, or a neutral startup if the former is not possible. There is nearly always something positive to acknowledge even if it is only the selection of topics. My goal is always to simply acknowledge the work of authors and most importantly to motivate them to improve their work. As a reviewer or editor, I am more like an uncle (or aunt) than a parent.

In summary, the best reviews are critical and generative. They serve to help editors make informed decisions and help authors improve their work. Effective reviews begin with a positive or descriptive startup and continue that tone throughout while offering substantive critical analysis and commentary.

In the next column, I'll continue exploring the issues of tone and content as well as some additional qualities of effective reviews. In the interim, please visit the Facebook page for Crafting Scholarship and add your commentary and experiences in reviewing and being reviewed.