Crafting Scholarship: Masterful Reviews
"Crafting Scholarship" is a regular NCFR Report column by Robert Milardo, former editor of the Journal of Marriage and Family and founding editor of the Journal of Family Theory & Review. He is author of Crafting Scholarship for the Behavioral and Social Sciences: Writing, Reviewing and Editing (Fall 2014, Routledge).
In this article, we consider selected best practices in reviewing "substandard," and more developed, submissions to journals. In both instances, the purpose of a review is to guide an editor's decision and authors' future revisions. Understanding the elements of useful reviews informs the overall quality of reviews and helps authors in their initial preparation of manuscripts.
Reviewing elementary or substandard submissions
Occasionally, editors receive manuscripts for consideration that seem to have little chance of being published. Such manuscripts may be deficient in a number of ways. They may lack a clear understanding of the available literature, lack the data appropriate to the intended purpose, misinterpret analyses, or be otherwise substandard in fundamental ways. In some instances, an editor or editorial team will review submissions and reject a substantial percentage outright and without further review. For many journals, editors rely on the peer review process for informing nearly all decisions, including instances where at first glance a manuscript appears substandard.
Reviewers vary in how they approach manuscripts that they judge as inappropriate for the journal. They may pen a very brief review that states their evaluation and little more. An appropriate brief review might read: "This manuscript investigates an important topic in many regards. Work in this area is difficult; nonetheless, a true test of the hypotheses requires more direct measures of the key variables and data from both relationship partners." The reviewer should add a bit more detail on why the measures and sample are deficient, but not much else. I prefer reviews that close with a soft landing encouraging the author to continue their work, for example, "I do hope the authors find these comments helpful in their continuing work."
Some reviewers, even in the case of a decidedly marginal manuscript, provide substantial feedback to authors and in doing so invest much of their time. Either approach (i.e., a brief rejection note or a longer one with more thorough feedback) is appropriate when the recommendation is to forgo pursuing a manuscript. The only clear responsibility of the reviewer is to distinctly state the reasons for his or her evaluation.
Most submissions are not entirely substandard and often are accomplished in some regards. The literature review may be especially strong and well written even if the data were inappropriate or ineffectively analyzed. In these cases, longer reviews are not optional; they are required.
There are instances when substantial feedback to authors is helpful, benefiting the author and potentially the field more generally. Sarah Schoppe-Sullivan shared such a story. I asked her if she had recently had a particularly positive experience in having her work reviewed. Here is her response:
Yes. Recently my students and I submitted a paper to a journal that was rejected quickly, but the reviews were so helpful and constructive that we used them to completely revise the paper. When we sent it to the next journal, it got a very positive response and request for a revision and resubmission. The resubmission was accepted [by the Journal of Family Psychology]. So even though the original submission was rejected, the reviews were helpful, and it was a great experience for the students. They could clearly see that the paper was so much better after we made the revision. That's not the only positive experience I've had, but it's one of the most positive. It was the way things are supposed to work.
In this example, the initial reviewers were not obliged to provide suggestions for improving the manuscript but, because they had done so, Sarah and her colleagues took advantage of the suggestions, revised the paper, and sent it off to a second and very well-regarded journal, where it was finally accepted. Both the authors and the field were served well. Writing detailed reviews can be a service to our colleagues, informing the decisions of editors and directing authors in ways to improve their work. In the case of substandard submissions, reviewers may not always have the time to invest in a comprehensive review, but when they do it can generate a string of positive consequences.
Let's consider the core of a review that is helpful to editors and authors. After all, writing technical articles is no easy task, and having peers read our work and provide suggestions for improvement is a significant part of crafting great work.
The best reviews, or what we might address as masterful reviews, are detailed and evaluate each of the core parts of a manuscript. Throughout a manuscript, reviewers, like readers, expect some precision in the use of language. For instance, the following statement sounds intriguing in principle but is terribly unhelpful in particulars: "Social psychological theory suggests a certain relational outcome in family communication." Just what is social psychological theory? Quite possibly the author had something specific in mind. The reviewer's mission is to encourage the full development of the underlying argument and the degree of precision in the use of language. In this case, we need a clear explanation of the term social psychological theory and probably a more particular explanation of the precise theory the author had in mind.
In this way, great reviews help to push advancements, for instance, pushing theory through a series of challenging questions, as in the following example: "Can you move beyond Erikson's fairly simple notion of stages? Are the developmental trajectories of secure and avoidant children likely to be unique? Can you extend these arguments?"
Great reviews often request added refinements to existing arguments or analyses. "I would like to see the author buttress her arguments with some mention of the earlier work by Larry Kurdek on gay and lesbian couples." Or in regard to a series of regressions, a reviewer might ask for a complementary comparison of means. Reviews often call for more nuanced arguments and in this way encourage authors to advance their thinking and perhaps their underlying theory or the complexity of design, measurement, or analyses. For example, in a literature review one author noted how cohabitation before marriage raises the risk of later divorce, relative to couples who do not cohabitate before marrying. This is true, but more recent work by Jay Teachman and others adds a bit of nuance to this long-standing finding in that the effect is not consistent across all cohabiting couples.
Effective reviews prioritize commentary
The most useful reviews center on a few core issues, and the overall clarity of the presentation is among the most important. Reviewers expect a clear, unambiguous statement of the purpose of a manuscript, and they expect this within the first few paragraphs. Heather Helms, an accomplished author and reviewer, commented on this issue: "In reviewing an article, if I don't know by the end of the third paragraph what the study is about I am not optimistic about the paper."
Great reviews, and the reviewers who write them, look for the clarity of the breadcrumb trail. All elements of the manuscript need to be clearly linked: the introduction clear and concise, the literature review appropriate to the research questions, the data and analyses pertinent to the questions being asked, and the conclusions clearly derivative of the analyses. There should be a "clear conceptual pathway through the thicket of ideas," as one reviewer noted. Often this is evidenced by the author's selection of headings.
Reviewers may usefully question the scope of a manuscript and in this way question the overall contribution. For instance, in a paper on intervention programs for violent adolescent offenders, a reviewer may question the exclusion of programs directed at adults, or the importance of distinguishing programs that are designed specifically for young women rather than young men. For authors, this suggests that they should anticipate the issues a reviewer is apt to raise and provide explanations for the overall design and any likely concerns.
Reviewers can benefit authors when they read rather literally for clarity and consistency of language. The clarity of any paper is largely contingent on the clarity of key concepts. Oddly enough, for instance, the term family is used imprecisely, sometimes referring to spouses and their children or the individuals living in a single household and sometimes referring to multiple households and a wider tracing of people related by birth, marriage or strong positive sentiments (e.g., fictive kin). In this case, even the most common terms (i.e., family) may need clarification. Authors can assemble manuscripts with an eye toward clearly defining and consistently using key concepts.
Identifying matters of style
There are a variety of additional issues to consider as well, largely minor concerns in that they are relatively easy to remedy. For example, as an editor I am not terribly concerned with minor lapses in style or the occasional typographical or grammatical error. Once a manuscript is accepted, editors of major journals have professional copyeditors review the manuscript for all of these issues. Editors vary in their concern about such issues and the degree of support they receive from their publishers (e.g., to pay for expert copyediting). Journals sponsored by professional organizations (like NCFR) are typically well supported in these regards, and editors and reviewers can focus on substantive issues where their expertise resides. Reviewers, at their own discretion, may include suggestions for authors regarding minor issues of style; such comments are typically included at the end of a review.
In contrast, significant lapses in preparation of a manuscript raise questions about the things that really do matter: How carefully did the author construct this particular research report, case study, review, or theory paper? I recently received a manuscript that had a glaring grammatical error in the title, and not fewer than eight simple errors on the first page. If the authors can't proofread their first page, what confidence can I have that their data are clean? A manuscript replete with simple errors invites questions from editors and reviewers regarding more substantive issues, such as the quality of the literature review, design, measurement, analysis, and interpretation.
In short, a few lapses in style or presentation are minor issues; anything more than that is a distraction for reviewers.
Reviewers often suggest additional work they regard as relevant that was not cited originally by authors. This can be quite helpful when it services our mutual goal to produce the best possible scholarship. Given the complexity of publishing and the breadth of sophistication needed to assemble an article, it is not surprising when an author misses what could be a useful source. The simple omission of a few pertinent publications is important to note but has little impact on the editorial outcome, and it shouldn't. There are exceptions.
More significant are occasions when substantial work is omitted and, in particular, work that might have informed an initial research design, the development of a theoretical model, or a policy implication. In these cases, the omission can figure prominently in the reviewer's recommendation and, eventually, that of the editor. Given the sophistication of methods for searching literatures, authors can largely avoid such problems with a bit of effort. But what if a scholarâ€”for instance, a developmental psychologistâ€”is working in the area of adolescent siblings and in an otherwise-talented report misses pertinent material published in an education journal? This kind of omission happens quite regularly, and the questions are was the theoretical modeling or the manuscript's research design affected and can the omission be corrected easily?
In journals representing multiple disciplines and diverse readerships, there is an expectation that authors will be familiar with work in neighboring disciplines. The journals sponsored by inherently multidisciplinary disciplines like education, social work, communications, or family science are unique in that there is an expectation that authors write for multidisciplinary audiences, at least up to a point. Some disciplines are implicitly regarded as too distant. We would not expect someone writing on kinship in family science to cite work in comparative biology (e.g., sisterhood in rodents). In fact, animal studies, including work with primates, is rarely addressed in fields that regularly publish work on human relationships and families, although it would be useful if there were more cross-disciplinary fertilization.
At times, reviewers face thorny ethical concerns. A reviewer might recommend a variety of work to an author, including the reviewer's own publications. A reviewer may prefer a more conservative position and communicate the potential relevance of a self-citation to the editor and let her decide if it is appropriate to pass the recommendation along to an author. In nearly all cases, a light touch is appropriate, and authors should have the prerogative to include suggested references or not. In the case of the latter, authors can communicate the rationale for their decision to exclude a reference in the correspondence that accompanies a revision. At the very least, the editor and reviewers need to know the issue was considered thoughtfully.
In the end, write the reviews you would like to receive: encouraging, generative, and critical. Our goal is to create great science in a supportive community.
Please share your experiences in reviewing and being reviewed on our Facebook page "Crafting Scholarship."