My friend and colleague Matt Karush has consistently argued that the present system has more merit than people like me want to recognize. I asked him to write a post describing his concerns, and so below, the first guest blogger at theaporetic.
As I have listened to some of my colleagues press for the reform of peer review in history through the adoption of new media technologies, I have been surprised by the conservatism of my own reaction. This post is an effort to think through some of my own concerns about these proposals and, optimistically, to make some small contribution to reform efforts. There is a lot of common ground between folks on all sides of these debates: we all want to strengthen the profession and to find ways of leveraging the enormous potential of the internet in order to do that.
What strikes me about arguments in support of open peer review is that they are often premised on a utopian vision of our digital future and a dystopian view of our analog present. The utopianism is neither surprising nor problematic. Proponents of change are understandably enthusiastic. Once experiments are launched, some of this enthusiasm will be tempered by experience. There is every reason to expect these models to improve through trial and error.
I am concerned, though, about the extremely negative views of current peer review practices that seem to motivate the reform movement. There is no doubt that problems exist and that travesties occur. But I think it would be a mistake to design a new system on the basis of an exaggerated critique of the old one. I’m worried, in other words, that unless we recognize what’s valuable in our current practices, we will throw the baby out with the bathwater.
So, the question is: what’s worth preserving?
Peer review, I think, has three main functions. First, it promotes intellectual exchange among scholars. This strikes me as the function that the current system performs least well but it’s also probably the least important of the three. As proponents of reform note, blind peer review short-circuits scholarly give and take by limiting critique to two or three hand-picked experts and by cutting the author out of the discussion. But if this problem is real, it’s hardly urgent; satisfying and productive intellectual exchange thrives in myriad channels outside of peer review. We all go to conferences, and we all share our work with colleagues. To judge by the acknowledgements we all write, these are profoundly important moments for scholarly communication. Incidentally, this is an area where it’s easy to see the benefits that the internet can bring: we ought to continue to multiply on-line arenas and tools (blogs, wikis, Zotero, etc) for real-time scholarly discussion, collaboration and debate. But creating those spaces doesn’t require reform of the peer review system.
Second, peer review is a system of quality control (although it’s hardly the only such system we rely on). Skeptics argue that traditional peer review does a bad job of this: senior scholars working under the protection of anonymity hijack the process in order to promote their friends and intellectual comrades at the expense of innovative or contrarian work. Frankly, this line of argument seems a little paranoid to me. I am sure that some reviewers somewhere have blocked the publication of work they didn’t like, but that’s hardly the norm. In any case, there are a ton of journals in the humanities. Articles that are rejected in one journal are published elsewhere. And thanks in part to databases and search capabilities, scholarship published in less prestigious journals does find an audience. More fundamentally, I don’t buy the idea that traditional peer review has led to a homogenization of historical scholarship. On the contrary, I think there’s a pretty deeply ingrained bias in favor of the new. Certainly, the critique that I have most often leveled at work that I have reviewed for journals and academic presses is that it doesn’t say anything new.
I want to make one other point about quality control. We historians don’t actually rely on peer review as a quality control mechanism in our own narrowly defined fields. Instead, what peer review gives us is a filter for sorting through reams of research in more-or-less distant fields. My own recent work has been on radio and the movies in 1930s Argentina. Peer review plays no role for me as a filter on scholarship in this narrow area: I read (or at least look at) everything I can find, published or not, academic or not, and I make my own judgments. However, to make a meaningful contribution, my work needs to engage with scholarship in a broad range of fields that I couldn’t possibly master: film studies, musicology, US. cultural history, the history of melodrama, etc, etc. As an aid for navigating these oceans of material, our current practices – a wide variety of journals and academic presses using traditional peer review – perform pretty well, I think.
Finally, peer review is a credentialing device. Tenure and promotion committees and university administrations use it to help determine what sort of publications ought to “count” and how much. I’ve seen the argument that this constitutes the outsourcing of judgments that departments ought to be making for themselves. But as in the case of quality control, peer review in its credentialing function really allows historians to evaluate work outside their own field, work that they are not in the best position to evaluate for themselves. And insofar as administrators are outsourcing these judgments, they are relying on peer review conducted by historians. Isn’t that outsourcing we can support? Finally, peer review merely constitutes a base line for evaluation in promotion and tenure cases; thankfully, most departments carry out in-depth reviews that go far beyond the peer review system for publishing. This is hardly a perfect system for guaranteeing meritocracy in academia, but it’s not bad.
OK, so in my view, traditional peer review accomplishes two of its three tasks fairly well. Can open peer review – the “first publish, then filter” model – do any of this better? Provided that participation rates are high, I think there is little doubt that an open peer review system would allow for more effective intellectual exchange. Regarding quality control and credentialing, I’m less convinced. Open peer review comes with its own pitfalls, notably the danger of producing what Cass Sunstein calls “information cocoons.” Once we move to “crowdsourcing” peer review, how do we avoid the formation of like-minded communities, in which the pressure to conform yields the very intellectual homogeneity reformers seek to avoid? In her sophisticated discussion of alternative peer review models, Kathleen Fitzpatrick has some convincing solutions to this problem:
Using such new technologies for purposes of deliberation, however, requires that all members of the network be equally empowered — and in fact, equally compelled — to contribute their ideas and voice their dissent, lest the network fall prey to a new mode of self-reinforcing group-think . . . The success of a community-based review system will hinge on the evaluation of one’s contributions to reviewing being considered as important as, if not even more important than, one’s own individual projects. Genuine peer-to-peer review will require prioritizing members’ work on behalf of the community within the community’s reward structures.
Fitzpatrick envisions a community of active scholar-reviewers commenting constructively on each others work. A complex algorithm would rate the reviewers (rather than the scholarly product), providing users with a much clearer sense of the value of a piece of scholarship than they now get from peer review.
Fitzpatrick’s vision is certainly fascinating and not without its appeal. All I would point out is that it would represent a dramatic transformation in the nature of our work. Reviewing would constitute a much more significant part of what we do – an ironic outcome, considering that critics of traditional peer review generally complain about how unpleasant and thankless the task is. Of course, in Fitzpatrick’s model, the reviewers now get “credit” for their efforts, since it is no longer anonymous and secret. But, frankly, I’m not sure I would want to make that trade.
Open peer review would seem to have the most potential in well-defined fields that have a certain critical mass of scholars: Shakespeare studies, for example. Here, you already have a networked community; the open peer review system merely uses technology to increase scholarly interaction. But most of us belong to many distinct, overlapping communities. I read and do peer reviews for Latin American Studies journals, Latin American history journals, general history journals, etc, etc. Current practices do a pretty good job of sustaining these multiple intellectual connections. Fitzpatrick’s model, given its labor intensive design, would tend to limit each of us to active participation in one community. If that community is narrow – “Argentine history,” say – then the result would be atomization; if it’s broad – “history” in general – then it would seem to flatten out important distinctions in expertise: I could comment on French history articles with the same authority as a historian of France.
Efforts to use new technology to nurture intellectual exchange among scholars are to be applauded. The techno-morons among us owe the innovators at CHNM and elsewhere a huge debt of gratitude. But I think the question of peer review is a distinct case. Efforts to reform the system ought to start with the recognition that in a lot of ways, current practices perform amazingly well.
[…] The Aporetic: What strikes me about arguments in support of open peer review is that they are often premised on a utopian vision of our digital future and a dystopian view of our analog present. The utopianism is neither surprising nor problematic. Proponents of change are understandably enthusiastic. Once experiments are launched, some of this enthusiasm will be tempered by experience. There is every reason to expect these models to improve through trial and error. […]
Thanks for this extremely thoughtful response, Matt, much of which I actually agree with. I don’t like to think of myself as utopian about this matter—indeed, as Roy and I put it in the introduction to Digital History, we want to be between the “cyberenthusiasts” and the “technoskeptics.” The concerns you mention (e.g., the echo chamber) are quite real, and any advocate of new methods will have to deal with these concerns. Hopefully, however, as creative intellectuals we can find ways beyond the information cocoons—indeed, as Sunstein himself notes, one easy way is link to those with whom you disagree, something I’ve seen quite a bit of in the humanities blogosphere.
But I think your vision of peer review is too myopic here. An unfortunate byproduct of the current system of peer review is that it tends to lock up our scholarship through rights concessions or gating. If those exploring new methods are utopian at all, it is in the quest to have it all: maintain scholarly values through peer review, yes, but also maximal dissemination. As surveys show, the conservative reaction of which you speak, very commmon, strangely enough, in the liberal humanities, means that scholars favor maintaining the current system of peer review even if it means walling off scholarship to independent scholars or the third world.
Finally, I also don’t share your experience of interdisciplinarity at all. In my experience, open scholarship like my blog has put me into conversations with many more scholars in different disciplines than journal publishing. Perhaps that’s a function of the fields we are in, but I would like to hear others on this point.
Anyway, thanks again—you should start a blog. 😉
Peer review works very well at protecting the positions of a small number of people. I say that not to be snarky–well maybe to be a little snarky–but to suggest that its a structure designed for a different era. And saying it works well is kind of like saying the Model T assembly line works well. It works well, but the thing it works well at producing is worth some critical analysis.
It works well at protecting me, the peer reviewer, from the author’s wrath if I say jerky things. It works well at shielding the editor from the author’s wrath if he/she says no to publication. And it works well at producing a professionally recognizable product, which is good and bad.
But put that aside: I’d still argue the model I suggested, a very conservative model, kept all the benefits of peer review intact but upped the community part. How is that at all bad?
It’s true we have other outlets for engagement with peers, but I’d agree with Dan that “maximal dissemination” is a good goal
Thanks, Matt, for doing a great job of articulating the strengths and weaknesses of the current system of peer review versus the “first publish, then filter” system. But to speak to Dan’s point: Why should “maximal dissemination” be valorized above other goals for scholarship? Some kinds of research fields and questions are more likely to benefit from wide review and maximal dissemination and others are not. For example, scholarly articles that analyze the exact timing of the transition from white indentured servitude to black slavery in 17th century VA require the investigation of complicated primary sources and obscure legal issues. Only a small group of specialists have the knowledge to address (or are likely to care). In contrast, an article exploring the relationship between slavery and freedom in the American Revolution might benefit from wider dissemination and exposure to a wider audience. There’s no “one size fits all” solution to this issue. But to go back to Mike’s initial question in his previous blog on this issue: the American Historical Review, which claims to speak to all historians in all fields of history–might actually benefit from adopting some version of the “publish, then filter” approach. It would push the AHR to publish articles of more general interest to historians and open the articles to responses from knowledgable historians in other fields. (See–you’ve made a convert!)
LOL!
I think that there IS a small but passionate audience for the transition from white servitude to slavery, and it’s an audience that loves documents: genealogists. They have a fondness for detail, a zeal for collecting, and a fascination with the past. What group could benefit more from being brought into the historical process?
Any time you want to make a post, let me know!
Leaving aside maximal dissemination for a bit (let’s focus on us scholars! forget the little people!) isn’t what’s happening here instructive? We can actually have an interaction about a piece of writing (Matt’s), and that interaction happens in such a way that others can join in and refer to it at a permanent home (a URL). This is one of many advantages of open, online publishing that traditional peer-review publications eschew.
How many journals have comments on articles? Why not? What I’ve called “recursive review” (rather than just pre- or post-peer review) is one of the critical characteristics of well-designed digital scholarship. I happen to think that this is just one way in which the traditional method is actually less accountable and scholarly than newer methods. A journal article is inert.
Dan, agreed on the importance of comments– not only for following up on a scholar’s arguments but also for exploring related issues. It’s a little bit weird that an journal which mentions a digital source collection’s omissions in passing has no obvious mechanism for following up on that item. It’s not a big enough issue to warrant a journal article in its own right, and relatively few humanities journals that I’ve ever read even maintain a letters-to-the-editors section.
(I’ve followed up on the digital-collections omission mentioned in that blog post; there’s an interesting technical explanation behind it, and I’ll be writing a followup post in the next week or so. Will either item be easily findable by the readers of that journal article? Probably not, but that doesn’t mean it wasn’t worth chasing down.)
Dan, yes, we’ve had a good exchange. But what does it amount to? We’ve reached no conclusions which a subsequent scholar can point to as a definitive “interpretation” and which they can use as a point of departure for their own work. And would an article, say, by Mike O’Malley on the variations in paper currency as a medium of exchange in the 1840s be likely to generate much debate, even among scholars? Common-place, an online journal for early American historians, doesn’t produce much in the way of lively exchange, even though it’s designed to do that. A journal article is only inert until subsequent scholars bring it to life through discussion and citation. At that level, the only difference between online and printed articles is the speed of the recorded responses. And there is the danger there that glibness may replace thoughtfulness.
Common-Place is in some ways a great journal but it’s a bad example. It doesn’t allow discussion of articles any more, and it used to confine to a forum: that was an experiment that failed as they often do.
What have we accomplished? Well, we’ve found some common ground. I’ve got a much better sense of what I think an all digital journal in the humanities should or could look like.
So next step: the journal. Dan’s working on it, and I hope to be as well. Will we get lively discussion about an article about bank notes in the 1840s? Well I just gave a talk about that exact subject in Cincinnati. The conversation was pretty lively. I made some blog posts on related subjects and some of them have gotten thousands of readers.
But it doesn’t have to be grand. What conferences do you like best? The small conferences, where a small group of interested people discuss common interests. Outcomes don’t have to be grand to be better.
I’ve seen plenty of blog posts that have become definitive interpretations and points of departure for others. To give just one example, Jay Rosen’s blog post on “The View from Nowhere” has driven much of the academic theory and discussion of journalistic objectivity in the last year. And he continues to develop the ideas via blog posts at PressThink and through his Twitter account. It is true that scholars can bring journal articles to life, but it seems to me the vitality of online work is more likely, since the author (like Rosen) can (and often feels obliged) to answer criticisms and develop themes further over time. It’s not as relaxing, however, as sending off your final edits to a journal and being done with it.
Yes to recursive review! Yes to maximal dissemination! (You digital guys have the coolest slogans.) Both of these things can (and should) happen. I just they think they should happen after peer review. But obviously, if there is some necessary connection between peer review and pay walls that keep out the third world, then I’ll have to stop defending peer review.
Basically, I’m just suggesting that when we discuss peer review, we shouldn’t automatically treat it as part of the digital vs. analog debate. We can imagine a digital world with old-fashioned peer review.
My utopian scenario: all print journals go digital and take advantage of the possibilities that enables: recursive review, real-time debate, etc. They are distributed free of charge (ie not owned by academic presses). But they continue to use something like traditional peer review. It’s “filter, publish, then disseminate and debate.” See: I’m no good at slogans.
But Matt, don’t you sometimes feel that writing a peer review is like dropping something down a well? And NOT hearing a splash?
And also, we all filter first and always will, most likely. We filter when we give our work to colleagues before sending it off, or dragoon the spouse into reading.
I can see your point, in that I rarely have to go outside the US: there’s more work than I can keep up with. Your work requires articles you can trust and the apparatus of blind peer review makes that easier. I’m not convinced that there aren’t ways to make it easier still
Thanks — this is useful. I hope you’ll publish something about this, focusing on the cocoons bit. Are you following the altmetrics community?
excellent publish, very informative. I ponder why the other experts of this sector do
not realize this. You must continue your writing.
I am sure, you’ve a great readers’ base already!
I am regular visitor, how are you everybody?
This paragraph posted at this website is genuinely pleasant.