My friend and colleague Matt Karush has consistently argued that the present system has more merit than people like me want to recognize. I asked him to write a post describing his concerns, and so below, the first guest blogger at theaporetic.
As I have listened to some of my colleagues press for the reform of peer review in history through the adoption of new media technologies, I have been surprised by the conservatism of my own reaction. This post is an effort to think through some of my own concerns about these proposals and, optimistically, to make some small contribution to reform efforts. There is a lot of common ground between folks on all sides of these debates: we all want to strengthen the profession and to find ways of leveraging the enormous potential of the internet in order to do that.
What strikes me about arguments in support of open peer review is that they are often premised on a utopian vision of our digital future and a dystopian view of our analog present. The utopianism is neither surprising nor problematic. Proponents of change are understandably enthusiastic. Once experiments are launched, some of this enthusiasm will be tempered by experience. There is every reason to expect these models to improve through trial and error.
I am concerned, though, about the extremely negative views of current peer review practices that seem to motivate the reform movement. There is no doubt that problems exist and that travesties occur. But I think it would be a mistake to design a new system on the basis of an exaggerated critique of the old one. I’m worried, in other words, that unless we recognize what’s valuable in our current practices, we will throw the baby out with the bathwater.
So, the question is: what’s worth preserving?
Peer review, I think, has three main functions. First, it promotes intellectual exchange among scholars. This strikes me as the function that the current system performs least well but it’s also probably the least important of the three. As proponents of reform note, blind peer review short-circuits scholarly give and take by limiting critique to two or three hand-picked experts and by cutting the author out of the discussion. But if this problem is real, it’s hardly urgent; satisfying and productive intellectual exchange thrives in myriad channels outside of peer review. We all go to conferences, and we all share our work with colleagues. To judge by the acknowledgements we all write, these are profoundly important moments for scholarly communication. Incidentally, this is an area where it’s easy to see the benefits that the internet can bring: we ought to continue to multiply on-line arenas and tools (blogs, wikis, Zotero, etc) for real-time scholarly discussion, collaboration and debate. But creating those spaces doesn’t require reform of the peer review system.
Second, peer review is a system of quality control (although it’s hardly the only such system we rely on). Skeptics argue that traditional peer review does a bad job of this: senior scholars working under the protection of anonymity hijack the process in order to promote their friends and intellectual comrades at the expense of innovative or contrarian work. Frankly, this line of argument seems a little paranoid to me. I am sure that some reviewers somewhere have blocked the publication of work they didn’t like, but that’s hardly the norm. In any case, there are a ton of journals in the humanities. Articles that are rejected in one journal are published elsewhere. And thanks in part to databases and search capabilities, scholarship published in less prestigious journals does find an audience. More fundamentally, I don’t buy the idea that traditional peer review has led to a homogenization of historical scholarship. On the contrary, I think there’s a pretty deeply ingrained bias in favor of the new. Certainly, the critique that I have most often leveled at work that I have reviewed for journals and academic presses is that it doesn’t say anything new.
I want to make one other point about quality control. We historians don’t actually rely on peer review as a quality control mechanism in our own narrowly defined fields. Instead, what peer review gives us is a filter for sorting through reams of research in more-or-less distant fields. My own recent work has been on radio and the movies in 1930s Argentina. Peer review plays no role for me as a filter on scholarship in this narrow area: I read (or at least look at) everything I can find, published or not, academic or not, and I make my own judgments. However, to make a meaningful contribution, my work needs to engage with scholarship in a broad range of fields that I couldn’t possibly master: film studies, musicology, US. cultural history, the history of melodrama, etc, etc. As an aid for navigating these oceans of material, our current practices – a wide variety of journals and academic presses using traditional peer review – perform pretty well, I think.
Finally, peer review is a credentialing device. Tenure and promotion committees and university administrations use it to help determine what sort of publications ought to “count” and how much. I’ve seen the argument that this constitutes the outsourcing of judgments that departments ought to be making for themselves. But as in the case of quality control, peer review in its credentialing function really allows historians to evaluate work outside their own field, work that they are not in the best position to evaluate for themselves. And insofar as administrators are outsourcing these judgments, they are relying on peer review conducted by historians. Isn’t that outsourcing we can support? Finally, peer review merely constitutes a base line for evaluation in promotion and tenure cases; thankfully, most departments carry out in-depth reviews that go far beyond the peer review system for publishing. This is hardly a perfect system for guaranteeing meritocracy in academia, but it’s not bad.
OK, so in my view, traditional peer review accomplishes two of its three tasks fairly well. Can open peer review – the “first publish, then filter” model – do any of this better? Provided that participation rates are high, I think there is little doubt that an open peer review system would allow for more effective intellectual exchange. Regarding quality control and credentialing, I’m less convinced. Open peer review comes with its own pitfalls, notably the danger of producing what Cass Sunstein calls “information cocoons.” Once we move to “crowdsourcing” peer review, how do we avoid the formation of like-minded communities, in which the pressure to conform yields the very intellectual homogeneity reformers seek to avoid? In her sophisticated discussion of alternative peer review models, Kathleen Fitzpatrick has some convincing solutions to this problem:
Using such new technologies for purposes of deliberation, however, requires that all members of the network be equally empowered — and in fact, equally compelled — to contribute their ideas and voice their dissent, lest the network fall prey to a new mode of self-reinforcing group-think … The success of a community-based review system will hinge on the evaluation of one’s contributions to reviewing being considered as important as, if not even more important than, one’s own individual projects. Genuine peer-to-peer review will require prioritizing members’ work on behalf of the community within the community’s reward structures.
Fitzpatrick envisions a community of active scholar-reviewers commenting constructively on each others work. A complex algorithm would rate the reviewers (rather than the scholarly product), providing users with a much clearer sense of the value of a piece of scholarship than they now get from peer review.
Fitzpatrick’s vision is certainly fascinating and not without its appeal. All I would point out is that it would represent a dramatic transformation in the nature of our work. Reviewing would constitute a much more significant part of what we do – an ironic outcome, considering that critics of traditional peer review generally complain about how unpleasant and thankless the task is. Of course, in Fitzpatrick’s model, the reviewers now get “credit” for their efforts, since it is no longer anonymous and secret. But, frankly, I’m not sure I would want to make that trade.
Open peer review would seem to have the most potential in well-defined fields that have a certain critical mass of scholars: Shakespeare studies, for example. Here, you already have a networked community; the open peer review system merely uses technology to increase scholarly interaction. But most of us belong to many distinct, overlapping communities. I read and do peer reviews for Latin American Studies journals, Latin American history journals, general history journals, etc, etc. Current practices do a pretty good job of sustaining these multiple intellectual connections. Fitzpatrick’s model, given its labor intensive design, would tend to limit each of us to active participation in one community. If that community is narrow – “Argentine history,” say – then the result would be atomization; if it’s broad – “history” in general – then it would seem to flatten out important distinctions in expertise: I could comment on French history articles with the same authority as a historian of France.
Efforts to use new technology to nurture intellectual exchange among scholars are to be applauded. The techno-morons among us owe the innovators at CHNM and elsewhere a huge debt of gratitude. But I think the question of peer review is a distinct case. Efforts to reform the system ought to start with the recognition that in a lot of ways, current practices perform amazingly well.