Guest Post: Defending Current Practice

My friend and col­league Matt Karush has con­sis­tently argued that the present sys­tem has more merit than peo­ple like me want to rec­og­nize. I asked him to write a post describ­ing his con­cerns, and so below, the first guest blog­ger at thea­poretic.

 

As I have lis­tened to some of my col­leagues press for the reform of peer review in his­tory through the adop­tion of new media tech­nolo­gies, I have been sur­prised by the con­ser­vatism of my own reac­tion. This post is an effort to think through some of my own con­cerns about these pro­pos­als and, opti­misti­cally, to make some small con­tri­bu­tion to reform efforts. There is a lot of com­mon ground between folks on all sides of these debates: we all want to strengthen the pro­fes­sion and to find ways of lever­ag­ing the enor­mous poten­tial of the inter­net in order to do that.

What strikes me about argu­ments in sup­port of open peer review is that they are often premised on a utopian vision of our dig­i­tal future and a dystopian view of our ana­log present. The utopi­anism is nei­ther sur­pris­ing nor prob­lem­atic. Pro­po­nents of change are under­stand­ably enthu­si­as­tic. Once exper­i­ments are launched, some of this enthu­si­asm will be tem­pered by expe­ri­ence. There is every rea­son to expect these mod­els to improve through trial and error.

I am con­cerned, though, about the extremely neg­a­tive views of cur­rent peer review prac­tices that seem to moti­vate the reform move­ment. There is no doubt that prob­lems exist and that trav­es­ties occur. But I think it would be a mis­take to design a new sys­tem on the basis of an exag­ger­ated cri­tique of the old one. I’m wor­ried, in other words, that unless we rec­og­nize what’s valu­able in our cur­rent prac­tices, we will throw the baby out with the bathwater.

So, the ques­tion is: what’s worth preserving?

Peer review, I think, has three main func­tions. First, it pro­motes intel­lec­tual exchange among schol­ars. This strikes me as the func­tion that the cur­rent sys­tem per­forms least well but it’s also prob­a­bly the least impor­tant of the three. As pro­po­nents of reform note, blind peer review short-circuits schol­arly give and take by lim­it­ing cri­tique to two or three hand-picked experts and by cut­ting the author out of the dis­cus­sion. But if this prob­lem is real, it’s hardly urgent; sat­is­fy­ing and pro­duc­tive intel­lec­tual exchange thrives in myr­iad chan­nels out­side of peer review. We all go to con­fer­ences, and we all share our work with col­leagues. To judge by the acknowl­edge­ments we all write, these are pro­foundly impor­tant moments for schol­arly com­mu­ni­ca­tion. Inci­den­tally, this is an area where it’s easy to see the ben­e­fits that the inter­net can bring: we ought to con­tinue to mul­ti­ply on-line are­nas and tools (blogs, wikis, Zotero, etc) for real-time schol­arly dis­cus­sion, col­lab­o­ra­tion and debate. But cre­at­ing those spaces doesn’t require reform of the peer review system.

Sec­ond, peer review is a sys­tem of qual­ity con­trol (although it’s hardly the only such sys­tem we rely on). Skep­tics argue that tra­di­tional peer review does a bad job of this: senior schol­ars work­ing under the pro­tec­tion of anonymity hijack the process in order to pro­mote their friends and intel­lec­tual com­rades at the expense of inno­v­a­tive or con­trar­ian work. Frankly, this line of argu­ment seems a lit­tle para­noid to me. I am sure that some review­ers some­where have blocked the pub­li­ca­tion of work they didn’t like, but that’s hardly the norm. In any case, there are a ton of jour­nals in the human­i­ties. Arti­cles that are rejected in one jour­nal are pub­lished else­where. And thanks in part to data­bases and search capa­bil­i­ties, schol­ar­ship pub­lished in less pres­ti­gious jour­nals does find an audi­ence. More fun­da­men­tally, I don’t buy the idea that tra­di­tional peer review has led to a homog­e­niza­tion of his­tor­i­cal schol­ar­ship. On the con­trary, I think there’s a pretty deeply ingrained bias in favor of the new. Cer­tainly, the cri­tique that I have most often lev­eled at work that I have reviewed for jour­nals and aca­d­e­mic presses is that it doesn’t say any­thing new.

I want to make one other point about qual­ity con­trol. We his­to­ri­ans don’t actu­ally rely on peer review as a qual­ity con­trol mech­a­nism in our own nar­rowly defined fields. Instead, what peer review gives us is a fil­ter for sort­ing through reams of research in more-or-less dis­tant fields. My own recent work has been on radio and the movies in 1930s Argentina. Peer review plays no role for me as a fil­ter on schol­ar­ship in this nar­row area: I read (or at least look at) every­thing I can find, pub­lished or not, aca­d­e­mic or not, and I make my own judg­ments. How­ever, to make a mean­ing­ful con­tri­bu­tion, my work needs to engage with schol­ar­ship in a broad range of fields that I couldn’t pos­si­bly mas­ter: film stud­ies, musi­col­ogy, US. cul­tural his­tory, the his­tory of melo­drama, etc, etc. As an aid for nav­i­gat­ing these oceans of mate­r­ial, our cur­rent prac­tices – a wide vari­ety of jour­nals and aca­d­e­mic presses using tra­di­tional peer review – per­form pretty well, I think.

Finally, peer review is a cre­den­tial­ing device. Tenure and pro­mo­tion com­mit­tees and uni­ver­sity admin­is­tra­tions use it to help deter­mine what sort of pub­li­ca­tions ought to “count” and how much. I’ve seen the argu­ment that this con­sti­tutes the out­sourc­ing of judg­ments that depart­ments ought to be mak­ing for them­selves. But as in the case of qual­ity con­trol, peer review in its cre­den­tial­ing func­tion really allows his­to­ri­ans to eval­u­ate work out­side their own field, work that they are not in the best posi­tion to eval­u­ate for them­selves. And inso­far as admin­is­tra­tors are out­sourc­ing these judg­ments, they are rely­ing on peer review con­ducted by his­to­ri­ans. Isn’t that out­sourc­ing we can sup­port? Finally, peer review merely con­sti­tutes a base line for eval­u­a­tion in pro­mo­tion and tenure cases; thank­fully, most depart­ments carry out in-depth reviews that go far beyond the peer review sys­tem for pub­lish­ing. This is hardly a per­fect sys­tem for guar­an­tee­ing mer­i­toc­racy in acad­e­mia, but it’s not bad.

OK, so in my view, tra­di­tional peer review accom­plishes two of its three tasks fairly well. Can open peer review – the “first pub­lish, then fil­ter” model – do any of this bet­ter? Pro­vided that par­tic­i­pa­tion rates are high, I think there is lit­tle doubt that an open peer review sys­tem would allow for more effec­tive intel­lec­tual exchange. Regard­ing qual­ity con­trol and cre­den­tial­ing, I’m less con­vinced. Open peer review comes with its own pit­falls, notably the dan­ger of pro­duc­ing what Cass Sun­stein calls “infor­ma­tion cocoons.” Once we move to “crowd­sourc­ing” peer review, how do we avoid the for­ma­tion of like-minded com­mu­ni­ties, in which the pres­sure to con­form yields the very intel­lec­tual homo­gene­ity reform­ers seek to avoid? In her sophis­ti­cated dis­cus­sion of alter­na­tive peer review mod­els, Kath­leen Fitz­patrick has some con­vinc­ing solu­tions to this problem:

Using such new tech­nolo­gies for pur­poses of delib­er­a­tion, how­ever, requires that all mem­bers of the net­work be equally empow­ered — and in fact, equally com­pelled — to con­tribute their ideas and voice their dis­sent, lest the net­work fall prey to a new mode of self-reinforcing group-think … The suc­cess of a community-based review sys­tem will hinge on the eval­u­a­tion of one’s con­tri­bu­tions to review­ing being con­sid­ered as impor­tant as, if not even more impor­tant than, one’s own indi­vid­ual projects. Gen­uine peer-to-peer review will require pri­or­i­tiz­ing mem­bers’ work on behalf of the com­mu­nity within the community’s reward structures.

 

Fitz­patrick envi­sions a com­mu­nity of active scholar-reviewers com­ment­ing con­struc­tively on each oth­ers work. A com­plex algo­rithm would rate the review­ers (rather than the schol­arly prod­uct), pro­vid­ing users with a much clearer sense of the value of a piece of schol­ar­ship than they now get from peer review.

Fitzpatrick’s vision is cer­tainly fas­ci­nat­ing and not with­out its appeal. All I would point out is that it would rep­re­sent a dra­matic trans­for­ma­tion in the nature of our work. Review­ing would con­sti­tute a much more sig­nif­i­cant part of what we do – an ironic out­come, con­sid­er­ing that crit­ics of tra­di­tional peer review gen­er­ally com­plain about how unpleas­ant and thank­less the task is. Of course, in Fitzpatrick’s model, the review­ers now get “credit” for their efforts, since it is no longer anony­mous and secret. But, frankly, I’m not sure I would want to make that trade.

Open peer review would seem to have the most poten­tial in well-defined fields that have a cer­tain crit­i­cal mass of schol­ars: Shake­speare stud­ies, for exam­ple. Here, you already have a net­worked com­mu­nity; the open peer review sys­tem merely uses tech­nol­ogy to increase schol­arly inter­ac­tion. But most of us belong to many dis­tinct, over­lap­ping com­mu­ni­ties. I read and do peer reviews for Latin Amer­i­can Stud­ies jour­nals, Latin Amer­i­can his­tory jour­nals, gen­eral his­tory jour­nals, etc, etc. Cur­rent prac­tices do a pretty good job of sus­tain­ing these mul­ti­ple intel­lec­tual con­nec­tions. Fitzpatrick’s model, given its labor inten­sive design, would tend to limit each of us to active par­tic­i­pa­tion in one com­mu­nity. If that com­mu­nity is nar­row – “Argen­tine his­tory,” say – then the result would be atom­iza­tion; if it’s broad – “his­tory” in gen­eral – then it would seem to flat­ten out impor­tant dis­tinc­tions in exper­tise: I could com­ment on French his­tory arti­cles with the same author­ity as a his­to­rian of France.

Efforts to use new tech­nol­ogy to nur­ture intel­lec­tual exchange among schol­ars are to be applauded. The techno-morons among us owe the inno­va­tors at CHNM and else­where a huge debt of grat­i­tude. But I think the ques­tion of peer review is a dis­tinct case. Efforts to reform the sys­tem ought to start with the recog­ni­tion that in a lot of ways, cur­rent prac­tices per­form amaz­ingly well.

 

13 Comments

  • […] The Aporetic: What strikes me about argu­ments in sup­port of open peer review is that they are often premised on a utopian vision of our dig­i­tal future and a dystopian view of our ana­log present. The utopi­anism is nei­ther sur­pris­ing nor prob­lem­atic. Pro­po­nents of change are under­stand­ably enthu­si­as­tic. Once exper­i­ments are launched, some of this enthu­si­asm will be tem­pered by expe­ri­ence. There is every rea­son to expect these mod­els to improve through trial and error. […]

  • Thanks for this extremely thought­ful response, Matt, much of which I actu­ally agree with. I don’t like to think of myself as utopian about this matter—indeed, as Roy and I put it in the intro­duc­tion to Dig­i­tal His­tory, we want to be between the “cyber­en­thu­si­asts” and the “tech­noskep­tics.” The con­cerns you men­tion (e.g., the echo cham­ber) are quite real, and any advo­cate of new meth­ods will have to deal with these con­cerns. Hope­fully, how­ever, as cre­ative intel­lec­tu­als we can find ways beyond the infor­ma­tion cocoons—indeed, as Sun­stein him­self notes, one easy way is link to those with whom you dis­agree, some­thing I’ve seen quite a bit of in the human­i­ties blogosphere.

    But I think your vision of peer review is too myopic here. An unfor­tu­nate byprod­uct of the cur­rent sys­tem of peer review is that it tends to lock up our schol­ar­ship through rights con­ces­sions or gat­ing. If those explor­ing new meth­ods are utopian at all, it is in the quest to have it all: main­tain schol­arly val­ues through peer review, yes, but also max­i­mal dis­sem­i­na­tion. As sur­veys show, the con­ser­v­a­tive reac­tion of which you speak, very com­m­mon, strangely enough, in the lib­eral human­i­ties, means that schol­ars favor main­tain­ing the cur­rent sys­tem of peer review even if it means walling off schol­ar­ship to inde­pen­dent schol­ars or the third world.

    Finally, I also don’t share your expe­ri­ence of inter­dis­ci­pli­nar­ity at all. In my expe­ri­ence, open schol­ar­ship like my blog has put me into con­ver­sa­tions with many more schol­ars in dif­fer­ent dis­ci­plines than jour­nal pub­lish­ing. Per­haps that’s a func­tion of the fields we are in, but I would like to hear oth­ers on this point.

    Any­way, thanks again—you should start a blog. ;-)

  • Peer review works very well at pro­tect­ing the posi­tions of a small num­ber of peo­ple. I say that not to be snarky–well maybe to be a lit­tle snarky–but to sug­gest that its a struc­ture designed for a dif­fer­ent era. And say­ing it works well is kind of like say­ing the Model T assem­bly line works well. It works well, but the thing it works well at pro­duc­ing is worth some crit­i­cal analysis.

    It works well at pro­tect­ing me, the peer reviewer, from the author’s wrath if I say jerky things. It works well at shield­ing the edi­tor from the author’s wrath if he/she says no to pub­li­ca­tion. And it works well at pro­duc­ing a pro­fes­sion­ally rec­og­niz­able prod­uct, which is good and bad.

    But put that aside: I’d still argue the model I sug­gested, a very con­ser­v­a­tive model, kept all the ben­e­fits of peer review intact but upped the com­mu­nity part. How is that at all bad?

    It’s true we have other out­lets for engage­ment with peers, but I’d agree with Dan that “max­i­mal dis­sem­i­na­tion” is a good goal

  • Thanks, Matt, for doing a great job of artic­u­lat­ing the strengths and weak­nesses of the cur­rent sys­tem of peer review ver­sus the “first pub­lish, then fil­ter” sys­tem. But to speak to Dan’s point: Why should “max­i­mal dis­sem­i­na­tion” be val­orized above other goals for schol­ar­ship? Some kinds of research fields and ques­tions are more likely to ben­e­fit from wide review and max­i­mal dis­sem­i­na­tion and oth­ers are not. For exam­ple, schol­arly arti­cles that ana­lyze the exact tim­ing of the tran­si­tion from white inden­tured servi­tude to black slav­ery in 17th cen­tury VA require the inves­ti­ga­tion of com­pli­cated pri­mary sources and obscure legal issues. Only a small group of spe­cial­ists have the knowl­edge to address (or are likely to care). In con­trast, an arti­cle explor­ing the rela­tion­ship between slav­ery and free­dom in the Amer­i­can Rev­o­lu­tion might ben­e­fit from wider dis­sem­i­na­tion and expo­sure to a wider audi­ence. There’s no “one size fits all” solu­tion to this issue. But to go back to Mike’s ini­tial ques­tion in his pre­vi­ous blog on this issue: the Amer­i­can His­tor­i­cal Review, which claims to speak to all his­to­ri­ans in all fields of history–might actu­ally ben­e­fit from adopt­ing some ver­sion of the “pub­lish, then fil­ter” approach. It would push the AHR to pub­lish arti­cles of more gen­eral inter­est to his­to­ri­ans and open the arti­cles to responses from knowl­edgable his­to­ri­ans in other fields. (See–you’ve made a convert!)

  • LOL!

    I think that there IS a small but pas­sion­ate audi­ence for the tran­si­tion from white servi­tude to slav­ery, and it’s an audi­ence that loves doc­u­ments: geneal­o­gists. They have a fond­ness for detail, a zeal for col­lect­ing, and a fas­ci­na­tion with the past. What group could ben­e­fit more from being brought into the his­tor­i­cal process?

    Any time you want to make a post, let me know!

  • Leav­ing aside max­i­mal dis­sem­i­na­tion for a bit (let’s focus on us schol­ars! for­get the lit­tle peo­ple!) isn’t what’s hap­pen­ing here instruc­tive? We can actu­ally have an inter­ac­tion about a piece of writ­ing (Matt’s), and that inter­ac­tion hap­pens in such a way that oth­ers can join in and refer to it at a per­ma­nent home (a URL). This is one of many advan­tages of open, online pub­lish­ing that tra­di­tional peer-review pub­li­ca­tions eschew.

    How many jour­nals have com­ments on arti­cles? Why not? What I’ve called “recur­sive review” (rather than just pre– or post-peer review) is one of the crit­i­cal char­ac­ter­is­tics of well-designed dig­i­tal schol­ar­ship. I hap­pen to think that this is just one way in which the tra­di­tional method is actu­ally less account­able and schol­arly than newer meth­ods. A jour­nal arti­cle is inert.

  • Dan, agreed on the impor­tance of com­ments– not only for fol­low­ing up on a scholar’s argu­ments but also for explor­ing related issues. It’s a lit­tle bit weird that an jour­nal which men­tions a dig­i­tal source collection’s omis­sions in pass­ing has no obvi­ous mech­a­nism for fol­low­ing up on that item. It’s not a big enough issue to war­rant a jour­nal arti­cle in its own right, and rel­a­tively few human­i­ties jour­nals that I’ve ever read even main­tain a letters-to-the-editors section.

    (I’ve fol­lowed up on the digital-collections omis­sion men­tioned in that blog post; there’s an inter­est­ing tech­ni­cal expla­na­tion behind it, and I’ll be writ­ing a fol­lowup post in the next week or so. Will either item be eas­ily find­able by the read­ers of that jour­nal arti­cle? Prob­a­bly not, but that doesn’t mean it wasn’t worth chas­ing down.)

  • Dan, yes, we’ve had a good exchange. But what does it amount to? We’ve reached no con­clu­sions which a sub­se­quent scholar can point to as a defin­i­tive “inter­pre­ta­tion” and which they can use as a point of depar­ture for their own work. And would an arti­cle, say, by Mike O’Malley on the vari­a­tions in paper cur­rency as a medium of exchange in the 1840s be likely to gen­er­ate much debate, even among schol­ars? Common-place, an online jour­nal for early Amer­i­can his­to­ri­ans, doesn’t pro­duce much in the way of lively exchange, even though it’s designed to do that. A jour­nal arti­cle is only inert until sub­se­quent schol­ars bring it to life through dis­cus­sion and cita­tion. At that level, the only dif­fer­ence between online and printed arti­cles is the speed of the recorded responses. And there is the dan­ger there that glib­ness may replace thoughtfulness.

  • Common-Place is in some ways a great jour­nal but it’s a bad exam­ple. It doesn’t allow dis­cus­sion of arti­cles any more, and it used to con­fine to a forum: that was an exper­i­ment that failed as they often do.

    What have we accom­plished? Well, we’ve found some com­mon ground. I’ve got a much bet­ter sense of what I think an all dig­i­tal jour­nal in the human­i­ties should or could look like.

    So next step: the jour­nal. Dan’s work­ing on it, and I hope to be as well. Will we get lively dis­cus­sion about an arti­cle about bank notes in the 1840s? Well I just gave a talk about that exact sub­ject in Cincin­nati. The con­ver­sa­tion was pretty lively. I made some blog posts on related sub­jects and some of them have got­ten thou­sands of readers.

    But it doesn’t have to be grand. What con­fer­ences do you like best? The small con­fer­ences, where a small group of inter­ested peo­ple dis­cuss com­mon inter­ests. Out­comes don’t have to be grand to be better.

  • I’ve seen plenty of blog posts that have become defin­i­tive inter­pre­ta­tions and points of depar­ture for oth­ers. To give just one exam­ple, Jay Rosen’s blog post on “The View from Nowhere” has dri­ven much of the aca­d­e­mic the­ory and dis­cus­sion of jour­nal­is­tic objec­tiv­ity in the last year. And he con­tin­ues to develop the ideas via blog posts at Press­Think and through his Twit­ter account. It is true that schol­ars can bring jour­nal arti­cles to life, but it seems to me the vital­ity of online work is more likely, since the author (like Rosen) can (and often feels obliged) to answer crit­i­cisms and develop themes fur­ther over time. It’s not as relax­ing, how­ever, as send­ing off your final edits to a jour­nal and being done with it.

  • Yes to recur­sive review! Yes to max­i­mal dis­sem­i­na­tion! (You dig­i­tal guys have the coolest slo­gans.) Both of these things can (and should) hap­pen. I just they think they should hap­pen after peer review. But obvi­ously, if there is some nec­es­sary con­nec­tion between peer review and pay walls that keep out the third world, then I’ll have to stop defend­ing peer review.

    Basi­cally, I’m just sug­gest­ing that when we dis­cuss peer review, we shouldn’t auto­mat­i­cally treat it as part of the dig­i­tal vs. ana­log debate. We can imag­ine a dig­i­tal world with old-fashioned peer review.

    My utopian sce­nario: all print jour­nals go dig­i­tal and take advan­tage of the pos­si­bil­i­ties that enables: recur­sive review, real-time debate, etc. They are dis­trib­uted free of charge (ie not owned by aca­d­e­mic presses). But they con­tinue to use some­thing like tra­di­tional peer review. It’s “fil­ter, pub­lish, then dis­sem­i­nate and debate.” See: I’m no good at slogans.

  • But Matt, don’t you some­times feel that writ­ing a peer review is like drop­ping some­thing down a well? And NOT hear­ing a splash?

    And also, we all fil­ter first and always will, most likely. We fil­ter when we give our work to col­leagues before send­ing it off, or dra­goon the spouse into reading.

    I can see your point, in that I rarely have to go out­side the US: there’s more work than I can keep up with. Your work requires arti­cles you can trust and the appa­ra­tus of blind peer review makes that eas­ier. I’m not con­vinced that there aren’t ways to make it eas­ier still

  • Thanks — this is use­ful. I hope you’ll pub­lish some­thing about this, focus­ing on the cocoons bit. Are you fol­low­ing the alt­met­rics com­mu­nity?

Leave a Reply

Your email is never shared.Required fields are marked *