Googling Peer Review, Part Two

Talk­ing with a friend about peer review, it occurred to me that the stuff which has been most influ­en­tial in my intel­lec­tual life, the stuff that’s been most pro­found and use­ful, is pro­found and use­ful in ways that have noth­ing at all to do with peer review.

Was Foucault’s Dis­ci­pline and Pun­ish peer reviewed? It sure doesn’t read as if it was. His­tory of Sex­u­al­ity, v.1? I’m guess­ing no. Both books had a pro­found influ­ence. Both books are deeply flawed in terms of evi­dence etc. etc. but both had pow­er­ful and provoca­tive things to say. Is Judith Butler’s work provoca­tive and use­ful  because of double-blind blind peer review? Did blind peer review dra­mat­i­cally improve the core assump­tions and frame of Mak­ing of the Eng­lish Work­ing Class? Peer review is to those books as pea shoot­ers are to air­craft carriers.

Was Geertz’s essay on cock­fight­ing in Bali dra­mat­i­cally improved by peer review? I seri­ously doubt it–what’s valu­able about that famous essay is the clar­ity of his prose and the nature of the insights. Maybe peer review pushed him to make it a lit­tle bet­ter, but the value comes from the method, the intel­lec­tual core, not some fine tun­ing on Bali­nese vil­lage cus­toms forced by Geertz’s dis­ci­pli­nary rivals.

A few years ago the JAH pub­lished a round table on the anniver­sary of the Brown v. Brd. case. There were a cou­ple arti­cles in there that showed me an entirely new way to think about the Brown deci­sion and civil rights. Really good stuff which I had not con­sid­ered before.1

Here again, what was valu­able about those arti­cles had zip to do with peer review, and every­thing to do with the fact that the authors had found a new frame for a famil­iar topic. They had seen evi­dence lying in plain sight in a new way.

So when you think back on the books and arti­cles that most influ­enced you, is your first thought “hell of a job on the peer review?”

Now the obvi­ous objec­tion is that peer review is sup­posed to be invis­i­ble, and present us, the gen­eral pub­lic, with a reli­able, vet­ted, accu­rate prod­uct. I sup­pose one could argue that in the exam­ples I cited, it worked as it was sup­posed to. But again it’s not the fact that they were peer reviewed that makes these pieces worth­while: peer review is to their worth as the pars­ley gar­nish is to the blue plate special.

I’m tempted to advance “O’Malley’s law:” the inverse rela­tion­ship between peer review and endur­ing intel­lec­tual value. Good work is gen­er­ally good because it has some­thing valu­able to say, not because it has appeased other pro­fes­sors. Work that is good because of peer review is prob­a­bly not very good.

Now of course most of us are not bril­liant thinkers, and even bril­liant thinkers get help. No doubt Geertz, Fou­cault and other post­mod­ern wor­thies worked in a com­mu­nity, and ben­e­fited from exchange with their peers.  We all want that input on our work: we want to clar­ify our think­ing and gain from the insights of peo­ple we respect. We want to make that eas­ier, not harder: more fluid and less cumbersome.

The biggest objec­tion I hear when I talk about this is from aca­d­e­mics who worry about hav­ing to wade through a bunch of junk to find some­thing valu­able. I don’t think that will be a prob­lem, and this sce­nario explains why.

Imag­ine a web­site dev­toed to your field–I’ll use mine as an exam­ple, gilded age US. The web­site is trans­par­ent to google–that is, google can “crawl” it and index its con­tents. I post an arti­cle, a piece of research, to the web­site.  I “tag” it with sub­ject words, and I make com­ment­ing available.

Inter­ested per­sons have reg­is­tered at the site. As reg­is­tered mem­bers, they can make com­ments, post links, sug­gest revi­sions: they can do what we ask our friends to do. They can also “tag” the arti­cle in var­i­ous ways–maybe they can rate it, per­haps as “must read” or “needs work” or “in progress” or “unper­sua­sive.” They would not be anony­mous, which ide­ally would restrain nas­ti­ness, and they would be your key audience–motivated schol­ars who care about the sub­ject. Your piece of research would accu­mu­late a tag cloud of things schol­ars said about it, the things they liked, the things they found prob­lem­atic. As the author, you could tag other people’s com­ments, and com­ment on them in turn–some body writes an unfair review, you could tag it as “hos­tile” or “antag­o­nis­tic.”  Schol­ars vis­it­ing the site could rate oth­ers’ comments–“useful” or “not use­ful.” So a nasty com­ment that added noth­ing would quickly be isolated.

You could choose to revise and resub­mit the piece. Or you could let your com­men­tary stand as revi­sion, and present the piece as a dia­logue. Or you could decide to live with it as a fin­ished piece, as we do upon pub­li­ca­tion now. Schol­ars and the gen­eral pub­lic search­ing for your research could find it by going directly to the site, or by a search engine like Google. It’s cheap, it’s effec­tive, it requires no “too busy to get to the MS” blind peers, and sys­tems like it are in place now, all over the web.

Aca­d­e­mics always worry about the great unwashed flock­ing to such a site, but really, let’s face it, there are prob­a­bly not 1000 peo­ple inter­ested in your lat­est research, or mine. There are prob­a­bly not 100: there may not be a dozen who are inter­ested enough to read and com­ment. Too much infor­ma­tion is not going to be the prob­lem most of the time.

The spec­tor of Ama­zon alarms schol­ars; peo­ple would post unsup­ported  half-assed crap and we’d have to wade through it, wast­ing time, whereas now we know that some other guy has already waded through it for us. But really, it’s not hard to fig­ure out if the evi­dence is lack­ing or an argu­ment is mud­dled and unclear. How long does it take to read an arti­cle in your spe­cialty and fig­ure that out? And again, the site I’m imag­in­ing is extremely unlikely  to have really large, Amazon-like num­bers of peo­ple vis­it­ing it. And even if it did, so what? We want a larger audi­ence, no? The mech­a­nisms I’ve described is self policing–lazy and ill-conceived work is quickly called out; need­lessly hos­tile and unpro­duc­tive com­ments are flagged: in time they would sim­ply be fil­tered out, lost in the noise of more use­ful posts and comments.

What would it take to start a site like this? I’m not entirely sure. I need to ask at CHNM. How hard would soft­ware like that be to mount and main­tain? If I were edit­ing The Jour­nal of the Gilded Age and Pro­gres­sive Era this is the model I’d look to. If you go to the web­site of that jour­nal, you find that they are try­ing to adjust to the dig­i­tal age, but their model remains the print jour­nals and the pre-vetted arti­cle. The result is an uncom­fort­able fit, nei­ther fish nor fowl.

Who knows how to set up a site like that?

  1. Mary L. Dudziak. “Brown as a Cold War Case;” and Daryl Michael Scott, “Post­war Plu­ral­ism, Brown v. Board of Edu­ca­tion, and the Ori­gins of Mul­ti­cul­tural Edu­ca­tion,” in Jour­nal of Amer­i­can His­tory Vol. 91, No. 1 June 2004.

11 Comments

  • […] This post was men­tioned on Twit­ter by Dan Cohen and Alex Gil, mabel rosen­heck. mabel rosen­heck said: RT @dancohen: http://bit.ly/aiNThd // “think back on the work that most influ­enced you, is your 1st thought “hell of a job on peer review?”” […]

  • This is a fas­ci­nat­ing topic. I used to work for an aca­d­e­mic jour­nal in the sci­ences, where peer review was essen­tial. It was also a free ser­vice expected of doc­tors who at times looked at it as a chore akin to going to get a root canal. Do you fore­see any reluc­tance of aca­d­e­mics drown­ing in their own work to go fur­ther than deter­min­ing the gen­eral overview of the arti­cle and take the time to do a sys­tem­atic, thought­ful review? Or will it be more of a crowd­sourc­ing tech­nique? Can any­one register?

    Also, I’m won­der­ing exactly what your def­i­n­i­tion of “aca­d­e­mic” is. Would PhDs, pro­fes­sors, pro­fes­sional his­to­ri­ans be sub­mit­ting their work? Would PhD stu­dents be included? MA students?

  • Well I’d like to see the def­i­n­i­tion of “aca­d­e­mic” broad­ened, to include seri­ous stu­dents of his­tory of all kinds. I’m imag­in­ing crowd sourc­ing peer review, and to peo­ple who worry about too many peo­ple being involved, I say “you’re flat­ter­ing yourself!

  • They’re also wor­ry­ing because they flat­ter them­selves that grift­ing grad­u­ate stu­dents and preda­tory pro­fes­sors are stand­ing by to poach their pre­cious research (on tooth­pick fac­tory work­place con­di­tions, which actu­ally sounds great).

  • You could get pretty close to what you need with a wiki of some sort, though you’d prob­a­bly want to mod­er­ate mem­ber­ship to some degree. Cer­tainly it would be rea­son­able to start it as an exper­i­ment and then see what you needed to change about the soft­ware to make it more inclu­sive, and allow bet­ter “peer com­ment­ing on peer” functionality.

  • […] 15, 2010 by rosendof Leave a Com­ment I thought this arti­cle on the future of peer review was par­tic­u­larly rel­e­vant to this class. Namely, the issues of open-source schol­ar­ship and how it […]

  • […] claims not to believe this. He writes, Talk­ing with a friend about peer review, it occurred to me that the stuff which has […]

  • K. Hering wrote:

    Of course Fou­cault had an edi­tor — many of his works were pub­lished with the Edi­tions Gal­li­mard, like the Order of Things (in French) in the mid 1960s, and Foucault’s edi­tor was Pierre Nora, who devel­oped Gallimard’s social sci­ence division…Whether you call it peer review or not, you can’t under­stand any of his works with­out sit­u­at­ing it in the intense intel­lec­tual cul­ture and exchanges about struc­tural­ism in France at the time, and Gal­li­mard and its (paid) edi­tors played an impor­tant role in advanc­ing these debates.

  • The argu­ment was not that Fou­cault had an editor–he clearly did. The point was about whether or not the process of “peer review” as prac­ticed now–double blind review by per­son cho­sen by an editor–happened at all and if it hap­pened at all, made a mate­r­ial improve­ment in the work.

    There’s no doubt that all per­sons are embed­ded in a social con­text and that they ben­e­fit form the exchange that con­text enables

  • […] ini­tial post,  but as Michael O’Malley (among oth­ers) has argued in a pair of blog posts, peer review itself imposes sig­nif­i­cant costs on the scholarly […]

Leave a Reply

Your email is never shared.Required fields are marked *