Peer Review and the Public Sphere

Peer review has not only served us badly: it’s cost aca­d­e­mics more and more cul­tural author­ity. The gen­eral pub­lic, hav­ing more sources avail­able online, is less will­ing to trust experts, and  sees peer review as akin to the mon­keys in Kipling’s Jun­gle Book: “We all say so, and so it must be true.”

Kath­leen Fitz­patrick made an excel­lent post on peer review. I highly rec­om­mend it as a deeper and more nuanced take than my ear­lier polem­i­cal version.

Fitz­patrick explains more about what Google does,  and how unclear Google is about how it ranks pages: since peer review is cen­tral to pro­mo­tion and tenure and career eval­u­a­tion, it’s dou­bly prob­lem­atic that Google hides its meth­ods. She adds that talk­ing about open sourc­ing peer review  it won’t pro­duce peo­ple will­ing to do the hard work. She’s right on all points.

But I still think a case can be made for ignor­ing a specif­i­cally aca­d­e­mic audi­ence for peer review, just ignor­ing it, and enter­ing aca­d­e­mic work in the gen­eral inter­net fray. Here’s why.

When GOP Can­di­date for Con­gress Rich Iott was revealed to be a Nazi reen­ac­tor in his spare time, there was a brief blog frenzy and I made a cou­ple posts about the value/utility of his­tor­i­cal reen­act­ing.1

I got a com­ment from a very thought­ful guy, “gepaeck, a seri­ous reen­ac­tor who said that more and more, peo­ple in the reen­ac­tor com­mu­nity sim­ply were not inter­ested in what his­to­ri­ans had to say: his­to­ri­ans, and the doc­u­ments they used, were all “biased” and sub­jec­tive and there­fore far less reli­able than the phys­i­cal expe­ri­ence of play­ing sol­dier in the hot sun. “Gepaeck” said he was quit­ting reen­act­ing for this rea­son: in the com­mu­nity of reen­ac­tors the act itself, the itchy wool and the loaded pack, had acquired an unim­peach­able author­ity and replaced schol­arly research.

Yes, sources are biased and clearly, the skep­ti­cal reen­ac­tors don’t really under­stand how aca­d­e­mic his­tory works.

But the larger point, rel­e­vant espe­cially in today’s polit­i­cal cli­mate, is that the prac­tice of peer review no longer claims the cul­tural author­ity it once claimed. Reen­ac­tors, who are fas­ci­nated by his­tory and should be a nat­ural audi­ence for well grounded research, increas­ingly dis­miss it altogether.

It’s not just his­tory: global warm­ing skep­tics sim­ply deride peer review as a small com­mu­nity of biased schol­ars, an echo cham­ber. Global warm­ing skep­tics are not some kind of odd­ball fringe: they’re pretty much syn­ony­mous with the Repub­li­can Party at this moment: VA’s Attor­ney Gen­eral: the Sen­ate minor­ity leader, Sarah Palin.

So not only does peer review not serve the needs of the in-group (us) very well: it does not enhance the pres­tige of the in-group among the out group, which as I take it was prob­a­bly the whole point of peer review to begin with.2

And this skep­ti­cism about peer review among the gen­eral pub­lic is not just some momen­tary blip: because global warm­ing skep­tics  and Nazi reen­ac­tors can find any kind of agree­able crap on the inter­net, they have no rea­son to depend on or accept the wis­dom of peer reviewed aca­d­e­mics. And so  we find our­selves in a sit­u­a­tion, here in Vir­ginia for exam­ple,  where the State’s fourth grade text­book cur­rently claims Stonewall Jack­son com­manded two bat­tal­ions of black Confederates.

It’s not as if I don’t partly agree with these skep­tics: peer review does tend towards in-group received wis­dom, and a shared secret lan­guage. That’s not all it does, of course. But at this cul­tural moment, more and more, no amount of in-group peer review can over­come the internet’s abil­ity to present you with what you want to believe, when you want to believe it.

So I want to imag­ine googling peer review not just because I want to break out of an inef­fec­tive and inef­fi­cient in-group sys­tem: it’s partly because I’d like to restore an active voice for thought­ful, “deep,” ver­i­fi­able work that’s well doc­u­mented and well argued, and that enters into the pub­lic sphere. Peer review that only speaks to and among “peers” won’t get it.

I’ll say right out that I think aca­d­e­mics should have more cul­tural author­ity: the work we do is vital to a healthy repub­lic and a healthy cul­ture. And the way to reclaim it is to get our process as much into pub­lic view as pos­si­ble: to stop speak­ing to “peers” in the sense of fel­low club mem­bers and start speak­ing to “peers” in the sense of thought­ful con­cerned fel­low cit­i­zens. That way we can make it clear how solid argu­ments derive from “biased” sources, and show what rea­soned and care­ful argu­ments look like.

If you want to see a good exam­ple of what I mean, look at the extra­or­di­nary exchange on the Civil War con­ducted at the Atlantic blog of Ta-Nehisi Coates, which you can find col­lected at this link. Or look at a spe­cific thread from that series, like this one. Here we have a blog­ger with a deep com­mu­nity of informed read­ers, engaged in a respect­ful dia­logue, con­cerned with both hard evi­dence and empa­thy and with the emo­tional res­o­nance of the past. This is a group of peers in the best sense, doing the pub­lic work of mak­ing sense of the past from a vari­ety of dis­ci­pli­nary perspectives.

Here is another excel­lent exam­ple of the same dynamic. I made a post about the alleged black Con­fed­er­ates, in which I pointed out some rea­sons why the main source for the claim that there were large num­bers of black con­fed­er­ates was sus­pect. I linked it in com­ments at Ta-Nehisi Coates’ blog. The blog­ger at “Dead Con­fed­er­ates” took it, ampli­fied it, and did more research and con­tex­tu­al­iz­ing to dra­mat­i­cally improve it. He used freely avail­able resources and framed it in a way that speaks directly to the com­mu­nity most con­cerned. My orig­i­nal post was in effect peer reviewed, sourced, and improved upon, then set in a con­text where it speaks to the peo­ple most engaged in the debate. This is exactly the kind of thing peer review should do. His blog lists more than  16, 000 hits: do that many peo­ple read a schol­arly article?

It should be clear that this argu­ment calls for recon­sid­er­ing not just peer review, but the shape and form of schol­arly work. I think we all know and under­stand the value of tra­di­tional schol­arly work: care­fully argued, scrupu­lously doc­u­mented, it argued very fine points with accu­racy  and pre­ci­sion. Sadly, that value is rapidly declin­ing. Schol­arly work in the future needs to get out of the box: keep the fine point, and the accu­racy and pre­ci­sion, but broaden the mean­ing of “peer” and engage in the ques­tions that con­cern the larger public.

I don’t mean pan­der, or dumb down: I’m not imag­in­ing vast audi­ences. But in the mod­ern era vir­tu­ally any sub­ject has a large non­spe­cial­ist con­stituency. These are the peo­ple we need to engage.

I don’t know if I can per­suade a tenure and pro­mo­tion review com­mit­tee to accept this as seri­ous work, but if I can’t, it will be for the same rea­son the first mate couldn’t con­vince the cap­tain of Titanic to watch out for ice­bergs. 3

  1. and
  2. A rea­son­able response might be that aca­d­e­mics never enjoyed much cul­tural author­ity, and I’m guilty of imag­in­ing a golden age. I would cer­tainly agree that aca­d­e­mics never enjoyed unchecked cul­tural author­ity in the past, and I’d add that they prob­a­bly shouldn’t–skepticism is a good thing for the most part. But I’m very com­fort­able argu­ing that the cul­tural author­ity of peer reviewed aca­d­e­mic work has declined in the last decade.
  3. now we’ll see if the com­mu­nity of titanic enthu­si­asts takes me to task for mis­un­der­stand­ing the causes of the ship’s demise. I hope so!


  • Are you lim­it­ing your dis­cus­sion of peer review to his­tory, or to the human­i­ties? Because once we move into the realm of the sci­ences, peer review looks a lit­tle dif­fer­ent. Yes, it’s under attack there, too, but there are a vari­ety of other avenues for dis­sem­i­na­tion and eval­u­a­tion that have sprung up., for exam­ple, pro­vides sci­en­tists a place to pub­lish their preprints for review and cir­cu­la­tion. Any­one can access these arti­cles freely. As Dan noted on his blog the other day, PNAS pro­vides non-traditional avenues for peer review.

    I’m also not con­vinced that peer review is as dis­trusted in the sci­ences as you make it out to be with your men­tion of global warm­ing deniers. Peo­ple want their sci­ences for the TV, the erec­tile dys­func­tion med­ica­tions (is your site going to get traf­fic now!), and the iPhones. I think politi­cians are more or less sen­si­tive to this fact, and that’s why things like the Coburn amend­ment tar­get the social sci­ences and not, say, physics.

    What­ever the case, I like what you’re propos­ing because it goes sig­nif­i­cantly beyond the typ­i­cal bound­aries of “open access,” which usu­ally applies only to read­ing the end prod­uct, not par­tic­i­pat­ing in the work-in-progress. When I’ve joined dis­cus­sions about “reform­ing” peer-review, they have often orbited around the idea of mobi­liz­ing the broader mem­ber­ship of schol­arly soci­eties (e.g. the Soci­ety for French His­tor­i­cal Stud­ies) to assume the role held by jour­nals. While this would be a step in the right direc­tion, it falls well short of what you’re propos­ing and would do noth­ing to elim­i­nate the echo cham­ber effect.

    You say that any sub­ject has “a large non­spe­cial­ist con­stituency” that can be tapped, but I won­der whether that’s really the case for some (or most) fields. I guess there’s only one way to find out.

  • I agree with almost every point you make here, Mike. But (and you knew there was one) I think the declin­ing cul­tural author­ity of peer review is part of a much larger story that has to do with the Viet­nam War, the six­ties, and the grow­ing sense that the human­i­ties and social sci­ences had been taken over by the left. In other words, our lack of pres­tige and our inabil­ity to shape dis­cus­sion in the pub­lic sphere are not pri­mar­ily a result of prob­lems with the peer review process. Vir­ginia doesn’t ask aca­d­e­mic his­to­ri­ans to review (or, God for­bid, write) its text­books, because aca­d­e­mics are all “biased” left­ists.
    Mean­while, I have no prob­lem with devel­op­ing new meth­ods of inject­ing our work into the pub­lic sphere via the inter­net, but I don’t see why those should replace peer review. Despite all the frus­tra­tions of peer review, I want my arti­cles vet­ted by peo­ple in my pro­fes­sion, and then I want to revise them on the basis of their sug­ges­tions. I know it sounds quaint, but I think my work improves from the process. Then, I can (and prob­a­bly should) go ahead and write provoca­tive blog posts for a larger audi­ence. Can’t we do both?

  • My wife, who works for the gvt. in “global cli­mate secu­rity” is always telling me how impor­tant peer reviewed sci­en­tific work is, and that she does not want to mess with under-researched par­ti­san junk, and I agree totally, but what draws me up short is the fact that no amount of peer reviewed work will change Sarah Palin’s mind.

    I think you’re right that this is a legacy of the 60s and “cul­ture war,” but the rem­edy must be to open up the “black box” of peer review, so that it’s clear how and why con­clu­sions are reached. Instead of “we met, we talked, you’re wrong,” we could estab­lish some­thing like an open dia­logue about evi­dence. I think both provoca­tive blog posts and peer reviewed spe­cial­ist work ought to be open sourced as much as pos­si­ble, so that it’s clear that while both the author and his sources are “biased,” the work is rea­soned and plausible.

  • I com­pletely agree that the dis­junc­tion between experts and inter­ested non­spe­cial­ists is a seri­ous prob­lem that needs to be addressed with sub­stan­tive changes to how the acad­emy eval­u­ates schol­ar­ship. To me, there’s no more dra­matic illus­tra­tion of this than the “con­tro­versy” over the link between vac­cines and autism. Even though sev­eral large epi­demi­o­log­i­cal stud­ies (includ­ing one study involv­ing over 500,000 Dan­ish chil­dren) have shown no evi­dence of such a link, there exists a very per­sis­tent rejec­tion­ism among a group of peo­ple who in my expe­ri­ence are edu­cated and actively research­ing the ques­tion. The con­se­quences of even a small group of par­ents opt­ing out of vac­cines is poten­tially very severe, but the med­ical research community’s response seems to be to just com­plain that the media is not argu­ing their case well enough. What’s inter­est­ing is that there’s no easy right/left split here and no large inter­est group like the energy lobby delib­er­ately obfus­cat­ing the data. If any­thing the sci­en­tists are dis­trusted because they are seen as too close to inter­ested eco­nomic par­ties like phar­ma­ceu­ti­cal com­pa­nies. The research com­mu­nity, in this case charged with pro­tect­ing chil­dren, sim­ply has very lit­tle author­ity among a sig­nif­i­cant minor­ity of par­ents. It’s hard not to see the aca­d­e­mic community’s prac­tice of reward­ing writ­ing for each other as a major part of the problem.

  • Excel­lent exam­ple on vac­cines, which as you say does not fit easy left/right splits but nev­er­the­less reflects mass dis­trust of the peer review process.

    I can’t think of a way to over­come this other than to have more trans­parency in peer review.

  • […] This post was men­tioned on Twit­ter by Ed Yong , Mary Carmichael. Mary Carmichael said: Read­ing today: Google and decline of peer review; @cshirky on how com­plex­ity begets col­lapse […]

  • Good dis­cus­sion. Per­haps it should be “pier” review, just like “Faux” News, to skewer the con­cept with a double-entendre, reverse Franglais.

    A few thoughts based on the blog and comments:

    A pur­ported strength of peer review, the anonymized and secret com­ments made known only to the author, are in fact the great­est weak­ness of the sys­tem. The qual­ity con­trol becomes the echo cham­ber in the black box to the unwashed–and suspicious–masses, who dis­miss con­sid­ered results, if they care at all, as con­spir­a­to­r­ial agen­das or ivory tower mus­ings. Peer review thus does noth­ing to con­vince the unin­formed or even the informed sceptics.

    A lot of peer review is also pro forma. Yes, rel­e­vant experts are asked to review and com­ment, but how thor­ough a job do they do? Who peer reviews the peer review­ers? Who selects them? How many are there? Why should I trust them if I myself have a dif­fer­ent bias or expertise?

    There have been some notable lapses in the most respected sci­en­tific jour­nals in recent years–some neg­li­gent and at least one intentional–that got through the vaunted peer review process at the most pres­ti­gious pub­li­ca­tions. It thus raises sus­pi­cions, to some extent valid in my view, that there is a larger frac­tion of faulty results con­stantly being pro­duced, even if most research results are of high qual­ity and present unim­peach­able con­clu­sions. Why should the scep­ti­cal “we” trust the process or the results themselves?

    A friend of mine is well-informed and a suc­cess­ful entre­pre­neur in the infor­ma­tion ser­vices area. He is both a cli­mate warm­ing scep­tic (I am not) and dis­dain­ful of uni­ver­sity inel­li­gen­cia. He believes that the grants that researchers get cor­rupt the process (the herd men­tal­ity), skews the results, and cre­ates the echo cham­ber. The fact that the arti­cles of 99% of cli­mate change researchers are peer reviewed by experts does not impress him, but just shows it is part of the larger echo cham­ber process and effect.

    It is true that most research is not of great social con­se­quence and fre­quently is not read, even by other researchers. A lot of pub­li­ca­tions are publish-or-perish quantity-mills, rather than qual­i­ta­tively new. And a lot of arti­cles can’t be under­stood by experts work­ing in other fields. But the peer review sys­tem does not do much to address these prob­lems, whether or not the arti­cles are of larger social sig­nif­i­cance or not.

    The net­work allows the intro­duc­tion of open pre-publication and post-publication reviews. These processes add value and build con­fi­dence in (valid) results. They also inevitably take time from the author in mov­ing on to new research. But in con­tentious areas, either in the intro­duc­tion and defense of new research par­a­digms or in soci­etally rel­e­vant areas, authors need to take the time to defend their views and thereby con­tribute more fully to their com­mu­ni­ties of prac­tice and to soci­ety at large.

  • Nathan, I’m not so sure that “the dis­junc­tion between experts and inter­ested non­spe­cial­ists” is some­thing we should be address­ing, at least if it’s the sort exem­pli­fied by the “vac­cines cause autism” folks. While they’re not quite as high up the crack­pot scale as, say, young Earth cre­ation­ists, they also appear to inhabit a world not of skep­ti­cal inquiry but of gath­er­ing what­ever evi­dence they can to sup­port their beliefs and attack the other side: they are sim­ply never going to change their minds, no mat­ter what hap­pens. That’s not sci­ence, that’s a form of religion.

    The med­ical community’s com­plaint about this (“the media is not argu­ing their case well enough”), while cer­tainly not effec­tive, is fair in cer­tain ways. Part of the rea­son that this can con­tinue on so long is because the media don’t in gen­eral have the sophis­ti­ca­tion to deal with sci­ence report­ing (or really, report­ing of schol­arly results in gen­eral) and see no ben­e­fit in devel­op­ing such sophis­ti­ca­tion. But researchers are per­fectly jus­ti­fied in not want­ing to become hard-core advo­cates and sales­men for a cer­tain point of view, both in that they have bet­ter things (such as research) to do, and because becom­ing such can com­pro­mise your abil­ity to good research. (The first thing you want as an advo­cate, and the last thing you want as a researcher, is an inabil­ity to admit that you were wrong.)

    Peer review is a tool used by researchers to improve research, and I think we need to keep that end in sight, and be care­ful about com­pro­mis­ing that aim of it in order to do bat­tle with cer­tain groups in the pub­lic that pre­tend to work within within research com­mu­ni­ties but are actu­ally under­min­ing the “real work” of research in order to pro­mote their own par­tic­u­lar belief sys­tems. Peer review is not and never will be a good tool for keep­ing cre­ation­ism out of grade school biol­ogy classrooms.

    That said, if along the way of attempt­ing to fix some of the man­i­fold prob­lems of our cur­rent peer review sys­tems, we man­age to open things up more to the true inter­ested non­spe­cial­ists (i.e., those with­out a political–I use this term loosely–aim), that’s great. I myself am one of those in sev­eral areas. But the level of knowl­edge and time for seri­ous research that I have for those things doesn’t even begin to approach the level at which I work in my pro­fes­sional career, and so it’s really a mat­ter for me of indulging and inter­est rather than being able to con­tribute any­thing useful.

Leave a Reply

Your email is never shared.Required fields are marked *