Peer review has not only served us badly: it’s cost academics more and more cultural authority. The general public, having more sources available online, is less willing to trust experts, and sees peer review as akin to the monkeys in Kipling’s Jungle Book: “We all say so, and so it must be true.”
Kathleen Fitzpatrick made an excellent post on peer review. I highly recommend it as a deeper and more nuanced take than my earlier polemical version.
Fitzpatrick explains more about what Google does, and how unclear Google is about how it ranks pages: since peer review is central to promotion and tenure and career evaluation, it’s doubly problematic that Google hides its methods. She adds that talking about open sourcing peer review it won’t produce people willing to do the hard work. She’s right on all points.
But I still think a case can be made for ignoring a specifically academic audience for peer review, just ignoring it, and entering academic work in the general internet fray. Here’s why.
When GOP Candidate for Congress Rich Iott was revealed to be a Nazi reenactor in his spare time, there was a brief blog frenzy and I made a couple posts about the value/utility of historical reenacting.[1. http://theaporetic.com/?p=413 and http://theaporetic.com/?p=427]
I got a comment from a very thoughtful guy, “gepaeck,“ a serious reenactor who said that more and more, people in the reenactor community simply were not interested in what historians had to say: historians, and the documents they used, were all “biased” and subjective and therefore far less reliable than the physical experience of playing soldier in the hot sun. “Gepaeck” said he was quitting reenacting for this reason: in the community of reenactors the act itself, the itchy wool and the loaded pack, had acquired an unimpeachable authority and replaced scholarly research.
Yes, sources are biased and clearly, the skeptical reenactors don’t really understand how academic history works.
But the larger point, relevant especially in today’s political climate, is that the practice of peer review no longer claims the cultural authority it once claimed. Reenactors, who are fascinated by history and should be a natural audience for well grounded research, increasingly dismiss it altogether.
It’s not just history: global warming skeptics simply deride peer review as a small community of biased scholars, an echo chamber. Global warming skeptics are not some kind of oddball fringe: they’re pretty much synonymous with the Republican Party at this moment: VA’s Attorney General: the Senate minority leader, Sarah Palin.
So not only does peer review not serve the needs of the in-group (us) very well: it does not enhance the prestige of the in-group among the out group, which as I take it was probably the whole point of peer review to begin with.[1. A reasonable response might be that academics never enjoyed much cultural authority, and I’m guilty of imagining a golden age. I would certainly agree that academics never enjoyed unchecked cultural authority in the past, and I’d add that they probably shouldn’t–skepticism is a good thing for the most part. But I’m very comfortable arguing that the cultural authority of peer reviewed academic work has declined in the last decade.]
And this skepticism about peer review among the general public is not just some momentary blip: because global warming skeptics and Nazi reenactors can find any kind of agreeable crap on the internet, t
hey have no reason to depend on or accept the wisdom of peer reviewed academics. And so we find ourselves in a situation, here in Virginia for example, where the State’s fourth grade textbook currently claims Stonewall Jackson commanded two battalions of black Confederates.
It’s not as if I don’t partly agree with these skeptics: peer review does tend towards in-group received wisdom, and a shared secret language. That’s not all it does, of course. But at this cultural moment, more and more, no amount of in-group peer review can overcome the internet’s ability to present you with what you want to believe, when you want to believe it.
So I want to imagine googling peer review not just because I want to break out of an ineffective and inefficient in-group system: it’s partly because I’d like to restore an active voice for thoughtful, “deep,” verifiable work that’s well documented and well argued, and that enters into the public sphere. Peer review that only speaks to and among “peers” won’t get it.
I’ll say right out that I think academics should have more cultural authority: the work we do is vital to a healthy republic and a healthy culture. And the way to reclaim it is to get our process as much into public view as possible: to stop speaking to “peers” in the sense of fellow club members and start speaking to “peers” in the sense of thoughtful concerned fellow citizens. That way we can make it clear how solid arguments derive from “biased” sources, and show what reasoned and careful arguments look like.
If you want to see a good example of what I mean, look at the extraordinary exchange on the Civil War conducted at the Atlantic blog of Ta-Nehisi Coates, which you can find collected at this link. Or look at a specific thread from that series, like this one. Here we have a blogger with a deep community of informed readers, engaged in a respectful dialogue, concerned with both hard evidence and empathy and with the emotional resonance of the past. This is a group of peers in the best sense, doing the public work of making sense of the past from a variety of disciplinary perspectives.
Here is another excellent example of the same dynamic. I made a post about the alleged black Confederates, in which I pointed out some reasons why the main source for the claim that there were large numbers of black confederates was suspect. I linked it in comments at Ta-Nehisi Coates’ blog. The blogger at “Dead Confederates” took it, amplified it, and did more research and contextualizing to dramatically improve it. He used freely available resources and framed it in a way that speaks directly to the community most concerned. My original post was in effect peer reviewed, sourced, and improved upon, then set in a context where it speaks to the people most engaged in the debate. This is exactly the kind of thing peer review should do. His blog lists more than 16, 000 hits: do that many people read a scholarly article?
It should be clear that this argument calls for reconsidering not just peer review, but the shape and form of scholarly work. I think we all know and understand the value of traditional scholarly work: carefully argued, scrupulously documented, it argued very fine points with accuracy and precision. Sadly, that value is rapidly declining. Scholarly work in the future needs to get out of the box: keep the fine point, and the accuracy and precision, but broaden the meaning of “peer” and engage in the questions that concern the larger public.
I don’t mean pander, or dumb down: I’m not imagining vast audiences. But in the modern era virtually any subject has a large nonspecialist constituency. These are the people we need to engage.
I don’t know if I can persuade a tenure and promotion review committee to accept this as serious work, but if I can’t, it will be for the same reason the first mate couldn’t convince the captain of Titanic to watch out for icebergs. [1. now we’ll see if the community of titanic enthusiasts takes me to task for misunderstanding the causes of the ship’s demise. I hope so!]
Are you limiting your discussion of peer review to history, or to the humanities? Because once we move into the realm of the sciences, peer review looks a little different. Yes, it’s under attack there, too, but there are a variety of other avenues for dissemination and evaluation that have sprung up. arXiv.org, for example, provides scientists a place to publish their preprints for review and circulation. Anyone can access these articles freely. As Dan noted on his blog the other day, PNAS provides non-traditional avenues for peer review.
I’m also not convinced that peer review is as distrusted in the sciences as you make it out to be with your mention of global warming deniers. People want their sciences for the TV, the erectile dysfunction medications (is your site going to get traffic now!), and the iPhones. I think politicians are more or less sensitive to this fact, and that’s why things like the Coburn amendment target the social sciences and not, say, physics.
Whatever the case, I like what you’re proposing because it goes significantly beyond the typical boundaries of “open access,” which usually applies only to reading the end product, not participating in the work-in-progress. When I’ve joined discussions about “reforming” peer-review, they have often orbited around the idea of mobilizing the broader membership of scholarly societies (e.g. the Society for French Historical Studies) to assume the role held by journals. While this would be a step in the right direction, it falls well short of what you’re proposing and would do nothing to eliminate the echo chamber effect.
You say that any subject has “a large nonspecialist constituency” that can be tapped, but I wonder whether that’s really the case for some (or most) fields. I guess there’s only one way to find out.
I agree with almost every point you make here, Mike. But (and you knew there was one) I think the declining cultural authority of peer review is part of a much larger story that has to do with the Vietnam War, the sixties, and the growing sense that the humanities and social sciences had been taken over by the left. In other words, our lack of prestige and our inability to shape discussion in the public sphere are not primarily a result of problems with the peer review process. Virginia doesn’t ask academic historians to review (or, God forbid, write) its textbooks, because academics are all “biased” leftists.
Meanwhile, I have no problem with developing new methods of injecting our work into the public sphere via the internet, but I don’t see why those should replace peer review. Despite all the frustrations of peer review, I want my articles vetted by people in my profession, and then I want to revise them on the basis of their suggestions. I know it sounds quaint, but I think my work improves from the process. Then, I can (and probably should) go ahead and write provocative blog posts for a larger audience. Can’t we do both?
My wife, who works for the gvt. in “global climate security” is always telling me how important peer reviewed scientific work is, and that she does not want to mess with under-researched partisan junk, and I agree totally, but what draws me up short is the fact that no amount of peer reviewed work will change Sarah Palin’s mind.
I think you’re right that this is a legacy of the 60s and “culture war,” but the remedy must be to open up the “black box” of peer review, so that it’s clear how and why conclusions are reached. Instead of “we met, we talked, you’re wrong,” we could establish something like an open dialogue about evidence. I think both provocative blog posts and peer reviewed specialist work ought to be open sourced as much as possible, so that it’s clear that while both the author and his sources are “biased,” the work is reasoned and plausible.
I completely agree that the disjunction between experts and interested nonspecialists is a serious problem that needs to be addressed with substantive changes to how the academy evaluates scholarship. To me, there’s no more dramatic illustration of this than the “controversy” over the link between vaccines and autism. Even though several large epidemiological studies (including one study involving over 500,000 Danish children) have shown no evidence of such a link, there exists a very persistent rejectionism among a group of people who in my experience are educated and actively researching the question. The consequences of even a small group of parents opting out of vaccines is potentially very severe, but the medical research community’s response seems to be to just complain that the media is not arguing their case well enough. What’s interesting is that there’s no easy right/left split here and no large interest group like the energy lobby deliberately obfuscating the data. If anything the scientists are distrusted because they are seen as too close to interested economic parties like pharmaceutical companies. The research community, in this case charged with protecting children, simply has very little authority among a significant minority of parents. It’s hard not to see the academic community’s practice of rewarding writing for each other as a major part of the problem.
Excellent example on vaccines, which as you say does not fit easy left/right splits but nevertheless reflects mass distrust of the peer review process.
I can’t think of a way to overcome this other than to have more transparency in peer review.
[…] This post was mentioned on Twitter by Ed Yong , Mary Carmichael. Mary Carmichael said: Reading today: Google and decline of peer review http://bit.ly/auqtlY; @cshirky on how complexity begets collapse http://bit.ly/dad3MZ […]
Good discussion. Perhaps it should be “pier” review, just like “Faux” News, to skewer the concept with a double-entendre, reverse Franglais.
A few thoughts based on the blog and comments:
A purported strength of peer review, the anonymized and secret comments made known only to the author, are in fact the greatest weakness of the system. The quality control becomes the echo chamber in the black box to the unwashed–and suspicious–masses, who dismiss considered results, if they care at all, as conspiratorial agendas or ivory tower musings. Peer review thus does nothing to convince the uninformed or even the informed sceptics.
A lot of peer review is also pro forma. Yes, relevant experts are asked to review and comment, but how thorough a job do they do? Who peer reviews the peer reviewers? Who selects them? How many are there? Why should I trust them if I myself have a different bias or expertise?
There have been some notable lapses in the most respected scientific journals in recent years–some negligent and at least one intentional–that got through the vaunted peer review process at the most prestigious publications. It thus raises suspicions, to some extent valid in my view, that there is a larger fraction of faulty results constantly being produced, even if most research results are of high quality and present unimpeachable conclusions. Why should the sceptical “we” trust the process or the results themselves?
A friend of mine is well-informed and a successful entrepreneur in the information services area. He is both a climate warming sceptic (I am not) and disdainful of university inelligencia. He believes that the grants that researchers get corrupt the process (the herd mentality), skews the results, and creates the echo chamber. The fact that the articles of 99% of climate change researchers are peer reviewed by experts does not impress him, but just shows it is part of the larger echo chamber process and effect.
It is true that most research is not of great social consequence and frequently is not read, even by other researchers. A lot of publications are publish-or-perish quantity-mills, rather than qualitatively new. And a lot of articles can’t be understood by experts working in other fields. But the peer review system does not do much to address these problems, whether or not the articles are of larger social significance or not.
The network allows the introduction of open pre-publication and post-publication reviews. These processes add value and build confidence in (valid) results. They also inevitably take time from the author in moving on to new research. But in contentious areas, either in the introduction and defense of new research paradigms or in societally relevant areas, authors need to take the time to defend their views and thereby contribute more fully to their communities of practice and to society at large.
Nathan, I’m not so sure that “the disjunction between experts and interested nonspecialists” is something we should be addressing, at least if it’s the sort exemplified by the “vaccines cause autism” folks. While they’re not quite as high up the crackpot scale as, say, young Earth creationists, they also appear to inhabit a world not of skeptical inquiry but of gathering whatever evidence they can to support their beliefs and attack the other side: they are simply never going to change their minds, no matter what happens. That’s not science, that’s a form of religion.
The medical community’s complaint about this (“the media is not arguing their case well enough”), while certainly not effective, is fair in certain ways. Part of the reason that this can continue on so long is because the media don’t in general have the sophistication to deal with science reporting (or really, reporting of scholarly results in general) and see no benefit in developing such sophistication. But researchers are perfectly justified in not wanting to become hard-core advocates and salesmen for a certain point of view, both in that they have better things (such as research) to do, and because becoming such can compromise your ability to good research. (The first thing you want as an advocate, and the last thing you want as a researcher, is an inability to admit that you were wrong.)
Peer review is a tool used by researchers to improve research, and I think we need to keep that end in sight, and be careful about compromising that aim of it in order to do battle with certain groups in the public that pretend to work within within research communities but are actually undermining the “real work” of research in order to promote their own particular belief systems. Peer review is not and never will be a good tool for keeping creationism out of grade school biology classrooms.
That said, if along the way of attempting to fix some of the manifold problems of our current peer review systems, we manage to open things up more to the true interested nonspecialists (i.e., those without a political–I use this term loosely–aim), that’s great. I myself am one of those in several areas. But the level of knowledge and time for serious research that I have for those things doesn’t even begin to approach the level at which I work in my professional career, and so it’s really a matter for me of indulging and interest rather than being able to contribute anything useful.
[…] read more […]