Dr. Joe Pickrell has a follow up to his widely discussed post on updating scientific publication for the 21st century. One section jumped out at me, not because it was revolutionary, but because it made explicit a complaint that I had often heard:
The solution to this problem relies on a simple observation–in my field, I am completely indifferent to whether a paper has been “peer-reviewed” for the basic reason that I consider myself a “peer”. I do not think it extremely hubristic to say that I am reasonably capable of evaluating whether a paper in my field is worth reading, and then if so, of judging its merits. The opinions of other people in the field are of course important, but in no way does the fact that two or three nameless people thought a paper worth publishing influence my opinion of it. This immediately suggests a system in which papers are posted online as soon as the authors think they are ready (on so-called pre-print servers). This system is the default in many physics, math, and economics communities, among others, and as far as I can tell it’s been quite successful.
The reality is that often the "peers" are not peers. How else to explain the publication of the longevity study in Science, now retracted? Or the non-canonical RNA editing? (presumably this is less common of a problem in specialized journals). And sometimes the feedback of peers can indicate that they don't really know what they're talking about. For example, I was once told that the authors of a phylogenetics paper which used Bayesian methods were asked to reanalyze their data with a max likelihood framework (jump to the last sentence of this section to see why this is peculiar). The theory of classical peer review made sense in the pre-internet age. But now there are a plenty of reasons why we might need to revisit this.* * Not to mention that "peer review" is a somewhat subjective concept. Richard A. Muller has gotten into a back & forth on this issue whether his latest work has undergone peer review. He claims it has, others claim not. I suspect most traditional biologists would be skeptical of Muller's claim, but physicists would accept it.
