What constitutes "peer review"?

Question

What constitutes "peer review"?

Answer 1

I cannot speak from experience since I'm still only a high schooler, but I can speak from what I've learned from a bit of research:



In the general form, David is right in that peer-review can take many forms. In many fields it is the submission of a paper by an editor to usually two or three other individuals who are experts that understand the work. Most often, actually, these 'referees' are independent from each other and do not communicate their thoughts with each other.



However, this is different for several other journals that have more stringent requirements. For "Nature" and "Science," which are very broad in their publications' focus, a two-step review process is used. Here is where an editorial board is used - if the board does not find that the work is a breakthrough in the field (enough to publish in the journal, anyways), it will be rejected. If the paper passes, then it is sent out to referees for scrutiny.



In this sense, the editorial board is peer-review and yet is not at the same time. The board does not check for errors as the collection of referees would, but instead focuses more on the implications of the work. These journals receive a wide range of (and many) submissions and must filter the best out. The review of the work is still largely up to independent referees.





As to whether or not Einstein's papers were peer-reviewed, I'm not sure if I can comment on that. If Einstein's papers were read and revised by Planck and Wein, then they were peer-reviewed. However, it's not necessarily that simple. It would appear that Annalen der Physik very strictly peer-reviews eery paper before it is considered for publication, and a mere review by two people does not seem to fit into this standard. It would help if I had access to the Wiley InterScience website so I can do a bit more research on the journal's peer-review process, but it's inconveniently down right now.



If the standards for peer review were less 105 years ago, and if Planck and Wien actually made suggestions for revisions and did not just read the papers, then it can be said the papers were peer reviewed. I am not sure though what constituted proper peer review in 1905 for Annalen. I need more info.

Answer 2

In the form with which I'm familiar, an author(s) of a paper for submission to a journal submit the paper to the journal. The editor locates other workers who do research in a similar area and asks them to read the paper and evaluate its suitability for publication. Quite often, these "referees" find flaws in the data and/or presentation and the paper is returned to the author for revision, or the addition of new data. Sometimes the paper is rejected and sometimes it is accepted without further revision.



Lik all processes, this can be abused. To follow up on the global warming denialist issue, there are "journals" set up by denialists, and the papers are "peer-reviewed" by "experts," some of whom have little or no science background (this is by no means limited to global warming denialists, by the way). In other cases (again not only on global warming) a prestigious enough author can pull strings to have the paper reviewed by "friendly" referees. Many practicing scientists see through these ploys, but those not "skilled in the art" are unfortunately duped all too often. For that reason, I pay no attention to articles claiming the unreality of global warming unless they appear in the very highest caliber journals (Science, Nature, Proceedings of the National Academy of Sciences, etc.), since they have the highest standards, and convincing evidence would be of such momentous importance (given the more than a century of work) that no self-respecting author who was convinced of the accuracy of such work would submit it anywhere else.

Answer 3

What constitutes peer review has changed over time.



Now, for most journals I have any experience with, a paper is reviewed by an editorial board and then sent to anonymous reviewers who are considered experts in the field.



A century ago science was much different, it was smaller and less formalized. I would be surprised if that journal even had a proper editorial board at that time. It was still reviewed by two experts prior to publication, so it was peer reviewed.



Alias's complaint is ridiculous. Scientists were less specialized then. Given how revolutionary Einstein's work is, I doubt there were many people who would have met his definition of experts in that field.



Obviously some editorial on the internet went through no review whatsoever. Of course peer review isn't necessarily a marker of quality these days. Science denialists of all stripes have set up their own journals to publish their crap.



I'm very surprised they aren't using Watson and Crick's paper as an example. It wasn't peer reviewed. Nature had the full formal process at that time, but the editor just ignored it. It was too obvious they were right.

Answer 4

The better journals (i.e. ones I don't edit) will send an article to a third party to determine whether to reject an article, accept it as written, or ask for some revisions. Not every submission to a peer-reviewed journal gets peer review. Sometimes the editor can tell right off that the article is not slanted toward that journal or is too long or too short. If the article is otherwise acceptable, the editor will ask an expert in the field for comment. The reviewer may or may not be biased. For example, if the article says that oikos' article is full of beans, oikos may be asked to review it. Or it might go to someone else who knows the field. The writer usually does not know who wrote the review but the reviewer knows who the writer is.



Sending out the article for comments before submission is a good idea (and is almost always acknowledged in the paper) but it is not what is meant by peer review in scientific publication. There need not be a formal editorial board, though.

Answer 5

Science evolves, and peer review has evolved with it. In 1899, when the American Physical Society was founded, there were just 36 members--now there are about 48,000. The most knowedgeable physicists, such as Planck and Einstein, could be aware of all the major research going on. No doubt many of the journal editors had this kind of breadth of knowledge also. This is simply not the case today. Along the way, peer review has evolved also. As part of my current research, I have papers from the Monthly Weather Review from 1906 and 1922 (as well as many more recent papers). It's doubtful that any peer review was in place in 1906, other than the review by the journal editor. The 1922 paper includes reviews by three peers, along with the paper. This is actually very useful to be able to see the reviewers' comments and it's too bad that we don't do that today.



These days peer review is a useful tool for winnowing out research that obviously flawed, but it certainly does not mean that what has been published in the peer reviewed literature should be accepted unquestioningly. On the other hand one should be suspicious of research that intentionally avoids peer review and publishes instead on advocacy websites or publications. There are no Einsteins publishing on "wattsupwiththat." I would make the same statement for Greenpeace's website--real science is not published on advocacy websites.



EDIT for Andy: Clearly you don't have the faintest understanding of peer review. If you don't know what you are talking about you should not answer questions.

Answer 6

In my experience, the editor-in-chief selects the most knowledgeable of his editors to oversee the review of the paper. The selected editor selects 3 people, based on his/her own general knowledge of the field, to review a submitted paper. The editor may have a list of potential reviewers, make select a person in the paper's list of references, may have personal knowledge of a colleague or may have asked the author to suggest reviewers. (On one occasion I shot down a paper whose author had suggested me. He had fanciful ideas that contracted basic physical laws.)



The 3 reviewers (if they agree they have the knowledge on the topic and the time to do the review work) read the paper, and its references, and look at other material they are familiar with and look for errors, omissions, misunderstanding of theory, etc. They return to their comments, hopefully after due diligence. If all recommend, usually subject to minor corrections, the editor asks the author to make the minor corrections and accepts the paper.



If none of the reviewers recommend the paper, or if the reviewers disagree, the editor, after reading their reasons, rejects the paper or asks the author for major revisions.



The author may made further explanations or revisions to get a rejected paper published.



The system does not produce error-free papers, but they are greatly improved. After publication, a paper published in a scientific journal is part of the scientific record. Experts in the field can rebut the paper with new lab experiments, point out contradictory evidence, correct arithmetic mistakes, etc.



The peer-review process filters most, not all of the utter dreck. Researchers can be reasonably confident citing the results.



I think of an "editorial board" as the editors available to supervise the reviews of a paper. Only 1 editor is used per paper.



[I don't know what German practice 100 years ago was like, or how it is an example of anything. Silly argument. I can't think of anybody in German physics in 1905 more able to review Einstein's papers than Planck and Wien.]

Answer 7

As a history guy who once had a paper peer reviewed I can tell you that the process is both formal, very formal, informal and every other kind process the human mind can think of. But in the end the folks that actually do the review tend to be relentless bastards that would like nothing better than to shoot your research down in flames, have you flogged and drummed out of the 'service' with a yellow stripe down the center of your back. No mercy is shown. Nit picking is the name of the game and God help you if your thesis is provably wrong....'yer dead meat! Believe me, anyone who intends to submit a paper for 'peer review' sweats bullets over it. If you read it in a 'peer reviewed' publication you can bet that it's right on the money. If it's not the folks that ultimately read the submission will be more than glad to point out the errors while firing flaming arrows into the backs of both the guy that submitted the paper and the guys that reviewed it and allowed it see the light of day. If a publication allows too many of these poorly reviewed articles to appear that publication is going to catch hell. As most, if not all the scientific publications have been around for a long time it stands to reason that they don't allow many mistakes to slip by. The peer review business has been around long enough to win a whale of a lot of respect...is it perfect? No, sometimes later science will blow a widely held scientific belief out of the water...it happens, but it doesn't happen often and when it does it happens because of better science...not for some political reasons as the climate deniers are attempting to do.

Answer 8

The better journals (i.e. ones I don't edit) will send an article to a third party to determine whether to reject an article, accept it as written, or ask for some revisions. Not every submission to a peer-reviewed journal gets peer review. Sometimes the editor can tell right off that the article is not slanted toward that journal or is too long or too short. If the article is otherwise acceptable, the editor will ask an expert in the field for comment. The reviewer may or may not be biased. For example, if the article says that oikos' article is full of beans, oikos may be asked to review it. Or it might go to someone else who knows the field. The writer usually does not know who wrote the review but the reviewer knows who the writer is.



Sending out the article for comments before submission is a good idea (and is almost always acknowledged in the paper) but it is not what is meant by peer review in scientific publication. There need not be a formal editorial board, though.

Source(s):

I have reviewed articles and had mine reviewed. I have also asked colleagues to review articles submitted to me, on an informal basis (informal review, not informal submission).

Answer 9

“What constitutes "peer review"? I say it consists of having material submitted read by a competent person or persons before the editor decides whether to publish.”



In respect of published works then I would certainly go along with that description. Often it is the editor (or an editor) who will circulate the material to relevant experts for critical evaluation. More often than not the editor isn’t an expert on the specific subject in question but will have a broad understanding of the wider subject.



As a simple example… an editor of a medical journal will have a good understanding of medicine as a whole, but if s/he receives a paper relating to a proposed link between the toxoplasma gondii protozoa and schizophrenia then it’s unlikely that s/he will be sufficiently competent in this area to adequately critique the paper. As such, the editor would request that suitable experts review and return the material.



Upon completion of the review the work will be returned to the editor with comments added. These comments could relate to anything, perhaps to the methodologies or processes or the way that conclusions have been drawn. The reviewer will, where appropriate, highlight errors and uncertainties and suggest improvements.



The standard procedure is for each reviewer to work independently, unaware of the contributions made from other reviewers. Perhaps more so in scientific journals, the reviewers will submit their recommendations to the editor, they may recommend that the work be published as is, be published but subject to certain criteria or be rejected.



Comments are fed back to the authors and if needs be, the work will be revised in accordance with feedback from the reviewers. After revision the material may be resubmitted for consideration.



Ultimately, as in most publications, it is the editor who makes the final decision. Technically they could publish regardless of the outcome of the review process but this defeats the whole purpose of the exercise. The norm is to take on board the comments of the reviewers and make a decision whether to publish, and what to publish, based on the feedback received.



In some situations an editor may feel it is necessary to obtain further reviews before reaching a decision on publication. This is likely to happen if the original reviewers return conflicting opinions.



Contrary to some opinions, it is not a condition of the review process that a consensus be reached prior to publication; in reality, this isn’t even the objective.



A few years ago Nature conducted an in-depth debate about the peer review process, this looked at the many aspects involved and raised a lot of interesting points.

http://www.nature.com/nature/peerreview/debate/index.html



Further points…



Never having been an editor for anything other than a school magazine and a charity newsletter means that I am not particularly au-fait with the workings of the editorial process. I have had papers reviewed, some have been published and some haven’t, and I’ve reviewed the work of others.



As regards Einstein, Wien, Planck etc, although the papers may not have been formally reviewed in the manner that a submission to Science or the Journal of Applied Meteorology and Climatology (JAM) would be reviewed, the material has still been subjected to the same critical analysis and has undergone the peer review process.



I would have thought that being critiqued by Wien and Planck, would represent the ultimate in peer reviewing. I wonder how much of Watt’s or Goddard’s posts would have passed muster with the likes of these people, I suspect much of it would have ended up being used as toilet paper.

Answer 10

The peer review process varies from one journal to another, but publishers establish a set of common policy standards. For example, one of the journals with which I have direct experience (JRS) is subject to the standards given below.

http://wipjp.blackwellpublishing.com/bw/publicationethics/#_Toc149460081

Journals with high standards of peer review attract manuscripts from the best authors and are widely read and cited within the research community. Typically, manuscripts that appear in these journals are reviewed first by the editor, who may reject a manuscript as unsuitable for the journal by reason of content or quality. The editor then asks three scientists with significant research experience in similar subject matter and no conflicts of interest (see Wiley guidelines) to review the manuscript. The reviewers change from paper to paper. There is not one formal group of reviewers that looks at all the manuscripts. The reviewers rate the content of the manuscript and comment on questions such as: 1. Were the experiments performed properly? I once rejected a paper on the grounds that the spectra shown were from O-ring grease rather than the sample claimed in the manuscript. The author was new to the field and lacked the experience to recognize an error. 2. Are the experimental results reproducible and reasonable? Reviewers may repeat some of the experiments and or compare the results with prior research. I have repeated experiments to ensure that the result is broadly correct. If the results are not consistent with prior research or the reviewer's experiment, the reviewer asks the author for an explanation. 3. Is the work original? I get really snarly with authors that either plagiarize others work or try to publish the same research in multiple journals to increase their publication count. 4. Is the manuscript of interest and important to the field? This is an opinion about whether the manuscript will be cited and promote further research in the field.



While deniers may set their own standards of peer review, the process should be transparent. In the case of Watts, the manuscript must be consistent with Watts point of view and there is no requirement that the manuscript has a factual basis. Both Nature and Watts have the esteem they deserve, based on the rigor of their respective review processes. Personally, I would advise Watts to publish his works on rolls of soft tissue paper so that they can serve a useful function.



Edit: I see that Trevor has the same utilitarian use for Watts blogs.

Answer 11

I’m going with a general definition of peer review as “some kind” of formal (official) evaluation of a scholarly work by one or more “experts” prior to publication.



It seems to me that the heart of the disagreement between Paul and Alias is that Alias insists on narrowly defining the peer review process only as it currently exists - without explicitly saying so. He could have, and probably should have, simply answered that in his opinion peer review did not exist prior to the mid-twentieth century - for example, “Science” and “The Journal of the American Medical Association” did not use outside reviewers until the 1940s (I’m not going to argue an exact date - because I don’t care).



Paul, and I think most scientists, trace the peer review process to either the Royal Society of Edinburgh in 1731 or to the Royal Society of London’s creation of a “Committee on Papers” to oversee the review and selection of texts for publication in 1752. The July 28, 1905 issue of “Science” contains the following information for authors: “MIS. intended for publication and books, etc.. intended for review should he sent to the Editor of Science”.



http://www.sciencemag.org/cgi/issue_pdf/toc_pdf/22/552.pdf



• >>Peer review is where the paper is reviewed by people who aere specialists in the NARROW subspecialty that the paper involves.<<



• << Planck was quite competant, but he did not work in the subfield. Ironically a Relativity Denier who did work in the subfield but was unreasonable would be a genuine peer reviewer.>>



These are bogus arguments because, again, they exist in a historical vacuum. Look at the graph of, “Number of physics PhDs conferred in the United States, 1900 to 2001” on page 7 here:



http://www.aps.org/programs/education/chairs/2002/upload/czjuko.pdf



It is/was a different world. I have heard tales from my older colleagues of major conferences being held in someone’s damn living room. That universe was quantitatively and qualitatively smaller than the one created by the post WWII explosion in the number of scientists.



As a side note, the “scientist” label is also relatively recent – back in the day, they were called Men of Science. Stuff changes, but that does not mean that its history is irrelevant or illegitimate.



You raise an interesting question, but the debate is far more subtle and detailed than most Y!A participants can handle or will allow.

Answer 12

Peer process is designed to ensure the submitted study is worthy of the time and considerate of other scientists. The peer-reviewed journals exist for scientists to share what they have learned, allow others to replicate and/or build on, and to advance mankinds understanding of science with every study published.



Whatupwith that is an editorial blog with no assurance that anything written there is at all factual. In fact, much of it is not factual an something published that cannot be assumed to be accurate or well-designed. In fact, the reason it is published the is to avoid the eyes of other climate science. The audience is people uneducated in climate science -- the audience is not other scientists.



What is published in peer-reviewed journals is meant to be challenged by other scientists. What is written on any blog is there because the writer does not want to be challenged by other scientist or because the quality of the work is not up to the standards of the journals.



I could start my own blog, many people have them. We can put whatever we want on it and call it science. But that does not mean it should be given any credibility.

Answer 13

The peer review process can take many forms.



Most peer reviewed or refereed journals have a list of subject matter specialists who are willing to voluntarily review manuscripts. These people will then either recommend publication, revision & resubmission or rejection. Generally the identities of the people in the peer group are kept confidential. Journals will also generally have certain editorial standards: article length, relevance of the research, newness or uniqueness of the work, etc.



Peer review groups rarely check the underlying work. These are generally volunteers who have their own work to do and don't really have time to check the accuracy or repeatability of the underlying research and/or experiments. But it is a generally good process for selecting publication-quality work.



There are problems in the peer review process. Counter-paradigm papers almost always faces more scrutiny than paradigm-reinforcing papers. The reviewers are generally anonymous while the authors are not; so their affinity or animosity to the authors is unknown. Many scientific disciplines consist of relatively closed and small communities. These problems exist in many fields of study.



Peer review is a useful thing; but it is not a certification that the paper is correct in its conclusions or that the underlying work is free of major errors.



If a peer reviewed paper and a non-peer reviewed paper provide contrasting conclusions from the same data sets and the data are available... Anyone with a modicum of scientific literacy can check the work and draw their own conclusions,

Answer 14

Einstein is a terrible example. His circumstance were so exceptional that you cannot make a rational comparison to ordinary peer review.



Have you read any of Einstein's four 1905 papers? They are all simple. If they weren't, they probably wouldn't have been published. They were submitted by a patent clerk who despite his Phd from Zurich, couldn't get a job teaching.



Apart from the brownian motion, the reason that they were published was that they were dirt simple, the math added up and they explained experimental results within the known parameters at the time. No one in their right mind (except for Einstein, possibly) believed that they were true -- just that they were a cute contrarian explanation that might lead somewhere, but in many ways they were nonsensical and heretical. Their beauty was in their accessibility and their simplicity -- not for their correctness or revolutionariness.



There was no one alive who had the 'expertise' to review Einstein's papers. Who was an expert in Relativity? It is my understanding that Einstein's papers were referred to Planck (who may have being the leading theoretician at the time) because they were such a challenge to the status quo and because Planck had expertise in energy quantum that was used to explain the photo electric effect. Like Planck, Einstein's papers were a mathematical explanation for otherwise unexplainable phenomena.



But even though the papers were accepted, Einstein's ideas were not. It was only in 1908 that Einstein got a part-time teaching job as a lecturer in Bern. He still had to work at the patent office, though.



Then things started to roll, real academic positions: back to Zurich in 1909 then on to Prague and Berlin.



Planck got his Nobel Prize in 1918. His work was now fairly well accepted. And Einstein in 1921 got his Nobel Prize "for his services to Theoretical Physics, and especially for his discovery of the law of the photoelectric effect". Relativity was still too controversial.





Bottom line: Einstein was lucky to have his seminal papers published. They are easy to read, the math is easy to do and they explain experiments -- but the consequences were revolutionary and were only slowly accepted. Einstein was lucky to have Planck champion his papers.



Further, I think your Alias has some basis for his assertion -- but he really misses the point. Einstein's papers are rare and exceptional. Most peer reviewed papers are not transcendent but humdrum. There is no need to refer these papers out to the true experts in the field to see if they make sense.



But, I really think that you should look in the mirror and ask yourself, "am I worthy of being trolled." Unless you are, the guy whose trolling you is a loser and is best ignored. The guy needs to grow up and get his own identity instead of trying to bask in your reflected limelight.

Answer 15

From my limited understanding, you're right in that "I say it consists of having material submitted read by a competent person or persons before the editor decides whether to publish." PA2 isn't someone you should get worked up about, as he believes that within ten years Antarctica will be gone.



I must ask, though, why you made a follow-up question in response to PA2, and then called his extreme-alarmism a "denialist ploy". I have trouble believing that you think that PA2 is a skeptic, so it seems you're just blaming random crap on the "other team".

Answer 16

Generally modern peer-review is a formal process. You submit your paper to a journal which passes it along to some other scientists with expertise in the field. They comment and decide whether it needs changes prior to publication.



But the purpose is simply for experts in the field to ensure that the science in the paper is sound. It doesn't necessarily have to be such a formal process, and in the case of Einstein's work being reviewed by Planck and Wien, that certainly qualifies as peer-review, as all had expertise in physics (that's an understatement).



Publishing an article on a blog is not peer-review. I suppose theoretically a person could post a paper on a blog, have it reviewed by experts in the scientific field in question, revise the paper based on their comments, and then post it again, and that could qualify as informal peer-review. But of course, that's not what Watts and Goddard do. They just throw together some incredibly amateur analysis, toss it up on the blog, and leave it unchanged. Posts on Watts' blog are rarely edited, and certainly aren't reviewed by climate scientists.

Answer 17

I see my answer was removed, Paul B.



So pathetic.



Einstein's paper was NOT peer reviewed, and as I said before you were just plain lying, just as you lied in claiming that you were a journal editor.



<>



That is not correct either. The editor is not some grammar guy, but rather is an "expert" but not necessarily in the very specific subfield your paper is in, and he, in practice, has final approval.



<>



Peer review is where the paper is reviewed by people who aere specialists in the NARROW subspecialty that the paper involves. Neither Planck nor Wein were doing anything related to the moving chasrges paradox. Had a referee system been in place at the time there was no way they would have been chosen as referees.



<< I meant specifically "read by a competent person or persons " *in order to decide whether or not to publish*; >>



Again not true. Planck was quite competant, but he did not work in the subfield. Ironically a Relativity Denier who did work in the subfield but was unreasonable would be a genuine peer reviewer.



<>



That's not true. There were actually lots of people working in the field at the time of Special Relativity. It was a hot topic.



Dana <>



It is interesting thatt Dana for whom "peer review" is so important doesn't know what it is. It does not depend on how good a physicist the reviewer is, it depends on whether he works in that field. Planck was working in a totally different field. It is interesting how the Moderates make such a big deal about the opinions specifically of "climaste scientists". By your reasoning Planck should not be in a position to have an independent opinion on global warming.

Answer 18

Peer review consists of at least one (other) person who remotely understands what it is you have done. I think of editors as people more concerned with grammar, spelling, and proper use of the language. My own work often had at least one "reviewer" who put a signature on it, and an "approver" who might have been simply a manager who trusted the reviewers. These might have been biased, in that they wanted the corporation / enterprise to succeed (profit) but they might have also been "zero-sum" people who might feel they could win by making the other guy look bad.

Answer 19

Your peers are people who equal you in some specific way. Who decides what makes a genuine peer review? Especially when you're planning to review new ideas. Lot's of opportunity there for circular arguments.

Answer 20

1

Answer 21

Actually, all peer reviewed means is that people that agree with your theories have read your material and concur. This is totally different then the real scientific method where it usually takes a double blind test done by two independent groups to come to the same conclusion. Considering that in global warming we don't really have independent groups doing any type of confirmation and they rely instead on peer review as "proof". Peer review can never proof that someone's theory is sound.

Answer 22

Adolf Hitler was recently exonerated by his peers: Heinrich Himmler, Joseph Goebbels, Hermann Goring, Albert Speer and Martin Bormann, editors of "Justice in Germany" have cleared Adolf Hitler of any wrongdoing during the 1930's and 1940's....



Peer review can used to claim anything --as long as you've collected the right group of peers-- so it is a completely worthless political exercise, not a scientific one.