Researchers from two of the nation鈥檚 most eminent universities, Harvard and Stanford, recently unveiled studies on different, but equally controversial, education issues. One was on school accountability policy, the other on teacher certification, but critics鈥� responses were the same.
They complained that the studies were 鈥渘ot quite ready for prime time鈥� because they had not undergone the independent scrutiny that usually comes when studies are published in academic journals.
It鈥檚 a refrain that has become common lately as research on hot-button issues moves directly from the author鈥檚 computer printer to a press release. The criticism has some scholars wondering whether education researchers need to do a better job of policing themselves.
鈥淚 think there have been abuses on all sides,鈥� said James. G. Cibulka, the dean of the college of education at the University of Kentucky, in Lexington. 鈥淎s a profession, we really need to have some agreed-upon criteria for vetting studies before they make their way to the public.鈥�
Whether the buzz about the need for giving studies some level of professional review before they hit the public arena is new, or even limited to education research, is an open question.
Studies generally undergo an evaluation process before they are accepted for publication in an academic journal, presented at a conference, or released by a think tank.
Peer review for articles published in a professional journal typically involves three or more reviewers who critique an author鈥檚 work in writing. The reviewers are 鈥渂lind,鈥� meaning that they are chosen by the journal editor and that the author does not know their identities.
Peer review for presentations at a professional conference is a less-rigorous procedure. Reviewers often accept or reject a proposed presentation on the basis of an abstract or a three- or four-page paper.
Peer review at a nonpartisan think tank, such as the New York City-based MDRC or the Washington-based American Institutes for Research, takes place mostly in-house as think tank researchers review one another鈥檚 work. In the later stages of its three-tiered review process, MDRC also enlists board members, outside experts in the topic under study, and participants from the study sites. For the Manhattan Institute, a think tank widely seen to support conservative perspectives, researcher Jay P. Greene said he typically circulates his papers to several outside colleagues before releasing them to the public.
鈥淥ne of the traditions of the education research enterprise has been the high frequency of advocacy research,鈥� said Grover J. 鈥淩uss鈥� Whitehurst, the director of the U.S. Department of Education鈥檚 Institute of Education Sciences. 鈥淚 would be skeptical that there鈥檚 been an uptick on this.鈥�
Others disagree. They say the debate over the need for tighter peer-review mechanisms sharpened last summer after the American Federation of Teachers produced an unreviewed study suggesting that some students in public elementary schools outperformed their counterparts in the same grades in charter schools on national tests of reading and mathematics. (鈥淎FT Charter School Study Sparks Heated National Debate,鈥� Sept. 1, 2004.)
When the AFT study鈥檚 findings became the subject of the lead story in The New York Times on Aug. 17, supporters of charter schools mounted a counterattack that included editorials, letters to the editor, and a full-page advertisement in national newspapers. The ad, signed by 31 academics, mostly criticized the study on methodological grounds, but it also called for closer scrutiny of research before findings percolate into the mainstream media. It said that 鈥渟uch studies need to be vetted by independent scholars, as is commonly done in coverage of research on the biological and physical sciences.鈥�
Since then, what has irked some other researchers is that three of the signers went on to do just what the teachers鈥� union had done: They took research findings to the press before the studies had undergone a formal peer-review process. That drew charges of hypocrisy from some of their research colleagues.
Jay P. Greene, one of the signers who was criticized, said he saw the ad as more of an admonition to journalists than to researchers.
鈥淭he ad never said all studies have to be peer-reviewed before people talk about them in public,鈥� said Mr. Greene, a senior research fellow who works at the Davie, Fla., branch of the New York City-based Manhattan Institute. 鈥淚 don鈥檛 think that鈥檚 a reasonable standard.鈥�
Another signer, David N. Figlio, an economist at the University of Florida, in Gainesville, said he viewed the ad similarly. He was not one of the researchers who later publicized unreviewed work.
Mr. Figlio said he signed the original ad to protest the prominent play the Times had given to research that had never been professionally vetted. When the same ad appeared in 澳门跑狗论坛, he said, he removed his name because he thought its coverage of the study had been more balanced.
鈥淚 feel strongly no unrefereed study should get front-page, above-the-fold treatment,鈥� he said.
Setting Clear Standards
But Mr. Cibulka of the University of Kentucky, who also signed the ad, says the responsibility belongs to researchers from the start. 鈥淭he problem is not with journalists,鈥� he said. 鈥淭he problem is with the profession for having failed to promulgate some clear standards.鈥�
Mr. Cibulka said a trend toward making premature research findings public began as education became more prominent in federal and state policymaking, and as think tanks sprang up to try to influence those policymaking decisions.
David C. Berliner, an education professor at Arizona State University, in Tempe, added that 鈥渆verybody has to have news all the time, and you can get stuff out there if you can tap into that.鈥�
Mr. Greene said that one reason he sends his reports directly to the news media is that it鈥檚 quicker than publishing in traditional journals, which can take up to two years.
鈥淚n that time, people who are not researchers are commenting all the time on the merits of certain policies, and they鈥檙e doing so in the absence of evidence,鈥� he said. 鈥淚f we say that researchers can鈥檛 comment with evidence until that research has been peer-reviewed, ... we鈥檒l actually have worse public-policy decisionmaking.鈥�
Such practices, in comparison, would be taboo in biomedical science, according to Joann Rodgers, the director of media relations for Johns Hopkins medical school. She said her office won鈥檛 issue a press release on a researcher鈥檚 study unless it鈥檚 been accepted for publication in a peer-reviewed journal. She said medical school researchers respect those guidelines because the penalty for violating them is heavy. Journals such as Nature, The New England Journal of Medicine, and Science won鈥檛 accept a study if its findings have appeared elsewhere first, she said.
Peer-Review Critique
Even so, many researchers say peer review is never a guarantee of quality in any field.
鈥淣ot only do some lower-quality studies get published when the peer-review process is not rigorous enough, some important and high-quality studies may not get published because there is not a suitable outlet for their topic, methodology, or length,鈥� Linda Darling-Hammond, the Stanford University education professor whose teacher-certification study was criticized last month, said in an e-mail interview.
鈥淪o any discussion of barriers to making research results public raises the possibility of staunching the free flow of ideas that is necessary for knowledge to move forward,鈥� she added in her e-mail.
Ms. Darling-Hammond issued a press release on her study after circulating it among colleagues and then presenting it in Montreal last month at the annual meeting of the American Educational Research Association. (鈥淪tudy Sees Positive Effects of Teacher Certification,鈥� April 27, 2005.)
The Harvard University researchers who were criticized last month, Paul E. Peterson and Martin R. West, followed the same procedure when they presented their school accountability study at a British economics conference.
Felice J. Levine, the executive director of the 22,000-member, Washington-based AERA, said it鈥檚 important for researchers to be able to present their findings at public professional conferences because that鈥檚 one way to gather feedback from colleagues on how to improve the work.
Conference presentations do undergo some review in order to land a spot on the AERA agenda, she pointed out. But it鈥檚 not quite the same as undergoing a journal review, which typically involves a careful reading by at least three 鈥渂lind鈥� reviewers鈥攃olleagues, in other words, who have been chosen by the editor and whose identity is unknown to the author.
Perhaps what the profession needs, some education researchers said last week, is a review mechanism in the middle. One model, they said, might be the National Bureau of Education Research, which posts 鈥渨orking papers鈥� on its Web site for colleagues鈥� reaction.
Mr. Cibulka suggested creating a national panel that could set standards for when to publicize research.
In retrospect, said Mr. Figlio of the University of Florida, 鈥淚 think the charter school dust-up may have done us all a big favor in terms of thinking about how do we balance readiness and timeliness.鈥�
