Optimism was running high in September 2002 when the U.S. Department of Education unveiled its plans for a clearinghouse to vet effectiveness studies on educational programs and practices. The hope was that the What Works Clearinghouse, within just a few years, would become an online one-stop shop where practitioners and policymakers could get the lowdown on what research has to say about what works in education.
But four years and nearly $23.4 million later, critics call the What Works Clearinghouse the 鈥渘othing works鈥� clearinghouse. As of last week, its Web site had posted reports for only 32 interventions鈥攐nly eight of which have a 鈥減ositive鈥� or 鈥減otentially positive鈥� effectiveness rating.
鈥淚 can鈥檛 tell if that鈥檚 because so little works, or because the clearinghouse has been so sluggish or so meticulous,鈥� said Chester E. Finn Jr., an assistant secretary for educational research and improvement in the U.S. Department of Education during the Reagan administration. He is now the president of the Thomas B. Fordham Foundation, a Washington-based think tank.
The clearinghouse posted its first reports鈥�10 study reviews on middle school math and peer-assisted learning strategies鈥攊n July 2004. Nearly two more years passed before another batch of products rolled off the line.
Acknowledging that the Institute of Education Science鈥檚 clearinghouse has not yet lived up to its promise, the site鈥檚 developers this month stepped up report production. Seven new reports on programs in the areas of character development, elementary school mathematics, and English-language learning were added on Sept. 14, and project directors plan to publish 40 to 50 more over the next six months.
鈥淚t鈥檚 been much harder than anybody thought,鈥� said Grover J. 鈥淩uss鈥� Whitehurst, the director of the IES, the Education Department鈥檚 main research arm and the agency in charge of the clearinghouse. 鈥淣ow we鈥檝e finally gotten to the point where we can get something out of the investment.鈥�
Rules-Based Process
Still, some critics wonder if the flurry of reports is too little, too late. With growing numbers of schools failing to hit their improvement targets under the No Child Left Behind Act, it鈥檚 more important than ever to get practitioners the information they need to raise student achievement, according to analysts.
Since its launch in 2002, the What Works Clearinghouse has reviewed research in four topical areas, including 31 studies on character education, 76 on middle school math, one on English-language learning, and 61 on elementary school math.
SOURCE: What Works Clearinghouse
鈥淚t鈥檚 gotten to a point where it鈥檚 very frustrating to the field, and they feel there are no products coming out,鈥� said Christopher T. Cross. Currently the chairman of Cross & Joftus, an educational consulting firm based in Bethesda, Md., and Danville, Calif., Mr. Cross served in the Education Department as assistant secretary for research under President George H.W. Bush.
The research pickings at the clearinghouse are slim, observers say, because federal officials overestimated the number of studies in the field that could meet the rigorous methodological standards they set.
Even some developers of curricular programs that get comparatively favorable reviews from the clearinghouse contend the screening criteria is unrealistic.
鈥淚 find it annoying that even the best-researched mathematics curriculum gets at best left-handed compliments,鈥� said Andrew C. Isaacs, an author of Everyday Mathematics, an elementary school program rated 鈥減otentially positive鈥� by the clearinghouse. (鈥淢uch-Used Elementary Math Program Gets Qualified Nod From U.S. Ed. Dept.,鈥� Sept. 20, 2006.)
Rebecca S. Herman, the project director for the clearinghouse, said it also took longer than anticipated to build an infrastructure for the systematic reviews.
鈥淲e had a pretty good sense of what the standards for review were from the beginning,鈥� she said. 鈥淏ut once you get into reviewing studies and findings, you find lots of subtle variations. If you want to have a rules-based process, then you have to develop all kinds of very specific rules.鈥�
From 2002 to 2004, Ms. Herman said, developers revamped procedures to make reports more useful. The early study reviews critiqued individual studies on particular interventions, giving them check marks, for instance, if they were deemed to 鈥渕eet the standard鈥� or 鈥渕eet the standard with reservations.鈥�
The more user-friendly 鈥渋ntervention reports鈥� added this year contain program descriptions, implementation costs, and simple Consumer Reports-style ratings that summarize a program鈥檚 overall effectiveness.
The revision also added a 鈥減otentially promising鈥� effectiveness rating to broaden practitioners鈥� choices.
Rival Database?
In the interim, though, other What Works-style projects are stepping into the breach left open by the paucity of scientifically proven programs.
The Education Department鈥檚 own office of planning, evaluation, and policy, for instance, is developing a database of 鈥減romising practices鈥� that are used in schools and districts with a record of raising achievement among non-English-speaking students. Thomas W. Luce III, the outgoing assistant secretary for that office, said U.S. Secretary of Education Margaret Spellings called for the database in response to practitioners鈥� requests for more help.
鈥淪he said, look, you all need to start finding out what are we learning that has a good chance of working,鈥� he said. 鈥淚 don鈥檛 know that it in any way reflects negatively on IES.鈥� The standards for inclusion in the new database, though not yet fully developed, will be admittedly lower than the screens that What Works reviewers use, Mr. Luce added.
Mr. Whitehurst, the head of the IES, declined in an interview this month to comment on the database, which is expected to be unveiled this winter. 鈥淚 believe and I hope,鈥� he said, 鈥渢hat the department doesn鈥檛 want to send out conflicting signals on what the standards are.鈥�
