Concerned about the credibility of federally financed education studies, Congress passed a law in the fall of 2002 that replaced the U.S. Department of Education鈥檚 top research office with the Institute of Education Sciences.
More focused and politically independent than its predecessor agency, the new institute also had a new mission: to foster 鈥渟cientifically based鈥� research that might offer definitive insights on how to improve schooling.
Now, four years later, all the pieces of that overhaul are in place, and critics and outside experts are giving it mixed, but mostly positive, grades.
The shift has meant that it鈥檚 taking a long time for federally funded studies to produce findings, but their methodological quality is better, analysts say. And while some researchers worry that certain kinds of studies鈥攖hose that address messy policy questions, for instance, or that use different kinds of research designs鈥攁re getting short shrift, others see the agency鈥檚 newfound focus as long overdue.
鈥淲e鈥檙e in a better place than we were,鈥� said C. Kent McGuire, who headed the research office under President Clinton and is now the dean of Temple University鈥檚 education school. 鈥淏ut it takes a while for the cake to bake, and what does the field do while it鈥檚 waiting for those studies to come out of the oven?鈥�
Just as important, observers ask, will practitioners and policymakers know鈥攐r care鈥攈ow to make use of the findings once they鈥檙e 鈥渂aked鈥�? Not unless the institute鈥檚 leaders pay as much attention to finding ways to make research useful as they do to making it more rigorous, some suggest.
鈥淭he quality of the research is improving,鈥� said Gerald E. Sroufe, the director of government relations for the 22,000-member, American Educational Research Association. 鈥淲hether it will have more impact on schools remains to be seen.鈥�
Dramatic Changes
The dawn of the Institute of Education Sciences marked the fourth overhaul of the federal educational research apparatus since the 1972 creation of the National Institute of Education. But longtime experts in the field say the changes this time around were especially sweeping.
To buffer the research operation from political interference, Congress said the agency should have a separate director serving a six-year-term鈥攍ong enough to span more than one presidential term.
President Bush chose Grover J. 鈥淩uss鈥� Whitehurst for the job, a former university-based researcher with a strong methodological bent. He had already been leading the agency, then known as the office of educational research and improvement, or OERI, for a year when Congress passed the Education Sciences Reform Act, the law that established the new institute.
Mr. Whitehurst quickly initiated a series of changes to the 185-person agency. For one, he de-emphasized the competitive grant program for field-initiated studies, which let researchers across the field propose their own study ideas, and replaced it with more-focused grant programs in cognitive science, early-childhood education, and other areas.
He shut down the far-flung Educational Resources and Information Center, or ERIC, clearinghouse system and created a leaner, more centralized ERIC to replace it. Mr. Whitehurst complained that the system, known as the nation鈥檚 electronic education library, had become like an old attic: creaky and inefficient. Instead of exhaustively archiving every piece research ever presented or written on a topic, the revamped system contains only published studies.
The federally funded regional education laboratories were revised, and the national research-and-development centers likewise were trimmed and given new missions and more-specific research tasks. The IES also put in place a new peer-review process that looks more like the procedure used at the National Institutes of Health to screen study proposals.
To nurture a generation of researchers capable of doing the kinds of studies Mr. Whitehurst had in mind, he established interdisciplinary pre- and post-doctoral fellowship programs. Begun in 2004, the programs so far have underwritten training for 177 researchers around the country. 鈥淭here is hardly anything in place now that is like it was in previous regimes,鈥� said Mr. Sroufe.
Yet the institute鈥檚 emphasis on scientifically based research, part of a wider movement in the federal government to transform education into an evidence-based field akin to medicine, was perhaps its most disputed innovation.
While 鈥渟cientifically based research鈥� encompasses a range of study designs, experts agree that pure experiments or randomized-control studies are the gold standard for determining whether an intervention really works. Though common in medicine, such studies are rarely done in education鈥攑artly because they cost so much, and partly because they are difficult, and sometimes unethical, to do.
Striking a Balance
From the start, researchers worried that randomized-control trials would squeeze other forms of research out of the federal-funding pie. But Mr. Whitehurst calls such concerns 鈥渙utdated and uninformed.鈥� While the percentage of experimental projects funded by the institute has increased since 2002, such studies account for just a third of its overall research portfolio, he says.
In comparison, nearly half of the grants the institute has awarded since 2002 pay for development activity鈥攑rojects that synthesize research for use in real-world contexts.
鈥淭hat鈥檚 a reasonable ratio from my point of view,鈥� said Robert C. Granger, the chairman of the national board for education sciences, which Congress established to guide the institute鈥檚 work. 鈥淲hat the staff at the institute has been able to do is get very focused around a set of questions that leave some things in and some things out.鈥�
Of all the grant proposals to the Institute of Education Sciences, nearly half concerned program development. The four categories of applications are:
*Click image to see the full chart.
SOURCE: National Board for Education Sciences
Given the Education Department鈥檚 small share of federal research dollars, it could never generate solid findings without a focused funding strategy, other experts say.
鈥淚f there鈥檚 not enough productivity coming out of the thing, it鈥檚 always going to be seen as a backwater,鈥� said James W. Kohlmoos, the president of the National Education Knowledge Industry Association, a Washington trade group known as NEKIA.
Funded at $575 million a year, the institute ranks near the bottom among federal agencies in funding for research and development, according to the Washington-based American Association for the Advancement of Science. Of 16 large federal agencies that receive such funding, only the Agency for International Development, the Smithsonian, and the Nuclear Regulatory Commission receive less from Congress.
Still, some of the department鈥檚 concentrated research investments are poised to bear fruit, observers say. A case in point is its program of cognitive-science research, which is starting to yield insights on ways to help students retain more of what they learn. (鈥淐ognition Studies Offer Insights on Academic Tactics,鈥� Aug. 30, 2006.)
At the same time, though, some experts鈥擬r. Kohlmoos included鈥攃ontinue to believe the institute鈥檚 methodological and topical focus is too narrow.
鈥淚 am very much a proponent of what鈥檚 called scientific research in education,鈥� said Ellen Condliffe Lagemann, a professor of history of American education and former dean at Harvard University鈥檚 graduate school of education. 鈥淥n the other hand, I don鈥檛 think it鈥檚 the only way of knowing about education. I think we need support for other kinds of research, too.鈥�
Ignoring Policy?
One such gap in the institute鈥檚 research portfolio may be in analyses of policy initiatives such as the federal No Child Left Behind Act, said Christopher T. Cross, a former assistant secretary for educational research and improvement under President George H.W. Bush.
鈥淚鈥檝e been on meetings on the Hill where people are very anxious to know how No Child Left Behind is being implemented, and they can鈥檛 get a handle on it because almost no research has been done,鈥� said Mr. Cross, now the chairman of Cross & Joftus, an educational consulting firm based in Bethesda, Md., and Danville, Calif.
鈥淲e all want to know how various things are working, and why they are,鈥� said Susan H. Fuhrman, the president of Teachers College, Columbia University.
鈥淲hat鈥檚 happening to the big high schools left over when small high schools are taken out?鈥� she added, citing another current initiative by way of example. 鈥淣ow that鈥檚 a big policy question, and it鈥檚 hard to think of where in IES you could propose this.鈥�
The Center for Policy Research in Education, or CPRE, a five-university consortium that Ms. Fuhrman helped found, was one of the long-standing federal research centers that lost out when the institute reshaped the centers鈥� missions.
Mr. Whitehurst, the IES chief, noted that the institute has continued to underwrite studies and national research centers, such as the Center on Teacher Incentives, based at Vanderbilt University in Nashville, Tenn., that address policy-related issues.
鈥淚t is true, however, that the challenges, both methodological and logistical, of doing policy work are formidable,鈥� he said. 鈥淥ur ability to address broad policy perspective with our $2 million to $3 million grants is limited, 鈥� so we nibble around the edges.鈥�
Other experts suggest Congress and the Institute of Education Sciences missed a chance to build leadership, expertise, and outreach capabilities in key topics when they cut the size of the 10 national education research centers and refocused their missions on new topics and targeted research questions. Now funded at roughly $2 million to $5 million a year, the centers are on average about half the size they were under the old research office. 鈥淭hat鈥檚 not a conception of what a center ought to be,鈥� said Frederic A. Mosher, a senior consultant for CPRE. 鈥淭hat鈥檚 a conception of what a big research project ought to be.鈥�
Mr. Mosher and others also say the institute has not paid enough attention to studying how to repackage what researchers have already learned in forms that educators and policymakers can use.
鈥淲hat Russ [Whitehurst] is missing is the full realization of how little capacity there is to do good design and development work in the education systems,鈥� he said. 鈥淚 think there鈥檚 even less capacity for that than there is for randomized trials.鈥�
In the research office鈥檚 previous incarnations, that job largely fell to the regional educational laboratories, which are located across the country. Now, the labs, too, are charged with undertaking experimental work. But they also continue to serve the field by producing 鈥渇ast response鈥� analyses intended to give policymakers in those regions quick answers to pressing questions.
The What Works Clearinghouse, an $23.4 million federal project to create an electronic version of Consumer Reports for education research, was supposed to be the institute鈥檚 main vehicle for making research more useful and user-friendly.
Read the related story,
鈥極ne Stop鈥� Research Shop Seen as Slow to Yield Views That Educators Can Use
But observers agree that the clearinghouse has not yet lived up to that promise. After four years in development, the clearinghouse as of last week had only highlighted eight interventions with 鈥減ositive鈥� or 鈥減otentially positive鈥� evidence of effectiveness.
Mollifying Critics
But, among the agency鈥檚 three goals of rigor, relevance, and real-world applicability, rigor had to come first, Mr. Whitehurst says. Still, he said, 鈥淚鈥檝e been acutely aware that a What Works Clearinghouse that identifies nothing is not particularly relevant for a school superintendent who is trying to make a decision tomorrow.鈥�
To fill in the gap, the IES over the past six months or so has been stepping up its efforts to be more relevant to practitioners, tinkering with the What Works Clearinghouse, making plans to develop practical policy guides, and forming task forces on urban education that put researchers in touch with educators from the trenches.
The efforts appear to be mollifying some critics. One is Michael D. Casserly, the executive director of the Council of the Great City Schools, a Washington group made up of large urban districts. Last year, in an appearance before the national board for education sciences, he said the Education Department鈥檚 research arm produced studies that were 鈥渘arrow and irrelevant鈥� and lacked the answers that practicing educators needed to meet the demands of the No Child Left Behind Act.
鈥淚鈥檓 increasingly hopeful that education research will prove useful to practitioners,鈥� Mr. Casserly said. 鈥淚t鈥檚 very clear to me that IES has listened hard to our concerns.鈥�
Others have been harder to convince. 鈥淓ach new version of what used to be called the NIE has become more and more political,鈥� said Bruce Hunter, the chief lobbyist for the Arlington, Va.-based American Association of School Administrators. 鈥淭hey are more serving of the purposes of the political administration, and less serving of the purpose of American education.鈥�
But most observers, in contrast, say Mr. Whitehurst鈥檚 management of the IES has been remarkably independent, given the context in which the agency operates.
They point out, for example, that Mr. Whitehurst was among a handful of federal officials who offered a dose of scholarly caution last year when U.S. Secretary of Education Margaret Spellings trumpeted historically high results for 9-year-olds on national reading and math tests as 鈥減roof that No Child Left Behind was working.鈥�
鈥淚 think Whitehurst has done an admirable job of trying to stay out of making political statements when he has been pressed to,鈥� said NEKIA鈥檚 Mr. Kohlmoos.
Mr. Whitehurst said the department鈥檚 top officials so far have not interfered in the institute鈥檚 work. 鈥淚t is a negotiation,鈥� he said, 鈥渁nd sometimes issues come up for which there are no rules, but I think that it has worked out OK.鈥�
