With the end of the six-year period of Reading First on the horizon, no clear empirical picture has emerged of how well the federal program is doing at a national level in bringing struggling readers to proficiency.
Preliminary findings from the long-awaited suggested that the $1 billion-a-year funding for the program has had no impact on students鈥� reading comprehension.
But those findings did not answer many of the research questions that lawmakers required the evaluation of the program to address when they created it under the federal No Child Left Behind Act. Those questions, which many in the field say are critical, include whether the program and the research base undergirding it have been effective, and which approaches, programs, and assessments for teaching struggling readers are the most promising.
Dueling analyses by critics and advocates of the program have tapped the report鈥檚 contents to claim, alternately, that Reading First is a wholesale failure and a qualified success. Flaws in the research design, proponents say, may skew the findings against the program, which has delivered improvements on local- and state-level tests and gained fans among many school officials and educators.
鈥楢n Economic Question鈥�
Federal officials stand by the study, saying it is the best national evaluation possible, though they caution that it is too early to come to the kinds of conclusions that program critics have drawn. But while the final report later this year is expected to include a deeper analysis of key aspects of Reading First, and its impact on student achievement over a longer period, even in its final form it is unlikely to paint a comprehensive picture of the program鈥檚 effectiveness. Nor is it likely to clarify how best to translate research findings on effective instruction into classroom practice.
鈥淚 would say you have to wait for the final report before it would be reasonable for people to draw conclusions about the Reading First study,鈥� said Grover J. 鈥淩uss鈥� Whitehurst, the director of the Institute of Education Sciences, the research arm of the U.S. Department of Education that commissioned the congressionally mandated study. One difficulty in doing the study, he said, is that the treatment is not clearly defined, and implementation of the program varies from site to site.
鈥淭he 鈥榠t鈥� [what is being measured] is more ambiguous than it might be in certain other impact studies,鈥� he added. 鈥淭here鈥檚 not a manual that you can get on the Web and order that is Reading First.鈥�
The Reading First legislation provides a detailed list of 10 analyses required by the law, most of which will not be included in the five-year, $35 million impact study. Four other studies that have already been released or are pending will address some of those questions.
Previous federal reports on the program determined through surveys of state and local officials that the program has changed classroom practice or raised test scores in many places. But no experiments have been conducted to find out the extent of the progress or reasons behind it.
The latest study, released May 1, was expected to provide some answers, and the final report later this year may include deeper insights into the program鈥檚 effectiveness. (鈥淩eading First Doesn鈥檛 Help Pupils 鈥楪et it鈥�,鈥� May 7, 2008.)
The Reading First Impact Study was planned as a controlled study that would randomly assign schools to take part in the federal reading program or as a comparison school. The study, however, was initiated after most states had started to distribute Reading First grants to districts, making random assignment unfeasible.
Instead, the study used a 鈥渞egression discontinuity鈥� design, in which participants are assigned to program or comparison groups solely on the basis of a cutoff score on a pre-program measure. Units (such as schools) on one side of the cutoff (such as a score based on the percentage of children in poverty) are assigned to the treatment condition. Those on the other side of the cutoff are assigned to the control condition. This type of design is distinguished from randomized clinical trials by its unique method of assignment. Regression-discontinuity designs are appropriate when a program or treatment is targeted to those who most need or deserve it. Causal inferences that are drawn from a well-implemented regression-discontinuity design are comparable with conclusions from randomized experiments.
One site, a school district, was studied using randomized assignment of the Reading First program to five schools and comparing them with five schools in a control group.
The regression-continuity design is 鈥渢he strongest quasi-experimental method that exists for estimating program impacts,鈥� according to the report.
But it is not an evaluation of the policies and practices put into place in Reading First schools.
G. Reid Lyon, who helped write the Reading First legislation as chief of the branch of the National Institute of Child Health and Human Development that financed reading research, said people were misreading 鈥渆xactly what is being studied here.鈥�
鈥淚t is an economic question: Do schools that get Reading First money have a greater impact on reading comprehension than eligible schools that didn鈥檛 get the money?鈥� he said. 鈥淏ut people are interpreting that it鈥檚 about what the money buys, the [instructional] programs or packages.鈥�
Mr. Lyon said the evaluation outlined in the law would have included reviews of the commercial reading programs and assessments used by grantees, as well as teacher preparation and professional development in the subject.
The final impact study will include a second year of test results on reading comprehension, and will also compare students鈥� reading fluency in Reading First schools with nonparticipating schools in the same districts. Research on the relationship between Reading First instruction and student achievement will also be analyzed.
Little Evidence
鈥� Eligible schools be ranked for receiving grants using indicators such as student reading performance or poverty levels of students.
鈥� A cutoff point be set to determine which schools would and would not receive funding.
鈥� Schools with approved grant applications had reached the cutoff score, while those whose grant applications were denied had fallen below the cutoff score.
The study included Reading First schools that just reached the cutoff score and comparison schools that were ranked just below the cutoff score.
SOURCES: Institute of Education Sciences; Research Methods Knowledge Base
The design has generated confusion, as well as criticism about the quality and accuracy of the study.
Mr. Lyon said it was well designed to measure the financing impact, but the comparison sample is 鈥渃ontaminated鈥� because of the likelihood that those schools were following many of the principles and practices of the participating schools. He also suggested the limited results should not be used to draw sweeping conclusions for or against the program.
Many people, however, are doing just that, and are using the study to make their case for continuing, reconstituting, or killing Reading First.
鈥淎 whole bunch of my colleagues who have been against this forever are choosing to jump on any piece of evidence that supports their case,鈥� said Michael Kamil, a reading researcher at Stanford University and a member of the National Reading Panel. 鈥淭his study represents an aberrant data point. It鈥檚 the only piece of evidence we have that it doesn鈥檛 work, so you鈥檝e got to explain why [other studies] found significant improvements in Reading First schools.鈥�
Much of the other evidence Mr. Kamil referred to does not include data on reading comprehension, a skill that is difficult to measure in young children.
Other researchers, however, argue that it was predictable that the program would not improve reading comprehension.
鈥淚鈥檝e always believed that the way that Reading First was implemented in many states put a premium on paying attention to a relatively narrow body of research about kids鈥� learning and their acquisition of reading skill,鈥� said P. David Pearson, a professor of language and literacy and the dean of the graduate school of education at the University of California, Berkeley. 鈥淣o one, including the National Reading Panel, ever meant for the big five to be all there was for teaching reading, but that鈥檚 what we got in Reading First.鈥�
Mr. Pearson was referring to the elements of reading instruction that the influential reading panel, in its 2000 report, determined should be taught explicitly to students. Those elements鈥攑honics, phonemic awareness, fluency, vocabulary, and comprehension鈥攁re required in Reading First schools.
Several former officials with the Education Department say it is how the study was conducted and the size of the sample, not the program, that are faulty.
The nonparticipating schools that were compared with those in the program, for example, were likely to have adopted many of the practices required in Reading First schools鈥攊ncluding scientifically based instruction as defined by the National Reading Panel鈥攎aking it less likely that they would have significant differences in reading performance, those officials say. Most states, they argue, have reported dramatic improvements in Reading First schools on measures of basic reading skills鈥攕uch as decoding and fluency鈥攖hat are the foundations of early instruction.
鈥淲hen the Reading First schools make bigger gains than the rest of the state on state reading tests, that鈥檚 telling you something,鈥� said Sandi Jacobs, who was an administrator of the Reading First program at the Education Department until last year.
She said that she and her colleagues raised concerns that the study, as designed, would not properly gauge the program鈥檚 effect. 鈥淚f you look at the implementation study [released by the Education Department last year], it shows dramatic change in teacher practice and significant differences along every reported measure.鈥� (鈥淩eading First Schools: More Reading Going On, Study Finds,鈥� Aug. 9, 2006.)
That implementation study was based mostly on survey data and state-reported test gains and would not have met the rigorous standard for evidence the reading panel had required.
Gains on state tests occur more rapidly in low-performing schools once efforts kick in to raise them, said Robert E. Slavin, the founder of the Baltimore-based Success for All Foundation.
Mr. Slavin agreed that the study would be more useful if it measured a broader range of skills, but said the most troubling aspect of it is that it has taken more than six years to produce quantitative data on the program.
鈥淚 do think it鈥檚 sad that they rolled the dice on this one study, and that there couldn鈥檛 have been a variety of things going on that started earlier, so that we could have taken things into account to improve the program,鈥� he said. 鈥淣ow, we鈥檙e almost seven years into this, and we don鈥檛 have a whole lot to say about how the program is working.鈥�
Unjustified Criticism?
Mr. Whitehurst, however, said that by research standards, the study sample is large and valid. A representative national sample of Reading First schools would have been impossible to study, he said. Other criticisms of the design, he added, are not justified either.
鈥淭he schools in this study and the students served by them are very similar to schools served by Reading First nationally and the children served by Reading First nationally,鈥� he said. 鈥淚s it possible that other schools and districts participating in Reading First are producing stronger impacts on achievement. Yes.鈥�
The findings, Mr. Whitehurst said, do not support the arguments made by some critics that the Reading First principles, or the research-based approach to instruction overall, are ineffective.
鈥淪cientifically based reading instruction has to work, because by definition it is based on practices that have been shown to work,鈥� he said. 鈥淪o this almost gleeful conclusion that because of this report we can ignore cumulative evidence on effective reading instruction is simply inappropriate.鈥�
