From school turnaround to family engagement to new teacher mentoring, the Every Student Succeeds Act gives states greater flexibility and a more nuanced approach to using research to guide programs and policy.
Still, it remains to be seen whether states have the capacity to shoulder greater responsibility for deciding what constitutes solid evidence in education.
While the No Child Left Behind Act famously cited 鈥渟cientifically based research鈥� more than 100 times, and earlier reauthorization attempts spoke of 鈥渟cientifically valid鈥� research, ESSA calls simply for programs and interventions to be 鈥渆vidence-based.鈥� It also leaves most of the responsibility to states to decide the quality of that evidence.
鈥淚 think this is actually great opportunity, because every state is going to need to think about what evidence-based means for them,鈥� said Carrie Conaway, an associate commissioner for planning, research and delivery systems for Massachusetts鈥� education department. State officials and researchers, she said, need more conversations 鈥渢o think about what strong evidence looks like鈥攁nd where we do and do not have strong evidence. There are plenty of places where programs funded by ESSA that require evidence now, where we don鈥檛 currently have great evidence on what does and doesn鈥檛 work.鈥�
For example, Conaway noted that school turnaround is a critical piece of ESSA鈥攚hich requires states to focus on their lowest-performing 5 percent of schools鈥攂ut there are only a handful of improvement models that have been gone through the most rigorous evaluations. 鈥淭hat鈥檚 a really nascent area of research. The good news is people are starting to build evidence there, but it鈥檚 an area where we don鈥檛 know a lot about what works and why.鈥�
From Gold Standard to Tiers
Under ESSA, any strategy or intervention can be considered evidence-based if it shows a statistically significant effect on improving student outcomes or other relevant outcomes. The definition actually encompasses four separate standards of research. The first three tiers, modeled on the Investing in Innovation, or i3, grant program, are required for the law鈥檚 accountability-related school improvement programs:
- Strong evidence includes at least one well-designed and -implemented experimental study, meaning a randomized controlled trial. This is the highest bar, and very few school improvement models have actually met it so far.
- Moderate evidence includes at least one well-designed and -implemented quasi-experimental study. For example, a program evaluation could use a regression discontinuity analysis, in which researchers might look at differences in outcomes for students who scored a point above and below the entrance cutoff score for a particular program or intervention.
- Promising evidence includes at least one well-designed and -implemented correlational study that controls for selection bias, the potential differences between the types of students who choose to participate in a particular program and those who don鈥檛.
The tiers mirror evidence structures that are already required in other IES research grants and in several other agencies. Felice Levine, the executive director of the American Educational Research Association, argued that the Education Department should base ESSA guidance to states on a framework for judging evidence quality that has been developed by the IES and the National Science Foundation.
鈥淚 think Congress did a nice job of transitioning from scientifically based research that was ultimately more than what was needed in NCLB to giving states the opportunity to look at what they are going to do under the tiered approach,鈥� said Paul Kimmelman, a senior education adviser for the American Institutes of Research and a longtime federal education policy watcher. I have confidence that the state education chiefs will try to use top-quality evidence to make decisions.鈥�
Grover 鈥淩uss鈥� Whitehurst, a Brookings Institution fellow and the first federal Institute of Education Sciences director, said it makes sense for Congress to take a step back from requiring specific types of research in ESSA. NCLB erred, Whitehurst said, by 鈥渄emanding the use of [scientifically based research] when it didn鈥檛 exist ... [and] getting Congress into the business of defining SBR in education,鈥� Whitehurst said. 鈥淧erhaps these were justified at the time because the state of education research was really awful. But neither makes sense today.鈥�
Accountability and Training
The law, however, doesn鈥檛 actually make a distinction among the three levels鈥攁ll of them are considered 鈥渆vidence,鈥� but states make the ultimate decision of how to use the tiers. Moreover, everything else in the law that must be evidence-based can use those tiers鈥攂ut if there is nothing that meets any of the levels, there鈥檚 a fourth tier. If a state or provider can show that a program鈥檚 rationale is based on high-quality research or a positive evaluation that suggests it is likely to improve student or other important outcomes, that is enough, as long as the program or intervention has ongoing self-evaluation as well.
Massachusetts鈥� Conaway praised the fourth tier for making it easier to test and evaluate promising interventions and programs鈥斺�淚 think it will really help us build good research if it鈥檚 used well,鈥� she said鈥攂ut there鈥檚 also little in ESSA that the Education Department can use to hold states accountable for not scrutinizing evidence. Some states, like California, have their own laws in place requiring educators to use 鈥渃urrent and confirmed research鈥� in areas like reading instruction, but others will be starting from scratch.
鈥淚鈥檓 a little skeptical of how this will work,鈥� said Johannes Bos, a senior vice president at the AIR and an expert in randomized controlled trials. 鈥淚t鈥檚 difficult to do well and easy to cherry-pick evidence to support what you want to support. There will have to be a lot of discipline and a lot of review and training and surrounding infrastructure to make it work.鈥�
AERA鈥檚 Levine also had concerns about the tiered approach. 鈥淲hile these tiers might be useful in the context of smaller scale, more locally based evaluation decisions,鈥� Levine said, 鈥渢hey are less useful in supporting state policymakers鈥� and education administrators鈥� use of cumulative knowledge in education.鈥�
Under ESSA, states make the ultimate decision on whether evidence on a subject is 鈥渞easonably available鈥� when deciding how it should be required. That standard is likely to come into play in areas like teacher professional development, for which research on effectiveness has been mixed.
Across the board, experts argued that the Education Department will need to invest heavily in more training and technical assistance to boost state capacity to gauge research quality, as well as invest in more cross-state comparisons to find best practices as states begin to implement the new law. 鈥淵ou are going to have all these states with lots of freedom to make different decisions about things,鈥� Bos said, 鈥渁nd you need to treat that as 50 different experiments and really learn from what these states are doing differently.鈥�
Conaway agreed, suggesting the need for evaluations of implementation in different states could encourage more researchers to replicate prior studies, building a stronger evidence base for programs.
That could open new opportunities for partnerships between researchers and state education officials. The No Child Left Behind Act spurred a major expansion of education research in both topic areas, like school turnaround models, and in research methods. Experimental studies, for example, were relatively uncommon in education research because of time, expense, and the difficulty of finding large enough groups of students and then withholding interventions from some of them to create an appropriate comparison group. Federal guidance and grants in the NCLB years greatly expanded the number of such experimental studies conducted in education.
Prior federal research projects, such as the Institute of Education Sciences鈥� research alliance grants and grants to build up state longitudinal student databases, have helped lay the groundwork for ESSA鈥檚 approach, said Paige Kowalski, the vice president of policy and advocacy for the Data Quality Campaign, said. 鈥淔ive years ago, we just weren鈥檛 having these conversations and state education agencies weren鈥檛 really in the position to act in this way,鈥� Kowalski said. 鈥淲e were just having the conversations around moving from a model of compliance to one of service.鈥�
Next Generation i3
President Obama鈥檚 Investing in Innovation grant program, which lent its tiered-evidence framework to the law, has also survived in a more targeted form as Education Innovation and Research grants.
The new grant competition will, like i3, include early-phase grants to develop, implement, and test new and promising programs. Second-level grants would support full, rigorous evaluations of programs that have already been successfully implemented.
The third level of grants would target programs that have shown 鈥渟izable, important impacts鈥� in those evaluations. These grants would support both scaling up and, interestingly, replication studies to make sure the original results were correct and 鈥渋dentify the conditions in which the program is most effective.鈥�
The innovation and research grants were a big win for Jon Baron, the vice president of evidence-based policy at the Laura and John Arnold Foundation. 鈥淚 think it鈥檚 an important toehold, and potentially at some point, it is a step forward,鈥� he said.
