UPDATED
The District of Columbia鈥檚 closely-watched system for evaluating teachers and providing bonus pay appears to have motivated weak teachers to make improvements, and to spur already-effective teachers to even higher levels of performance, concludes.
Teachers on the cusp of dismissal under D.C.'s IMPACT evaluation system improved their performance by statistically significant margins, as did those on the cusp of winning a large financial bonus, according to the study, which was published as a working paper Thursday by the Cambridge, Mass.-based National Bureau of Economic Research.
The provocative study raises new questions about how recently revamped teacher-evaluation systems鈥攁nd pay schedules linked to them鈥攕hape teacher behavior. It is among the first research studies to look at teacher evaluation empirically.
As an actual policy rather than a pilot, IMPACT was somewhat different in scope from other programs aimed at boosting teacher skill. Teachers with lower initial ratings also received access to instructional coaches to improve areas of their practice, one possible explanation for the gains.
鈥淎ll of the actors involved are taking this system seriously, and it鈥檚 in those situations where you might expect greater responses to the incentives,鈥� said James H. Wyckoff, a University of Virginia professor of education and policy, and one of the study鈥檚 authors.
The study does not look at whether the improvement in teachers鈥� scores translated into more learning for their students, though, and the researchers cautioned that the results apply only to the populations close to the performance thresholds studied. They do not depict an 鈥渙verall鈥� effect of IMPACT.
Dividing Line
Established in the 2009-10 school year, D.C.'s IMPACT evaluation system relies on a complex mix of factors to score each teacher, including both multiple observations and measures of student achievement. Teachers deemed ineffective under the system can be dismissed, while those scoring at the 鈥渕inimally effective鈥� level, the second lowest, get one year to improve. Those teachers who earn the 鈥渉ighly effective鈥� rating are eligible for of up to $25,000. Earning successive 鈥渉ighly effective鈥� ratings also permits teachers to on the salary scale.
Since its rollout, IMPACT has led to the dismissal of several hundred teachers. The system, instituted by then-Chancellor Michelle Rhee, began operation just as the federal Race to the Top initiative rocketed teacher evaluation to the top of policymakers鈥� agendas. The performance-bonus system, initially funded through the support of several private foundations, began a year later, the product of contract negotiations with the Washington Teachers鈥� Union.
Though IMPACT has won accolades in policy circles, some teachers have been less happy with the system. In surveys and focus groups, they have complained about feeling monitored, and that it鈥檚 harder for teachers in the most challenging schools to get top scores on the evaluation.
For the study, Wyckoff and Stanford University Professor Thomas Dee analyzed records for D.C. teachers and their students from 2009-10 through 2011-12, including more than 2,600 teachers each year.
The researchers examined those teachers falling just above or just below the dividing line separating 鈥渕inimally effective鈥� from 鈥渆ffective鈥� teachers and compared how they fared in subsequent rounds of the evaluation. Similarly, they analyzed the performance of those teachers just above and just below the decision point for awarding the financial bonuses, and compared how they did in the next year鈥檚 evaluation scoring.
The idea behind this methodology, known as a regression-discontinuity design, is that there aren鈥檛 major differences in characteristics between these groups, so their subsequent performance can be analyzed as a function of how they responded to their initial IMPACT scores. The design helps to rule out other factors that could have affected the results, though it is not as conclusive as a true randomized experiment.
Teachers with two 鈥渕inimally effective鈥� ratings first were fired in the 2010-11 school year. The researchers found that in the 2011-12 year, those teachers at risk of losing their jobs pulled up their performance by about 13 IMPACT points. Teachers on the cusp of winning a performance bonus in 2010-11, the first year they were offered, also improved by about 11 IMPACT points.
To put those gains in perspective, they鈥檙e about 50 percent and 41 percent, respectively, of the growth the average D.C. teacher makes in his or her first three years on the job.
Both effects are relative to the population of teachers that just made it over the IMPACT bar.
D.C. schools Chancellor Kaya Henderson said in an interview that the study validates IMPACT鈥檚 theory of action, which was to reward the best teachers, give assistance to struggling ones, and dismiss those not able to perform.
鈥淭he simple fact that this study tells us that the things we set out do to on our evaluation system are getting done makes me super proud, and hopeful about where we鈥檙e going,鈥� she said.
Adding to the Literature
The study also found that in 2010-11, teachers deemed minimally effective were also less likely to stay, with their retention rate falling by about 11 percentage points鈥攁n increase of 50 percent over the population of teachers that just made the 鈥渆ffective鈥� cut.
Rates of teacher turnover in Washington have also earned the district its , although the paper shows that highly effective teachers had high overall retention rates. Also, the study notes, teachers newly hired in 2011-12 had average IMPACT scores that were more than 25 points higher than the teachers who left.
Henderson was unapologetic about those findings.
鈥淲hen I came to the job as the deputy chancellor of human capital, our goal was to reset the standard about what it meant to teach in DCPS. There were a lot of people who rose to the challenge, and some who didn鈥檛, or couldn鈥檛 or wouldn鈥檛,鈥� she said. 鈥淚 want to know that the best teachers are sticking around, and if the minimally effective teachers are choosing to go elsewhere, I鈥檓 encouraged by that, too.
鈥淭his is not every other district in America; it is a really tough place, and we are asking our teachers to do amazing things.鈥�
The study鈥檚 suggestion that a high-stakes evaluation system linked to pay decisions and professional development can be a motivating factor for teachers to improve directly contrasts with a run of recent studies on bonus-pay programs showing little or no effects overall on student achievement.
Teacher evaluation and its effects have not been as widely scrutinized. Just one study indicates that improvements in teachers鈥� evaluation scores may have an effect on student learning. That 2011 paper found that on Cincinnati鈥檚 teacher-evaluation system correlated to increases in achievement among their students.
But the new study did not examine in detail whether Washington teachers鈥� improvements translated to higher student performance, a criticism local union officials highlighted. 鈥淭he goal of IMPACT was to improve student achievement, but in fact, [overall] student achievement has been flat,鈥� Washington Teachers Union President Elizabeth Davis said.
Still others praised the study鈥檚 design but pushed back against interpretations that teacher evaluations of the sort used in the District of Columbia will lead to widespread improvements in teaching quality.
鈥淧ut simply, what this study says is that if we take a group of otherwise similar teachers and randomly label some as 鈥極K鈥� and tell others they suck and their jobs are on the line, the latter group is more likely to seek employment elsewhere,鈥� wrote Bruce D. Baker, a professor in the graduate school of education at Rutgers University in New Brunswick, N.J., 鈥淣o big revelation there, and certainly no evidence that D.C. impact 鈥榳orks.鈥� 鈥�
UPDATED 1:51 p.m. with more reaction.