Harvard University researcher Daniel M. Koretz has some good news and some bad news for policymakers looking ahead to the reauthorization of the federal No Child Left Behind Act: The nation鈥檚 K-12 students are now attending Lake Wobegon schools.
Because of rampantly inflated standardized test scores, Mr. Koretz contended at a panel discussion here last week, all children seem to be above average, as in the fictional town made famous by the radio personality Garrison Keillor鈥攐r at least better academically than they actually are.
鈥淲e know that we are creating, in many cases, with our test-based accountability system, illusions of progress,鈥� Mr. Koretz told a roomful of policymakers and educators at a Sept. 22 forum hosted by the American Enterprise Institute, a Washington think tank.
Mr. Koretz, a professor of education at Harvard鈥檚 graduate school of education who has written a new book, , said that under the No Child Left Behind law, widespread teaching to the test, strategic reallocation of teaching talent, and other means of gaming the high-stakes testing system have conspired to produce scores on state standardized tests that are substantially better than students鈥� mastery of the material.
鈥淚f you tell people that performance on that tested sample is what matters, that鈥檚 what they worry about, so you can get inappropriate responses in the classroom and inflated test scores,鈥� he said.
Mr. Koretz pointed to research in the 1990s on the state standardized test then used in Kentucky, which was designed to measure similar aspects of proficiency as the National Assessment of Educational Progress, the federally sponsored testing program often called 鈥渢he nation鈥檚 report card.鈥�
Scores on both tests should have moved more or less in lock step, he said. But instead, 4th grade reading scores rose sharply on the Kentucky Instructional Results Information System test, which resembled an NCLB-test prototype, from 1992 to 1994, while sliding slightly on NAEP over the same period.
鈥淚f you鈥檙e a parent in Kentucky, what you care about is whether your kids can read, not how well they can do on the state test,鈥� Mr. Koretz said. 鈥淎nd the national assessment told us that, in fact, the gains in the state test were bogus.鈥�
Shortage of Studies
In his book, Mr. Koretz says a similar situation occurred during the 鈥淭exas miracle鈥� of the 1990s: Minority students posted large gains on the high-stakes Texas Assessment of Academic Skills, but much smaller ones on NAEP.
Mr. Koretz said the relative dearth to date of comparative studies on large-scale state assessments isn鈥檛 for lack of trying. He said he and other scholars have often been rebuffed after approaching officials about the possibility of studying their assessment systems.
鈥淭here have not been a lot of studies of this,鈥� Mr. Koretz said, 鈥渇or the simple reason that it鈥檚 politically rather hard to do, to come to a state chief and say, 鈥業鈥檇 like the chance to see whether your test scores are inflated.鈥�?鈥�
Bella Rosenberg, an education consultant and a longtime special assistant to Albert Shanker, the late president of the American Federation of Teachers, echoed many of Mr. Koretz鈥檚 critiques.
鈥淭his is officially sanctioned malpractice鈥攊f we did this in health, in medical practice, we鈥檇 all be dead by now,鈥� she said of the testing system under the NCLB law. 鈥淭he fact is, we鈥檙e doing a great job of obscuring instead of illuminating achievement gaps.鈥�
But fellow panelist Roberto Rodriguez, a senior education adviser to U.S. Sen. Edward M. Kennedy, D-Mass., the chairman of the Senate Health, Education, Labor, and Pensions Committee who helped shepherd the NCLB law through Congress in 2001, cautioned against 鈥渢hrowing the baby out with the bath water.鈥�
While a more effective accountability system 鈥渘eeds to be a goal, I would argue, of the upcoming reauthorization of the [Elementary and Secondary Education Act],鈥� he said, that 鈥渟ummative assessment is a critically important area of accountability.鈥� The No Child Left Behind law is the current version of the ESEA, first enacted in 1965.
On the tug of war between advocates of the current system and detractors who prefer that tests better reflect the course content that students are supposed to be mastering, Mr. Rodriguez said, 鈥淚 think we can have our cake and eat it too.鈥�
Incentives for Instruction
But Mr. Koretz said some parts of the nearly 7-year-old NCLB law need to be fundamentally reworked.
鈥淚 think we ought to mandate, as part of No Child Left Behind, independent evaluations of state accountability programs,鈥� he said. 鈥淭here is nobody involved in this system who has an incentive to look for good instruction anymore鈥攁ll the incentives are lined up in one direction: Increase scores on the summative tests at any costs.
鈥淲e need to create a system in which somebody 鈥� has incentives to make sure we鈥檙e not just gaming the system.鈥�
Williamson M. Evers, the U.S. Department of Education鈥檚 assistant secretary for planning, evaluation, and policy development, who attended the panel discussion, suggested that any changes to the ESEA law should preserve its essence.
鈥淪ome are proposing changes in accountability that remove consequences; this would make it accountability in name only,鈥� Mr. Evers said in an e-mail. 鈥淩esults should be comparable statewide, or parents, taxpayers, and teachers themselves can鈥檛 tell how students are doing.鈥�
Bruce Fuller, a professor of education and public policy at the University of California, Berkeley, praised the points Mr. Koretz makes about testing under NCLB in his book, and suggested steps to address the law鈥檚 flaws.
鈥淲ashington should first ensure that states use a common benchmark that defines proficient student achievement in reading, writing, and math,鈥� Mr. Fuller, who is the director of the Policy Analysis for California Education research center based at Berkeley and Stanford University, said in an e-mail. 鈥淪econd, neo-No Child policies should ensure that each state sets a valid way of tracking student growth over time, and provide states the resources to build necessary data systems.鈥�
