澳门跑狗论坛

MAP, iReady, Star鈥攖hese every-few-months tests are some of the most commonly used tools among the nation鈥檚 school districts, for a wide array of different purposes.

When school leaders say they champion 鈥渄ata-driven鈥� instruction, they鈥檙e often talking about the data from these assessments: Some use the tests to get a sense of how well kids are likely to score on year-end exams; some, to figure out a subset of skills students struggle with that teachers should prioritize for reteaching; still others use them to try to gauge whether a new tutoring initiative is working. (Or all three.)

But for all of their heavy use, there鈥檚 little external evaluation of their technical properties, or how well they align to the most commonly used curriculum. Nearly all of them are proprietary, owned by private companies.

Now, a recent attempt to look under the hood of these tools, officially known as 鈥渋nterim assessments,鈥� is on hold after several providers of the tests wouldn鈥檛 commit to participation.

The curriculum reviewer EdReports, known for its ratings of curriculum series, set its sights on evaluating interim assessments, a project that the group started working on in 2016.

The process that EdReports undertook, and the response from publishers, demonstrate the ways in which a key part of teaching and learning can be opaque for schools and the teachers who rely on the results.

The stakes are even higher now: Many districts are using the data from these tests to drive decisions about pandemic recovery.

鈥淒istricts are relying on this interim assessment data even more so to really gauge what happened during the pandemic,鈥� said Eric Hirsch, the executive director of EdReports.

What interim assessments measure

Because most of these tests don鈥檛 have external evaluations to demonstrate their effectiveness, EdReports wanted to take a closer look. It planned to include educator evaluators鈥攕imilar to its curriculum reviews鈥攁nd feedback from testing experts through collaboration with the Center for Assessment, a New Hampshire-based consulting group.

But unlike curriculum reviews, the assessment reviews would have required publisher participation to gain access to internal information about assessment design鈥攑articipation that EdReports didn鈥檛 get, for the most part.

Despite the clear value for district leaders, there鈥檚 little incentive for publishers to participate in such a review, said Gregory Cizek, a professor of educational measurement and statistics at the University of North Carolina Chapel Hill.

鈥淵ou have to be a pretty large, well-funded publisher who鈥檚 pretty confident that you鈥檙e going to look good,鈥� he said. 鈥淏ecause you don鈥檛 want these independent published reviews to say that your test is terrible.鈥�

One wrinkle concerns the wide range of tools that fall under the moniker 鈥渋nterim assessment.鈥� Different interim assessments serve different purposes.

Some claim that they gauge how students are doing in a particular subject area, like math or science, in such a way that would predict student performance on a state summative assessment.

Others are more geared toward providing information that can directly influence teaching. These assessments might evaluate students鈥� proficiency with specific skills鈥攏umber sense, for example鈥攕o that teachers or interventionists can focus on those skills with students, said Erika Landl, a senior associate with the Center for Assessment who worked with EdReports.

One of the goals of the EdReports analysis was transparency鈥攖o make it clear to district leaders what these assessment providers were claiming they could do, and what they weren鈥檛.

鈥淚 think there鈥檚 this black box about, 鈥榃hat does this mean, and what should I do with it?鈥欌�� said Landl, about interim assessment data.

She gave an example: As students progress throughout the school year, they will likely show improvement on an interim assessment that predicts performance on an end-of-year, summative test. But even though students are growing, they still might not have grown enough to reach the 鈥減roficient鈥� level on that summative assessment鈥攁 distinction that can be confusing.

EdReports also wanted to determine whether these tests actually measured the things that they claimed to measure. If an assessment provider says that its test predicts performance on an end-of-year state test, does it? If the company says it can pinpoint the skills that were causing students to struggle, can it?

A few organizations do this kind of analysis. The Buros Center for Testing at the University of Nebraska-Lincoln has reviewed more than 2,800 commercially available tests, and their reviews are available for purchase. But many tests aren鈥檛 included, said Cizek.

鈥淭hey often struggle to get some of the most high profile testing programs to participate. For example, the SAT is not in there. The NAEP is not in there,鈥� he said, referring respectively to the college-admissions tests and the federally administered 鈥渘ation鈥檚 report card.鈥�

The National Center on Intensive Intervention also 鈥攖hat they measure what they say they measure鈥攁nd reliability: whether the tool produces consistent results. But similarly to the Buros Center reviews, not every interim assessment provider participates.

To that landscape, EdReports review would have included educator feedback in addition to evaluation from psychometricians. Just as with curriculum, Hirsch said, it鈥檚 important for assessments to also be reviewed by 鈥渢hose who actually utilize them鈥濃�攁nd to package it in a user-friendly way.

鈥淢ost districts lack the capacity and the expertise to put these different component pieces together, interpret the technical manuals, and put these in a synthesized report,鈥� he said.

EdReports鈥� proposed assessment 鈥榞ateways鈥�

Similar to how it designs curriculum reviews, EdReports planned to organize the assessment review around different categories, or 鈥済ateways.鈥�

The first assessed alignment to college- and career鈥搑eady standards, as well as accessibility.

A second gateway examined technical quality, scored by independent, national psychometric experts. These questions probe whether the tests actually measure what they claim to measure.

The third gateway would have evaluated the clarity and usability of score reports, a task undertaken by both educators and technical experts.

The process asked publishers to submit a wealth of information, some proprietary: item examples, the algorithms that power adaptive tests, technical guides, and evidence guides.
The ask of companies was 鈥渟how us your work,鈥� Hirsch said.

The process was 鈥渞esource intensive and well worth it,鈥� said Kristen Huff, the vice president of assessment and research at Curriculum Associates, which had signed on to be part of the review before it was tabled in May.鈥淚t forced us to take all of these rich rationales that we had in our minds and in our words and put them on paper.鈥�

Other companies, though, were more hesitant. NWEA, which makes the suite of MAP tests, was invited to participate and declined. The company declined to comment. Renaissance, another assessment provider, hadn鈥檛 yet decided whether to participate in the review when EdReports paused the process.

鈥淭here鈥檚 value in this particular type of project because districts, states, others are very interested in how they evaluate one assessment provider over another. So I think it鈥檚 helpful to look at what those claims would be and whether those claims are met,鈥� said Darice Keating, the senior vice president of government affairs at Renaissance.

But other existing processes already gather much of this information鈥攍ike state requests for proposals, or the National Center on Intensive Intervention, noted Andy Frost, vice president of assessment product management at Renaissance. And it鈥檚 a 鈥渟ignificant鈥� effort to assemble all of the information that EdReports is requesting, he said.

鈥淭here would certainly be some utility. EdReports and the Center [are] tremendous organizations,鈥� he said. 鈥淭here would absolutely be some incremental value added. I think the question is how much and at what cost.鈥�

Hirsch acknowledged that the process could be 鈥渙nerous.鈥�

鈥淚 certainly don鈥檛 want to state in any way that this is an easy process on a publisher,鈥� he said.

Passing the baton to districts

Cizek, the UNC Chapel Hill professor, ultimately said getting companies to sign on is an uphill battle.

鈥淚t鈥檚 really a lose-lose for interim assessment providers,鈥� he said. Determining alignment and validity takes time and money. After seeing the review criteria, publishers might know that they don鈥檛 meet some of the requirements鈥攁round accessibility, for example鈥攁nd avoid the process as a result, he said.

鈥淚f their customers are happy with their product, why would they bother participating in this hassle?鈥� he asked.

Several companies that chose not to participate in EdReports鈥� reviews said that the criteria have still affected their decisionmaking.

鈥淲e are looking to those criteria as we build our new assessments. That鈥檚 big,鈥� said Laura Slover, the chief executive officer of CenterPoint Education Solutions, a nonprofit that works with districts to develop coherent instructional systems and creates assessments. She cited timing as the reason why the organization didn鈥檛 participate.

鈥淭o that extent, I think EdReports is already starting to have the impact, perhaps indirectly that they were hoping to have,鈥� Slover said.

The time and effort EdReports and the Center for Assessment put into the project isn鈥檛 wasted, either.

Landl and Susan Lyons, a principal consultant for Lyons Assessment Consulting, recently released for district leaders based on the criteria the group had put together for the EdReports reviews. Its aim is to be a version of the review process that districts can use to inform their decisionmaking鈥攅ven if they don鈥檛 have access to testing experts, Landl said.

The tool is a way to pass the baton to districts Hirsch said, 鈥渁nd say, 鈥榊ou may not be able to answer all of the questions we were trying to answer. But you may be able to answer some of them.鈥�