As two state consortia work to finish new assessments aligned with the Common Core State Standards, some mathematics experts say they鈥檙e worried that the computer-based testing platforms will hamper a key element of the exams: open-ended math-performance tasks that test students鈥� ability to apply their knowledge.
Unlike previous state assessments, those being developed by the two federally funded consortia will include complex, multipart word problems that students will answer on screen. While some of those questions will provide built-in tools that allow students to put points on a graph or draw lines on a ready-made picture, other questions will ask them to write their answers in narrative form, using a keyboard.
Some experts contend that forcing students to write a solution doesn鈥檛 match the expectations of the common-core math standards, which ask students to model mathematics using diagrams, graphs, and flowcharts, among other means.
鈥淚t鈥檚 not like, during the year in classrooms, these kids are solving these problems on the computer,鈥� said David Foster, the executive director of the Morgan Hill, Calif.-based Silicon Valley Mathematics Initiative, which provides professional development for math teachers, creates assessments, and has worked with both consortia. 鈥淚t鈥檚 such an artificial idea that now it鈥檚 test time, so you have to solve these problems on computers.鈥�
But experts with the two state consortia鈥�the Partnership for Assessment of Readiness for College and Careers, or PARCC, and the Smarter Balanced Assessment Consortium鈥攕ay the tests that are slated to debut this school year, while imperfect, are a huge step forward in assessment methods, and have advantages over paper-and-pencil tests of the past.
The tests will also continue to improve as developers analyze the field-test results over the next few months, and in coming years as freehand drawing tools and other capabilities are incorporated, they say. For now, though, costs, security issues, and timing have prevented the use of more advanced technology.
Questions about the accuracy of the new tests are especially critical in states where teacher evaluations will be tied to the 2015 results. Many states agreed to link teacher evaluation and student achievement to improve their chances of getting federal Race to the Top grant money or No Child Left Behind Act waivers. While the U.S. Department of Education recently gave states the flexibility to hold off on tying test scores to teacher evaluations, many states did not take that option.
Classroom vs. Testing
Each performance task on the common-core tests asks students to apply a variety of skills to a novel scenario.
For instance, a sample middle school PARCC item provides a table with the number of brass and percussion instruments in three different marching bands. Students are asked to calculate the proportion of brass to percussion players using an on-screen 鈥渆quation editor,鈥� which they click on to put in numbers and symbols. Eventually, they鈥檙e asked to solve a multistep problem about the makeup of a new band and 鈥渟how or explain鈥� all their steps.
For Smarter Balanced, students will do one performance task; it will generally take 45 to 90 minutes, depending on the grade level. Students taking the PARCC assessment will likely encounter two sessions of about 50 to 100 minutes with up to 20 shorter performance-based tasks, though these details have yet to be finalized. For both consortia, parts of the performance tasks will be scored by hand. Students will be allowed to use scratch paper throughout both consortia鈥檚 tests, but those sheets will not be turned in or scored.
Currently, at least 26 states and the District of Columbia are planning to use the assessments developed by either PARCC or Smarter Balanced. Also, some of the states that have backed away from earlier plans to use the consortia tests may well use individual test items.
Math educators are quick to differentiate between concerns about the content of the tests and the testing platforms.
鈥淚 think there are some potentially really good questions [on the tests], but when you give kids a great question but all they can use to solve this problem is the keyboard in front of them, you sort of put handcuffs on the kids,鈥� Mr. Foster said.
Martin Gartzman, the executive director of the Center for Elementary Mathematics and Science Education at the University of Chicago, has been leading the charge among a group of Illinois math educators who are worried about the PARCC testing platform.
鈥淭he primary tools kids use to solve problems and justify their answers every day in math class and required by the common core aren鈥檛 available for kids to use during these tests,鈥� he said. 鈥淲e鈥檙e going to get tests that are just going to frustrate people and are going to really underrepresent what students know.鈥�
Thought Processes
But being able to communicate a thought process in words is a critical skill, said Doug Sovde, PARCC鈥檚 director of content and instructional supports. 鈥淚 think it鈥檚 entirely reasonable for somebody to work through something on scratch paper and be able to share their reasoning after they worked through the task,鈥� he said. 鈥淚t鈥檚 not unlike a student making an outline before they write their research paper.鈥�
Jan Mulqueeny, an assistant superintendent in District 126 in Alsip, Ill., said she witnessed students struggling with the transfer from scratch paper during PARCC field testing last spring.
鈥淚 saw [4th grade] students writing their answers to the problems through drawings and creating tables and a variety of ways. But when they tried to replicate that in the text box and with the equation editor, they were totally stymied,鈥� she said. 鈥淲hen you picked up the student scratch paper and saw the work they did, and saw what was on the screen, there was certainly a discrepancy.鈥�
PARCC is planning to eventually provide a drawing tool for students, said Mr. Sovde.
鈥淲e are working to develop something that works on all devices, all browsers, Java, et cetera,鈥� he said. 鈥淭hat鈥檚 going to take some time to get it right, and we are pursuing that functionality.鈥�
Shelbi Cole, the director of mathematics for the Smarter Balanced Assessment Consortium, said it鈥檚 important to understand that a lot of research has gone into item development to help ensure test-score validity. The group is analyzing field-testing results now, including comparisons of how students did on the same items using computers vs. paper and pencil.
鈥淭he stuff that ends up in the operational pool is what lives after we鈥檝e done all this research,鈥� Ms. Cole said.
Further, she said, computer-based administration provides opportunities that aren鈥檛 possible with paper-and-pencil tests, such as the use of spreadsheets or problems that mimic the development of an app. Those aren鈥檛 ready now, but 鈥渨e have a plan in place鈥� for such features down the road, she said.
More broadly, 鈥淚 don鈥檛 know that math class should be a paper-and-pencil subject anymore,鈥� Ms. Cole said. 鈥淭here are lots of great tools out there people are using.鈥�
Mr. Foster of the Silicon Valley Mathematics Initiative, who consulted for PARCC and wrote exemplar performance tasks for Smarter Balanced, strongly disagreed.
鈥淚鈥檓 a mathematician, and I never solve problems by merely sitting at the keyboard. I have to take out paper and pencil and sketch and doodle and tinker around and draw charts,鈥� he said. 鈥淥f course, I use spreadsheets all the time, but I don鈥檛 even start a spreadsheet until I know what I want to put in the cells.
鈥淎ll Smarter Balanced and PARCC are going to look at is the final explanation that is written down,鈥� he said, 鈥渁nd if there鈥檚 a flaw in the logic, there鈥檚 no way to award kids for the work they really did and thought about.鈥�
Mr. Foster added: 鈥淚鈥檝e played with the platform, and it makes me sick. And I鈥檝e done it with problems I鈥檝e written.鈥�
What the Standards Say
The common-core math standards explicitly set the expectation that students should at times create visuals to represent their understanding. The Standards for Mathematical Practice, which describe the processes that proficient math students use in solving problems, say students should be 鈥渁ble to identify important quantities in a practical situation and map their relationships using such tools as diagrams, two-way tables, graphs, flowcharts, and formulas.鈥�
A 7th grade geometry standard asks students to 鈥渄raw (freehand, with ruler and protractor, and with technology) geometric shapes with given conditions.鈥� A majority of the fractions standards in grades 3-5 ask students to solve problems by, for example, 鈥渦sing a visual fraction model.鈥�
鈥淲e know there鈥檚 a gap between what the standards say, what students are asked to do in the classroom, and what we鈥檙e measuring,鈥� Ms. Cole of Smarter Balanced said. 鈥淐ertain graphical sketches probably won鈥檛 be assessed in the exact way they鈥檙e called for in the standards. ... But these are isolated pieces in a much larger system in what we鈥檙e trying to do. We鈥檝e come a really long way.鈥�
Mr. Sovde of PARCC also emphasized that the new tests, while still being improved, are 鈥渁 massive step forward from where state assessments have been up to this point.鈥�
A math-assessment specialist who helped with item development and review for Smarter Balanced said she believes the platform works well.
鈥淚 feel like I鈥檓 able to understand and to hear what the student is thinking鈥� by reading his or her response, said Jennifer Judkins, who works in the Washington state education agency and has graded more than a thousand student responses to open-ended math questions for Smarter Balanced.
鈥淚 can鈥檛 say it鈥檚 100 percent perfect, but, for the most part,鈥� she said, 鈥淚 could understand what they were doing.鈥�
But, as James W. Pellegrino, a professor of education at the University of Illinois-Chicago who serves on the technical-advisory committees of both consortia, points out, students can solve a single problem in any number of ways, not all of which are easy to explain in words.
鈥淭he worry is [the platform] narrows the scope of what students can do, and the evidence they can provide about what they understand,鈥� he said. 鈥淭hat leads to questions about the validity of the inferences you can make about whether students really developed the knowledge and skills that are part of the common core.鈥�
鈥楴o Free Lunch鈥�
As Mr. Foster sees it, there鈥檚 at least one simple solution that could be incorporated soon: scan students鈥� scratch paper into the system.
鈥淪canning technology is so easy and so cheap in this day and age,鈥� he said. 鈥淭hat鈥檚 not high-tech stuff.鈥�
Most schools have had scanning technology for many years, he said.
Mr. Sovde of PARCC said the group is considering that option.
Even so, Scott Marion, an associate director of the National Center for the Improvement of Educational Assessment, in Dover, N.H., which advises both consortia, argues that scanning has its own set of problems. For instance, the papers may not be legible or they may not scan well, Mr. Marion said. Plus, 鈥測ou鈥檙e shipping these boxes and somebody in school is handling them, and that makes it easier for cheating.鈥�
He added: 鈥淭here鈥檚 no free lunch in this. It鈥檚 going to be a challenge no matter what way we do it.鈥�
