If Students Aren't Trying on International Tests, Can We Still Compare Countries' Results?

Save to favorites
Print

Copy URL

Many students seem to be blowing off a major international exam, leading some researchers to argue that the results paint a distorted picture of where countries stand in education rankings.

Worldwide, a high percentage of students either skip questions, spend insufficient time answering them, or quit the Program for International Student Assessment test early. As a result, a handful of countries fall lower in overall PISA rankings than they might if their students applied themselves, .

PISA, administered every three years to 15-year-olds all over the world, is one of the primary international tests used to get a snapshot of student performance worldwide and to facilitate policy comparisons. The new research raises fresh questions about such comparisons.

鈥淧ISA is taken as the gold standard, and countries live and die by their PISA rankings,鈥� said Kala Krishna, a liberal arts professor of economics at Pennsylvania State University, who conducted the research with two colleagues. 鈥淭he United States has been very upset because it doesn鈥檛 do so well on PISA, and that has led to a lot of concern that maybe we鈥檙e not using our money wisely.鈥�

In response to a series of emailed questions, an official at the Organization for Economic Cooperation and Development, the Paris-based group that runs PISA, faulted what she called the study鈥檚鈥渟ensational tone.鈥�

Shifts in ranking would be far smaller or even insignificant if the measurement errors associated with the exam, which is based on samples of students, are taken into account, said Miyako Ikeda, a senior analyst at the OECD鈥檚 directorate for education and skills.

Examining Online Test Data

Krishna and two colleagues, S. Pelin Akyol of Bilkent University in Turkey, and Jinweng Wang, also of Penn State, conducted the study, which was released as a working paper this month by the National Bureau for Economic Research. It has not yet been peer reviewed.

For the study, the authors mined keystroke data鈥攈ow long students spent on each test question and how they responded to various kinds of test questions鈥攆rom the 2015 online administration of PISA, which was taken by students in more than 58 countries, including the United States. The test measures skills and knowledge including math, reading, and problem-solving, among others, but PISA officials emphasize a different topic each administration. In 2015, the emphasis was on science, and the new research is based on data from that portion of the test.

From the data supplied by the OECD, the researchers devised criteria to identify students who weren鈥檛 putting their best foot forward on the test. Among those, for example, were students whose keystrokes showed that they answered questions in such a short period of time that they couldn鈥檛 possibly have read each of them.

They also highlighted test-takers who quit the exam early or didn鈥檛 answer more questions even when plenty of exam time remained. In all, the study found that students tended to bypass harder questions or open-ended ones more often than easy questions or those given in a multiple-choice format.

Both Krishna and Ikeda noted that those measures are judgment calls, since they measure test-taking behavior as a proxy for student effort.

Certain student and contextual attributes were linked with higher incidences of goofing off, the study found. Wealthier students and lower-skilled students tended to take the exam less seriously. Testing context seemed to matter, too. Countries in which students reported sitting for more 鈥渉igh stakes鈥� exams had a higher proportion of students blowing off PISA, which does not carry stakes for students.

In all, the proportion of students exhibiting what the researchers called 鈥渘onserious鈥� behavior on the test ranged from a low of 14 percent in Korea to a high of 67 percent in Brazil. In the United States, about 23 percent of test-takers fell into that category.

A Change in Rankings?

To find out how lack of student effort affected each country鈥檚 ranking, the researchers used a statistical technique to plug in the missing answers, based on how each student likely would have performed given his or her other answers and those of similarly skilled students.

Portugal, which ranked 31st on PISA, might have jumped up to 19th or 16th place if its students had made every effort on the test, depending on which model was used. Sweden would potentially have moved up as many as 11 spots and Norway as many as nine, they found.

High rates of nonserious behavior didn鈥檛 always affect rankings. Most countries at the very top and bottom of the rankings would stay there or move only a few slots. The United States鈥� overall rank in science, 27th, might improve about two to five notches depending on the statistical technique used, if all students did their best, the researchers estimated.

The OECD鈥檚 Ikeda pushed back on those estimates. If all students in all countries made a serious effort on the test, then the changes in ranking would be far less dramatic than the country-by-country examples the researchers chose to highlight, she said. Many if not all of those changes wouldn鈥檛 be significant when the sampling error associated with the test is taken into account, she contended.

For that reason, she rejected the researchers鈥� call to publish a set of adjusted rankings alongside the regular ones.

鈥淭hat is not to say that effort is not important鈥攚e agree that it significantly affects absolute performance鈥攂ut it does not affect relative performance rankings,鈥� she said.

The Downside of Low Stakes

Regardless of how the researchers鈥� thought experiment is interpreted, the results highlight two testing truisms.

First, comparisons of international student performance are fraught with caveats, 鈥渂ut ifs,鈥� and interpretive challenges.

And second, as students age, they don鈥檛 always take 鈥渓ow stakes鈥� exams like PISA seriously鈥攗nlike tests that stand to affect their grades or access to college. The United States鈥� major low-stakes exam is the National Assessment of Educational Progress, given as a dipstick of student learning in America in much the same way that PISA is internationally.

NAEP officials have . But Peggy Carr, the associate commissioner for the assessment division of the U.S. Department of Education鈥檚 National Center for Education Statistics, which administers NAEP, said that those students who do take the exam seem to be doing their earnest best. The percentage of students skipping questions on the 12th grade exam in 2015 was comparable to the rates for 4th and 8th graders.

Forthcoming keystroke data from the 2017 12th grade NAEP, which was given on tablets instead of bubble sheets, should provide even richer information on how students are engaging with test items, she said.