Traditional methods of studying social-emotional skills will have to evolve in more reliable, less subjective ways if educators and policymakers expect to incorporate them validly into accountability systems and school improvement plans, education researchers meeting here last week cautioned.
The federal Every Student Succeeds Act broadens the definition of school success, requiring states and districts to include nonacademic factors in their accountability systems. Concepts like growth mindset鈥攖he belief that intelligence and other skills are not fixed, but can be improved through effort鈥攁nd grit鈥攖he ability to sustain interest over a long period and persist in a task in the face of boredom or challenges鈥攈ave garnered great interest as potential levers to lift student academic achievement through non-academic skills.
鈥淚t鈥檚 only in the last five years that mindset has moved from an influential academic theory to an educational phenomenon,鈥� David Miele, an education professor at Boston University, said during a symposium on the research at the American Educational Research Association鈥檚 annual conference. The gathering drew more than 16,500 researchers from around the world.
But the research studying the skills hasn鈥檛 quite caught up with their rising popularity, some scholars said.
In an analysis of 167 National Science Foundation-funded studies of those skills鈥攊ncluding 88 interventions designed to improve such qualities as motivation, self-efficacy, and persistence鈥擩olene Jesse, a program director in the National Science Foundation鈥檚 education and human resources directorate, said the instruments being developed are largely self-reports.
For example, to identify changes in a student鈥檚 鈥済rittiness,鈥� a researcher might ask students to rate, on a scale of 1 to 5, how much they agree with such statements as 鈥淪etbacks don鈥檛 discourage me鈥� or 鈥淚 finish whatever I begin.鈥�
Beliefs vs. Actions
Such methods are long-established, but in the symposium and several other discussions at the AERA meeting, researchers warned that surveys like these can be muddier and vulnerable to biases.
鈥淟et鈥檚 say we were studying students鈥� math ability,鈥� said Evan Heit, the director of the NSF鈥檚 division of research on learning. 鈥淗ow satisfied would we all be if we did that by simply asking students, 鈥極K, how good are you in math?鈥� We probably would not be very satisfied. So should we be using self-reports for [social-emotional learning]?鈥�
For example, Lee Shumow, an educational psychology professor at Northern Illinois University, evaluated how an intervention affected teachers鈥� and students鈥� growth mindsets. After conducting teacher training and a six-week curriculum designed to boost growth mindsets among teachers and students, Shumow used self-reported surveys but also observed 10 7th grade and 15 high school science classes.
She measured how often teachers used feedback to students that supported growth mindset鈥攆or example, 鈥淵ou did well on this test; see how your studying is paying off?"鈥攁nd how often they made comments that would undermine a growth mindset, such as comparing students鈥� test scores or saying, 鈥淵ou aced this test; see how smart you are?鈥�
Students whose teachers provided more growth-oriented feedback had better performance and higher levels of growth mindset.
During teacher training, the middle and high school teachers participating in the growth-mindset intervention answered survey questions in ways that suggested each had a strong growth mindset.
鈥淲e did measure teachers鈥� mindsets using a standard survey to measure mindset,鈥� Shumow explained, 鈥渂ut teachers are pretty savvy. ... They knew exactly how they were supposed to respond, and they answered [the questions] accordingly.鈥�
And sure enough, during classroom observations, the teachers who were part of the intervention made more comments supporting growth mindset than did teachers who had not taken part in the intervention鈥攂ut the intervention teachers also made more undermining comments than the teachers in the control group.
鈥淲e think the statements the teachers made and the behavior they displayed in the classroom鈥� reflect a conflict between the explicit beliefs they are learning and implicit beliefs that may be more fixed, Shumow said. 鈥淲e think the implicit beliefs are where it鈥檚 at.鈥�
Heit, of the NSF, pointed to a recent Brookings Institution report calling for researchers studying social skills to distinguish better between character traits and the related behaviors that can be trained. Heit and his colleague Jesse called for more observational protocols that would help teachers and researchers understand what grit or a growth mindset looks like in day-to-day practice.
True Grit? It Depends
Even when students or teachers answer surveys frankly, they may be influenced by comparing their own behavior with that of their peers鈥攚hat researchers call reference bias.
That鈥檚 why prior studies have found very high-performing students in competitive schools often report being less hardworking than they actually are.
It can also be why between-school comparisons of grit don鈥檛 always work well, according to another study presented at the meeting by University of Pennsylvania psychology professor Angela Duckworth, who coined the term 鈥済rit"; mindset researcher David Yaeger at the University of Texas at Austin; and colleagues at Stanford University and the University of Notre Dame.
The researchers compared two types of student surveys of academic persistence with a behavioral test measuring how long high school students would continue to perform a difficult but mundane task while being distracted.
Students鈥� own reported perseverance predicted how likely they were to complete their first year of college in comparison with others at their own school. But the student reports were not accurate for comparing the college-persistence rates between one high school and another.
By contrast, performance on the behavior task did accurately predict differences in college persistence both within the high schools and between them.
As children grow to adolescence, Duckworth said, they may compare themselves more to their peers, which may also affect how accurately they report their own persistence, self-efficacy, or self-control.
One of the limitations of using student self-reports to measure grit, Duckworth said, is they are 鈥渁 judgment based on a mental model that鈥檚 influenced by many things other than the objective behavior.鈥�
In fact, in a separate study, Duckworth and postdoctoral researcher Lauren Eskreis-Winkler flipped the grit research structure on its head: They used a survey designed to get students thinking about other, younger students as the intervention to increase students鈥� own gritty behavior.
In a randomized controlled experiment, the researchers gave 550 middle school students basic information on grit and randomly assigned some to fill out a survey giving 鈥渢ips鈥� to 4th graders on how to be more persistent and gritty. Then the researchers asked all the students to complete challenging math problems in an online program, but added that they could 鈥渢ake a break鈥� and play simple entertainment games whenever they wanted.
In both cases, students who had acted as 鈥渕entors鈥� to other students via the surveys persisted in the math task longer and with fewer breaks than those who had not been mentors. The effect was strongest for students who were initially deemed low-performing in math.
The results have also been repeated with three groups of adults: at-risk community college students, unemployed workers trying to find a new job, and smokers seeking to quit.
