Have a student ask an artificial intelligence image generator, like Google鈥檚 Gemini, for a picture of a doctor and it will probably pull up a white man in a lab coat and stethoscope.
Ask a generative AI tool like ChatGPT to and it will give higher marks to the classical lover than the rap fan.
Robots and computers are machines. They don鈥檛 have their own opinions about a user鈥檚 race, income, or fondness for Beethoven over Kendrick Lamar.
But AI technology is only as good as the data it鈥檚 been trained on. And that data often reflects the biases inherent in society.
鈥淲e assume [AI] is neutral. We assume that it鈥檚 not prone to [human] biases,鈥� said Punya Mishra, the associate dean of scholarship & innovation at Mary Lou Fulton Teachers College at Arizona State University.
But AI tools are 鈥渁s biased as we are because they have been trained on us,鈥� he said. 鈥淭hey are a black mirror to us.鈥�
As AI transforms K-12 education鈥攑roviding everything from lesson planning assistance for overworked teachers to chatbot tutors for students鈥� educators must be aware of how societal biases reflected in the data that underpins AI can shape its responses, experts say.
Those biases aren鈥檛 limited to race, gender, or family income issues, emphasized Erin Mote, the co-founder and executive director of InnovateEDU, a national nonprofit that seeks to build school districts鈥� capacity on policy and technology. They can include whether a student is an English learner, has learning and thinking differences, or even whether they are performing on grade-level.
鈥淎I can鈥檛 think about context. It鈥檚 really just about patterns and inconsistencies,鈥� Mote said.
She encourages educators to embrace the benefits that AI offers, while ensuring the tech never has the final say on decisions about students. If AI was the ultimate decisionmaker, schools would be 鈥渞eally, really taking out the valuable knowledge that educators bring,鈥� she said.
In other words, if they blindly trust AI鈥檚 recommendations, educators risk using cutting edge technology to double down on the very types of discrimination schools are working to move past.
And putting products that are untested for bias into classrooms could come at a high cost for schools and ed-tech developers, warned Nathan Kriha, a P-12 policy analyst for The Education Trust, a civil rights organization.
It could mean 鈥渨e will have some potentially incredibly problematic tools embedded into school systems鈥� across the country, Kriha said.
Bias problems emerge due to lack of information
Computer scientists and even many educators were aware of AI鈥檚 bias problems even before ChatGPT came on the scene in late 2022.
AI-powered-assessment algorithms used to figure out criminal sentences , studies have shown. Amazon鈥檚 former AI-powered hiring system appeared to screen out female applicants, Facial recognition technology is
鈥淲e have a big challenge because the underlying technology that this was built on is inherently biased and we know it,鈥� Mote said.
Often, the bias problems emerge because the technology doesn鈥檛 have nearly as much information about one group as it does about another.
In a K-12 context, AI might look at different types of students. And while most datasets contain a lot of information about native English speakers and neurotypical students, they don鈥檛 have nearly as much to draw from on English learners or students in special education.
By the same token, generative AI tools like ChatGPT develop鈥攕ome say 鈥済et smarter"鈥攂y absorbing huge swaths of the internet. But most online content is written in English and originated in Western countries. That opens the door for huge potential bias in its recommendations.
鈥淲e know that the output is going to be more white-centered, it鈥檚 going to be more male centered,鈥� Kriha said. 鈥淲hat happens if [responses] have historical inaccuracies? Or lack cultural context? There鈥檚 so much potential here for things to go wrong and for bias to bubble up to the surface.鈥�
In some cases, AI鈥檚 biases and what鈥檚 triggering them are obvious, Mishra said. But that鈥檚 not always the case because 鈥渢hese things are black boxes,鈥� he added.
That means even the tech鈥檚 creators can鈥檛 precisely trace where the information they synthesize and spit back out to users comes from鈥攁nd therefore can鈥檛 explain why AI might give very different responses to similar queries.
Some experts predict that the biases in widely used large language models like ChatGPT will eventually weaken, since people all over the world are training the technology over time through their queries and responses.
To illustrate this idea, Zafer Unal, a professor in the College of Education at the University of South Florida, notes that his own grasp of English has improved dramatically since he came to the United States from Turkey nearly two decades ago.
Mastering the new language took 鈥減ractice, practice, practice, practice, feedback, practice, feedback, practice,鈥� Unal recalled. In general, he suggests 鈥渋t鈥檚 the same with AI. AI is practicing with the use of data. If we give it data, will ChatGPT or other AI improve? Oh, definitely.鈥�
But Mote believes the biases will stick鈥攁nd perhaps even be amplified鈥攗nless AI technology incorporates more representative data, even if that data must be carefully curated or .
Real consequences if machines trump human judgment
If educators aren鈥檛 aware of AI鈥檚 biases鈥攁nd don鈥檛 ultimately rely on their own judgment over a machine鈥檚鈥攖here could be real consequences for students as these tools take hold, experts warn.
Case-in-point: One of the most exciting possibilities of AI for K-12 educators is its potential for personalizing lessons for students.
But AI鈥檚 feedback on student work 鈥渕ight be right for, say, a neurotypical child and maybe not right for a neuroatypical child,鈥� said Ashok Goel, a professor of computer science and human-centered computing in the School of Interactive Computing at Georgia Institute of Technology, who is developing and testing an AI chatbot to assist adult learners. 鈥淏ut the AI will not be able to make that distinction because it doesn鈥檛 have the data on neuroatypical children鈥� since that population is 鈥渉arder to collect data on.鈥�
Similarly, voice recognition software used to gauge a student鈥檚 reading level may not accurately assess students with strong regional accents or those whose first language isn鈥檛 English.
AI-powered chatbots could help a student who is essentially on grade-level but may not have the sophistication to serve students who are further behind, warned Jeremy Roschelle, the co-executive director of learning science research for Digital Promise, a nonprofit organization that works on equity and technology issues in schools.
And an AI-powered system relying on historical data to decide whether a particular student should be suspended, or which kids should be invited to take Advanced Placement Biology might dole out tougher punishments for students of color and send more white students to gifted classes.
鈥淚f models are trained on [discriminatory] data, then they will produce recommendations that are discriminatory,鈥� said D鈥橝ndre Weaver, the chief digital equity officer for Digital Promise.
That鈥檚 a problem for even the most responsible ed-tech developers, Mote said.
鈥淢any of these companies are super well-meaning and really do want to be good stewards of equity, but the only data they have to train their models on is their own,鈥� she said.
Compounding matters: There鈥檚 a race underway among ed-tech providers to get AI-powered K-12 tools to market, experts say. As developers rush to create these products, they may not be carefully testing for biases.
鈥淵ou have got to figure that the future has scandals,鈥� in which researchers realize an AI tool is biased long after it has been used to inform teaching and learning in many schools, Roschelle said.
Think of AI as a brainstorming partner, not an authority
Thoughtful regulation of education technology at the federal and state levels could head off or alleviate that nightmare scenario, Roschelle added.
But absent those policy safeguards, there are immediate steps schools can take to limit potential problems.
Number one on districts鈥� to-do lists should be training teachers carefully on how to use these tools, including a strong 鈥渕essaging campaign鈥� about bias, Kriha said.
鈥淚鈥檓 worried that teachers might not even know that these biases exist in the first place,鈥� Kriha said. 鈥淗opefully, [the messaging campaign] would make teachers a bit more conscious about over relying on these tools in classroom settings.鈥�
Mishra suggested teachers think of AI as a brainstorming partner, not an authority.
鈥淲e need to be seeing this鈥攁t least in the K-12 context鈥攁s a possibility engine rather than helping us do actual tasks because it鈥檚 not there yet,鈥� he said. Schools need to be asking themselves 鈥渉ow do we build critical thinking skills in our educators and in our students not to take this [AI] stuff at face value?鈥�
Schools can also ask some basic questions of vendors to get a sense of whether their tools have been investigated for bias, experts said. Those could include: What is the purpose of this tool? Has the tool been proven to be effective for that purpose? Does the vendor have studies, research, or other evidence that the tool is effective with different types of students, including English learners and students in special education?
鈥淭here are a lot of questions that vendors should have good answers to, and if they don鈥檛, that鈥檚 a red flag,鈥� said Hannah Quay-de la Vallee, a senior technologist for the Center for Democracy & Technology.
Schools should be wary if a vendor can鈥檛 鈥渢ell you what they鈥檝e done to assess their tool for bias鈥� as opposed to being able to say, 鈥淗ere鈥檚 our data, here鈥檚 what we鈥檝e found,鈥� she explained. Another warning signal, according to Quay-de la Vallee: 鈥溾�楧on鈥檛 worry, we checked but it鈥檚 a trade secret.鈥欌��
Schools also need to be aware that a particular tool may not be effective with their students. For instance, if an AI-powered application was trained on English learners, but they were all native Spanish speakers, it might not be a good choice for a district where most English learners speak another language, such as Chinese, at home.
鈥淵ou can have the best vendor with the best tech, and it still might not work in your context,鈥� Quay-de la Vallee said.
To think through these issues, it may be a good idea for a district technology office to work with staffers who specialize in equity or diversity, Quay-de la Vallee said.
鈥淎bsolutely talk to your diversity, equity, and inclusion office,鈥� she suggested. 鈥淗ave them do a first pass of like, 鈥極oh, that鈥檚 like pinging some red flags. We need to do some checks and balances there.鈥欌��
AI detectors are popular among educators but bring big bias problems
Schools should also stop using AI-related tools that are shown to be ineffective or biased. Case-in-point: Right now, one of the most problematic AI-related tools educators use regularly is, ironically, AI detectors.
More than two-thirds鈥�68 percent鈥攐f teachers report using an AI detection tool regularly, according to a survey of 460 6th to 12th grade public school teachers conducted for the Center for Democracy & Technology, a nonprofit organization that aims to shape technology policy.
But researchers ran essays written by Chinese students for the Test of English as a Foreign Language, or TOEFL, through seven widely used detectors. They did the same with a sample of essays written by U.S. 8th graders who were native English speakers.
In fact, Kristin Woelfel, a policy counsel specializing in equity and civic technology for the center, believes that schools that continue to use the detection software may be committing a 鈥減otential civil rights violation where these kids are going to be flagged more often and potentially either disciplined or given some kind of academic consequence like a change in grades.鈥�
The burden of creating opportunities for all students to learn about AI
K-12 schools must address another AI equity problem: Even though an understanding of AI is expected to be a highly valued skill in the future job market, most students don鈥檛 have access to AI coursework.
For now, it is unclear which schools make AI part of the curriculum鈥攁nd whether students from a variety of backgrounds benefit. But Black, Hispanic, and Native American students have less access than white and Asian students to classes in computer science, the broader subject of which AI is a subdiscipline, according to Code.Org, a nonprofit that seeks to expand computer science coursework.
Those students, as well as those from low-income families and children who live in rural areas, are also less likely to have a strong broadband connection, a computing device, and a teacher who understands how technology can effectively advance instruction.
Historically, the nation has a 鈥渉orrendous track record of providing equitable access to new technologies for students of color, and students from low-income backgrounds,鈥� said The Education Trust鈥檚 Kriha.
鈥淎nd if we as a society are living under this assumption that these tools are going to be important for professional life, there needs to be open conversation about this equity of access鈥� to AI tools and coursework in both the technical aspects of AI and AI literacy, he said.
Ensuring students from all types of backgrounds are exposed to AI and given the chance to consider careers in this burgeoning corner of computer science isn鈥檛 just an equity imperative, experts say.
It may be the best possible defense against bias.
鈥淭his is not just a high level like, 鈥極h, wouldn鈥檛 it be nice,鈥欌�� said Quay-de la Vallee from the Center for Democracy & Technology. 鈥淲e see in the technology we build that these are failures because we build [technology] for the circumstances that are most obvious to the people who built it.鈥�
If more people from a Muslim background had worked on AI-powered facial recognition software, it might have become obvious sooner that many women wear veils called niq膩bs and , she said.
鈥淚t鈥檚 not like people building the tech don鈥檛 care about these populations. It鈥檚 that a lot of these disparities are invisible to them,鈥� Quay-de la Vallee added. 鈥淭hat is just going to continue to get worse if we are not building a pipeline that brings in people who have lived experiences that allow you to clock [bias problems] very early on in the development process.鈥�
