Beta testing is one of the most basic steps on the path to getting education products and ideas into the classroom鈥攁nd, researchers and developers say, one of the trickiest to get right.
As applied to K-12, the term generally refers to the early stage of testing almost any product in schools鈥攁 game, an assessment, a software package, a personal computer鈥攁nd then refining it, based on how well it works.
School districts typically agree to take part in beta testing because they will get access to a product or service they believe will help them鈥攚hich could be a new learning tool for their students or a professional-development program for their staff members.
Yet beta testing can also pose challenges for both schools and product developers. When superintendents and school boards agree to beta-test a product, they know that they may be creating extra work for teachers and administrators who need to be trained in how to use it, and that their schools may be forced to carve out class time for trying it out.
Those concerns, among others, sometimes lead districts to reject offers to stage beta tests, to the frustration of developers desperate to take products for test flights in schools.
To complicate matters, beta testing can mean very different things within the research community, and within schools.
As in other fields, like medicine, education beta tests are often defined as efforts to test something through a series of pilots or experiments to see if a product does what it鈥檚 supposed to before it goes live on the market, said Grover J. 鈥淩uss鈥� Whitehurst, a former director of the Institute of Education Sciences, the main research arm of the U.S. Department of Education.
In other cases, a beta test can include a far more structured process to evaluate something in a scientifically rigorous fashion, such as through a randomized control trial, said Mr. Whitehurst, who now directs the Brown Center, a research division at the Brookings Institution in Washington focused on school improvement. In those experiments, one group of students might be exposed to a product, while a second one is not, and results are compared between the two groups.
In any case, the process is 鈥渃ritical for any product that has any degree of innovation in it,鈥� said Mr. Whitehurst, who conducted beta tests on reading and assessment products he helped develop before joining the federal government. 鈥淚f the developer intends for it to accomplish something that hasn鈥檛 been accomplished before, it鈥檚 absolutely necessary,鈥� he said.
Strengths and Shortcomings
Finding a district willing to act as a partner in beta testing is one thing; staging a successful pilot test that leads to a product breaking into the K-12 market is something else.
Elliot Soloway, a professor of computer science and engineering at the University of Michigan in Ann Arbor, who has conducted several beta tests as a product developer, has experienced that reality firsthand.
Several years ago, Mr. Soloway said, he worked with a company that was beta-testing pocket personal computers鈥攎eant to serve as 鈥渕obile learning environments"鈥攁mong a relatively small group of teachers and about 150 students in a school district in an Eastern state. (He declined to name the school system or the state.) The teachers in the group quickly became adept at using the tool. Students鈥� test scores improved. So did their record in turning in homework on time.
鈥淚t went swimmingly,鈥� Mr. Soloway recalled.
The trouble started when the product was scaled up for use by more than 50 teachers and 1,500 students. Costs rose. While the system鈥檚 software and hardware worked well in the original test group, unexpected problems emerged when teachers in the larger group used the devices, probably because those educators lacked the tech-savvy to resolve glitches on their own, Mr. Soloway speculated. Teachers confused by the technology tended to stop using the devices 鈥渄ead in their tracks,鈥� without providing developers with enough information to diagnose the problem, he said.
Another complication: While teachers in the original test group were able to craft learning activities to make use of the technology鈥檚 features, teachers in the larger group were unable to figure out how to make the technology work for their curricular needs鈥攁nd believed doing so simply added to their workload, he added.
The pocket personal computer system never made it to the marketplace.
The project unraveled because of a computer 鈥渂ug rebellion鈥� and a 鈥渢eacher rebellion,鈥� Mr. Soloway said. 鈥淚t was horrible. ... We got lured into thinking we had a successful pilot; we can roll it into the rest of the school. Not a chance.鈥�
The most obvious temptation among developers is to structure the beta test in a way that will heighten the chances a product will succeed, which can undermine the process, Mr. Soloway said.
In those circumstances, 鈥渙f course it鈥檚 going to succeed,鈥� he said, but the developer finds 鈥測ou don鈥檛 learn enough鈥� about the product鈥檚 true strengths and shortcomings.
Mr. Soloway still consults with education companies, and he still does beta testing, but he says he has learned from earlier setbacks. Today, he says he pays more attention to providing teachers with tech support, and help crafting lessons that mesh with the technology. He adds that experience has taught him that developers and district officials need to work closely together to choose a range of teachers and students with different backgrounds, who are likely to have different degrees of willingness to accept the technology, rather than cherry-picking participants.
Michael Casserly, the executive director of the Washington-based Council of the Great City Schools, said that when district leaders resist allowing beta testing in their schools, it鈥檚 often because they worry that process isn鈥檛 transparent, and that the results may not be portrayed accurately. Concerns about creating new administrative burdens and training tend to be less important, he said.
The council will agree to help coordinate or recommend beta tests among its 67 member districts only if developers agree to make the results of those tests public鈥攅ven if they produce undesirable results, Mr. Casserly said.
His group also expects developers to choose independent evaluators, rather than people predisposed to cast the results in a positive light. In addition, the council recommends that the individual districts it represents follow those same guidelines when considering allowing beta testing, independently of the organization.
When developers are told of the council鈥檚 expectations, 鈥渁ll of a sudden they鈥檙e not interested,鈥� Mr. Casserly said. 鈥淭he last thing they want is to test a product, then have all of the nation鈥檚 major school districts know that it doesn鈥檛 work.鈥�
Evidence-Based Decisions
Developers sometimes conduct beta tests in different phases, with different goals in mind.
That was the case with an experiment being conducted by Melanie Stegman鈥攖he director of the learning technologies program at the Federation of American Scientists, in Washington鈥攚ho is conducting a beta test of Immune Defense, a video game designed to teach students about immunology in an interactive and entertaining way.
Ms. Stegman is testing the game among students studying video-game design and other subjects at McKinley Technology High School, a public school in Washington. Those students are relatively adept with technology, but that鈥檚 OK, Stegman says鈥攖he goal of this part of the test is to understand on a basic level whether the game works without glitches, whether students understand how to play the game, and whether they enjoy it.
She said she also asks those teenagers for suggestions about its design.
Ms. Stegman is now recruiting teachers鈥攕ome tech-savvy, some not鈥攖o participate in the next phase of her research, a randomized control trial which will ask them to use the game over three separate days, for one class period each day. The experiment will help determine whether students鈥� knowledge of cell biology increases as a result of playing the game, compared with other classroom activities considered to be best practices, she said.
At that point, 鈥渨e鈥檒l be able to test whether the game is really working,鈥� she said. 鈥淚s it teaching kids biology?鈥�
Chris Dasenbrook, a teacher who is overseeing the classes where Immune Defense is being tested, said the project appeals to him because it gives students insights on the challenges faced by actual game developers. He has carved out time for students to test the game in three different classes, including ones on interactive media and 2-D concepts.
鈥淎 lot of my kids play video games,鈥� Mr. Dasenbrook said, but 鈥渋t鈥檚 different when you鈥檙e pilot-testing. You鈥檙e chasing down bugs. You鈥檙e putting the program through its paces.鈥�
Ultimately, he said, the beta test shows students the challenges in transforming an idea for a game into a 鈥渃ommercially viable product.鈥�
Not all developers and researchers are able to find willing partners in K-12 to help them test their products.
When he was the director of the Institute of Education Sciences, Mr. Whitehurst said, members of the research and development community occasionally asked him to consider supporting a requirement that schools take part in beta testing in exchange for federal funding, such as Title I money鈥攁 step he considered a major overreach.
Still, Mr. Whitehurst said, federal officials should consider providing incentives to districts, possibly through grants provided for education research, to encourage beta testing.
Developers, he added, could make the process more attractive to districts by agreeing to carve out time during their beta tests to conduct research specific to the needs of those school systems and their personnel. Mr. Casserly of the Council of Great City Schools, meanwhile, believes districts and developers would benefit if school systems set clearer procedures for how beta tests are to be conducted and managed.
Encouraging more schools to take part in beta testing would have the side benefit of creating a new, more sophisticated class of consumers of educational products, Mr. Whitehurst added.
鈥淚t would be a way to get school leaders to buy into the importance of using evidence to make decisions,鈥� he said.
