A Legacy of Fairness in Testing

By David Lariviere

Many aspects of American society have changed dramatically since ETS was founded in 1948. And ETS has had to adapt to remain an educational testing leader.

However, what has been constant throughout the years is the organization’s commitment to fairness.  From the 1960s and addressing the challenges brought on by the civil rights movement to incorporating new technologies of today, adhering to principles of fairness has never waned.

Rob Durso, Chairman of the Fairness Steering Committee, says, “We’re doing the right thing here. The tests have to be fair, it’s good measurement, and there needs to be a process in place to actually make it happen. They can impact people’s lives so we want to make sure they’re fair.”

Barbara Kirsh, who serves as Executive Director in the Legal division, credits the commitment to leadership at the top. “ETS President and CEO Walt MacDonald has always been a champion for fairness for all, including people with disabilities and accessibility. All our ETS presidents have been,” she said.

Michael Zieky has been a principal contributor to numerous versions of guidelines (the most recent edition published in 2014) that serve as the foundation for evaluating and assuring the quality and fairness of ETS products and services.

Before that, he became involved in the formal documented fairness review process, which was set up in 1980 and continues through today. The process is organized and managed by an independent body within ETS in order to maintain its credibility. “It’s a series of independent judgments and was set up that way on purpose,” Zieky says.

A challenge facing assessment developers, along with many others in education, is how fairness is defined. “The most common public definition is, well it’s not fair if it’s harder for one group than another. We categorically reject that definition,” Zieky says. “A difference in difficulty is not proof of being unfair. A common example is that women are usually shorter than men. That doesn’t mean the yardsticks are biased.”

For ETS, the best way to approach the ideal of fairness to all test takers is to make what are called sources of construct irrelevant variance as small as possible. These are aspects of a test item that are irrelevant to what’s being measured.

An example, cites Zieky, is sports, as ETS has a rule against requiring specific knowledge such as how many players are on a football team when the knowledge is not relevant to what is being measured. It would be required, for example, if it was a test for a physical education teacher in the United States.

A sensitivity review is conducted to avoid any stereotyping or anything being potentially offensive. Test questions are also analyzed for Differential Item Functioning, or DIF, a statistical methodology that was advanced at ETS and began to be a formal part of the test development process in the 1980s. The focus of DIF is to determine whether test takers from particular race, gender or ethnic groups who score about the same in general on a particular test (so are considered at about the same level of overall proficiency) perform less well on a particular question. If so, then the content of that particular question is reviewed to see if it included construct-irrelevant information.

The training for test developers is rigorous and Zieky says it “takes about a year to learn” to write good and fair test questions. “It’s a learned skill and it’s a very difficult thing to do.” Often, ETS test developers come to the organization as experts in a subject, like math or chemistry, and are further trained by assessment development experts within the organization on how to create effective and fair subject-relevant test questions.

Fairness guidelines are formally revised about every five years, but ETS is constantly evaluating its practices and procedures to make sure they are up to date. “We’re re-evaluating the DIF process right now,” Durso says.

Kirsh has spent much of her career working on issues related to fairness, serving as the first leader of the Office of Disability Policy, founded in 1997. “I’m currently the leader of an accessibility initiative at ETS corporate wide. Recent successes were making the GRE test digitally accessible and hosting (thanks especially to Director of Digital Accessibility Mark Hakkinen’s efforts) a Global Accessibility Awareness Day (in May) that wound up stretching into a full week,” she says. On behalf of the Accessibility Strategy Advisory team, she also makes quarterly presentations to senior officers on the Executive Accessibility Steering Committee led by MacDonald.

We are now at the beginning of a new era (with computerized testing) and we’re continuously learning how to incorporate fairness principles into new testing modalities.

~ Michael Zieky

Mary Pitoniak, the current Executive Director of the Office of Professional Standards Compliance and the Office of Disability Policy, says that both offices are involved with helping to ensure fairness in ETS programs and products. “The Office of Professional Standards Compliance conducts audits of testing programs using the ETS Standards for Quality and Fairness and works with programs to remedy any issues that arise related to fairness. The Office of Disability Policy collaborates with Disability Services to help test takers with disabilities receive reasonable and fair accommodations.”

Technology has certainly significantly changed testing in the last 20 years. “When we started out most of our tests were paper and pencil and we were very much at the peak of that technology,” recalls Zieky. “We are now at the beginning of a new era (with computerized testing) and we’re continuously learning how to incorporate fairness principles into new testing modalities.”

Rebecca Zwick, a member of the Foundational Psychometrics and Statistical Research group at ETS, has done a lot of research and writing on fairness. She thinks ETS is “very concerned with fairness issues” but also acknowledges that the public sometimes views fairness differently, emphasizing the need for a constant two-way dialogue.

ETS is committed to that effort going forward, says Durso. “We have to remain vigilant in thinking about fairness in what we do because things happen in the world that could affect people and their performance.”