In 2014, a team of engineers at Amazon began working on a project to automate hiring at their company. Their task was to build an algorithm that could review resumes and determine which applicants Amazon should bring on board. But, according to a report this week, the project was canned just a year later, when it became clear that the tool systematically discriminated against women applying for technical jobs, such as software engineer positions.
It shouldn’t surprise us at all that the tool developed this kind of bias. The existing pool of Amazon software engineers is overwhelmingly male, and the new software was fed data about those engineers’ resumes. If you simply ask software to discover other resumes that look like the resumes in a “training” data set, reproducing the demographics of the existing workforce is virtually guaranteed.
In the case of the Amazon project, there were a few ways this happened. For example, the tool disadvantaged candidates who went to certain women’s colleges presumably not attended by many existing Amazon engineers. It similarly downgraded resumes that included the word “women’s” — as in “women’s rugby team.” And it privileged resumes with the kinds of verbs that men tend to use, like “executed” and “captured.”
Fortunately, Amazon stopped using the software program when it became clear the problem wasn’t going to go away despite programmers’ efforts to fix it. But recruiting tools that are likely similarly flawed are being used by hundreds of companies large and small, and their use is spreading.
HOW ARTIFICIAL INTELLIGENCE IS CHANGING THE WORKPLACE
There are many different models out there. Some machine learning programs — which learn how to complete a task based on the data they’re fed — scan resume text, while others analyze video interviews or performance on a game of some kind. Regardless, all such tools used for hiring measure success by looking for candidates who are in some way like a group of people (usually, current employees) designated as qualified or desirable by a human. As a result, these tools are not eliminating human bias — they are merely laundering it through software.
And it’s not just gender discrimination we should be concerned about. Think about all the ways in which looking at resume features might similarly cluster candidates by race: zip code, membership in a Black student union or a Latino professional association, or languages spoken. With video analysis, patterns of speech and eye contact have cultural components that can similarly lead to the exclusion of people from particular ethnic or racial groups. The same goes for certain physical or psychological disabilities.
We’ve seen these types of problems with artificial intelligence in many other contexts. For example, when we used Amazon’s facial recognition tool to compare members of Congress against a database of mugshots, we got 28 incorrect matches — and the rate for false matches was higher for members of color. This is due, in part, to the fact that the mugshot database itself had a disproportionately high number of people of color because of racial biases in the criminal justice system.
These tools are not eliminating human bias — they are merely laundering it through software.
Algorithms that disproportionately weed out job candidates of a particular gender, race, or religion are illegal under Title VII, the federal law prohibiting discrimination in employment. And that’s true regardless of whether employers or toolmakers intended to discriminate — “disparate impact discrimination” is enough to make such practices illegal.
But it can be difficult to sue over disparate impact, particularly in “failure-to-hire” cases. Such lawsuits are very rare because it’s so hard for someone who never got an interview to identify the policy or practice that led to her rejection.
That’s why transparency around recruiting programs and other algorithms used by both companies and the government is so crucial. Many vendors who market these hiring tools claim that they test for bias and in fact are less biased than humans. But their software is proprietary, and there’s currently no way to verify their claims. In some cases, careful work by outside auditors may be able to uncover bias, but their research is thwarted by various obstacles. We’re challenging one such obstacle — a federal law that can criminalize testing of employment websites for discrimination.
But even this kind of outside research can’t give us the full picture. We need regulators to examine not only the software itself but also applicant pools and hiring outcomes for companies that deploy the software. The Equal Employment Opportunity Commission, the federal agency that enforces laws against job discrimination, has begun to the implications of algorithms for fair employment, and we urge the agency to do more. EEOC should issue guidance for employers considering using these tools, detailing their potential liability for biased outcomes and steps they can take to test for and prevent bias. It should also include questions about data-driven bias in all of its investigations.
Big-data algorithms will replicate and even magnify the biases that exist in society at large — unless they are designed and monitored very, very carefully. The right kind of oversight is required to make sure that happens.
Learn More About the Issues on This Page
-
Press ReleaseOct 2024
LGBTQ Rights
+2 Issues
At Supreme Court, Tennessee Seeks to Expand Reach of Dobbs to Ban Health Care for Transgender YouthAt Supreme Court, Tennessee Seeks to Expand Reach of Dobbs to Ban Health Care for Transgender Youth
WASHINGTON – In their brief defending the state’s ban on gender-affirming medical care for transgender youth, Tennessee Attorney General Jonathan Skrmetti has asked the Supreme Court to expand its ruling overturning Roe v. Wade and allow the state to target transgender people’s autonomy over their own bodies, too. In U.S. v. Skrmetti, three Tennessee transgender youth and their families are challenging a state law that prohibits medical providers from prescribing medical treatments to transgender youth, such as puberty blockers and hormone replacement therapies, that are allowed for minors who are not transgender. Represented by the American Civil Liberties Union, the ACLU of Tennessee, Lambda Legal, and Akin Gump Strauss Hauer & Feld LLP, the families argue that the ban violates their Equal Protection rights under the 14th Amendment. In the brief filed today, Tennessee relies on the Supreme Court’s 2022 ruling in Dobbs v. Jackson Whole Women’s Health in an attempt to justify its ban on gender-affirming health care for transgender people. The brief cites to Dobbs at least 10 times in sweeping arguments to justify government sex discrimination. “We’ve seen just how far extreme politicians will push to deny us our reproductive freedom, from banning abortion to threatening IVF to even threatening to put doctors in jail for providing emergency care, with deadly consequences for women’s lives,” said Jennifer Dalven, director of the ACLU’s Reproductive Freedom Project. “The same politicians who are trying to control women have now set their sights on transgender people and their families and are trying to control their bodies and lives. Allowing politicians to continue down this road could hold severe implications for the freedom of all people to decide what is right for their own body.” Tennessee claims that their ban on gender-affirming care does not discriminate on the basis of sex even though it bans minors of one sex from accessing health care it allows to members of another sex. A transgender girl is barred from taking doctor-prescribed estrogen because Tennessee considers it “inconsistent” with her birth-assigned sex of male but a cisgender girl with a birth-assigned sex of female is permitted to take estrogen for any purpose including to affirm her female gender identity. “Since this Project’s founding by Justice Ruth Bader Ginsburg, we have adamantly challenged efforts to limit who we can be based on our gender or ability to bear children,” said Ria Tabacco Mar, director of the ACLU’s Women’s Rights Project. “Tennessee’s attempt to limit who young people can become based on their sex shares a through-line with our nation’s history of subjugating women in the name of biology. The fight for each of us to live fully and authentically must include trans people. There is no ‘transgender’ exception to the U.S. Constitution.” In April 2023, the American Civil Liberties Union, the ACLU of Tennessee, Lambda Legal, and Akin Gump Strauss Hauer & Feld LLP sued the state of Tennessee to block S.B. 1, which prohibits medical providers from prescribing medical treatments to transgender youth, such as puberty blockers and hormone replacement therapies, that it exempts for minors who are not transgender. Following a decision by the Sixth Circuit Court of Appeals, S.B. 1 took effect in July 2023. Since 2021, 24 states have banned hormone therapies for transgender youth. “Laws like Tennessee’s are not benign regulations of medical care; they are discriminatory efforts to exclude transgender people from the protections of the Constitution,” said Chase Strangio, co-director of the ACLU’s LGBTQ & HIV Project. “These bans represent a dangerous and discriminatory affront to the well-being of transgender youth across the country and their constitutional right to equal protection under the law.” -
Press ReleaseSep 2024
Women's Rights
Civil Rights Groups Tell Federal Appeals Court that Protections for Pregnant Workers Cover Abortion CareCivil Rights Groups Tell Federal Appeals Court that Protections for Pregnant Workers Cover Abortion Care
WASHINGTON, D.C. - On Friday, a broad coalition of labor, workers’ rights, and gender justice organizations — led by the American Civil Liberties Union (ACLU), the ACLU of Arkansas, and the National Women’s Law Center (NWLC) — filed an amicus brief in the U.S. Court of Appeals for the Eighth Circuit in support of the Equal Employment Opportunity Commission’s (EEOC) regulations implementing the landmark Pregnant Workers Fairness Act (PWFA), including the regulations’ explicit protection for workers who have abortions. The PWFA, which took effect last year, was the culmination of a decade-long campaign to secure access to reasonable accommodations for workers with temporary limitations caused by “pregnancy, childbirth, and related medical conditions.” The EEOC’s regulations provide comprehensive guidance to workers, employers, and the courts about the statute’s range of protections, including job-protected time off for medical treatment and recovery. The amicus brief filed in the U.S. Court of Appeals for the Eighth Circuit supports the EEOC in a challenge by 17 red states that are trying to block the PWFA regulations’ coverage of accommodations for abortion-related care. In June, the U.S. District Court for the Eastern District of Arkansas dismissed the states’ challenge, and the states appealed to the Eighth Circuit. Their appeal includes a request that the Eighth Circuit issue a preliminary injunction of the regulations’ abortion provisions. The ACLU-NWLC brief details the federal law’s longstanding ban on discrimination against workers who obtain abortions – protection the PWFA intended to continue. The brief also documents the accounts of workers whose health and jobs were put at risk when their employers denied them accommodations, including time off, for abortion care. Such accounts illustrate the urgent need for clarity with respect to the PWFA’s coverage of abortion-related accommodations, and the devastating medical and financial consequences if such accommodations are denied. “Abortion has long been recognized as part of the full spectrum of workers’ pregnancy-related needs that are protected by the law, ” said Gillian Thomas, senior staff attorney at the ACLU’s Women's Rights Project. “The states’ challenge has created uncertainty, and seeks to deprive workers of essential, job-protected time off for abortion care, posing significant health risks and leading to forced pregnancies. The EEOC regulations are vital to ensure pregnant workers don't have to choose between their health and their jobs.” “Extremist politicians are demonstrating repeatedly their willingness to risk women’s health to advance their anti-abortion agenda,” said Gaylynn Burroughs, vice president for Education and Workplace Justice at the National Women’s Law Center (NWLC). “Let’s be clear: abortion care is pregnancy-related care, and stripping essential protections for abortion from the PWFA would endanger the health and economic security of pregnant workers.” “The Pregnant Workers Fairness Act and its regulations provide a lifeline for workers who need accommodations during the most critical periods of their lives. Attempting to strip away these protections, particularly for abortion-related needs, is an affront to the rights and well-being of pregnant workers,” said John Williams, legal director of the ACLU of Arkansas. “All people in all aspects of pregnancy, including abortion, should be treated with the care and consideration they deserve. We stand firmly with the EEOC in defending these essential protections.” A copy of the brief can be found here. -
PodcastSep 2024
Women's Rights
How Sue Bird Is Supercharging Women's SportsHow Sue Bird Is Supercharging Women's Sports
-
Press ReleaseJul 2024
Women's Rights
LGBTQ Rights
ACLU Condemns House Measure Against Department of Education Title IX RuleACLU Condemns House Measure Against Department of Education Title IX Rule
WASHINGTON – Today, the U.S. House of Representatives voted to pass H.J. Res. 165, a Congressional Review Act resolution disapproving of the U.S. Department of Education’s final Title IX rule providing guidance to schools on protecting students from sexual harassment, pregnancy discrimination, anti-LGBTQ discrimination, and other forms of sex discrimination. Mike Zamore, ACLU National Policy and Government Affairs Director, had the following response: “Today’s vote to completely repeal the Title IX rule takes a sledgehammer to the government’s ability to ensure that all students have the opportunity to learn free from sex discrimination and harassment. Stripping critical civil rights protections from LGBTQ students and students who are pregnant and parenting - as this resolution would do - is something that all members of Congress should strongly oppose. This is also not how Congress should act to protect the rights of students who face disciplinary action under Title IX because passing this resolution would prevent future rulemaking to ensure fair processes in school disciplinary proceedings.” The ACLU’s letter urging a “No” vote on H.J. Res. 165 can be found here: /documents/aclu-urges-no-vote-on-h-j-res-165 The ACLU’s statement on the final Title IX rule can be found here: /press-releases/155090