When I joined the artificial intelligence company Clarifai in early 2017, you could practically taste the promise in the air. My colleagues were brilliant, dedicated, and committed to making the world a better place.
We founded Clarifai 4 Good where we helped students and charities, and we donated our software to researchers around the world whose projects had a socially beneficial goal. We were determined to be the one AI company that took our social responsibility seriously.
I never could have predicted that two years later, I would have to quit this job on moral grounds. And I certainly never thought it would happen over building weapons that escalate and shift the paradigm of war.
Some background: In 2014, Stephen Hawking and Elon Musk led an effort with thousands of AI to collectively pledge never to contribute research to the development of lethal autonomous weapons systems — weapons that could seek out a target and end a life without a human in the decision-making loop. The researchers argued that the technology to create such weapons was just around the corner and would be disastrous for humanity.
I signed that pledge, and in January of this year, I wrote a letter to the CEO of Clarifai, Matt Zeiler, asking him to sign it, too. I was not at all expecting his response. As The New York Times on Friday, Matt called a companywide meeting and announced that he was totally willing to sell autonomous weapons technology to our government.
I could not abide being part of that, so I quit. My objections were not based just on a fear that killer robots are “just around the corner.” They already exist and are used in combat today.
Now, don’t go running for the underground bunker just yet. We’re not talking about something like the Terminator or Hal from “2001: A Space Odyssey.” A scenario like that would require something like artificial general intelligence, or AGI — basically, a sentient being.
In my opinion, that won’t happen in my lifetime. On the other end of the spectrum, there are some people who describe things like landmines or homing missiles as “semiautonomous.” That’s not what I’m talking about either.
The core issue is whether a robot should be able to select and acquire its own target from a list of potential ones and attack that target without a human approving each kill. One example of a fully autonomous weapon that’s in use today is the Israeli Harpy 2 drone (or Harop), which seeks out enemy radar signals on its own. If it finds a signal, the drone goes into a kamikaze dive and blows up its target.
Fortunately, there are only a few of these kinds of machines in operation today, and as of right now, they usually operate with a human as the one who decides whether to pull the trigger. Those supporting the creation of autonomous weapons systems would prefer that human to be not “in the loop but “on the loop” — supervising the quick work of the robot in selecting and destroying targets, but not having to approve every last kill.
When presented with the Harop, a lot of people look at it and say, “It’s scary, but it’s not genuinely freaking me out.” But imagine a drone acquiring a target with a technology like face recognition. Imagine this: You’re walking down the street when a drone pops into your field of vision, scans your face, and makes a decision about whether you get to live or die.
Suddenly, the question — “Where in the decision loop does the human belong?” — becomes a deadly serious one.
What the generals are thinking
On the battlefield, human-controlled drones already play a critical role in surveillance and target location. If you add machine learning to the mix, you’re looking at a system that can sift through exponentially increasing numbers of potential threats over a vast area.
But there are vast technical challenges with streaming high definition video halfway around the world. Say you’re a remote drone pilot and you’ve just found a terrorist about to do something bad. You’re authorized to stop them, and all of a sudden, you lose your video feed. Even if it’s just for a few seconds, by the time the drone recovers, it might be too late.
What if you can’t stream at all? Signal jamming is pretty common in warfare today. Your person in the loop is now completely useless.
That’s where generals get to thinking: Wouldn’t it be great if you didn’t have to depend on a video link at all? What if you could program your drone to be self-contained? Give it clear instructions, and just press Go.
That’s the argument for autonomous weapons: Machine learning will make war more efficient. Plus there's the fact that Russia and China are already working on this technology, so we might as well do the same.
Sounds reasonable, right?
Okay, here are six reasons why killer robots are genuinely terrifying
There are a number of reasons why we shouldn’t accept these arguments:
1. Accidents. Predictive technologies like face recognition or object localization are guaranteed to have error rates, meaning a case of mistaken identity can be deadly. Often these technologies fail disproportionately on people with darker skin tones or certain facial features, meaning their lives would be doubly subject to this threat.
Also, drones go rogue sometimes. It doesn’t happen often, but software always has bugs. Imagine a self-contained, solar-powered drone that has instructions to find a certain individual whose face is programmed into its memory. Now imagine it rejecting your command to shut it down.
2. Hacking. If your killer robot has a way to receive commands at all (for example, by executing a “kill switch” to turn it off), it is vulnerable to hacking. That means a powerful swarm of drone weapons could be turned off — or turned against us.
3. The “black box” problem. AI has an “explainability” problem. Your algorithm did XYZ, and everyone wants to know why, but because of the way that machine learning works, even its programmers often can’t know why an algorithm reached the outcome that it did. It’s a black box. Now, when you enter the realm of autonomous weapons, and ask, “Why did you kill that person,” the complete lack of an answer simply will not do — morally, legally, or practically.
4. Morality & Context. A robot doesn’t have moral context to prioritize one kind of life over another. A robot will only see that you’re carrying a weapon and “know” that its mission is to shoot with deadly force. It should not be news that terrorists often exploit locals and innocents. In such scenarios, a soldier will be able to use their human, moral judgment in deciding how to react — and can be held accountable for those decisions. The best object localization software today is able to look at a video and say, “I found a person.” That’s all. It can’t tell whether that person was somehow coerced into doing work for the enemy.
5. War at Machine Speed. How long does it take you to multiply 35 by 12? A machine can do thousands of such calculations in the time it takes us to blink. If a machine is programmed to make quick decisions about how and when to fire a weapon, it’s going to do it in ways we humans can’t even anticipate. Early experiments with swarm technology have shown that no matter how you structure the inter-drone communications, the outcomes are different every time. The humans simply press the button, watch the fight, and wait for it to be over so that they can try to understand the what, when, and why of it.
Add 3D printing to the mix, and now it’s cheap and easy to create an army of millions of tiny (but lethal) robots, each one thousands of times faster than a human being. Such a swarm could overwhelm a city in minutes. There will be no way for a human to defend themselves against an enemy of that scale or speed — or even understand what’s happening.
6. Escalation. Autonomous drones would further distance the trigger-pullers from the violence itself and generally make killing more cost-free for governments. If you don’t have to put your soldiers in harm’s way, it becomes that much easier to decide to take lives. This distance also puts up a psychological barrier between the humans dispatching the drones and their targets.
Humans actually find it very difficult to kill, even in military combat. In his book “Men Against Fire,” S.L.A. Marshall reports that over 70 percent of bullets fired in WWII were not aimed with the intent to kill. Think about firing squads. Why would you have seven people line up to shoot a single person? It’s to protect the shooters’ psychological safety, of course. No one will know whose bullet it truly was that did the deed.
If you turn your robot on and it decides to kill a child, was it really you who destroyed that life?
There is still time to ask our government to agree to ban this technology outright
In the end, there are many companies out there working to “democratize” powerful technology like face recognition and object localization. But these technologies are “dual-use,” meaning they can be used not only for everyday civilian purposes but also for targeting people with killer drones.
Project Maven, for instance, is a Defense Department contract that’s currently being worked on at Microsoft and Amazon (as well as in startups like Clarifai). Google employees were successful in persuading the company to from the contract because they feared it would be used to this end. Project Maven might just be about “counting things” as the Pentagon claims. It might also be a targeting system for autonomous killer drones, and there is absolutely no way for a tech worker to tell.
With so many tech companies participating in work that contributes to the reality of killer robots in our future, it’s important to remember that major powers won’t be the only ones to have autonomous drones. Even if killer robots won’t be the Gatling gun of World War III, they will do a lot of damage to populations living in countries all around the world.
We must remind our government that humanity has been successful in instituting international norms that condemn the use of chemical and biological weapons. When the stakes are this high and the people of the world object, there are steps that governments can take to prevent mass killings. We can do the same thing with autonomous weapons systems, but the time to act is now.
Official Defense Department states currently that there must be a “human in the loop” for every kill decision, but that is under debate right now, and there’s a loophole in the policy that would allow for an autonomous weapon to be approved. We must work together to ensure this loophole is closed.
That’s why I refuse to work for any company that participates in Project Maven, or who otherwise contributes dual-use research to the Pentagon. Policymakers need to promise us that they will stop the pursuit of autonomous lethal weapons systems once and for all.
I don’t regret my time at Clarifai. I do miss everyone I left behind. I truly hope that the industry changes course and agrees to take responsibility for its work to ensure that the things we build in the private sector won’t be used for killing people. More importantly, I hope our government begins working internationally to ensure that autonomous weapons are banned to the same degree as biological ones.
Learn More About the Issues on This Page
-
Press ReleaseDec 2024
Free Speech
National Security
ACLU and Partners Urge Supreme Court to Block TikTok BanACLU and Partners Urge Supreme Court to Block TikTok Ban
WASHINGTON — Today, the American Civil Liberties Union, the Electronic Frontier Foundation (EFF), and the Knight First Amendment Institute at Columbia University filed an amicus brief urging the Supreme Court to block the enforcement of a law that would effectively ban people in America from using TikTok as soon as January 19, 2025. Earlier this month, the U.S. Court of Appeals for the D.C. Circuit rejected TikTok’s challenge to the law. TikTok has asked the Supreme Court for a temporary injunction to prevent the app from being banned while the court considers whether to take the case, saying that unless the justices intervene, the law will “shutter one of America’s most popular speech platforms the day before a presidential inauguration.” “The Constitution imposes an extraordinarily high bar on this kind of mass censorship,”said Patrick Toomey, deputy director of ACLU’s National Security Project. “The Supreme Court should take up this important case and protect the rights of millions of Americans to freely express themselves and engage with others around the world.” The brief argues that the D.C. Circuit failed to fully address the law’s profound implications for the First Amendment rights of the 170 million Americans who use TikTok. While the lower court’s decision correctly recognized that the statute triggers First Amendment scrutiny, it barely addressed users’ First Amendment interests in speaking, sharing, and receiving information on the platform. The court also perplexingly attempted to cast the government’s ban on TikTok as a vindication of users’ First Amendment rights, which it is not. The rights groups also explain that the law was intended to suppress certain content and viewpoints that many legislators believe could be amplified on TikTok, including the risk of foreign “propaganda.” But under the First Amendment, the government must meet a very high bar to restrict speech based on concerns about its “motivating ideology” or “perspective,” and the government has not come close to meeting that bar here. “The government should not be able to restrict speech, especially to the extent here, based on guessing about the mere possibility of uncertain future harm,” said David Greene, civil liberties director at EFF. “The Supreme Court should put the TikTok ban on hold while it considers the DC Circuit’s erroneous ruling.” Finally, the brief underscores that the government can’t impose this type of sweeping ban unless it’s necessary to prevent extremely serious and imminent harm to national security. But the government has not provided evidence of impending harm, or evidence that banning TikTok is the only available way to address its concerns. As the brief explains, the D.C. Circuit improperly treated the government’s invocation of “national security” as a trump card and failed to hold the government to its burden. “Restricting citizens’ access to foreign media is a practice that has long been associated with repressive regimes, and we should be very wary of letting the practice take root here,” said Jameel Jaffer, executive director at the Knight First Amendment Institute at Columbia University. “It would do lasting damage to the First Amendment and our democracy if the Supreme Court let this ban go into effect even temporarily.” You can find the brief online here: https://www.supremecourt.gov/DocketPDF/24/24A587/335380/20241217144322392_24A587%20TikTok%20v%20Garland%20Amicus%20Brief%20pdfa.pdf -
Press ReleaseDec 2024
National Security
Religious Liberty
ACLU Statement on New White House Strategy to Counter IslamophobiaACLU Statement on New White House Strategy to Counter Islamophobia
WASHINGTON – The Biden administration today released the first-ever national strategy to counter Islamophobia and related forms of bias and discrimination, including hate against Arab, Sikh, and South Asian Americans. In advance of the strategy, American Civil Liberties Union and its partners had urged the administration to overhaul government programs that reflect anti-Muslim discrimination. In particular, we have called for urgent action to constrain governmental agencies from continuing to exercise their authorities and technology to wrongly surveil and investigate, watchlist, and question and detain Muslims at the border, as well as deny immigration benefits to people from Muslim-majority countries. While the White House raised expectations that many of these issues would be addressed, the final strategy ended up falling far short. The following is a statement from Hina Shamsi, director of the ACLU’s National Security Project: “While this strategy acknowledges discrimination and its harms, it does little to end them and is a squandered opportunity. For decades, American officials have invoked national security to pass laws and implement programs that disproportionately harm Muslims and people perceived to be Muslim. A serious anti-discrimination strategy would concretely address multiple bias-infused government practices that deny our communities equal participation in civic life and our democracy, like federal watchlisting, surveillance, and investigation. We’re profoundly frustrated that the administration didn’t take even the basic, overdue step of recognizing that anti-Muslim discrimination is uniquely normalized and embedded in government policies.” -
VirginiaDec 2024
National Security
Trabelsi v. Crawford, et al. – Lawsuit Challenging Unlawful Detention and Inhumane Treatment of Acquitted ManTrabelsi v. Crawford, et al. – Lawsuit Challenging Unlawful Detention and Inhumane Treatment of Acquitted Man
Our client, Nizar Trabelsi, is in the United States against his will. The federal government brought him here from Belgium more than a decade ago and charged him with terrorism-related crimes. At trial, the government’s case failed: a federal jury found Mr. Trabelsi not guilty. But instead of allowing Mr. Trabelsi to return to Belgium after his acquittal, the United States placed him in highly restrictive immigration detention and began an ongoing effort to force him to Tunisia, where he was born and where he will very likely be tortured. Mr. Trabelsi’s detention violates the Constitution, immigration law, and the extradition treaty between the United States and Belgium. Through this lawsuit, he seeks to return to Belgium, and he demands an immediate improvement of his detention conditions. -
Press ReleaseDec 2024
National Security
Immigrants' Rights
Federal Judge Rejects Acquitted Man’s Argument That His Immigration Detention Is UnlawfulFederal Judge Rejects Acquitted Man’s Argument That His Immigration Detention Is Unlawful
ALEXANDRIA, Va. — This week, the U.S. District Court for the Eastern District of Virginia ruled that the court does not have jurisdiction to decide whether the government can continue to detain Nizar Trabelsi, a Tunisian national who was brought to the United States against his will for prosecution and was cleared of all charges last year. The ruling means the United States will be able to continue to detain Mr. Trabelsi while it attempts to deport him to Tunisia, where he was born and where, as an immigration judge ruled earlier this year, he will likely face torture. The court also concluded that even if it did have jurisdiction over Mr. Trabelsi’s case, his claims challenging the government’s detention of him would fail. “Reviewing claims that the government is holding someone unlawfully is at the core of the judicial function, and we’re heartbroken for our client that the court got this so wrong,” said Brett Max Kaufman, senior staff attorney with the ACLU’s Center for Democracy. “The government’s arguments in defense of Mr. Trabelsi’s detention makes a hash of the Constitution, immigration laws, and its own extradition treaty.” In 2013, the United States forcibly extradited Mr. Trabelsi from Belgium to face criminal charges in the United States. In July 2023, after almost 10 years of highly restrictive pretrial detention, a federal jury cleared Mr. Trabelsi of all charges. But instead of releasing Mr. Trabelsi or returning him to Belgium, the U.S. transferred him to immigration detention, wrongly treating him as an applicant for admission and placing him in the deportation process. Over the years, the Belgian government has issued multiple formal diplomatic requests asking the U.S. to facilitate his return, but the U.S. has refused to send him back. Mr. Trabelsi filed a lawsuit challenging the government's authority to detain him. He also sought immediate improvements to his detention conditions. Those claims were not part of the court’s ruling today, and will continue to be litigated on a separate track. “Mr. Trabelsi only wants to return to Belgium after being illegally extradited to the United States, held for ten years, and then acquitted by a jury of any crimes,” said Nicole Hallett, clinical professor of law and director of the Immigrants’ Rights Clinic at the University of Chicago Law School. “He cannot be held indefinitely and we will continue to fight to make sure that justice ultimately prevails.” Mr. Trabelsi is represented by the American Civil Liberties Union, the University of Chicago Law School’s Immigrants’ Rights Clinic, the ACLU of Virginia, and Professor Jonathan Hafetz of Seton Hall Law School. The complaint in Trabelsi v. Crawford was filed against Jeffrey Crawford, warden of the Farmville Detention Center where Mr. Trabelsi is being held; Liana Castano, ICE field office director for the Washington Field Office; Alejandro Mayorkas, secretary of the Department of Homeland Security; and Merrick Garland, U.S. attorney general.