WASHINGTON – A Senate committee that oversees the Pentagon has adopted a measure from Arizona Sen. Mark Kelly intended to require “ultimate human responsibility” in the use of force.
The U.S. and other nations are increasingly deploying autonomous weapons systems, and the use of artificial intelligence has escalated rapidly in the U.S. war with Iran and the Russia-Ukraine war.
“Congress is responding to a public and allied concern that the United States could find itself legally and ethically exposed if autonomous systems cause civilian casualties without any clear human decision-maker in the chain,” Zaza Tsotniashvili, a professor at Caucasus International University in Tbilisi, Georgia, said by email.
Kelly, a Democrat who served as a Navy combat pilot, added his provision to the Senate version of the National Defense Authorization Act. It codifies a 2023 Defense Department policy called Directive 3000.09, which requires human oversight in the use of autonomous weapons – military systems that operate without human intervention.
On June 5, President Donald Trump issued a presidential memorandum directing the Pentagon to “eliminate unnecessary barriers to rapid deployment” of AI and to provide an updated version of Directive 3000.09 within 90 days. Kelly’s amendment would effectively override that, if it survives the legislative process, by blocking the elimination of human involvement in the “kill chain.”
The Senate Armed Services Committee approved the amendment Thursday and sent the bill to the full Senate on an 18-9 vote.
Kelly ultimately voted against the NDAA despite the inclusion of the AI provision and other amendments he supported. He cited the $1.15 trillion price tag and lack of transparency, as shown by Trump’s defiance of Congress in going to war with Iran.
“We cannot write them another blank check,” Kelly said last week in a statement explaining his opposition to the bill.
AI use by the U.S. armed forces has been in the spotlight in recent months.
In January, Anthropic’s AI tool, Claude, was used in the capture of Venezuelan President Nicolás Maduro.
Claude was also used on the first day of Operation Epic Fury in Iran, with speculation that it was involved in a missile attack on a school near a military base that killed 175 people, mostly young girls.
Following that strike, technology scholar Kevin Baker argued that the AI program used to identify targets – known as Maven Smart System – and the officials behind it bore much more responsibility than Claude. Baker said old information led Maven to target the school, which was part of an Islamic Revolutionary Guard Corps compound until 2016 at the latest.
“People failed to update a database, and other people built a system fast enough to make that failure lethal,” Baker wrote in The Guardian.
Maven, engineered by Palantir Technologies in 2017, was built to analyze intelligence data and evaluate what targets to strike. The New Yorker reported that it took a person four clicks to go from target identification to target destruction, making Maven a multi-year project in using AI to compress the kill chain.
Kelly’s amendment also comes while Trump works to speed up the military’s adoption of AI.
The effort to put legal guardrails on AI when it comes to lethal use of force earned praise from some experts.
Paul Lushenko, a non-resident expert at the nonprofit organization RegulatingAI and lecturer at George Washington University, called it a “legal, moral, ethical imperative.”
“There’s a perception shared among the military and the public that there ought to be human oversight of these capabilities,” he said.
Tools like Maven have been strong assets for ensuring national defense, Lushenko said, but guidelines are imperative for how they are used and who is held accountable for misuse.
Such guidelines will have to come from civilian overseers, internal military culture and technological guardrails.
Tsotniashvili said the Defense Department’s policies aren’t stable enough for Congress to remain silent.
“DoD directives can be reinterpreted, waived or quietly revised by the executive branch without congressional approval,” he said.
Trump’s memorandum said the deployment of AI technologies must come while “maintaining rigorous oversight.” It does not specify who would conduct such oversight or what that would look like.
“Ambiguity serves the interest of those deploying these systems,” Tsotniashvili said.
Sen. Ruben Gallego sent a letter Monday to Secretary of Defense Pete Hegseth demanding clarification on how the updated directive would ensure the U.S. and its servicemembers are not harmed by autonomous weapons.
The Arizona Democrat asserted that Trump’s update to Directive 3000.09 will increase the risk of harm to civilians.
Defining “human oversight” is a challenge, Lushenko said. The concepts of meaningful human control and appropriate human judgment are open to interpretation, and that’s complicated by an argument that checks can be incorporated into the design of autonomous weapons.
But Jovana Davidovic, a professor at the University of Iowa who studies military ethics, said developers have a responsibility for the actions of any AI system they create.
“The more autonomous the system is, the more the decisions that developers make ultimately drive outcomes,” she said by email. Tsotniashvili said codifying Directive 3000.09 is necessary but insufficient, and more specifics are needed to ensure human oversight is “meaningful rather than ceremonial.”
“The 2027 NDAA provision is a step forward,” he said. “But without accompanying accountability mechanisms, it may succeed in preserving the appearance of human control while the substance of it continues to erode.”
Republish our articles for free, online or in print, under a Creative Commons license.
