By: Kelsey Atherton
Bob Work, in his last months as deputy secretary of defense, wanted everything in place so that the Pentagon could share in the sweeping advances in data processing already enjoyed by the thriving tech sector.
A memo dated April 26, 2017, established an “Algorithmic Warfare Cross-Functional Team,” a.k.a. “Project Maven.” Within a year, the details of Google's role in that program, disseminated internally among its employees and then shared with the public, would call into question the specific rationale of the task and the greater question of how the tech community should go about building algorithms for war, if at all.
Project Maven, as envisioned, was about building a tool that could process drone footage quickly and in a useful way. Work specifically tied this task to the Defeat-ISIS campaign. Drones are intelligence, surveillance and reconnaissance platforms first and foremost. The unblinking eyes of Reapers, Global Hawks and Gray Eagles record hours and hours of footage every mission, imagery that takes a long time for human analysts to scan for salient details. While human analysts process footage, the ground situation is likely changing, so even the most labor-intensive approach to analyzing drone video delivers delayed results.
In July 2017, Marine Corps Col. Drew Cukor, the chief of the Algorithmic Warfare Cross-Function Team, presented on artificial intelligence and Project Maven at a defense conference. Cukor noted, “AI will not be selecting a target [in combat] ... any time soon. What AI will do is complement the human operator.”
As Cukor outlined, the algorithm would allow human analysts to process two or three times as much data within the same timeframe. To get there, though, the algorithm to detect weapons and other objects has to be built and trained. This training is at the heart of neural networks and deep learning, where the computer program can see an unfamiliar object and classify it based on its resemblance to other, more familiar objects. Cukor said that before deploying to battle “you've got to have your data ready and you've got to prepare and you need the computational infrastructure for training.”
At the time, the contractor who would develop the training and image-processing algorithms for Project Maven was unknown, though Cukor did specifically remark on how impressive Google was as an AI company. Google's role in developing Maven would not come to light until March 2018, when Gizmodo reported that Google is helping the Pentagon build AI for drones. Google's role in the project was discussed internally in the company, and elements of that discussion were shared with reporters.
“Some Google employees were outraged that the company would offer resources to the military for surveillance technology involved in drone operations,” wrote Kate Conger and Dell Cameron, “while others argued that the project raised important ethical questions about the development and use of machine learning.”
A petition by the Tech Workers Coalition that circulated in mid-April called upon not just Google to pull out of Pentagon contracts, but for Amazon, Microsoft and IBM to refuse to pick up the work of Project Maven. (The petition attracted 300 signatures at the time of this story.)
Silicon Valley's discord over the project surprised many in positions of leadership within the Pentagon. During the 17th annual C4ISRNET Conference, Justin Poole, the deputy director of the National Geospatial-Intelligence Agency, was asked how the intelligence community can respond to skepticism in the tech world. Poole's answer was to highlight the role of intelligence services in reducing risk to war fighters.
Disagreement between some of the people working for Google and the desire of the company's leadership to continue pursuing Pentagon contracts exacerbated tension in the company throughout spring. By May, nearly a dozen Google employees had resigned from the company over its involvement with Maven, and an internal petition asking the company to cancel the contract and avoid future military projects garnered thousands of employee signatures. To calm tensions, Google would need to find a way to reconcile the values of its employees with the desire of its leadership to develop further AI projects for a growing range of clients.
That list of clients, of course, includes the federal government and the Department of Defense.
While efforts to convince the tech community at large to refuse Pentagon work have stalled, the pressure within Google resulted in multiple tangible changes. First, Google leadership announced the company's plan to not renew the Project Maven contract when it expired in 2019. Then, the company's leaders released principles for AI, saying it would not develop intelligence for weapons or surveillance applications.
After outlining how Google intends to build AI in the future, with efforts to mitigate bias, aid safety and be accountable, Google CEO Sundar Pichai set out categories of AI work that the company will not pursue. This means refusing to design or deploy “technologies that cause or are likely to cause overall harm,” including an explicit prohibition on weapons principally designed to harm people, as well as surveillance tech that violates international norms.
Taken together, these principles amount to a hard-no only on developing AI specifically intended for weapons. The rest are softer no's, objections that can change with interpretations of international law, norms, and even in how a problem set is described.
After all, when Poole was asked how to sell collaboration with the intelligence community to technology companies, he framed the task as one about saving the lives of war fighters.
The “how” of that lifesaving is ambiguous: It could equally mean better and faster intelligence analysis that gives a unit on patrol the information it needs to avoid an ambush, or it could be the advance info that facilitates an attack on an adversary's encampment when the guard shift is particularly understaffed. Image processing with AI is so ambiguous a technology, so inherently open to dual-use, that the former almost certainly isn't a violation of Google's second objection to AI use, but the latter example absolutely would be.
In other words, the long-term surveillance that goes into targeted killing operations above Afghanistan and elsewhere is likely out of bounds. However, the same technology used over Iraq for the fight against ISIS might be permissible. And software built to process drone footage in the latter context would be identical to the software built to process images for the former.
The lines between what this does and doesn't prevent becomes even murkier when one takes into account that Google built its software for Project Maven on top of TensorFlow, an open-source software library. This makes it much harder to build in proprietary constraints on the code, and it means that once the Pentagon has a trainable algorithm on hand, it can continue to develop and refine its object-recognition AI as it chooses.
But the window for Google to be involved in such a project, whether to the joy or dismay of its employees and executive leadership, is likely closing.
In late June, the Pentagon announced creation of a Joint Artificial Intelligence Center, which among other functions would take over Project Maven from the Algorithmic Warfare Cross-Functional Team. The defense sector is vast, and with Google proving to be a complicated contractor for the Pentagon, new leadership may simply take its AI contracts worth million elsewhere with to see if it can get the programming it needs. And Maven itself still receives accolades within the Pentagon.
Gen. Mike Holmes, commander of Air Combat Command, praised Project Maven at a June 28 defense writers group breakfast, saying that the use of learning machines and algorithms will speed up the process by which humans process information and pass on useful insights to decisions makers.
Inasmuch as the Pentagon has a consensus view of explaining tools like Maven, it is about focusing on the role of the human in the process. The software will do the first pass through the imagery collected, and then as designed highlight other details for a human to review and act upon. Holmes was adamant that fears of malicious AIs hunting humans, like Skynet from the “Terminator” movies, are beyond premature.
“We're going to have to work through as Americans our comfort level on how technologies are used and how they're applied,” said Holmes. “I'd make the case that our job is to compete with these world-class peer competitors that we have, and by competing and by setting this competition on terms that we can compete without going to conflict, it's better for everybody.”
AI of the tiger
Project Maven, from the start, is a program specifically sold and built for the work of fighting a violent nonstate actor, identifying the weapons and tools of an insurgency that sometimes holds swaths of territory.
“Our responsibility is to help people understand what the intent is with the capability that we are helping to develop. ... Maven is focused on minimizing collateral damage on the battlefield. There's goodness in that,” said Capt. Sean Heritage, acting managing partner of Defense Innovation Unit Experimental (DIUx).
“There's always risk in how it will be used down the road, and I guess that's where a small pocket of people at Google's heads were. But, as Mr. Work pointed out during his panel at Defense One, they don't seem to have as challenging of a time contributing to AI capability development in China.”
Google's fight over Project Maven is partly about the present — the state of AI, the role of the United States in pursuing insurgencies abroad. It is also a fight about how the next AI will be built, and who that AI will be built to be used against. And the Pentagon seems to understand this, too. In the same meeting where Holmes advocated for Maven as a useful tool for now, he argued that it was important for the United States to develop and field tools that can match peer or near-peer rivals in a major conflict.
That's a far cry from selling the tool to Silicon Valley as one of immediate concern, to protect the people fighting America's wars presently through providing superior real-time information.
“The idea of a technology being built and then used for war, even if that wasn't the original intent,” says author Malka Older, “is what science fiction writers call a ‘classic trope.' ”
Older's novels, set two or three generations in the near-future, focus on the ways in which people, governments and corporations handle massive flows of data, and provide one possible vision of a future where the same kinds and volumes of data are collected, but where that data is also held by a government entity and shared transparently.
While radical transparency in data is alien to much of the defense establishment, it's an essential part of the open-source technology community for security concerns both genuine and sometimes not-so genuine. Building open source means publishing code and letting outsiders find flaws and vulnerabilities in the algorithm, without looking at any of the sensitive data the algorithm is built to process.
And Project Maven is built on top of open-source framework.
“One of the dangerous concepts that we have of technology is that progress only goes in one direction,” says Older.
“There's constantly choices being made of where technology goes and where concepts go and what we are trying to do.”
While it's entirely possible that the Pentagon will be able to continue the work of Project Maven and other AI programs with new contractors, if it wanted to reach out to those skeptical of how the algorithm would interpret images, it could try justifying the mission not just with national security concerns, but with transparency.
“Part of being an American is that Americans have expectations about what their government does and whether the government uses tech and tools to infringe upon their rights or not,” said Holmes. “And, so, we have really high standards as a nation that the things that we bring forward as military tools have to live up to.”
To work with the coders of the future, it may not be enough to say that the code — open source or not — is going to be used in ways consistent with their values. The Pentagon may have to find ways to transparently prove it.
https://www.c4isrnet.com/it-networks/2018/07/27/targeting-the-future-of-the-dods-controversial-project-maven-initiative/
