Back to news

February 7, 2019 | International, C4ISR

DARPA: Defending Against Adversarial Artificial Intelligence

Today, machine learning (ML) is coming into its own, ready to serve mankind in a diverse array of applications – from highly efficient manufacturing, medicine and massive information analysis to self-driving transportation, and beyond. However, if misapplied, misused or subverted, ML holds the potential for great harm – this is the double-edged sword of machine learning.

“Over the last decade, researchers have focused on realizing practical ML capable of accomplishing real-world tasks and making them more efficient,” said Dr. Hava Siegelmann, program manager in DARPA's Information Innovation Office (I2O). “We're already benefitting from that work, and rapidly incorporating ML into a number of enterprises. But, in a very real way, we've rushed ahead, paying little attention to vulnerabilities inherent in ML platforms – particularly in terms of altering, corrupting or deceiving these systems.”

In a commonly cited example, ML used by a self-driving car was tricked by visual alterations to a stop sign. While a human viewing the altered sign would have no difficulty interpreting its meaning, the ML erroneously interpreted the stop sign as a 45 mph speed limit posting. In a real-world attack like this, the self-driving car would accelerate through the stop sign, potentially causing a disastrous outcome. This is just one of many recently discovered attacks applicable to virtually any ML application.

To get ahead of this acute safety challenge, DARPA created the Guaranteeing AI Robustness against Deception (GARD) program. GARD aims to develop a new generation of defenses against adversarial deception attacks on ML models. Current defense efforts were designed to protect against specific, pre-defined adversarial attacks and, remained vulnerable to attacks outside their design parameters when tested. GARD seeks to approach ML defense differently – by developing broad-based defenses that address the numerous possible attacks in a given scenario.

“There is a critical need for ML defense as the technology is increasingly incorporated into some of our most critical infrastructure. The GARD program seeks to prevent the chaos that could ensue in the near future when attack methodologies, now in their infancy, have matured to a more destructive level. We must ensure ML is safe and incapable of being deceived,” stated Siegelmann.

GARD's novel response to adversarial AI will focus on three main objectives: 1) the development of theoretical foundations for defensible ML and a lexicon of new defense mechanisms based on them; 2) the creation and testing of defensible systems in a diverse range of settings; and 3) the construction of a new testbed for characterizing ML defensibility relative to threat scenarios. Through these interdependent program elements, GARD aims to create deception-resistant ML technologies with stringent criteria for evaluating their robustness.

GARD will explore many research directions for potential defenses, including biology. “The kind of broad scenario-based defense we're looking to generate can be seen, for example, in the immune system, which identifies attacks, wins and remembers the attack to create a more effective response during future engagements,” said Siegelmann.

GARD will work on addressing present needs, but is keeping future challenges in mind as well. The program will initially concentrate on state-of-the-art image-based ML, then progress to video, audio and more complex systems – including multi-sensor and multi-modality variations. It will also seek to address ML capable of predictions, decisions and adapting during its lifetime.

A Proposers Day will be held on February 6, 2019, from 9:00 AM to 2:00 PM (EST) at the DARPA Conference Center, located at 675 N. Randolph Street, Arlington, Virginia, 22203 to provide greater detail about the GARD program's technical goals and challenges.

Additional information will be available in the forthcoming Broad Agency Announcement, which will be posted to www.fbo.gov.

https://www.darpa.mil/news-events/2019-02-06

On the same subject

  • Future US Navy weapons will need lots of power. That’s a huge engineering challenge.

    June 26, 2018 | International, Naval

    Future US Navy weapons will need lots of power. That’s a huge engineering challenge.

    David B. Larter WASHINGTON ― The U.S. Navy is convinced that the next generation of ships will need to integrate lasers, electromagnetic rail guns and other power-hungry weapons and sensors to take on peer competitors in the coming decades. However, integrating futuristic technologies onto existing platforms, even on some of the newer ships with plenty of excess power capacity, will still be an incredibly difficult engineering challenge, experts say. Capt. Mark Vandroff, the current commanding officer of the Carderock Division of the Naval Surface Warfare Center and the former Arleigh Burke-class destroyer program manager who worked on the DDG Flight III, told the audience at last week's American Society of Naval Engineers symposium that adding extra electric-power capacity in ships currently in design was a good idea, but that the weapons and systems of tomorrow will pose a significant challenge to naval engineers when it comes time to back-fit them to existing platforms. “Electrical architecture on ships is hard,” Vandroff said. Vandroff considered adding a several-megawatt system to a ship with plenty of power to spare, comparing it with simultaneously turning on everything in a house. “When you turn everything on in your house that you can think of, you don't make a significant change to the load for [the power company],” Vandroff explained. “On a ship, if you have single loads that are [a] major part of the ship's total load, [it can be a challenge]. This is something we had to look at for DDG Flight III where the air and missile defense radar was going to be a major percentage of the total electric load ― greater than anything that we had experienced in the previous ships in the class. That's a real technical challenge. “We worked long and hard at that in order to get ourselves to a place with Flight III where we were confident that when you turned things on and off the way you wanted to in combat, you weren't going to light any of your switchboards on fire. That was not a back-of-the-envelope problem, that was a lot of folks in the Navy technical community ... doing a lot of work to make sure we could get to that place, and eventually we did.” In order to get AMDR, or SPY-6, installed on the DDG design, Vandroff and the team at the DDG-51 program had to redesign nearly half the ship — about 45 percent all told. Even on ships with the extra electric-power capacity, major modifications might be necessary, he warned. “We're going to say that in the future we are going to be flexible, we are going to have a lot of extra power,” Vandroff said. “That will not automatically solve the problem going forward. If you have a big enough load that comes along for a war-fighting application or any other application you might want, it is going to take technical work and potential future modification in order to get there.” Even the powerhouse Zumwalt class will struggle with new systems that take up a large percentage of the ship's power load, Vandroff said. “Take DDG-1000 ― potentially has 80-odd megawatts of power. If you have a 5- or 6-megawatt load that goes on or off, that is a big enough percentage of total load that it's going to be accounted for. Electrical architecture in the future is still an area that is going to require a lot of effort and a lot of tailoring, whatever your platform is, to accommodate those large loads,” he said. In 2016, when the Navy was planning to install a rail gun on an expeditionary fast transport vessel as a demonstration, service officials viewed the electric-power puzzle as the reason the service has not moved more aggressively to field rail gun on the Zumwalt class. Then-director of surface warfare Rear Adm. Pete Fanta told Defense News that he wanted to move ahead with a rail gun demonstration on the JHSV because of issues with the load. “I would rather get an operational unit out there faster than do a demonstration that just does a demonstration,” Fanta said, “primarily because it will slow the engineering work that I have to do to get that power transference that I need to get multiple repeatable shots that I can now install in a ship.” https://www.defensenews.com/naval/2018/06/24/future-navy-weapons-will-need-lots-power-thats-a-huge-engineering-challenge/

  • No more ‘must-wins’: Defense firms growing warier of fixed-price deals

    January 25, 2024 | International, Naval

    No more ‘must-wins’: Defense firms growing warier of fixed-price deals

  • L3Harris receives contract to advance technology for intelligence community

    May 23, 2023 | International, C4ISR

    L3Harris receives contract to advance technology for intelligence community

    L3Harris will lay the groundwork for generating and analyzing human activities that produce data captured by GPS, Bluetooth and other systems

All news