Back to news

December 31, 2018 | International, C4ISR

US Spies Want to Know How to Spot Compromised AI

BY DAVE GERSHGORN

What if you were training an AI, and an adversary slipped a few altered images into its study set?

The US government's research arm for intelligence organizations, IARPA, is looking for ideas on how to detect “Trojan” attacks on artificial intelligence, according to government procurement documents.

Here's the problem the agency wants to solve: At a simple level, modern image-recognition AI learns from analyzing many images of an object. If you want to train an algorithm to detect pictures of a road signs, you have to supply it with pictures of different signs from all different angles. The algorithm learns the relationships between the pixels of the images, and how the structures and patterns of stop signs differ from those of speed-limit signs.

But suppose that, during the AI-training phase, an adversary slipped a few extra images (Trojan horses) into your speed-limit-sign detector, ones showing stop signs with sticky notes on them. Now, if the adversary wants to trick your AI in the real world into thinking a stop sign is a speed-limit sign, it just has to put a sticky note on it. Imagine this in the world of autonomous cars; it could be a nightmare scenario.

The kinds of tools that IARPA (Intelligence Advanced Research Projects Activity) wants would be able to detect issues or anomalies after the algorithm has been trained to recognize different objects in images.

This isn't the only kind of attack on AI that's possible. Security researchers have also warned about inherent flaws in the way artificial intelligence perceives the world, making it possible to alter physical objects like stop signs to make AI algorithms miscategorize them without ever messing with how it was trained, called “adversarial examples.”

While neither Trojan attacks nor the adversarial examples are known to have been used by malicious parties in the real world, researchers have said they're increasingly possible. IARPA is looking at a short timeline as well, expecting the program to conclude after a maximum of two years.

https://www.defenseone.com/technology/2018/12/us-spies-want-know-how-spot-compromised-ai/153826

On the same subject

  • European Union tees up new military-cooperation proposals

    May 8, 2019 | International, Aerospace, Naval, Land, C4ISR, Security, Other Defence

    European Union tees up new military-cooperation proposals

    By: Martin Banks The European Union will launch a call for a new batch of proposals as part of Europe's new Permanent Structured Cooperation (PESCO) initiative this week. At the same time, officials said that the 34 existing PESCO projects are still considered to be at a very initial “incubation phase,” meaning they have yet to come to fruition. The PESCO defense pact – a show of unity and a tangible step in EU integration – was set up in December 2017 between EU governments and involved two phases of joint initiatives, each consisting of 17 projects. The third and latest phase, to be launched this week, is for an unspecified number of new projects. The founding PESCO members, including France, Germany and Italy, have been asked to table proposals by the summer with a view to these being approved by the end of 2019. The new batch of projects is likely to be smaller than the previous two, the second of which was launched last November, and is expected to be more “mature” when it comes to the projects' setup, including support by member states, one official said. EU members are responsible for developing and implementing PESCO projects. An EU defence source said, “They are still at an initial stage, or incubation phase.” Twelve of the existing 34 schemes are expected to reach initial operational capability by 2022, with four of these due to be implemented later this year, according to the source. The 34 schemes include a harbor and maritime surveillance and protection (HARMSPRO) project, designed to deliver a new maritime capability with the ability to conduct surveillance and protection of specified maritime areas, from harbors up to littoral waters. Another is the Training Mission Competence Centre which aims to improve the availability and professionalism of personnel for EU training missions. The list also includes a European armoured infantry vehicle and cyber rapid response teams. Other projects involve developing new equipment, such as infantry fighting vehicles, amphibious assault vehicles, light armored vehicles, indirect fire support, strategic command-and-control systems for EU defense missions, minesweeping drones, upgrading maritime surveillance and developing a joint secure software defined radio. Long blocked by London, PESCO, is one of the most tangible steps in EU integration since Britons voted to leave the bloc, as militaries begin to plan, spend and deploy together. The eventual aim of PESCO is to develop and deploy forces together, backed by a multi-billion-euro fund for defense research and development. The idea aims to bring together European countries with a military capacity and political desire to collaborate on planning, carry out joint analyses of emerging crises and to react to them quickly. Speaking recently in the European parliament in Brussels, Finnish Prime Minister Juha Sipilä welcomed the establishment of PESCO as a “step in the right direction” but said PESCO members “should now concentrate on implementation and reaching results.” The EU source said, “Some might find it surprising that the 34 projects are still at the ideation phase but you have to remember that the PESCO project was launched only recently so the record is not bad. We are not talking about a ‘project factory' but a commitment on the part of participating members to work more closely in the area of security and defence.” EU defence expert Paul Taylor wrote, “It is worth noting that the PESCO effort is still at a relatively early stage of development.” Jamie Shea, a senior fellow at Friends of Europe, a leading Brussels think tank, commented, “It is welcome news that the number of PESCO projects is likely soon to grow still further beyond the current 34. But to sustain political and public interest in this initiative it is important that we see soon the first deliverables to show that the good intentions are being followed with real and new European military capabilities.” Shea added, “Moreover the key test for the success of PESCO will not just be to generate more multinational efforts but also to produce capabilities that plug the current shortfalls in the EU's most urgent requirements and move it towards its goal of strategic autonomy.” https://www.defensenews.com/global/europe/2019/05/06/european-union-tees-up-new-military-cooperation-proposals/

  • France qualifies Serval armored patrol vehicle variant for its Army

    December 2, 2022 | International, Land

    France qualifies Serval armored patrol vehicle variant for its Army

    The green light signifies a step forward for realizing the ground service's large-scale Scorpion modernization program.

  • House lawmakers propose $1.3 billion for Pentagon innovation

    June 4, 2024 | International, Security

    House lawmakers propose $1.3 billion for Pentagon innovation

    The bill would add more than $500 million for DIU initiatives to quickly deliver capabilities to combatant commands and the military services.

All news