Back to news

April 20, 2018 | International, C4ISR, Security

DARPA official: To build trust in AI, machines must explain themselves

By:

Artificially intelligent systems must be able to explain themselves to operators if they are to be trusted, according to an expert from the Defense Advanced Research Agency, who voiced concern that methods used by current AI systems are often masked by mysterious algorithms.

“A lot of the machine learning algorithms we're using today, I would tell you ‘good luck,” Fred Kennedy, the director of DARPA's Tactical Technology Office during a panel at Navy League's Sea-Air-Space on April 10. “We have no idea why they know the difference between a cat and a baboon.”

“If you start diving down into the neural net that's controlling it,” Kennedy continued, “you quickly discover that the features these algorithms are picking out have very little to do with how humans identify things.”

Kennedy's comments were in response to Deputy Assistant Secretary of the Navy for Unmanned Systems Frank Kelley, who described the leap of faith operators must make when dealing with artificially intelligent systems.

“You're throwing a master switch on and just praying to God that [Naval Research Laboratory] and John's Hopkins knew what the hell that they were doing,” Kelley said of the process.

The key to building trust, according to Kennedy, lies with the machines.

“The system has to tell us what it's thinking,” Dr. Kennedy said. “That's where the trust gets built. That's how we start to use and understand them.”

DARPA's Explainable Artificial Intelligence program seeks to teach AI how to do just that. The program envisions systems that will have the ability to explain the rationale behind their decisions, characterize their strengths and weaknesses, and describe how they will behave in the future. Such capabilities are designed to improve teamwork between man and machine by encouraging warfighters to trust artificially intelligent systems.

“It's always going to be about human-unmanned teaming,” said Kennedy. “There is no doubt about that.”

https://www.defensenews.com/home/2018/04/10/darpa-official-to-build-trust-in-ai-machines-must-explain-themselves/

On the same subject

  • USS Laboon shoots down four drones in Red Sea

    December 24, 2023 | International, Naval

    USS Laboon shoots down four drones in Red Sea

    U.S. Central Command said the drones “were inbound” to the destroyer.

  • Cyber Command’s measure of success? Outcomes

    July 14, 2020 | International, C4ISR, Security

    Cyber Command’s measure of success? Outcomes

    Mark Pomerleau A U.S. Cyber Command official said that when they examine whether any given operation or even when a strategy has been successful, they're not looking at metrics, but rather outcomes. “It's really about: have we enabled the collective defense of the nation,” Maj. Gen. John Morrison, Cyber Command's outgoing chief of staff, told C4ISRNET in a July interview. Roughly two years ago, Cyber Command and the Department of Defense started a paradigm shift for cyber policy and operations. The 2018 DoD cyber strategy tasked Cyber Command to “defend forward,” which is best described as operators working on foreign networks to prevent attacks before they happen. The way Cyber Command meets those goals is through persistent engagement, which means challenging adversary activities wherever they operate. Part of the need for a change was that adversaries were achieving their objectives but doing so below the threshold of armed conflict – in the so-called gray zone – through cyberspace. DoD wanted to stop that from happening through more assertive cyberspace action. Some in the academic community have wanted to see some way in which the command can measure the success of these new approaches. But Morrison explained that these outcomes, or intended effects during operations, could be enabling other partners – foreign or other agencies within the U.S. government – to take action in defense of the nation. For example, he said that when Cyber Command teams encounter malware they haven't seen before, they share it with partners in government, such as FBI or DHS, which can lead to the greater national collective defense. He also noted that building partnerships enables a sense of collective defense in cyberspace and can help significantly in the future against sophisticated adversaries. Morrison will be replaced at Cyber Command by Maj. Gen. David Isaacson. It is unclear where Morrison is headed next. The need for flexibility As Cyber Command has gained more authorities in recent years, it has been able to conduct significantly more operations and different types of operations as well, Morrison said. Throughout these missions, leaders have learned they must be flexible, be it in tactics, structure of teams, or the capabilities they need or develop. “We have thinking adversaries that we go against every single day. That drives us to change how we operate,” Morrison said. “You change your tactics, techniques and procedures but that's also going to drive changes in how we train and what we train ... It drives how we do capability development and development of capabilities and the employment of those capabilities, which again ties back to training at a much faster pace in this space.” Morrison noted that this includes how teams are organized. He explained the way defensive cyber protection teams were first envisioned when they were created in 2012-2013 is not at all how they fight now. To keep up with dynamic adversaries, Cyber Command is keeping closer watch on readiness metrics developed by the command for its cyber teams. This is a framework that details standards for how teams are equipped, manned and supplied. Cyber protection teams were detailed first and now Cyber Command has readiness metrics for combat mission teams, the offensive teams that support combatant commands, and intelligence/support teams. Officials are still working through metrics for what are called national teams that are charged with defending the nation. The command also needs to improve the way it feeds operational requirements into capabilities cyber warriors can use, Morrison said. This includes improving acquisition practices for both of the programs of record Cyber Command is executing through its Joint Cyber Warfighting Architecture — which guides capability development priorities and includes the Unified Platform and Persistent Cyber Training Environment — and the more rapidly developed tools needed on the fly. “That's where you've got the ability inside the command now to rapidly produce that capability through a variety of means and get it into the hands of our operators as quick as possible,” he said. In fact, the Army has begun to embed tool developers and coders alongside operators through the Rapid Cyber Development Network to more quickly meet urgent needs. This allows them in almost in near real time to develop or change tools to meet requirements. “How do we do capability development in a much smoother fashion than we sometimes do today where we're able to rapidly assess, prioritize, resource operational requirements to produce a capability that we can then get back into the hands of our operators as quickly as possible,” Morrison said. From these capabilities that are developed for shorter term needs, he said the key will be deciding if they want to move them into a program of record. Will it be a longer term capability, will it adjust tactics, techniques and procedures or training? “We've got to work those pieces,” he explained. On the longer term, program of record capabilities, he noted officials still want the iterative development associated with more software-centric systems as opposed to more traditional military hardware. Integration with combatant commands Cyber is much more ingrained in military planning and operations than it was in years prior, Morrison said, however, work remains. There is now a closer link between the combatant commands and Cyber Command elements that plan, coordinate, synchronize and conduct cyber operations on their behalf, Morrison said, noting that they are still maturing. These include the Joint Force Headquarters-Cybers‚ which are commanded by each of the service cyber component commanders, and plan, synchronize and conduct operations for combatant commands they're assigned to, and new entities being created called cyber operations-integrated planning elements. These are forward extensions of the Joint Force Headquarters resident within the combatant commands to better coordinate cyber planning with other operations for the combatant commander. These entities all enable a greater central connective tissue from a Cyber Command perspective as they can feed from the theater level back to the command providing a global cyberspace picture. “You have to take not only a regional view of anything that you're doing, but, when you can bring the power of a global enterprise behind it, that's a pretty powerful capability for our nation,” Morrison said. “We are in the process of building every one of our CO-IPEs but I definitely think that we are heading in the right direction, especially as [the CO-IPEs] get built and they integrate closer and closer with their supported combatant commands.” https://www.c4isrnet.com/cyber/2020/07/10/cyber-commands-measure-of-success-outcomes/

  • Inside South Korea's race to become one of the world’s biggest arms dealers

    May 29, 2023 | International, Other Defence

    Inside South Korea's race to become one of the world’s biggest arms dealers

    South Korea is using a $13.7 billion arms deal with Poland to lay the groundwork for a military-industrial juggernaut.

All news