Back to news

April 20, 2018 | International, C4ISR, Security

DARPA official: To build trust in AI, machines must explain themselves

By:

Artificially intelligent systems must be able to explain themselves to operators if they are to be trusted, according to an expert from the Defense Advanced Research Agency, who voiced concern that methods used by current AI systems are often masked by mysterious algorithms.

“A lot of the machine learning algorithms we're using today, I would tell you ‘good luck,” Fred Kennedy, the director of DARPA's Tactical Technology Office during a panel at Navy League's Sea-Air-Space on April 10. “We have no idea why they know the difference between a cat and a baboon.”

“If you start diving down into the neural net that's controlling it,” Kennedy continued, “you quickly discover that the features these algorithms are picking out have very little to do with how humans identify things.”

Kennedy's comments were in response to Deputy Assistant Secretary of the Navy for Unmanned Systems Frank Kelley, who described the leap of faith operators must make when dealing with artificially intelligent systems.

“You're throwing a master switch on and just praying to God that [Naval Research Laboratory] and John's Hopkins knew what the hell that they were doing,” Kelley said of the process.

The key to building trust, according to Kennedy, lies with the machines.

“The system has to tell us what it's thinking,” Dr. Kennedy said. “That's where the trust gets built. That's how we start to use and understand them.”

DARPA's Explainable Artificial Intelligence program seeks to teach AI how to do just that. The program envisions systems that will have the ability to explain the rationale behind their decisions, characterize their strengths and weaknesses, and describe how they will behave in the future. Such capabilities are designed to improve teamwork between man and machine by encouraging warfighters to trust artificially intelligent systems.

“It's always going to be about human-unmanned teaming,” said Kennedy. “There is no doubt about that.”

https://www.defensenews.com/home/2018/04/10/darpa-official-to-build-trust-in-ai-machines-must-explain-themselves/

On the same subject

  • Four big questions for cybersecurity in 2019

    January 2, 2019 | International, C4ISR

    Four big questions for cybersecurity in 2019

    By: Justin Lynch How will cybersecurity experts remember 2018? In the past year, the Trump administration announced it would take more offensive hacking operations against foreign countries, the Department of Justice announcedsweeping indictments against Chinese hackers and the U.S. intelligence community reported that foreign countries continued to interfere in American elections. So what comes next? Here are four overarching questions for the cybersecurity community in 2019: What will the new Pentagon chief do with expanded cyber powers? In August, the president gave the secretary of Defense the ability to conduct cyberattacks against foreign countries so long as they do not interfere with the national interest of the United States, according to four current and former White House and intelligence officials. But the resignation of Jim Mattis, the Defense secretary, means the next Pentagon chief will have a broad arsenal of cyber authorities. For the cyber community, Patrick Shanahan, the current acting secretary, is a relative unknown. He has not given significant insight into how he views the role of offensive cyberattacks for the Pentagon, and his scheduled Jan. 1 elevation comes as some in the Trump administration and U.S. Cyber Command have pushed for even more authorities. However, he has spoken at length about the need for the defense industry to bolster its own cyber practices. Although the appointment of Shanahan as acting Pentagon chief is temporary, he is on the short list of officials who may take on the job full time. The new Pentagon chief may also have to decide when the National Security Agency and U.S. Cyber Command should split. Both bodies are led by Gen. Paul Nakasone, but that may change. Cyber Command is in the process of gaining its own infrastructure to conduct offensive cyberattacks, and a Pentagon official told Fifth Domain in November that it appeared the split was all but certain to happen in the coming years, although no formal decision as been made. What comes next in the U.S.-China cyber relationship? The Department of Justice released a flurry of indictments against Chinese hackers in 2018, accusing Beijing's cyber sleuths of infiltrating American government agencies and defense contractors. The most recent round of allegations came Dec. 18, and the legal action could continue in 2019. While announcing the most recent indictments, Deputy Attorney General Rod Rosenstein accused China of breaking an agreement not to use hacked materials for commercial use, although he did not offer evidence. The hacking allegations come amid a broader trade war between the United States and China. Experts have told Fifth Domain a trade war could increase digital tension between the two nations. If the trade war continues, experts say they see little incentive for China to limit its cyberattacks. Will America suffer blowback for more offensive cyber operations? When the Trump administration announced the United States would take more offensive actions in cyberspace, some in the federal cybersecurity community criticized the plan as faulty. “The side effects of the strategy of ‘persistent engagement' and ‘defend forward' are still ill-understood,” Max Smeets and Herb Lin, experts at Stanford University wrote for Lawfare. “A United States that is more powerful in cyberspace does not necessarily mean one that is more stable or secure.” Experts also warn of making any rush judgments about the effectiveness of these offensive cyberattacks. Current and former intelligence officials worry that uncovering and attributing a hack can take more than a year, and, even then, that process is not perfect. One former official pointed to the leaked documents about Russian targeting of American election infrastructure in 2016 that was sent to the news organization the Intercept. It took months for the intelligence community to understand the full extent of the hack, the official said, an example of how long it takes to detect a cyberattack. However, all of that means it is reasonable to expect that the merits of the new offensive cyber operations may not be known publicly for years. Will Congress take action to streamline cybersecurity contracting and research? Yes, changing the way government does business is ambitious. But experts argue that if the United States wants to keep up with digital innovations from China and other countries it is necessary to change the American government's relationship with the private sector and academia. The effort to streamline cybersecurity funding and research will fall to the new Congress, in which Democrats will take over the House of Representatives. But when it comes to the U.S. government's relationship with the cyber industry, structural barriers to innovation remain. On average, it takes roughly seven years for an idea to get a contract inside the U.S. government. In that length of time, a product is already two generations old. Former Pentagon officials have used the digital fight against the Islamic State as an example of how long the process takes. It took roughly two years for Cyber Command to receive the proper equipment and training after the order to digitally defeat the Islamic State, officials told Fifth Domain. In addition, the cybersecurity industry is watching a series of bills in Congress. Sen. Mark Warner, D-Va., has pushed for a streamlined security clearance process, and industry officials told Fifth Domain they expect him to continue the effort in the new year. The bill could make it easier and cheaper to get a security clearance. And many in the federal cybersecurity community have called for a change in academia's relationship with cybersecurity. The universities and research institutions in the United States focusing on quantum computing are “subpar,” George Barnes, deputy director at the NSA said in June. Experts say that quantum computers will make traditional cybersecurity methods obsolete because of the expansive computing power. However, new investments in artificial intelligence and a new Solarium Commission, which was created to help contextualize cyber in the broader national and economic security discussion, may provide solutions to these problems. https://www.fifthdomain.com/industry/2018/12/31/four-big-questions-for-cybersecurity-in-2019

  • EU mulls billions in funding to quicken artillery shell production

    March 2, 2023 | International, Land

    EU mulls billions in funding to quicken artillery shell production

    The move comes as Ukraine's fortunes depend on the West's ability to continue supplying weapons against Russian attacks.

  • Northrop Grumman avoids Triton production pause

    March 30, 2021 | International, Aerospace

    Northrop Grumman avoids Triton production pause

    Northrop Grumman has avoided an anticipated gap in production of its MQ-4C Triton high-altitude, long-endurance (HALE) unmanned aerial vehicle (UAV), with the announcement that Congress had reintroduced a previously removed aircraft from the...

All news