Back to news

April 20, 2018 | International, C4ISR, Security

DARPA official: To build trust in AI, machines must explain themselves

By:

Artificially intelligent systems must be able to explain themselves to operators if they are to be trusted, according to an expert from the Defense Advanced Research Agency, who voiced concern that methods used by current AI systems are often masked by mysterious algorithms.

“A lot of the machine learning algorithms we're using today, I would tell you ‘good luck,” Fred Kennedy, the director of DARPA's Tactical Technology Office during a panel at Navy League's Sea-Air-Space on April 10. “We have no idea why they know the difference between a cat and a baboon.”

“If you start diving down into the neural net that's controlling it,” Kennedy continued, “you quickly discover that the features these algorithms are picking out have very little to do with how humans identify things.”

Kennedy's comments were in response to Deputy Assistant Secretary of the Navy for Unmanned Systems Frank Kelley, who described the leap of faith operators must make when dealing with artificially intelligent systems.

“You're throwing a master switch on and just praying to God that [Naval Research Laboratory] and John's Hopkins knew what the hell that they were doing,” Kelley said of the process.

The key to building trust, according to Kennedy, lies with the machines.

“The system has to tell us what it's thinking,” Dr. Kennedy said. “That's where the trust gets built. That's how we start to use and understand them.”

DARPA's Explainable Artificial Intelligence program seeks to teach AI how to do just that. The program envisions systems that will have the ability to explain the rationale behind their decisions, characterize their strengths and weaknesses, and describe how they will behave in the future. Such capabilities are designed to improve teamwork between man and machine by encouraging warfighters to trust artificially intelligent systems.

“It's always going to be about human-unmanned teaming,” said Kennedy. “There is no doubt about that.”

https://www.defensenews.com/home/2018/04/10/darpa-official-to-build-trust-in-ai-machines-must-explain-themselves/

On the same subject

  • Lockheed and Bell will compete head-to-head to build US Army’s future attack recon aircraft

    March 26, 2020 | International, Aerospace

    Lockheed and Bell will compete head-to-head to build US Army’s future attack recon aircraft

    By: Jen Judson WASHINGTON — Sikorsky, a Lockheed Martin-owned company, and Bell have been selected to build and fly Future Attack Reconnaissance Aircraft (FARA) prototypes for the U.S. Army in a head-to-head competition, according to a March 25 Army statement. The Army is planning to procure both a FARA and Future Long-Range Assault Aircraft (FLRAA) that will slowly replace the current fleet of Sikorsky-manufactured UH-60 Black Hawks utility helicopters and Boeing-made AH-64 Apache attack helicopters. The service plans to initially field both in the 2030s. FARA will fill a critical capability gap currently being filled by AH-64E Apache attack helicopters teamed with Shadow unmanned aircraft following the retirement of the OH-58D Kiowa Warrior helicopters. The service has tried and failed three times to fill the gap with an aircraft. The Army had selected five teams to provide FARA designs last spring: AVX Aircraft Co. partnered with L-3 Communications Integrated Systems, Bell, Boeing, Karem Aircraft and Sikorsky. The Army laid out a handful of mandatory requirements that the vendors had to meet and also a list of desired requirements for initial designs, Col. Craig Alia, the Future Vertical Lift Cross Functional Team chief of staff, told reporters last year. The service also looked at the vendors' execution plans and evaluated timing as well as funding profile requirements. “The ones that were selected were clearly meeting the mandatory requirements and were in the acceptable risk level of the execution plan and the desired requirements," Dan Bailey, who is the FARA competitive prototype program manager, added. The prototype program falls under the purview of the Army Combat Capabilities Development Command Aviation and Missile Center's Aviation Development Directorate. Lockheed unveiled its design — Raider X — at the Association of the U.S. Army's annual conference in October 2019. The company's design uses its Raider X2 coaxial technology with a focus on how it will perform “at the X.” “One thing that always comes out is the importance of this aircraft at the X,” Tim Malia, Sikorsky's director of future vertical lift light, told Defense News in an interview last fall. “The ‘X' is defined by the Army as the terminal area where they actually have to go do the work, do the reconnaissance, do the attack mission. The operation at the X is really critical for this program and for this platform.” That's where Raider X comes in. It's a slightly larger version of the Raider coaxial helicopter that Lockheed has been flying for several years at its West Palm Beach, Florida, flight test center. “Through our mature S-97 RAIDER technology demonstrator, we continue to optimize our FARA solution, which will provide the Army with an integrated weapon system that combines speed, range, maneuverability, survivability and operational flexibility," Andy Adams, Sikorsky's vice president of future vertical lift, said in a March 25 statement. This approach is driving down risk and will result in an aircraft solution that is capable of executing the Army's joint all-domain operations." Bell revealed its design — the Bell 360 Invictus — which is based on 525 technology. But its design features several key differences, including its size in order to adhere to the Army requirement of 40-foot in diameter rotor blades. The design includes a single main rotor helicopter in a four-blade configuration, a low-drag tandem cockpit fuselage and transportability in a C-17. “Bell is proud to continue work on the Bell 360 Invictus as part of the Army's Future Attack Reconnaissance Aircraft Competitive Prototype competition,” Keith Flail, Bell's vice president of advanced vertical lift systems, told Defense News on March 25. “We have made significant investment and begun manufacturing in order to preserve the Army's schedule for FARA CP and we are thrilled to continue our work on the Invictus,” he said. “Bell's approach from the beginning has been tailored to deliver the advanced performance required without excessive complexity in order to get critical capability in the hands of the warfighter quickly and affordably.” The AVX and L3 team was the earliest to unveil its design for the FARA competition at the Army Aviation Association of America's annual summit in Nashville, Tennessee, in April 2019. The design uses AVX's compound coaxial and ducted fans technology. The companies said its single-engine design meets 100 percent of the Army's mandatory requirements and 70 percent of its desired attributes. Karem announced it would team with Northrop and Raytheon and came out with its design at AUSA — its AR-40 — with a single main rotor, tilting compound wings and a rotating tail rotor. Boeing was the last to reveal its design, coming out with it March 3. The company's clean-sheet design featured a hingeless, six-bladed, high-solidity main rotor; a four-bladed conventional tail rotor; and a propeller on the back. The prototype aircraft are expected to start flying in the fourth quarter of fiscal 2022, and the flight test is expected to run through 2023. The engineering and manufacturing development phase is expect to begin in FY24. https://www.defensenews.com/smr/army-modernization/2020/03/25/lockheed-and-bell-will-compete-head-to-head-to-build-us-armys-future-attack-recon-aircraft

  • THE DOD’S APP STORE DOES THIS ONE CRUCIAL THING TO STAY SECURE

    July 5, 2018 | International, C4ISR

    THE DOD’S APP STORE DOES THIS ONE CRUCIAL THING TO STAY SECURE

    Lily Hay Newman EVERY DAY, COMPANIES like Google and Apple wage a constant battle to keep malicious apps out of their marketplaces and off people's phones. And while they do catch a lot of malware before it does any damage, there are always a few nasty infiltrators that manage to sneak by and end up getting downloaded by thousands of consumers. No one wants these mistakes to happen, but when you're a crucial app store for the Department of Defense, these mistakes can't happen. That was the problem facing the National Geospatial-Intelligence Agency as it set about creating a flexible yet ultrasecure app store in 2012. NGA is a combat support organization that primarily assesses and distributes geospatial intelligence. The agency wanted to provide sensitive and mission-critical apps to groups across the DOD through a platform that had the security and resilience of a government defense product, while also offering a streamlined, up-to-date user experience similar to ubiquitous commercial app stores. "We recognized that we did not know everything when it came to apps, and we wanted to be using the innovation that was happening in the commercial sector," says Joedy Saffel, division chief and source director of NGA who has worked on the GEOINT App Store from the beginning. "But how do we do that in a safe, secure manner? How do we do that from a contractual perspective? And how do we do that in a way that nontraditional vendors will trust doing business with the government? It was a great challenge." The key, Saffel says, is getting developers to agree to hand over the source code of their apps for in-depth analysis and review. Whether an app is a simple time/speed/distance calculator for a pilot or a hyper-specialized classified tool, sharing source code is a big risk for developers, because it means trusting third parties with the core intellectual property they have built their businesses on. But NGA soon realized that full access was the only way its project could work. So NGA's GEOINT App Store runs its security protections and screening processes in a way a commercial platform never could. Need To Know You can browse through the GEOINT App Store yourself today and see many of the mapping, aeronautical, weather-forecasting, location-sharing, and travel-alert services that it hosts for Android, iOS, desktop, and web. But that's just the public unclassified section—one crucial aspect of designing the platform was building segmentation controls so DOD employees with different levels of clearance, or simply different needs, could have gated access to different apps. "We built the App Store to be a completely unclassified environment that's open to the public," says Ben Foster, a technical director at NGA who is the product manager for the app store. "But it also has identity management that uses a federated approach to authentication. It's even flexible enough to integrate with other identity-management platforms across DOD. If a user is a helicopter pilot, they might see and get different apps then someone who is a tactical operator in the Army." This system also works with the platform's pricing variations: Some apps are free to everyone, some downloads come with a fee that needs to be taken out of a particular department's budget, and some apps are licensed by NGA or another agency. The most radical part of the GEOINT App Store from a government perspective is the speed with which NGA can process apps and get them live in the store. In general, government acquisition processes take many months or years, a clear problem when it comes to constantly evolving software. So NGA worked with its chief information officer, IT Directorate, legal team, international affairs division, and contracting office to establish a streamlined app-vetting process that would be acceptable under federal acquisition regulations. The agency also contracted with a private firm called Engility to directly manage the outreach, acquisition, and development environment for customizing prospective apps to NGA's requirements. The process, known as the Innovative GEOINT Application Provider Program, or IGAPP, minimizes bureaucratic hurdles and guides developers who want to submit an app through a pipeline that vets, modifies, and generally grooms apps for NGA's store. "What we focused on early on was providing tools so developers can bring their app and do a lot of the pre-testing and development with Engility," NGA's Saffel says. "We're able to be flexible with that because it's being done outside of the government footprint in a brokered environment. And then NGA has a governance board that meets every week, and the whole process has matured enough that by the time an app comes to NGA, we can review it and get that application into the app store and exposed within two weeks' time." Though the process might be even faster if NGA only did the minimum vetting required, Saffel says that the GEOINT team worked to find a balance where the apps go live quickly, but there's still time for the automated code analyses and human audits that commercial app stores can't do. Check It Out After a developer submits their app, Engility does extensive source code analysis and vulnerability scanning and produces an initial findings report. John Holcomb, the IGAPP program manager from Engility, notes that an initial vulnerability report can have as many as 1,000 items on it that a developer needs to address. "It's a little intimidating at first," Holcomb says. "But we walk them through it, and they go back and modify their code—it's their code, we don't modify it for them. We might go through four runs of that on a brand-new app, but by the time we're done, they will have remediated their code down to the level that the government needs. There are still going to be bureaucratic hurdles, but it's our job to break through those." In addition to digging deep into source code, IGAPP also tests how apps function in practice, to make sure that there aren't benign-looking aspects of the code that actually underlie a shady function. "We take the compiled application and we watch what it does," Holcomb says. "Who does it phone home to? Is it sending private information unencrypted?" After an app gets approved for inclusion in the GEOINT App Store, developers continue to work with IGAPP on developing and vetting software updates so that patches and improvements can be pushed out quickly. The brokered vetting process means that the government never holds developers' source code directly. The inspection is always mediated by Engility, which signs nondisclosure agreements with developers and isn't a software maker itself. Holcomb says that the company carefully guards app data while storing it, and once a project is done, Engility doesn't just do a soft data deletion; it hard-purges the information from its cloud servers within 30 days. NGA's Saffel and Holcomb both note that developers were apprehensive about the unusual workflow at first, but over the years the app store has gained credibility. Developers say they benefit from the IGAPP process both by securing lucrative government contracts and by integrating the improvements from the IGAPP development into their commercial products. The code audits and security vetting IGAPP offers are expensive, so developers generally don't do such extensive assessment on their own. "Everyone's dream is to sell to the government, but it normally takes years of effort to get to a position where you can. In our case, I was able to sell to the government in less than a month," says Bill DeWeese, CEO of the firm Aviation Mobile Apps, which has had six apps accepted into the GEOINT App Store. "You do feel a little anxiety about sharing source code, you worry about your IP leaking and someone getting ahold of it. But I haven't had any issues, and the benefit is the increased quality of your products at no cost—you get the analysis for free and you can put it in your commercial offerings." NGA's Saffel says the governance board that evaluates the apps at the end of the process is careful to stay vigilant so nothing goes into the store by accident. The board will still push back on apps or turn them away when warranted, but Saffel says the process has matured such that most of what the board sees these days is ready or very near ready to go live. And IGAPP prioritizes its patching process and infrastructure, to make it easy for developers to push bug fixes and improvements throughout the life of an app. All of this means a consumer-grade turnaround time for critical Department of Defense tools without the consumer-grade security concerns. "NGA is kind of a unique combat-support agency," Saffel says. "With the GEOINT App Store we chose to go into a very risky new frontier for DOD and the government in general, but I think we've demonstrated that we can do things differently and still be secure and still control access. We're supporting a lot of different mission sets, and I expect that the app store will keep growing." https://www.wired.com/story/dod-app-store-does-this-one-crucial-thing-to-stay-secure/

  • Trump Signs Executive Order Establishing First National Artificial Intelligence Initiative

    February 19, 2019 | International, C4ISR, Other Defence

    Trump Signs Executive Order Establishing First National Artificial Intelligence Initiative

    President Trump on Monday signed an executive order establishing the U.S.' first artificial intelligence initiative, which directs federal agencies to prioritize research funds for AI development to stay ahead of peer competitors such as China. https://www.defensedaily.com/trump-signs-executive-order-establishing-first-national-artificial-intelligence-initiative

All news