Back to news

January 21, 2020 | International, C4ISR

Trustworthy AI: A Conversation with NIST's Chuck Romine

By: Charles Romine

Artificial Intelligence (AI) promises to grow the economy and improve our lives, but with these benefits, it also brings new risks that society is grappling with. How can we be sure this new technology is not just innovative and helpful, but also trustworthy, unbiased, and resilient in the face of attack? We sat down with NIST Information Technology Lab Director Chuck Romine to learn how measurement science can help provide answers.

How would you define artificial intelligence? How is it different from regular computing?

One of the challenges with defining artificial intelligence is that if you put 10 people in a room, you get 11 different definitions. It's a moving target. We haven't converged yet on exactly what the definition is, but I think NIST can play an important role here. What we can't do, and what we never do, is go off in a room and think deep thoughts and say we have the definition. We engage the community.

That said, we're using a narrow working definition specifically for the satisfaction of the Executive Order on Maintaining American Leadership in Artificial Intelligence, which makes us responsible for providing guidance to the federal government on how it should engage in the standards arena for AI. We acknowledge that there are multiple definitions out there, but from our perspective, an AI system is one that exhibits reasoning and performs some sort of automated decision-making without the interference of a human.

There's a lot of talk at NIST about “trustworthy” AI. What is trustworthy AI? Why do we need AI systems to be trustworthy?

AI systems will need to exhibit characteristics like resilience, security and privacy if they're going to be useful and people can adopt them without fear. That's what we mean by trustworthy. Our aim is to help ensure these desirable characteristics. We want systems that are capable of either combating cybersecurity attacks, or, perhaps more importantly, at least recognizing when they are being attacked. We need to protect people's privacy. If systems are going to operate in life-or-death type of environments, whether it's in medicine or transportation, people need to be able to trust AI will make the right decisions and not jeopardize their health or well-being.

Resilience is important. An artificial intelligence system needs to be able to fail gracefully. For example, let's say you train an artificial intelligence system to operate in a certain environment. Well, what if the system is taken out of its comfort zone, so to speak? One very real possibility is catastrophic failure. That's clearly not desirable, especially if you have the AI deployed in systems that operate critical infrastructure or our transportation systems. So, if the AI is outside of the boundaries of its nominal operating environment, can it fail in such a way that it doesn't cause a disaster, and can it recover from that in a way that allows it to continue to operate? These are the characteristics that we're looking for in a trustworthy artificial intelligence system.

NIST is supposed to be helping industry before they even know they needed us to. What are we thinking about in this area that is beyond the present state of development of AI?

Industry has a remarkable ability to innovate and to provide new capabilities that people don't even realize that they need or want. And they're doing that now in the AI consumer space. What they don't often do is to combine that push to market with deep thought about how to measure characteristics that are going to be important in the future. And we're talking about, again, privacy, security and resilience ... trustworthiness. Those things are critically important, but many companies that are developing and marketing new AI capabilities and products may not have taken those characteristics into consideration. Ultimately, I think there's a risk of a consumer backlash where people may start saying these things are too easy to compromise and they're betraying too much of my personal information, so get them out of my house.

What we can do to help, and the reason that we've prioritized trustworthy AI, is we can provide that foundational work that people in the consumer space need to manage those risks overall. And I think that the drumbeat for that will get increasingly louder as AI systems begin to be marketed for more than entertainment. Especially at the point when they start to operate critical infrastructure, we're going to need a little more assurance.

That's where NIST can come together with industry to think about those things, and we've already had some conversations with industry about what trustworthy AI means and how we can get there.

I'm often asked, how is it even possible to influence a trillion-dollar, multitrillion-dollar industry on a budget of $150 million? And the answer is, if we were sitting in our offices doing our own work independent of industry, we would never be able to. But that's not what we do. We can work in partnership with industry, and we do that routinely. And they trust us, they're thrilled when we show up, and they're eager to work with us.

AI is a scary idea for some people. They've seen “I, Robot,” or “The Matrix,” or “The Terminator.” What would you say to help them allay these fears?

I think some of this has been overhyped. At the same time, I think it's important to acknowledge that risks are there, and that they can be pretty high if they're not managed ahead of time. For the foreseeable future, however, these systems are going to be too fragile and too dependent on us to worry about them taking over. I think the biggest revolution is not AI taking over, but AI augmenting human intelligence.

We're seeing examples of that now, for instance, in the area of face recognition. The algorithms for face recognition have improved at an astonishing rate over the last seven years. We're now at the point where, under controlled circumstances, the best artificial intelligence algorithms perform on par with the best human face recognizers. A fascinating thing we learned recently, and published in a report, is that if you take two trained human face recognizers and put them together, the dual system doesn't perform appreciably better than either one of them alone. If you take two top-performing algorithms, the combination of the two doesn't really perform much better than either one of them alone. But if you put the best algorithm together with a trained recognizer, that system performs substantially better than either one of them alone. So, I think, human augmentation by AI is going to be the revolution.

What's next?

I think one of the things that is going to be necessary for us is pulling out the desirable characteristics like usability, interoperability, resilience, security, privacy and all the things that will require a certain amount of care to build into the systems, and get innovators to start incorporating them. Guidance and standards can help to do that.

Last year, we published our plan for how the federal government should engage in the AI standards development process. I think there's general agreement that guidance will be needed for interoperability, security, reliability, robustness, these characteristics that we want AI systems to exhibit if they're going to be trusted.

https://www.nist.gov/blogs/taking-measure/trustworthy-ai-conversation-nists-chuck-romine

On the same subject

  • Trump orders creation of independent space force - but Congress will still have its say

    June 19, 2018 | International, Aerospace, Naval

    Trump orders creation of independent space force - but Congress will still have its say

    Valerie Insinna and Aaron Mehta WASHINGTON — President Donald Trump on Monday appeared to sign an executive order directing the Pentagon to create a new ”Space Force,” a move that could radically transform the U.S. military by pulling space functions variously owned by the Air Force, Navy and other military branches into a single independent service. But while the president's support for a new military branch is notable, experts -- and a powerful member of Congress -- believe Trump still needs the support of Congress to make a space force happen. “I am hereby directing the Department of Defense and Pentagon to immediately begin the process necessary to establish a Space Force as the sixth branch of the armed forces,” Trump said during a meeting of the National Space Council. “That's a big statement. We are going to have the Air Force and we are going to have the Space Force. Separate but equal. It is going to be something. So important,” Trump added. “General Dunford, if you would carry that assignment out, I would be very greatly honored.” Dunford responded in the affirmative, telling Trump, “We got you.” According to a White House pool report, the president signed the executive order establishing the Space Force at about 12:36 p.m. EST. However, a readout issued from the White House later that day of the executive order contained no language related to the creation of a new military branch, leaving open the question of whether Trump has actually issued formal guidance to the military. The Air Force referred all questions to the Office of the Secretary of Defense, which did not respond immediately to requests for comment. However, a defense official, speaking on background, said “The Joint Staff will work closely with the Office of the Secretary of Defense, other DoD stakeholders and the Congress to implement the President's guidance." Trump's support for creating a separate branch for space is a break from his own adminsitration's stance last year, as well as that of Secretary of Defense Jim Mattis. “At a time when we are trying to integrate the Department's joint warfighting functions, I do not wish to add a separate service that would likely present a narrower and even parochial approach to space operations vice an integrated one we're constructing under our current approach,” Mattis wrote in a 2017 letter to members of Congress. But in recent months, Trump has signaled he was intrigued by the idea of a stand alone space force, saying in a May 1 speech that “We're actually thinking of a sixth” military branch for space. At the time, that statement confounded Air Force leaders who had publicly opposed the creation of a separate space service, leading them to adopt a softer tone when talking about the potential for Space Force to avoid being seen as out of step with Trump. This time, however, Trump's announcement tracks with the Pentagon's schedule for an interim report on whether to establish an independent space corps. Deputy Secretary of Defense Patrick Shanahan said in April that it was on track to be wrapped up on June 1. The final report, which would be sent to Congress, is due in August. Trump's announcement was characteristically vague, but experts say that any new branch would have to come through an act of Congress. “The Congress alone has the power to establish a new branch of the military and to establish the positions of senior executive officials to lead such a department,” said Jonathan Turley, a professor at Georgetown University's law school who has studied constitutional issues relating to the military. “While the Pentagon can informally create study or working groups, it has no such authority.” The president can have the military lay the groundwork for a future new branch, Turley said, which is close to what Trump seemed to be getting at. By: Kelsey Atherton “What the President can do is to order the study and proposal for a new branch, which would ultimately go to Congress of any authorization and appropriations,” he said. Todd Harrison, an expert with the Center for Strategic and International Studies, agreed, tweeting Monday that “The president can't just create a new military service on his own. It requires congressional authorization..” “So the near-term practical effect of all this is that the president can direct DoD to come up with a plan and start preparing to create a Space Force, but he still needs congress to authorize it,” Harrison continued. And while sources on Capitol Hill said they believe Trump does have the authority to establish the new military branch, and that their attention will now turn to funding and missions for the new Space Force, at least one Republican member of Congress made his stance clear. “Establishing a service branch requires congressional action,” House Armed Services Tactical Air and Land Forces Subcommittee chair Mike Turner, R-Ohio. “We still don't know what a Space Force would do, who is going to be in it, or how much is it going to cost. “The congressionally mandated report evaluating a Space Force to answer those questions is due in August,” Turner added. “After we get the report that we required as a legislative body and the President signed off on, then this issue can be appropriately evaluated for what's best for national security.” Congress reacts Trump's announcement also left it unclear whether this new space force will rest under the Department of the Air Force — much like the Marine Corps is a component of the Department of the Navy — or whether a new “Department of the Space Force” will also be created. Rep. Mike Rogers, R-Ala., the head of the House Armed Services strategic forces subcommittee, tweeted out his support for Trump's order. Rogers had previously proposed a separate space service as part of Congress' annual defense policy bill. However, lawmakers and experts also immediately registered their opposition to the announcement. Sen. Bill Nelson, (D-Fla.), the top Democrat on the Senate Commerce Committee that oversees nonmilitary space programs, tweeted that now was not the right time to establish a separate space force. Harrison noted that the infrastructure may already exist to smooth the creation of a space force. “Creating a Space Force would not necessarily mean a huge increase in funding. We already have space forces within the military, this would just be reorganizing them under a single chain of command,” he tweeted. “Yes, there would be some extra overhead costs, but it doesn't have to be huge.” But David Deptula, a retired Air Force lieutenant general and currently dean of the Mitchell Institute for Aerospace Studies, questioned whether the administration had hammered down the details needed to successfully consolidate the military's space functions into a single service. “This is another case of ready, fire, aim,” he said. David Larter, Joe Gould, Tara Copp and Leo Shane III contributed to this report. This story is developing.

  • Spirit selected to lead MOSQUITO uncrewed fighter development

    February 1, 2021 | International, Aerospace

    Spirit selected to lead MOSQUITO uncrewed fighter development

    Thursday, 28 January 2021 Autonomous ‘loyal wingman' aircraft create the opportunity to expand, diversify and rapidly upgrade combat Air Forces in a cost-effective way - so the UK has committed funding for a new prototype. The UK's first fleet of uncrewed fighter aircraft is one step closer to reality following a £30-million contract to design and manufacture a prototype in a three-year deal supporting more than 100 jobs in Belfast. The uncrewed combat aircraft will be designed to fly at high-speed alongside fighter jets, armed with missiles, surveillance and electronic warfare technology to provide a battle-winning advantage over hostile forces. Known as a ‘loyal wingman', these aircraft will be the UK's first uncrewed platforms able to target and shoot down enemy aircraft and survive against surface to air missiles. In a boost for Northern Ireland's defence industry, Spirit AeroSystems, Belfast, has been selected to lead Team MOSQUITO in the next phase of the project. Using ground-breaking engineering techniques, the team will further develop the RAF's Lightweight Affordable Novel Combat Aircraft (LANCA) concept, with a full-scale vehicle flight-test programme expected by the end of 2023. Northern Ireland Secretary, Brandon Lewis says: "This is fantastic news and underlines the distinct strengths in Northern Ireland's economy, through its advanced engineering and manufacturing capabilities. This ground-breaking project will involve significant investment which will not only support local employment, but also reinforce Northern Ireland's contribution to the security of our nation." Team MOSQUITO, which also includes Northrop Grumman UK, will mature the designs and manufacture a technology demonstrator to generate evidence for a follow-on LANCA programme. If successful, Project Mosquito's findings could lead to this revolutionary capability being deployed alongside the Typhoon and F-35 Lightning jets by the end of the decade. Defence Minister, Jeremy Quin said: "This is a great win for the Northern Ireland defence industry and will showcase some of the most pioneering engineering work currently being undertaken in the UK. The £30 million project will accelerate the development of the UK's future air power by delivering cutting-edge uncrewed aircraft, maintaining our position as a world leader in emerging technologies." Working with innovative partners from across the UK, Project Mosquito is transforming traditional approaches to combat air to enable the rapid development of technology. By utilising the latest software development techniques and civilian aerospace engineering and manufacturing expertise, the project will deliver dramatic reductions in costs and development timelines, so their innovations can reach the front-line quicker than ever before. This game changing research and development project will ensure the final aircraft design will be capable of being easily and affordably updated with the latest technology so we remain one step ahead of our adversaries. The aircraft's flexibility will provide the optimum protection, survivability and information as it flies alongside Typhoon, F-35 Lightning, and later, Tempest as part of our future combat air system. Air Chief Marshal Mike Wigston, Chief of the Air Staff says: "We're taking a revolutionary approach, looking at a game-changing mix of swarming drones and uncrewed fighter aircraft like Mosquito, alongside piloted fighters like Tempest, that will transform the combat battlespace in a way not seen since the advent of the jet age." Director Future Combat Air, Richard Berthon adds: "Project Mosquito is a vital element of our approach to Future Combat Air, rapidly bringing to life design, build and test skills for next generation combat air capabilities. Autonomous ‘loyal wingman' aircraft create the opportunity to expand, diversify and rapidly upgrade Combat Air Forces in a cost-effective way, now and in the future." As announced by the Prime Minister in November 2020, the UK's Future Combat Air System (FCAS) programme is set to benefit from a portion of the extra £1.5 billion investment into military research and development, which will help ensure our Armed Forces are prepared to meet the threats of tomorrow. LANCA originated in 2015 in Dstl to understand innovative Combat Air technologies and concepts that offer radical reductions in cost and development time and is a RAF Rapid Capabilities Office led project under the Future Combat Air System Technology Initiative (FCAS TI). The UK MOD's Defence Science and Technology Laboratory (Dstl) provides the project management and is the MOD's technical authority for LANCA and Project Mosquito on behalf of the RCO. https://www.airframer.com/news_story.html?report=20244

  • DSEI :Raytheon anticipates international boom in counterdrone sales

    September 12, 2019 | International, Aerospace

    DSEI :Raytheon anticipates international boom in counterdrone sales

    By: Jen Judson LONDON — Raytheon is expecting a boom in international sales of its counter-UAS system already battle-tested with the U.S. Army. The Howler system — which includes a Ku-band Radio Frequency Sensor, a command-and-control system, and a Coyote unmanned aircraft system designed to take out enemy drones — could soon see an abundance of buyers. The system could also include a high-energy laser defeat solution and a high-powered microwave capability to provide a non-kinetic approach to knocking drone threats out of commission, according to company officials. “We have experienced quite a bit of interest from our international friends, partners and allies,” James McGovern, Raytheon vice president of mission systems and sensors in its Integrated Defense Systems business, told Defense News during an interview at DSEI, a defense exposition in London. “It's exploded: Counter-unmanned aircraft systems is the in-vogue discussion on weapon space and solution set at every trade show we've been to. It's a nonstop revolving door of interested customers in our solutions,” he added. Over the past five years, drone threats — cheap, commercial off-the-shelf ones — have proliferated in use, posing a threat on the battlefield as well as to airports, sports stadiums, government buildings and urban areas. Raytheon is preparing to reach initial operational capability with the U.S. Army of its Block II version of the Coyote, which is a variant that makes the Block more missile-like in appearance. The IOC goal is for the first quarter of 2020, according to Pete Mangelsdorf, director of the Coyote and rapid development programs within the land warfare systems portfolio at Raytheon. In the meantime, the company is soon expecting congressional notification for a sale to its first foreign country, Mangelsdorf said. Raytheon expects to see roughly 15 more countries issue letters of request for the Howler system to include Block II Coyote rapidly following the first congressional notification, he added. The company has license to separately sell sensors abroad, but generally customers are not just interested in what the sense-and-detect capability sensors would bring but rather want full-up systems that include all of Howler's elements. Raytheon's Howler system is flexible and can integrate into other systems and platforms, according to McGovern. Currently, Howler is used on a U.S. Army truck, but it could be integrated onto a pallet at a fixed site or mounted on a different truck depending on customer needs, McGovern said. The U.S. Army system was developed in response to a joint urgent operational need statement to find something that could counter drones as well as rockets, artillery and mortars, according to Mangelsdorf. The Block I version of the Coyote, which looks more like a plane or loitering munition than a missile, was the interim capability in response to the joint urgent operational need statement. The system is scalable in size. McGovern said Raytheon has used its gallium nitride technology to scale down the radar array while maintaining range and detection sensitivity, and while fitting it onto a smaller vehicle in the event a customer has a need for increased mobility, like in the case of special operations forces. The radar has the ability to see singular drones and identify drone swarms with high fidelity. Other radars might just pick up a drone swarm as one big blob, McGovern noted. https://www.defensenews.com/digital-show-dailies/dsei/2019/09/11/raytheon-anticipates-international-boom-in-counter-drone-sales

All news