Back to news

June 22, 2020 | International, C4ISR

Artificial intelligence systems need ‘checks and balances’ throughout development

The Pentagon's primary artificial intelligence hub is already studying how to aim a laser at the correct spot on an enemy vehicle, pinpointing which area to target to inflict the most damage, and identifying the most important messages headed to commanders, officials said June 16.

But as part of that work, the Department of Defense needs to carefully implement checks and balances into the development process, experts urged June 16.

“Fundamentally I would say there's a requirement ... that there's going to be a mixture of measures taken to ensure the governability of the system from the first stage of the design of the system all the way up through the operations of the system in a combat scenario,” said Greg Allen, the Joint Artificial Intelligence Center's chief of strategy and communications at the Joint Artificial Intelligence Center, at the Defense One Tech Summit June 16.

The JAIC is working on several lethality projects through its new joint warfighting initiative, boosted by a new contract award to Booz Allen potentially worth $800 million.

“With this new contract vehicle, we have the potential to do even more this next year than we did in the past,” Allen said.

Meanwhile, the Army's Artificial Intelligence Task Force is working on an advanced threat recognition project. DARPA is exploring complementing AI systems that would identify available combat support assets and quickly plan their route to the area.

Throughout all of the development work, experts from the military and from academia stressed that human involvement and experimentation was critical to ensuring that artificial intelligence assets are trustworthy. The department has released a document of five artificial intelligence ethical principles, but the challenge remains implementing those principles into projects across a department with disparate services working on separate artificial intelligence projects.

“We want safe, reliable and robust systems deployed to our warfighters,” said Heather Roff, senior research analyst at the Johns Hopkins Applied Physics Lab. “We want to be able to trust those systems. We want to have some sort of measure of predictability even if those systems act unpredictably.”

Brig. Gen. Matt Easley, director of the artificial intelligence task force at Army Futures Command, said the service is grappling with those exact challenges, trying to understand how the service can insert “checks and balances” as it trains systems and soldiers.

Easley added that the unmanned systems under development by the Army will have to be adaptable to different environments, such as an urban or desert scenarios. In order to ensure that the systems and soldiers are ready for those scenarios, the Army has to complete a series of tests, just like the autonomous vehicle industry.

“We don't think these systems are going to be 100 percent capable right out of the box,” Easley said on the webinar. “If you look at a lot of the evolution of the self-driving cars throughout our society today, they're doing a lot of experimentation. They're doing lots of testing, lots of learning every day. We in the Army have to learn how to go from doing one to two to three vehicle experiments to have many experiments going on every day across all our camp posts and stations.”

Increasingly autonomous systems also mean that there needs to a cultural shift in among all levels of military personnel who will need to better understand how artificial intelligence is used. Roff said that operators, commanders and judge advocate generals will need to better understand how systems are supposed “to ensure that the human responsibility and governability is there.”

“We need to make sure that we have training, tactics, procedures, as well as policies, ensuring where we know the human decision maker is,” Roff said.

https://www.c4isrnet.com/it-networks/2020/06/18/artificial-intelligence-systems-need-checks-and-balances-throughout-development/

On the same subject

  • U.S. Army signs agreement with BAE Systems for new M777 structures

    January 6, 2024 | International, Land

    U.S. Army signs agreement with BAE Systems for new M777 structures

    BAE Systems will work with its supply chain in the UK and the US to produce the major M777 titanium structures, which form the basis of the gun

  • Pentagon, Industry Struggle To Chart Impacts Of COVID On Arsenal

    April 22, 2020 | International, Naval

    Pentagon, Industry Struggle To Chart Impacts Of COVID On Arsenal

    "I think a three month slow-down in program activity is an optimistic projection based on the level of damage the economy is currently sustaining." By PAUL MCLEARYon April 21, 2020 at 3:33 PM WASHINGTON: Defense industry and Pentagon leaders are struggling to assess what impact the COVID-19 pandemic will have on building and maintaining the global US arsenal, but early assessments agree there'll be some disruption and delay as the global economy teeters. How much, how long, and where those disruptions will occur are something of an open question. With massive supply chains across the globe that run from small mom and pop manufacturing shops to massive global conglomerates, there's no one formula to assess what will happen to the industrial base in the weeks and months to come. On Monday, Pentagon acquisition chief Ellen Lord said she expects the largest programs to see three-month delays, but some analysts say that assessment could be too rosy. “I think a three month slow-down in program activity is an optimistic projection based on the level of damage the economy is currently sustaining,” said Andrew Hunter, former chief of staff to two heads of Pentagon acquisition. Hunter is now at the Center for Strategic and International Studies. The designation of the defense industry as critical infrastructure has ensured that shipyards, factories, and shops have generally remained open, as workers stagger shifts and companies provide liberal leave and teleworking where possible. That has placed the defense industry in a good position relative to other parts of the economy. But the supply chain those companies rely on “is tightly interlinked with the commercial economy, especially the aviation and automotive sectors, and this will transmit a degree of economic disruption into defense in the coming months,” Hunter said. Lord singled out the aerospace sector, along with shipyards and space launch as areas most at risk of slowdown and disruption. So far, though, the Navy and its largest shipbuilder say they're mostly working through the issues, and are staying away from putting a specific length of time on any delays. “We do not have a list of programs that are delayed, but as Ms. Lord and [Navy acquisition head James] Guerts have both said, while it's too soon to identify specific delays, generally we are expecting and planning for program disruptions,” Navy spokesman Capt. Danny Hernandez told me in an email. Geurts has moved to accelerate funding for some work and parts supplies so industry can eventually ramp up once the entire workforce is back on the job. Speaking to reporters last week, Geurts said over the “next three four weeks we'll get a better view of the exact delay and disruption and then how we might mitigate that, or quite frankly, where do we have opportunities where we can accelerate things” due to some excess capacity in the supply base because it's not being consumed by commercial aviation or shipbuilding. At the country's largest shipbuilder, Huntington Ingalls, company executives told me recently they're not seeing significant reductions in the number of parts they're receiving. “We are working with a few critical suppliers that are having challenges, but I think for the most part are going to be able to get through that,” said Lucas Hicks, vice president of new construction aircraft carrier programs. “We have reached out to all the suppliers and are working with them to try to help them.” There will be some pain, though how much is up for debate. Analyst Byron Callan of Capital Alpha Partners said in a note Monday that a three-month impact might not force companies to drastically change their expectations for the whole year. Any slippages this spring “could conceivably be made up in subsequent months, but that may be a challenge. Contractors could sustain prior guidance and just call out COVID-19 as a risk for the full year, or drop guidance altogether. No one has a perfect crystal ball.” Hawk Carlisle, president of the National Defense Industrial Association, said on a Monday conference call organized by the Jewish Institute for National Security of America that the slowdown in the global economy “is going to cause things to cost more, whether it's service agreements or products and manufacturing capability.” He's looking to the next congressional stimulus package for an acknowledgment “that these programs are going to exceed budget,” he added, “because of this two to three months of delays, partial workforce, paying the workforce, inability to perform on contract.” But Hunter thinks, so far, the Pentagon has handled the situation as well as can be expected: “To the department's credit, they have been aggressively looking for issues in the supply chain, which means that Ms. Lord's estimate becomes a lot more likely if they succeed in staying ahead of these problems.” https://breakingdefense.com/2020/04/pentagon-industry-struggle-to-chart-impacts-of-covid-on-arsenal

  • El Al must make Airbus, Boeing order decision soon, CEO says

    June 3, 2024 | International, Aerospace

    El Al must make Airbus, Boeing order decision soon, CEO says

All news