Back to news

June 22, 2020 | International, C4ISR

Artificial intelligence systems need ‘checks and balances’ throughout development

The Pentagon's primary artificial intelligence hub is already studying how to aim a laser at the correct spot on an enemy vehicle, pinpointing which area to target to inflict the most damage, and identifying the most important messages headed to commanders, officials said June 16.

But as part of that work, the Department of Defense needs to carefully implement checks and balances into the development process, experts urged June 16.

“Fundamentally I would say there's a requirement ... that there's going to be a mixture of measures taken to ensure the governability of the system from the first stage of the design of the system all the way up through the operations of the system in a combat scenario,” said Greg Allen, the Joint Artificial Intelligence Center's chief of strategy and communications at the Joint Artificial Intelligence Center, at the Defense One Tech Summit June 16.

The JAIC is working on several lethality projects through its new joint warfighting initiative, boosted by a new contract award to Booz Allen potentially worth $800 million.

“With this new contract vehicle, we have the potential to do even more this next year than we did in the past,” Allen said.

Meanwhile, the Army's Artificial Intelligence Task Force is working on an advanced threat recognition project. DARPA is exploring complementing AI systems that would identify available combat support assets and quickly plan their route to the area.

Throughout all of the development work, experts from the military and from academia stressed that human involvement and experimentation was critical to ensuring that artificial intelligence assets are trustworthy. The department has released a document of five artificial intelligence ethical principles, but the challenge remains implementing those principles into projects across a department with disparate services working on separate artificial intelligence projects.

“We want safe, reliable and robust systems deployed to our warfighters,” said Heather Roff, senior research analyst at the Johns Hopkins Applied Physics Lab. “We want to be able to trust those systems. We want to have some sort of measure of predictability even if those systems act unpredictably.”

Brig. Gen. Matt Easley, director of the artificial intelligence task force at Army Futures Command, said the service is grappling with those exact challenges, trying to understand how the service can insert “checks and balances” as it trains systems and soldiers.

Easley added that the unmanned systems under development by the Army will have to be adaptable to different environments, such as an urban or desert scenarios. In order to ensure that the systems and soldiers are ready for those scenarios, the Army has to complete a series of tests, just like the autonomous vehicle industry.

“We don't think these systems are going to be 100 percent capable right out of the box,” Easley said on the webinar. “If you look at a lot of the evolution of the self-driving cars throughout our society today, they're doing a lot of experimentation. They're doing lots of testing, lots of learning every day. We in the Army have to learn how to go from doing one to two to three vehicle experiments to have many experiments going on every day across all our camp posts and stations.”

Increasingly autonomous systems also mean that there needs to a cultural shift in among all levels of military personnel who will need to better understand how artificial intelligence is used. Roff said that operators, commanders and judge advocate generals will need to better understand how systems are supposed “to ensure that the human responsibility and governability is there.”

“We need to make sure that we have training, tactics, procedures, as well as policies, ensuring where we know the human decision maker is,” Roff said.

https://www.c4isrnet.com/it-networks/2020/06/18/artificial-intelligence-systems-need-checks-and-balances-throughout-development/

On the same subject

  • The report card is in for the US defense industry’s health

    February 3, 2021 | International, Naval

    The report card is in for the US defense industry’s health

    By: Joe Gould WASHINGTON ― The health of America's defense industrial base ranks a middling “C” due to growing cyber vulnerabilities, a poor ability to surge production in a crisis, and political obstacles for defense budgeting, according to a lead defense trade group's new study. According to the National Defense Industrial Association's second annual “Vital Signs” report, the defense industrial base entered the COVID-19 pandemic in a weakened state despite healthier marks for competition, profitability and demand. The report, released Tuesday, graded eight performance areas from 0 to 100 for an average score of 74 for the year 2020. Government data firm Govini co-produced the data-driven report, which used the Pentagon's 2018 assessment of the defense industrial base as a jumping off point. “The defense industrial base is facing multiple headwinds: industrial security threats, flat future defense budgets, a shortage of skilled, cleared workers and decreased investments in the sciences that fuel U.S. innovation,” NDIA's chief executive, Hawk Carlisle, said in a statement. “Add to these the increased regulatory burdens and barriers for new entrants, which continue to shrink both the number of companies that participate in the DIB and the number of new companies entering the defense marketplace.” He added that the report, which makes no recommendations, ought to drive discussions as the Pentagon ramps up for competition with China and Russia. Though the study predates both COVID-19 and the revelation that elite cyber spies have spent months secretly exploiting SolarWinds' software to peer into computer networks, it raised alarms over industrial security. As data breaches and cybersecurity vulnerabilities both surged, industrial security overall showed “clear and continued deterioration,” ranking the lowest of all with a 56. The industrial base's ability to meet surge demand during a crisis received a failing grade of 66. Companies NDIA surveyed said that in the first 30 days, the industrial base could ramp up quickly but the rate of progress would slow soon thereafter; more than a 100 percent increase would take 180 days. More than half of firms said the availability of skilled labor would be a factor in increasing defense production, and 16.5 percent said a gap in U.S.-based human capital was the most vulnerable part of their supply chain ― in part fueled by a security clearance backlog. The size of the defense industry workforce fell to about 1.1 million people from its mid-1980s peak of 3.2 million, the report said. An analysis of public opinion, congressional action and regulatory action downgraded the “political and regulatory” outlook by 10 points since 2018 ― even prior to the pandemic and a related emphasis on domestic spending. A key factor was a new Cybersecurity Maturity Model Certification framework and its “additional regulatory burdens for all defense contractors,” the report said. Also, public opinion in favor of defense spending saw its largest decline since the Reagan-era defense buildup of the 1980s: A Gallup poll found that 17 percent of respondents felt the United States is spending too little on national defense and military purposes, down from 25 percent in 2019 and 33 percent in 2018. Industry can be pleased the “Demand” category jumped 16 points, corresponding with an increase in contract obligations issued by the Department of Defense. DoD contract obligations grew from $329 billion in fiscal 2017 to $394 billion in fiscal 2020, marking about a 20 percent increase. Foreign military sales also grew by nearly 20 percent over the same time period. Among all categories, major defense platforms ― aircraft, naval vessels and land vehicles ― were awarded the largest share of total contract obligation value, but contract obligation value for electronics and communication services grew 89 percent, leading all service categories. https://www.defensenews.com/2021/02/02/c-grade-for-us-defense-industrys-health-warns-trade-group-report

  • Brazil's Embraer to build NATO-approved aircraft in Portugal

    April 24, 2023 | International, Aerospace

    Brazil's Embraer to build NATO-approved aircraft in Portugal

    Brazil's Defence Minister Jose Mucio said on Friday his country's planemaker Embraer SA would manufacture aircraft that meet NATO's requirements in a partnership with Portuguese aerospace company OGMA.

  • ANALYSIS | The political tides of war are shifting — and may be taking Ukraine with them | CBC News

    February 10, 2024 | International, Land

    ANALYSIS | The political tides of war are shifting — and may be taking Ukraine with them | CBC News

    We have been warned — over and over again lately — that Russia’s invasion of Ukraine is an inflection point in history. And while the soaring political rhetoric may have sounded good over the last few years, it's very likely that this week, we truly arrived at that turning point.

All news