August 21, 2019 | International, C4ISR
4 big problems the intelligence community faces moving to a new data system
By: Nathan Strout The Defense Intelligence Agency wants to move quickly in developing the Machine-assisted Analytic Rapid-repository System, but the massive project, which will transform how the intelligence community uses data, faces some hard problems. The Defense Intelligence Agency is responsible for informing war fighters and policy makers on the military capabilities of foreign nations. Currently, the agency relies on the Modernized Integrated Database to house foundational military intelligence, but the 20-year old database wasn't built for the 21st century data landscape. The DIA wants to replace MIDB with MARS, a comprehensive, adaptable, scalable and rigorous data environment. With more data that is better labeled and organized, MARS will allow analysts to use applications to sort and process that data to make connections they couldn't otherwise. Last year, the DIA issued a broad agency announcement to solicit industry feedback from MARS. They've spent the year going over that information and are now preparing to actually build it. “2019 was about learning,” said Terry Busch, chief of DIA's integrated analysis and methodologies division. “This was the year of prepping to get started. In 2020 we get started ... We're going to move from designing to building very very quickly." Irving Townsend, also of the DIA, added that the agency was working to make some components of MARS available to the United States' closest allies in 2020 so leaders in those countries can begin looking at how they can use that data. In summer 2020, the MARS application programming interface will be released to the public, Busch said. But even as the DIA gears up to begin building MARS, the agency has four big problems left to solve. Resolving data inconsistency Perhaps the most pressing problem with creating a unified resource such as MARS is ensuring that the data fed into the system has been labeled and handled in a uniform way. That's easier said than done. According to Busch, there are 1,300 different data standards in the Department of Defense. “I work a lot with (the National Geospatial-Intelligence Agency) and DIA because we have a very similar path for our data,” said Busch. “The NGA has done some wonderful work with some of their data standardization and modeling (...) because NGA's been in the data making business for a long time.” Making the data collected by the various agencies and services interoperable is essential for the MARS enterprise. Data storage The amount of data collected by the intelligence community that MARS needs to encompass is staggering. Hosting images and videos, like those collected by the National Reconnaissance Office, for instance, takes up a lot of space. Because of this, MARS will not actually host all of the data itself. Instead, MARS will refer to intelligence hosted by other agencies. Instead, MARS needs to be able to index that intelligence. Theoretically, users will be able to click a link to access that data hosted on other servers. The DIA will have to figure out how this solution for MARS to operate effectively. The black box problem In developing a massive dataset of intelligence, the DIA wants all intelligence to be explainable, meaning that analysts need to be able to see how the intelligence was arrived at. In other words, can analysts and systems show their work? “It's really, really important to understand that we're not going to accept a black box,” said Townsend. “Our analysts are not going to accept that.” This is a problem for intelligence agencies and contractors who don't want to reveal their methods or proprietary information. Townsend noted that companies are going to have to figure out how to explain their intelligence without giving away that information. Legacy systems Another problem with moving to MARS? Many legacy systems will not be able to move to the MARS framework. While MARS will incorporate all of the MIDB data, some legacy systems will not be able to use the new system and will instead remain reliant on MIDB. “The transition is difficult. Many of our war fighters are impinged by legacy technology,” said Busch. “There is not turning MIDB off. Not for the foreseeable future.” Busch noted that while Congress may not like funding both programs simultaneously, it is necessary until those legacy systems can be replaced or upgraded. The DIA will discuss these problems at a MARS industry day Sept. 10, where they are expected to explain what the agency wants from industry. https://www.c4isrnet.com/intel-geoint/2019/08/19/4-big-problems-the-intelligence-community-faces-moving-to-a-new-data-system/