Keeping tabs on workers in contact with the federal government is high priority in an era that is not only post-Edward Snowden, but post-Fort Hood, post-Navy Yard and generally post-general-sense-of-security. The government's struggles to maintain thorough backgrounds of its workers are well-documented, but now officials hope the use of big data will fix many of the problems.

After the September 2013 shooting at the Navy Yard, the Defense Department directed the establishment of the DoD Insider Threat Management Analysis Center to try to close some of the gaps that allowed a credentialed military contractor into the Navy Yard facilities – and that also made it possible for other security breaches by cleared personnel.

Part of that reconciliation will hinge on the use of continuous evaluation, an IT-powered version of the background checks regularly performed on military personnel. A pilot program employing continuous evaluation currently monitors roughly 250,000 personnel and will scale to 500,000 by the end of the year, according to Carrie Wibben, director of the security policy operations directorate in DoD's Office of the Under Secretary of Defense (Intelligence).

Wibben spoke as part of a panel held by Defense One in Arlington, Virginia, on July 16.

"We've spent a lot of time and energy establishing a centralized DoD element to gather and fuse relevant information from different data sources within the department," Wibben said. "And I imagine continuous evaluation will be one of those key data sources…in producing information on our workforce that will give us a better understanding and more holistic picture of the people working for and on behalf of the department."

Continuous evaluation is a critical mission of the DITMAC. The center also relies on the service components to provide their own data they've collected from capabilities that currently are in development, the panelists said. The undertaking additionally relies on the vast troves of publicly available data online – including social media and other sources of public information.

"The mission is to enable information-sharing, collaboration, analysis and risk management across the DoD components to address current and emerging threats to DoD personnel and missions, including their information," said Mark Nehmer, deputy chief of implementation at DITMAC. "Our job is to actually fuse the information we can find with the information they've already got at the component level, query inside and outside the department for additional information, bundle that and hand it back to the component for action as fast as possible. We're an information exchange and enabling function."

DITMAC's mission and the Pentagon's focus on maintaining up-to-date personnel background information are at the crosshairs of ongoing debates: What constitutes an insider threat versus a whistleblower, or what data is legal – or ethical – for the government to collect and keep?

"It's not the technical issues anymore," said Patricia Larsen, co-director of the National Insider Threat Task Force at the Office of the Director of National Intelligence. "It's the legal questions, the non-technical constrains that are really the problem. As soon as you try to build a national-level program, you stumble across the different legal and civil liberties interpretations."

But Wibben emphasized it's less about personal information being collected and used by the government, and more about fixing the problems – including the failure to identify and flag certain behaviors and patterns – that led to the Navy Yard, Fort Hood and Edward Snowden.

"There are gaps in our periodic re-investigations where we only look at people once every five to 10 years. A lot can happen in five to 10 years, so those are gaps where we as a federal government can no longer assume that risk," Wibben said. "It's much easier to do this type of work, especially on the predictive analytics side, if you just de-identify the data. If you truly take the identification indicators out of it and look at just the data itself, the trends and patterns, and then only re-associate if you see something problematic, it makes things a lot easier. Those are things that we're looking at from a capabilities standpoint."