Defense Department and intelligence community officials are capitalizing on tech tools that make sense of vast quantities of big data, including with a new big data hub and app marketplace at the CIA.

The CIA next month will start using Cloudera's Enterprise Data Hub platform, which will pool standardized data from intelligence analysts across agencies for use by intelligence community (IC) decision-makers, according to CIO Doug Wolfe. The open-source big data platform and marketplace will be hosted on the IC's Amazon Web Services C2S private cloud, Wolfe said Feb. 25 at the Cloudera Federal Forum in McLean, Va.

The enterprise data hub agreements are set to last no more than three years as agency officials throughout DoD and IC look to exploit fast-moving technology – and move more quickly themselves. The goal is to keep pace with emerging threats and avoid getting stuck in proprietary solutions, focusing instead on open-source technology.

"I am concerned on the analytic front that we get locked in to certain solutions – that we believe we should put all our data in one place and we believe we have to run all of our data against any given tool or any given solution," Wolfe said. "I've got to believe that two years from now there's going to be the next generation of Spark or Hadoop or whatever that is and we're going to say we really need access to that and we really need to leverage that to make our mission work."

At DoD, officials are hoping to emulate the IC's strides in the cloud.

"You can talk about big data, you can talk about all kinds of new technology…but at the end of the day what does that mean? It means change," said Dave Bennett, director of the Defense Information Systems Agency's implementation and sustainment center. "DoD is not a big change agent. We are slow, we are cautious. If you expect us to move with any sort of speed, I would say good luck with that."

Yet despite DoD's caution in implementing new technologies, services and processes, it's something that must be done, if for no other reason than to maintain network security, Bennett noted.

"The analytics and the tools give you the ability to parse through the information and be able to react to it in a timely fashion. If you're looking five minutes behind you in terms of what's going on, you're six minutes late. The reality is in the world we live in now, a cyber attack happens at the speed of electrons," Bennett said. "We have to get to the point where we are able to understand an issue, and react to it, before it's had a chance to take effect. We aren't there yet, and we've got to leverage the tools becoming increasingly available that give us the flexibility to do that."