WASHINGTON — For a seven-member team inside the Pentagon, March 19 last year is known as the Thursday that never ended.
That day, COVID-19 cases in the U.S. had soared 40 percent in 24 hours, and the California governor issued the nation’s first stay-at-home order for residents. A few days earlier, the Department of Defense began sending employees home.
To prepare for the impending national emergency, the small IT staff sprinted to enable more than 1 million employees to work remotely at once, up from 95,000 on an average day before the pandemic.
The race to set up telework for the largest employer in the U.S. — and likely the world — included increasing circuits, upgrading infrastructure, expanding virtual private networks and drastically boosting bandwidth. But the department needed a collaboration platform so employees’ work could continue uninterrupted.
The scramble led to the Commercial Virtual Remote Environment, or CVR, a platform with Microsoft Teams collaboration tools and other Office 365 products that gave employees new telework tools such as chat, video conferencing and document sharing.
RELATED
In a matter of weeks, the DoD seemed to overcome all of its perennial IT challenges at once: speed, scaling and training. With CVR, the department quickly and incrementally deployed a new IT tool to meet the needs of disparate components across the globe.
The accelerated timeline was unprecedented, cutting the many months or even years often needed to adopt software to one month.
“I would argue that if we had brought in Teams and had done the normal research, testing, socialization, pilot rollouts … [it would’ve taken] the better part of a year,” DoD Chief Information Officer Dana Deasy said.
Officials shared in-depth details for the first time with C4ISRNET about what it took to roll out CVR, which boasts about 1.3 million users.
The story is one of a department overcoming typical IT contracting barriers to fast-track software development processes on a grandiose scale. DoD senior leaders and program managers who have pushed for quicker adoption herald the project as the ultimate example of the flexibility and speed capable at the department.
To hear Cloud Computing Program Office Director Sharon Woods tell it, people gave up full nights of sleep to deploy the software. Naps came intermittently throughout the day. Showers? Rare. Baseball caps covered unwashed hair. Coffee was king.
“Looking back on it, it’s almost a little surreal,” she said.
Inside the rollout
In early March, a telework task force created by Deasy’s office recognized in its daily meetings that the department needed to prepare in case the coronavirus spread widely. The issue was so important that the group included a who’s who collection of Pentagon IT leaders: all service chief information officers; senior staff from Deasy’s office; and representatives from the Joint Chiefs of Staff, Cyber Command, the National Security Agency and the Defense Information Systems Agency.
“A big part of that was, what does that mean in terms of the requirements for the workforce, if and when they need to telework in unprecedented numbers from what we’ve ever done in the past,” said Vice Adm. Nancy Norton, director of DISA, which serves as the Pentagon’s IT support agency.
DISA and the services rushed to prepare networks for increased cyber risks posed by working from home:
• Expand the virtual private network used for more secure access.
• Increase network bandwidth to handle remote traffic spikes.
• Upgrade IT infrastructure to handle more users online.
• Then, develop the employee collaboration platform.
“There was a point where we were having everybody coming after us with all kinds of requests to use just about every collaboration tool you can imagine,” Deasy said.
“I remember tasking a team literally to get together [during] a week and a weekend and sit down and come back and say: ‘Bring us the solution that, you know, we can contractually make happen quickly, will integrate well into our environments, that we know can scale to a million-plus, we know we can secure with U.S. Cyber Command.’”
IT leaders suddenly had to deliver a suite of tools to employees that differed from the ones with which they were familiar. Beyond scaling the collaboration platform to more than a million people, the next challenge became clear: training those employees to use it. Before that could happen, the IT team needed to understand the collaboration requirements of component after component.
All the top IT leaders pointed to the critical role of the meetings by the high-level coronavirus telework task force, in which participants gathered requirements for the collaboration platform — typically a prolonged process.
“Oftentimes you spend the first few years just trying to make sure that everybody clearly and completely agrees on the requirements,” Norton said. “Well, we didn’t do that for this at all.
We said: ‘OK, here are the basics for the requirements. Now let’s start building and let’s start delivering, and that’s that minimum viable product.’”
The CVR Environment on which the task force settled is a temporary platform accredited at cybersecurity impact level 2, which allows users to work with unclassified, noncritical mission information. The group decided early on to deliver the platform using a software development process that Deasy and other IT leadership have backed for years: DevSecOps, where a product is delivered with minimum capabilities, and improvements are added incrementally.
On March 17, the DoD CIO told the Cloud Computing Program Office that workers needed a collaboration platform.
“Less than 24 hours after first hearing about that, folks from my team contacted me saying: ‘Hey, we want to do this, we think we can do this, and we think we can roll it out in a matter of weeks,’” Woods said. “It’s hard to not have your initial reaction be: ‘You’re crazy. How are we supposed to do this?’”
Delivering software in record time
For the small Cloud Computing Program Office team, March 19 marked the start of a nearly nonstop push to build and roll out the collaboration platform. That workday stretched into days of around-the-clock activity.
“We worked 24 hours a day, seven days a week, and that’s not an exaggeration,” Woods said.
The minimum viable product that the CCPO introduced was a “bare bones” version of Teams, Woods said, with chat function and video conferencing. When emails introducing the software first landed in employee inboxes, a “bunch” of workers forwarded the notifications to the help desk thinking that they were phishing attacks, Woods said. One person forwarded it with the message “not today, ISIS,” referring to the militant Islamic State group.
The IT team brought the first test accounts online March 20, about 24 hours after the group started working on CVR. By March 25, the first pilot users were live. Those testers reached the highest levels of the Pentagon: flag officers, general officers and senior executives. Within a week, senior leaders cleared CVR for departmentwide use.
The department ramped up the number of users it added daily, starting with hundreds a night, growing to thousands, to ultimately adding 250,000 users in one night. As employees established more accounts, requests poured in for new capabilities, such as conference calls.
The team had a growing and flexible list of capabilities it wanted to add.
“The list got reprioritized based on the demand from the users,” Woods said.
The CVR delivery embodied the DevSecOps approach that IT officials preached for the last few years, with a continuous loop of user feedback. Every night, the CVR team combed through comments and tweaked the platform.
For example, Woods said, the CCPO team received feedback about information that would be helpful in the CVR welcome email.
“Every night, we would make small adjustments to that welcome email to provide greater clarity and understanding about what was happening,” Woods said. “Yes, it went out really quickly. There was maybe a degree of confusion amongst some of the users. But we took all of that feedback, rapidly reacted to it, and it resulted in a better product.”
As the CVR journey continued, the CCPO team added an automated self-service portal to respond to an influx of basic questions about user accounts, saving the department more than $10 million on help desk costs, according to Norton.
The delivery didn’t come without hiccups. Some agencies wanted to know when the CVR platform would reach their offices so leaders would have ample time to prepare their workforce. But because Woods’ team moved so fast, delivery sometimes came when agencies weren’t expecting it.
“Those are just some of the missteps [involved when] speed was so critical because of the urgency of the requirement,” Woods said. “And you do have these little missteps, and that’s OK. And I know that’s very uncomfortable, especially when you’re looking at the traditional way that DoD has often done IT development.
“It was so critical to why CVR was successful. Embracing some of that discomfort so that we didn’t lose speed, but that we could also react quickly to course-correct every day.”
Faster in the future?
The Cloud Computing Program Office continued to strengthen the collaboration platform throughout the pandemic spikes of the summer and fall, which kept DoD workers mostly at home. The success led the department to develop a permanent remote collaboration solution with stronger cybersecurity that will replace the first version this coming summer.
The unparalleled speed at which the department implemented CVR, along with several other projects to enable telework, raises a significant question: How does the accomplishment shift expectations for IT projects?
The experience established a playbook for faster deployment, officials said.
“CVR just obliterated the myth that it takes years for the department to deploy a capability at scale,” Woods said. “The big piece of the shift is you want speed. The question is: How do you get speed? And I think this is where authentic agile practices become so critical.”
However, it’s not feasible for department personnel and contractors to stay up all night for several days in a row to set up user accounts and deliver new tools outside of a national emergency like the COVID-19 outbreak.
But the work during the pandemic proved that the department can move faster on regular IT projects outside of all-out crisis mode if it successfully uses the DevSecOps concept.
“Anytime you’re able to clearly align all of the different enablers to understand what the focus is, then you don’t have slowdowns and barriers” caused by different offices like finance, contracting or sustainment professionals, Norton said. “When everybody’s on the same sheet of music, focused on delivery, then you can do this for any number of projects.”
Mark Valentine, general manager of Microsoft’s national security team, said developing CVR with the department left lasting lessons in the value of a close relationship between the DoD and a contractor, leading to openness about departmental challenges.
“There’s always risk in doing anything new,” he said. “And therefore I think that the trust that we had from the relationship and continued to build through the crisis was absolutely instrumental into making it a success.”
Proactive support from senior leadership was the cornerstone of the project. Executives removed identified barriers, making “hard choices” that balanced capability, security, mission and speed, cutting the timeline from months to weeks, Woods said.
“That was a really big deal, and not necessarily the way that you would expect leadership at the highest levels to engage, but they did,” Woods said. “It [the quick decision-making] let them comfort and support their communities. And then it really helped us make sure that it [the platform] was meeting the needs of the department.”
Woods pointed to the services’ desire to rapidly deploy a permanent Office 365 solution for remote work. Instead of releasing the capability as a “big bang,” IT teams are incrementally shipping out new tools as part of a minimum viable product, she explained.
“Everything that we did on the teleworking front shows that the DoD technologists could move quickly and show real creativity and out-of-box thinking like the private sector can do,” Deasy said.
From those early days in March, the uncertainty about the pandemic’s severity drove a stronger platform, a standard that should be the goal of future DoD projects, Deasy said.
“The way we built it is one word: enduring,” he said. “We didn’t stand up something that was fragile. … We always designed everything to be enduring, meaning that we didn’t know how long the pandemic would go on. But we knew that whatever we built, it had to be built in a way that could become a lasting asset for the department.”
As the pandemic still rages in the U.S., the department faces new uncertainties. Traditionally, political appointees such as Deasy, the department’s CIO, leave with a new administration. Norton’s tenure at DISA ends in a few months. Their roles in the pandemic response rewrote the expectations on future IT projects. But will the lessons they learned endure?
Andrew Eversden covers all things defense technology for C4ISRNET. He previously reported on federal IT and cybersecurity for Federal Times and Fifth Domain, and worked as a congressional reporting fellow for the Texas Tribune. He was also a Washington intern for the Durango Herald. Andrew is a graduate of American University.