In 2015, Chris Lynch, a veteran of the Seattle tech scene, came to the Pentagon as the founding director of the Defense Digital Service. In the four years since, he guided DDS from a fringe unit to an organization embraced by the services, while maintaining a culture unlike any within the Pentagon. His “Rebel Alliance” is at the center of department efforts to move to commercial cloud, modernizing the ground control units for GPS and digitizing personnel records.
Now, after years of considering leaving, Lynch will be replaced by Brett Goldstein, a former special adviser to the Navy. The outgoing director explained what makes him angry, what he tells new hires and what led to the spread of DevOps. He spoke with Aaron Mehta from C4ISRNET sister publication Defense News.
C4ISRNET: DDS is for the first time not going to be about you. Are you worried about the culture?
CHRIS LYNCH: No. One of the really neat things about DDS is that early on I brought in this little LEGO Star Wars photo that hangs on the wall, right next to the poster that says, “Get Sh*t Done.”
I’ll go away on a trip, and I’ll come back and there’s something new. The chief of the Air Force brought down a Rebel Alliance flag for us. We’ve gotten things from people all over the building and all over the world.
The greatest organizations in the world don’t have to tell people their culture by putting it down on a piece of paper and explaining it on the day that they show up. The best cultures in the world are ones where you bring people in, you recruit, you weed out the ones that are all wrong, and you put people into a place in which it is obvious what it means, not because you told them but because they saw it and were part of it. Not by a series of instruction, or values written on walls ... none of that. It is all very much built around this idea that it attracts a type of person, brings in a type of person, and will make a certain type of individual very successful here.
C4ISRNET: What’s the thing that you’re most proud of?
LYNCH: Oh jeez. At first, I would have been like “Oh wow, we did the first ever federal bug bounty; oh wow, we named a program Hack the Pentagon; oh wow, we named something Joint Enterprise Defense Infrastructure; oh wow, we’re bringing commercial cloud to the Department of Defense.” There are so many things.
But you know what my favorite part about it is? That I actually don’t have a favorite project. Because of how we built out all the criteria of how we do projects they all are ridiculously important. By the time that we actually start working on projects, everybody’s bought in.
A project that is not incredibly impactful will not gain traction here. We’ve built out a culture that organizes itself around killing really bad project ideas, and that’s really amazing. We’ve built out a thing that literally the criteria are around changing and saving lives. They’re all really good, they’re all exceptional. Somebody is going to send us an email and they’ll say, “Wow, this mattered.”
I’m incredibly proud because we never worked on somebody’s stupid dashboard product. We didn’t create PowerPoint presentations. We did the work.
C4ISRNET: What are areas where progress isn’t evident yet?
LYNCH: Years ago, we did a project called [GPS Next Generation Operational Control System], ground control software for next-generation GPS. We went in, worked with the Air Force and one of the vendors on this. The phrase that we were using then was “DevOps.”
The program was not going well. We worked with them on automation, all this type of stuff, and using commercial cloud in order to do deployments. Then all of a sudden, the Air Force was like, “Wow, DevOps is amazing, we’re going to do DevOps on everything.”
Then, all of a sudden, they were like, “What do you think of this other thing?” We looked at a program like [air operations center software], which was also not doing well. Then they worked with some people in the Air Force, with the Defense Innovation Board, and all of a sudden you have something born out of that called Kessel Run. I hear DevOps all the time; I think it was incredible.
C4ISRNET: What’s the biggest frustration you ran into?
LYNCH: At some point in this job, you’re going to think to yourself this seemingly innocuous thing should not be this hard. Those add up, and there are a lot of them. Sometimes it’s over such small things that you wonder how it happens without the super powers that Defense Digital Services manages to acquire over time.
That is hard and has a deep emotional weight to it because you realize that we get to do a lot of things that a lot of others in this department do not get to do. Not because they’re not important, but because they’re not the usual way of doing things. The greatest frustrations that we run into are where it is very clear what the right outcome is.
But either the incentive structures put in place in the building drive incorrect behavior despite saying the right words around it or bad decisions where non-technical individuals have made highly technical decisions at the cost of mission incorrectly has taken a toll. Those things frustrate me to no end.
C4ISRNET: Can you give you a specific example of the latter?
LYNCH: Every single one of our projects we run into basically. One of the very first things I worked on were called treatment records. That system was using a product not intended to be used as a database and they used it as the database for some strange acquisition requirement reason.
Doctors could scan medical documents into a file format that would never be supported over at the Department of Veteran Affairs and had no idea the consequence of their actions. Stuff like that is incredibly hard to see because sometimes the fix is weeks or days. Sometimes the consequences of those decisions have been in place for years.
I’ll tell stories about some of these things and I will actually get angry. There’s a lot to be done. If you want to know why I spent so much time talking about the need for incredibly technical individuals to show up in the mission of national defense and sit at the table to help drive those decisions … if you want to know the reason that I care so much about that, it’s because the things that we do here literally matter. They’re going to change or save a life.
C4ISRNET: How has thinking in the Defense Department changed from when you started?
LYNCH: There is a colonel in the Army who once said to me, “I work for an institution that is several hundred years old and that had a requirement to carry a heavy rucksack and fire a rifle and I do neither of those things.” We are exceptional at so many things here and we have the greatest learnings in history of what that means. But there are fundamental changes that are not an option to ignore.
At one point if you wanted to work on really cool technology you worked in some form on things for the mission of national defense. You wanted to work on encryption? You were at NSA. None of those things are true anymore. It used to be that geographic boundaries and distance were huge advantages; it’s not necessarily true anymore.
It used to be that it was very hard to have a broad view of very large areas and see things that are happening; those are also not true. The world is changing. All of that is driven by this flawless execution of technology and so the Department of Defense must grapple with that fundamental shift.
Is it better than when I showed up? We’re saying more of the right things; I’m seeing openings and opportunities. You see things, like GPS OCX, and then eventually you will see things like Kessel Run will show up; you’ll see some iterations of how we look at DIU and DIUX.
C4ISRNET: But …
LYNCH: We still have an incredibly difficult time attracting top-tier talent that is highly technical to come here. Defense Digital Service is really good at that, but I don’t see that across the board.
We struggle with telling people that there’s a place for them in that mission and they should show up and work on that alongside with us. Even if you don’t want to work on weapons, even if you don’t want to work on the wartime effort that we have, even if you just want to work on making people’s lives here better, there is a place for you. We have a hard time saying that.
Because we have adversaries that are really good at software, they don’t need us. It’s easy for us to look back at history and judge our adversaries by the timelines and evolution of our own growth in certain areas. I don’t think that that’s a good way to do things.
C4ISRNET: How do you create a situation where a four-star general doesn’t just talk cyber because cyber gets money in the budget right now, but talks cyber because he understands it and he wants to really go at it?
LYNCH: That might be a bridge too far. We have a group of generals who are coming up who get it. We’re trying to show them that we can also work side-by-side with them on their mission and have an order of magnitude impact.
Sometimes we have to lean on the people who are going to be very open to different ways of doing things and aren’t afraid to take some risks. Either way we’re going to grapple with it. Groups like ours are important because we punch above our weight limit and we get to work with all those people, we get to advise, we get to work side-by-side, we get to actually help.
C4ISRNET: You’ve seen the comments in the last year about Google. As somebody who comes from that outside tech world and now has been immersed in this thing for a couple of years, what’s your view?
LYNCH: There is a vocal minority of people in the United States right now in the tech world that do not want to work with the Department of Defense. I would say to all those people it is better to be at the table and shaping the discussion and the decisions that are being made. If enough people don’t show up, we are ceding those to non-technical people who are then going to turn to other non-technical people, and they are going to make terrible products that don’t give us any outcome that anybody wants.