Open-source software is every company’s vulnerable backbone

Nearly every aspect of modern life is augmented or controlled by software systems, ensuring that human needs are fulfilled. This runs the gamut from the trivial to the life-threatening, but the code structures involved don’t all come from private corporations with full ownership and a lock on the source. JC Herz is a national expert on open-source software, the supply chain, and the culture of technology creation. Herz is also the CEO and founder of Ion Channel, and an adviser for government agencies, including DARPA.

ABERMAN: Well, I want to talk first of all about how you describe that the quadrant of software design, and how that really affects how we think about things like logistics and technology.

HERZ: So, I like to use an informal quadrant of, on one axis easy and difficult, and on the other axis, sexy and boring. So, most of Silicon Valley lives in easy and sexy, and it’s a revolutionary new way to order pizza. The real studs out there do sexy and difficult, like space, and self-driving cars. Easy and boring is most enterprise software, and then difficult and boring is where government lives. And that’s why it’s so hard to cover, and it’s so hard to discuss, because the minute you start trotting out NIST Standards, the eyes begin to glaze. It’s very complicated. It’s very difficult. It’s excruciatingly boring, and it’s hyper-consequential. And to some degree, these are the problems, now, of all enterprises, because it’s not just government agencies that are getting attacked a million times a day by state actors.

ABERMAN: Well, I think in some ways it’s boring, but I think of it as almost commodity-like, that a lot of the things that go into a software deployment now are little building blocks, Lego blocks, of what I would describe as open-source software.

Subscribe to the What’s Working in Washington podcast on iTunes.

HERZ: Right. And most people don’t realize that eighty percent of proprietary software, commercial vendor IT, is open-source. And so, large global corporations are building their ten to twenty percent special sauce on top of open-source components. And if they don’t keep it patched, and updated, you’re vulnerable. So, this creates huge supply chain vulnerabilities for everyone.

ABERMAN: And this is where the cultural aspects come to play. My understanding of open-source software, one of the big aspects of it is, somebody puts the code together a certain way, and says to the world, here it is, have at it, use it however you want, but no warranty, no representation it’s going to work. But here it is. And my understanding is that most of the products that we take advantage of, and use today, have many of these different open-source modules built into them that are freely available, making it easier to build software, but hard to make sure it works right.

HERZ: Yeah, and I think that we have to replace our frame, in a way, because people are still talking about commercial vs open-source, like back from when Steve Ballmer was in charge of Microsoft, calling Linux a cancer, everyone got that narrative. Microsoft just bought Github, which is the leading platform for open-source software development, so, things have changed. The other thing is that this distinction between commercial software, which, by the way, comes with no warranty, too, if you read the end user licensing, and open-source is that it forces people to look at all open-source like it’s the same, but there’s open-source which is maintained with huge corporate support, with hundreds of developers watching and fixing and evolving it constantly, and then there are single-committer components where it’s just one guy.

And to lump it all in starts to create some really dangerous myopia, because you’re calling apples, mangoes, oranges, and concrete bricks the same thing. It’s not even all fruit. So, you have to start to understand that supply chain. In Ion Channel, one of the things that we return in our bodies of evidence is, how many committers are developing and maintaining code for each of these components that you’re using? And if you have single-committer components, that’s probably something you want to think about migrating away from, because of the bus factor. That guy gets hit by a bus, there’s no one maintaining the code. Or it could be a bad guy, or it’s just poor code because no one’s reviewing it.

ABERMAN: And then there’s the other issue, the cybersecurity issue, that if you have a piece of code that isn’t being updated and patched, it’s the back door into a system.

HERZ: The most attractive target for malware is a widely-used component that is poorly maintained. So, it’s not version one of something, because no one’s using version one of something. It’s version 6.1 of something, but everyone is moved on, and so in our data platform, what we see is this kind of migration, wherein a programming language becomes unfashionable, and people abandon it for a new one. It’s almost like an ecosystem, the wildebeest are migrating, and you don’t want to go to that dry watering hole that is this programming language, because no one is maintaining components written in this language anymore.

ABERMAN: Or the software’s old enough to be on eight-inch floppy disks.

HERZ: Absolutely! And people don’t realize this when they procure software or abdot software. It’s almost like they want to treat it as a capital good, like this is a car, it’s great, I’m going to amortize it over five years. It’s cherry. It’s awesome. But they have to understand that software ages like milk, not like fine wine, and the minute you implement something, it’s already depreciating and degrading, and unless you constantly monitor for vulnerabilities against that, and automate that, because there’s no other way to do it with your staff, there’s going to be a gotcha there.

ABERMAN: What are the practical problems? Why doe a complex supply chain matter to a business or consumer?

HERZ: The more complex the supply chain is, the more places there are for it to go wrong. So, Heartbleed, that happened a few years ago, OpenSSL was a component that created a serious vulnerability, and it wasn’t just in the OpenSSL that people had, it was baked into Oracle. It was baked into IBM, it was hidden in everything. So, unless you’ve got an automated view, and a full roster, of what you’ve implemented, and you’re monitoring that, the more that supply chain expands, the more behind the curve you’re going to be, because you can’t do it manually with people anymore.

ABERMAN: And then there’s also the issue of just disaster. You gave me the example before you came on the air, of Ford, with the F150 truck.

HERZ: Right. So, there was one factory on the coast of Japan that made the pigment that renders the paint sparkly on a car, and after Fukushima, if you wanted a Ford F150 in tuxedo black, you were out of luck for several months. That’s a single point of failure, and that’s akin to what I talk about with these single-committer components. It’s a single point of failure, and if you start racking those up into an application that has 1300, 1400, 2000 software dependencies, and that’s not unusual, there’s just the odds that somewhere, something is going to go wrong. And if you don’t know that it’s gone wrong, you’re going to be vulnerable.

ABERMAN: JC, I’ll tell you, the big takeaway from me is that software is not fine wine. It’s milk. But more importantly, if we don’t take the time to look under the hood, our businesses are very much at risk.

HERZ: Yeah. And I think automation is the secret to that, because the volume of velocity of software and software change now, has made it impossible to even contemplate hiring the six or or twelve or twenty people you would need to do the job, if you could hire them.

ABERMAN: Thanks for taking the time in unpacking this issue, JC, we very much appreciate it.

HERZ: Thanks for having me.