There has been plenty of discussion in the federal community about the Office of Management and Budget’s 30-day cyber sprint and whether it made any difference or not.
Some experts say the cyber sprint was just window dressing on long-standing problems. Others pointed to finally forcing agencies to use their smart identity cards to log-on to their networks and computers, and that was, at least, the type of difference maker that had been missing over the last decade.
A new document obtained by Federal News Radio shows just how bad a shape agencies were in as of June, including how many critical vulnerabilities that existed for more than 30 days and how many potential holes in individual agency networks, and just how far they’ve come over the summer.
When OMB released results of the cyber sprint, federal Chief Information Officer Tony Scott highlighted governmentwide progress in using secure identity cards to log on to networks. But it’s what Scott didn’t talk about publicly that creates both hope and fear.
The document from June, when the sprint began, shows just how bad agencies were doing in closing critical vulnerabilities, including those open for more than 30 days. It also details indicators of compromise that agencies said they had and the number of privileged users who had access to the network.
At the request of an administration official citing national security concerns, Federal News Radio decided not to publish the chart or talk about specific agencies.
But generally speaking, the “before” picture was scary.
Agencies listed more than a dozen indicators of compromise, including one agency with a majority of them. An indicator of compromise means strong evidence exists the system or network has been compromised by hackers.
Agencies also said they had more than 50 critical vulnerabilities open for more than 30 days and more than 75 active critical vulnerabilities. Several agencies listed double-digit vulnerabilities open for more than 30 days and/or active problems.
Now 45 days later, a government official said the picture is much brighter, but far from perfect.
“The indicators of compromise were all false positives,” said the official, who spoke on condition of anonymity in order to address the sensitive data in the chart. “Some of that is the agencies are learning how to search for indicators the Homeland Security Department sends out, and some of that was on DHS for writing better descriptions of what to look for.”
The official said all agency-reported indicators of compromise were investigated either on site or by DHS reviewing the computer images agencies sent them.
The official said if an agency did have a compromise, it’s a big deal so there was a huge effort to figure out what exactly was going on.
Agencies are reporting new potential indicators regularly, and the official said they expect to have some false positives.
“If we don’t get false positives, we aren’t looking hard enough,” the official added.
Another federal cyber expert took a more pragmatic spin on the situation.
The official, who also spoke on condition of anonymity in order to address the sensitive nature of the data, said it’s definitely good news that the indicators of compromise were false positives.
“Your agency has to have a good cyber threat management arm to its cyber program. If you have a good one, they can identify indicators of compromise and validate them,” the official said. “But most agencies don’t have good threat management program to fully identify indicators of compromise. It’s very serious to have an indicator of compromise, especially if you don’t have a good threat management process.”
The official said agencies also need to have a strong relationship with DHS, the FBI and the intelligence community, which can validate the threats and help you remediate or protect data.
“If you are on your own, you will not get far,” the official said.
As for the critical vulnerabilities, the news is less cheery.
As DHS Secretary Jeh Jonhson said publicly, agencies patched or remediated about 60 percent of the critical vulnerabilities The official said that number continues to increase.
“The trick with these vulnerabilities is two-fold: they are constantly replenished. For example, we just had Windows 2003 server go out of support mode, so any agency with that server will have critical vulnerability where they didn’t have it 30 days ago,” the official said. “Our goal is to get to the point where few if any are around for more than 30 days. Really to have none around for 30 days. We know that will be a constant battle and demand a lot of attention, so the trick is to maintain focus for the next two-three-five years.”
The second challenges with critical vulnerabilities is agencies may not necessarily have updated the IP addresses that DHS scans regularly and there may be problems with the address, but the agency no longer had data on that system, the official said.
A third related challenge is some software vendors don’t update their version numbers when they push out a new, more secure application, and that too causes false positives.
The federal cyber expert said the progress against the critical vulnerabilities data was like adding a new shine to a beat-up car.
“There is no reason why DHS should be telling us about our critical vulnerabilities on our Internet facing systems,” the expert said. “It puts a huge spotlight on agencies that they can’t manage their resources. But when you step back and look at the bigger picture, most of those vulnerabilities were looking at Windows 2003 server vulnerability, that, yes, could be used as entrance vector. When you look at external systems, those are systems we know of that we can see, but there are a lot that we can’t see, called internal systems, and there are many, many more critical vulnerabilities in them.”
The expert said they know of one agency that had tens of thousands of critical vulnerabilities on internal facing systems.
“Internal systems are behind the firewall but most of the systems are on the same network. So if a spear phishing hack is successful, someone could get into a system, move laterally until they find a system that hosts sensitive data and has critical vulnerability and they hop on that system,” the second official said. “The hacker also can move system-to-system or application-to-application so they are using those vulnerabilities to steal data just the same.”
This is why OMB seems to be marching toward mandating or at least outlining the specifics of what a “defense in-depth strategy” would look like governmentwide.
This could include reassessing how quickly agencies can install two-factor authentication on applications or systems. OMB initially told agencies to meet the 75 percent goal during the sprint, but sources said they backed away and required employees to use two-factor to authenticate only to the desktop.
The security expert said while two-factor to the desktop helps, it’s not closing a huge gap that hackers commonly exploit.
“In today’s environment, the technique we see most often is to spear phish and if the hacker does it 100 times and 10 let him in, then two-factor at the desktop is a joke,” the security expert said. “The hacker just has to wait for the right person to let him in and then move laterally in environment because patches are missing on internal facing systems. The move to a defense in-depth concept means if you have a crack in one layer, then you can survive because you have the protection at other layers. But if you have crack at many levels, then you are in trouble.”
The first government official recognized there is plenty of more work that needs to be done.
“Government cybersecurity is not where we want it to be , but the sprint accomplished what it should and moved us forward rapidly,” the official said. “There are now far fewer privileged users who can log in with only a password and that is a significant achievement. We now have to turn the sprint into a marathon. We have done a burst of activity and that was valuable, and now we have to sustain our pace.”
This post is part of Jason Miller’s Inside the Reporter’s Notebook feature. Read more from this edition of Jason’s Notebook.