Too often, agencies are erasing key forensic evidence after a cyber attack.
The Homeland Security Department is trying to change the approach of system administrators to preserve key data.
Ann Barron-DiCamillo, the director of the U.S. Computer Emergency Response Team, said there are some steps agency chief information security officers, system administrators and others should keep in mind when they realize they’ve been hacked.
As federal chief information officer Tony Scott has said many times, there are two kinds of organizations: Those who have been hacked and know it. And those who have been hacked and don’t know it.
So with that rule to live by, US-CERT offers these best practices:
Hacked organizations shouldn’t automatically initiate reactive measures to the network without first consulting incident response experts. Barron-DiCamillo said US-CERT and a host of other companies do incident responses for a living as opposed systems administrators or other IT experts who respond to cyber problems only when they happen.
“This can cause loss of volatile data such as memory and other host-based artifacts. We also see them touching adversary infrastructure. It seems unusual, but we do,” she said. “They are pinging or doing name server (NS) look up, browsing to certain sites. Agency staff is trying to investigate the incident, naturally, and they want to conduct the analysis on suspicious domains or IPs. However, these actions can tip off the adversaries that they have been detected. Again, a no-no. You don’t want to do that.”
Barron-DiCamillo said preemptive blocking of an adversary’s website or IP address also is discouraged because, again, it will tip-off the hackers that they have been found. She said this could cause the bad actors to change infrastructure, and incident responders could lose the limited visibility they have on the hackers.
Barron-DiCamillo said too often system administrators change the network and system passwords before US-CERT or a vendor can fully understand what “hooks” the hackers have into the network.
“The adversary, again, sees this and will change tactics and you’ve lost the visibility you’ve had as an investigator,” Barron-DiCamillo said after a panel at the NextGov Prime 2015 conference in Washington.
Don’t erase audit logs
Agencies commonly write-over logs to preserve storage space, that’s an industry best practice. But when a cyber breach occurs, Barron-DiCamillo said without those audit logs, investigators can’t look at key forensic because the logs don’t go back far enough.
“We are trying to evangelize to our community so they understand to contact incident response teams at the very beginning of an event so we can ensure the forensic evidence that we need to confirm the kinds of activity that has happened is going to be there for them when they get there on-site,” she said.
US-CERT holds a weekly call with system administrators, CISOs and others to educate and discuss these trends. Barron-DiCamillo said it’s important for agencies to know they can call US-CERT whenever there is a concern, even if it’s a false alarm.
“I think there is an education part,” she said. “But the world has changed a bit so a lot of these system administrators never had the hat of doing cybersecurity work they have been doing and so we have to continue to educate them as they roll into new roles that they’re unfamiliar with and their common response as a help desk is to reimage the machine or the box so you have to educate them as they are rolling into new roles what they really need to do with these kinds of events.”
Another challenge is agencies have over the last 15 years gotten rid of the technical expertise and relied on government vendors to provide cybersecurity services.
But contractors are bound by the terms and conditions of the contract they are working under so the CISO or system administrator needs to understand and take control of the actions after the organization has discovered a breach.
“You want to ensure you have the flexibility in your contract to allow for incident response investigation, to allow the investigators to access the raw data even if it’s held at a third-party site. We saw that happen last year in some cases,” Barron-DiCamillo said. “So making sure you contractor language allows for that is something we’ve worked with entities from the General Services Administration and others over the past year to ensure those caveats are being added and modifications to existing contracts because they will follow it to the letter which sometimes can really hamper the investigation from a cyber perspective.”
Changes to data breach regulations
In 2007, the Office of Management and Budget issued a policy and framework to expand how agencies should respond to a cyber attack.
OMB was supposed to update data breach notification standards by the end of 2014. It didn’t release a new policy, at least publicly.
Barron-DiCamillo said other regulatory bodies, specifically around contracting, over the last two years have modified regulations to ensure the data necessary for an investigation is maintained.
So what should you do if discover a hack?
Barron-DiCamillo said US-CERT’s incident response teams focus on federal networks first and foremost, particularly those which involve advanced persistent threats (APTs).
“We are trying to work with our dot-gov clients and make sure we are there when we see adversary type activity that is associated with what we call focused operations or APT,” she said. “We are focused on those because those are typically the types of investigations where you will need a mature incident response team not only to ferret out the actor but understand how deep and how wide they have gotten inside your network.”
Barron-DiCamillo said there also are a lot of third-party vendors who provide incident response services.
US-CERT also provides “red team” and “blue team” services. These experts test network defenses and give agencies recommendations for improvement.
Barron-DiCamillo said the waiting list for these services is long so US-CERT is focusing on those agencies with high-value data assets or those with large amounts of data that are commonly targeted.