I recently participated in an investigation that serves as a cautionary tale for organizations that (knowingly or unknowingly) fail to keep log data.
The incident involved a major media company that was already a Pivot Point Security client. They contacted us in a panic saying they were victims of a cyber attack. They were desperate to find out what happened and who did what and so on.
It turned out that the IT department had been working on “performance issues” with their Exchange e-mail server with Microsoft for about a week. It was Microsoft that, after being in the server all that time and doing all kinds of testing and tuning, had finally suggested that the problem could be the result of malicious activity. Prior to that, all eyes had been on performance and none on security.
As a first step, we checked OSCAR, our security event management and log management platform, which was already operating in the client’s environment. OSCAR automatically detects anomalous patterns in security logs. At a glance we found nothing unusual, so we started digging for additional data that would hopefully tell us how the hackers got in, and how the incident had evolved.
Naturally we wanted to look at the firewall logs that related to communications inbound and outbound to and from the machine in question. To our dismay, we found that the client’s firewall wasn’t configured to capture data on traffic it allowed into or out from the mail server. So there was no record of what might have occurred.
Next we asked the client for a copy of the logs from the Exchange server itself. We learned that the server was configured to keep only seven days worth of logs. They weren’t copied anywhere and there were no backup snapshots; the logs just stayed on the server and were overwritten every seven days. At this point the incident had happened eight or nine days previously, so all the data we could have used to analyze the situation had been overwritten—there was nothing anywhere to show us what might have happened.
Our client had a hunch that the attack might have been initiated from inside their firewall, and they shipped us a desktop system that they had shut down. But once again, the shutdown took place over a week after the incident had occurred. No logs…
We had to have a difficult conversation with this client, explaining that they hadn’t kept sufficient data for us (or anyone) to determine exactly what had happened. Our recommendations were as follows:
- Anytime you think you have a performance problem with no obvious solution, or you encounter any “problem of unknown origin,” take a snapshot of the relevant logs or export a copy of them as an initial step in your troubleshooting. And keep the log data until the incident is fully resolved.
- Turn on comprehensive logging at your firewall. To analyze security incidents it’s vital to capture data on both inbound and outbound traffic, both accepted and rejected, from every critical system on your firewall.
- Store server logs for a longer period of time. (In this case, they decided to retain the logs for two weeks instead of just one week.)
Unfortunately for this client, all they know for sure about this incident is that they don’t know what happened. Was it a security breach? Was sensitive data compromised? Was this just an accident caused by Microsoft, or possibly a server operator? Most likely they will never find out.
It’s surprising how many companies find themselves in this position. Often it’s because IT lacked the security know-how or funding to put basic safeguards in place. In other cases it’s because the business or the legal department didn’t want IT to capture the log data in the first place.
If you think what you don’t know won’t hurt you (and so you won’t have to act on it), think again. The potential cost and risk associated with not collecting basic log information is large and growing, and it’s only a question of when, not if, you’ll be attacked.
The post When Troubleshooting A Performance Incident Suddenly Becomes A Security Incident appeared first on Pivot Point Security.