Tracking down malware hosts identified by OpenDNS
I've always been a fan of OpenDNS. Even before they switched to being a security outfit, I used them for content filtering in the office and at home. If you're not familiar with their offering I'd recommend visiting their website and taking a look for yourself.
Earlier this year, a colleague overseas got in touch to let me know he'd put OpenDNS in at a few office locations based in other countries. The good news was that change had been seamless and that OpenDNS had instantly started to spot malicious DNS requests originating from both sites. The bad news was that he had no way or idea of working out which desktops or servers were making the bad requests as he had almost no visibility on those networks and no body on site able to help.
Having your public DNS alert you to malicious requests originating from an office or country is a bit like having a smoke alarm on the 40th floor. Sure, it lets you know you have a fire, but it can't tell you what floor its on, how bad the fire is, where it started or whether or not its spreading. For us, the 'identity' as defined by OpenDNS was the country.
These sites were remote, isolated and with little to no logging or monitoring in place. After various discussions the only logs we could lay our hands on were the gateway firewall logs. Not much use you might think in trying to find the infected hosts as they don't contain the DNS query contents.
After spending a bit of time observing, reviewing and testing how OpenDNS worked I found a way to identify the infected hosts from the sites firewall logs alone, not through looking at the DNS requests but through looking at the responses OpenDNS serves for malicious requests. I'm sharing it here because the scenario is common enough that this may come in use to someonelse, someday.
When OpenDNS receives a DNS request attempting to resolve a 'bad domain' it responds to the lookup request rather than ignoring it:
|Manual lookup against OpenDNS Vs. Google DNS|
OpenDNS does this in order to present the user with a holding page which matches the type of threat category OpenDNS classify that particular domain under:
|OpenDNS malware holding page|
Now, although the malware in question is unlikely to render such a warning page to the user, it will still probably go off and attempt to connect to the recently resolved domain to do whatever unspeakable evil it was programmed to do. This means that, although you might not have visibility to differentiate between good and bad DNS requests, you will probably have visibility of which of your desktops or servers are trying to connect to the OpenDNS holding IP addresses by looking at your firewall logs for any connection attempts to 18.104.22.168. This isn't OpenDNS' only 'repsonse' IP address, not by a long way, but you can always perform a manual lookup against OpenDNS of any domains they flag as evil using nslookup or similar (see above) and then search for connections to that 'response' IP in your logs.
So for me, with a bit of grep and awking through the logs I was able to work out which of my workstations were in trouble:
Also, whilst I was there I was able to check which systems were still using other DNS services and then request that they be changed: