Walled Off: Handling DNS Poisoning At Scale

As part of the Adversary Sight engine that powers visibility within the watchTowr Platform - we collect troves of attack surface data about our clients for continued and continuous review - providing the foundations of our ability to identify vulnerabilities that impact an organisation at any level.

This includes actively and continuously enumerating our client's assets across the Internet to build comprehensive visibility of their external attack surface.

We map many different types of external assets - legacy assets like domains, subdomains and IP addresses - all the way through to external assets that are more reflective of the world we live in today - SaaS platforms, cloud environments, source code repositories and more.

When hunting for legacy assets like subdomains, we employ a myriad of techniques for discovery - aligned to the techniques we see real-world attackers use. One such technique that we employ to identify attractive assets on the Internet that belong to our clients is subdomain enumeration - a common and supposedly solved process - which can be achieved both passively and actively.

For this post, we will focus on subdomain bruteforcing - an active subdomain enumeration technique used to discover resolvable subdomains by appending keywords as a prefix to the target domain.

For example, given a domain of invalidcontoso.com, we want to discover potentially hidden domains like admin.invalidcontoso.com, secret.invalidcontoso.com, or backup.invalidcontoso.com.

One way we can bruteforce subdomains is by spamming a wordlist at a DNS resolver, and makes for easy results - if the DNS resolver gives us a DNS record in return as a result, we've found a valid subdomain. Some would suggest it's literal magic. Many would disagree.

I hear you say - this isn't new! Subdomain enumeration is a solved "challenge"!

However, when you look at large amounts of data as we do at scale at watchTowr, we see strange things.

Through the course of our activity, our Adversary Sight engine began to flag on a large number of strange looking enumerated subdomains that were sporadically responding to DNS A record queries. This aroused our suspicion. As always, we dived down the rabbit hole to work out why these seemingly random subdomains were supposedly valid - but definitely were not.

This behaviour was unpredictable - DNS A record queries made for the same subdomain resolved to various seemingly random IP addresses within a short time frame or not at all. These IP addresses, which we attributed to various companies via their ASNs, did not appear to be correlated.

Amidst our troubleshooting efforts, we noticed a common denominator - these DNS queries made against global public resolvers were, at some point, routed to authoritative nameservers based in mainland China. Going further, we were able to consistently replicate this behaviour across our immense asset visibility, and began to identify specific strings that, if part of the hostname requested for resolution, would attract poisoned results.

Strange things abound, we began to build a thesis, otherwise known as 'a hunch and a feeling backed up by experience', that these DNS queries were being intercepted and tampered with by China's DNS-based internet filtering apparatus – popularly known as 'The Great Firewall' (GFW).

You can see this for yourself (no matter your location) by querying a domain name which triggers this filtering apparatus. For example, we can see that the vipfashion.com domain is supported by authoritative nameservers based in mainland China:

$ dig vipfashion.com NS +short
dns1.hichina.com.
dns2.hichina.com.

If we attempt to resolve the (non-existent) subdomain kman.vipfashion.com, we can see that the domain resolves to four separate IP addresses belonging to Twitter, Dropbox, and Facebook.

$ dig -t A kman.vipfashion.com +short
192.133.77.191 [Twitter (AS13414)]

$ dig -t A kman.vipfashion.com +short
108.160.169.181 [Dropbox (AS19679)]

$ dig -t A kman.vipfashion.com +short
69.63.186.30 [Facebook (AS32934)]

$ dig -t A kman.vipfashion.com +short
108.160.165.62 [Dropbox (AS19679)]

While this example worked at the time of writing, the continuous nature of changes made to the GFW's configuration means you may see different results. Some experimentation may be required!

Gone unchecked, this would corrupt the dataset we keep of client assets, create difficulty in accurately attributing assets to clients, and even create legal challenges if we were probe systems unrelated to our actual clients. Clearly, this is unacceptable and a solution had to be found.

GFW and DNS Tampering

Different countries use varying forms of DNS injection to flag content they deem to be inappropriate, most commonly by redirecting users to a warning page or by responding with NXDOMAIN (indicating that the queried domain does not exist) when resolving unacceptable queries. This would cause, for example, a domain associated with pirated media to fail to resolve (or so we are told).

The GFW, however, employs a unique approach - it instead opts to inject incorrect IP addresses, even if these IP addresses have already been allocated to an existing, unrelated organisation (as illustrated in our example from earlier).

This phenomenon occurs when the resolution of an IP address must be made on an authoritative nameserver located behind the GFW. The GFW monitors each DNS query routed through mainland Chinese network infrastructure and injects a fake DNS reply with an incorrect IP address when a query containing an offending keyword is detected.

This behaviour is also bidirectional, although it is definitely more prevalent for resolutions made from within China rather than outside of it.

We theorise that this form of DNS tampering significantly impacts our clients with mainland China presence, as these organisations often utilise authoritative nameservers hosted from within mainland China.

To illustrate, we have outlined one example of how the GFW can tamper with DNS queries.

Possible Solutions

So what can we do about this behaviour? Well, the GFW's DNS tampering has been around for some time and has been discussed at length by the security community. Let's explore some of the common solutions proposed and evaluate them to see if they are appropriate for the purposes of maintaining the integrity of an organisation's discovered attack surface - a pivotal point of maintaining a live picture of an organisation's attack surface.

DNSSEC

The Domain Name System Security Extensions (DNSSEC) is a feature of DNS that authenticates responses to domain name lookups, which is designed to prevent attackers from manipulating or poisoning the responses to DNS requests.

On paper, this sounds like the perfect solution. However, to maintain a live picture of an organisation's attack surface, this is unfortunately not relevant - attackers do not have the luxury of implementing DNSSEC on behalf of their targets.

Additional complexities required to set up DNSSEC, such as Key-Signing Keys (KSK) and Zone-Signing Keys (ZSK), may also deter its adoption (nobody likes managing keys, ever - and don't pretend otherwise).

Even if DNSSEC is implemented and the resolver correctly rejects the injected response, the real response would never make it back to the resolver, resulting in a denial-of-service.

Perform Multiple Queries And Observe for Anomalies

One potential circumvention strategy proposed to weed out fake DNS responses is not to hastily accept the first response returned by the resolver, but instead to perform multiple queries for each unique DNS record and attempt to detect the behaviour of the GFW by analysing them for anomalies.

For example, if we were to bruteforce a subdomain for a client with a known bad keyword, and we were to receive responses for both Facebook and Dropbox, we could conclude that the response had been poisoned by the GFW, since a host with this configuration is an unlikely scenario.

While it might seem like this rather straightforward approach would work, having to perform multiple resolutions each time we want to resolve a domain would be computationally costly (we use DNS a lot!), which would potentially have a performance impact on other functional components. This does not seem to be a scalable solution.

The efficacy of this 'anomaly detection' would also be dependent on the analysis technique applied. While we outlined a simple approach above, other approaches (such as those based on observing a discrepancy between TTL values returned from each query) have been shown to be effective in select scenarios. However, as alluded to previously, the computational cost associated with each technique must be considered and evaluated to determine its viability.

Tl;dr single solutions that don't scale are not real solutions.

GFWatch to the rescue!

Fortunately, in our search for a better solution, we came across the work of the researchers at GFWatch.org, who have developed a system to continuously monitor the GFW's filtering policy.

Amongst other things, GFWatch aims to identify the domains censored by the GFW, as well as maintaining a list of the forged IP addresses that have been used to spoof DNS responses, complete with historical archives as well. Neat!

That's awesome and super useful! Using this, we can extrapolate the censored domains captured by this project to generate a list of banned keywords to avoid when enumerating subdomains.

With this list, we can assume - with a reasonably high degree of confidence - that a DNS record has been faked if a query contains an offending keyword and also resolves to a known forged IP address.

The assurance this list provides makes for a world of difference, as our data pipeline can now accurately attribute assets when processing subdomains with near-linear time complexity. Furthermore, this insight can be extended retroactively to historical data within our data warehouse if we suspect that the integrity of data has been tainted by the GFW.

Although invalid addresses still appear from time to time, the valuable data collected by this project has allowed us to quickly filter out 99% of false positives and in turn, allow us to focus our time on finding vulnerabilities in real assets that belong to our clients.

Limitations

Unfortunately, despite being able to leverage the amazing data collected by the GFWatch project for asset attribution, the strategy discussed is not a silver bullet. Tampered DNS replies still slip by occasionally as new keywords are added by GFW operators, or as new IP addresses are forged.

However, while it isn't possible to detect all DNS tampering, knowledge of how the GFW operates allows us to design a solution that can quickly react to these changes, mitigating the potential damage caused.

Conclusion

I hope this post has provided you with a glimpse of the challenges we face when mapping attack surfaces at scale - when you do enumeration at scale, you inevitably see weird things that turn supposedly simple actions into a little more complex actions.

I am especially grateful for the extensive academic articles shared by researchers around the world. The research documented within has allowed us to understand the capabilities and mannerisms of the GFW quite quickly, which in turn allowed us to find a solution swiftly.

If you'd like to learn more about how the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, can support your organisation, please get in touch.