DNS Shotgun is a bechmarking tool specifically developed for realistic performance testing of DNS resolvers. Its goal is to simulate real clients and their behaviour, including timing of queries and realistic connection management, which are areas where traditional tools are lacking.
Proper query timing is essential to simulate realistic cache-hit rate, while connection management becomes very important when benchmarking connection-oriented protocols. In this blog post we will focus on benchmarking DNS-over-TLS and DNS-over-HTTPS protocols. More information about UDP, TCP and other DNS Shotgun’s features and methodology can be found in the documentation.
Motivation
The rise in popularity of DNS-over-TLS (DoT) and DNS-over-HTTPS (DoH) presents us with new challenges. These connection-oriented protocols are built on top of TCP and TLS. The major difference from UDP is that they are no longer stateless. They require an established connection to exchange DNS messages. This introduces additional round-trips and performance overhead.
We wanted to measure this impact on both the clients and the servers, at scale. What happens when thousands of clients switch from UDP to DoT/DoH? How does the client behaviour and protocol choice affect their latency and server performance cost? How much do you need to scale your infrastructure to handle the increased load of DoT/DoH?
Connection Management
To answer these questions, we needed a tool that would be capable of simulating real clients as realistically as possible. Other existing DNS performance tools, such as dnsperf or flamethrower, establish a set number of connections at the start of the test and then send queries using these connections. Real clients, however, will keep establishing and closing their connections continuously.
The other issue with existing tools is the limited number of connections you can establish. DNS Shotgun has no such limit. It allows binding to multiple source IP addresses and using as many TCP connections as your hardware can handle. We’ve verified this can scale to at least hundreds of thousands of active established connections, illustrated by the following charts.
Features
DNS Shotgun supports UDP, TCP, DoT and DoH. Traffic can be replayed over IPv4 or IPv6. The toolchain consists of various scripts for data processing, replaying traffic to simulate real clients and visualizing the results with charts such as the ones above.
The tool can be used to replay traffic over a single protocol or to use multiple protocols simultaneously. For example, it is possible to simulate the following scenario:
- 80 % of clients use UDP
- 10 % of clients use DNS-over-TLS
- 10 % of clients use DNS-over-HTTPS
The results for each protocol can be interpreted separately and then compared to each other. You can find more information about DNS Shotgun, its features and usage in our comprehensive documentation. It describes the benchmark process step by step and also includes some performance tuning tips.
More Resources
We’ve presented the results of our benchmarks of DNS-over-TLS and DNS-over-HTTPS at DNS OARC 33. Some of our UDP measurements were presented at RIPE 79.
A more general talk with our recommendations for DoT/DoH deployments based on our benchmark experience are also available in this video which was originally presented at DoH webinar organized by eco & CENTR.
Acknowledgements
We’d like to thank the Comcast Innovation Fund for sponsoring the work to support the use of TCP, DoT and DoH protocols.
DNS Shogun is built of top of the dnsjit engine. We’d like to thank DNS-OARC and Jerry Lundström for the development and continued support of dnsjit.