Since the end of January 2021, the data from all authoritative DNS servers operated by CZ.NIC about DNS transactions (queries and responses) is being collected exclusively using the new standard Compacted-DNS (C-DNS) format defined in RFC 8618.  For data acquisition on the servers we use the DNS Probe software, developed by CZ.NIC Labs in cooperation with Brno Technical University. This milestone marks the end of a six-month transition period in which we migrated all servers from the traditional PCAP format that we used previously. During that period we heavily tested and improved the performance and stability of DNS Probe, and also compared the results obtained in both the old and new format.

Unlike PCAP, the C-DNS format was created specially for storage and transmission of data about large sets of DNS transactions. It was designed so as to be as effective and flexible as possible. The price for these qualities is the relative high complexity of the format. A detailed description of C-DNS is outside the scope of this blog post, but here are at least the essential characteristics:

• CBOR binary data encoding (RFC 7049) – updated: RFC 8949
• variable precision of time stamps
• it is not necessary to store all available data; on the other hand, it is possible to add items that are not in the standard
• data about a DNS query and the corresponding response is stored together, common items are not duplicated
• large sets of transactions are put into blocks of a variable size (the recommended size is about 10 thousand transactions)
• blocks further contain certain metadata and statistics, and also implement additional data compression.

The task of the DNS Probe software is to extract clients’ DNS queries and server’s replies from the network traffic (both UDP and TCP), pair those that belong together and generate C-DNS data according to the configuration. The output data can be either stored on a local disk or immediately sent for further processing via an encrypted TLS connection. A part of the DNS Probe project therefore is a data receiver (dns-probe-collector package). In our pipeline, we then store the data received this way straight to the Apache Hadoop database.

The most visible benefit of the C-DNS format in comparison to PCAP should be a significant decrease in the volume of stored and/or transmitted data. With our data it is indeed the case: the following table shows the volumes of data in both formats collected during December 2020 from nine production DNS servers (xz compression was used for both formats). The mean value of the last column (C-DNS/PCAP ratio) is 15.6%.

When we compared the data, it turned out that DNS Probe systematically reports some transactions that we don’t get from PCAP data. The difference is fairly negligible in UDP traffic (at most 1%), but for TCP it is usually 5–10 %, and for some servers even considerably more. The reasons may be due to both the known problems of the PCAP format (e.g. the response may sometimes precede the corresponding query), and possibly also flaws in the software tools that we used for looking up and pairing queries and responses.

Looking into the future, we also plan to utilize the extensibility of C-DNS and add optional data items related to the clients (resolvers): the autonomous system number (ASN) and the GeoIP-based country code.