Depending on your age, you either might or might not have used Telnet to connect to remote computers in the past. But regardless of your age, you would probably not consider Telnet for anything you currently use. SSH has become the de facto standard when it comes to remote shell connection as it offers higher security, data encryption and much more besides.

When we created our first honeypots for the Turris project (see our older blog articles – 1, 2, 3), we started with SSH and Telnet, because both offer interactive console access and thus are very interesting for potential attackers. But SSH was our main goal, while Telnet was more of a complimentary feature. It came as a great surprise to discover that the traffic we drew to the Telnet honeypots is three orders of magnitude higher than in the case of SSH (note the logarithmic scale of the plot below). Though there is a small apples-to-oranges issue, as we compare the number of login attempts for Telnet with the number of issued commands for SSH, the huge difference is obvious and is also visible in other aspects, such as in the number of unique attacker IP addresses.

And what is more, as you can see from the picture above, the activity at the Telnet honeypot has risen significantly and rapidly at the end of May 2016. The difference is even more pronounced when we use a linear rather than logarithmic scale (below).

Because we constantly monitor the honeypots for interesting new activity, we became very curious about what the reason behind this might be. Is this heightened activity from previously seen attackers or are new attackers starting to be active? If it is the latter, where are they coming from? The following charts should help us with the answers.

On the first chart below, you can see how the number of connection attempts per attacking IP address evolves in time.

There is some increase, but it clearly does not correspond to the huge rise in activity we have observed. Therefore, new attackers must have cropped up as well. This is demonstrated on the following chart.

The number of attackers has at one point risen from roughly 30,000 unique IP addresses per day to more than 100,000. And even though it has fallen somewhat in the meantime, it is still at at least double of the previous values.

This means that something must have happened – either an entity (probably a botnet) not previously active in Telnet scanning came into play, or an existing botnet has quickly gathered new members. To get a better idea of what really happened, we did a more in-depth analysis of the data.

Geography

At first we had a look at the geographical origin of the attackers and how it changes in time. The most active countries are shown below.

The plot is somewhat cluttered, but it is obvious that besides China, which was very active even before, most countries increased their activity at the same time. The plot below shows selected countries for which there was the most significant rise in the number of attacking IP addresses.

So now we know where most of the attacks are coming from, but we are still not much closer to discovering what is actually attacking us.

Devices

For this, we need to get more information about the attackers than the IP address alone can provide. We would like to know what kind of device is hidden behind the IP address at hand. For this we could either do an active scan of the address or use a third party service. We decided to go with the latter option and used Shodan.io to look up information about each attacker’s IP address (and as a side-effect we managed to significantly speed up the official python library for this service). From the data we gathered, we took a special interest in data usable for identification of the type of product behind the address. For this purpose, we found out that the “Server:” header from either HTTP or HTTP-like services is very useful. We were able to obtain the value of this header for more than 1.8 million IP addresses from the total of over 6.5 million observed attackers, i.e. slightly above 27 %.

The following plot shows the most active products as identified by the Server header. Please note that the header does not necessarily mean one specific product, but probably a family of products using the same or similar software. Also, one device might have more than one service enabled.

In first place we find the RomPager/4.07 HTTP server, which is an old version of an HTTP server used in many home routers and other embedded devices known for having serious security vulnerabilities in the past. In second place was gSOAP/2.7, which is an older version of a popular toolkit for web services used, again, often in embedded devices. H264DVR 1.0 is an identifier for a RTSP (Real Time Streaming Protocol) server used in online DVR products, such as security cameras, etc. From the names of the other products, you can see that many others come from the embedded world, such as the Dahua Rtsp Server which again points to CCTV cameras. This product also had a serious security vulnerability found in the past.

From what was shown above, it is clear that, at least from the devices we could identify, a large number are in fact embedded devices, such as cameras, routers, etc. These devices often run outdated software which are known to have security holes and an attacker with such knowledge can easily compromise a large number of hosts by a single exploit. But what we did not yet explore is the question of whether any of these devices might be behind the recent increase in Telnet attack activity. The following plots should shed some light on this.

The plot above shows the activity for the most often seen products: RomPager/4.07 and gSOAP/2.7. Both have seen almost an order of magnitude increase in activity since May 2016. But what is even more interesting is the situation of the H264DVR 1.0 product. In this case, there was almost no activity until April 2016, when about 7,000 unique IP addresses with this server started to scan Telnet for about a month. Then, after a brief period of calm, the number of attackers from these devices rose even higher to about 20,000 unique IP addresses per day.

The third plot shows a similar trend, but without the intermittent period of activity, for the Dahua Rtsp Server.

From this we can conclude that the rise of Telnet activity is driven by attacks from compromised embedded devices. We could speculate that an attacker was able to target these devices using some known vulnerability and after taking them over, uses them to spread the botnet even further. What is even worse than the number of attacking devices is the trend. The following plot shows the number of new IP addresses seen on the Telnet honeypot each day.

Currently, each day some 20,000 new devices appear. The composition of these newly attacking devices is similar to the overall picture. Thus, many of those are again embedded devices as shown below.

In fact, the situation for some of the products is such, that the number of attacking devices of a certain type forms a large proportion of these devices visible on the Internet. The plot below is a copy of one previously shown, but it contains annotations showing a comparison between the number of such devices seen in attacks and the total number of such devices visible on the Internet as observed by Shodan.io. The most notable case is the H264DVR 1.0 server, where more than two thirds of all visible devices seem to be “infected”.

Conclusion

Our Telnet honeypot has seen a significant rise in activity since May 2016. From the available data, we conclude that many of the attack attempts come from embedded devices, such as CCTV cameras, routers, etc. These devices form an easy target as there is usually a “monoculture” of these devices, all having the same setup and same vulnerabilities. It is very likely that an adversary is specifically targeting some of these devices to form a botnet. It even seems that in some cases, a large proportion of online devices of a specific type are already taken over.

In the course of our investigation, we were able to obtain one “infected” CCTV camera. We were not able to find any obvious malware in its firmware and thus we conclude that the attacks are probably performed remotely without permanent changes to the firmware. However, these results are preliminary and we will continue to investigate the case in more depth.

As we have already seen more than 6 million unique IP addresses attacking our Telnet honeypot, we decided to make it possible for people to check if their equipment was compromised. Therefore, we have setup an online service AmIHacked.turris.cz which offers a simple way to test if an IP address was observed on our honeypot.

Though I sincerely wish that you don’t find your IP address in our database, I urge you to try it out. Especially if you have a product matching any of those we have mentioned above. And regardless of the result, I recommend taking some time to review which of your ‘smart’ devices are accessible from the Internet and consider limiting access them by using a firewall. No system is perfect and devices with outdated software without regular updates are a security risk. It is very likely that what we observe here with Telnet is just the tip of the iceberg of what is going on out there on the Internet.

Autor: Bedřich Košata