Little bit of history
Apart from operating .CZ top level domain, CZ.NIC does a lot of other interesting things contributing to the common good. Part of it is running Czech national CSIRT team, doing security research and raising awareness about potential security issues. As part of our security research, we started wondering a long time ago how much are the average Joes and Janes attacked, by who and how. People that are just connected to the internet, run no public service and are just consumers. If only there was some kind of probe that would allow us to see what is going on there…
And that is how project Turris started. We made a HW probe to tell us what is going on. We gave it away for free in Czech Republic to people who were interested to help us with our research. They were required to use it as their main means to connect to the Internet and let us run some traps. The output of those traps were automatically send to our servers for our security researchers to look at. As security research takes some time and we were getting plenty plenty of data, we also processed those data automatically with a simple goal – we tried to somehow automatically detect wannabe attackers. Whenever somebody (on the Internet trying to connect to our router) raised attention of our automatic system, we put him on our automatically generated greylist and also distributed that new list of the “bad guys” to all our probes to block them. That means that if you tried attacking few of our routers, the rest was already warned and refused to even talk to you. Off-course, copy of all the data was also available for security researchers to look at and analyze who, why and how.
Apart from the security research aspect, those probes had some extra features like multiple gigabit ports and Wi-Fi to make people want to use them as their main Internet gateway. But they also featured some USB ports, mPCIe slots and off course, since we were doing some hardware, we did it the right way. We used open-source software, Linux distribution with plenty of packages available, we committed to providing software updates and gave people root accounts as any decent hardware vendor should.
People loved those devices and based on their side features started calling our probes routers. As the message spread, people abroad started asking us for those. Some people were also asking to get those with disables data collecting service to use as regular router and were trying to bribe us! With money! In the end we decided to do Indiegogo campaign to figure out whether people are really interested in such a device and even willing to pay for it for real. In the end we actually did two campaigns and both of them succeeded. The message was clear, people are indeed very interested in open and secure routers.
But what about our security research? Well, we hoped that some of those people would enable this feature even thought it was voluntary. And it really happened. People started buying our routers and thanks to the sales of Omnia and nowadays MOX, we have now probes around the world collecting data about various attack attempts. That itself is awesome, but apart from requests to buy our routers, we got other types of requests as well. People were interested in somehow joining the fight against the attackers.
HaaS joins the party
Finding out that people actually have a burning desire to help fighting back those pesky attackers and would like to be mean to them was a really interesting outcome. Everybody can run a honeypot and try catching some attackers. But it is not entirely easy to setup and manage, hard to figure out what to do with the data and actually use it for something.
To make things much easier, CZ.NIC created a project HaaS – Honeypot as a Service. It is a simple service, really easy to install and get running. All it does is that it forwards the attacker to a honeypot managed by CZ.NIC. No danger to the user even if attacker manages to get out and nice statistics easily available on the web. Apart from that, recorded sessions are available to all security researchers and directly used by people from CSIRT team.
This service got quite some attention and became popular. It was the first part of data collection that we made available elsewhere and people loved it. So apart from data from the home users we started getting data from people who were running our software on their servers in various countries.
Sentinel is born
Our data collection software was originally developed more than half a decade ago with really specific idea in mind. We knew what we wanted to collect, how, we knew how many routers will there be and we were only unsure how long would this project last. Over the time, we started making more and more routers, we learned from our data where we need more granular data and on the other hand what data were pretty useless although they looked really interesting at first. What we were certain of is that we need something more modular, extensible and scaleable. So we came with the idea of complete rewrite from scratch. And during the rewrite, one of the features we would like to achieve is to be able to let more people to participate, similar to HaaS.
What is the current state? We are making great progress, we have designed how it should work in the end, we have all the building blocks ready and we even have few components migrated over already. Currently it is available on our routers to early adopters and is not entirely user friendly/intuitive. That is one of things we are working on right now. Once we finish that, we will migrate all the routers that are part of our data collection network nowadays. And there is also some tuning to do – where to set a line between attacker and confused user.
Although development is still in progress, we already have one component that you can use even on your own servers. Dynamic firewall. Component that fetches list of attackers from our servers and protects you against them. Currently no way to contribute back if you don’t have our router apart from HaaS (we get data from there), but that will change.
How to deploy
Last part of this post is a simple howto for those of you who want to get our dynamic firewall running on their servers. I tried to do that for my servers, so I created some packages for rpm based distributions. You can find them on OBS. The package you are after is named sentinel-dynfw-client and I tested it on openSUSE. But as your distribution might behave differently I’ll describe what I did and why here as well.
What is part of the package is a simple python program, that connects to our server, fetches the list of bad guys, listens for any changes and synchronizes that with the locally managed ipset. To make sure, we are connecting to the right server, we need a certificate. I integrated downloading it into unit file for the whole service.
Now when we have an ipset with bad guys, we need to somehow integrate it into firewall. In the past, every distribution had their own way of managing firewall, but nowadays, everybody has firewalld, which makes writing howtos simpler 🙂 For firewalld you need tho configuration files. One introduces our ipset and other one specifies a firewall zone where all the bad guys in the ipset will end. And there, they will get blocked.
One last note, as mentioned, list is currently quite small as we are just starting testing it and obviously it makes sense to integrate it not only with Linux firewall but also with many other services. To do so, we have some python skeleton that is easy to extend and can be used to integrate with something else. If you manage to come up with some cool use, please let us know!