Locked Shields is the largest international cyber security drill. It is regularly organised since 2010 by NATO CCDOE (Cooperative Cyber Defence Centre of Excellence), and the focus of the drill is a clash between two teams. The red team attacks the blue team, which plays the role of the defender. This year, the drill was attended by a total of 19 blue teams. The teams were charged with the defense of a diverse computer infrastructure of a fictional country’s military base consisting of different servers, numerous workstations, SCADA systems, etc. The defenders were to face attackers, whose objective was to damage, compromise, or completely take down the network or its elements, or at least to make things complicated for the defenders. In addition to the technical part, the drill is focused also on strategic decision-making, cooperation with the press and the handling legal matters. We were invited by colleagues from GovCert and assigned to the “Linux team”.
The event was held over two weeks, for two and three days of each of the weeks. On the first week we were mainly getting familiar with team members, the infrastructure and systems. During these first two days, it was important to find vulnerabilities of all kinds and identify malicious software or unnecessary running services. Then, access to the so-called Gamenet was closed and all machines were set to their original state.
The second week, we had only the first day to test the prepared patches and then, again, all the machines were reset. On the second day, it all started. After the Gamenet was open, we had 30 minutes for “defending” before the red team started its attacks. We had to use these 30 minutes to apply everything we prepared during the previous days. Here, automation was revealed in its full glory. The rest of the second day and the entire third day were the time of unrestrained attacks – 15 and a half hours in total.
The overall atmosphere was very tense, IP addresses used by the attackers from the red team that needed to be blocked could be heard in the corridors, and in the room where we were sitting (with the “linux guys”), colleagues were giving advices and helping each other. Since each of us had to take care of a different system, here we will try to bring you at least two viewpoints:
Radka:
In the first week, after servers and services were assigned, I already knew that I was going to take care of a server with a military accessories e-shop. So I looked for holes in configuration of the Tomcat used to run the e-shop. I also took a look at the local database to see if system binaries were modified, unnecessary services were running or unwanted ports were open. At the same time I admired the skillful work of other team members on their services. Many of them already had experience from previous years, so they studied the materials and the rules of the game more aptly or just had more practice with the areas in question. I repeatedly asked myself if I was going to be of any help there 🙂 Even after we went our separate ways in the evening, at home, many of us including me were considering the ways to improve things.
The next day, we again tried all kinds of magic on our respective servers. We had another day for such testing during our second week, but this time on fully reset machines, so I could try out the things I omitted the previous week.
And the next day it all started. At nine in the morning we were let into the GameNet, where we had half an hour to “fix” everything. Here we could observe the advantages of automation. The team members who were in charge of this part have created many useful Ansible playbooks in the previous days that they could apply right away, so we had a decent foundation. Around the thirty-fifth minute, my initial enthusiasm was spoiled by defacement, which misused an e-shop forum that was not treated to withstand an SQL Injection attack. After a while, I found and patched the hole, also giving the fullest details on the incident.
Fortunately, nothing worse than that happened to the e-shop since (or is it just sweet ignorance?), although more attempts could be seen in the logs. While investigating problems in my section, I closely followed others whenever I could and helped them where I could, and often admired how easily they dealt with much more complicated cases under the pressure.
By the end of the last day, the atmosphere became more relaxed, but the tension rose again when we looked at the score, because our team was on the top positions. At 4 PM, the GameNet was closed and we eagerly watched the last points being credited for reports or communication with users. Around 5 PM we already began to suspect that we could really end up on the first place, but the second team was only few points behind us. The next day a webinar was held, where leaders of different teams (red and also green and white support teams) in Tallinn shared their impressions with us, but most importantly, also announced the results from which we finally knew for sure that we were FIRST.
Overall, it was a great experience for me, I had a lot of fun while also learning a lot thanks to the advice of more experienced colleagues. I also had the opportunity to meet other experts from the field and see these masters at work.
Martin:
My job was to take care of operation of three servers. The first two served as DNS resolvers including authoritative servers for our domain and were also responsible for time synchronization via NTP. The third one was a VPN server for remote user connections. Here was the first hitch. I was looking forward to the classic OpenVPN, but what was waiting for me was an OpenVPN Access Server, which is a more commercial version, different from the classic free-distributed version, so I had to get to know it first.
In addition to the above, among the many things that needed to be found/verified, or even removed right at the beginning, I can mention the Linux packages debsums and rkhunter that enjoyed quite a success in our team. It was relatively easy to patch the configurations of DNS servers — for example, zone transfer was allowed in configuration files. It was a bit trickier for the VPN server though, because all the configurations including logs were in SQLite, and familiarization with the structure wasn’t so straightforward, but by the beginning of the drill I managed to prepare a script for deleting unneeded accounts and unathorized methods of authentication. The deletion of SUID bit for /bin/nano was just the icing on the cake.
During the “battle” itself, it was relatively quiet on my servers. Sometimes, in addition to IP addresses, some domains could be heard from the “corridors”. Such domains had to be blocked at the DNS server.
Finally, the clock stroke 4 PM and our eyes turned to the score, which at the time was not yet final and points were still being added/subtracted. The air grew thick with tension and at one point, the score froze when we were one point away from being on the first place (which was really sad, considering the total score was over 30,000). But then it started changing again and, eventually, it was announced unofficially that the first three places will stay unchanged. By that moment, we were first! It should be added that the second place was only about 400 points behind and it’s really not as much (congrats Estonia).
In conclusion, we can only say that all in all, the participation in the Locked Shields was a really great experience for us. We would like to once again thank all our team members and also GovCERT for the invitation. We are happy that we could be part of such a success.
Radka Nepejchalová, Martin Kunc