Turris: secure open-source routers

The Czech Republic top-level domain registrar, CZ.NIC, wondered about the safety of home routers, so it set out to gather some information on the prevalence of attacks against them. It turns out that one good way to do that is to create a home router that logs statistics and other information. Michal Hrušecký from CZ.NIC came to the 2019 Southern California Linux Expo (SCALE 17x) in Pasadena, CA to describe the experiment and how it grew into a larger project that makes and sells open-source routers.

CZ.NIC is legally an association of competing companies, but in reality it is run like a non-profit, Hrušecký said. Beyond just domain registration, CZ.NIC has various other activities around making the internet more accessible and secure. That includes projects like the BIRD internet routing daemon and the Knot DNS resolver, as well as books, translations, and even a television series on "How to handle the internet". Beyond that, the Czech Computer Security Incident Response Team (CSIRT) is part of CZ.NIC.

One of the other things it is doing is creating open-source home routers. It started because CZ.NIC wondered about how safe home users are from network attacks. Are there active attacks against home users? And, if so, how frequent are they and what kinds of attacks are being made? To figure out the answer, the organization created Project Turris to create a secure router that it gave away. These routers would monitor the network and report suspicious traffic back to the project. They also served as endpoints for some honeypots that the project was running.

CZ.NIC wanted to make the Turris router "the right way", he said, so the organization made it all open source. The router has automatic security updates and users are given root access on the device. It also sported some "interesting hardware", Hrušecký said; it had a two-core PowerPC CPU, 2GB of RAM, and 256MB of NAND flash.

Based on the information provided by the Turris routers, CZ.NIC researchers started publishing reports about what they were finding. That led some people to ask if they could get the routers themselves, because they felt that other router makers were "not doing things right". That led to the creation of commercial Turris routers: the Turris Omnia (which was reviewed here in 2016) and the upcoming Turris Mox. Those routers will still allow people to participate in the research if they choose to.

Building the routers with free and open-source software (FOSS) is really the only way to go, he said. The project knew that it was not going to be able to compete with small, cheap routers, so it created routers with lots of capability that would allow them to run lots of different kinds of services. FOSS makes it easy to get started on a project like this because there is lots of available software that can be easily integrated into the OS.

These routers allow users to do whatever they want and people believe they are more capable than they truly are, Hrušecký said. That means they break things in "really creative ways". Sometimes they will make custom changes, completely outside of the OS framework, which get overwritten with the next automatic update. These are "tricky problems" to handle; the project would not have if it locked its users out. At some "dark moments" he understands why some companies do that.

Another tricky piece is upstreaming, he said. Turris works on getting its code upstream, but it takes longer than "anyone would want", he said. The project can take shortcuts that the upstream project will find lacking. Upstream projects want the code to be polished and generalized, which takes time. The "upstream project" in this case is OpenWrt, which is a distribution for routers. But OpenWrt is optimized for routers with far less resources than the Turris routers.

Typically, OpenWrt installs a highly compressed filesystem image into flash and has a small overlay where packages can be installed—generally only a few, however. Turris is not using the compressed image, but is instead using the "coolest filesystem for Linux": Btrfs. The project is using Btrfs snapshots and "went crazy" with them. It does a snapshot automatically weekly and before any update; it also allows manual snapshots. A "factory" reset can go back to the previous snapshot, the factory snapshot, or reflash the system.

Turris created its own web interface for the router, which is simpler than the standard OpenWrt interface. OpenWrt is targeted at more-technical users, while Turris wanted an interface for less-technical users, but to still allow them to use advanced features, such as VPNs or adding a guest WiFi network. Since Turris does things a bit differently, it sometimes runs into problems that OpenWrt does not have. In addition, OpenWrt packages are sometimes too trimmed down feature-wise, so Turris must build its own versions. Some packages use LXC containers, which may seem crazy, he said, but does make sense in some cases; it requires a different kernel configuration from the standard OpenWrt kernels, though.

Hrušecký introduced the honeypots as a service (HaaS) project by saying: "Honeypots are cool, right? Everyone wants a honeypot at home." But it takes time to set up and maintain a honeypot and there is some risk, so why not have someone else run it for you? CZ.NIC will run the honeypot; users just need to run the HaaS proxy on their system, which will relay potentially malicious traffic (e.g. connections to the SSH port) to a honeypot on the HaaS server. It will simulate a device and record what is sent by the attack. Users can then check out the attacks aimed at their server on the HaaS web site. HaaS is something that came from the security research but has now been separated out from the router project so it can be used elsewhere.

Turris Sentinel is a work in progress that will make some of the other security-research pieces available outside of the router framework. It will collect firewall logs and send them to a central location. It also has "minipots", which pretend to be a service on some port (e.g. telnet, HTTP), ask for login credentials that get logged, then close the connection. There was an earlier version of this, but it was closely tied to the Turris routers, so it has been rewritten to be more general. The Turris project was a bit surprised how willing people are to provide this data to it. The data will be made available on the site eventually, but is currently being shared with the Czech and other countries' CSIRT organizations.

The project has integrated the Suricata open-source intrusion-detection system (IDS) and intrusion-prevention system (IPS). It can look deeply into the packets and log or block a network flow based on its rules. For unencrypted communication, it has access to all of the information exchanged. But even for encrypted connections, there is a fair amount of information that can be extracted from things like the IP and MAC addresses, parts of certificate exchange, the length of the connection, and the amount of data transferred.

Suricata can be used to monitor untrusted devices and detect suspicious anomalies. There are open-source rules available to detect malware attacks, which can be used to block the traffic, for example. There is also the PaKon tool for Suricata that will aggregate information about the traffic on the network. It will alert when a new computer connects to the network. It will allow you to find out what your refrigerator is doing on the network when you are not at home, Hrušecký said with a grin.

Something that has come out of the research that CZ.NIC is doing is a "list of bad guys". If certain hosts are repeatedly attacking servers and routers that are reporting back, they will get added to the list, which is sent out to all of the routers. They can then block those IP addresses to reduce the malicious traffic they are handling.

Something that people have been asking for is Nextcloud support. It makes good sense, he said, because the Turris routers are the ultimate in self-hosting. They live at your home and you are root, so it is natural fit. Turris is also working with Nextcloud on a device that will specifically target hosting that service. So far, much of the software side is working, though there are still some areas that need work.

One of the "little bit crazier" uses is to turn the router into a digital video recorder (DVR). Adding a USB DVB-T device and a disk drive gives you a DVR, he said. Adding TVHeadend along with Nextcloud turns the Turris device into a router and home server combo box.

Hrušecký demonstrated the HaaS interface and took questions at the end of the talk. He showed how you can look at the kinds of attacks that are being attempted against your router (but are actually being handled by the CZ.NIC HaaS server) including the credentials used, the commands the attacks are trying, and the locations where they are trying to download code from. The Turris routers cost around €300, he said in response to an attendee question. They are not directly available in the US, but that is being worked on; there is lots of paperwork that needs to be completed. Until then, he suggested looking on eBay and similar sites.

A YouTube video of the talk is available.

[I would like to thank LWN's travel sponsor, the Linux Foundation, for travel assistance to Pasadena for SCALE.]