Security

Network security in the microservice environment

We have seen that a microservice architecture is intimately tied to the use of a TCP/IP network as the interconnecting fabric, so when Bernard Van De Walle from Aporeto gave a talk at CloudNativeCon and KubeCon Europe 2017 on why we shouldn't bother securing that network, it seemed a pretty provocative idea.

Back in the old days, said Van De Walle, the enterprise had its computing infrastructure in data centers — big racks of servers, all with workloads running on them. The interconnecting network was divided into zones according to functional and security requirements. The zones were separated by big firewalls which filtered traffic based on source and destination IP address (or subnet) and port range. Modulo a few disasters, this worked OK.

But then came microservices, which took away workload predictability. No longer could one point at a box and say "this is a Java box, it runs the JVM", or "this is an Apache box, it runs the web server". Now any given pod may end up running on any node, at least by default, and this is a big challenge to the traditional model of firewalls. One could run one firewall per node, but people have started to deploy VPNs and software-defined networks, and once you do that, each new pod deployment requires updating the node firewalls across the entire deployment. The traditional firewall-as-gatekeeper model has real problems in this world.

So we should step back and think about this again; perhaps the network is not the best place to provide our interconnection security. Indeed, Van De Walle went on, we could embrace the contrary and assume that the network provides no security at all. This changes our threat model; the internal network is now just as untrusted as the external one. We cannot now rely on IP addresses and port numbers as authenticators; each flow must be authorized some other way. Kubernetes, he said, defaults to zero-trust networking; that is, by default the network is flat from the ground up. The IP address is dynamically assigned and carries no real information. Instead, Kubernetes has objects with associated identifiers — name, namespace, and labels — these are the foundation for your identity.

Kubernetes has made attempts to address the flatness of the network before. In v1.4, noted Van De Walle, network policies were introduced. The policy engine was based on a whitelist model; that is, everything was forbidden unless you explicitly authorized it. Policies are ingress-only, so you get to police traffic coming into your pod, but not that going out of it. Network policies are not active by default; they are activated on a per-namespace basis by the use of an annotation in the relevant YAML file. (For those who are not familiar with Kubernetes, I should point out that nearly everything involves YAML. The creation, modification, or destruction of just about any kind of resource involves the changes being specified in a YAML file, which is then referenced on the command line. If you're going to start using Kubernetes, prepare to start dreaming in YAML.)

In Van De Walle's opinion, Kubernetes network policies are fairly good. Rules are applied to specific pods, which are selected by a role label. Although the filtering is ingress filtering, you specify which traffic is allowed to enter a pod based on the role of the sender. For example, a policy might apply to all nodes whose role was backend; that policy would permit any traffic originating from a pod whose role was frontend. Rules are additive; each rule allows a new class of traffic, and traffic need only match any one rule in order to be permitted ingress.

There are already a number of projects working with (and extending) the network policies mechanism on the market; Van De Walle named Project Calico as an example, and there are others. But these implementations are tied to the networking backend, because, at the end of the day, policing is still based on IP addresses. So Aporeto has developed Trireme, a way of securing data flows that is independent of IP address information or the network fabric, and based entirely around the pod label.

Trireme adds a signed-identity-exchange phase on top of TCP's three-way handshake. The signed identity is then used to implement Kubernetes's native network policies with an enhanced degree of reliability. The iptables "mangle" table is used to send packets involved in the handshake via Trireme's user-space daemon, which must be installed on each authenticating node and which adds the identity exchange on top of the TCP handshake. Signatures are authenticated via a pre-shared key, or full PKI. If the pre-shared key option is used — only recommended for small deployments — Aporeto suggests distributing the key as a Kubernetes secret; if PKI is used, each node must be supplied with its private key and the relevant certificates.

Because this authentication is implemented on the node level, the pod is completely unaware that it is happening; it just never hears incoming traffic that fails the Trireme test. At this point, Van De Walle did a pleasingly quirky demo where a server pod was deployed that took requests over the network for the "Beer of the Day", which it returned from a random list of German beers. Two client pods were also deployed, both running a tiny application that continuously retrieved Beers of the Day from the server; one client possessed the tokens to assert an identity via Trireme, and one did not. When no network policy was in force, both clients were able to retrieve the beer of the day; when a network policy allowing only the approved client to connect to the server was applied, the non-approved client could no longer retrieve the daily beer.

Trireme is particularly helpful when not everything is happening in a single Kubernetes cluster. The ability to federate clusters across multiple data centers is coming; because this will almost inherently involve network address translation (NAT), authentication via source IP becomes extremely difficult. But as long as a TCP connection can be made, Trireme can layer its identity exchange on top. Future plans for Trireme include the ability to require encryption on-demand on a connection-by-connection basis, though this will slow data flows as now all packets in a flow will need to go via the user-space daemon, for encryption to be applied/removed.

There are problems, or at least corner cases. Because TCP is stateful, and the netfilter state engine is used to recognize the packets involved in a new connection in order to send those (and only those) via the Trireme user-space daemon, every connection set up before any policies are applied remains valid after policy application, even if the policy should have forbidden it. Aporeto is experimenting with flushing the TCP connection table in order to address this problem.

The slides from Van De Walle's talk are available those who are interested. Trireme is an elegant implementation of a clever idea, but for me its greatest value may be in encouraging me to recognize that zero-trust networking is a good way to think in a containerized microservice environment; that the old days, when access to the private network bolstered or indeed established your right to access the information stored thereon, might just be passing away.

[Thanks to the Linux Foundation, LWN's travel sponsor, for assistance in getting to Berlin for CNC and KubeCon.]

Comments (16 posted)

Pandavirtualization: Exploiting the Xen hypervisor (Project Zero)

The latest installment from Google's Project Zero covers the development of an exploit for this unpleasant Xen vulnerability. "To demonstrate the impact of the issue, I created an exploit that, when executed in one 64-bit PV guest with root privileges, will execute a shell command as root in all other 64-bit PV guests (including dom0) on the same physical machine."

Comments (8 posted)

Over The Air: Exploiting Broadcom’s Wi-Fi Stack (Part 2) (Project Zero)

Here's the second part in the detailed Google Project Zero series on using the Broadcom WiFi stack to compromise the host system. "In this post, we’ll explore two distinct avenues for attacking the host operating system. In the first part, we’ll discover and exploit vulnerabilities in the communication protocols between the Wi-Fi firmware and the host, resulting in code execution within the kernel. Along the way, we’ll also observe a curious vulnerability which persisted until quite recently, using which attackers were able to directly attack the internal communication protocols without having to exploit the Wi-Fi SoC in the first place! In the second part, we’ll explore hardware design choices allowing the Wi-Fi SoC in its current configuration to fully control the host without requiring a vulnerability in the first place."

Comments (24 posted)

Alert summary April 6, 2017 to April 12, 2017

Dist.	ID	Release	Package	Date
Arch Linux	ASA-201704-3		mediawiki	2017-04-10
Arch Linux	ASA-201704-2		python-django	2017-04-09
Arch Linux	ASA-201704-1		python2-django	2017-04-09
Debian	DLA-893-1	LTS	bouncycastle	2017-04-10
Debian	DSA-3829-1	stable	bouncycastle	2017-04-11
Debian	DSA-3828-1	stable	dovecot	2017-04-10
Debian	DSA-3828-2	stable	dovecot	2017-04-11
Debian	DSA-3827-1	stable	jasper	2017-04-07
Debian	DLA-887-1	LTS	libdatetime-timezone-perl	2017-04-07
Debian	DLA-891-1	LTS	libnl	2017-04-10
Debian	DLA-892-1	LTS	libnl3	2017-04-10
Debian	DLA-888-1	LTS	logback	2017-04-08
Debian	DLA-890-1	LTS	ming	2017-04-10
Debian	DLA-889-1	LTS	potrace	2017-04-09
Debian	DLA-894-1	LTS	samba	2017-04-11
Debian	DLA-886-1	LTS	tzdata	2017-04-07
Fedora	FEDORA-2017-b38b98727e	F25	curl	2017-04-09
Fedora	FEDORA-2017-174cb400d7	F24	flatpak	2017-04-11
Fedora	FEDORA-2017-047cffb598	F25	ghostscript	2017-04-09
Fedora	FEDORA-2017-712a186f5f	F24	icecat	2017-04-07
Fedora	FEDORA-2017-674d306f51	F25	icecat	2017-04-07
Fedora	FEDORA-2017-ab3acddd21	F25	libtiff	2017-04-10
Fedora	FEDORA-2017-51979161f4	F25	tigervnc	2017-04-07
Fedora	FEDORA-2017-7e5b5201e7	F25	xen	2017-04-05
Fedora	FEDORA-2017-054729ab08	F25	xen	2017-04-09
Gentoo	201704-02		chromium	2017-04-10
Gentoo	201704-01		qemu	2017-04-10
Gentoo	201704-03		xorg-server	2017-04-10
openSUSE	openSUSE-SU-2017:0969-1	42.1 42.2	apparmor	2017-04-10
openSUSE	openSUSE-SU-2017:0955-1	42.1 42.2	clamav-database	2017-04-06
openSUSE	openSUSE-SU-2017:0961-1		ffmpeg	2017-04-07
openSUSE	openSUSE-SU-2017:0958-1	42.2	ffmpeg	2017-04-07
openSUSE	openSUSE-SU-2017:0942-1	42.1 42.2	libpng12	2017-04-05
openSUSE	openSUSE-SU-2017:0937-1	42.1 42.2	libpng16	2017-04-05
openSUSE	openSUSE-SU-2017:0941-1	42.2	nodejs4	2017-04-05
openSUSE	openSUSE-SU-2017:0982-1	42.2	php7	2017-04-11
openSUSE	openSUSE-SU-2017:0973-1	42.2	pidgin	2017-04-11
openSUSE	openSUSE-SU-2017:0935-1	42.1	samba	2017-04-05
openSUSE	openSUSE-SU-2017:0944-1	42.2	samba	2017-04-05
openSUSE	openSUSE-SU-2017:0980-1	42.1 42.2	slrn	2017-04-11
Oracle	ELSA-2017-0893	OL6	389-ds-base	2017-04-11
Oracle	ELSA-2017-0892	OL6	kernel	2017-04-11
Red Hat	RHSA-2017:0893-01	EL6	389-ds-base	2017-04-11
Red Hat	RHSA-2017:0892-01	EL6	kernel	2017-04-11
Red Hat	RHSA-2017:0933-01	EL7	kernel	2017-04-12
Red Hat	RHSA-2017:0931-01	EL7	kernel-rt	2017-04-12
Red Hat	RHSA-2017:0932-01	MRG/EL6	kernel-rt	2017-04-12
Scientific Linux	SLSA-2017:0893-1	SL6	389-ds-base	2017-04-11
Scientific Linux	SLSA-2017:0892-1	SL6	kernel	2017-04-11
Scientific Linux	SLSA-2017:0630-1	SL6	tigervnc	2017-04-05
Slackware	SSA:2017-098-01		libtiff	2017-04-08
Slackware	SSA:2017-100-01		vim	2017-04-10
SUSE	SUSE-SU-2017:0946-1	SLE11	jasper	2017-04-05
SUSE	SUSE-SU-2017:0983-1	SLE12	xen	2017-04-11
Ubuntu	USN-3258-2	16.04 16.10	dovecot	2017-04-11
Ubuntu	USN-3258-1	16.04 16.10	dovecot	2017-04-10
Ubuntu	USN-3257-1	16.04 16.10	webkit2gtk	2017-04-10

Full Story (comments: none)