Bose and Kubernetes

Dylan O'Mahony, the cloud architecture manager for Bose, opened a presentation at KubeCon + CloudNativeCon North America 2018 by noting that many attendees may be wondering why a "50-year-old audio company" would be part of a presentation on Kubernetes. It turns out that Bose was looking for ways to support its smart-speaker products and found the existing solutions to be lacking. Bose partnered with Connected, "a product development company from Toronto", to use Kubernetes as part of that solution, so O'Mahony and David Doyle from Connected were at the conference to describe the prototype that they built.

Problem

As a way to demonstrate the problem they were trying to solve, O'Mahony spoke to an Amazon "Alexa" device (an Echo Dot) and asked it to play a particular song "on stage". That led the nearby Bose smart speaker to start playing the tune. Since both devices have wireless interfaces, it would seem like making that work would not be all that difficult, he said. But it turns out to be harder than it looks. There is no direct interface between the two devices; it all must be handled in the cloud. So it takes hundreds of miles of cable to bridge the three-foot gap between the two devices on stage.

The Amazon device does all of its voice processing in the Amazon cloud, which then hands off instructions to the Bose cloud. The speaker is not directly exposed on the internet; it can send out messages, but it is unable to receive random messages from the net. The easiest way to handle that is to have the speaker make a persistent connection to the Bose cloud when it powers up. MQTT was chosen as the protocol; a persistent bidirectional WebSocket connection is made between each speaker and the cloud service.

The "crux of the problem" is scaling; solutions abound for thousands of connected devices. When he looked around a few years ago for Internet of Things (IoT) products, he couldn't find any that could handle the five-million (or more) connections envisioned for the system. Some managed services would scale to hundreds of thousands of connected devices, but not to millions, he said. That is why Bose engaged with Connected, which was able to help prototype a system that could handle that many connections using Kubernetes.

Solution

Doyle then stepped up to fill in what was done—and how. He noted that scaling for web applications is largely a solved problem, but that is not the case for scaling messaging applications. Four people at Connected worked on the project in two teams: makers and breakers. The makers were tasked with building the system, while the breakers were building the "test rig". People moved around between the teams to get exposure to both sides of the problem.

There were a lot of choices they could have made for the different parts that made up the stack, but their methodology was to find the first thing that would work and run with it. The Bose cloud infrastructure (nicknamed "Galapagos") was Kubernetes on AWS; it does not use Amazon's managed Kubernetes product (EKS). That part was handled by a small team at Bose. Each of Doyle's team's members had a full rollout of the stack available to be used; Minikube is fine, but "if you are going to build something at scale, you really have to test at scale", he said.

As O'Mahony had mentioned, MQTT was used. It is the dominant communications protocol for IoT applications because it is lightweight. The devices set up a persistent WebSocket connection to the cloud, but something other than "magic" is needed to handle all those connections. For that, they used VerneMQ.

The high-level picture (which can be seen in their slides [PDF]) has HAProxy as a front-end load balancer feeding VerneMQ acting as a message broker. They created their own "uncreatively named" service, called "the listening service", which picks up the device status messages from the speakers and stores a "shadow state" of the device in an Apache Cassandra database. The shadow state consists of the speaker volume, what it is playing, its firmware revision, and so on.

There are plenty of examples of configuring Kubernetes for HTTP ingress handling, but there were not many examples for ingress handling of straight TCP, he said. They were not using HTTP, but could not find examples of what they were trying to do so they chose HAProxy since it is the de facto standard for proxying and load balancing. It has been a good choice; it is "rock solid" and uses a "ridiculously low" amount of resources for a huge number of connections.

VerneMQ stood out as a message broker for a number of reasons. It is written in Erlang and the Open Telecom Platform (OTP), which were originally used in telephony applications; that makes it a good base for clustering and concurrency, he said. It also easily scales both vertically and horizontally. In addition, it is an open-source solution, which was important. Other solutions did not fit the bill because they were proprietary or lacked other features needed. VerneMQ provides bridging, which will be useful in some of the future plans as well as shared subscriptions (from MQTT version 5). The latter allowed VerneMQ to load balance between the different listening services in a round-robin fashion. Beyond that, VerneMQ maintained the time order for messages, which is important when handling speaker commands.

The glue is the listening service, which they wrote in Go. It is pretty straightforward; it only contains about 100 lines of code. It simply subscribes to the VerneMQ stream as a shared subscription, processes any status change messages, and writes the changes to the shadow state to the database. It is lightweight and scales easily.

The shadow store is a digital duplicate of the state of the device in the cloud. One of the questions they had was whether Cassandra could keep up with the volume of messages, which it handled easily. It is a non-relational database that turned out to be performant and fault tolerant, while also being "massively scalable" and quite stable. It was an important part of the proof of concept, but Doyle said he would not be mentioning it much more in the talk since they had few problems with it.

He then described how they put it all together on Kubernetes. The container images were all built from scratch using Alpine Linux, which resulted in small images. Because they are clustering services, VerneMQ and Cassandra are run as Kubernetes StatefulSets; that way, Kubernetes brings them up in order and gives them a consistent name in DNS. HAProxy was configured as a DaemonSet to ensure that it was deployed on each ingress node. The listening service, Prometheus, and Grafana were handled as Deployments; the latter two provided visibility into the cluster for management and diagnosing problems.

Testing

For the "test rig", they chose Locust. The "breakers" half of the team looked into other options, such as MZBench and Apache JMeter, but found them to be less flexible and not made for publish/subscribe (pub/sub) models. Locust is Python-based with a master node that instructs workers to simulate various loads and types of traffic. These were deployed on bare AWS EC2 instances and were not part of the production Kubernetes cluster.

Doyle then described some of the problems that were encountered on the way to supporting five million connections, but noted that it was "by no means an exhaustive list"; there were plenty of other problems in both the production code and the testing setup that were encountered and surmounted.

To start, they used one Locust worker to see how many connections it could make; the first result was an "underwhelming" 340. It turned out to be a problem with Python file-descriptor limits; three were consumed for each MQTT connection and Python was limited to 1024 in total. They tried replacing select() in the Eclipse Paho MQTT library with asyncio, but that did not play nicely with the Locust concurrency model. In the end, they simply rebuilt Python to increase the hard-coded size of file-descriptor sets. That resulted in roughly 10,000 connections per worker.

Moving on to a multi-worker test, they hit a barrier at around 700,000 concurrent connections. That was due to configuration defaults for HAProxy and VerneMQ as well as some problems with an additional layer of network address translation (NAT) because VerneMQ was set up as a Kubernetes Service abstraction. The workaround for that was to reconfigure everything, add more listeners for VerneMQ, bypass the Service abstraction, increase the network and I/O bandwidth of the HAProxy nodes, and ensure that HAProxy was doing a round-robin of VerneMQ nodes.

That allowed them to break the one-million barrier, but only to 1.1 million connections. At that point, subscriptions were failing due to VerneMQ nodes being terminated; Kubernetes brought them back up, but it impacted the number of concurrent connections that could be supported. This problem was harder to figure out, he said; they tried to incrementally scale VerneMQ from ten to 80 nodes, but that made no difference. Grafana and Prometheus were "absolutely key" to tracking the problem down. They eventually found that the broker would immediately subscribe connecting clients, which could overwhelm the other VerneMQ servers. The solution was modify the clients to add an exponential backoff delay between connecting and subscribing; that allowed VerneMQ to recover.

The next barrier was at 1.5 million connections; at that point, the Erlang OTP schedulers all went to 100% CPU utilization. Erlang has lightweight threads and for each CPU available to it, Erlang adds a scheduler for that CPU. Adding more resources for VerneMQ, which is the Erlang component, did not help. It turned out that Erlang OTP is not control-group aware, so even if it had 12 virtual CPUs (vCPUs) available to it, only four would be used. The solution was to directly configure the number of vCPUs in Erlang.

There were some other stopping points along the way, Doyle said, but eventually they were able to reach 4.85 million connections. One could perhaps claim that was five million ("what's 150,000 between friends?") but it still wouldn't feel quite right. The problem was resources again, but testing was expensive. It took around five hours to bring up all of the connections; at one point, they "blew through" their monthly testing budget in a day. "But we were getting so close", he said.

It came time to do an internal demo. They upped the resources one more time and were able to get to five million and one active, concurrent WebSocket connections, he said to applause. The average latency for a published message to reach a subscriber was 69ms, which is below the "magic number" of 250ms so humans will not perceive a delay. The system was publishing nearly ten thousand messages per second. There were 12 different testing scenarios that were used and one of the others could reach 25 thousand messages per second.

Lessons

There were a number of lessons that came from this project, which he wanted to pass along to attendees. First up was to pay attention to your dependencies. By design Kubernetes does not track or manage dependencies between the various parts of the system, but that was a little surprising to him as an application developer. They bring up the services in a particular order to try to get a handle on that, but there are still areas that needed some thinking. For example, when is a service actually up? Is it when one replica has started? Or half of them? Or all of them?

He recommended experimenting with resource limits. The proper values are hard to figure out; try out lots of different workloads and scenarios. "Do component-level load testing", he said; there will be a need for rate limiting, it can't be avoided for running at scale. Troubleshooting is made more difficult by all of the different layers in play. The more layers there are, the more problems there are in figuring out where problems lie. "This is why setting up monitoring early matters so much."

Starting out at scale is different than getting there via organic growth. When the system is growing, it will hit certain walls, as they did, but each needs to be surmounted before (sometimes almost immediately) smacking into the next one. Effortlessly scaling a system is a boring feature, but it is a "killer feature of Kubernetes". It is incredible to be able to simply double the size of a cluster at the snap of your fingers, he said.

Finally, he was surprised by how much cheaper their solution is compared to the alternatives; they expected the cost to be less, but the projections show it to be more expensive at the start, but decreasing (on a cost per device per year measure) over time to a much lower cost than alternatives. No actual numbers were shown, but that is what Bose has calculated. All of that was "only possible through the flexibility of Kubernetes", Doyle concluded. In the Q&A, O'Mahony noted how happy Bose is with this work and that moving to Kubernetes is a "really good choice" for the company.

A YouTube video of the presentation is available.

[I would like to thank LWN's travel sponsor, The Linux Foundation, for assistance in traveling to Seattle for KubeCon NA.]