|
|
Log in / Subscribe / Register

Borg, Omega, and Kubernetes (ACM Queue)

Five Google developers share the lessons from ten years of container development in this ACM Queue article. "To cope with these kinds of requirements, configuration-management systems tend to invent a domain-specific configuration language that (eventually) becomes Turing complete, starting from the desire to perform computation on the data in the configuration (e.g., to adjust the amount of memory to give a server as a function of the number of shards in the service). The result is the kind of inscrutable 'configuration is code' that people were trying to avoid by eliminating hard-coded parameters in the application's source code. It doesn't reduce operational complexity or make the configurations easier to debug or change; it just moves the computations from a real programming language to a domain-specific one, which typically has weaker development tools (e.g., debuggers, unit test frameworks, etc)."

to post comments

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 3, 2016 15:52 UTC (Thu) by sgourichon (guest, #88605) [Link] (11 responses)

Interesting. The "configuration is code" slippery slope is an interesting parallel to the "html document is html+js code" slippery slope that clobbers accessibility, search engine visibility and ultimately causes people to generate "SEO version" of documents to recover something that, until people killed it, was granted to begin with (e.g. https://flippingbook.com/features/digital-publication/seo ).

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 3, 2016 17:25 UTC (Thu) by ledow (guest, #11753) [Link] (10 responses)

But the fact that they require JS in the first place quite leads me to believe that maybe HTML on it's own isn't sufficient. For instance, no real kind of movement, or reaction to user input, is really possible in base HTML. And like how HTML wasn't sufficient and required CSS to make sensible use of it.

If you aren't able to get done what you want to get done, you'll invent features or import them to do it. That plain HTML cannot do those things might be a misuse of HTML, an inadequacy in the specification, or not even coverage of the specification by browsers. But people still want to do it so rather than let them bodge a way to do it, maybe we should be providing languages that are already complete and planned so the planned route is used rather than a third-party add-on.

Of course, just "being" HTML5 isn't the be-all-and-end-all and quite often in my industry (education), I see HTML5 touted as the Flash/Java/Plugin replacement of today. Sure, it can do more, but can it do all that those plugins did? And if it can't, either users or content creators will complain.

And then you get into things like Node.js, emscripten, etc. where - actually - even Javascript isn't sufficient and people ARE using it to recompile C99 code written in SDL to display in a browser window without requiring plugins (replacing things like Unity web plugin, for instance). I've done it. It's magical, miracle-performing, time-saving fabulousness. But still the inner me reels at the thought that such things are necessary, or even so easy to do and still be held as "HTML5 compliant".

A part of me just wants a universal language. That a browser can understand the same as a computer. Vector graphics, for instance, and well-defined bitmap routines. Scalable interfaces. And controls on what portion of the functions the browser will allow or deny rather than having to have OpenGL and WebGL and artificial constructs to impose those limitations. We can't put insert, say, a native-looking application command button, reliably into a web page. So web and desktop-application will always be different, and maybe that's correct but I guarantee that SOMEONE somewhere just wants to recompile their desktop client application into a web client that looks identical.

But there is no simple answer. The less features you give, the more inventions are tacked on, in several different, competing and non-standard ways. The more features you give, the more complex the interaction and you lose the purity of what you were trying to achieve (e.g. have all text searchable or accessible, etc.).

But with the world moving towards virtual machines, containers, apps, etc. I can't see how each webpage being considered an "app" with just resource limits but full capability (subject to operating system security) is really a bad thing. The websites that want to do fancy stuff can. Everyone else can just bundle a basic "generic HTML5 display app" which just renders their old website. That's where Chromebook is basically going. It's where most mobile apps already are. Maybe we should just embrace it and spend time making sure the sandbox is secure and the "instruction set" used consistent and reliable instead of trying to impose perfect purpose-built languages for each part that will all be out of fashion in a year anyway, just as browsers start to support them fully.

I'd honestly prefer www.mybank.com to be a standardised VM-like contained app that had the run of a general purpose instruction set in a limited environment than have to force them to let me use a vaguely modern browser, or stick plugins into IE (still need to do that for my employer's business banking!), etc. So long as I can control resource limits per page, it would probably work out better and if they want to suck in their own proprietary display library, or an Adobe-Flash like library from a remote website, that seems much more sensible than trying to make ME install all those things and keep them up to date forever more and then blaming me when I can't see the website.

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 1:16 UTC (Fri) by raven667 (subscriber, #5198) [Link]

I think you are probably right and are not the only person who thinks this way, take a look at Atlantis and James Mickens at MS Research, they've been working on the exact thing you've specified, rethinking the browser as a VM container manager with HTML/JS/CSS being the degenerate application.

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 12:39 UTC (Fri) by Wol (subscriber, #4433) [Link] (8 responses)

> If you aren't able to get done what you want to get done, you'll invent features or import them to do it. That plain HTML cannot do those things might be a misuse of HTML, an inadequacy in the specification, or not even coverage of the specification by browsers. But people still want to do it so rather than let them bodge a way to do it, maybe we should be providing languages that are already complete and planned so the planned route is used rather than a third-party add-on.

So when you design a hammer, you should build in the ability to drive screws?

I'm sorry, but you're "fixing" a problem that should never have arisen!!! If you've got a job to do, don't pick the wrong tool and then complain that the result is a mess!!!

Plain HTML was intended as a simple *DISPLAY* format. Part of its design brief was that *ALL* formatting should be done on the *CLIENT* device because the client knew what it was capable of. Idiotic web designers thought they knew better, and pushed in features so the server could control how stuff was displayed, which quickly lead to unreadable web pages because the server said "minimum width 800 pixels" on a screen which was 640 pixels wide. And so on.

And pdf suffers from exactly the same problem - it is a Page Description Language, meant for exactly the opposite of html - it dictates exactly how the page is laid out, so you find a device that can print it and you're guaranteed to get exactly the same result regardless of printer (well, almost :-) Except it's now being used for data entry ...

> If you aren't able to get done what you want to get done, you'll invent features or import them to do it. That plain HTML cannot do those things might be a misuse of HTML, an inadequacy in the specification, or not even coverage of the specification by browsers. But people still want to do it so rather than let them bodge a way to do it, maybe we should be providing languages that are already complete and planned so the planned route is used rather than a third-party add-on.

Single use is a VALUABLE feature. "languages that are already complete" are a ******* liability!!! You shouldn't be inventing or importing features, you should be using a different ******* tool!!!

Unfortunately, this just seems endemic across the industry - "the tool I understand doesn't do what I want, so you need to add completely irrelevant (to the original task) functionality, so I can use the wrong tool for the job".

How many people use Excel as a database? How many people use Word tables as a spreadsheet? For f**** sake my wife uses Word as a graphics editor!!!

Personally, I would like an HTML that actually does the job it was originally intended to - the tool has been subverted so much that it is now crap at doing what it was originally meant for - DISPLAYING INFORMATION ON ANY DEVICE. (How many web pages do you regularly come across that won't print? Most of them?)

Cheers,
Wol

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 16:59 UTC (Fri) by flussence (guest, #85566) [Link] (7 responses)

>Idiotic web designers thought they knew better, and pushed in features so the server could control how stuff was displayed, which quickly lead to unreadable web pages because the server said "minimum width 800 pixels" on a screen which was 640 pixels wide. And so on.

If only the amateur-hour web 2.0 design world was that generous! 800 pixel width hasn't been usable for years; even 960px (1920/2) isn't enough in a lot of places. I've surrendered to using an awkward window manager arrangement to give the browser 1000px of width, but even then I've noticed the trademark Horizontal Scrollbar Of Clueless Design made a comeback on Github.

And the bitter irony in all this is that the same blingy, tacky premade templates... sorry, "web frameworks"... that promise consistency and usability across devices are the ones *destroying* it.

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 19:32 UTC (Fri) by niner (guest, #26151) [Link] (6 responses)

Another irony is that 15 years ago, when CSS became widely supported, we carefully crafted websites so that they would be usable on different screen and window sizes. Then came the web "designers" claiming that everyone is on 1024x786 or 1280x1024 and browses in a full screen Internet Explorer and made completely static layouts.

A decade later, those designers finally grokked that the web isn't static and claimed that they have found the solution with so called "responsive design" which is to carefully craft websites so that they would be usable on different screen and window sizes. And of course we have to admire them and treat them like this was something totally new...

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 20:13 UTC (Fri) by raven667 (subscriber, #5198) [Link] (5 responses)

I would call that a consequence of most of the people with any design experience were coming from the print media world, where there is hundreds of years of collective knowledge that computer scientists don't have, based on media that comes in fixed sizes.

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 5, 2016 13:27 UTC (Sat) by Wol (subscriber, #4433) [Link] (4 responses)

So why is it that web pages won't print?

Cheers,
Wol

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 5, 2016 13:52 UTC (Sat) by hummassa (guest, #307) [Link] (3 responses)

Care to elaborate? I print web pages all the time...

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 5, 2016 17:48 UTC (Sat) by kvaml (guest, #61841) [Link] (2 responses)

(interjecting)
I work on a website where many of the screens are meant to be printable. After the web designer finished mangling pages so that various fixed width elements no longer fit on a page, I had to introduce a new CSS sheet
<link media="print" href="/css/print.css" type="text/css" rel="stylesheet">
to undo the web designer's work and restore printability.

My wife often encounters print issues when attempting to print screens from shopping sites. Very little of the site actually makes it onto the printed page.

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 5, 2016 18:43 UTC (Sat) by hummassa (guest, #307) [Link] (1 responses)

Your interjection is welcome!

This seems to be one of those YMMV things... :D

As Chromium shows the thing to be printed before sending it to the printer, I probably make some cut-and-paste (sometimes combined with deleting parts on the DOM inspector thingy) before printing some pages, but I don't recall as of lately having to jump thru too many hoops before printing, hence my questioning.

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 5, 2016 19:44 UTC (Sat) by Wol (subscriber, #4433) [Link]

That was exactly what I meant - and I think shopping sites tend to be the worst.

The number of times I've hit "print", and about all that fits on the page is the header, the footer, and the sidebars. The stuff I *actually* *want* is all about three foot to the right of the left margin, and my paper is only eight inches wide ...

Cheers,
Wol

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 3, 2016 16:29 UTC (Thu) by niner (guest, #26151) [Link] (12 responses)

After all these years of hype, this article finally presents a real world use case where I really understand the advantages of containers. For a company like Google where there are probably thousands of applications with varying degrees of resource requirements in development all the time, resource utilization really must be a huge issue. Having a rather small and standardized interface between applications and the underlying host system allows for fully automated distribution of applications to the available hardware. Containers are this interface.

From an administration perspective they are also a clear boundary of responsibilities between the comparatively small team of system administrators who take care of hardware and the operating systems running on the bare metal and the thousands of application teams who are responsible for their respective workload. No central point involved in applications coming or going.

This also explains why to date I haven't found a single advantage containers would have for our shop where despite us being in the web business and us developing a lot of software, we'll soon be able to consolidate all our servers onto just one failover cluster consisting of two machines. Because affordable hardware is insanely powerful these days.

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 3, 2016 17:48 UTC (Thu) by farnz (subscriber, #17727) [Link] (5 responses)

It's a bit like the "Big Data" fad. Tools like Hadoop can handle incredible amounts of data on a cluster, but you probably don't need them. If you buy something like a Dell PowerEdge R920 to run your traditional tooling on then $150,000 gets you 3 TiB of RAM, $300,000 gets you a single server with 6 TiB of RAM; yes, that's a lot of money, but you'd also pay a lot of money for a Hadoop cluster that has 6 TiB of total RAM; you'd be looking at $100,000 for a cluster of 100 machines with a total of 6 TiB RAM. For most businesses, 1/3rd the capital cost for over 10x the running costs (the 6 TiB RAM server needs 3.3 kW peak power, the cluster needs about 40 kW peak power) and more complex software is not a good tradeoff.

That said, once you've got a dataset that's so large that 6 TiB isn't enough RAM for a working set, you start needing the cluster; if that's you in the foreseeable future, you might want to get going with Hadoop now. Most businesses, though, aren't looking at needing to keep over 6 TiB in RAM for the duration of processing - if your entire data set is under 1 TiB, you don't need Big Data yet.

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 3, 2016 18:08 UTC (Thu) by SEJeff (guest, #51588) [Link] (4 responses)

This so much! This is what I call the difference between what I call "medium" and "big" data. Medium data can fit on a single, albeit exotic machine in memory. Big data simply can't be processed or stored on a single machine. Also, 256G of ram isn't insane anymore and a 16 node hadoop cluster is a relatively small one now adays.

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 9:36 UTC (Fri) by farnz (subscriber, #17727) [Link]

I wouldn't call a PowerEdge R920 exotic, personally. It's just the highest end of Intel server - and is thus a direct drop in replacement from every perspective to the littler server you've already got for data processing. Similar applies to a big IBM Power server - it's like a little machine but more powerful.

Exotic would be something like an SGI UV of some description - custom hardware that needs specialist administration to keep it running.

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 12:42 UTC (Fri) by Wol (subscriber, #4433) [Link] (2 responses)

And if you use a *sensible* database, you don't even need to store it all in ram ... :-)

The only reason database servers need to be so big and powerful, is that relational is insanely wasteful of resources ... (Don't bite, you'll regret it :-)

Cheers,
Wol

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 14:17 UTC (Fri) by farnz (subscriber, #17727) [Link] (1 responses)

Even Pick has a non-zero working set size - and no usable data processing environment is so inefficient that a dataset 1/6th the size of RAM results in a working set size larger than RAM.

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 17:43 UTC (Fri) by Wol (subscriber, #4433) [Link]

Yep. Except Pick - for any given "real world" dataset - has a far smaller disk footprint than relational, and Pick was designed to retrieve data quickly from disk. It was designed - in the 60s - to treat the disk as virtual memory and access it as quickly and efficiently as possible.

To quote the Pick FAQ - "Relational optimises the easy task of finding data in RAM. Pick optimises the far harder task of getting into RAM in the first place."

Cheers,
Wol

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 3, 2016 18:37 UTC (Thu) by drag (guest, #31333) [Link] (4 responses)

The whole point is just to make managing applications easier on Linux.

Linux has massive capabilities for newtorking and all that jazz. Multiple addresses per physical port, for example. There is really no need to worry about having multiple applications wanting to use port 80 anymore. It's all a bit gimped by the lack of IPv4 address space, unfortunately. Other examples are things like cgroups, distributed files systems, layered file systems, namespace isolation, and so on and so forth. Containers represent a relatively easy way to take advantage of this. It's simply something that you can't do just with 'apt-get' or 'yum'. And it's easier then dealing with trying to use puppet or ansible to do everything.

Sure you can use VMs by themselves if you want, but really the point to all virtualization is really just to make it easier and cheaper to deploy and manage applications. That's really the entire point. People can go on about security or whatever, but I think anything other then 'cheaper and easier to run apps' is really a red herring.

Then there is the ability to just manage the entire datacenter through declarative configs and restful APIs. This is the 'devops' side of things is really about. No longer scripting is about stringing various shell commands together. The HTTP API is the new command line environment.

Just give me any 'cloud' you want.. whether it's something like Vmware, Ovirt/RHEV, or Openstack or some public cloud like Rackspace or Amazon I can write a python program with libcurl that can deploy a identical kubernetes cluster on any of them. I'll just keep a bunch of json files and the script in a git repository or something like that and it's not a big deal to setup a identical application environments regardless of the underlying hardware or whatever. The challenge just is to find what balance of managed services, cheapness, performance, and reliability you want.

It can just as easily be on a 2/3-node cluster in somebody's office closet or it could be Amazon. (provided the app devs are able to resist the temptation into buying into proprietary services that can only be obtained from a particular cloud vendor)

You'll see people go on with things like 'hyper convergence' and whatever... but the stuff that makes sense is really code for 'Lets get rid of Cicso, EMC, blade servers and all that 'big enterprise stuff' and replace it with cheap software running on cheap hardware'. (After all: 10GbE is enough for anybody, right?)

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 0:08 UTC (Fri) by SEJeff (guest, #51588) [Link] (3 responses)

*ahem* 40G is enough for anybody, right?

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 1:22 UTC (Fri) by raven667 (subscriber, #5198) [Link] (2 responses)

40 Gbe is already dead, the QSFP form factor and related electronics have had their respective ASICs basic data rate upgraded from 4x10Gbit channels to 4x25Gbit channels, making 100Gbe at the same cost and form factor as 40Gbe. You can now get 16-port or more 100Gbe switches for the same cost as 40G last year, not from the big brand name vendors though as they will take years to suck the money out of their customers while they trail the whitebox OEM vendors who just repackage Broadcom reference designs..

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 2:33 UTC (Fri) by SEJeff (guest, #51588) [Link] (1 responses)

I know, and use both 57G and 100G networks for $job. That was total sarcasm which you missed

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 4, 2016 18:48 UTC (Fri) by raven667 (subscriber, #5198) [Link]

> That was total sarcasm which you missed

Oh, well... carry on then. 8-)

Borg, Omega, and Kubernetes (ACM Queue)

Posted Mar 3, 2016 21:27 UTC (Thu) by diederich (subscriber, #26007) [Link]

Huge scale is one of the main advantages of this kind of focus on containerization.

The bigger advantage is about focusing on the application as the fundamental unit, instead of the OS. That's the big win, and it's a win at any scale, in my opinion.

I'm not saying that it's not possible to so manage in other ways, but for me, the natural fit has been this container-o-centric approach.


Copyright © 2016, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds