An introduction to OpenStack [LWN.net]

By Jake Edge
June 5, 2013

CloudOpen Japan 2013

In a CloudOpen Japan talk that included equal parts advocacy and information, Rackspace's Muharem Hrnjadovic looked at OpenStack, one of the entrants in the crowded open source "cloud" software derby. In the "tl;dr" that he helpfully provided, Hrnjadovic posited that "cloud computing is the future" and that OpenStack is the "cloud of the future". He backed those statements up with lots of graphs and statistics, but the more interesting piece was the introduction to what cloud computing is all about, as well as where OpenStack fits in that landscape.

Just fashion?

Is "cloud computing" just a fashion trend, or is it something else, he asked. He believes that it is no mere fashion, but that cloud computing will turn the IT world "upside-down". To illustrate why, he put up a graph from an Amazon presentation that showed how data centers used to be built out. It was a step-wise function as discrete parts of the data center were added to handle new capacity, with each step taking a sizable chunk of capital. Overlaying that graph was the actual demand for the services, which would sometimes be above the build-out (thus losing customers) or below it (thus wasting money on unused capacity). The answer, he said, is elastic capacity and the ability to easily increase or decrease the amount of computation available based on the demand.

There are other reasons driving the adoption of cloud computing, he said. The public cloud today has effectively infinite scale. It is also "pay as you go", so you don't have sink hundreds of thousands of dollars into a data center, you just get a bill at the end of the month. Cloud computing is "self-service" in that one can get a system ready to use without going through the IT department, which can sometimes take a long time.

Spikes in the need for capacity over a short period of time (like for a holiday sale) are good uses of cloud resources, rather than building more data center capacity to handle a one-time (or rare) event. Finally, by automating the process of configuring servers, storage, and the like, a company will become more efficient, so it either needs fewer people or can retrain some of those people to "new tricks". Cloud computing creates a "data center with an API", he said.

OpenStack background

There are lots of reasons to believe that OpenStack is the cloud of the future, Hrnjadovic said. OpenStack has been called the "Linux of the cloud" because it is following the Linux growth path. In just three years, support for OpenStack from companies in the IT sector has "exploded". It was originally started by the US National Aeronautics and Space Administration (NASA) and Rackspace, though NASA eventually withdrew because OpenStack didn't fit with its organizational goals. When that happened, an independent foundation was created to "establish a level playing field". That made OpenStack into a credible project, he said, which helped get more companies on board.

The project is "vibrant", with an active community whose size is "skyrocketing". The graph of the number of contributors to OpenStack shows the classic "hockey stick" shape that is so pleasing to venture capitalists and other investors. Some of those graphs come from this blog post. There were 500+ contributors to the latest "Grizzly" release, which had twice as many changes as the "Essex" release one year earlier. The contributor base is a "huge force", he said; "think of what you could do with 500 developers at your disposal".

Where do these developers come from? Are they hobbyists? No, most of them are earning their paycheck by developing OpenStack, Hrnjadovic said. When companies enter the foundation, they have to provide developers to help with the project, which is part of why the project is progressing so quickly.

Another indication of OpenStack's momentum is the demand for OpenStack skills in the job market. Once again, that graph shows "hockey stick" growth. Beyond that, Google Trends shows that OpenStack has good mindshare, which means that if you want to use OpenStack, you will be able to find answers to your questions, he said.

OpenStack consists of more than 330,000 lines of Python code broken up into multiple components. That includes the Nova compute component, various components for storage (block, image, and object), an identity component for authentication and authorization, a network management component, and a web-based dashboard to configure and control the cloud resources.

There is an incubation process to add new components to OpenStack proper. Two features went through the incubation process in the Grizzly cycle and are now being integrated into OpenStack: Heat, which is an orchestration service to specify and manage multi-tier applications, and Ceilometer, which allows measuring and metering resources. Several other projects (Marconi, Reddwarf, and Moniker) are in various stages of the incubation process now. The project is "developing at a fast clip", Hrnjadovic said.

There are a number of advantages that OpenStack has, he said. It is free, so you don't have to ask anyone to start using it. It is also open source (Apache licensed), so you "can look under the hood". It has a nice community where everyone is welcomed. The project is moving fast, both in squashing bugs and adding features. It is written in Python, which is "much more expressive" than C or Java.

A revolution

"There are some early warning signs that what we have here is a revolution", Hrnjadovic said. Cloud computing is an equalizer that allows individuals or startups to be able to "play the same games" as big companies. Because it has a low barrier to entry, you can "bootstrap a startup on a very low budget". Another sign that there is a revolution underway is that cloud computing is disruptive; the server industry is being upended. He quoted Jim Zemlin's keynote that for every $1 consumed in cloud services, there is $4 not being spent on data centers. Beyond that, there is little or no waiting for cloud servers, unlike physical servers that need to be installed in a data center, which can take some time. Lastly, cloud technologies provide "new possibilities" and allow doing things "you couldn't do otherwise".

In the face of a revolution, "you want to be on the winning side". Obviously, Hrnjadovic thinks that is OpenStack, but many of his arguments in the talk could equally apply to other open source cloud choices (Eucalyptus, CloudStack, OpenNebula, ...).

These days, everything is scaling horizontally (out) rather than vertically (up), because it is too expensive to keep upgrading to more and more powerful servers. So, people are throwing "gazillions" of machines—virtual machine instances, bare metal, "whatever"—at the problems. That many machines requires automation, he said. You can take care of five machines without automating things, but you can't handle 5000 machines that way.

Scaling out also implies "no more snowflakes". That means there are no special setups for servers, they are all stamped out the same. An analogy he has heard is that it is the difference between pets and cattle. If a pet gets injured, you take them to the veterinarian to get them fixed, but if one of a herd of cattle is injured, you "slaughter it brutally and move on". That's just what you do with a broken server in the cloud scenario; it "sounds brutal" but is the right approach.

Meanwhile, by picking OpenStack, you can learn about creating applications on an "industrial strength" operating system like Linux, learn how to automate repetitive tasks with Chef or puppet, and pick up a bit of Python programming along the way. It is a versatile system that can be installed on anything from laptops to servers and can be deployed as a public or private cloud. Hybrid clouds are possible as well, where the base demand is handled by a private cloud and any overage in demand is sent to the public cloud; a recent slogan he has heard: "own the base and rent the spike".

Hrnjadovic finished with an example of "crazy stuff" that can be done with OpenStack. A German company called AoTerra is selling home heating systems that actually consist of servers running OpenStack. It is, in effect, a distributed OpenStack cloud that uses its waste heat to affordably heat homes. AoTerra was able to raise €750,000 via crowd funding to create one of the biggest OpenStack clouds in Germany—and sell a few heaters in the deal.

He closed by encouraging everyone to "play with" OpenStack. Developers, users, and administrators would all be doing themselves a service by looking at it.

[I would like to thank the Linux Foundation for travel assistance to Tokyo for CloudOpen Japan.]

Index entries for this article
Conference	CloudOpen Japan/2013

to post comments

Does the "Cloud" run on physical servers in datacen

Posted Jun 6, 2013 7:20 UTC (Thu) by bourbaki (guest, #84259) [Link] (32 responses)

> Overlaying that graph was the actual demand for the services, which would sometimes be above the build-out (thus losing customers) or below it (thus wasting money on unused capacity). The answer, he said, is elastic capacity and the ability to easily increase or decrease the amount of computation available based on the demand.

So with the "Cloud", you can increase or decrease your computing power without having hardware servers idling in datacenters, ready to take this new charge ?

In the whole article, it looks like the speaker believes that the "cloud computing" resources are made up of thin air...

What am I missing ?

Does the "Cloud" run on physical servers in datacen

Posted Jun 6, 2013 7:59 UTC (Thu) by dlang (guest, #313) [Link] (29 responses)

someone still has to predict the future load and build out accordingly. That someone has to charge you for system that they built that nobody is using. They also get to charge you enough to make a profit.

Now, if you have a bursty load so that you can turn off a large percentage of your systems for a large percentage of the time, it can be far cheaper to pay like this and let someone else who has bursts of load at a different time than you do pay for another chunk of the same systems.

and for a small business, it gives the ability to scale up rapidly based on usage. So if you create something that's wildly popular, you can use your income from your first users to pay for the servers for you next users and ramp up in a matter of minutes to hours rather than weeks as you order and rack new systems

of course, this assumes that you have done the extra engineering work to make sure that your application can actually scale horizontally. This is not trivial for anything other than the most trivial applications (but fortunately there really are a lot of those :-) so it's very possible that you end up in a situation where you would like to add another hundred systems, but doing so won't actually help (due to things like syncing data between them, or bottlenecks to back-end resources)

Also, running many small systems can be less efficient than running fewer larger systems.

with lots of small systems, you pay for the memory/cpu to run a copy of the kernel and the system daemons for every node.

you can also run into problems keeping all of the nodes consistent (when someone changes something, do you have to distribute that change to all other nodes?) This is a larger version of the locking problem that you can have on a huge single node between many threads. But all your communications are much slower.

cloud computing has some places where it's a huge win, but unfortunately comments like "for every $1 consumed in cloud services, there is $4 not being spent on data centers" can be very misleading and cause people to make bad decisions based on bad information.

I've seen companies look at their internal IT budget that includes sysadmins, security, audit, etc salaries and compare it with Amazon's list prices for hardware and decide that they can save a fortune by switching everything to AWS. But this comparison doesn't recognize the fact that they are still going to need sysadmins, security, auditors, etc.

Yes, switching how you build systems to have the builds be automated, having all your systems be the same, etc are wins (some of them gigantic wins compared to how companies are building and managing servers now), but you don't have to switch to the cloud to get those wins, you can change your processes in your existing datacenters and get the benefit.

A few years ago, everyone was talking about how virtualizing your systems was such a huge win, companies spent millions doing so. But if they just switched to running on virtual servers without changing how they built and administered the servers, they found that their problems got larger instead of smaller.

For example, if you are building and patching each system manually, going from 100 bare metal systems to 100 bare metal systems hosting 1000 virtual systems makes your job MUCH harder, you need to change so that you aren't manually building and patching your systems, and at that point the difference between doing so on bare metal and doing so on virtual machines really isn't that large.

Cloud computing is the next step in this process

The hype also ignores the additional complexity (and therefor costs) related to cloud computing

The fact that security when running in someone else's datacenter is more complicated than when running in your own datacenter. In your own, you can protect networks instead of individual systems.

The fact that you have significantly less reliability for individual components.

the fact that there are things you are just not allowed to do (broadcasts for example)

and the fact that there are frequently hidden performance issues (disk and network latency tends to be significantly higher in a cloud environment than in most datacenters.

none of these things mean that you can't run something in a cloud environment, it just means that it may take more effort and/or more systems to do so.

Given the number of shortcuts and "we'll implement that later" delays that happen in normal software development, it's easy to be burned if you rush and don't have the experience.

Early Adopters tend to be the cream of the crop, able to deal with all sorts of special cases and adapt rapidly, but that doesn't mean that the main body of companies are going to be as successful as the early adopters were, even with the benefit of the better tools. They just don't have the experience and skill to deal with everything that the early adopters can.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 7, 2013 23:21 UTC (Fri) by giraffedata (guest, #1954) [Link] (26 responses)

The speaker's explanation of why cloud computing is great is actually a sales pitch for potential cloud customers, rather than an explanation of why cloud technology works in general. I hope he didn't present it as the latter, because if so, he indeed missed the other side of the question: why doesn't the pain spared the cloud user simply shift to the cloud provider?

In the example of the holiday peak, I'm kind of curious as to which of Rackspace's customers use fewer servers during holidays so that Rackspace can give those resources to the holiday retailers.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 7, 2013 23:33 UTC (Fri) by dlang (guest, #313) [Link] (25 responses)

well, financial services companies do far less business during the holidays, so there is some possible gain from that sort of thing.

But mostly the cloud providers just charge enough to cover provisioning for such peak times.

But they run a far more efficient organization than most IT departments, they aren't spending time manually installing the OS and applications on each system, maintaining old operating systems/software, etc. They leave such mistakes up to the users (who think that just because it's in the cloud they don't have to worry about anything.....)

Does the "Cloud" run on physical servers in datacenters

Posted Jun 8, 2013 19:15 UTC (Sat) by giraffedata (guest, #1954) [Link] (24 responses)

But mostly the cloud providers just charge enough to cover provisioning for such peak times.

To the extent that is true, the argument that by being a cloud customer, a retailer doesn't have to pay for a year's worth of computers when they are needed only during the holidays, doesn't wash. And that was the OP's point.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 8, 2013 19:50 UTC (Sat) by dlang (guest, #313) [Link] (22 responses)

Yep, if it looked like I was disagreeing with that point, it was due to poor wording on my part.

A few years ago I went and did some looking and found that to get equivalent performance to a cheap ServerBeach system in AWS, the break-even point was that if I needed the AWS system for 2 hours a day it was the same cost.

Now prices have shifted a bit, and this wasn't taking into account reserved instances and so on, so the numbers have probably shifted a bit.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 11, 2013 11:30 UTC (Tue) by massimiliano (guest, #3048) [Link] (21 responses)

the break-even point was that if I needed the AWS system for 2 hours a day it was the same cost.

Could you please explain this a bit better?

Specifically, does it mean that if you use the AWS machine for three hours per day it would cost less to own it (and pay the electrical bill) or does it mean something else?

Thanks!
_ Massimiliano

Does the "Cloud" run on physical servers in datacenters

Posted Jun 11, 2013 17:54 UTC (Tue) by dlang (guest, #313) [Link] (20 responses)

> Specifically, does it mean that if you use the AWS machine for three hours per day it would cost less to own it (and pay the electrical bill)

specifically, it cost less to rent it from a colo facility like ServerBeach and pay their profit margin than to run an equivalent server 3 hours a day in AWS.

now, as I said, this was a couple years ago, and Amazon has reduced their prices some since (but systems are more powerful today than they were then), and I was not accounting for reserved instances or spot pricing.

Remember that there are 8760 hours in a year (~720 in a month), a $99/month colo server is 0.1375/hour which is just over the cost of a M1 medium instance, which is a slowish single-core box with 4G of ram

a 4-core box with 16G of ram is nothing really special in a colo or datacenter, in AWS the rough equivalent (before you take into overhead that reduce the performance) will cost you ~360/month or ~4380/year. Even if you cut these numbers in half by using a reserved instance, that's still pretty expensive

Does the "Cloud" run on physical servers in datacenters

Posted Jun 11, 2013 21:49 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (19 responses)

You can get c1.xlarge (8 cores, 8Gb RAM) for 10 cents per hour if you buy a reserved node. With spot nodes you can usually go down to 5 cents per hour.

Technically, EC2 is more expensive than a no-name VPS hosting (and you're not going to get a dedicated server for $99 a month) but from my experience it's significantly more reliable. And also has very nice services like S3 (which is great for static files).

And the main advantage - you can get additional nodes extremely easy. And delete them even easier.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 11, 2013 22:35 UTC (Tue) by dlang (guest, #313) [Link] (18 responses)

remember that not all cores are the same, the cores for the c1.xlarge are slower than some of the other cores (2.5 vs 3.5 compute units per core)

The pricing I am seeing is 0.58/hour. the price you are seeing may be for a different availability zone, they vary. The pricing for reserved nodes depends on the type of reservation. I was using the ballpark figure of saving 50%, in extreme cases you can save up to 65%. Spot pricing allows you to save more, as long as it is acceptable to loose your system at high demand times.

8 core real servers start at <$200/month. I haven't bothered to jump through the hoops, but it used to be that you could get smaller servers starting at $99/month from Server Beach, Rackspace, etc. Taking a quick look at their sites, it looks like they won't give you pricing without involving a salesman, something I'm not going to do just to get numbers for a discussion here :-)

It makes sense that if you guarantee that you are paying for the system for three years, that you will get better pricing. This also shows that you really are paying for more than just the time that you use the server, you are also paying a share of the time they expect the server to be idle

Look, I'm not saying that AWS is horrible and should be avoided. I'm just disputing the "put it in the cloud and save tons of money" mantra. Sometimes it will save you money, sometimes it will cost you money.

First do the math and decide what the best answer for your business is. It may be worth spending more for the flexibility, it may not be.

> And the main advantage - you can get additional nodes extremely easy. And delete them even easier.

as long as the extra nodes are not reserved nodes.

at this point we are back to the higher, standard pricing.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 11, 2013 22:41 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (17 responses)

I've actually tried to get quotes for lowest-possible priced VPS and colocated servers. You're not going to get anything reliable enough for less than $200 a month.

> Look, I'm not saying that AWS is horrible and should be avoided. I'm just disputing the "put it in the cloud and save tons of money" mantra. Sometimes it will save you money, sometimes it will cost you money.
It almost always "save your money" now...

It's scary how much of a typical corporate datacenter Amazon can replace. They have storage, computing, archival storage, etc.

> at this point we are back to the higher, standard pricing.
Unless you need it for a couple of hours to handle peak loads or to run a large computation.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 0:16 UTC (Wed) by dlang (guest, #313) [Link] (16 responses)

I guess it depends on your definition of reliable. I don't see anything fundamentally wrong with the providers who charge less.

While Amazon does have some purchasing advantages due to it's size, the margins on hardware are rather small (single digit percentages), so it's not decisive. they all have to pay for the systems, the electricity, and the maintenance of the systems. To this figure, they then tack on their profit margin and the cost of whatever value-add that they provide over giving you the raw servers.

It's in the cost of this value-add that Amazon has an advantage due to their size. they are spending a lot more in developing and maintaining the software infrastructure, but they have a much larger number of servers to split this cost over.

But if you pay for only part of the time of the server, all of them have to pay their overhead for the full time of the server.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 2:18 UTC (Wed) by giraffedata (guest, #1954) [Link] (15 responses)

They then tack on their profit margin

The profit margin doesn't figure in. You pay that even if you do it all yourself.

In a competitive (non-monopolistic) situation, there is no real profit - products are sold at cost. What some accountings report as profit is actually the cost of the capital - what investors had to be paid to provide the money to build the data center. Even if you build your own data center, you have to buy the capital to do it - from a bank, bondholders, stockholders/owners, or whatever, and it will cost the same.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 3:51 UTC (Wed) by dlang (guest, #313) [Link] (14 responses)

you have obviously never run a business of any kind :-)

competition doesn't mean zero profit, it just puts a (soft) cap on the profit margin

But the more layers that there are who need to either make a profit, or pay the 'cost of capital' the higher the price will be.

this is part of the reason why the giant companies got so large, by eliminating the multiple layers of profit, they were able to lower their cost of the final product, either keeping all the profit themselves, or being able to lower the retail cost below what other, less integrated companies could afford.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 5:01 UTC (Wed) by nybble41 (subscriber, #55106) [Link] (2 responses)

The grandparent is correct; under ideal conditions, in a state of equilibrium, no business will have any significant _economic_ profit. Economic profit is defined in terms of opportunity cost; the accounting profit you refer to is measured by revenues and expenditures. The accounting profit margin has to be measured against other potential investment opportunities to determine whether a particular business is a _good_ investment; economic profit already takes these alternatives into account.

If there were an economic profit to be had in any particular business, then by definition the market would not be in a state of equilibrium. Capital would be reallocated toward that area until balance was achieved and the expected marginal return no longer exceeded the opportunity cost.

In practice economic profit still tends toward zero, but uncertainty and constantly changing conditions ensure that it fluctuates (with a wide margin of error) rather than settling there. You see a few outliers with a short-term profit or loss, but the low-risk businesses around for the long term are all about equal, and tend toward the market rate of return on capital (the "risk-free" interest rate adjusted for changes in the supply of currency). This rate of accounting return is equivalent to zero economic return.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 5:57 UTC (Wed) by dlang (guest, #313) [Link] (1 responses)

> under ideal conditions, in a state of equilibrium

spoken like a true Economist (no insult to you intended, to "Economists" there is some insult intended :-)

however, in the real world there are not ideal conditions, and no field of business is in a state of equilibrium.

you are also ignoring the fact that businesses convert labor into money, including the efforts of the owners.

So business owners who start off with little money earn this fictional "cost of capital" on their time and effort, without necessarily having the capital to start with.

There is also the matter of convenience and trust. These are not things that can be measured precisely (and they differ from person to person) but they can significantly change the amount of profit that a company can make.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 15:57 UTC (Wed) by nybble41 (subscriber, #55106) [Link]

> however, in the real world there are not ideal conditions, and no field of business is in a state of equilibrium.

Actually, I addressed this point. Even under non-ideal conditions, the economic profit tends toward zero. It just never reaches zero and stays there.

> you are also ignoring the fact that businesses convert labor into money, including the efforts of the owners.

No, I'm not. The labor invested into the business is part of the opportunity cost. The same capacity for labor could be put to use in another line of work, or in leisure. Accounting profit does not consider the value of the owners' labor; economic profit does. This is part of the reason accounting profit overstates the business's actual profitability.

> There is also the matter of convenience and trust. ... they can significantly change the amount of profit that a company can make.

Sure, but there is a cost to providing convenience and trust. Some of that cost may be measured and quantified; the rest is not factored in to accounting profit, again exaggerating the overall profitability of the venture. Economic profit considers the total cost.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 5:43 UTC (Wed) by giraffedata (guest, #1954) [Link] (10 responses)

But the more layers that there are who need to either make a profit, or pay the 'cost of capital' the higher the price will be.

This actually gives us the proof of the theorem that in a competitive, free, and efficient market, there is no profit. If each stage in the supply chain withdrew value from it, then the most profitable supply chain would be the shortest. Companies would be striving to "cut out the middleman." But the fact is that the middleman always contributes services worth whatever his "markup" is, and in modern times, we see supply chains growing ever longer. The successful companies are the ones who outsource everything they can and use their own capital to grow horizontally within a thin layer of the stack.

The total amount of capital throughout the supply chain is the same whether you divide it into 10 stages or one, so whatever amounts you call profit add up to the same final cost at the end of the chain.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 12:34 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (9 responses)

> This actually gives us the proof of the theorem that in a competitive, free, and efficient market, there is no profit.
You forgot to add "at the equilibrium conditions". Which doesn't happen because we (as a civilization) experience an economic growth. That in turn creates imbalances and this allows the profit to exceed the breakeven point.

Once the growth stops, then yes - there'll be no profit.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 12:49 UTC (Wed) by massimiliano (guest, #3048) [Link] (6 responses)

I would have never thought that a simple clarification on a price comparison could spawn such an interesting thread about economy... but of course this is LWN!

Once the growth stops, then yes - there'll be no profit.

So, is it for this reason that when the growth stops we have a crisis, and there are movements stating that our economic system is flawed because a perpetual growth is not possible and-or sustainable?

Cheers,
_ Massimiliano

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 13:12 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

Well, all exponential growth is unsustainable in the end. But you don't need large growth for our economic system to work, just a couple percents a year is enough. And this can be sustained for at least a couple hundred years more.

After that? Who knows, maybe we'll live in a post-scarcity society or as uploaded electronic intellects.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 13:36 UTC (Wed) by khim (subscriber, #9252) [Link] (4 responses)

So, is it for this reason that when the growth stops we have a crisis, and there are movements stating that our economic system is flawed because a perpetual growth is not possible and-or sustainable?

Yup. Growth actually stopped a few years back and now we are looking on the world where most businesses actually produce negative profits.

I've already tried to discuss that: you can easily find all the times where expansion happened and new markets arrived on a graph. But discussion quickly derailed because people tend to state all the right facts then carefully ignore the implications.

Here is an example:

Slashing the cost of a $500 widget to $50 immediately opens up a lot of internal market.

Think about it: if wages are going down 10x when gadgets price goes down 10x then market actually shrinks. Ok, it does not work. So we have wages constant and price of gadgets are 10x cheaper. Cool - but now we have all the businesses deep in red. We can issue the debt (government debt or private debt) and compensate this (which is exactly what happens today around the world) - but how long it'll work? The answer: 5, may 10 years if you are lucky. Not a long term solution. Now we find the real answer: if gadgets price goes down and wages are kept up then "internal market" grows only if you move productions of all these gadgets overseas. Where you can easily create new debt. Which explains the paradox:

In fact, US exported hardly anything during the first post-war years (hardly anybody in Europe could buy imported stuff).

If you'll think about it then it's obvious that main export product in the first phase of expansion should be green paper with some watermarks - and this is exactly what happened.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 14:25 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

>If you'll think about it then it's obvious that main export product in the first phase of expansion should be green paper with some watermarks - and this is exactly what happened.
That's not a place to discuss it, but this is incorrect. Total 'export' of US currency is just a small fraction of US GDP (I think around 0.2% or about 50-100 billion).

Anyway, growth has not stopped, but politicians' heads are firmly stuck in their nether places. So the Depression goes on, doing its job (depressing the economy). The current job situation is not a fundamental change, it's just an evidence of mismanagement.

And if you want a test case - look at Japan. They are on the road back to robust growth.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 14:59 UTC (Wed) by khim (subscriber, #9252) [Link] (2 responses)

Total 'export' of US currency is just a small fraction of US GDP (I think around 0.2% or about 50-100 billion).

And again you ignore the facts. Yes, US mainly exports not physical currency, but bonds. But that's now - at the final stages of game. In the first stages it actually loaned money, then have gotten actual physical goods back. The first dose is for free :-).

It's not as if that's some new or unknown process - that's just how explansion to the new territory happens. Only half-millenia ago conquistadors actually brought mirros and glassbeads while half-century ago just some papers were used.

The current job situation is not a fundamental change, it's just an evidence of mismanagement.

It's not a fundamental change, I agree. That happens every time all the available territories are fully exploited. The only difference: this time there are no new territories coming. Globe is more-or-less completely covered.

And if you want a test case - look at Japan. They are on the road back to robust growth.

Really? What kind of growth? In debt? Yes, here they are leaders, but not the number one. Greece won this race few years back and now reap all the benefits: over 25% unemployment rate and all other nice side benefits. Japan is next, but it's quite possible that some small countries will outrun it on the road to ruin (like Cyprus did recently). Of course they are not members of EU thus they will have more choices but not that many.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 15:12 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

>And again you ignore the facts. Yes, US mainly exports not physical currency, but bonds. But that's now - at the final stages of game. In the first stages it actually loaned money, then have gotten actual physical goods back. The first dose is for free :-).
Nope. You do.

The notion that US exports 'money' is one of those 'simple, neat and wrong' answers. The reality is much more complicated.

US had lived with a near zero trade balance for the most part of the last 50 years. And it had a really good economic growth. And what's more, the peaks of this growth actually fall in the time when US had had minimal import _and_ export (btw, that's a coincidence, not a causal relation).

>Really? What kind of growth? In debt? Yes, here they are leaders, but not the number one.
Want to bet? I propose the following: during the next 2 years Japan GDP will grow at least 2% each year above the level of inflation. There'll be no additional significant unemployment and there'll be no debt crises (even though the debt level will actually grow).

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 17:31 UTC (Wed) by khim (subscriber, #9252) [Link]

The notion that US exports 'money' is one of those 'simple, neat and wrong' answers. The reality is much more complicated.

Well, sure. There are some pretty complex structures involved. But that's the core engine which moves the whole mechanism.

And what's more, the peaks of this growth actually fall in the time when US had had minimal import _and_ export (btw, that's a coincidence, not a causal relation).

No, it's not a coincidence. If your economic machine is built around building debt then most of growth will happen when new territories are just added and new debt attached to these new people is rapidly created. This is when you actually have minimal import and export because there debt machine uses future imports and exports as collateral. Once you start to actually grow import and export future prospects become worse and worse and when it becomes obvious that future imports and exports are overvalued growth stops. This is similar process to the Apple's stock growth and fall: it reached maximum and started going down not when Apple's actual sales reached maximum but when expectation reached maximum.

I propose the following: during the next 2 years Japan GDP will grow at least 2% each year above the level of inflation. There'll be no additional significant unemployment and there'll be no debt crises (even though the debt level will actually grow).

This will be just a pointless gamble. Two factors are known:
1. Expansion of markets is finished and now we are in the purely speculative pyramid-style phase of the game.
2. This particular pyramid is extremely slowly-moving one (note how you mention "2% each year", not "20% each year" or "200% each year").

These two factors mean that this whole thing will fall apart at some point, but that timescale for this process is measured in decades, not mere years. And since this is not a zero-sum game yet (when Greece collapses Japan and US pyramids get new lease on life, e.g.) it's practically impossible to say which countries will collapse when. By all rights Japan should have collapsed before Greece, but because most of it's debt is issued by large internal creditors and because Japan is not a member of EU Greece went under first. Will Italy or Japan collapse first? Nobody knows: again, objectively purely in economic sense Japan's situation is much worse, but Italy's hands are tied by EU.

When I complained last time about the lack of detailed theory I was not pretending: over century old Mark's theory is still the most potent theory which describes this process, but it is over century old and thus it obviously misses some modern trends. Which is a pity. We desperately need an adequate theory to make this destructive process less disastrous, but so far most "professional" economists refuse to even acknowledge that process is underway.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 15:55 UTC (Wed) by giraffedata (guest, #1954) [Link] (1 responses)

You forgot to add "at the equilibrium conditions". Which doesn't happen because we (as a civilization) experience an economic growth. That in turn creates imbalances and this allows the profit to exceed the breakeven point.

That was included in "efficient." A (theoretical) efficient market reaches equilibrium instantly.

Coming back to analyzing whether it should be cheaper to rent from ServerBeach of own your own, if one believes that inefficiency/growth means we have to add in ServerBeach's profit, one should indicate what particular costs of the inefficiency that profit represents.

All I'm saying is that in making a comparison like this, if you add in one of the supplier's book profits, you'll be double counting. Just add up everyone's costs, not counting transfers from one to another, and compare the totals.

(I say should be cheaper above to distinguish from the alternative method of looking at actual price lists, in which case book profits are again irrelevant).

Does the "Cloud" run on physical servers in datacenters

Posted Jun 12, 2013 16:06 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

That depends on the definition of 'efficient'. Real market is definitely not perfect, because doing stuff like building new plants or training workers takes time. Even simple money transactions are not instantaneous.

> Coming back to analyzing whether it should be cheaper to rent from ServerBeach of own your own, if one believes that inefficiency/growth means we have to add in ServerBeach's profit, one should indicate what particular costs of the inefficiency that profit represents.
For individuals it's a more complicated calculation. Any company consists of humans who have finite attention spans.

For example, if your core competency is at producing welded tubes then do you really want to spend your time and energy on maintaining your own Exchange installation? Of course, high-level management won't necessarily know all the details of installation, but they still would have to understand proposals, approve budgets, etc.

And it's compounded by the fact that the 'classic' IT infrastructure still is not at the 'commodity' level like, say, HVAC. It still requires a fair amount of attention to details even for fairly simple cases.

Then there's a question of reliability - it's much easies to guarantee it if you have 100000 servers than if you have just 2.

And so on.

Does the "Cloud" run on physical servers in datacenters

Posted Jun 8, 2013 20:25 UTC (Sat) by dark (guest, #8483) [Link]

Well... a cloud provider does have the option of powering down unused machines until they're needed for peak capacity. With power being an increasing part of the cost of servers, this can be a significant savings.

For small customers the savings probably wouldn't be worth the overhead of setting up the infrastructure for managing this. Besides, I don't know any data center that would actually give a discount for powered-down servers :) Maybe if it catches on...

Also, even if small customers could do this with their own machines, a cloud provider could do it more aggressively because their total load is more predictable.

Does the "Cloud" run on physical servers in datacen

Posted Jun 8, 2013 4:26 UTC (Sat) by eternaleye (guest, #67051) [Link]

One other use case: Build your own datacenter using OpenStack (allowing fast reallocation of resources), and if you're over capacity farm out to the cloud.

If you remain over capacity for a while, scale your own datacenter.

Does the "Cloud" run on physical servers in datacen

Posted Jun 10, 2013 20:41 UTC (Mon) by yodermk (guest, #3803) [Link]

Hi, if I can just address a bit of this. (Disclaimer: I work for Rackspace.)

> The hype also ignores the additional complexity (and therefor costs) related to cloud computing

> The fact that security when running in someone else's datacenter is more complicated than when running in your own datacenter. In your own, you can protect networks instead of individual systems.

Well Openstack Quantum (networks) does allow you to isolate instances in their own VLAN, just as you might do in your own DC. You can have firewall instances between them and the outside, running appropriate firewall rules.

> The fact that you have significantly less reliability for individual components.

Not sure what you're referring to here. But it should be noted that one of the core pillars of developing applications in the cloud is that you should design for failure. Not that the cloud is inherently faulty, but things can and do go wrong, whether it be in your own DC or in a public cloud. Netflix has its famous "chaos monkey" which randomly kills things
just to be sure its application is built with fault tolerance in mind. If something goes wrong, something should detect that and use the cloud API to kill the faulty instance and re-create it. Obviously you need enough load balanced nodes that this shouldn't be noticed by the user.

> the fact that there are things you are just not allowed to do (broadcasts for example)

Quantum Networks gives you an isolated layer 2 network that allows broadcasts.

> and the fact that there are frequently hidden performance issues (disk and network latency tends to be significantly higher in a cloud environment than in most datacenters.

That is something to be aware of for some apps - particularly disk latency (not sure about network latency). You have many tenants on a server competing for the disk I/O and scheduling resources of the host, so indeed I/O heavy applications such as databases can perform poorly in the cloud. That is one reason why some data heavy applications may want to make use of both dedicated servers for databases and cloud servers to serve their application. Of course "average" sites, which are not constantly pushing hardware limits, will work just fine with their DB in the cloud.

The cloud isn't perfect for everything, of course, but for a lot of organizations it is a real win. Over time as these issues are further addressed the percentage of applications where it is a good fit will only increase.

Does the "Cloud" run on physical servers in datacen

Posted Jun 6, 2013 12:29 UTC (Thu) by pboddie (guest, #50784) [Link]

Even though there are likely to be architectural differences I'm ignoring, the terms "grid computing" or "utility computing" probably express the "limitless resources" concept used by the speaker better than the "cloud" marketing term: people don't consider where things like electrical power or water come from, but merely that they can demand more or less at various times and their needs are magically met.

Of course, this all requires robust infrastructure so that more of the resource is "on tap", and one might argue that people should be slightly more aware of where these resources come from and how they are produced in the first place.

Does the "Cloud" run on physical servers in datacen

Posted Jun 6, 2013 18:08 UTC (Thu) by dvdeug (subscriber, #10998) [Link]

Nitrogen/oxygen air isn't free to make; you've got to have a star fusing all that hydrogen into heaver elements and then you've got to extract it from the star (usually by blowing it up) and then you've got to actively keep the oxygen in the atmosphere because it wants to combine with carbon and other elements.

So yeah, it's just like thin air; it's incredibly hard to make and keep available, but doing so just isn't your problem.