LWN: Comments on "The true costs of hosting in the cloud"

Comment about the human-operation cost

yshemesh — Wed, 29 May 2019 17:47:56 +0000

When referring to the human factor ("Staff - FTE"), you would also need 2-3 engineers to do 24x7 management of the cloud.
One more important point is that with Kubernetes and CNCF eco-system, you can minimize the amount of high-touch SRE engineers to minimum so you can gain even better competitive advantage with clouds.

The true costs of hosting in the cloud

Cyberax — Fri, 04 May 2018 20:54:13 +0000

Couldn't you just relaunch instances?

The true costs of hosting in the cloud

nicram — Thu, 03 May 2018 23:38:13 +0000

From my experience with AWS and Azure:
- System slow down to almost unusable state - 2 days for resolution very fast contact but slow fix
- Some ports become not accessible (changes in routers etc. infrastructure) - 4 days for resolution again fast contact.

They are not IT gods just highly trained monkeys with low salaries.

The true costs of hosting in the cloud

seneca6 — Thu, 08 Mar 2018 19:29:55 +0000

I second the call for looking at dedicated servers. Between colocating and cloud, it can be a very viable option, especially if all you want is some VMs or containers and you have no great need for scalability. Here in France, I'm often surprised at how much cheaper dedicated servers can be (with best-effort traffic included in the fixed monthly fee!).

Concerning communities and small-scale servers: Some years ago, tutorials for self-hosting your web site popped up all over the net; right now, where dedicated servers - but without RAID disks or service level agreement - are available for ridiculous prices, I'd love to see the same wave of tutorials for fail-over database clusters or distributed filesystems! Of course they exist, but it's still not trivial to set up such systems. Self-hosting -> self-clustering!

The true costs of hosting in the cloud

Wol — Thu, 08 Mar 2018 14:48:35 +0000

SUSE Linux?

:-)

Cheers,
Wol

The true costs of hosting in the cloud

Wol — Thu, 08 Mar 2018 14:46:42 +0000

> Do you think you'll be able to beat the economy of scale that AWS (or Azure) enjoys?

For somebody like the Canadian Government, setting up their own private cloud would probably be a very good idea ... :-)

Cheers,
Wol

The true costs of hosting in the cloud

gfernandes — Tue, 06 Mar 2018 16:49:05 +0000

>>And if you go to IBM for your software then you truly deserve what you'll get

Hear! Hear!
:)

The true costs of hosting in the cloud

Cyberax — Mon, 05 Mar 2018 02:18:35 +0000

Well, AWS has dedicated hosts and dedicated instances (two different products) that ensure that your VMs will not share the same physical hardware as other customers.

And if you go to IBM for your software then you truly deserve what you'll get.

The true costs of hosting in the cloud - cable cost

anarcat — Mon, 05 Mar 2018 01:26:41 +0000

The article says cable costs are the majority of the hardware costs in a data center. I think that's a mistake - it refers to an article that says the majority of network hardware cost is cables.

True, that's a typo on my part.

That said, I wasn't referring to the cost of actual cabling hardware, but more the human cost of managing all those cables and faults. Most cables are actually pretty cheap, really, unless you go to end-to-end fiber or something. The real cost is, again, labor: labeling, documentation, coloring, DoS attacks and who knows what the heck is happening on that network... ;)

The true costs of hosting in the cloud

giraffedata — Sun, 04 Mar 2018 23:40:31 +0000

I believe that is within IBM's range of products. IBM will sell you hardware and software to set up your own data center, will supply the labor to construct and/or operate your data center (running your own cloud or not), will rent you bare metal machines in an IBM data center, will rent you virtual machines in an IBM data center, or will sell you services such as a database running in an IBM data center.

The only thing I think is missing from this product spectrum is renting you space in an IBM data center to place your own equipment.

I've heard many times that people are willing to pay a premium to have their own data center because of the risk that some other tenant of a public cloud will hack them. It seems like a low risk to me, but then we have news like Spectre/Meltdown where ostensibly a program running in Company A's AWS virtual machine can see the data in Company B's AWS virtual machine, and I can believe people are willing to pay that premium.

The true costs of hosting in the cloud - cable cost

giraffedata — Sun, 04 Mar 2018 23:14:33 +0000

The article says cable costs are the majority of the hardware costs in a data center. I think that's a mistake - it refers to an article that says the majority of network hardware cost is cables.

The fastest network cables (which include transceivers) cost over a thousand dollars, so it's easy to believe that when you have lots of them for every switch the cable costs are the majority of the networking cost. But they don't outweigh the costs of the servers and other hardware in the data center.

The true costs of hosting in the cloud

giraffedata — Sun, 04 Mar 2018 23:08:31 +0000

The author touched briefly on "demand management" but that turns out to be a crucial deciding factor in favor of the cloud

I thought the author stated this as a crucial factor, in that it is what pushed the cost of colo, as he developed it, over the cost of cloud. You have to buy bigger servers than you need because a) you might have underestimated your need; b) you might grow; and c) a server might break.

With cloud, all that is included in the hourly rate and because it's combined with the same margins for a thousand other tenants, it should increase the hourly rate by less than it increases the colo costs.

The true costs of hosting in the cloud

kpfleming — Fri, 02 Mar 2018 23:44:03 +0000

And there's the in-between model, like Packet. Someone else owns the hardware, networking, and data center space, but you have complete control over the machines. More variables to consider.

The true costs of hosting in the cloud

Cyberax — Fri, 02 Mar 2018 19:51:01 +0000

> Doesn't that question also apply to doing it yourself and employing people full-time to look after your infrastructure exclusively, as the article describes? I would expect this way to be less expensive than that, whereby "you get what you pay for" applies.
I work at AWS but I'd drank Cloud Computing brand of KoolAid long before joining.

I would say that $100k per month is too low to even _start_ thinking about moving to your own infrastructure. The article's author, for example, doesn't mention multi-region and multi-AZ (Availability Zone) deployments.

On AWS it's trivially easy to launch instances in multiple datacenters (AZs) or multiple geographic regions. Just click a button on the console and you're done.

But if you're spending $100k per month then you'll have to physically dispatch your engineer to a remote DC to set up your (likely) multi-rack infrastructure. Then you'll have to worry about your supply chain. If a server in Frankfurt fails, do you have a local contact there that can supply a replacement within at most 24 hours?

> And of course, having people you know physically in charge of the infrastructure has certain trust implications.
I'd argue that cost/benefit of Amazon abusing AWS to access competitors infrastructure is simply ridiculous. Getting caught at it would result in multi-billion loss of business. And there aren't that many secrets that important to steal.

> Not to mention that you are then generally less of a target than AWS (though also less expert at dealing with attackers, though you can also choose how much expertise you need to pay for).
AWS is huge. Even if someone hacks their way into the internal control plane then they probably won't be interested in targeting your company. There are more juicy targets out there.

The true costs of hosting in the cloud

mjthayer — Fri, 02 Mar 2018 07:07:22 +0000

> So basically you're describing a home-grown AWS lookalike, with all the infrastructure needed for automatic instance management.
>
> Do you think you'll be able to beat the economy of scale that AWS (or Azure) enjoys?

Doesn't that question also apply to doing it yourself and employing people full-time to look after your infrastructure exclusively, as the article describes? I would expect this way to be less expensive than that, whereby "you get what you pay for" applies.

And of course, having people you know physically in charge of the infrastructure has certain trust implications. Not to mention that you are then generally less of a target than AWS (though also less expert at dealing with attackers, though you can also choose how much expertise you need to pay for).

The true costs of hosting in the cloud

k8to — Fri, 02 Mar 2018 01:47:07 +0000

Dealing with problems on bare metal have the full range of options available to implementing in amazon and some additional ones. The question is how much does it cost to implement those options starting from bare metal and how much does it cost to implement those solutions in AWS?

The true costs of hosting in the cloud

csd — Thu, 01 Mar 2018 21:30:22 +0000

The author touched briefly on "demand management" but that turns out to be a crucial deciding factor in favor of the cloud. And 2 clear examples are:
- you are a smaller company, and you are peddling your product/solution to get an uptick in traffic. A physical deployment needs 1m-6m advance planning so you have to dream of how successful your product will be 6 months from now and start buying today. As opposed to turning on the VMs as demand actually knocks on your door. This alone can account an absurd amount of unnecessary or too-early capital expenses, or even worse an under-capacity server farm that can't keep up with quickly growing demand providing poor service to users.
- Seasonal changes - this is particularly true for storefronts which see a spike of 100% in Nov/Dec over the other 10 months of the year. A few years back I heard from someone from Ebay that 50% of their servers were pretty much off Jan-Oct...

Having worked in this space for a few years, the equation does tend to favor in-house if you compare machines-vs-machines (i.e. taking planned capacity instead of true utilization as the factor) but things change quickly if you take machines-vs-used-capacity - which is a truer comparison todo because one you buy and own while the other you rent as necessary. And if you take the mix of changing your 'lease period' and the discounts that cloud providers give you, you can move capacity you know you'll be using with a high degree of confidence to a longer lease (reserved instances in AWS-speak) at a much lower hourly rate while keeping the bursty capacity on more flexible per-hour lease, while still having the option to wait-and-see until the demand actually shows up. So you end up with something like 30% on long-term lease, 30% on mid-term lease and the remaining 40% no lease (pay-as-you go). And revise the numbers and ratios constantly.
For companies that run so big that burstiness becomes more line line-noise then I can see this being true. It took dropbox 8 years of strong growth before they decided to switch...

The true costs of hosting in the cloud

sjfriedl — Thu, 01 Mar 2018 18:35:07 +0000

> It turned out the small-medium AWS instances aren't cheaper than leasing bare metal

Which may be true only while it's running: what happens when that bare metal breaks and you're down for 2 hours waiting for remote hands or 4 days waiting for a new part?

It may be hard to put a dollar value on downtime, but it's probably not $0.

The true costs of hosting in the cloud

Cyberax — Thu, 01 Mar 2018 17:35:14 +0000

So basically you're describing a home-grown AWS lookalike, with all the infrastructure needed for automatic instance management.

Do you think you'll be able to beat the economy of scale that AWS (or Azure) enjoys?

Correct answer for almost all engineering questions: It depends

david.a.wheeler — Thu, 01 Mar 2018 16:15:47 +0000

This is a great example of a general principle: The correct answer for almost all engineering questions is, "It depends".

I am tired of people just following the latest fad (because it's the latest fad), or holding onto an old way (just because it's the old way). There are usually pros and cons to different approaches, and people should approach technical decisions as an engineering trade-off (looking at issues like cost, time-to-develop, functionality, execution performance, reliability, security, maintainability, etc.). Look at the pros and cons, and then make the best decision for that particular situation.

Thanks for the article.

The true costs of hosting in the cloud

dskoll — Thu, 01 Mar 2018 12:24:10 +0000

Our particular service is spam-filtering and email archiving. That's about a worst-case price scenario for AWS: high bandwidth, high disk I/O and large amounts of storage.

The true costs of hosting in the cloud

mjthayer — Thu, 01 Mar 2018 10:14:59 +0000

What about a half-way house of companies with the expertise to set up a local data centre which can either set up servers for you on-premise, and visit to maintain them when necessary (and do as much as possible remotely), or even help set up a shared data centre in e.g. an industrial area with multiple local users, who could spill over onto each other's infrastructure in case of spikes?

The true costs of hosting in the cloud

madhatter — Thu, 01 Mar 2018 09:39:06 +0000

Thanks for an excellent choice of talk, well reviewed. If anything, it is a salutary reminder that there is no need to have a belief about whether cloud or physical hosting is cheaper: this should be a data-driven decision. If you're not collecting and periodically reviewing data on your current arrangements and possible alternatives, relying instead on some rule of thumb or personal mantra, you may not be making the best possible decisions.

The true costs of hosting in the cloud

Cyberax — Thu, 01 Mar 2018 08:02:53 +0000

AWS is really the cheapest solution if you want a (somewhat) reliable computing infrastructure and if you don't have some unique requirements. Even large companies can't beat it easily.

The main attraction is that hardware management is Someone Else's Problem. This can not be overstated. If your AWS instance has a problem then simply stop it and then resume it to get it moved to a different hardware node. No messing around with remote hands in a DC or waiting for a replacement part to arrive.

Then there's a question of disaster preparedness. It's easy to run servers in several AWS regions. You'd be hard-pressed to do that using colocated servers.

And there are other goodies, like you can use EC2 Spot or Google Preemptible VMs to get dirt-cheap capacity if you need some number crunching. Or you can use AWS Lightsail as a replacement for dodgy VPS providers.

Now, AWS does have an Achilles heel - it's the high price of outbound data transfer. If you need to serve a lot of content, you might be better off running your own hosts and making your own interconnect agreements with tier ISPs.

The true costs of hosting in the cloud

Cyberax — Thu, 01 Mar 2018 04:38:45 +0000

IPs on Amazon don't cost anything if they are used. Seriously, you can have tens of IPs and if they are attached to a running instance then you are not charged for them.

IPv6 is also fully supported in most regions.

The true costs of hosting in the cloud

rahulsundaram — Thu, 01 Mar 2018 00:51:13 +0000

> Is there really anyone outside of the San Francisco Bay area making this doing datacenter networking and sysadmin work? I mean, really?

Sure. Look at the salary range for an experienced sysadmin in the coastal areas in U.S. Keep in mind, these are HCOL areas so the numbers are inflated in general.

The true costs of hosting in the cloud

yokem_55 — Wed, 28 Feb 2018 22:42:22 +0000

Which brings us to the sensitive question of staff costs; Dyachuk described those as "substantial". These costs are for the system and network administrators who are needed to buy, order, test, configure, and deploy everything. Evaluating those costs is subjective: for example, salaries will vary between different countries. He fixed the person yearly salary costs at $250,000 (counting overhead and an actual $150,000 salary) and accounted for three people on staff. Those costs may also vary with the colocation service; some will include remote hands and networking, but he assumed in his calculations that the costs would end up being roughly the same because providers will charge extra for those services.

Is there really anyone outside of the San Francisco Bay area making this doing datacenter networking and sysadmin work? I mean, really?

The true costs of hosting in the cloud

dskoll — Wed, 28 Feb 2018 21:55:33 +0000

The problem with cloud services is that while they have a low cost to get started, they have relatively high incremental costs as you add users compared to your own colocated hardware.

We looked at Amazon, but even for our relatively small company, we concluded it made more sense to use our own colocated hardware.

I do agree with the other poster who said emergency scaling up onto cloud instances can be a good way to handle sudden spikes in capacity. We haven't needed this yet, but we have the capability in our code to do that.

hybrid setups are interesting

anarcat — Wed, 28 Feb 2018 20:02:24 +0000

[snipped description of a hybrid cloud/colo setup] (Disclaimer: I have not yet had a chance to follow any of the links for further details, so if this hybrid concept was already mentioned I apologize.)

It has not, thanks for bringing this up. It's definitely something that was brought up in other talks at Kubecon, but it's something lots of people are still struggling with. In other articles about Kubecon, I mentioned how Kubernetes is one way to standardize those applications and allow cross-cloud migrations, or at least make those possible. I think it's why large cloud providers like Google, Amazon and Microsoft are embracing it: it provides an on-ramp to their services. And I think having the possibility to have hybrid infrastructures like what you are proposing is probably the best, as it resolves the main problem with colocation, which is when the plain fails and you run out of capacity or you have catastrophic outages. The possibility of rebuilding in the cloud is an amazing fallback.

That said, one big problem with the cloud is when you start using custom extensions like Amazon's serverless stuff or Google's large datasets. Those are "heavy" in that they are a "gravity center" that pull services towards them and make it hard to find the "escape velocity" to leave the service again when you need to. You become dependent on those APIs or large datasets that cannot be abstracted away. So that's also something to be careful about when considering the cloud.

"!holy wars"

anarcat — Wed, 28 Feb 2018 19:57:00 +0000

So before people drive themselves up the wall about Amazon here, I would invite everyone to keep in mind the guidelines here: "Please try to be polite, respectful, and informative, and to provide a useful subject line". Or, in the words of the speaker:

I'm not here to start a holy war. There are true believers of the cloud, there are true believers of bare metal. We are not here to engage in a heated discussions about pros and cons of those. I'm here to talk about money. That is the main objective of this talk: to estimate how much it's going to cost. And all the technical advantages and disadvantages, well there are other talks where those [have been covered].

Also, nitpickers will certainly argue with the math Dyachuk has come up with: that would also be missing the point. This is just one model: if you prefer Dell servers instead of Supermicro or Linode instead of AWS, yes, those prices will change. It's an example, take it with a grain of salt.

Thank you and have a nice day! :)

The true costs of hosting in the cloud

admalledd — Wed, 28 Feb 2018 19:42:18 +0000

Reasons mentioned in the article are exactly why for small/medium sized platforms the approach my current employer takes is perhaps the best-of-both. Granted, it requires you to plan for it from the start, but tooling for the cloud itself requires different concepts anyways when moving from bare hardware (or even VMs).

Here, we have a few rows of racks in a co-location DC. These support all our 24/7 "base load", then if we have any large spike that cannot be processed within the DC it is shipped to our cloud environments where they scale out to a few thousand within a few minutes. Of course the "magic" to do this is platform/application specific, but they *do* exist for most anything in one form or another.

At least for our estimations, the staffing cost of cloud-vs-dc was near-to-nothing different for our size, and we only need one/two who specialize in pure hardware, the rest of OPS don't need to particularly care between. As mentioned it is really only when we start hitting that wall of "Intricate high-speed inter-node networking at $SIZE" that we then move excess burst to the cloud.

By no means is this a perfect answer, but from what little I have seen of others and talked to, it is an attractive solution if it can be made to fit.

(Disclaimer: I have not yet had a chance to follow any of the links for further details, so if this hybrid concept was already mentioned I apologize.)

The true costs of hosting in the cloud

flussence — Wed, 28 Feb 2018 19:21:38 +0000

I have no experience with colo, but I did go shopping around for a dedicated server a while back. It turned out the small-medium AWS instances aren't cheaper than leasing bare metal, and if for any reason you need an extra IP address the AWS monthly cost quickly shoots off into the stratosphere while it's a one-off purchase elsewhere. (On that subject, are they still doing that insane server-side CGNAT thing instead of IPv6?)