>“All of our engineers are focused” on reducing the costs of computing
If this is so, how is it that a startup like Digital Ocean that's only 2 years old find a way to provide $5/month plan that's running on SSD while m1.small on EC2 which is slower than DO costing me around $40/month? At this rate, I should probably buy a server myself and run it at home.
I like AWS and its ecosystem, but I am tired of Amazon continuing to provide shitty service for a ridiculous cost.
I use DO and AWS for different projects because they meet different needs. Of course there are going to be places where one is better suited than the other in both performance and price.
For example, one of my projects requires 1TB of drive space. At DO, I have no choice but to go to the $960/mo 1 TB plan. But I don't need the SSD speed, 96 GB of RAM, 24 cores, or the 10 TB of monthly transfer. But with EC2, I can just use EBS @ $100/mo/TB and pair it with the right-size EC2 instance for the jobs at hand. As an aside, the real question for me is to decide whether I want to invest in a proper dedicated server where I can get good performance and online storage at a lower cost.
DO has done a fantastic job at finding a spot in the VPS market, which is why I'm one of their customers. It doesn't mean Amazon is doing it wrong.
If all you're after is the storage, that is very expensive unless you keep moving your storage tiers up and down very rapidly. E.g. Hetzner has servers at 49 euro (ca. 66 USD) per month for 2TB.
I have a similar need for memcached, but I'm still using DO. I'm running the $5 instances running Twitter's fatcache. For $250/month, I can cheaply get 1TB of memcached storage online and available in our cluster (out of NY2).
Thrown in a some consistent hashing and I can scale up/down as needed. Fatcache is completely kick ass.
Cloud is great when you're in the discovery phase as it provides loads of flexibility for a small startup to try stuff out without investing in hardware, but at the point you're spending $50k/month in the cloud, racking servers and building a private cloud can give you better performance at 20-30% of the cloud cost... The main problem here is most startups don't have the expertise in house to make it happen.
Everyone thinks cloud is cheaper, but it's more about flexibility and avoiding capex, but at a certain scale, it makes sense to host it yourself - or pay someone to build it out for you.
I'm working in this cloud -> private-cloud migration space currently if anyone is interested in discussing.
This obviously depends on your specific scenario, but paying the maintenance/flexibility premium for something like AWS can be cheaper than something more rigid (see Netflix). I hesitate to make any generalizations like "building a private cloud can give you better performance at 20-30% of the cloud cost", because this is not true for all applications/businesses (it's certainly not for ours). For example, those with incredibly bursty traffic (Netflix, Amazon, other online retailers), can see some big wins with AWS. Not having lots of idle capacity that still has to be managed and paid for, not having to beef up the Ops/IT staff, not having to own your own/manage/maintain hardware, etc.
My advice to others is to be wary about generalized figures like "20-30% of cloud cost" from anyone with a vested interest in getting you to switch. Generalizations are dangerous when every usage case is different.
Also keep in mind that there is a lot of room for middle ground (even with AWS). For some (but not all), going hybrid can be the best of both worlds.
Yes, every situation is different, but replacing capacity that you use at peak on AWS will cost you much less to buy, rack, and power yourself considering hardware and hosting costs.
Elastic scaling will save you money on AWS, but you generally can't scale everything (databases, caches, storage are harder to elastically scale up and down). And you need to pay at peak for whatever your peak is and deal with additional complexity of elastic scaling to begin with. No doubt you're paying smart people already to figure this out - it doesn't come for free.
In any account - if you take a largish box you might rack as an example (Dual-socket Sandy Bridge Xeon, ~256GB RAM, SSD, approximately a cr1.8xlarge), that box costs about $30k annually on AWS. You can buy it for around $8k. Over two years with hosting costs, you're looking at around $10k vs $60k roughly. Numbers work out similarly for other instance types although you would need some scale - perhaps a dozen of these systems - for this to make sense for your application.
Netflix I think is a special case - video delivery which they almost certainly don't do from AWS is their biggest cost - api/web traffic would be dwarfed in comparison, so who cares if it's expensive on AWS in that case? They probably enjoy the flexibility of AWS more than the little bit of money they might save by doing it themselves. Many other largish companies have gone the private cloud route; Zynga comes to mind: http://www.informationweek.com/mobile/inside-zyngas-big-move...). It's not the right fit for everybody, but it's a consideration.
Hybrid is interesting too -- provision your own hardware for the big iron - and burst into cloud for the rest of it.
All the hype of "cloud" makes people think that it's the only and/or best way to do it - certainly not the case for every sort of business.
> Yes, every situation is different, but replacing capacity that you use at peak on AWS will cost you much less to buy, rack, and power yourself considering hardware and hosting costs.
The important thing to keep in mind with a DIY situation is that there are additional costs besides hardware/rack/space, and a lot of things that you may have a hard time putting a dollar value on (flexibility). It's not accurate to just throw out the cost for a Sandy Bridge Xeon + rack space without factoring in the other stuff (upgrades, maintenance, if you need to add an employee, the lessened mobility, the increased responsibility, etc).
At our current scale, if we went the DIY route, we'd need to hire someone else to take over more of the Ops situation for us. An additional employee would cost a lot more than paying AWS a bit more (salary, management overhead, good benefits, communication). We'd also have more "stuff" to worry about, whereas right now our small team spends 98% of our time working on our product, marketing, and sales. This is entirely anecdotal and specific to our situation, but I mention it as an example.
> All the hype of "cloud" makes people think that it's the only and/or best way to do it - certainly not the case for every sort of business.
I personally can't stand the "cloud" buzz word, because it doesn't adequately describe anything. I am no "cloud" zealot, but on the same token I think some bare-metal-always-wins advocates don't really "get" what AWS is doing or when it's appropriate to use them (even if they appear more expensive at face value). As a developer, I've seen the kind of crazy neat things we can do with the flexibility and the shifting of responsibilities to AWS. For us, AWS is a no-brainer, though we do wish we had a more affordable route to going hybrid than DirectConnect or their dedicated machine offerings. But like I said, I wouldn't dare make a generalized assertion of AWS' value to others based on our situation. Every company/staff/application is different.
I get what AWS offers, I've been a customer there for many years, my point is that as a startup, eventually you figure something out that is more stable and gets some traction (or you hope to at least). At that point, moving to a less flexible but overall vastly cheaper solution is often the answer.
So, the question becomes, "How much do I spend on AWS per year?" and something like "If I could hire one person to save me $500k+/year, would I do that?".
If you're a small business with an uncertain future, and smallish AWS spend (<$50k/month) -- sure, stay with AWS until you get into the situation I outlined above.
It can be cheaper for very limited situations, sure.
I have never seen a case where AWS makes financial sense for the base load, or for anyone without that severely bursty traffic, though. For batch jobs or handling peaks of less than 6-8 hours, sure. But fewer people have peaks that are economical to handle that way than they think. And for those who do - e.g. retailers that needs to handle the christmas rush, depending on a system where servers can in theory become unavailable at any time is a high risk game.
Most of the time you can get to 30% of AWS with rented managed servers. Racking your own kit can get substantially below that.
Going hybrid is interesting, though, as it actually makes the case for actually using AWS much even weaker for most people:
Whereas previously you'd need your managed servers to be able to handle sufficient peaks to give you time to spin up more servers, once you have support for spinning up cloud instances for peaks, you can often aim for near 100% utilisation during normal days peaks, which magnifies the price difference substantially.
Even for most e-commerce sites, unless your site has a very localised audience, peaks are rarely high enough or sharp enough to make AWS economical if you have a system that's prepped for a hybrid situation - you can let things get very close to the wire, and start spinning up VMs, but you very rarely will find situations where it's worth doing for more than a few days before it'll be apparent it'll be cheaper to spin up a managed machine billed monthly, and/or putting in an order for more kit to put in your own rack.
> Not having lots of idle capacity that still has to be managed and paid for,
You are still paying for it with AWS with every single hour of VM's you pay for - it's one of the reasons AWS is so expensive: You have to fund all the spare capacity they need to have on standby to prevent running out of hardware during peaks.
> not having to beef up the Ops/IT staff,
This attitude scares me. You have to be prepared to deal with outages of pretty much every kind with AWS too. If you run managed servers, there's pretty much no difference, other than that you have to account for whatever the hosts turnaround time in bringing up new servers is (which one of those places where a hybrid setup might help you - especially if you're using a host like e.g. Softlayer or Iweb that offers the full range from managed servers to cloud servers in the same data centres).
The one thing you don't have to deal with is if you use a colo and it saves you people to cable up and rack servers and install the OS. But you can hire people to do that too.
> not having to own your own/manage/maintain hardware, etc.
... but you don't need to do this to beat AWS hands down, either: If you rent managed servers on a month by month basis, with no contracted minimum term, you can still easily come out vastly cheaper for the base load and most peaks.
> My advice to others is to be wary about generalized figures like "20-30% of cloud cost" from anyone with a vested interest in getting you to switch. Generalizations are dangerous when every usage case is different.
Sure. It's sound advice to do your own maths. But every time I have, for a variety of different sites, AWS has come out ludicrously expensive for anything but batch jobs and very short, sharp peaks.
> AWS has turned infrastructure into an api and software which is just simply fantastic, but you pay for that sort of flexibility.
There's nothing wrong with paying for the flexibility, though. We gladly do it because it's a great fit for our traffic patterns and our staffing level.
Is it possible to spin up a few thousand of those on DO?
Sure most people use AWS as a glorified VPS provider, and for that it is rather expensive. The fact that DO is only 2 years old actually scares me more since they probably haven't faced as many unique challenges as a company who has been in the business for a longer period of time has.
This is something that I feel so many people don't understand about EC2 (ELASTIC Cloud Compute). If you are using it like a traditional VPS provider, you are likely over-paying for something you could get cheaper elsewhere. If you are just looking at a price comparison to another VPS provider or even the private cloud, you are doing it wrong. One must consider how much (or little) ability they need to scale/up AND down at very short notice.
EC2 is a boon for those that need to add and remove capacity quickly. This is the appeal of EC2. It's not going to blow your socks off as far as performance per dollar. It is going to blow you away with its flexibility. If you don't need to scale up AND down quickly and regularly, EC2 is probably not a good fit for you. With EC2, you are paying for that flexibility.
Netflix is one of the penultimate examples of "doing EC2 correctly". Their traffic is extremely bursty. They are constantly automatically scaling up/down, and they do this and so many other things without human intervention.
Right now, probably so. Though, to AWS' credit, they've been "diversifying" in developing some of these other complementary services (RDS, Route53, ElastiCache, Search, DynamoDB, Elastic Transcoder). It's getting to the point where EC2 is a good fit for those who want less infrastructure responsibility (and are willing to pay for it).
If RDS/ElastiCache were cheaper, I think it'd be a lot easier to make the case for EC2 for startups and smaller companies that don't need the auto-scaling features. Even as a smaller business, it's a lot better for us to spend a little more on infrastructure in order to avoid hiring additional people to keep us running.
Compare their cost reductions with the decrease in computing costs and you'll very rapidly discover that they don't actually pass on savings to customers. Quite the opposite.
Is that assuming they are replacing all existing hardware with newer models as they become available, or is it based on some reported replacement schedule Amazon has stated? Also, is it based on Moore, or the actual market prices?
I would expect their costs are based on the mixture of systems they have in place and for it to go down somewhat similar to the market cost for the component, but with a somewhat large lag time as older systems are phased out and newer ones phased in.
You're paying for a lot more than just the hardware your instance runs on with EC2. You're also indirectly paying for the other systems they make available to you within EC2 (auto-scaling, ELB, EBS, CloudWatch, etc).
If you don't need these additional services, AWS is indeed a bad value. If you use a good chunk of them like we do, it's money well spent, though it won't fit neatly into graphs like the previously mentioned one.
I mentioned this elsewhere in the thread, but you pay the premium with EC2 for flexibility, not so much for raw computing power. If you don't need to be able to scale up AND down with regularity, EC2 is indeed a lot more expensive (since you're not considering the core advantage of EC2 in that case).
There are a ton of simple VPS providers that would probably get you closer to the baseline for computing costs, but to be fair, those providers are a completely different beast.
In a sense, if you only care about net profit, where you can always reinvest gross profits to drive the net to zero.
Analysts estimate Amazon Web Services will have $3.8 billion in revenue in 2013, with a 100% gross margin. It's big enough that it's raising the gross margin of the whole company.
Perhaps, but discussing profitability of AWS in gross margin terms is somewhat pointless. At 100%, that means not a single cost of operating the business has been accounted for. No computers, no engineers, no data centers, no electricity, nothing. It's a meaningless number.
I'm not suggesting the business is not profitable, by all accounts it is, but to understand how it's benefiting the company as a whole, you need to look at the operating line.
Yes, gross margin is (revenue - COGS) / revenue. But per your parent comment, Amazon is not accounting for their AWS costs in COGS. Instead, it is primarily included in the technology and content expense line, which is below the gross margin.
If gross margin = 100%, then by definition COGS = 0. Thus, gross margin is a meaningless number in this case.
Full-disclosure: I'm an engineer at Amazon Web Services.
"Mr. Hamilton later said privately that Amazon had developed original statistical methods to limit damage from catastrophic failures."
I don't think it's what James is referring to (as it's just the tip of the iceberg), but at Re:Invent we also released Infima - a library for service-level fault isolation and fault tolerance using Route 53 for endpoint discovery;
The techniques in Infima give some insight into how we think about multi-tenancy and availability, as well as provide a framework for building other services on top of EC2, Route 53 and so on.
Interesting that they focus on all the infrastructure they design and build/commission themselves. They are also a big customer of my company (http://www.newisys.com/Products/products.shtml). Coincidentally, they call out Facebook at the bottom of the article, in context of the Open Compute Project. Facebook had airflow and overheating problems with their intended OCP design and our Newisys engineers helped with some design adjustments to resolve those, too. Now Facebook is buying cheap Taiwanese-made (Quanta & Compal) OCP-design servers and shipping them to us (http://sanmina.com/) for racking, systems test, and shipping to their DC in Lulea, Sweden (https://www.facebook.com/LuleaDataCenter).
So yes, these guys all do lots of tough engineering work, but it's not in isolation and not without assistance from industry veterans whose careers it has been to ensure properly functioning data centers and networks.
> Google publishes some technical papers about its big computers, but does not talk about things like the semiconductors it has developed internally to handle computing at this scale.
They're also the world's largest computer manufacturer (for internal data center hardware).
Software people tend to think chips and hardware design are magic skills known by only three revered people in the world. It's just another track of school you could have chosen. ("Google writes its own software now?")
They used merchant silicon in their custom switches. Modern switch ASICs are 2+ billion transistor devices with lots of fancy custom logic for SRAMs and TCAMs and analog magic for PHYs. Even Google can't afford to do that by itself.
> 1.5 million requests a second, and holds trillions of “objects”
My god. I used to think I was a bit of a rock star, with (rails) web sites exceeding 100 requests/second on my resume. 1.5 million requests per second ... I cannot even really imagine that.
These get requests are likely sharded among a large number of machines. Viewed that way, it's much more manageable. Want to service more requests? Just add more machines.
1.5M relational queries on the other hand--that would be impressive.
These are spread over an enormous number of machines.
OT, but for truly impressive single machine performance, check out the LMAX FX trading platform, which can handle 6 million orders per second on a single core on commodity hardware.....in Java. [1]
It all depends on how many servers you have, as well as what sort of databases/database complexity.
For a laughably trivial site, a single host could do a lot of requests per second without too much effort. For a normal Rails app running on a handful of hosts, 100 requests per second is nothing to scoff at, especially if you're on your own.
They are in Equinix SY3 (Sydney)[1]. Pretty sure Dell doesn't do DC design for Equinix. I heard some stories about the truck loads of servers going in when they were rolling it out, but I don't remember the vendor.
What is rather mind boggling is that Google, Amazon, Facebook, Microsoft, Yahoo all have multiple, huge data center installations and, they are all profitable companies. It does not seem to be a zero-sum game yet for market share, and the growth in the world's appetite for storage and access is still not fully served. Can they all wind up 'winning' with profitable operations founded on web infrastructure?
How much storage does one Internet citizen occupy, anyway?
"How much storage does one Internet citizen occupy, anyway?"
That number chases toward infinite. Its growth may slow down, as we reach the limit of how many 'actions' a person can reasonable take that generates content, but it'll never stop growing just on the basis of the unlimited desire for higher quality experiences (mixed with a general moore's law ride that makes it all financially reasonable).
And just when you think storage demand is going to slow down, we'll jump to some kind of virtual reality life recording, that makes it possible for others to walk around in your daily life as though they're actually there, and storage demand will explode another magnitude.
You can remove Facebook and Yahoo from that list though. Their operations are funded by their core businesses and they don't compete in the AWS-style cloud segment. That is, it doesn't make sense to discuss Facebook in regard to whether it can win or make money, compared to AWS or Azure.
So is there room for Google, AWS, Azure, Rackspace, IBM etc? Yep. It's a future hundred billion dollar plus business (AWS will probably be at $10b+ in a few years). No reason there can't be three or four big players in such a large market.
Yep, I get that. But they IPO'd in 1997. Again, not to detract, and in fact I'm amazed by how they've been able to compete both on the Brick & Mortar stage and Digital Services stage. I've just never seen a company given such love/leniency for so long by Wall St.
On Wall Street, the fake love often turns to a company being out of favor quickly (with both directions being equally irrational, in the overvaluing and then undervaluing). Indeed, the less fundamentals you have propping you up, the more likely this is to eventually happen, and happen hard. That exact scenario has played out numerous times with Netflix, for example.
Amazon's valuation is almost purely an emotional basis, as all future returns are being pulled forward to the extent that they're overvalued and the calculations about their future earnings expectations are correct. Is Amazon worth $50 billion? $100b? $168b? That's an emotional judgment today, as they have no profits and little history of profitability, and if you want to speculate on tomorrow... well if their valuation keeps bubbling up, they'll soon catch Walmart, which has $17 billion in annual profit (meaning it may only take... 20 years or so for Amazon to fully realize that market value via profitability). Amazon is going to be an excellent short at the end of this stock market bubble, they'll lose half their value on the down swing (my opinion).
However keep in mind, this is not a company that has always been loved by Wall Street. For most of its life, it has been hated by Wall Street. If you run an average price from early 2003 through the end of 2008, the stock was essentially flat lined for five years. And obviously it was viewed with near universal disgust after the dotcom bubble imploded, and the stock cratered to the $5's after 9/11. Amazon.Toast and all that.
It's a better business idea for Amazon to build and invest in the company than it is to go profiteering right now. How valuable would a company be if it could get same day deliveries anywhere in America? In Europe? In China? All of the world? One of these days Amazon is going to stop building as much and their profits are going to shoot through the roof.
The price will bottom out eventually and Amazon knows this. What we are really seeing is an infrastructure arms race and that is why AWS is costly. They are not building AWS for the individual user long term. They are building it for the enterprise. Eventually as bandwidth and computing power increases and cost decreases it may be feasible to run your personal blog out of your house or use a VPS on Digital Ocean. However, as we become a global community you will have to think about your application on that scale to be successful and that's where AWS can find longevity.
>> Mr. Hamilton also said Amazon was not interested in participating in industry standards discussions, which could publicize its many technical advances.
What are the side effects of this in a few years, when all of the high-end stuff has been designed 4 different ways by 4 different companies?
What happens when Dell and HP's server businesses are shells of their former selves? When Cisco is but a memory as their biggest customers have began designing solutions in-house?
You'll have to go with the cloud. You just won't be able to get the best of the best equipment without doing so.
Or, you know, pay for the product. I don't understand the logic behind complaining about paying $3.75 a week for a product that you use so frequently. It should be embarrassing.
10 articles a month isn't exactly frequently. At the low end, you are suggesting that people throw more than a dollar per article at them. You shouldn't find it surprising that many people will baulk at that.
I'm a bit skeptical on the fact that such companies do not disclose their custom infrastructure design. So their value is both in hardware optimizations as well as software ones ?
being a grad student myself, I'm wondering if we'll ever see any specific technical aspects on infrastructures that could drive systems research and provide real problems to be attacked...
I'm skeptical that there's anything that innovative going on behind the veil.
I suspect if you had access you would look at a whole bunch of evolutionary improvements and probably trace them back to which papers you've seen it sort of mentioned in.
Probably not as much custom silicon as the article implies either, simply because on the scale of things Amazon is not a semiconductor company and Intel pretty unambiguously own the cutting edge there.
If this is so, how is it that a startup like Digital Ocean that's only 2 years old find a way to provide $5/month plan that's running on SSD while m1.small on EC2 which is slower than DO costing me around $40/month? At this rate, I should probably buy a server myself and run it at home.
I like AWS and its ecosystem, but I am tired of Amazon continuing to provide shitty service for a ridiculous cost.