New-age Transactional Systems - Not Your Grandpa's OLTP
John Hugg discusses high volume transaction processing applications with high and low frequency profiles, and how VoltDB can be used for that purpose.
News
The content has been bookmarked!
There was an error bookmarking this content! Please retry.
A Look At Hidden Costs In Cloud Solutions
Posted by Dilip Krishnan on Aug 25, 2010
In a recent post David Pallman takes a look at the hidden costs of moving to the cloud, specifically in the context of Azure.
Cloud computing has real business benefits that can help the bottom line of most organizations. However, you may have heard about (or directly experienced) cases of sticker shock where actual costs were higher than expectations.
“These costs aren’t really hidden, of course: it’s more that they’re overlooked, misunderstood, or underestimated.” He says, as he examines and identifies these commonly overlooked costs in cloud based solutions.
Hidden Cost #1: Dimensions of Pricing
According to David the #1 source of surprises is not taking into account the various dimensions the provider services are metered. Every service utilized adds more facets the offering can be metered in terms of bandwidth, storage, transaction costs, service fees etc.
In effect, everything in the cloud is cheap but every kind of service represents an additional level of charge. To make it worse, as new features and services are added to the platform the number of billing considerations continues to increase.
He suggests consumers use service specific ROI calculators such as the Windows Azure TCO Calculator, Neudesic’s Azure ROI Calculator, or the official Windows Azure pricing information.
Hidden Cost #2: Bandwidth
”Bandwidth is often overlooked or underappreciated in estimating cloud computing charges.” he says, He suggests that we model the bandwidth usage using tools such as Fiddler to give an give a ballpark estimate based on key usage scenarios. One could also throttle bandwidth overages or model the architecture to provide the path of least traffic given any usage scenario.
Hidden Cost #3: Leaving the Faucet Running
He suggests we review the usage charges and billing often to avoid surprises at the end
Leaving an application deployed that you forgot about is a surefire way to get a surprising bill. Once you put applications or data into the cloud, they continue to cost you money, month after month, until such time as you remove them. It’s very easy to put something in the cloud and forget about it.
Hidden Cost #4: Compute Charges Are Not Based on Usage
Hidden Cost #6: A Suspended Application is a Billable Application
He emphasizes that if the application is not used or suspended it does not mean the billing charges do not apply. He urges users to check the billing policies.
Since the general message of cloud computing is consumption-based pricing, some people assume their hourly compute charges are based on how much their application is used. It’s not the case: hourly charges for compute time do not work that way in Windows Azure.
Hidden Cost #5: Staging Costs the Same as Production
Many have mistakenly concluded that only Production is billed for when in fact Production and Staging are both charged for, and at the same rates.
Use Staging as a temporary area and set policies that anything deployed there must be promoted to Production or shut down within a certain amount of time. Give someone the job of checking for forgotten Staging deployments and deleting them—or even better, automate this process.
Hidden Cost #7: Seeing Double
[Y]ou need a minimum of 2 servers per farm if you want the Windows Azure SLA to be upheld, which boils down to 3 9’s of availability. If you’re not aware of this, your estimates of hosting costs could be off by 100%!
Hidden Cost #8: Polling
Polling data in the cloud is a costly activity and incurs transaction fees. Very soon the costs could add up based on the quantity of polling. He suggests
Either find an alternative to polling, or do your polling in a way that is cost-efficient. There is an efficient way to implement polling using an algorithm that varies the sleep time between polls based on whether any data has been seen recently.
Hidden Cost #9: Unwanted Traffic and Denial of Service Attacks
He warns that unintended traffic in the form of DOS attacks or spiders etc. could increase traffic in unexpected ways. he suggests the best way to deal with such unintended charges is to audit the security of the application and provide measures of controls such as CAPTCHA’s.
If your application is hosted in the cloud, you may find it is being accessed by more than your intended user base. That can include curious or accidental web users, search engine spiders, and openly hostile denial of service attacks by hackers or competitors. What happens to your bandwidth charges if your web site or storage assets are being constantly accessed by a bot?
Hidden Cost #10: Management
Finally he states that as a result of all of these factors clouds come with an inherent cost to manage these services for efficiency in usage and consequently billing.
Regularly monitor the health of your applications. Regularly monitor your billing. Regularly review whether what’s in the cloud still needs to be in the cloud. Regularly monitor the amount of load on your applications. Adjust the size of your deployments to match load.
The cloud’s marvelous IT dollar efficiency is based on adjusting deployment larger or smaller to fit demand. This only works if you regularly perform monitoring and adjustment.
He concludes his article saying that its very possible that one might not get things right the first time around and to expect some experimentation; possibly get a assessment of the infrastructure and seek the guidance from experts.
Cloud computing is too valuable to pass by and too important to remain a diamond in the rough.
Be sure to check out the original post and do enrich the comments section with your experiences.
- This article is part of a featured topic series on SOA
- See more SOA content at: http://www.infoq.com/soa
- Other recent content items in this topic
Costs
by
Chris Czarnecki
Posted
Re: Costs
by
William Louth
Posted
Re: Costs
by
Chris Czarnecki
Posted
GridGain 3.0
by
Nikita Ivanov
Posted
Re: GridGain 3.0
by
William Louth
Posted
Re: GridGain 3.0
by
Nikita Ivanov
Posted
Re: GridGain 3.0
by
William Louth
Posted
Re: GridGain 3.0
by
William Louth
Posted
Re: GridGain 3.0
by
Nikita Ivanov
Posted
Re: GridGain 3.0
by
William Louth
Posted
-
Great article. Its important to highlight these costs as they can quickly accumulate. Its also important to highlight that there are other, maybe more cost effective cloud solutions for .NET developers than Azure. More details can be found here.
-
And why not use a costing technique and model that is ideally suited to the (metered) service industry - activity based costing.
costicity.jinspired.com/?page_id=43
An metering approach based on ABC has advantages across many mgmt domains including performance, capacity, security, reliability/availability,.....
williamlouth.wordpress.com
We should also push for standardization of how metering information is made available in particular we need it at the point of interaction which in the real world is generally the norm
opencore.jinspired.com/?p=1583 -
I agree totally.
-
GridGain 3.0 includes Pay-Per-Usage pricing with our unique Idle Detection Technology that prevents charging for an active and running node if it is idling for more than hour. Idling is defined as not performing any user operations and not storing any user data as part of the data grid.
This technology allows GridGain customers to safely over-provision yet pay for the actual use only. This removes the cost penalty for such frequent over-provisioning scenarios such as planned over-capacity for anticipated load spikes, staging and QA environments, hot standbys, scheduled builds, disaster recovery sites, etc.
--
Nikita Ivanov
GridGain Systems
GridGain = Compute + Data + Grid -
Nikita with OpenCore customers have a solution that works with any stack and allows them to add important context to the costing model as well as ensuring they can bring cost, performance and other business centric metrics under the same model. In this regard there seems to be a huge amount of waste in the cloud with re-invention of such technology whereas it would be best the industry standardize on an API and model which allow vendors like yourself to register meter specific to your service within runtimes and at metering points. This is what OpenCore (metering) is looking to achieve and fortunately this time we have standard based on a foundation that is also technical superior to anything else available.
Remember you don't manage cost you manage its cause. Cause is specific to the customers goal and generally inaccessible to generic runtime frameworks.
Metering points for remote service integration: opencore.jinspired.com/?p=1583
Meter plugins for local runtime integration: williamlouth.wordpress.com/2010/04/03/if-it-can... -
I don't see it...
1. What is the incentive for us (vendor) to even look at it? We have something that is 100% integrated into our system/billing and no standard will ever provide anything close to what we specifically need.
2. *ANY* standards in cloud industry are pie in the sky right now. I'm saying it with pretty dead-on authority...
Idea is good, but I think we are 3-5 years away from even considering it at this point...
Best,
Nikita. -
It is also worth nothing that whilst at the beginning applications might operate in isolation within one particular service/cloud domain this is going to eventually change and thus we need a cost model that spans multiple runtimes, providers, and services. If my job makes a call to a metered service then I need to see this information tallied with the execution behavior and cost of the job within the G3 env.
We are going to move to a (composite) service supply chain. Whilst its nice to make local cost & performance optimization at one service point the bigger savings will be made in managing the complete supply chain cost & value models. -
"What is the incentive for us (vendor)"
This is the problem with a real lack of competition and when the market is not sufficiently mature to protect itself and effectively manage & audit a suppliers charges. That said it is not an all or nothing choice. You can still have you billing but still make available execution context (thread generally) specific meters.
Metering != Billing, Metering > Billing
williamlouth.wordpress.com/2010/03/15/metering-... -
Cost models are incredibly important in task/job scheduling. Customers should be able to introduce multiple cost based meters beyond what the grid scheduling considers. Check-out what Microsoft is doing with this information.
Quincy: Fair Scheduling for Distributed Computing Clusters
research.microsoft.com/apps/pubs/default.aspx?i...
"We introduce a powerful and flexible new framework for scheduling concurrent distributed obs with fine-grain resource sharing. The scheduling problem is mapped to a graph datastructure, where edge weights and capacities encode the competing demands of data locality, fairness, and starvation-freedom, and a standard solver computes the optimal online schedule according to a global cost model."
OpenCore can fully support such a solution today that takes into costs associated with metered services and resources consumed directly and indirectly as a result of them. Maybe it is still too far off but hey we need to live up to the company name. -
Bill, I'm with you buddy :)
Again, these ideas are great and some will come to fruition at some day. But we need to have something today and most grid and cloud standards have been an object failure. I'm on JSR-107 ("Data Grid" JSR). It's been... almost 10 years (if I'm not mistaken) and nothing is done. Metering and billing is a way more generalized than that.
I wish we could have an industry wide standard for that - we'd jump on it. Until then - we'll provide our own technology.
Best,
Nikita Ivanov.
GridGain = Compute + Data + Cloud
10 comments
Watch Thread Reply