Amazon EC2 recently gained two-minute warnings before Spot Instances are reclaimed, a feature "formally known as a Spot Instance Termination Notice". The goal is to "allow more types of applications to benefit from the scale and low price of Spot Instances" by giving them time to save state, upload log files, or deregister from other components such as load balancers or cluster managers.
Amazon EC2 Spot Instances are spare EC2 computing capacity users can bid on. Instances are launched whenever a bid exceeds the current spot price, which varies in real-time based on supply and demand. The Spot Instance pricing model complements the regular On-Demand and Reserved Instance purchasing options and prices can be considerably lower than On-Demand (recently 86% lower, on average). They are best used for time-flexible, interruption-tolerant tasks, though advanced users are also capable of facilitating them for mission critical 24/7 production workloads by means of respective fallback and adaptive Auto Scaling strategies.
The simplified Spot Instance lifecycle comprises three states (a complete spot bid status diagram and explanation is also available):
- A spot request is created which includes the bid price, desired number of instances of a particular type and optionally additional constraints. Spot requests can also be persistent so that a new bid is automatically placed after an instance has been terminated.
- When a bid price exceeds the current spot price, the instances are run.
- When the current spot price rises above the bid price or the capacity decreases, the instances are reclaimed by AWS.
AWS now introduced a two-minute warning before terminating a Spot Instance to enable additional use cases and simplify dealing with Spot Instance interruptions, which had previously only been possible after the fact. Besides cleanup tasks like saving state and uploading log files, this also enables early corrective measures for the pending instance loss. For example, users could scale up an On-Demand fallback cluster or bid on less contended instance types or availability zones.
The termination notice is made available as part of the instance metadata at "http://169.254.169.254/latest/meta-data/spot/termination-time". The URL yields a HTTP 404 status code during normal operation, but "will become available when the instance has been marked for termination (step 3, above), and will contain the time when a shutdown signal will be sent to the instance’s operating system". The new bid status "marked_for_termination" is also accessible via the DescribeSpotInstanceRequests API action, though it doesn't include the shutdown signal time.
Users have voiced their surprise about the need to poll the API or metadata at a recommended 5 second interval rather than being offered Amazon SNS notifications – Mitch Garnaat tweets:
The new spot instance termination notice (aws.amazon.com/blogs/aws/new-ec2- …) is great but having to poll instance metadata? Kludgey.
AWS recently accelerated the pace towards event-driven computing architectures with the introduction of AWS Lambda, DynamoDB Streams and Amazon S3 event notifications (see previous coverage). The latter are based on SNS notifications indeed, just like many other AWS services’ notifications, for example Auto Scaling lifecycle notifications (also previously covered) – Avram Lyon agrees:
@jeffbarr Can't be SNS like ASG LifecycleHooks? Still, now adding support for termination notices to github.com/scopely/shudder
The referenced shudder is Scopely's "service for facilitating graceful shutdowns in AWS Auto Scaling groups" and has meanwhile gained said support for handling termination notices too – Martin Camitz also prefers to receive SNS notifications:
So now the instance can find out that it is being terminated and can take appropriate action and that's really great. But I want to know as well!
He created a C# based service EC2SpotStop2SNS for EC2 Windows instances as a workaround.
Amazon offers an EC2 Spot Instance curriculum with video tutorials covering common use cases, bidding strategies and interruption management strategies. Custom bidding strategies can facilitate a Spot Instance pricing history for the last 90 days and a Spot Instance data feed with usage and pricing information per account. Support is provided in the Amazon Elastic Compute Cloud (EC2) forum.