Reference: Spot Instances

Spot Instance pricing is treated as an auction where a bid is placed on an instance and compared to the market demand and therefore associated price. If the bid is acceptable then the instance can be acquired at a significant discount over the On Demand price. As a result the price of Spot instance is variable and this presents an opportunity to minimize the processing costs for our customers.

Volatile Spot Instances

These are EC2 Instances purchased as Spot Instances during Qubole AutoScaling - the bidding policy is set in the Volatile Spot Instance Settings section of the Cluster Configuration. Note that the Spot Percentage is not applied to the entire cluster but to the growth capacity or (Max Slaves) - (Min Slaves).

Maximum Bid Price

The percentage of the standard On Demand price you are willing to pay.

Request Timeout

The length of time you are willing to wait for AWS to respond to your bid.

When this box is checked the Master Node as well as the Core Slave Nodes (designated as the Min Slave Nodes) will be Spot Instances. Additionally all Instances added during Qubole AutoScaling will be Spot Instances.

Maximum Bid Price

The percentage of the standard On Demand price you are willing to pay.

Request Timeout

The length of time you are willing to wait for AWS to respond to your bid.

Spot Instance Availability

AWS does not guarantee that the Spot instances requested will be granted – if the current Spot price exceeds the bid price, or there is not enough spare capacity, the Spot instance request will be rejected. Administrators can ensure that AutoScaling proceeds successfully by selecting the “Fall to On Demand” option in the Volatile Spot Instance Settings. However, if Stable Spot Instances are being used, the Cluster will fail to come up because AWS did not give any Spot Instances and therefore a Master Node could not be established inside of EC2.

Spot Instance Loss

AWS may take Spot Instances away at any time and Administrators can reduce the probability of this event by using higher bid prices. Administrators may also mitigate the impact to processing by selecting “Use Qubole Placement Policy” in the Cluster Configuration. However if more than 50% of the Cluster is Spot Instances then there will not be enough On Demand Instances to back up all of the Spot Instances. If Stable Spot Instances are being used and in addition to the Slave Nodes the Master Node is taken away by AWS then the entire Cluster will be taken offline because there is no more Master Node in EC2.

For a list of AWS Reason's for Spot Instance Loss refer to: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html