Spot instances can be terminated without price change (and new ones gotten after long time)

[gaganapplatix]

We noticed that spot instances were Terminated without any bid or spot price changes. It seems AWS can terminate them and not give new ones immediately. We may need to switch to on-demand when this is happening. Will be a little hard to decide but need some mechanism

[gaganapplatix]

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-bid-status.html

Your Spot Instances continue to run as long as your maximum price is at or above the Spot price, there is available capacity for your instance type, and you don't terminate the instance. If a change in the Spot price or available capacity requires Amazon EC2 to terminate your Spot Instances, the Spot request goes into a terminal state. For example, if your price equals the Spot price but Spot Instances are not available, the status code is instance-terminated-capacity-oversubscribed. A request also goes into the terminal state if you cancel the Spot request or terminate the Spot Instances.

Sounds like capacity is a factor here.

[shrinandj]

Commit 631afe7 from @alantang888 addresses this.

We need to do a new release of minion-manager that will include this fix.