Using the AWS Ruby SDK to launch and tag AWS EC2 Spot Requests

Amazon Web Services provide a number of SDKs that support the various scripting languages. Because we love to automate manual repetitive tasks, we utilized the Ruby AWS SDK to develop a script that: –

1. Creates a specified number of spot instance requests with associated launch parameters
2. Spreads the launch requests over a specified number range of launch zones
3. Waits for the instances to be fulfilled and informs you
4. Tags the instances with your tagname so they appear with a visible description in the web portal name

This is a working version but feel free to enhance to parameterise obvious variables

There are a few prerequisites you will need before you can run the script.

1. You obviously need an AWS account
2. You will need to install latest version of Ruby
3. Build a config.yml file that has your security credentials eg.
access_key_id: ABCDEFGHIJK
secret_access_key: ABCDEFGHIJ12345
4. You may need to setup HTTP_PROXY variable if you are installing ruby gems over a proxy
eg set HTTP_PROXY=http://userid:[email protected]:8080
5. gem install aws-sdk
6. gem install net_ssh

Now heres the script. As mentioned it currently has some hard coded spt request attributes but feel free to parameterise whatever makes sense.

# request spot instance requests

require 'aws-sdk'
require 'open-uri'
require 'base64'
require 'yaml'
 
class Starter
	def initialize(ec2)
		@ec2 = ec2
	end
 
	def start!
		request_ids = request_instances
		spot_ids = wait_for_spots(request_ids)
		instances = spot_ids.map { |iid| @ec2.instances[iid] }
		tag_instances(instances)
#		instance_ids = wait_for_instances(instances)
		instances
	end
 
	private
 
	def request_instances
		$stderr.puts('Requesting spot request instances')
#		zones = %w(eu-west-1a eu-west-1b eu-west-1c)
		zones = %w(us-west-1c)
		responses = zones.map do |zone|
			@ec2.client.request_spot_instances(spot_request_spec(zone))
		end
		responses.flat_map { |r| r[:spot_instance_request_set].map { |rr| rr[:spot_instance_request_id] } }
	end
 
	def spot_request_spec(zone) {
		spot_price: "0.15",
		instance_count: 20,
		launch_specification: {
			image_id: 'ami-9999999d',
			key_name: 'us-keypair2',
			instance_type: 'm1.large',
			security_groups: ['default'],
			user_data: Base64.encode64('TagName'),
			placement: {
				availability_zone: zone
			}
		}
	   }
	end
 
	def wait_for_spots(request_ids)
		$stderr.puts('Waiting for spot request instances')
		loop do
			response = @ec2.client.describe_spot_instance_requests(:spot_instance_request_ids => request_ids)
			statuses = response[:spot_instance_request_set].map { |rr| rr[:state] }
      if statuses.all? { |s| s == 'active' }
        $stderr.puts('Spot request instances fulfilled')
        return response[:spot_instance_request_set].map { |rr| rr[:instance_id] }
      end
      if statuses.all? { |s| s == 'closed' }
        $stderr.puts('Spot request instances cancelled')
        return response[:spot_instance_request_set].map { |rr| rr[:instance_id] }
      end
		  result = {}
		  statuses.uniq.each{|status| result[status] = statuses.count(status)}
		  $stderr.puts("Spot instances in states: #{result}")
		#	$stderr.puts("Spot request statuses: #{statuses.join(', ')}")
			sleep(5)
		end
	end
 
 	def wait_for_instances(instances)
		$stderr.puts('Waiting for instances........')
		response = @ec2.client.describe_instance_status(:instance_ids => instances)
		puts response
	  statuses = response[:instance_status_set].map { |rr| rr[:system_status] }
	  puts statuses
		loop do
			instances.each do |instance|
			   puts 'Instance console output' + instance.console_output
			sleep(2)
			end
		end
	end
	
	def tag_instances(instances)
		instances.each do |instance|
		  zone = instance.availability_zone.split('-').last
		  instance.tag('Name', value: "TagName")
	  end
	  $stderr.puts('Instances tagged')
	end

end
 
AWS.config(YAML.load(File.read('config.yml')))
ec2 = AWS::EC2.new(:region => "us-west-1")
Starter.new(ec2).start!