Amazon Glacier with Ruby FOG

Create an Amazon user with the following IAM security policy and save their credentials


Here’s some shitty code you can try
the “multipart_chunk_size” must be a power of 2 multiple of 1MB

#!/usr/bin/env ruby

require 'rubygems'
require 'fog'

glacier =
    :aws_access_key_id => 'MYACCESSKEY',
    :aws_secret_access_key => 'MYSUPERSECRETKEY')

vault = glacier.vaults.create :id => 'myvault'
archive1 = vault.archives.create :body =>'MYFILE.tar.gz'), :multipart_chunk_size => 1024*1024, :description => "adding some archive BLAH"
puts archive1.inspect

The output should show you some info about your upload job

You’re going to want to store your job id so and the object related information for later incase you want to do a fetch of the object

 description="adding some archive BLAH",

Retreiving archives is a two step process
1. Create a job to pull the archive into a downloadable state ( archive-retrieval )
2. Pull down the bytes after the job is done and the archive is ready for download ( get archive-output )


Ruby – Using RVM to create your ruby jail

* This has only been tested with ubuntu 12.04 – you also already need gcc and ruby of some sort installed
These instructions allow you to run your own version of ruby and rubygems from your home folder

Download and install rvm
Set a couple of environment variables

bash -s stable < <(curl -s 

echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"' >> ~/.bash_profile 
echo 'PATH=$PATH:$HOME/.rvm/usr/bin # Add RVM to PATH for scripting' >> ~/.bash_profile
. ~/.bash_profile

Install Ruby 1.9.3

rvm install 1.9.3
rvm use 1.9.3 --default

Install some gnu tools you need to install gems

tar xzvf m4-1.4.16.tar.gz && cd m4-1.4.16/
./configure --prefix=$HOME/.rvm/usr
make && make install

tar xzvf gperf-3.0.4.tar.gz
cd gperf-3.0.4/
./configure --prefix=$HOME/.rvm/usr
make && make install

tar xzvf byacc.tar.gz
cd byacc-20121003/
./configure --prefix=$HOME/.rvm/usr
make && make install

tar xzvf termcap-1.3.1.tar.gz
cd termcap-1.3.1/
./configure --prefix=$HOME/.rvm/usr
make && make install

tar xzvf ncurses-5.9.tar.gz
cd ncurses-5.9/
./configure --prefix=$HOME/.rvm/usr CFLAGS=-fPIC
make && make install

tar xzvf texinfo-4.13a.tar.gz
cd texinfo-4.13/
./configure --prefix=$HOME/.rvm/usr LDFLAGS=-L$HOME/.rvm/usr/lib CPPFLAGS=-I$HOME/.rvm/usr/include/ncurses
make && make install

Install some more tools you need to install gems
This time just use the ones that rvm has packaged

for i in curl zlib readline openssl iconv pkgconfig autoconf libxml2 libxslt libyaml ; do rvm pkg install $i --verify-downloads 1 --with-opt-dir=$HOME/.rvm/usr ; done

Reinstall ruby 1.9.3 with the new path of your tools compiled in

rvm reinstall 1.9.3 --with-opt-dir=$HOME/.rvm/usr

Install the ‘fog’ gem

gem install fog

Your home folder will now be 1.4GB large but you’ll have a self contained ruby and rubygems installation with the fog library available

Amazon EC2 – Clone system into AMI without reboot

This requires you to install the Amazon CLI API tools located at:
This also requires java installed
Api Reference here:

This is for the lazy people like myself to reference
This will snapshot all ebs volumes that are also attached to the system you’re creating an image from.
If you’re not pausing this instance when you take a snapshot you can get an inconsistent state, but if you can’t afford downtime then just do this and if it doesn’t work when you boot a new system do it again

james.tran$ ec2cim i-111a111 -n my_newami_name -d my_description --no-reboot --region us-west-1 -W AWSUSER_SECRET_KEY -O AWSUSER_ACCESS_KEY
IMAGE     ami-543106

Amazon EC2 – Snapshotting EBS script

I’ve stolen the example from here but i’ve made my own adjustments

1. Create an Amazon IAM user for snapshotting and save the credentials file. ( You’ll need them to use the amazon cli api , it comes in csv format)
Create an Amazon IAM Group for snapshot permissions
add a “Custom Policy” and paste the code block below
Example IAM Policy:

  "Statement": [
      "Action": [
      "Effect": "Allow",
      "Resource": [

2. Install the Amazon CLI Tools

$ wget
$ unzip
$ mv ec2-api-tools /opt
$ ln -s /opt/ec2-api-tools /opt/aws

3. Install the below script in cron on your snapshot interval and change the “Constants” and also plugin your AWS Keys
Script Below

# EBS Snapshot volume script
# Constants - You'll want to edit these
export EC2_HOME
export JAVA_HOME


# Retention in days

instance_id=`wget -q -O-`

# Dates
datecheck_7d=`date +%Y-%m-%d --date "$RETENTION days ago"`
datecheck_s_7d=`date --date="$datecheck_7d" +%s`
datenow=`date +%Y-%m-%d-%H:%M:%S`

# Add entry in logfile for run begin
echo "${datenow} ======= BEGIN SNAPSHOT SCRIPT =========" 2>&1 >> $LOGFILE
# Get all volume info and copy to temp file
$ec2_bin/ec2-describe-volumes -O $access_key -W $secret_key  --filter "attachment.instance-id=$instance_id" > $VOLTMPFILE 2>&1

# Get all snapshot info
$ec2_bin/ec2-describe-snapshots -O $access_key -W $secret_key | grep "$instance_id" > $TMPFILE 2>&1

# Loop to remove any snapshots older than 7 days
for obj0 in $(cat $TMPFILE | awk '{print $5}')
        snapshot_name=`cat $TMPFILE | grep "$obj0" | awk '{print $2}'`
        datecheck_old=`cat $TMPFILE | grep "$snapshot_name" | awk '{print $5}' | awk -F "T" '{print $1}'`
        datecheck_s_old=`date --date="$datecheck_old" +%s`

        # Check if snapshot is older than retention days
        if (( $datecheck_s_old <= $datecheck_s_7d ));
                echo "deleting snapshot $snapshot_name ... older than $RETENTION days" 2>&1 >> $LOGFILE
                $ec2_bin/ec2-delete-snapshot -O $access_key -W $secret_key $snapshot_name
                echo "not deleting snapshot $snapshot_name ... not older than $RETENTION days" 2>&1 >> $LOGFILE

# Create snapshot
for volume in $(cat $VOLTMPFILE | grep "VOLUME" | awk '{print $2}')
        # Description cannot have spaces
        description="instance-id:${instance_id}_vol-id:${volume}_`hostname`_backup-`date +%Y-%m-%d`"
        echo "Creating Snapshot for the volume: $volume with description: $description" 2>&1 >> $LOGFILE
        $ec2_bin/ec2-create-snapshot -O $access_key -W $secret_key -d "$description" $volume 2>&1 >> $LOGFILE