Amazon Glacier with Ruby FOG

Create an Amazon user with the following IAM security policy and save their credentials

{
    "Statement":[{
        "Effect":"Allow",
        "Resource":[
            "*"],
        "Action":[
            "glacier:*"]}
    ]
}

Here’s some shitty code you can try
the “multipart_chunk_size” must be a power of 2 multiple of 1MB

#!/usr/bin/env ruby

require 'rubygems'
require 'fog'

glacier = Fog::AWS::Glacier.new(
    :aws_access_key_id => 'MYACCESSKEY',
    :aws_secret_access_key => 'MYSUPERSECRETKEY')

vault = glacier.vaults.create :id => 'myvault'
archive1 = vault.archives.create :body => File.new('MYFILE.tar.gz'), :multipart_chunk_size => 1024*1024, :description => "adding some archive BLAH"
puts archive1.inspect

The output should show you some info about your upload job

You’re going to want to store your job id so and the object related information for later incase you want to do a fetch of the object

</pre>
<Fog::AWS::Glacier::Archive
 id="dAisPrlq.......jTzjr64Xeg",
 description="adding some archive BLAH",
 body=#<File:MyFile.tar.gz>
 >
<pre>

Retreiving archives is a two step process
1. Create a job to pull the archive into a downloadable state ( archive-retrieval )
2. Pull down the bytes after the job is done and the archive is ready for download ( get archive-output )
References:
http://www.spacevatican.org/2012/9/4/using-glacier-with-fog/
http://blog.vuksan.com/2010/07/20/provision-to-cloud-in-5-minutes-using-fog/

Advertisement

Ruby – Using RVM to create your ruby jail

* This has only been tested with ubuntu 12.04 – you also already need gcc and ruby of some sort installed
These instructions allow you to run your own version of ruby and rubygems from your home folder

Download and install rvm
Set a couple of environment variables

bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer) 

echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"' >> ~/.bash_profile 
echo 'PATH=$PATH:$HOME/.rvm/usr/bin # Add RVM to PATH for scripting' >> ~/.bash_profile
. ~/.bash_profile

Install Ruby 1.9.3

rvm install 1.9.3
rvm use 1.9.3 --default

Install some gnu tools you need to install gems

wget ftp://ftp.gnu.org/gnu/m4/m4-1.4.16.tar.gz 
tar xzvf m4-1.4.16.tar.gz && cd m4-1.4.16/
./configure --prefix=$HOME/.rvm/usr
make && make install

wget ftp://ftp.gnu.org/gnu/gperf/gperf-3.0.4.tar.gz
tar xzvf gperf-3.0.4.tar.gz
cd gperf-3.0.4/
./configure --prefix=$HOME/.rvm/usr
make && make install

wget ftp://invisible-island.net/byacc/byacc.tar.gz
tar xzvf byacc.tar.gz
cd byacc-20121003/
./configure --prefix=$HOME/.rvm/usr
make && make install

wget ftp://ftp.gnu.org/gnu/termcap/termcap-1.3.1.tar.gz
tar xzvf termcap-1.3.1.tar.gz
cd termcap-1.3.1/
./configure --prefix=$HOME/.rvm/usr
make && make install

wget ftp://ftp.gnu.org/gnu/ncurses/ncurses-5.9.tar.gz
tar xzvf ncurses-5.9.tar.gz
cd ncurses-5.9/
./configure --prefix=$HOME/.rvm/usr CFLAGS=-fPIC
make && make install

wget ftp://ftp.gnu.org/gnu/texinfo/texinfo-4.13a.tar.gz
tar xzvf texinfo-4.13a.tar.gz
cd texinfo-4.13/
./configure --prefix=$HOME/.rvm/usr LDFLAGS=-L$HOME/.rvm/usr/lib CPPFLAGS=-I$HOME/.rvm/usr/include/ncurses
make && make install

Install some more tools you need to install gems
This time just use the ones that rvm has packaged
# ORDER MATTERS !!!

for i in curl zlib readline openssl iconv pkgconfig autoconf libxml2 libxslt libyaml ; do rvm pkg install $i --verify-downloads 1 --with-opt-dir=$HOME/.rvm/usr ; done

Reinstall ruby 1.9.3 with the new path of your tools compiled in

rvm reinstall 1.9.3 --with-opt-dir=$HOME/.rvm/usr

Install the ‘fog’ gem

gem install fog

Your home folder will now be 1.4GB large but you’ll have a self contained ruby and rubygems installation with the fog library available

Amazon EC2 – Clone system into AMI without reboot

This requires you to install the Amazon CLI API tools located at: http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
This also requires java installed
Api Reference here: http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-CreateImage.html

This is for the lazy people like myself to reference
This will snapshot all ebs volumes that are also attached to the system you’re creating an image from.
If you’re not pausing this instance when you take a snapshot you can get an inconsistent state, but if you can’t afford downtime then just do this and if it doesn’t work when you boot a new system do it again

james.tran$ ec2cim i-111a111 -n my_newami_name -d my_description --no-reboot --region us-west-1 -W AWSUSER_SECRET_KEY -O AWSUSER_ACCESS_KEY
IMAGE     ami-543106

Amazon EC2 – Snapshotting EBS script

I’ve stolen the example from here but i’ve made my own adjustments

1. Create an Amazon IAM user for snapshotting and save the credentials file. ( You’ll need them to use the amazon cli api , it comes in csv format)
Create an Amazon IAM Group for snapshot permissions
add a “Custom Policy” and paste the code block below
Example IAM Policy:

{
  "Statement": [
    {
      "Action": [
        "ec2:DescribeVolumes",
        "ec2:CreateSnapshot",
        "ec2:DeleteSnapshot",
        "ec2:DescribeSnapshotAttribute",
        "ec2:DescribeSnapshots",
        "ec2:ModifySnapshotAttribute",
        "ec2:ResetSnapshotAttribute"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ]
}

2. Install the Amazon CLI Tools

$ wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
$ unzip ec2-api-tools.zip
$ mv ec2-api-tools /opt
$ ln -s /opt/ec2-api-tools /opt/aws

3. Install the below script in cron on your snapshot interval and change the “Constants” and also plugin your AWS Keys
Script Below

#!/bin/bash
# EBS Snapshot volume script
# Constants - You'll want to edit these
JAVA_HOME="/usr"
EC2_HOME="/opt/aws"
ec2_bin="/opt/aws/bin"
export EC2_HOME
export JAVA_HOME
LOGFILE='/var/log/aws_snapshot.log'
TMPFILE='/tmp/snap_info.txt'

VOLTMPFILE='/tmp/volume_info.txt'

# Retention in days
RETENTION="7"

# AWS ACCESS INFO
access_key='SOMEACCESSKEY'
secret_key='SOMESECRETKEY'
instance_id=`wget -q -O- http://169.254.169.254/latest/meta-data/instance-id`

# Dates
datecheck_7d=`date +%Y-%m-%d --date "$RETENTION days ago"`
datecheck_s_7d=`date --date="$datecheck_7d" +%s`
datenow=`date +%Y-%m-%d-%H:%M:%S`

# Add entry in logfile for run begin
echo "${datenow} ======= BEGIN SNAPSHOT SCRIPT =========" 2>&1 >> $LOGFILE
# Get all volume info and copy to temp file
$ec2_bin/ec2-describe-volumes -O $access_key -W $secret_key  --filter "attachment.instance-id=$instance_id" > $VOLTMPFILE 2>&1

# Get all snapshot info
$ec2_bin/ec2-describe-snapshots -O $access_key -W $secret_key | grep "$instance_id" > $TMPFILE 2>&1

# Loop to remove any snapshots older than 7 days
for obj0 in $(cat $TMPFILE | awk '{print $5}')
do
        snapshot_name=`cat $TMPFILE | grep "$obj0" | awk '{print $2}'`
        datecheck_old=`cat $TMPFILE | grep "$snapshot_name" | awk '{print $5}' | awk -F "T" '{print $1}'`
        datecheck_s_old=`date --date="$datecheck_old" +%s`

        # Check if snapshot is older than retention days
        if (( $datecheck_s_old <= $datecheck_s_7d ));
        then
                echo "deleting snapshot $snapshot_name ... older than $RETENTION days" 2>&1 >> $LOGFILE
                $ec2_bin/ec2-delete-snapshot -O $access_key -W $secret_key $snapshot_name
        else
                echo "not deleting snapshot $snapshot_name ... not older than $RETENTION days" 2>&1 >> $LOGFILE
        fi
done

# Create snapshot
for volume in $(cat $VOLTMPFILE | grep "VOLUME" | awk '{print $2}')
do
        # Description cannot have spaces
        description="instance-id:${instance_id}_vol-id:${volume}_`hostname`_backup-`date +%Y-%m-%d`"
        echo "Creating Snapshot for the volume: $volume with description: $description" 2>&1 >> $LOGFILE
        $ec2_bin/ec2-create-snapshot -O $access_key -W $secret_key -d "$description" $volume 2>&1 >> $LOGFILE
done