Ruby – Using RVM to create your ruby jail

* This has only been tested with ubuntu 12.04 – you also already need gcc and ruby of some sort installed
These instructions allow you to run your own version of ruby and rubygems from your home folder

Download and install rvm
Set a couple of environment variables

bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer) 

echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"' >> ~/.bash_profile 
echo 'PATH=$PATH:$HOME/.rvm/usr/bin # Add RVM to PATH for scripting' >> ~/.bash_profile
. ~/.bash_profile

Install Ruby 1.9.3

rvm install 1.9.3
rvm use 1.9.3 --default

Install some gnu tools you need to install gems

wget ftp://ftp.gnu.org/gnu/m4/m4-1.4.16.tar.gz 
tar xzvf m4-1.4.16.tar.gz && cd m4-1.4.16/
./configure --prefix=$HOME/.rvm/usr
make && make install

wget ftp://ftp.gnu.org/gnu/gperf/gperf-3.0.4.tar.gz
tar xzvf gperf-3.0.4.tar.gz
cd gperf-3.0.4/
./configure --prefix=$HOME/.rvm/usr
make && make install

wget ftp://invisible-island.net/byacc/byacc.tar.gz
tar xzvf byacc.tar.gz
cd byacc-20121003/
./configure --prefix=$HOME/.rvm/usr
make && make install

wget ftp://ftp.gnu.org/gnu/termcap/termcap-1.3.1.tar.gz
tar xzvf termcap-1.3.1.tar.gz
cd termcap-1.3.1/
./configure --prefix=$HOME/.rvm/usr
make && make install

wget ftp://ftp.gnu.org/gnu/ncurses/ncurses-5.9.tar.gz
tar xzvf ncurses-5.9.tar.gz
cd ncurses-5.9/
./configure --prefix=$HOME/.rvm/usr CFLAGS=-fPIC
make && make install

wget ftp://ftp.gnu.org/gnu/texinfo/texinfo-4.13a.tar.gz
tar xzvf texinfo-4.13a.tar.gz
cd texinfo-4.13/
./configure --prefix=$HOME/.rvm/usr LDFLAGS=-L$HOME/.rvm/usr/lib CPPFLAGS=-I$HOME/.rvm/usr/include/ncurses
make && make install

Install some more tools you need to install gems
This time just use the ones that rvm has packaged
# ORDER MATTERS !!!

for i in curl zlib readline openssl iconv pkgconfig autoconf libxml2 libxslt libyaml ; do rvm pkg install $i --verify-downloads 1 --with-opt-dir=$HOME/.rvm/usr ; done

Reinstall ruby 1.9.3 with the new path of your tools compiled in

rvm reinstall 1.9.3 --with-opt-dir=$HOME/.rvm/usr

Install the ‘fog’ gem

gem install fog

Your home folder will now be 1.4GB large but you’ll have a self contained ruby and rubygems installation with the fog library available

Advertisements

Amazon EC2 – Clone system into AMI without reboot

This requires you to install the Amazon CLI API tools located at: http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
This also requires java installed
Api Reference here: http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-CreateImage.html

This is for the lazy people like myself to reference
This will snapshot all ebs volumes that are also attached to the system you’re creating an image from.
If you’re not pausing this instance when you take a snapshot you can get an inconsistent state, but if you can’t afford downtime then just do this and if it doesn’t work when you boot a new system do it again

james.tran$ ec2cim i-111a111 -n my_newami_name -d my_description --no-reboot --region us-west-1 -W AWSUSER_SECRET_KEY -O AWSUSER_ACCESS_KEY
IMAGE     ami-543106

Amazon EC2 – Snapshotting EBS script

I’ve stolen the example from here but i’ve made my own adjustments

1. Create an Amazon IAM user for snapshotting and save the credentials file. ( You’ll need them to use the amazon cli api , it comes in csv format)
Create an Amazon IAM Group for snapshot permissions
add a “Custom Policy” and paste the code block below
Example IAM Policy:

{
  "Statement": [
    {
      "Action": [
        "ec2:DescribeVolumes",
        "ec2:CreateSnapshot",
        "ec2:DeleteSnapshot",
        "ec2:DescribeSnapshotAttribute",
        "ec2:DescribeSnapshots",
        "ec2:ModifySnapshotAttribute",
        "ec2:ResetSnapshotAttribute"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ]
}

2. Install the Amazon CLI Tools

$ wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
$ unzip ec2-api-tools.zip
$ mv ec2-api-tools /opt
$ ln -s /opt/ec2-api-tools /opt/aws

3. Install the below script in cron on your snapshot interval and change the “Constants” and also plugin your AWS Keys
Script Below

#!/bin/bash
# EBS Snapshot volume script
# Constants - You'll want to edit these
JAVA_HOME="/usr"
EC2_HOME="/opt/aws"
ec2_bin="/opt/aws/bin"
export EC2_HOME
export JAVA_HOME
LOGFILE='/var/log/aws_snapshot.log'
TMPFILE='/tmp/snap_info.txt'

VOLTMPFILE='/tmp/volume_info.txt'

# Retention in days
RETENTION="7"

# AWS ACCESS INFO
access_key='SOMEACCESSKEY'
secret_key='SOMESECRETKEY'
instance_id=`wget -q -O- http://169.254.169.254/latest/meta-data/instance-id`

# Dates
datecheck_7d=`date +%Y-%m-%d --date "$RETENTION days ago"`
datecheck_s_7d=`date --date="$datecheck_7d" +%s`
datenow=`date +%Y-%m-%d-%H:%M:%S`

# Add entry in logfile for run begin
echo "${datenow} ======= BEGIN SNAPSHOT SCRIPT =========" 2>&1 >> $LOGFILE
# Get all volume info and copy to temp file
$ec2_bin/ec2-describe-volumes -O $access_key -W $secret_key  --filter "attachment.instance-id=$instance_id" > $VOLTMPFILE 2>&1

# Get all snapshot info
$ec2_bin/ec2-describe-snapshots -O $access_key -W $secret_key | grep "$instance_id" > $TMPFILE 2>&1

# Loop to remove any snapshots older than 7 days
for obj0 in $(cat $TMPFILE | awk '{print $5}')
do
        snapshot_name=`cat $TMPFILE | grep "$obj0" | awk '{print $2}'`
        datecheck_old=`cat $TMPFILE | grep "$snapshot_name" | awk '{print $5}' | awk -F "T" '{print $1}'`
        datecheck_s_old=`date --date="$datecheck_old" +%s`

        # Check if snapshot is older than retention days
        if (( $datecheck_s_old <= $datecheck_s_7d ));
        then
                echo "deleting snapshot $snapshot_name ... older than $RETENTION days" 2>&1 >> $LOGFILE
                $ec2_bin/ec2-delete-snapshot -O $access_key -W $secret_key $snapshot_name
        else
                echo "not deleting snapshot $snapshot_name ... not older than $RETENTION days" 2>&1 >> $LOGFILE
        fi
done

# Create snapshot
for volume in $(cat $VOLTMPFILE | grep "VOLUME" | awk '{print $2}')
do
        # Description cannot have spaces
        description="instance-id:${instance_id}_vol-id:${volume}_`hostname`_backup-`date +%Y-%m-%d`"
        echo "Creating Snapshot for the volume: $volume with description: $description" 2>&1 >> $LOGFILE
        $ec2_bin/ec2-create-snapshot -O $access_key -W $secret_key -d "$description" $volume 2>&1 >> $LOGFILE
done

Adding EBS Volumes with Opscode’s AWS cookbook

1. Download opscode’s aws cookbook and put it into your own cookbook repo

$ git clone https://github.com/opscode/cookbooks.git opscode-cookbooks
$ cp -r opscode-cookbooks/aws my-cookbooks/
$ cd my-cookbooks 

2. Create a new cookbook that will utilize the aws cookbook

$ knife cookbook create aws-tests

3. Set the cookbook to have the dependency of the opscode aws cookbook

$ vi my-cookbooks/aws-tests/metadata.rb
maintainer       "YOUR_COMPANY_NAME"
maintainer_email "YOUR_EMAIL"
license          "All rights reserved"
description      "Installs/Configures aws_tests"
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version          "0.0.1"

depends "aws"

4. Create your recipe to create and attach a new EBS volume to your ec2 instance

$ vi vi my-cookbooks/aws-tests/recipes/default.rb
# Create and attach your new EBS volume
aws_ebs_volume "new_ebs_volume" do
  aws_access_key "MYAPIKEY"
  aws_secret_access_key "MYAPIKEYSECRET"
  size 1
  device "/dev/xvdi"
  action [ :create, :attach ]
end

5. Create a filesystem and mount your new volume

# Create your partition and filesystem for ext4
bash "create_filesystem" do
  user "root"
  code <<-EOH
    parted /dev/xvdi mklabel gpt
    parted /dev/xvdi mkpart logical ext4 1 -1
    parted /dev/xvdi set 1 lvm on
    yes | parted /dev/xvdi mkpart logical ext4 1 -- "-1"
    mkfs.ext4 /dev/xvdi1
  EOH
  not_if "parted /dev/xvdi1 |grep ext4"
end


directory "/mnt/test" do
  owner "root"
  group "root"
  mode "0755"
  recursive true
end

mount "/mnt/test" do
  device "/dev/xvdi1"
  options "rw noatime"
  fstype "ext4"
  action [ :enable, :mount ]
  not_if "cat /proc/mounts |grep /mnt/test"
end

6. Add aws and aws_tests recipes to your node

$ knife node edit i-fff4f8c
{
  "chef_environment": "_default",
  "name": "i-fff4f8",
  "run_list": [
    "recipe[aws]",
    "recipe[aws_tests]"
  ],
  "normal": {
    "tags": [

    ],
  }
}

7. Run chef-client on your node

$ chef-client