Ruby with the F5 BigIP API

Something i found kind of useful.
Original Instructions found on F5’s page:
https://devcentral.f5.com/tech-tips/articles/getting-started-with-ruby-and-icontrol

I assume you’re running ubuntu or debian

1. Install Ruby Gems

apt-get install ruby rubygems libopenssl-ruby

2. Download iControl
iControl Ruby Gem

3. Install iControl gem

gem install f5-icontrol-10.2.0.gem

Run one of the example files (located in /var/lib/gems/1.8/gems/f5-icontrol-10.2.0.a.1/examples/ if installed as ‘root’)

ruby get_version.rb <f5address> <username> <pass>
=> BIG-IP_v10.1.0

And Lastly Here’s a little script to grab the Down/Disabled/Active Pool Members of a given pool
This was constructed with the help of some of the example scripts that installed with the icontrol gem.
Example usage:

f5-pool-members.rb -b bigIPAddress -u user -p pass -n poolname
    -b bigip-address,                BIGIP Load balancer address
        --bigip-address
    -u, --bigip-user bigip-user      Username of BIGIP admin
    -p, --bigip-pass bigip-pass      Password of BIGIP admin
    -n, --pool-name pool-name        Name of pool
    -h, --help                       Display this screen
#!/usr/bin/env ruby
# == Synopsis
# f5-node-initiator - Quickly add nodes to pools
# == Usage
# f5-node-initiator [OPTIONS]
# -h, --help:
#    show help
#
# --bigip-address, -b [hostname]:
#    specify the destination BIG-IP for virtual and pool creation
#
# --bigip-user, -u [username]:
#    username for destination BIG-IP
#
# --bigip-pass, -p [password]:
#    password for destination BIG-IP
#
# --pool-name, -n [name]:
#    name of pool to add node to
#
# --node-definition, -d [ip address:port]:
#    definition for node being added to pool, example: 10.2.1.1:443

require 'rubygems'
require 'f5-icontrol'
require 'optparse'

bigip_address = ''
bigip_user = ''
bigip_pass = ''
pool_name = ''
node_address = ''
node_port = 80

# Current script's name
currentFile = File.basename(__FILE__)

# IF options undefined set to help option
if ARGV.empty?
  ARGV[0] = '-h'
end

 # This hash will hold all of the options
 opt = {}
 optparse = OptionParser.new do |opts|
   # Set a banner, displayed at the top of help screen
   opts.banner = "#{currentFile} -b bigIPAddress -u user -p pass -n poolname"

   # Define the options, and what they do
   opts.on( '-b', '--bigip-address bigip-address', 'BIGIP Load balancer address' ) do |x|
     bigip_address = x
   end
   opts.on( '-u', '--bigip-user bigip-user', 'Username of BIGIP admin' ) do |x|
     bigip_user = x
   end
   opts.on( '-p', '--bigip-pass bigip-pass', 'Password of BIGIP admin' ) do |x|
     bigip_pass = x
   end
   opts.on( '-n', '--pool-name pool-name', 'Name of pool' ) do |x|
     pool_name = x
   end
   #opts.on( '-d', '--node-definition node-definition', 'definition for node being added to pool, example: 10.2.1.1:443' ) do |x|
   #  node_definition = x
   #end

   # This displays the help screen
   opts.on( '-h', '--help', 'Display this screen' ) do
     puts opts
     exit 1
   end
 end


# Parse Command options
optparse.parse!

# Initiate SOAP RPC connection to BIG-IP
bigip = F5::IControl.new(bigip_address, bigip_user, bigip_pass, ['LocalLB.Pool']).get_interfaces

# Insure that target pool exists
unless bigip['LocalLB.Pool'].get_list.include? pool_name
  puts 'ERROR: target pool "' + pool_name +'" does not exist'
  exit 1
end

ActiveMembers = Array.new
DisabledMembers = Array.new
DownMembers = Array.new

bigip['LocalLB.Pool'].get_monitor_instance([ pool_name ])[0].collect do |pool_member1|
  puts
  node_addr = pool_member1['instance']['instance_definition']['ipport']['address'].to_s
  node_port = pool_member1['instance']['instance_definition']['ipport']['port'].to_s
 
  if pool_member1['instance_state'].to_s =~ /INSTANCE_STATE_DOWN/
    DownMembers.push node_addr
  elsif pool_member1['enabled_state'].to_s =~ /false/
    DisabledMembers.push node_addr
  else
    ActiveMembers.push node_addr
  end
  #puts "Node: #{node_addr}:#{node_port}"
  #puts "Node Health: #{pool_member1['instance_state']}"
  #puts "Enabled State: #{pool_member1['enabled_state']}"
end

puts "Poolname: " + pool_name
puts "=============== Unhealthy State Nodes ================"
DownMembers.each do |x|
  puts x
end
puts "=============== Disabled State Nodes ================"
DisabledMembers.each do |x|
  puts x
end
puts "=============== Active and Healthy State Nodes ================"
ActiveMembers.each do |x|
  puts x
end

Example Output:

Poolname: myf5_pool
=============== Unhealthy State Nodes ================
10.0.0.3
=============== Disabled State Nodes ================
10.0.0.4
=============== Active and Healthy State Nodes ================
10.0.0.2
10.0.0.1
Advertisements

cubism.js with graphite server

Recently i’d been looking for a better interface for displaying multiple server stats.
Low and behold i found one created by Square payments.
Here’s what this thing looks like

cubism_example

Their project and examples can be found here:
http://square.github.com/cubism/

I’m going to assume you have an apache server and graphite server running somewhere.

So here’s a small example of how to get this going and your metrics plotting.
It’s pretty much an html file with some javascript
Create an new html file in your apache docroot with the following information ( usually /var/www/ in most ubuntu systems )

1. Put in your page title and some css includes and js includes

Alot of this is going to pull the css and js source directly from Square’s page.
I would recommend you download these files and copy them locally for your own use and serving them from apache yourself.


<meta charset="utf-8" />
Cubism.js</pre>
<style>
@import url(//fonts.googleapis.com/css?family=Yanone+Kaffeesatz:400,700);
@import url(//square.github.com/cubism/style.css);
</style>
<div id="body">
<h2>Host01 Load Average</h2>
<div id="graphs"></div>
<script type="text/javascript" src="http://d3js.org/d3.v2.js"></script>
<script type="text/javascript" src="http://square.github.com/cubism/cubism.v1.js"></script>
<script type="text/javascript" src="http://square.github.com/cubism/highlight.min.js"></script>
<script type="text/javascript">
{font-family:arial,helvettica,sans-serif"}
</script>

2. Setup some cubism settings.

Set the time granularity with the “.step” setting
Set the number of metrics you want to display with the “.size” setting

<script type="text/javascript">
var context = cubism.context()
    .step( 1 * 60 * 1000 )   // 1 minute
    .size(960);  // 1 * 960 = 4 hours

Setup more cubism graphite settings

Set the address of your graphite webserver at the “context.graphite”
Set the height of each row of metric data at “.height”
Set the time shift of how many days you want to go back at “.shift”, like you want to see data from 7 days ago.

var graphite = context.graphite("http://graphite-server.foo-bar.net");
var horizon = context.horizon().metric(graphite.metric).height(100).shift( - 0 * 24 * 60 * 60 * 1000 );

Create a list of metrics you want to see in an array

var metrics = [
   'stats.host01.cpu.load.load',
   'stats.host02.cpu.load.load',
   'nonNegativeDerivative(stats.host02.network.eth0.interface_tx_bytes)'
]

Call d3 and apply a bunch of stuff to the div with the id=graphs

d3.select("#graphs").append("div")
    .attr("class", "axis")
    .call(context.axis().orient("top"));

d3.select("#graphs").append("div")
    .attr("class", "rule")
    .call(context.rule());

d3.select("#graphs").selectAll(".horizon")
    .data(metrics)
  .enter().append("div")
    .attr("class", "horizon")
    .call(horizon);
</script>

Put it all together

The fully constructed HTML file should look something like this


<meta charset="utf-8" />
Cubism.js</pre>
<style>
@import url(//fonts.googleapis.com/css?family=Yanone+Kaffeesatz:400,700);
@import url(//square.github.com/cubism/style.css);
</style>
<div id="body">
<h2>Host01 Load Average</h2>
<div id="graphs"></div>
<script type="text/javascript" src="http://d3js.org/d3.v2.js"></script>
<script type="text/javascript" src="http://square.github.com/cubism/cubism.v1.js"></script>
<script type="text/javascript" src="http://square.github.com/cubism/highlight.min.js"></script>
<script type="text/javascript">
{font-family:arial,helvettica,sans-serif"}
</script>

<script type="text/javascript">
var context = cubism.context()
    .step( 1 * 60 * 1000 )   // 1 minute
    .size(960);  // 1 * 960 = 4 hours

var graphite = context.graphite("http://graphite-server.foo-bar.net");
var horizon = context.horizon().metric(graphite.metric).height(100).shift( - 0 * 24 * 60 * 60 * 1000 );

var metrics = [
   'stats.host01.cpu.load.load',
   'stats.host02.cpu.load.load',
   'nonNegativeDerivative(stats.host02.network.eth0.interface_tx_bytes)'
]

d3.select("#graphs").append("div")
    .attr("class", "axis")
    .call(context.axis().orient("top"));

d3.select("#graphs").append("div")
    .attr("class", "rule")
    .call(context.rule());

d3.select("#graphs").selectAll(".horizon")
    .data(metrics)
  .enter().append("div")
    .attr("class", "horizon")
    .call(horizon);
</script>

A more complicated example using graphite.find function


<meta charset="utf-8" />
Cubism.js</pre>
<style>
@import url(//fonts.googleapis.com/css?family=Yanone+Kaffeesatz:400,700);
@import url(http://square.github.com/cubism/style.css);
</style>
<div id="body">
<h2>Host01 Load Average</h2>
<div id="graphs"></div>
<script type="text/javascript" src="http://d3js.org/d3.v2.js"></script>
<script type="text/javascript" src="http://square.github.com/cubism/cubism.v1.js"></script>
<script type="text/javascript" src="http://square.github.com/cubism/highlight.min.js"></script>
<script type="text/javascript">
{font-family:arial,helvettica,sans-serif"}
</script>

<script type="text/javascript">
var context = cubism.context()
    .step( 1 * 60 * 1000 )   // 1 minute
    .size(960);  // 1 * 960 = 4 hours

var graphite = context.graphite("http://graphite-server.foo-bar.net");
//////// Example: 'stats.host*.cpu.load.load'
graphFind = 'stats.host0*.network.eth*.interface_*_bytes'

// Set The Time Row on Top
d3.select("#graphs").append("div")
    .attr("class", "axis")
    .call(context.axis().orient("top"));

// Set the Vertical Line Bar
d3.select("#graphs").append("div")
    .attr("class", "rule")
    .call(context.rule());

graphite.find(graphFind, function(error, results) {

   // Map find results to array and set to graphite.metric object type
    var metrics = results.sort().map(function(i) {
      return graphite.metric(i);
      //// return it as a nonNegativeDerivative
      // return graphite.metric('nonNegativeDerivative('+i+')');
    });

   // loop through array and print stuff to "graphs" div and apply .height and .colors to object
   for (var i=0;i<metrics.length;i++){
    d3.select("#graphs").call(function(div) {
        div.append("div").selectAll(".horizon")
             .data([metrics[i]])
             .enter().append("div")
             .attr("class", "horizon")
            .call(context.horizon()
              .height(100)
              .colors(["#08519c","#3182bd","#6baed6","#bdd7e7","#bae4b3","#74c476","#31a354","#006d2c"])
            );
    });
   }
   // Set The Time Row on Bottom
   d3.select("#graphs").append("div")
       .attr("class", "axis")
       .call(context.axis().orient("bottom"));
});
</script>

Amazon Glacier with Ruby FOG

Create an Amazon user with the following IAM security policy and save their credentials

{
    "Statement":[{
        "Effect":"Allow",
        "Resource":[
            "*"],
        "Action":[
            "glacier:*"]}
    ]
}

Here’s some shitty code you can try
the “multipart_chunk_size” must be a power of 2 multiple of 1MB

#!/usr/bin/env ruby

require 'rubygems'
require 'fog'

glacier = Fog::AWS::Glacier.new(
    :aws_access_key_id => 'MYACCESSKEY',
    :aws_secret_access_key => 'MYSUPERSECRETKEY')

vault = glacier.vaults.create :id => 'myvault'
archive1 = vault.archives.create :body => File.new('MYFILE.tar.gz'), :multipart_chunk_size => 1024*1024, :description => "adding some archive BLAH"
puts archive1.inspect

The output should show you some info about your upload job

You’re going to want to store your job id so and the object related information for later incase you want to do a fetch of the object

</pre>
<Fog::AWS::Glacier::Archive
 id="dAisPrlq.......jTzjr64Xeg",
 description="adding some archive BLAH",
 body=#<File:MyFile.tar.gz>
 >
<pre>

Retreiving archives is a two step process
1. Create a job to pull the archive into a downloadable state ( archive-retrieval )
2. Pull down the bytes after the job is done and the archive is ready for download ( get archive-output )
References:
http://www.spacevatican.org/2012/9/4/using-glacier-with-fog/
http://blog.vuksan.com/2010/07/20/provision-to-cloud-in-5-minutes-using-fog/

Ruby – regex example

I thought i might throw out some simple examples of using regexes with ruby for when i forget

command = `mpstat -P ALL`
regex = /(?<NAME0>load)\s+average:\s+(?<NAME1>\S+),\s+(?<NAME2>\S+),\s+(?<NAME3>\S+)/x
result = command.match(regex)

# Print your regex
puts " #{result['NAME0']} #{result['NAME1']} #{result['NAME2']} #{result['NAME3']}"
#or
puts " #{result[1]} #{result[2]} #{result[3]} #{result[4]}"

annndd… something more complicated in context of something else


#!/usr/bin/env ruby
require "getopt/long"
require 'socket'

opt = Getopt::Long.getopts(
     ["--server", "-s", Getopt::REQUIRED],
     ["--port", "-p", Getopt::REQUIRED],
     ["--environment", "-e", Getopt::REQUIRED]
)

unless opt["s"] and opt["p"] and opt["e"]
  unless opt["p"] =~ /\d+/
    currentFile = File.basename(__FILE__)
    puts "usage: ./#{currentFile} -s graphiteServer -p graphitePort -e siteEnvironment"
    puts "usage: ./#{currentFile} -s someserver -p 2003 -e dev"
    exit 1
  end
end

statprefix = 'stats'
hostname = `hostname`.chomp
command = `mpstat -P ALL`
epoch = (Time.now.to_i).to_s
graphiteServer = opt["s"]
graphitePort = opt["p"]
siteEnv = opt["e"]

regexTitles = /(?<TITLEID>CPU\s.*)/x
partsTitle = command.match(regexTitles)
partsTitle = partsTitle['TITLEID'].split

regex = /(?<CPUID>all.*)/x
parts = command.match(regex)
parts = parts['CPUID'].split

hash = Hash[partsTitle.zip(parts)]
sock = TCPSocket.new(graphiteServer, graphitePort)
hash.each_pair do |title,value|
  title = title.sub(/^\%/,"")
  sock.puts "#{statprefix}.#{siteEnv}.#{hostname}.cpu.all.#{title} #{value} #{epoch}"
end
sock.close

Graphite – aggregating your second retention bucket

Make sure your storage-aggregator.py is running in order databases to compress down into 2nd and 3rd retention levels.

$ ls
league_number.wsp  user_number.wsp

Find out info on your database “league_number”
Seeing something like xFilesFactor .5 means that 50% of that time retention period has to show
if you have 10 second intervals but are only inputting once every 60 seconds then your xFilesFactor is only .2
Set the xFilesFactor = 0 if you don’t want to drop any of your data

$ /usr/local/bin/whisper-info.py league_number.wsp
maxRetention: 315360000
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 63097960

Archive 0
retention: 21600
secondsPerPoint: 10
points: 2160
size: 25920
offset: 40

Archive 1
retention: 315360000
secondsPerPoint: 60
points: 5256000
size: 63072000
offset: 25960

Resize your whisper database if data is not being aggregated properly

$ /usr/local/bin/whisper-resize.py league_number.wsp
Usage: whisper-resize.py path timePerPoint:timeToStore [timePerPoint:timeToStore]*

timePerPoint and timeToStore specify lengths of time, for example:

60:1440      60 seconds per datapoint, 1440 datapoints = 1 day of retention
15m:8        15 minutes per datapoint, 8 datapoints = 2 hours of retention
1h:7d        1 hour per datapoint, 7 days of retention
12h:2y       12 hours per datapoint, 2 years of retention

If you have fucked up graphs that are truncating data on the second retention size
align them like so

$ /usr/local/bin/whisper-resize.py user_number.wsp --xFilesFactor=0 --aggregationMethod=average 10:6h 1min:10y

Make sure you set your storage-aggregation correctly and make sure it’s started

$ cp storage-aggregation.conf.example storage-aggregation.conf

$ /opt/graphite/bin/carbon-aggregator.py status
carbon-aggregator (instance a) is not running
$ /opt/graphite/bin/carbon-aggregator.py start
Starting carbon-aggregator (instance a)

Ubuntu/Debian Preseeding with LVM

When you have your ubuntu or debian splash screen hit “tab”
then append:

auto url=http://mywebserver/mypreseed.cfg

You can use this example to bring up a quick webserver to server your preseed file: Python Simple HTTP Server

You’ll still probably be prompted for all the locale information because that is requested BEFORE debian/ubuntu loads the preseed configuration. ( yes , dumb i know ). These are the settings that don’t get pulled in.

# keyboard and locale settings
####################################################################

d-i debian-installer/locale string en_US
d-i console-keymaps-at/keymap select sg-latin1

# networking
####################################################################

d-i netcfg/disable_dhcp boolean false
d-i netcfg/get_hostname string localhost
d-i netcfg/get_domain string localdomain
d-i netcfg/choose_interface select eth0

You can get around this by rebuilding your boot cdrom or doing a pxe install that loads all the info ahead of time.

Example

auto url=http://mywebserver/mypreseed.cfg \
locale=en_US console-keymaps-at/keymap=sg-latin1 \
interface=eth0 hostname=localhost domain=localdomain --

Now for actually making your preseed config

Setup Locale info

d-i     debian-installer/locale string en_US
d-i     console-setup/ask_detect boolean false
d-i     time/zone string UTC
d-i     debian-installer/splash boolean false
d-i     debian-installer/language string en
d-i     debain-installer/country string US
d-i     console-keymaps-at/keymap select sg-latin1

Setup Your Networking

d-i     netcfg/get_nameservers  string 192.168.1.5
d-i     netcfg/get_ipaddress    string 192.168.1.10
d-i     netcfg/get_netmask      string 255.255.255.0
d-i     netcfg/get_gateway      string 192.168.1.1
d-i     netcfg/confirm_static   boolean true
d-i     netcfg/get_hostname string tempnode
d-i     netcfg/get_domain string localdomain

Setup your Repository and what directory in the repo to look for your install files

d-i     mirror/country string manual
d-i     mirror/http/hostname string http.us.debian.org
d-i     mirror/http/directory string /debian
d-i     mirror/http/proxy string

Partition Your disks and set them up as lvm

d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-lvm/device_remove_lvm_span boolean true
d-i partman-auto/purge_lvm_from_device  boolean true
d-i partman-auto-lvm/new_vg_name string system

Setup your logical Volumes in LVM and also non-lvm partitions
This will make
* BEWARE OF YOUR FORMATTING OF THIS PART – IT IS SUPER SENSITIVE

/boot : is a bootable filesystem with 300mb and ext3
/ : Is between 10GB and grows to the end of the disk and ext3
swap : uses 200% of the size of system ram for a disk or 8GB

#d-i partman-auto/init_automatically_partition \
#  select Guided - use entire disk and set up LVM
d-i partman-auto/expert_recipe string                         \
      boot-root ::                                            \
              40 300 300 ext3                                 \
                      $primary{ }                             \
                      $bootable{ }                            \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext3 }    \
                      mountpoint{ /boot }                     \
              .                                               \
              2000 10000 1000000000 ext3                      \
                      $lvmok{ }                               \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext3 }    \
                      mountpoint{ / }                         \
              .                                               \
              8000 8000 200% linux-swap                       \
                      $lvmok{ }                               \
                      method{ swap } format{ }                \
              .

Here’s another LVM Example

d-i partman-auto/expert_recipe string                         \
      boot-root ::                                            \
              40 300 300 ext4                                 \
                      $primary{ }                             \
                      $bootable{ }                            \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext4 }    \
                      mountpoint{ /boot }                     \
              .                                               \
              2000 10000 1000000000 ext4                      \
                      $lvmok{ }                               \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext4 }    \
                      mountpoint{ / }                         \
              .                                               \
              2000 1000 10000 ext4                            \
                      $lvmok{ }                               \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext4 }    \
                      mountpoint{ /var }                      \
              .                                               \
              2000 1000 60000 ext4                            \
                      $lvmok{ }                               \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext4 }    \
                      mountpoint{ /var/lib/mysql }                      \
              .                                               \
              2000 1000 30000 ext4                            \
                      $lvmok{ }                               \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext4 }    \
                      mountpoint{ /www }                      \
              .                                               \
              8000 8000 200% linux-swap                       \
                      $lvmok{ }                               \
                      method{ swap } format{ }                \
              .

Answer yes to all the partition and LVM prompts

d-i partman-lvm/confirm boolean true
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition select Finish partitioning and write changes to disk
d-i partman/confirm boolean true

Setup your Time Zone info

d-i     clock-setup/utc boolean true
d-i     clock-setup/ntp boolean true
d-i     clock-setup/ntp-server  string ntp.ubuntu.com
d-i     time/zone string US/Los_Angeles

Setup Root and First User info

d-i     base-installer/kernel/image     string linux-server
d-i     passwd/root-login       boolean true
d-i     passwd/root-password-crypted    password $1$VuPOoDRD$seX.C54E8TUdMkaAmKvjx0
d-i     passwd/user-fullname string Ubuntu User
d-i     passwd/username string ubuntu
d-i     passwd/user-password-crypted password $1$NvFY8IuR$BGqOozSN91ljvQB.pVLDw.
d-i     user-setup/encrypt-home boolean false
d-i     user-setup/allow-password-weak  boolean true
d-i     passwd/user-default-groups      string adm cdrom dialout lpadmin plugdev sambashare

Miscellaneous Stuffage

d-i     apt-setup/services-select       multiselect security
d-i     debian-installer/allow_unauthenticated  string true
d-i     pkgsel/upgrade  select safe-upgrade
d-i     pkgsel/language-packs   multiselect
d-i     pkgsel/update-policy    select none
d-i     pkgsel/updatedb boolean true
popularity-contest popularity-contest/participate boolean false
tasksel tasksel/first multiselect standard, openssh-server
d-i     grub-installer/skip     boolean false
d-i     lilo-installer/skip     boolean false
d-i     grub-installer/only_debian      boolean true
d-i     grub-installer/with_other_os    boolean false
d-i     finish-install/keep-consoles    boolean false
d-i     finish-install/reboot_in_progress       note
d-i     cdrom-detect/eject      boolean true
d-i     debian-installer/exit/halt      boolean false
d-i     debian-installer/exit/poweroff  boolean false

Choose your packages to install

d-i     pkgsel/include string vim openssh-server

What it looks like as one file:

d-i     debian-installer/locale string en_US
d-i     console-setup/ask_detect boolean false
d-i     time/zone string UTC
d-i     debian-installer/splash boolean false
d-i     debian-installer/language string en
d-i     debain-installer/country string US
d-i     netcfg/get_nameservers  string 192.168.1.5
d-i     netcfg/get_ipaddress    string 192.168.1.10
d-i     netcfg/get_netmask      string 255.255.255.0
d-i     netcfg/get_gateway      string 192.168.1.1
d-i     netcfg/confirm_static   boolean true
d-i     netcfg/get_hostname string tempnode
d-i     mirror/country string manual
d-i     mirror/http/hostname string http.us.debian.org
d-i     mirror/http/directory string /debian
d-i     mirror/http/proxy string
d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-lvm/device_remove_lvm_span boolean true
d-i partman-auto/purge_lvm_from_device  boolean true
d-i partman-auto-lvm/new_vg_name string system
#d-i partman-auto/init_automatically_partition \
#  select Guided - use entire disk and set up LVM
d-i partman-auto/expert_recipe string                         \
      boot-root ::                                            \
              40 300 300 ext3                                 \
                      $primary{ }                             \
                      $bootable{ }                            \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext3 }    \
                      mountpoint{ /boot }                     \
              .                                               \
              2000 10000 1000000000 ext3                      \
                      $lvmok{ }                               \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext3 }    \
                      mountpoint{ / }                         \
              .                                               \
              8000 8000 200% linux-swap                       \
                      $lvmok{ }                               \
                      method{ swap } format{ }                \
              .

d-i partman-lvm/confirm boolean true
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition select Finish partitioning and write changes to disk
d-i partman/confirm boolean true
d-i     clock-setup/utc boolean true
d-i     clock-setup/ntp boolean true
d-i     clock-setup/ntp-server  string ntp.ubuntu.com
d-i     time/zone string US/Los_Angeles
d-i     base-installer/kernel/image     string linux-server
d-i     base-installer/kernel/image     string linux-server
d-i     passwd/root-login       boolean true
d-i     passwd/root-password-crypted    password $1$VuPOoDRD$seX.C54E8TUdMkaAmKvjx0
d-i     passwd/user-fullname string Ubuntu User
d-i     user-setup/allow-password-weak  boolean true
d-i     user-setup/encrypt-home boolean false
d-i     passwd/user-default-groups      string adm cdrom dialout lpadmin plugdev sambashare
d-i     apt-setup/services-select       multiselect security
d-i     debian-installer/allow_unauthenticated  string true
d-i     pkgsel/upgrade  select safe-upgrade
d-i     pkgsel/language-packs   multiselect
d-i     pkgsel/update-policy    select none
d-i     pkgsel/updatedb boolean true
popularity-contest popularity-contest/participate boolean false
tasksel tasksel/first multiselect standard, openssh-server
d-i     grub-installer/skip     boolean false
d-i     lilo-installer/skip     boolean false
d-i     grub-installer/only_debian      boolean true
d-i     grub-installer/with_other_os    boolean false
d-i     finish-install/keep-consoles    boolean false
d-i     finish-install/reboot_in_progress       note
d-i     cdrom-detect/eject      boolean true
d-i     debian-installer/exit/halt      boolean false
d-i     debian-installer/exit/poweroff  boolean false
d-i     pkgsel/include string vim openssh-server

Info for this was stolen here ( Thanks ):
http://cptyesterday.wordpress.com/2012/06/17/notes-on-using-expert_recipe-in-debianubuntu-preseed-files/

Preseed Example files found here:

https://github.com/cookingclouds/Preseed

Graphite Install

*this assumes debian or ubuntu

Graphite Cookbook Install

$ git clone https://github.com/cookingclouds/cookbooks/tree/master/cc_graphite

I Know there is a cookbook for graphite in the opscode repo but it has dependencies.
This cookbook shouldn’t have dependencies and is also a lot more messy
Run after installing cookbook :

$ /opt/graphite/bin/carbon-cache.py start
$ cd /opt/graphite/webapp/graphite && python manage.py createsuperuser

Manual Install

Grab the sourcecode


$ wget https://launchpad.net/graphite/0.9/0.9.10/+download/graphite-web-0.9.10.tar.gz
$ wget https://launchpad.net/graphite/0.9/0.9.10/+download/carbon-0.9.10.tar.gz
$ wget https://launchpad.net/graphite/0.9/0.9.10/+download/whisper-0.9.10.tar.gz
$ wget https://launchpad.net/graphite/0.9/0.9.10/+download/check-dependencies.py

Install Whisper Database

$ ls |grep tar |while read i ; do tar xzvf $i ; done
$ apt-get -y install apache2 python-django python-django-tagging python-ldap python-memcache python-cairo
$ cd ~/whisper-0.9.10/
$ python setup.py install

Install Carbon

$ apt-get -y install python-twisted python-simplejson
$ cd ~/carbon-0.9.10/
$ python setup.py install
$ cd /opt/graphite/conf/
$ cp carbon.conf.example carbon.conf
$ cp storage-schemas.conf.example storage-schemas.conf
$ cd /opt/graphite/
$ ./bin/carbon-cache.py start

Install Graphite-web

$ cd ~/graphite-web-0.9.10/
$ python check-dependencies.py
$ python setup.py install
$ cp examples/example-graphite-vhost.conf /etc/apache2/sites-available/default
$ cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
$ mkdir /etc/apache2/run
$ apt-get -y install libapache2-mod-wsgi
$ /etc/init.d/apache2 reload
$ cd /opt/graphite/webapp/graphite/
$ python manage.py syncdb
$ chown -R www-data:www-data /opt/graphite/storage
$ cd /opt/graphite/webapp/graphite
$ cp local_settings.py.example local_settings.py
$ /etc/init.d/apache2 restart

Examples on how to send data to graphite

hostname = * you guessed it
test_001 = service name
1001 = value
$NOW = epoch time to input in graph

$ NOW=`date +%s` ; echo "hostname.test_001 1001 $NOW" |nc localhost 2003

or

$ cd /opt/graphite/examples
$ ./example-client.py

CouchDB 1.0.1 – Replication not completing

* on ubuntu 12.04 using couchdb 1.0.1

Doing this fails – Pulling replication from the slave server:

$ curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"source":"http://192.168.1.100:5984/chef","target":"chef","continuous":false}'

{"error":"http_request_failed","reason":"failed to replicate http://192.168.1.100:5984/chef/"}

Relevant Logs:
From the slave couchdb logs trying to pull replication:
https://gist.github.com/3019911
Example of the couchdb table from the master running: curl http://192.168.1.100:5984/chef/
https://gist.github.com/3020474
From the master couchdb logs:
https://gist.github.com/3020547

Doing this works – Pushing Replication from the master to the slave:

$ curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"target":"http://192.168.1.11:5984/chef","source":"chef","continuous":false}'

{"ok":true,"session_id":"7a0c0632a92e68983b3d0a2d07c47dd8","source_last_seq":23,"history":[{"session_id":"7a0c0632a92e68983b3d0a2d07c47dd8","start_time":"Fri, 29 Jun 2012 21:21:22 GMT","end_time":"Fri, 29 Jun 2012 21:21:22 GMT","start_last_seq":0,"end_last_seq":23,"recorded_seq":23,"missing_checked":0,"missing_found":3,"docs_read":3,"docs_written":3,"doc_write_failures":0}]}