Knife EC2 Extension – Install and Use


$ apt-get install -y libxslt-dev libxml2-dev
$ gem install knife-ec2
$ gem install net-ssh-multi

Get Your AWS Keys

Login to your AWS account at
Go to My Account/Console -> Security Credentials

Scroll Down to The Certificates and Secret Keys Menu and generate your new access keys as needed

Generate the Keypair associated with your new ec2 instances ( for ssh )

Create a new keypair. This should result in a pem file output to you. If you lose this file you will not be able to access any ec2 instances associated with it unless you have alternate accounts you can login with.


$ cd ~/.chef
$ vi knife.rb

Append the following to your knife.rb

### AWS Configuration ###

## The below lines allow you to use the ec2 api
knife[:aws_access_key_id] ='< AWS ACCESS KEY GOES HERE >'
knife[:aws_secret_access_key] ='< AWS SECRET KEY GOES HERE >'

## The below allow you to ssh into new ec2 instance that are associated with the keypair below
## You can alternately choose to specify the username and key location on the knife command line
# knife[:aws_ssh_key_id] ='james-aws'
# knife[:identity_file] ="/home/james/.ssh/james-aws.pem"

Test knife-ec2 command

bootstrap file squeeze.rb can be grabbed from here:

$ knife ec2 server list
$ knife ec2 server create -I ami-e00df089 -f t1.micro -Z us-east-1a -G "default_security" -k james-aws --ssh-key /home/james/.ssh/james-aws.pem --template-file /home/james/bootstrap/squeeze.rb

knife-ec2 command reference list

knife ec2 server create --help
knife ec2 server create (options)
    -Z, --availability-zone ZONE     The Availability Zone
    -A, --aws-access-key-id KEY      Your AWS Access Key ID
    -K SECRET,                       Your AWS API Secret Access Key
        --user-data USER_DATA_FILE   The EC2 User Data file to provision the instance with
        --bootstrap-version VERSION  The version of Chef to install
    -N, --node-name NAME             The Chef node name for your new node
        --server-url URL             Chef Server URL
    -k, --key KEY                    API Client Key
        --color                      Use colored output
    -c, --config CONFIG              The configuration file to use
        --defaults                   Accept default values for all questions
    -d, --distro DISTRO              Bootstrap a distro using a template
        --ebs-no-delete-on-term      Do not delete EBS volumn on instance termination
        --ebs-size SIZE              The size of the EBS volume in GB, for EBS-backed instances
    -e, --editor EDITOR              Set the editor to use for interactive commands
    -E, --environment ENVIRONMENT    Set the Chef environment
    -f, --flavor FLAVOR              The flavor of server (m1.small, m1.medium, etc)
    -F, --format FORMAT              Which format to use for output
    -i IDENTITY_FILE,                The SSH identity file used for authentication
    -I, --image IMAGE                The AMI for the server
        --no-color                   Don't use colors in the output
    -n, --no-editor                  Do not open EDITOR, just accept the data as is
        --no-host-key-verify         Disable host key verification
    -u, --user USER                  API Client Username
        --prerelease                 Install the pre-release chef gems
        --print-after                Show the data after a destructive operation
        --region REGION              Your AWS region
    -r, --run-list RUN_LIST          Comma separated list of roles/recipes to apply
    -G, --groups X,Y,Z               The security groups for this server
    -S, --ssh-key KEY                The AWS SSH key id
    -P, --ssh-password PASSWORD      The ssh password
    -x, --ssh-user USERNAME          The ssh username
    -s, --subnet SUBNET-ID           create node in this Virtual Private Cloud Subnet ID (implies VPC mode)
        --template-file TEMPLATE     Full path to location of template to use
    -V, --verbose                    More verbose output. Use twice for max verbosity
    -v, --version                    Show chef version
    -y, --yes                        Say yes to all prompts for confirmation
    -h, --help                       Show this message

Graphite Install

*this assumes debian or ubuntu

Graphite Cookbook Install

$ git clone

I Know there is a cookbook for graphite in the opscode repo but it has dependencies.
This cookbook shouldn’t have dependencies and is also a lot more messy
Run after installing cookbook :

$ /opt/graphite/bin/ start
$ cd /opt/graphite/webapp/graphite && python createsuperuser

Manual Install

Grab the sourcecode

$ wget
$ wget
$ wget
$ wget

Install Whisper Database

$ ls |grep tar |while read i ; do tar xzvf $i ; done
$ apt-get -y install apache2 python-django python-django-tagging python-ldap python-memcache python-cairo
$ cd ~/whisper-0.9.10/
$ python install

Install Carbon

$ apt-get -y install python-twisted python-simplejson
$ cd ~/carbon-0.9.10/
$ python install
$ cd /opt/graphite/conf/
$ cp carbon.conf.example carbon.conf
$ cp storage-schemas.conf.example storage-schemas.conf
$ cd /opt/graphite/
$ ./bin/ start

Install Graphite-web

$ cd ~/graphite-web-0.9.10/
$ python
$ python install
$ cp examples/example-graphite-vhost.conf /etc/apache2/sites-available/default
$ cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
$ mkdir /etc/apache2/run
$ apt-get -y install libapache2-mod-wsgi
$ /etc/init.d/apache2 reload
$ cd /opt/graphite/webapp/graphite/
$ python syncdb
$ chown -R www-data:www-data /opt/graphite/storage
$ cd /opt/graphite/webapp/graphite
$ cp
$ /etc/init.d/apache2 restart

Examples on how to send data to graphite

hostname = * you guessed it
test_001 = service name
1001 = value
$NOW = epoch time to input in graph

$ NOW=`date +%s` ; echo "hostname.test_001 1001 $NOW" |nc localhost 2003


$ cd /opt/graphite/examples
$ ./

CouchDB 1.0.1 – Replication not completing

* on ubuntu 12.04 using couchdb 1.0.1

Doing this fails – Pulling replication from the slave server:

$ curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"source":"","target":"chef","continuous":false}'

{"error":"http_request_failed","reason":"failed to replicate"}

Relevant Logs:
From the slave couchdb logs trying to pull replication:
Example of the couchdb table from the master running: curl
From the master couchdb logs:

Doing this works – Pushing Replication from the master to the slave:

$ curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"target":"","source":"chef","continuous":false}'

{"ok":true,"session_id":"7a0c0632a92e68983b3d0a2d07c47dd8","source_last_seq":23,"history":[{"session_id":"7a0c0632a92e68983b3d0a2d07c47dd8","start_time":"Fri, 29 Jun 2012 21:21:22 GMT","end_time":"Fri, 29 Jun 2012 21:21:22 GMT","start_last_seq":0,"end_last_seq":23,"recorded_seq":23,"missing_checked":0,"missing_found":3,"docs_read":3,"docs_written":3,"doc_write_failures":0}]}

Migrating Chef CouchDB to Multi-Master CouchDB

* assuming you are using ubuntu/debian
chef-server =
couchdb01 =
couchdb02 =

Enable chef-server couchdb to listen on all interfaces

root@chefserver:~# sed -i bak s/bind_address = = /etc/couchdb/default.ini
root@chefserver:~# /etc/init.d/couchdb restart

Install CouchDB on couchdb01/couchdb02 and set to listen on all interfaces

root@couchdb01:~# apt-get -y install couchdb
root@couchdb01:~# /etc/init.d/couchdb stop
root@couchdb01:~# sed -i bak s/bind_address = = /etc/couchdb/default.ini
root@couchdb01:~# /etc/init.d/couchdb start

Create the empty chef table on couchdb01/couchdb02

root@couchdb01:~# curl -X PUT http://localhost:5984/chef

root@couchdb02:~# curl -X PUT http://localhost:5984/chef

Push the chef table from chef-server to couchdb01/02 and enable a continuous replication

To Couchdb02

root@chef-server:/var/lib/couchdb# curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"source":"chef","target":"","continuous":true}'

To Couchdb01

root@chef-server:~# curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"source":"chef","target":"","continuous":true}'

Setup the multi-master replication for couchdb01/02

Enable continuous replication FROM couchdb01 to couchdb02

root@couchdb01:/var/lib/couchdb# curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"source":"chef","target":"","continuous":true}'

Enable continuous replication FROM couchdb02 to couchdb01

root@couchdb02:/var/lib/couchdb# curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"source":"chef","target":"","continuous":true}'

Install Apache and generate config on Chef Server

root@chefserver:~# apt-get -y install apache2
root@chefserver:~# mkdir -p /usr/share/chef-server/public
root@chefserver:~# for i in rewrite proxy status proxy_http proxy_balancer headers ; do a2enmod $i ; done
root@chefserver:~# cd /etc/apache2/sites-available
root@chefserver:~# echo "Listen 5984" | tee -a chef_couchdb_loadbalancer
root@chefserver:~# echo '<VirtualHost *:5984>' | tee -a chef_couchdb_loadbalancer
root@chefserver:~# MYHOST=$(hostname -f)
root@chefserver:~# echo "ServerName ${MYHOST}-couchdb" |tee -a chef_couchdb_loadbalancer
root@chefserver:~# cat>>chef_couchdb_loadbalancer<<EOF
<Proxy balancer://couchlb>
ProxyPass / balancer://couchlb
ProxyPassReverse / balancer://couchlb
DocumentRoot /usr/share/chef-server/public
LogLevel info
ErrorLog /var/log/chef/chef_couchdb_apache2-error.log
CustomLog /var/log/chef/chef_couchdb_apache2-access.log combined
RewriteEngine On
RewriteRule ^/(.*)$ balancer://couchlb%{REQUEST_URI} [P,QSA,L]

Stop Couchdb on Chef-Server

*This will stop the continuous replication to couchdb01/couchdb02

root@chefserver:~# /etc/init.d/couchdb stop

Start Apache load balancer on Chef-Server

root@chefserver:~# a2ensite chef_couchdb_loadbalancer
root@chefserver:~# /etc/init.d/apache2 restart

Test your couchdb balancer:


You probably also want to edit your init scripts on couchdb01/02 to automatically restart the continuous replication upon start or restart
The replication does not persist after you stop a couchdb instance unless you explicitly issue the command again

Chef Server – Threading Merb – Chef Server API Service

This will only make your api system faster if you have enough cpus to support it.
It should be 1 merb worker per core ( * i think * )

Install Apache and enable the necessary mods

$ apt-get -y install apache2
$ mkdir -p /usr/share/chef-server/public
$ for i in rewrite proxy status proxy_http proxy_balancer headers ; do a2enmod $i ; done

Stop Chef-Server

$ /etc/init.d/chef-server stop

Edit your Chef configuration file ( Replace Worker Threads and Port Numbers as needed )

$ sed -i s/PORT=4000/PORT=5000/g /etc/default/chef-server
$ echo "WORKERTHREADS=4" | tee -a /etc/default/chef-server

Edit the Chef init script ( Back it up first )

$ cp /etc/init.d/chef-server /etc/init.d/chef-server.original
$ sed -i '35s/DAEMON_OPTS="-p/DAEMON_OPTS="-c $WORKERTHREADS -p/g' /etc/init.d/chef-server
$ sed -i '42s/(ps/#(ps/g' /etc/init.d/chef-server
$ sed -i '/#(ps/ i (ps -fp $pid | egrep -q "merb.*( chef-server .*api.* spawner|worker .* $PORT)") || return 1' /etc/init.d/chef-server

Run diff to see the init script differences

$ diff /etc/init.d/chef-server /etc/init.d/chef-server.original
< (ps -fp $pid | egrep -q "merb.*( chef-server .*api.* spawner|worker .* $PORT)") || return 1
<   #(ps -fp $pid | egrep -q "merb.*(merb : master|worker.*$PORT)") || return 1 --- >   (ps -fp $pid | egrep -q "merb.*(merb : master|worker.*$PORT)") || return 1

Get variables from your Chef config for apache config construction

$ THREADSCT=$(grep "WORKERTHREADS" /etc/default/chef-server |awk -F"=" '{print $2}')
$ PORTPREFIX=$(grep "PORT" /etc/default/chef-server |awk -F"=" '{print $2}'| sed -e s/[0-9][0-9]$//g )
$ MYHOST=$(hostname -f)

Generate Your Apache Config – Generate the Load Balance Members

$ cd /etc/apache2/sites-available
$ echo "Listen 4000" |tee -a chef_loadbalancer
$ echo '<VirtualHost *:4000>' |tee -a chef_loadbalancer
$ echo "ServerName $MYHOST" |tee -a chef_loadbalancer
$ echo "" |tee -a chef_loadbalancer
$ echo "<Proxy balancer://cheflb>" | tee -a chef_loadbalancer
$ seq -w 00 $NEWCOUNT | while read i ; do echo "BalancerMember${PORTPREFIX}${i}" |tee -a chef_loadbalancer ; done

Append your Apache config with the rest of the relevant information

$ cat>>chef_loadbalancer<<EOF
ProxyPass / balancer://cheflb
ProxyPassReverse / balancer://cheflb
DocumentRoot /usr/share/chef-server/public
LogLevel info
ErrorLog /var/log/chef/chef_server_apache2-error.log
CustomLog /var/log/chef/chef_server_apache2-access.log combined
RewriteEngine On
RewriteRule ^/(.*)$ balancer://cheflb%{REQUEST_URI} [P,QSA,L]

Enable your new apache config and start Apache and Chef-Server

$ a2ensite chef_loadbalancer
$ /etc/init.d/apache2 restart
$ /etc/init.d/chef-server start

Test your chef server

$ time knife node list
$ time knife role list


Much of this information was stolen from :
If this technique is outdated please make me aware of it or if my apache configuration is awful ( which i’m sure it is, i’m just too lazy to improve it )

Chef Server – Backup and Restore

* This assumes you’re using the regular debian/ubuntu install
and that you have a system with a fresh install of chef-server laying around

Backup your files on the OLD/Original Chef Server
This backs up all the chef configs/certs and couchdb configs and data

$ tar czvf chef-backup-`date +%Y-%m-%d-%s`.tar.gz /etc/couchdb /var/lib/chef /var/lib/couchdb /var/cache/chef /var/log/chef /var/log/couchdb /etc/chef

Transfer your files to the NEW Chef Server

$ scp old-chefserver:~/chef-backup*.tar.gz /tmp

Stop all chef services on the NEW Chef Server

$ for i in chef-server-webui chef-server rabbitmq-server jetty couchdb chef-solr ; do /etc/init.d/${i} stop ; done

Extract the backup file on to the NEW Chef Server

$ cd /tmp
$ tar xzvf chef-backup*.tar.gz

Delete the old rabbitmq data from your default installation on the NEW Chef Server

$ rm -fr /var/lib/rabbitmq/mneisa/*

Start rabbitmq in the foreground and recreate your user and tables on the NEW Chef Server

$ rabbitmq-server
$ rabbitmqctl add_vhost /chef
Creating vhost "/chef" ...
$ rabbitmqctl add_user chef testing
Creating user "chef" ...
$ rabbitmqctl set_permissions -p /chef chef ".*" ".*" ".*"
Setting permissions for user "chef" in vhost "/chef" ...
$ rabbitmqctl stop
Stopping and halting node rabbit@ubuntu02 ...

Start all the Chef Services on NEW Chef Server

$ for i in chef-solr couchdb jetty rabbitmq-server chef-server chef-server-webui ; do /etc/init.d/${i} start ; done

Verify your Chef Server is restored
and verify the nodes and data or try the following knife commands

$ knife node list
$ knife node show mynode -a node

Alternate Backup Methods from Opscode:

Chef Server – Debian/Ubuntu Install

* Please make sure to change “mynewpassword” to whatever password you want to use.

$ apt-get -y install lsb-release wget debconf-utils
$ echo "deb `lsb_release -cs`-0.10 main" | tee /etc/apt/sources.list.d/opscode.list
$ gpg --keyserver --recv-keys 83EF826A
$ gpg --export | tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null
$ cat>answerfile<<EOF
chef-server-webui chef-server-webui/admin_password password mynewpassword
chef-solr chef-solr/amqp_password password mynewpassword
$ url=`hostname -f` ; echo "chef chef/chef_server_url string http://$url:4000" |tee -a answerfile
$ debconf-set-selections ./answerfile
$ echo "deb testing main" |tee -a /etc/apt/sources.list
$ wget
$ apt-key add rabbitmq-signing-key-public.asc
$ apt-get update
$ apt-get -y install chef-server opscode-keyring

Login at http://hostname:4040/
Username: admin
# This is the password you set in the answerfile above
Password: mynewpassword

Chef Server – Troubleshooting a fresh install on ubuntu 12.04

If your chef-server is not starting on 12.04
There is a relevant bug at :
This most likely happens if you didn’t run “apt-get update” before you installed chef-server

try running chef-server on it’s own and see what messages you get.
$ chef-server

Loading init file from /usr/share/chef-server-api/config/init.rb
[Mon, 25 Jun 2012 06:31:53 +0000] INFO: Loading roles
Loading /usr/share/chef-server-api/config/environments/development.rb
[Mon, 25 Jun 2012 06:31:53 +0000] INFO:
[Mon, 25 Jun 2012 06:31:53 +0000] INFO: ********************************************************************************
[Mon, 25 Jun 2012 06:31:53 +0000] INFO: *   Starting Chef Server in Development Mode.
[Mon, 25 Jun 2012 06:31:53 +0000] INFO: *   Start the server with `-e production` for normal use
[Mon, 25 Jun 2012 06:31:53 +0000] INFO: ********************************************************************************
[Mon, 25 Jun 2012 06:31:53 +0000] INFO:
NOTE: Gem.activate is deprecated, use Specification#activate. It will be removed on or after 2011-10-01.
Gem.activate called from /usr/lib/ruby/1.8/merb-core/core_ext/kernel.rb:142.
/usr/lib/ruby/vendor_ruby/1.8/rubygems.rb:232:in `_deprecated_activate': Deprecated use of Gem.activate(dep) (ArgumentError)
        from /usr/lib/ruby/vendor_ruby/1.8/rubygems/deprecate.rb:63:in `send'
        from /usr/lib/ruby/vendor_ruby/1.8/rubygems/deprecate.rb:63:in `activate'
        from /usr/lib/ruby/1.8/merb-core/core_ext/kernel.rb:142:in `load_dependency'
        from /usr/lib/ruby/1.8/merb-core/bootloader.rb:405:in `load_dependencies'
        from /usr/lib/ruby/1.8/merb-core/bootloader.rb:405:in `each'
        from /usr/lib/ruby/1.8/merb-core/bootloader.rb:405:in `load_dependencies'
        from /usr/lib/ruby/1.8/merb-core/bootloader.rb:393:in `run'
        from /usr/lib/ruby/1.8/merb-core/bootloader.rb:99:in `run'
        from /usr/lib/ruby/1.8/merb-core/server.rb:172:in `bootup'
        from /usr/lib/ruby/1.8/merb-core/server.rb:42:in `start'
        from /usr/lib/ruby/1.8/merb-core.rb:170:in `start'
        from /usr/sbin/chef-server:86

Fix the offending line:
$ sed -ie ‘142s/Gem.activate(dep)/dep.to_spec.activate/’ /usr/lib/ruby/1.8/merb-core/core_ext/kernel.rb
$ /etc/init.d/chef-server start
$ /etc/init.d/chef-server-webui start

Chef – Configure Knife Admin

Install Ruby, Ruby gems, and Chef-client


Install X-Code from Mac Appstore
Open Xcode and do the following :  Xcode >> Preferences >> Downloads >> Command Line Tools
$ cd /tmp
$ curl -O
$ tar zxf rubygems-1.8.24.tgz
$ cd rubygems-1.8.24
$ ruby setup.rb –no-format-executable
$ sudo gem install chef


$ yum -y install ruby ruby-devel ruby-ri ruby-rdoc ruby-shadow gcc gcc-c++ automake autoconf make curl dmidecode git
$ cd /tmp
$ curl -O
$ tar zxf rubygems-1.8.24.tgz
$ cd rubygems-1.8.24
$ ruby setup.rb –no-format-executable
$ sudo gem install chef


$ apt-get install -y curl git ruby1.9.1 rubygems1.9.1 chef

Copy the validation.pem and webui.pem

This is the key that allow you to register your host as a client to the chef server
$ scp root@chef-server:/etc/chef/validation.pem  /etc/chef/validation.pem

This is the key that allows you to authenticate yourself as an admin user to the chef server
Chef requires an existing admin to authorize a new admin
$ scp root@chef-server:/etc/chef/webui.pem  /etc/chef/webui.pem

Run Knife Configure

$ knife configure -i
Example Output:

Overwrite /root/.chef/knife.rb? (Y/N) y
Please enter the chef server URL: [http://chef-server:4000] http://chef-server:4000
Please enter a clientname for the new client: [root] jtran
Please enter the existing admin clientname: [chef-webui]
Please enter the location of the existing admin client's private key: [/etc/chef/webui.pem]
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef/validation.pem]
Please enter the path to a chef repository (or leave blank):
Creating initial API user...
Created client[jtran]
Configuration file written to /root/.chef/knife.rb

Verify Knife is working

$ knife client list
Example Output: