Chef – berkshelf lesson for dummies like me Ermahgerd!

I feel like some of the explanations on berkshelf on the internet are confusing.
So i felt like doing a small write up myself

berkshelf is pretty much a replacement for the “knife cookbook” command.
The big win with berkshelf is that it also resolves dependencies of a cookbook like apt or yum.
It reads a file called “Berksfile” for other cookbooks the current cookbook needs and what repositories to fetch them from and pulls them to your local system.

I will use the logstash cookbook at https://github.com/lusis/chef-logstash as an example
If you read the Berksfile at https://github.com/lusis/chef-logstash/blob/master/Berksfile
it will show you what other cookbooks the logstash cookbook needs

So in order to get going

gem install berkshelf
git clone git@github.com:lusis/chef-logstash.git
cd logstash
berks install
berks upload

That installed berkshelf, cloned the logstash cookbook, resolved dependencies for the logstash cookbook and uploaded logstash cookbook and its dependencies to your chef-server

Additionally berkshelf installs its configuration file at : ~/.berkshelf/config.json
You may need to edit some stuff there to match your ~/.chef/knife.rb file

Adding EBS Volumes with Opscode’s AWS cookbook

1. Download opscode’s aws cookbook and put it into your own cookbook repo

$ git clone https://github.com/opscode/cookbooks.git opscode-cookbooks
$ cp -r opscode-cookbooks/aws my-cookbooks/
$ cd my-cookbooks

2. Create a new cookbook that will utilize the aws cookbook

$ knife cookbook create aws-tests

3. Set the cookbook to have the dependency of the opscode aws cookbook

$ vi my-cookbooks/aws-tests/metadata.rb

maintainer       "YOUR_COMPANY_NAME"
maintainer_email "YOUR_EMAIL"
license          "All rights reserved"
description      "Installs/Configures aws_tests"
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version          "0.0.1"

depends "aws"

4. Create your recipe to create and attach a new EBS volume to your ec2 instance

$ vi vi my-cookbooks/aws-tests/recipes/default.rb

# Create and attach your new EBS volume
aws_ebs_volume "new_ebs_volume" do
  aws_access_key "MYAPIKEY"
  aws_secret_access_key "MYAPIKEYSECRET"
  size 1
  device "/dev/xvdi"
  action [ :create, :attach ]
end

5. Create a filesystem and mount your new volume

# Create your partition and filesystem for ext4
bash "create_filesystem" do
  user "root"
  code <<-EOH
    parted /dev/xvdi mklabel gpt
    parted /dev/xvdi mkpart logical ext4 1 -1
    parted /dev/xvdi set 1 lvm on
    yes | parted /dev/xvdi mkpart logical ext4 1 -- "-1"
    mkfs.ext4 /dev/xvdi1
  EOH
  not_if "parted /dev/xvdi1 |grep ext4"
end


directory "/mnt/test" do
  owner "root"
  group "root"
  mode "0755"
  recursive true
end

mount "/mnt/test" do
  device "/dev/xvdi1"
  options "rw noatime"
  fstype "ext4"
  action [ :enable, :mount ]
  not_if "cat /proc/mounts |grep /mnt/test"
end

6. Add aws and aws_tests recipes to your node

$ knife node edit i-fff4f8c

{
  "chef_environment": "_default",
  "name": "i-fff4f8",
  "run_list": [
    "recipe[aws]",
    "recipe[aws_tests]"
  ],
  "normal": {
    "tags": [

    ],
  }
}

7. Run chef-client on your node

$ chef-client

Debugging knife configure -i

If you ever see something like this


root@chefserver01:/etc/chef# knife configure -i -V
Overwrite /root/.chef/knife.rb? (Y/N) y
Please enter the chef server URL: [http://chefserver01:4000]
Please enter a clientname for the new client: [root] jtran7
Please enter the existing admin clientname: [chef-webui]
Please enter the location of the existing admin client's private key: [/etc/chef/webui.pem]
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef/validation.pem]
Please enter the path to a chef repository (or leave blank):
Creating initial API user...
INFO: HTTP Request Returned 500 Internal Server Error: Connection reset by peer
ERROR: Server returned error for http://chefserver01:4000/clients, retrying 1/5 in 4s
INFO: HTTP Request Returned 409 Conflict: Client already exists
INFO: HTTP Request Returned 500 Internal Server Error: Connection reset by peer
ERROR: Server returned error for http://chefserver01:4000/clients/jtran7, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Connection reset by peer
ERROR: Server returned error for http://chefserver01:4000/clients/jtran7, retrying 2/5 in 5s

it’s likely your ampq password is incorrect in /etc/chef/server.rb

# amqp_pass sets the password for the AMQP virtual host in rabbitmq-server.
amqp_pass "testing"

If you had deleted your rabbitmq mnesia table you had to do something like this.
The word “testing” is your ampq password

sudo rabbitmqctl add_vhost /chef
sudo rabbitmqctl add_user chef testing
sudo rabbitmqctl set_permissions -p /chef chef ".*" ".*" ".*"

Clustered AMQP Rabbit-MQ

RabbitMQ already has excellent documentation at: http://www.rabbitmq.com/clustering.html

But this will be a bit more chef specific
server01 = existing chef-server
server02 = new rabbitmq server to be added to cluster

On Server01

On your existing chef-server
*I assume you’re using the latest rabbitmq and that server02 will also install matching version
Check what the cluster output looks like

root@server01:~# rabbitmqctl cluster_status
Cluster status of node rabbit@server01 ..
[{nodes,[{disc,[rabbit@server01]}]},{running_nodes,[rabbit@server01]}]
...done.

Copy your rabbitmq cookie to server02

root@server01:~# scp /var/lib/rabbitmq/.erlang.cookie root@server02:/var/lib/rabbitmq/
Are you sure you want to continue connecting (yes/no)? yes
root@server02's password: 
.erlang.cookie                                                                                        100%   20     0.0KB/s   00:00

On Server02

Install rabbitmq

root@server02:~# echo "deb http://www.rabbitmq.com/debian/ testing main" |tee -a /etc/apt/sources.list
root@server02:~# wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
root@server02:~# apt-key add rabbitmq-signing-key-public.asc
root@server02:~# apt-get update
root@server02:/var/chef/cache# apt-get -y install rabbitmq-server

Delete existing mnesia database and start rabbitmq

root@server02:~# service rabbitmq-server stop
root@server02:~# rm -fr /var/lib/rabbitmq/mnesia 
root@server02:~# service rabbitmq-server start
 * Starting message broker rabbitmq-server
   ...done.

Join Server02 to Server01

root@server02:~# rabbitmqctl cluster_status
Cluster status of node rabbit@server02 ...
[{nodes,[{disc,[rabbit@server02]}]},{running_nodes,[rabbit@server02]}]
...done.
root@server02:~# rabbitmqctl stop_app
Stopping node rabbit@server02 ...
...done.
root@server02~# rabbitmqctl reset   
Resetting node rabbit@server02 ...
...done.
root@server02:~# rabbitmqctl cluster rabbit@server01 rabbit@server02
Clustering node rabbit@server02 with [rabbit@server01,rabbit@server02] ...
...done.
root@server02:~# rabbitmqctl start
root@server02:~# rabbitmqctl cluster_status
Cluster status of node rabbit@server01 ...
[{nodes,[{disc,[rabbit@server02,rabbit@server01]}]},
 {running_nodes,[rabbit@server02,rabbit@server01]}]
...done.

Verify on server01 that cluster shows up as expected

root@server01:~# rabbitmqctl cluster_status
Cluster status of node rabbit@server01 ...
[{nodes,[{disc,[rabbit@server02,rabbit@server01]}]},
 {running_nodes,[rabbit@server02,rabbit@server01]}]
...done.

Make sure chef still works

root@server01:~# knife node list

root@server01:~# knife client list
  chef-validator
  chef-webui
  root
  server01
root@server01:~# knife bootstrap -x root server02 --template-file ~/ubuntu12.04.rb 
Bootstrapping Chef on server02
Failed to authenticate root - trying password auth
Enter your password: 
server02 Updating installed gems

root@server01:~# knife node list
server02

Essential Knife Plugins for chef

Knife-Spork:
This plugin helps you control versioning of your cookbooks and also prevents you from accidently commiting things you don’t want to.

$ gem install knife-spork

https://github.com/jonlives/knife-spork

Knife-ec2:
Obviously this is a connector to ec2. I use Amazon ec2 so i am biased

$ gem install knife-ec2

https://github.com/opscode/knife-ec2/

Knife-preflight:
A preflight plugin for Chef::Knife which lets you see which nodes and roles use a particular cookbook before you upload it.

$ gem install knife-preflight

https://github.com/jonlives/knife-preflight

Knife-env-diff:
A plugin for Chef::Knife which will diff the cookbook versions of two or more environments.

$ gem install knife-env-diff

https://github.com/jgoulah/knife-env-diff

Other mentionables

Knife-slapchop:
Threaded bootstrapping of aws nodes
https://github.com/kryptek/knife-slapchop

Knife-ironfan
Cluster Deployments
https://github.com/infochimps-labs/ironfan

Search for more knife plugins:

$ gem search -r knife-

Multiple AWS Accounts with Knife Admin

I Recently stumbled across a predicament of multiple aws accounts.
This is a minor predicament but a predicament nonethless.
I have a situation where i have

1. A personal AWS account

2. A work AWS account

3. A vendor AWS account

These three AWS accounts all use the same chef-server. So to make my life easier i decided to organize them.
I created the following structure:

$ mkdir -p ~/chef-aws/{personal,work,thirdparty}/.chef

I copied my knife.rb from ~/.chef/knife.rb into each of these folders.

$ cp -p ~/.chef/knife.rb ~/chef-aws/personal
$ cp -p ~/.chef/knife.rb ~/chef-aws/work
$ cp -p ~/.chef/knife.rb ~/chef-aws/thirdparty

Here’s an example of the knife.rb file
You can find details on setting up knife with ec2 here : Knife-EC2 Configuration


current_dir = File.dirname(__FILE__)
log_level                :info
log_location             STDOUT
node_name                "neosirex"
client_key               "/home/James/.chef/myuser.pem"
validation_client_name   "neosirex-validator"
validation_key           "/home/James/.chef/random-validator.pem"
chef_server_url          "https://api.opscode.com/organizations/somemakebelieveaccount"
cache_type               'BasicFile'
cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
cookbook_path            ["#{current_dir}/../cookbooks"]

Here’s the snippet that’s added to each AWS specific knife.rb

knife[:aws_access_key_id] ='< AWS ACCESS KEY GOES HERE >'
knife[:aws_secret_access_key] ='< AWS SECRET KEY GOES HERE >'

So now in order to use different AWS accounts what i do is change into each of those aws directories and run knife commands from there.
Each of the following commands would give me the output only of the relevant AWS server

$ cd ~/chef-aws/personal && knife ec2 server list
$ cd ~/chef-aws/work && knife ec2 server list
$ cd ~/chef-aws/thirdparty && knife ec2 server list

I Leave my default ~/.chef/knife.rb file without AWS credentials in it.
This is because i don’t want to accidently deploy to the wrong AWS account.
There’s still room for human error but i suppose it’s better than nothing
If someone has a better approach to this i’d like to know about it.

Knife EC2 Extension – Install and Use

Install

$ apt-get install -y libxslt-dev libxml2-dev
$ gem install knife-ec2
$ gem install net-ssh-multi

Get Your AWS Keys

Scroll Down to The Certificates and Secret Keys Menu and generate your new access keys as needed

Generate the Keypair associated with your new ec2 instances ( for ssh )

Create a new keypair. This should result in a pem file output to you. If you lose this file you will not be able to access any ec2 instances associated with it unless you have alternate accounts you can login with.

Configure

$ cd ~/.chef
$ vi knife.rb

Append the following to your knife.rb

### AWS Configuration ###

## The below lines allow you to use the ec2 api
knife[:aws_access_key_id] ='< AWS ACCESS KEY GOES HERE >'
knife[:aws_secret_access_key] ='< AWS SECRET KEY GOES HERE >'

## The below allow you to ssh into new ec2 instance that are associated with the keypair below
## You can alternately choose to specify the username and key location on the knife command line
# knife[:aws_ssh_key_id] ='james-aws'
# knife[:identity_file] ="/home/james/.ssh/james-aws.pem"

Test knife-ec2 command

bootstrap file squeeze.rb can be grabbed from here:
https://github.com/cookingclouds/bootstrap/

$ knife ec2 server list
$ knife ec2 server create -I ami-e00df089 -f t1.micro -Z us-east-1a -G "default_security" -k james-aws --ssh-key /home/james/.ssh/james-aws.pem --template-file /home/james/bootstrap/squeeze.rb

knife-ec2 command reference list

knife ec2 server create --help
knife ec2 server create (options)
    -Z, --availability-zone ZONE     The Availability Zone
    -A, --aws-access-key-id KEY      Your AWS Access Key ID
    -K SECRET,                       Your AWS API Secret Access Key
        --aws-secret-access-key
        --user-data USER_DATA_FILE   The EC2 User Data file to provision the instance with
        --bootstrap-version VERSION  The version of Chef to install
    -N, --node-name NAME             The Chef node name for your new node
        --server-url URL             Chef Server URL
    -k, --key KEY                    API Client Key
        --color                      Use colored output
    -c, --config CONFIG              The configuration file to use
        --defaults                   Accept default values for all questions
    -d, --distro DISTRO              Bootstrap a distro using a template
        --ebs-no-delete-on-term      Do not delete EBS volumn on instance termination
        --ebs-size SIZE              The size of the EBS volume in GB, for EBS-backed instances
    -e, --editor EDITOR              Set the editor to use for interactive commands
    -E, --environment ENVIRONMENT    Set the Chef environment
    -f, --flavor FLAVOR              The flavor of server (m1.small, m1.medium, etc)
    -F, --format FORMAT              Which format to use for output
    -i IDENTITY_FILE,                The SSH identity file used for authentication
        --identity-file
    -I, --image IMAGE                The AMI for the server
        --no-color                   Don't use colors in the output
    -n, --no-editor                  Do not open EDITOR, just accept the data as is
        --no-host-key-verify         Disable host key verification
    -u, --user USER                  API Client Username
        --prerelease                 Install the pre-release chef gems
        --print-after                Show the data after a destructive operation
        --region REGION              Your AWS region
    -r, --run-list RUN_LIST          Comma separated list of roles/recipes to apply
    -G, --groups X,Y,Z               The security groups for this server
    -S, --ssh-key KEY                The AWS SSH key id
    -P, --ssh-password PASSWORD      The ssh password
    -x, --ssh-user USERNAME          The ssh username
    -s, --subnet SUBNET-ID           create node in this Virtual Private Cloud Subnet ID (implies VPC mode)
        --template-file TEMPLATE     Full path to location of template to use
    -V, --verbose                    More verbose output. Use twice for max verbosity
    -v, --version                    Show chef version
    -y, --yes                        Say yes to all prompts for confirmation
    -h, --help                       Show this message

Chef Server – Backup and Restore

* This assumes you’re using the regular debian/ubuntu install
and that you have a system with a fresh install of chef-server laying around

Backup your files on the OLD/Original Chef Server
This backs up all the chef configs/certs and couchdb configs and data

$ tar czvf chef-backup-`date +%Y-%m-%d-%s`.tar.gz /etc/couchdb /var/lib/chef /var/lib/couchdb /var/cache/chef /var/log/chef /var/log/couchdb /etc/chef

Transfer your files to the NEW Chef Server

$ scp old-chefserver:~/chef-backup*.tar.gz /tmp

Stop all chef services on the NEW Chef Server

$ for i in chef-server-webui chef-server rabbitmq-server jetty couchdb chef-solr ; do /etc/init.d/${i} stop ; done

Extract the backup file on to the NEW Chef Server

$ cd /tmp
$ tar xzvf chef-backup*.tar.gz

Delete the old rabbitmq data from your default installation on the NEW Chef Server

$ rm -fr /var/lib/rabbitmq/mneisa/*

Start rabbitmq in the foreground and recreate your user and tables on the NEW Chef Server

$ rabbitmq-server
$ rabbitmqctl add_vhost /chef
Creating vhost "/chef" ...
...done.
$ rabbitmqctl add_user chef testing
Creating user "chef" ...
...done.
$ rabbitmqctl set_permissions -p /chef chef ".*" ".*" ".*"
Setting permissions for user "chef" in vhost "/chef" ...
...done.
$ rabbitmqctl stop
Stopping and halting node rabbit@ubuntu02 ...
...done.

Start all the Chef Services on NEW Chef Server

$ for i in chef-solr couchdb jetty rabbitmq-server chef-server chef-server-webui ; do /etc/init.d/${i} start ; done

Verify your Chef Server is restored
http://NEW-CHEF-SERVER:4040
and verify the nodes and data or try the following knife commands

$ knife node list
$ knife node show mynode -a node

Alternate Backup Methods from Opscode:

http://wiki.opscode.com/display/chef/Backing+Up+Chef+Server

Chef – Configure Knife Admin

Install Ruby, Ruby gems, and Chef-client

Mac OSX

Install X-Code from Mac Appstore
Open Xcode and do the following : Xcode >> Preferences >> Downloads >> Command Line Tools
$ cd /tmp
$ curl -O http://production.cf.rubygems.org/rubygems/rubygems-1.8.24.tgz
$ tar zxf rubygems-1.8.24.tgz
$ cd rubygems-1.8.24
$ ruby setup.rb –no-format-executable
$ sudo gem install chef

CentOS6

$ yum -y install ruby ruby-devel ruby-ri ruby-rdoc ruby-shadow gcc gcc-c++ automake autoconf make curl dmidecode git
$ cd /tmp
$ curl -O http://production.cf.rubygems.org/rubygems/rubygems-1.8.24.tgz
$ tar zxf rubygems-1.8.24.tgz
$ cd rubygems-1.8.24
$ ruby setup.rb –no-format-executable
$ sudo gem install chef

Ubuntu

$ apt-get install -y curl git ruby1.9.1 rubygems1.9.1 chef

Copy the validation.pem and webui.pem

This is the key that allow you to register your host as a client to the chef server
$ scp root@chef-server:/etc/chef/validation.pem /etc/chef/validation.pem

This is the key that allows you to authenticate yourself as an admin user to the chef server
Chef requires an existing admin to authorize a new admin
$ scp root@chef-server:/etc/chef/webui.pem /etc/chef/webui.pem

Run Knife Configure

$ knife configure -i
Example Output:

Overwrite /root/.chef/knife.rb? (Y/N) y
Please enter the chef server URL: [http://chef-server:4000] http://chef-server:4000
Please enter a clientname for the new client: [root] jtran
Please enter the existing admin clientname: [chef-webui]
Please enter the location of the existing admin client's private key: [/etc/chef/webui.pem]
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef/validation.pem]
Please enter the path to a chef repository (or leave blank):
Creating initial API user...
Created client[jtran]
Configuration file written to /root/.chef/knife.rb

Verify Knife is working

$ knife client list
Example Output:

  jtran
  chef-webui
  chef-validator

Chef – Attribute overriding

Let’s pretend we have a cookbook named “wonka”

in ~/chef-repo/cookbooks/wonka/attributes/default.rb

default[:wonka][:gobstopper]  = "everlasting"
default[:wonka][:kazoozle]    = "alittle"

We want to override the value of [:wonka][:gobstopper] for a node named “oompaloompa01”

Without an override defined your json should look something like the following
You can issue:
$ knife node edit ooompaloopma01

{
  "run_list": [
    "recipe[chocolate_factory::default]"
  ],
  "chef_environment": "development",
  "name": "oompaloompa01",
  "normal": {
    "chef_environment": "development",
    "name": "oompaloopma01",
    "normal": {
      "tags": [

      ]
    },
    "tags": [

    ]
  }
}

Now let’s say you want to define default[:wonka][:gobstopper] = “allgone” with an override
Your json should look like the following

{
  "run_list": [
    "recipe[chocolate_factory::default]"
  ],
  "chef_environment": "development",
  "name": "oompaloompa01",
  "normal": {
    "chef_environment": "development",
    "name": "oompaloopma01",
    "wonka": {
      "gobstopper": [
        "allgone"
      ]
    },
    "normal": {
      "tags": [

      ]
    },
    "tags": [

    ]
  }
}

that should commit the override to that specific node.
you can check the values of a node’s attributes with the following command ( where the attribute group you want to look at is “wonka” )
$ knife node show oompaloompa01 -a wonka

Now just run chef-client on oompaloopa01 and you should see the override take effect.