One of the things not obvious how to do in Amazon EC2 is whitelisting traffic to your load balancers.
Here’s how to do it
In your security groups add the inbound traffic to security group “amazon-elb/amazon-elb-sg” for all TCP
Monthly Archives: July 2012
Essential Knife Plugins for chef
Knife-Spork:
This plugin helps you control versioning of your cookbooks and also prevents you from accidently commiting things you don’t want to.
$ gem install knife-spork
https://github.com/jonlives/knife-spork
Knife-ec2:
Obviously this is a connector to ec2. I use Amazon ec2 so i am biased
$ gem install knife-ec2
https://github.com/opscode/knife-ec2/
Knife-preflight:
A preflight plugin for Chef::Knife which lets you see which nodes and roles use a particular cookbook before you upload it.
$ gem install knife-preflight
https://github.com/jonlives/knife-preflight
Knife-env-diff:
A plugin for Chef::Knife which will diff the cookbook versions of two or more environments.
$ gem install knife-env-diff
https://github.com/jgoulah/knife-env-diff
Other mentionables
Knife-slapchop:
Threaded bootstrapping of aws nodes
https://github.com/kryptek/knife-slapchop
Knife-ironfan
Cluster Deployments
https://github.com/infochimps-labs/ironfan
Search for more knife plugins:
$ gem search -r knife-
Multiple AWS Accounts with Knife Admin
I Recently stumbled across a predicament of multiple aws accounts.
This is a minor predicament but a predicament nonethless.
I have a situation where i have
1. A personal AWS account
2. A work AWS account
3. A vendor AWS account
These three AWS accounts all use the same chef-server. So to make my life easier i decided to organize them.
I created the following structure:
$ mkdir -p ~/chef-aws/{personal,work,thirdparty}/.chef
I copied my knife.rb from ~/.chef/knife.rb into each of these folders.
$ cp -p ~/.chef/knife.rb ~/chef-aws/personal $ cp -p ~/.chef/knife.rb ~/chef-aws/work $ cp -p ~/.chef/knife.rb ~/chef-aws/thirdparty
Here’s an example of the knife.rb file
You can find details on setting up knife with ec2 here : Knife-EC2 Configuration
current_dir = File.dirname(__FILE__)
log_level :info
log_location STDOUT
node_name "neosirex"
client_key "/home/James/.chef/myuser.pem"
validation_client_name "neosirex-validator"
validation_key "/home/James/.chef/random-validator.pem"
chef_server_url "https://api.opscode.com/organizations/somemakebelieveaccount"
cache_type 'BasicFile'
cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
cookbook_path ["#{current_dir}/../cookbooks"]
Here’s the snippet that’s added to each AWS specific knife.rb
knife[:aws_access_key_id] ='< AWS ACCESS KEY GOES HERE >' knife[:aws_secret_access_key] ='< AWS SECRET KEY GOES HERE >'
So now in order to use different AWS accounts what i do is change into each of those aws directories and run knife commands from there.
Each of the following commands would give me the output only of the relevant AWS server
$ cd ~/chef-aws/personal && knife ec2 server list $ cd ~/chef-aws/work && knife ec2 server list $ cd ~/chef-aws/thirdparty && knife ec2 server list
I Leave my default ~/.chef/knife.rb file without AWS credentials in it.
This is because i don’t want to accidently deploy to the wrong AWS account.
There’s still room for human error but i suppose it’s better than nothing
If someone has a better approach to this i’d like to know about it.
Hash Keys and Values in General Ruby
Create a new Hash
node = Hash.new
Create a new Hash of Hash
node[:one] = Hash.new
Insert Values into your Hash of Hash
node[:one][:object] = "number1" node[:one][:block] = "number2"
Print Key and Value
node[:one].each_pair do |k,v| puts k puts v end
Print Keys only
node[:one].keys.each do |key| puts key end
Print Values Only
node[:one].values.each do |value| puts value end
Keys and Values from Hashes in Ruby Templates
Reference: http://ruby-doc.org/stdlib-1.9.3/libdoc/erb/rdoc/ERB.html
ERB recognizes certain tags in the provided template and converts them based on the rules below:
<% Ruby code -- inline with output %> <%= Ruby expression -- replace with result %> <%# comment -- ignored -- useful in testing %> % a line of Ruby code -- treated as <% line %> (optional -- see ERB.new) %% replaced with % if first thing on a line and % processing is used <%% or %%> -- replace with <% or %> respectively
Create a new “motd” cookbook
$ knife cookbook create motd
Example: ~/cookbooks/motd/recipes/default.rb
# # Cookbook Name:: motd # Recipe:: default # # Copyright 2012, YOUR_COMPANY_NAME # # All rights reserved - Do Not Redistribute # template "/home/motd" do source "motd.erb" owner "root" end
Example: ~/cookbooks/motd/template/default/motd.erb
* You can find out what hashes are defined on a system by running “ohai”
Hostname : <%= node["hostname"] %> Platform : <%= node["platform"] %> Memory Usage <% node["memory"].each_pair do |k,v| %> <%= k %> : <%= v%> <% end %> Block Devices <% node["block_device"].each_pair do |k,v| %> Key: <%= k %> - <%= v%> <% end %> Network Info <% node["network"]["interfaces"].keys.each do |k| %> Key: <%= k %> <% end %> <% node["network"]["interfaces"].values.each do |v| %> Value: <%= v %> <% end %>
Run chef-client with the “motd” cookbook installed and look at the output at
$ cat /home/motd
Hostname: ubuntu01 Platform: ubuntu Memory Usage vmalloc_total : 34359738367kB anon_pages : 129892kB writeback : 0kB dirty : 0kB vmalloc_used : 266104kB vmalloc_chunk : 34359469948kB active : 186264kB buffers : 31252kB commit_limit : 773540kB nfs_unstable : 0kB slab_unreclaim : 11684kB bounce : 0kB slab_reclaimable : 17644kB mapped : 11580kB cached : 190268kB slab : 29328kB inactive : 165136kB free : 105708kB total : 502612kB committed_as : 926460kB page_tables : 4324kB swap : cached0kBfree522236kBtotal522236kB Block Devices Key: sda - timeout30modelVMware Virtual Sremovable0vendorVMware,rev1.0size41943040staterunning Key: sr0 - timeout30modelVMware IDE CDR10removable1vendorNECVMWarrev1.00size1401432staterunning Key: fd0 - removable1size0 Key: loop7 - removable0size0 Key: loop6 - removable0size0 Key: loop5 - removable0size0 Key: loop4 - removable0size0 Key: loop3 - removable0size0 Key: loop2 - removable0size0 Key: loop1 - removable0size0 Key: loop0 - removable0size0 Key: ram15 - removable0size131072 Key: ram14 - removable0size131072 Key: ram13 - removable0size131072 Key: ram12 - removable0size131072 Key: ram11 - removable0size131072 Key: ram10 - removable0size131072 Key: ram9 - removable0size131072 Key: ram8 - removable0size131072 Key: ram7 - removable0size131072 Key: ram6 - removable0size131072 Key: ram5 - removable0size131072 Key: ram4 - removable0size131072 Key: ram3 - removable0size131072 Key: ram2 - removable0size131072 Key: ram1 - removable0size131072 Key: ram0 - removable0size131072 Network Info Key: eth0 Key: lo Value: mtu16436encapsulationLoopbackflagsLOOPBACKUPLOWER_U....< i truncated output > Value: mtu1500typeethencapsulationEthernetflagsBROADCASTM....< i truncated output >
Also for shits and giggles
Example Ohai output:
root@ubuntu01:~# ohai
{
"idletime": "35 minutes 35 seconds",
"uptime": "36 minutes 47 seconds",
"dmi": {
"base_board": {
"chassis_handle": "0x0000",
"location_in_chassis": "Not Specified",
"product_name": "440BX Desktop Reference Platform",
"serial_number": "None",
"manufacturer": "Intel Corporation",
"version": "None",
"type": "Unknown",
"features": "None",
"contained_object_handles": "0",
"all_records": [
.. < i truncated output >
Ubuntu/Debian Preseeding with LVM
When you have your ubuntu or debian splash screen hit “tab”
then append:
auto url=http://mywebserver/mypreseed.cfg
You can use this example to bring up a quick webserver to server your preseed file: Python Simple HTTP Server
You’ll still probably be prompted for all the locale information because that is requested BEFORE debian/ubuntu loads the preseed configuration. ( yes , dumb i know ). These are the settings that don’t get pulled in.
# keyboard and locale settings #################################################################### d-i debian-installer/locale string en_US d-i console-keymaps-at/keymap select sg-latin1 # networking #################################################################### d-i netcfg/disable_dhcp boolean false d-i netcfg/get_hostname string localhost d-i netcfg/get_domain string localdomain d-i netcfg/choose_interface select eth0
You can get around this by rebuilding your boot cdrom or doing a pxe install that loads all the info ahead of time.
Example
auto url=http://mywebserver/mypreseed.cfg \ locale=en_US console-keymaps-at/keymap=sg-latin1 \ interface=eth0 hostname=localhost domain=localdomain --
Now for actually making your preseed config
Setup Locale info
d-i debian-installer/locale string en_US d-i console-setup/ask_detect boolean false d-i time/zone string UTC d-i debian-installer/splash boolean false d-i debian-installer/language string en d-i debain-installer/country string US d-i console-keymaps-at/keymap select sg-latin1
Setup Your Networking
d-i netcfg/get_nameservers string 192.168.1.5 d-i netcfg/get_ipaddress string 192.168.1.10 d-i netcfg/get_netmask string 255.255.255.0 d-i netcfg/get_gateway string 192.168.1.1 d-i netcfg/confirm_static boolean true d-i netcfg/get_hostname string tempnode d-i netcfg/get_domain string localdomain
Setup your Repository and what directory in the repo to look for your install files
d-i mirror/country string manual d-i mirror/http/hostname string http.us.debian.org d-i mirror/http/directory string /debian d-i mirror/http/proxy string
Partition Your disks and set them up as lvm
d-i partman-auto/disk string /dev/sda d-i partman-auto/method string lvm d-i partman-lvm/device_remove_lvm boolean true d-i partman-lvm/device_remove_lvm_span boolean true d-i partman-auto/purge_lvm_from_device boolean true d-i partman-auto-lvm/new_vg_name string system
Setup your logical Volumes in LVM and also non-lvm partitions
This will make
* BEWARE OF YOUR FORMATTING OF THIS PART – IT IS SUPER SENSITIVE
/boot : is a bootable filesystem with 300mb and ext3
/ : Is between 10GB and grows to the end of the disk and ext3
swap : uses 200% of the size of system ram for a disk or 8GB
#d-i partman-auto/init_automatically_partition \
# select Guided - use entire disk and set up LVM
d-i partman-auto/expert_recipe string \
boot-root :: \
40 300 300 ext3 \
$primary{ } \
$bootable{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext3 } \
mountpoint{ /boot } \
. \
2000 10000 1000000000 ext3 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext3 } \
mountpoint{ / } \
. \
8000 8000 200% linux-swap \
$lvmok{ } \
method{ swap } format{ } \
.
Here’s another LVM Example
d-i partman-auto/expert_recipe string \
boot-root :: \
40 300 300 ext4 \
$primary{ } \
$bootable{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /boot } \
. \
2000 10000 1000000000 ext4 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ / } \
. \
2000 1000 10000 ext4 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /var } \
. \
2000 1000 60000 ext4 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /var/lib/mysql } \
. \
2000 1000 30000 ext4 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /www } \
. \
8000 8000 200% linux-swap \
$lvmok{ } \
method{ swap } format{ } \
.
Answer yes to all the partition and LVM prompts
d-i partman-lvm/confirm boolean true d-i partman/confirm_write_new_label boolean true d-i partman/choose_partition select Finish partitioning and write changes to disk d-i partman/confirm boolean true
Setup your Time Zone info
d-i clock-setup/utc boolean true d-i clock-setup/ntp boolean true d-i clock-setup/ntp-server string ntp.ubuntu.com d-i time/zone string US/Los_Angeles
Setup Root and First User info
d-i base-installer/kernel/image string linux-server d-i passwd/root-login boolean true d-i passwd/root-password-crypted password $1$VuPOoDRD$seX.C54E8TUdMkaAmKvjx0 d-i passwd/user-fullname string Ubuntu User d-i passwd/username string ubuntu d-i passwd/user-password-crypted password $1$NvFY8IuR$BGqOozSN91ljvQB.pVLDw. d-i user-setup/encrypt-home boolean false d-i user-setup/allow-password-weak boolean true d-i passwd/user-default-groups string adm cdrom dialout lpadmin plugdev sambashare
Miscellaneous Stuffage
d-i apt-setup/services-select multiselect security d-i debian-installer/allow_unauthenticated string true d-i pkgsel/upgrade select safe-upgrade d-i pkgsel/language-packs multiselect d-i pkgsel/update-policy select none d-i pkgsel/updatedb boolean true popularity-contest popularity-contest/participate boolean false tasksel tasksel/first multiselect standard, openssh-server d-i grub-installer/skip boolean false d-i lilo-installer/skip boolean false d-i grub-installer/only_debian boolean true d-i grub-installer/with_other_os boolean false d-i finish-install/keep-consoles boolean false d-i finish-install/reboot_in_progress note d-i cdrom-detect/eject boolean true d-i debian-installer/exit/halt boolean false d-i debian-installer/exit/poweroff boolean false
Choose your packages to install
d-i pkgsel/include string vim openssh-server
What it looks like as one file:
d-i debian-installer/locale string en_US
d-i console-setup/ask_detect boolean false
d-i time/zone string UTC
d-i debian-installer/splash boolean false
d-i debian-installer/language string en
d-i debain-installer/country string US
d-i netcfg/get_nameservers string 192.168.1.5
d-i netcfg/get_ipaddress string 192.168.1.10
d-i netcfg/get_netmask string 255.255.255.0
d-i netcfg/get_gateway string 192.168.1.1
d-i netcfg/confirm_static boolean true
d-i netcfg/get_hostname string tempnode
d-i mirror/country string manual
d-i mirror/http/hostname string http.us.debian.org
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string
d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-lvm/device_remove_lvm_span boolean true
d-i partman-auto/purge_lvm_from_device boolean true
d-i partman-auto-lvm/new_vg_name string system
#d-i partman-auto/init_automatically_partition \
# select Guided - use entire disk and set up LVM
d-i partman-auto/expert_recipe string \
boot-root :: \
40 300 300 ext3 \
$primary{ } \
$bootable{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext3 } \
mountpoint{ /boot } \
. \
2000 10000 1000000000 ext3 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext3 } \
mountpoint{ / } \
. \
8000 8000 200% linux-swap \
$lvmok{ } \
method{ swap } format{ } \
.
d-i partman-lvm/confirm boolean true
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition select Finish partitioning and write changes to disk
d-i partman/confirm boolean true
d-i clock-setup/utc boolean true
d-i clock-setup/ntp boolean true
d-i clock-setup/ntp-server string ntp.ubuntu.com
d-i time/zone string US/Los_Angeles
d-i base-installer/kernel/image string linux-server
d-i base-installer/kernel/image string linux-server
d-i passwd/root-login boolean true
d-i passwd/root-password-crypted password $1$VuPOoDRD$seX.C54E8TUdMkaAmKvjx0
d-i passwd/user-fullname string Ubuntu User
d-i user-setup/allow-password-weak boolean true
d-i user-setup/encrypt-home boolean false
d-i passwd/user-default-groups string adm cdrom dialout lpadmin plugdev sambashare
d-i apt-setup/services-select multiselect security
d-i debian-installer/allow_unauthenticated string true
d-i pkgsel/upgrade select safe-upgrade
d-i pkgsel/language-packs multiselect
d-i pkgsel/update-policy select none
d-i pkgsel/updatedb boolean true
popularity-contest popularity-contest/participate boolean false
tasksel tasksel/first multiselect standard, openssh-server
d-i grub-installer/skip boolean false
d-i lilo-installer/skip boolean false
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean false
d-i finish-install/keep-consoles boolean false
d-i finish-install/reboot_in_progress note
d-i cdrom-detect/eject boolean true
d-i debian-installer/exit/halt boolean false
d-i debian-installer/exit/poweroff boolean false
d-i pkgsel/include string vim openssh-server
Info for this was stolen here ( Thanks ):
http://cptyesterday.wordpress.com/2012/06/17/notes-on-using-expert_recipe-in-debianubuntu-preseed-files/
Preseed Example files found here:
Knife EC2 Extension – Install and Use
Install
$ apt-get install -y libxslt-dev libxml2-dev $ gem install knife-ec2 $ gem install net-ssh-multi
Get Your AWS Keys
Login to your AWS account at aws.amazon.com
Go to My Account/Console -> Security Credentials
Scroll Down to The Certificates and Secret Keys Menu and generate your new access keys as needed
Generate the Keypair associated with your new ec2 instances ( for ssh )
Create a new keypair. This should result in a pem file output to you. If you lose this file you will not be able to access any ec2 instances associated with it unless you have alternate accounts you can login with.
Configure
$ cd ~/.chef $ vi knife.rb
Append the following to your knife.rb
### AWS Configuration ### ## The below lines allow you to use the ec2 api knife[:aws_access_key_id] ='< AWS ACCESS KEY GOES HERE >' knife[:aws_secret_access_key] ='< AWS SECRET KEY GOES HERE >' ## The below allow you to ssh into new ec2 instance that are associated with the keypair below ## You can alternately choose to specify the username and key location on the knife command line # knife[:aws_ssh_key_id] ='james-aws' # knife[:identity_file] ="/home/james/.ssh/james-aws.pem"
Test knife-ec2 command
bootstrap file squeeze.rb can be grabbed from here:
https://github.com/cookingclouds/bootstrap/
$ knife ec2 server list $ knife ec2 server create -I ami-e00df089 -f t1.micro -Z us-east-1a -G "default_security" -k james-aws --ssh-key /home/james/.ssh/james-aws.pem --template-file /home/james/bootstrap/squeeze.rb
knife-ec2 command reference list
knife ec2 server create --help
knife ec2 server create (options)
-Z, --availability-zone ZONE The Availability Zone
-A, --aws-access-key-id KEY Your AWS Access Key ID
-K SECRET, Your AWS API Secret Access Key
--aws-secret-access-key
--user-data USER_DATA_FILE The EC2 User Data file to provision the instance with
--bootstrap-version VERSION The version of Chef to install
-N, --node-name NAME The Chef node name for your new node
--server-url URL Chef Server URL
-k, --key KEY API Client Key
--color Use colored output
-c, --config CONFIG The configuration file to use
--defaults Accept default values for all questions
-d, --distro DISTRO Bootstrap a distro using a template
--ebs-no-delete-on-term Do not delete EBS volumn on instance termination
--ebs-size SIZE The size of the EBS volume in GB, for EBS-backed instances
-e, --editor EDITOR Set the editor to use for interactive commands
-E, --environment ENVIRONMENT Set the Chef environment
-f, --flavor FLAVOR The flavor of server (m1.small, m1.medium, etc)
-F, --format FORMAT Which format to use for output
-i IDENTITY_FILE, The SSH identity file used for authentication
--identity-file
-I, --image IMAGE The AMI for the server
--no-color Don't use colors in the output
-n, --no-editor Do not open EDITOR, just accept the data as is
--no-host-key-verify Disable host key verification
-u, --user USER API Client Username
--prerelease Install the pre-release chef gems
--print-after Show the data after a destructive operation
--region REGION Your AWS region
-r, --run-list RUN_LIST Comma separated list of roles/recipes to apply
-G, --groups X,Y,Z The security groups for this server
-S, --ssh-key KEY The AWS SSH key id
-P, --ssh-password PASSWORD The ssh password
-x, --ssh-user USERNAME The ssh username
-s, --subnet SUBNET-ID create node in this Virtual Private Cloud Subnet ID (implies VPC mode)
--template-file TEMPLATE Full path to location of template to use
-V, --verbose More verbose output. Use twice for max verbosity
-v, --version Show chef version
-y, --yes Say yes to all prompts for confirmation
-h, --help Show this message
Graphite Install
*this assumes debian or ubuntu
Graphite Cookbook Install
$ git clone https://github.com/cookingclouds/cookbooks/tree/master/cc_graphite
I Know there is a cookbook for graphite in the opscode repo but it has dependencies.
This cookbook shouldn’t have dependencies and is also a lot more messy
Run after installing cookbook :
$ /opt/graphite/bin/carbon-cache.py start $ cd /opt/graphite/webapp/graphite && python manage.py createsuperuser
Manual Install
Grab the sourcecode
$ wget https://launchpad.net/graphite/0.9/0.9.10/+download/graphite-web-0.9.10.tar.gz $ wget https://launchpad.net/graphite/0.9/0.9.10/+download/carbon-0.9.10.tar.gz $ wget https://launchpad.net/graphite/0.9/0.9.10/+download/whisper-0.9.10.tar.gz $ wget https://launchpad.net/graphite/0.9/0.9.10/+download/check-dependencies.py
Install Whisper Database
$ ls |grep tar |while read i ; do tar xzvf $i ; done $ apt-get -y install apache2 python-django python-django-tagging python-ldap python-memcache python-cairo $ cd ~/whisper-0.9.10/ $ python setup.py install
Install Carbon
$ apt-get -y install python-twisted python-simplejson $ cd ~/carbon-0.9.10/ $ python setup.py install $ cd /opt/graphite/conf/ $ cp carbon.conf.example carbon.conf $ cp storage-schemas.conf.example storage-schemas.conf $ cd /opt/graphite/ $ ./bin/carbon-cache.py start
Install Graphite-web
$ cd ~/graphite-web-0.9.10/ $ python check-dependencies.py $ python setup.py install $ cp examples/example-graphite-vhost.conf /etc/apache2/sites-available/default $ cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi $ mkdir /etc/apache2/run $ apt-get -y install libapache2-mod-wsgi $ /etc/init.d/apache2 reload $ cd /opt/graphite/webapp/graphite/ $ python manage.py syncdb $ chown -R www-data:www-data /opt/graphite/storage $ cd /opt/graphite/webapp/graphite $ cp local_settings.py.example local_settings.py $ /etc/init.d/apache2 restart
Examples on how to send data to graphite
hostname = * you guessed it
test_001 = service name
1001 = value
$NOW = epoch time to input in graph
$ NOW=`date +%s` ; echo "hostname.test_001 1001 $NOW" |nc localhost 2003
or
$ cd /opt/graphite/examples $ ./example-client.py
cookingclouds 