CouchDB 1.0.1 – Replication not completing

* on ubuntu 12.04 using couchdb 1.0.1

Doing this fails – Pulling replication from the slave server:

$ curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"source":"http://192.168.1.100:5984/chef","target":"chef","continuous":false}'

{"error":"http_request_failed","reason":"failed to replicate http://192.168.1.100:5984/chef/"}

Relevant Logs:
From the slave couchdb logs trying to pull replication:
https://gist.github.com/3019911
Example of the couchdb table from the master running: curl http://192.168.1.100:5984/chef/
https://gist.github.com/3020474
From the master couchdb logs:
https://gist.github.com/3020547

Doing this works – Pushing Replication from the master to the slave:

$ curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"target":"http://192.168.1.11:5984/chef","source":"chef","continuous":false}'

{"ok":true,"session_id":"7a0c0632a92e68983b3d0a2d07c47dd8","source_last_seq":23,"history":[{"session_id":"7a0c0632a92e68983b3d0a2d07c47dd8","start_time":"Fri, 29 Jun 2012 21:21:22 GMT","end_time":"Fri, 29 Jun 2012 21:21:22 GMT","start_last_seq":0,"end_last_seq":23,"recorded_seq":23,"missing_checked":0,"missing_found":3,"docs_read":3,"docs_written":3,"doc_write_failures":0}]}
Advertisement

Migrating Chef CouchDB to Multi-Master CouchDB

* assuming you are using ubuntu/debian
chef-server = 192.168.1.10
couchdb01 = 192.168.1.11
couchdb02 = 192.168.1.12

Enable chef-server couchdb to listen on all interfaces

root@chefserver:~# sed -i bak s/bind_address = 127.0.0.1/bind_address = 0.0.0.0/g /etc/couchdb/default.ini
root@chefserver:~# /etc/init.d/couchdb restart

Install CouchDB on couchdb01/couchdb02 and set to listen on all interfaces

root@couchdb01:~# apt-get -y install couchdb
root@couchdb01:~# /etc/init.d/couchdb stop
root@couchdb01:~# sed -i bak s/bind_address = 127.0.0.1/bind_address = 0.0.0.0/g /etc/couchdb/default.ini
root@couchdb01:~# /etc/init.d/couchdb start

Create the empty chef table on couchdb01/couchdb02

root@couchdb01:~# curl -X PUT http://localhost:5984/chef
{"ok":true}

root@couchdb02:~# curl -X PUT http://localhost:5984/chef
{"ok":true}

Push the chef table from chef-server to couchdb01/02 and enable a continuous replication

To Couchdb02

root@chef-server:/var/lib/couchdb# curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"source":"chef","target":"http://192.168.1.12:5984/chef","continuous":true}'
{"ok":true,"_local_id":"77f057c373dca43097fac542c367b24f"}

To Couchdb01

root@chef-server:~# curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"source":"chef","target":"http://192.168.1.11:5984/chef","continuous":true}'
{"ok":true,"_local_id":"e926c9297e5776db862ae3c1be27bbde"}

Setup the multi-master replication for couchdb01/02

Enable continuous replication FROM couchdb01 to couchdb02

root@couchdb01:/var/lib/couchdb# curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"source":"chef","target":"http://192.168.1.12:5984/chef","continuous":true}'
{"ok":true,"_local_id":"77f057c373dca43097fac542c367b24f"}

Enable continuous replication FROM couchdb02 to couchdb01

root@couchdb02:/var/lib/couchdb# curl -X POST http://localhost:5984/_replicate -H "Content-Type: application/json" -d '{"source":"chef","target":"http://192.168.1.11:5984/chef","continuous":true}'
{"ok":true,"_local_id":"77f057c373dca43097fac542c367b24f"}

Install Apache and generate config on Chef Server

root@chefserver:~# apt-get -y install apache2
root@chefserver:~# mkdir -p /usr/share/chef-server/public
root@chefserver:~# for i in rewrite proxy status proxy_http proxy_balancer headers ; do a2enmod $i ; done
root@chefserver:~# cd /etc/apache2/sites-available
root@chefserver:~# echo "Listen 5984" | tee -a chef_couchdb_loadbalancer
root@chefserver:~# echo '<VirtualHost *:5984>' | tee -a chef_couchdb_loadbalancer
root@chefserver:~# MYHOST=$(hostname -f)
root@chefserver:~# echo "ServerName ${MYHOST}-couchdb" |tee -a chef_couchdb_loadbalancer
root@chefserver:~# cat>>chef_couchdb_loadbalancer<<EOF
<Proxy balancer://couchlb>
BalancerMember http://192.168.1.11:5984
BalancerMember http://192.168.1.12:5984
</Proxy>
ProxyPass / balancer://couchlb
ProxyPassReverse / balancer://couchlb
DocumentRoot /usr/share/chef-server/public
LogLevel info
ErrorLog /var/log/chef/chef_couchdb_apache2-error.log
CustomLog /var/log/chef/chef_couchdb_apache2-access.log combined
RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://couchlb%{REQUEST_URI} [P,QSA,L]
</VirtualHost>
EOF

Stop Couchdb on Chef-Server

*This will stop the continuous replication to couchdb01/couchdb02

root@chefserver:~# /etc/init.d/couchdb stop

Start Apache load balancer on Chef-Server

root@chefserver:~# a2ensite chef_couchdb_loadbalancer
root@chefserver:~# /etc/init.d/apache2 restart

Test your couchdb balancer:

http:://192.168.1.10:5984/_utils

You probably also want to edit your init scripts on couchdb01/02 to automatically restart the continuous replication upon start or restart
The replication does not persist after you stop a couchdb instance unless you explicitly issue the command again

Chef Server – Threading Merb – Chef Server API Service

This will only make your api system faster if you have enough cpus to support it.
It should be 1 merb worker per core ( * i think * )

Install Apache and enable the necessary mods

$ apt-get -y install apache2
$ mkdir -p /usr/share/chef-server/public
$ for i in rewrite proxy status proxy_http proxy_balancer headers ; do a2enmod $i ; done

Stop Chef-Server

$ /etc/init.d/chef-server stop

Edit your Chef configuration file ( Replace Worker Threads and Port Numbers as needed )

$ sed -i s/PORT=4000/PORT=5000/g /etc/default/chef-server
$ echo "WORKERTHREADS=4" | tee -a /etc/default/chef-server

Edit the Chef init script ( Back it up first )

$ cp /etc/init.d/chef-server /etc/init.d/chef-server.original
$ sed -i '35s/DAEMON_OPTS="-p/DAEMON_OPTS="-c $WORKERTHREADS -p/g' /etc/init.d/chef-server
$ sed -i '42s/(ps/#(ps/g' /etc/init.d/chef-server
$ sed -i '/#(ps/ i (ps -fp $pid | egrep -q "merb.*( chef-server .*api.* spawner|worker .* $PORT)") || return 1' /etc/init.d/chef-server

Run diff to see the init script differences

$ diff /etc/init.d/chef-server /etc/init.d/chef-server.original
35c35
< DAEMON_OPTS="-c $WORKERTHREADS -p $PORT -e production -d -a $ADAPTER -P $PIDFILE -L $LOGFILE -C $CONFIG -u $USER -G $GROUP -V" --- > DAEMON_OPTS="-p $PORT -e production -d -a $ADAPTER -P $PIDFILE -L $LOGFILE -C $CONFIG -u $USER -G $GROUP -V"
42,43c42
< (ps -fp $pid | egrep -q "merb.*( chef-server .*api.* spawner|worker .* $PORT)") || return 1
<   #(ps -fp $pid | egrep -q "merb.*(merb : master|worker.*$PORT)") || return 1 --- >   (ps -fp $pid | egrep -q "merb.*(merb : master|worker.*$PORT)") || return 1

Get variables from your Chef config for apache config construction

$ THREADSCT=$(grep "WORKERTHREADS" /etc/default/chef-server |awk -F"=" '{print $2}')
$ NEWCOUNT=$(( THREADSCT -- ))
$ PORTPREFIX=$(grep "PORT" /etc/default/chef-server |awk -F"=" '{print $2}'| sed -e s/[0-9][0-9]$//g )
$ MYHOST=$(hostname -f)

Generate Your Apache Config – Generate the Load Balance Members

$ cd /etc/apache2/sites-available
$ echo "Listen 4000" |tee -a chef_loadbalancer
$ echo '<VirtualHost *:4000>' |tee -a chef_loadbalancer
$ echo "ServerName $MYHOST" |tee -a chef_loadbalancer
$ echo "" |tee -a chef_loadbalancer
$ echo "<Proxy balancer://cheflb>" | tee -a chef_loadbalancer
$ seq -w 00 $NEWCOUNT | while read i ; do echo "BalancerMember http://127.0.0.1:${PORTPREFIX}${i}" |tee -a chef_loadbalancer ; done

Append your Apache config with the rest of the relevant information

$ cat>>chef_loadbalancer<<EOF
</Proxy>
ProxyPass / balancer://cheflb
ProxyPassReverse / balancer://cheflb
DocumentRoot /usr/share/chef-server/public
LogLevel info
ErrorLog /var/log/chef/chef_server_apache2-error.log
CustomLog /var/log/chef/chef_server_apache2-access.log combined
RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://cheflb%{REQUEST_URI} [P,QSA,L]
</VirtualHost>
EOF

Enable your new apache config and start Apache and Chef-Server

$ a2ensite chef_loadbalancer
$ /etc/init.d/apache2 restart
$ /etc/init.d/chef-server start

Test your chef server

$ time knife node list
$ time knife role list

Reference:

Much of this information was stolen from : http://mrmiller.nonesensedomains.com/2010/06/15/chef-performance-tuning-part-1/
If this technique is outdated please make me aware of it or if my apache configuration is awful ( which i’m sure it is, i’m just too lazy to improve it )

Chef Server – Backup and Restore

* This assumes you’re using the regular debian/ubuntu install
and that you have a system with a fresh install of chef-server laying around

Backup your files on the OLD/Original Chef Server
This backs up all the chef configs/certs and couchdb configs and data

$ tar czvf chef-backup-`date +%Y-%m-%d-%s`.tar.gz /etc/couchdb /var/lib/chef /var/lib/couchdb /var/cache/chef /var/log/chef /var/log/couchdb /etc/chef

Transfer your files to the NEW Chef Server

$ scp old-chefserver:~/chef-backup*.tar.gz /tmp

Stop all chef services on the NEW Chef Server

$ for i in chef-server-webui chef-server rabbitmq-server jetty couchdb chef-solr ; do /etc/init.d/${i} stop ; done

Extract the backup file on to the NEW Chef Server

$ cd /tmp
$ tar xzvf chef-backup*.tar.gz

Delete the old rabbitmq data from your default installation on the NEW Chef Server

$ rm -fr /var/lib/rabbitmq/mneisa/*

Start rabbitmq in the foreground and recreate your user and tables on the NEW Chef Server

$ rabbitmq-server
$ rabbitmqctl add_vhost /chef
Creating vhost "/chef" ...
...done.
$ rabbitmqctl add_user chef testing
Creating user "chef" ...
...done.
$ rabbitmqctl set_permissions -p /chef chef ".*" ".*" ".*"
Setting permissions for user "chef" in vhost "/chef" ...
...done.
$ rabbitmqctl stop
Stopping and halting node rabbit@ubuntu02 ...
...done.

Start all the Chef Services on NEW Chef Server

$ for i in chef-solr couchdb jetty rabbitmq-server chef-server chef-server-webui ; do /etc/init.d/${i} start ; done

Verify your Chef Server is restored
http://NEW-CHEF-SERVER:4040
and verify the nodes and data or try the following knife commands

$ knife node list
$ knife node show mynode -a node

Alternate Backup Methods from Opscode:

http://wiki.opscode.com/display/chef/Backing+Up+Chef+Server

Chef Server – Debian/Ubuntu Install

* Please make sure to change “mynewpassword” to whatever password you want to use.

$ apt-get -y install lsb-release wget debconf-utils
$ echo "deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main" | tee /etc/apt/sources.list.d/opscode.list
$ gpg --keyserver keys.gnupg.net --recv-keys 83EF826A
$ gpg --export packages@opscode.com | tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null
$ cat>answerfile<<EOF
chef-server-webui chef-server-webui/admin_password password mynewpassword
chef-solr chef-solr/amqp_password password mynewpassword
EOF
$ url=`hostname -f` ; echo "chef chef/chef_server_url string http://$url:4000" |tee -a answerfile
$ debconf-set-selections ./answerfile
$ echo "deb http://www.rabbitmq.com/debian/ testing main" |tee -a /etc/apt/sources.list
$ wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
$ apt-key add rabbitmq-signing-key-public.asc
$ apt-get update
$ apt-get -y install chef-server opscode-keyring

Login at http://hostname:4040/
Username: admin
# This is the password you set in the answerfile above
Password: mynewpassword

Chef Server – Troubleshooting a fresh install on ubuntu 12.04

If your chef-server is not starting on 12.04
There is a relevant bug at : http://tickets.opscode.com/browse/CHEF-2766
This most likely happens if you didn’t run “apt-get update” before you installed chef-server

try running chef-server on it’s own and see what messages you get.
$ chef-server

Loading init file from /usr/share/chef-server-api/config/init.rb
[Mon, 25 Jun 2012 06:31:53 +0000] INFO: Loading roles
Loading /usr/share/chef-server-api/config/environments/development.rb
[Mon, 25 Jun 2012 06:31:53 +0000] INFO:
[Mon, 25 Jun 2012 06:31:53 +0000] INFO: ********************************************************************************
[Mon, 25 Jun 2012 06:31:53 +0000] INFO: *   Starting Chef Server in Development Mode.
[Mon, 25 Jun 2012 06:31:53 +0000] INFO: *   Start the server with `-e production` for normal use
[Mon, 25 Jun 2012 06:31:53 +0000] INFO: ********************************************************************************
[Mon, 25 Jun 2012 06:31:53 +0000] INFO:
NOTE: Gem.activate is deprecated, use Specification#activate. It will be removed on or after 2011-10-01.
Gem.activate called from /usr/lib/ruby/1.8/merb-core/core_ext/kernel.rb:142.
/usr/lib/ruby/vendor_ruby/1.8/rubygems.rb:232:in `_deprecated_activate': Deprecated use of Gem.activate(dep) (ArgumentError)
        from /usr/lib/ruby/vendor_ruby/1.8/rubygems/deprecate.rb:63:in `send'
        from /usr/lib/ruby/vendor_ruby/1.8/rubygems/deprecate.rb:63:in `activate'
        from /usr/lib/ruby/1.8/merb-core/core_ext/kernel.rb:142:in `load_dependency'
        from /usr/lib/ruby/1.8/merb-core/bootloader.rb:405:in `load_dependencies'
        from /usr/lib/ruby/1.8/merb-core/bootloader.rb:405:in `each'
        from /usr/lib/ruby/1.8/merb-core/bootloader.rb:405:in `load_dependencies'
        from /usr/lib/ruby/1.8/merb-core/bootloader.rb:393:in `run'
        from /usr/lib/ruby/1.8/merb-core/bootloader.rb:99:in `run'
        from /usr/lib/ruby/1.8/merb-core/server.rb:172:in `bootup'
        from /usr/lib/ruby/1.8/merb-core/server.rb:42:in `start'
        from /usr/lib/ruby/1.8/merb-core.rb:170:in `start'
        from /usr/sbin/chef-server:86

Fix the offending line:
$ sed -ie ‘142s/Gem.activate(dep)/dep.to_spec.activate/’ /usr/lib/ruby/1.8/merb-core/core_ext/kernel.rb
$ /etc/init.d/chef-server start
$ /etc/init.d/chef-server-webui start

Chef – Configure Knife Admin

Install Ruby, Ruby gems, and Chef-client

Mac OSX

Install X-Code from Mac Appstore
Open Xcode and do the following :  Xcode >> Preferences >> Downloads >> Command Line Tools
$ cd /tmp
$ curl -O http://production.cf.rubygems.org/rubygems/rubygems-1.8.24.tgz
$ tar zxf rubygems-1.8.24.tgz
$ cd rubygems-1.8.24
$ ruby setup.rb –no-format-executable
$ sudo gem install chef

CentOS6

$ yum -y install ruby ruby-devel ruby-ri ruby-rdoc ruby-shadow gcc gcc-c++ automake autoconf make curl dmidecode git
$ cd /tmp
$ curl -O http://production.cf.rubygems.org/rubygems/rubygems-1.8.24.tgz
$ tar zxf rubygems-1.8.24.tgz
$ cd rubygems-1.8.24
$ ruby setup.rb –no-format-executable
$ sudo gem install chef

Ubuntu

$ apt-get install -y curl git ruby1.9.1 rubygems1.9.1 chef

Copy the validation.pem and webui.pem

This is the key that allow you to register your host as a client to the chef server
$ scp root@chef-server:/etc/chef/validation.pem  /etc/chef/validation.pem

This is the key that allows you to authenticate yourself as an admin user to the chef server
Chef requires an existing admin to authorize a new admin
$ scp root@chef-server:/etc/chef/webui.pem  /etc/chef/webui.pem

Run Knife Configure

$ knife configure -i
Example Output:

Overwrite /root/.chef/knife.rb? (Y/N) y
Please enter the chef server URL: [http://chef-server:4000] http://chef-server:4000
Please enter a clientname for the new client: [root] jtran
Please enter the existing admin clientname: [chef-webui]
Please enter the location of the existing admin client's private key: [/etc/chef/webui.pem]
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef/validation.pem]
Please enter the path to a chef repository (or leave blank):
Creating initial API user...
Created client[jtran]
Configuration file written to /root/.chef/knife.rb

Verify Knife is working

$ knife client list
Example Output:

  jtran
  chef-webui
  chef-validator

Chef – Attribute overriding

Let’s pretend we have a cookbook named “wonka”

in ~/chef-repo/cookbooks/wonka/attributes/default.rb

default[:wonka][:gobstopper]  = "everlasting"
default[:wonka][:kazoozle]    = "alittle"

We want to override the value of [:wonka][:gobstopper] for a node named “oompaloompa01”

Without an override defined your json should look something like the following
You can issue:
$ knife node edit ooompaloopma01

{
  "run_list": [
    "recipe[chocolate_factory::default]"
  ],
  "chef_environment": "development",
  "name": "oompaloompa01",
  "normal": {
    "chef_environment": "development",
    "name": "oompaloopma01",
    "normal": {
      "tags": [

      ]
    },
    "tags": [

    ]
  }
}

Now let’s say you want to define default[:wonka][:gobstopper] = “allgone” with an override
Your json should look like the following

{
  "run_list": [
    "recipe[chocolate_factory::default]"
  ],
  "chef_environment": "development",
  "name": "oompaloompa01",
  "normal": {
    "chef_environment": "development",
    "name": "oompaloopma01",
    "wonka": {
      "gobstopper": [
        "allgone"
      ]
    },
    "normal": {
      "tags": [

      ]
    },
    "tags": [

    ]
  }
}

that should commit the override to that specific node.
you can check the values of a node’s attributes with the following command ( where the attribute group you want to look at is “wonka” )
$ knife node show oompaloompa01 -a wonka

Now just run chef-client on oompaloopa01 and you should see the override take effect.

Chef – Encrypted Data-bags example

Create your encrypt/decrypt “key”

$ openssl rand -base64 512 > ~/.chef/encrypted_data_bag_secret

Create a new “data bag” named “mysecrets”

$ knife data bag create mysecrets

Create a new json with information that you want encrypted.

This will be stored inside the “data bag” named “mysecrets”
This will use the “key” you created earlier to encrypt
We will store this as “marioworld”

$ knife data bag create mysecrets marioworld –secret-file ~/.chef/encrypted_data_bag_secret
* This will prompt open an editor to add items to json

{ "id": "marioworld",
"user": "luigi"
"pass": "yahoo"
}

Now Create a simple recipe and template file that will utilize this encrypted “data bag”

Create the Recipe

$ knife cookbook create databag-test
$ cd ~/chef-repo/cookbooks/recipes/
$ vi default.rb

#
# Cookbook Name:: databag-test
# Recipe:: default
#
# Copyright 2012, James Tran
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# This is where you will store a copy of your key on the chef-client
secret = Chef::EncryptedDataBagItem.load_secret("/etc/chef/encrypted_data_bag_secret")

# This decrypts the data bag contents of "mysecrets->marioworld" and uses the key defined at variable "secret"
luigi_keys = Chef::EncryptedDataBagItem.load("mysecrets", "marioworld", secret)

template "/tmp/databag" do
     variables(:mypass => luigi_keys['pass'],
               :myuser => luigi_keys['user'])
     owner "root"
     mode  "0644"
     source "databag_test.erb"
end

Create the Template

$ cd ~/chef-repo/cookbooks/databag-test/templates
$ vi databag_test.erb

Username: <%= @myuser %>
Password: <%= @mypass %>

Copy your “key” to the node

$ scp ~/.chef/encrypted_data_bag_secret root@somenode:/etc/chef/

Add the recipe to a node and run chef-client

$ knife node run_list add somenode “recipe[databag-test]”
$ knife ssh “name:somenode” -x root “chef-client”

Verify the contents of the new file created at /tmp/databag

$ knife ssh “name:somenode” -x root “cat /tmp/databag”

Username: luigi
Password: yahoo