Clustered AMQP Rabbit-MQ

RabbitMQ already has excellent documentation at: http://www.rabbitmq.com/clustering.html

But this will be a bit more chef specific
server01 = existing chef-server
server02 = new rabbitmq server to be added to cluster

On Server01

On your existing chef-server
*I assume you’re using the latest rabbitmq and that server02 will also install matching version
Check what the cluster output looks like

root@server01:~# rabbitmqctl cluster_status
Cluster status of node rabbit@server01 ..
[{nodes,[{disc,[rabbit@server01]}]},{running_nodes,[rabbit@server01]}]
...done.

Copy your rabbitmq cookie to server02

root@server01:~# scp /var/lib/rabbitmq/.erlang.cookie root@server02:/var/lib/rabbitmq/
Are you sure you want to continue connecting (yes/no)? yes
root@server02's password: 
.erlang.cookie                                                                                        100%   20     0.0KB/s   00:00    

On Server02

Install rabbitmq

root@server02:~# echo "deb http://www.rabbitmq.com/debian/ testing main" |tee -a /etc/apt/sources.list
root@server02:~# wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
root@server02:~# apt-key add rabbitmq-signing-key-public.asc
root@server02:~# apt-get update
root@server02:/var/chef/cache# apt-get -y install rabbitmq-server 

Delete existing mnesia database and start rabbitmq

root@server02:~# service rabbitmq-server stop
root@server02:~# rm -fr /var/lib/rabbitmq/mnesia 
root@server02:~# service rabbitmq-server start
 * Starting message broker rabbitmq-server
   ...done.

Join Server02 to Server01

root@server02:~# rabbitmqctl cluster_status
Cluster status of node rabbit@server02 ...
[{nodes,[{disc,[rabbit@server02]}]},{running_nodes,[rabbit@server02]}]
...done.
root@server02:~# rabbitmqctl stop_app
Stopping node rabbit@server02 ...
...done.
root@server02~# rabbitmqctl reset   
Resetting node rabbit@server02 ...
...done.
root@server02:~# rabbitmqctl cluster rabbit@server01 rabbit@server02
Clustering node rabbit@server02 with [rabbit@server01,rabbit@server02] ...
...done.
root@server02:~# rabbitmqctl start
root@server02:~# rabbitmqctl cluster_status
Cluster status of node rabbit@server01 ...
[{nodes,[{disc,[rabbit@server02,rabbit@server01]}]},
 {running_nodes,[rabbit@server02,rabbit@server01]}]
...done.

Verify on server01 that cluster shows up as expected

root@server01:~# rabbitmqctl cluster_status
Cluster status of node rabbit@server01 ...
[{nodes,[{disc,[rabbit@server02,rabbit@server01]}]},
 {running_nodes,[rabbit@server02,rabbit@server01]}]
...done.

Make sure chef still works

root@server01:~# knife node list

root@server01:~# knife client list
  chef-validator
  chef-webui
  root
  server01
root@server01:~# knife bootstrap -x root server02 --template-file ~/ubuntu12.04.rb 
Bootstrapping Chef on server02
Failed to authenticate root - trying password auth
Enter your password: 
server02 Updating installed gems

root@server01:~# knife node list
server02