Graphite – aggregating your second retention bucket

Make sure your storage-aggregator.py is running in order databases to compress down into 2nd and 3rd retention levels.

$ ls
league_number.wsp  user_number.wsp

Find out info on your database “league_number”
Seeing something like xFilesFactor .5 means that 50% of that time retention period has to show
if you have 10 second intervals but are only inputting once every 60 seconds then your xFilesFactor is only .2
Set the xFilesFactor = 0 if you don’t want to drop any of your data

$ /usr/local/bin/whisper-info.py league_number.wsp
maxRetention: 315360000
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 63097960

Archive 0
retention: 21600
secondsPerPoint: 10
points: 2160
size: 25920
offset: 40

Archive 1
retention: 315360000
secondsPerPoint: 60
points: 5256000
size: 63072000
offset: 25960

Resize your whisper database if data is not being aggregated properly

$ /usr/local/bin/whisper-resize.py league_number.wsp
Usage: whisper-resize.py path timePerPoint:timeToStore [timePerPoint:timeToStore]*

timePerPoint and timeToStore specify lengths of time, for example:

60:1440      60 seconds per datapoint, 1440 datapoints = 1 day of retention
15m:8        15 minutes per datapoint, 8 datapoints = 2 hours of retention
1h:7d        1 hour per datapoint, 7 days of retention
12h:2y       12 hours per datapoint, 2 years of retention

If you have fucked up graphs that are truncating data on the second retention size
align them like so

$ /usr/local/bin/whisper-resize.py user_number.wsp --xFilesFactor=0 --aggregationMethod=average 10:6h 1min:10y

Make sure you set your storage-aggregation correctly and make sure it’s started

$ cp storage-aggregation.conf.example storage-aggregation.conf

$ /opt/graphite/bin/carbon-aggregator.py status
carbon-aggregator (instance a) is not running
$ /opt/graphite/bin/carbon-aggregator.py start
Starting carbon-aggregator (instance a)
Advertisements

Graphite Install

*this assumes debian or ubuntu

Graphite Cookbook Install

$ git clone https://github.com/cookingclouds/cookbooks/tree/master/cc_graphite

I Know there is a cookbook for graphite in the opscode repo but it has dependencies.
This cookbook shouldn’t have dependencies and is also a lot more messy
Run after installing cookbook :

$ /opt/graphite/bin/carbon-cache.py start
$ cd /opt/graphite/webapp/graphite && python manage.py createsuperuser

Manual Install

Grab the sourcecode


$ wget https://launchpad.net/graphite/0.9/0.9.10/+download/graphite-web-0.9.10.tar.gz
$ wget https://launchpad.net/graphite/0.9/0.9.10/+download/carbon-0.9.10.tar.gz
$ wget https://launchpad.net/graphite/0.9/0.9.10/+download/whisper-0.9.10.tar.gz
$ wget https://launchpad.net/graphite/0.9/0.9.10/+download/check-dependencies.py

Install Whisper Database

$ ls |grep tar |while read i ; do tar xzvf $i ; done
$ apt-get -y install apache2 python-django python-django-tagging python-ldap python-memcache python-cairo
$ cd ~/whisper-0.9.10/
$ python setup.py install

Install Carbon

$ apt-get -y install python-twisted python-simplejson
$ cd ~/carbon-0.9.10/
$ python setup.py install
$ cd /opt/graphite/conf/
$ cp carbon.conf.example carbon.conf
$ cp storage-schemas.conf.example storage-schemas.conf
$ cd /opt/graphite/
$ ./bin/carbon-cache.py start

Install Graphite-web

$ cd ~/graphite-web-0.9.10/
$ python check-dependencies.py
$ python setup.py install
$ cp examples/example-graphite-vhost.conf /etc/apache2/sites-available/default
$ cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
$ mkdir /etc/apache2/run
$ apt-get -y install libapache2-mod-wsgi
$ /etc/init.d/apache2 reload
$ cd /opt/graphite/webapp/graphite/
$ python manage.py syncdb
$ chown -R www-data:www-data /opt/graphite/storage
$ cd /opt/graphite/webapp/graphite
$ cp local_settings.py.example local_settings.py
$ /etc/init.d/apache2 restart

Examples on how to send data to graphite

hostname = * you guessed it
test_001 = service name
1001 = value
$NOW = epoch time to input in graph

$ NOW=`date +%s` ; echo "hostname.test_001 1001 $NOW" |nc localhost 2003

or

$ cd /opt/graphite/examples
$ ./example-client.py