RRDTOOL – How to remove spikes

We use Cacti at work to graph the usage of our clients links. It is a pretty popular feature with our customers. A problem (well not really. More like a gotcha) with rrdtool is the way it stores data. Here’ a quote from the rrd tutorial
“Round robin is a technique that works with a fixed amount of data, and a pointer to the current element. Think of a circle with some dots plotted on the edge, these dots are the places where data can be stored. Draw an arrow from the center of the circle to one of the dots, this is the pointer. When the current data is read or written, the pointer moves to the next element. As we are on a circle there is no beginning nor an end, you can go on and on. After a while, all the available places will be used and the process automatically reuses old locations. This way, the database will not grow in size and therefore requires no maintenance. RRDTool works with with Round Robin Databases (RRDs).”

So rrd stores the difference in values (between the last value and current) in the database, rather than the value itself. This creates a problem when routers are rebooted. The counters on the interfaces get cleared and rrd is fooled into thinking that there is a spike in usage. This results to “spikes” in the graphs. Sometimes you see that a 128kbps link has maxed out at 98mbps!!! :). The best way to stop this from happening is to set the correct min and max values for the ds names. Coming back to cacti again. When cacti creates a new rrd database, it does not really give one the option to setup the maximum and minimum speeds of a interface. It defaults to a max of 100000000 (i.e. 100mbps). Occasionally when we have to reboot our routers, I do the following to remove the spikes

cp filename.rrd filename.rrd.backup
Any good admin knows that before you mess with a file, you make a backup :).

rrdtool info filename.rrd | more
This gives us the chance to get the ds (data sources) names

rrdtool tune filename.rrd -a ds_name:MAXIMUM_VALUE
Set the maximum of the ds to the required

rrdtool dump filename.rrd > filename.xml
Export all data in the rrd to a xml file

mv filename.rrd filename.rrd.old
Rename the rrd to make way for the new one.

rrdtool restore filename.xml filename.rrd -r
Restore the rrd from the xml file with the -r (range check) option. So any values that are higher than the new maximum value are ignored.

And the spikes are gone..