Sunday, January 15, 2012

Pushing metrics / baselines via REST interface in RHQ

A few days ago I wrote about pulling raw metrics from RHQ via the REST interface.

It is also possible to push metrics as well as read and write baselines.

Pushing metrics

There are two ways to push numeric metric values into the server:

Single metric

curl -i -u rhqadmin:rhqadmin \
http://localhost:7080/rest/1/metric/data/10022/raw/1324656971 \
-X PUT \
-d @/tmp/foo \

With /tmp/foo containing:
{"timeStamp": 132465716178, "value": 123, "scheduleId":10023}

Note that you need to give the schedule id (10022) and the timestamp in the URL. The samples-project also contains an example in Python.

Multiple metrics

If you want to push multiple metrics at once you can use this call (of course it will also work for a single one):

curl -u rhqadmin:rhqadmin \
http://localhost:7080/rest/1/metric/data/raw \
-d @/tmp/foo \

with /tmp/foo containing:

{"timeStamp": 132465716078, "value": 123, "scheduleId":10022},
{"timeStamp": 132465716079, "value": 223, "scheduleId":10022}


Baselines are an interesting feature in the sense that they mark a band in which a dynamic metric usually oscillates. When the metric goes out of those bounds, you can get an alarm. The system usually computes those baselines by taking the minimum and maximum values from the existing dataset for the last n days (n is configurable). When a baseline is computed, they are also displayed on the large metric graphs. Here the baselines could also be manually set.

It is now (master/RHQ 4.3) possible to set the computation frequency to 0 to disable the automatic calculation by the server and push baseline data via the REST interface into the server. This allows to e.g. read metrics from a system like R, compute projections for future bands (e.g. via Holt-Winters) or x% quantiles of the existing data and write the results back as baseline data so that the normal alerting workflow can pick up that data.

The first call is to obtain the baseline for schedule with id 10013:

curl -u rhqadmin:rhqadmin \
http://localhost:7080/rest/1/metric/data/10013/baseline \

This call updates the schedule 10013 with new values:

curl -u rhqadmin:rhqadmin \
http://localhost:7080/rest/1/metric/data/10013/baseline \
-HContent-Type:application/json \
-HAccept:application/json \
-X PUT \
-d '{"max":2.58304512E9, \
"min":0.119968768E9, \
"mean":1.285011894659459E9, \

No comments: