As you may know I am currently working on RHQ-Metrics, a time series database with some charting extensions that will be embeddable into Angular apps.
One of the goals of the project is also to make the database also available for other consumers that may have other input sources deployed as well as their own graphing.
Over the weekend I looked a bit at Grafana which looks quite nice, so I started to combine RHQ-metrics with Grafana.
I did not (immediately) find a description of the data format between Grafana and Graphite, but saw that InfluxDB is supported, so I fired up WireShark and was able to write an endpoint for RHQ-Metrics that supports listing of metrics and retrieving of data.
The overall setup looks like this:
Setup used
A Ganglia gmond emits data over IP multicast, which is received with the protocol translator client, ptrans. This translates the data into requests that are pushed into the RHQ-metrics system (shown as the dotted box) and there stored in the Cassandra backend.
Grafana then talks to the RHQ-Metrics server and fetches the data to be displayed.
For this to work, I had to comment out the graphiteUrl in conf.js and use the influx backend like this:
datasources: {
influx: {
default: true,
type: 'influxdb',
username: 'test',
password: 'test',
url: 'http://localhost:8080/rhq-metrics/influx',
}
},As you can see, the URL points to the RHQ-Metrics server.
The code for the new handler is still "very rough" and thus not yet in the normal source repository. To use it, you can clone my version of the RHQ-Metrics repository and then run start.sh in the root directory to get the RHQ-Metrics server (with the built-in memory backend) running.
I have now added code (still in my repository) to support some of the aggregating functions of Influx like min, max, mean, count and sum. The following shows a chart that uses those functions:
11 comments:
Great work :) Is this now merged upstream? Looking to get something like this integrated into Fabric8.
Yes, that is in the latest rhq-metrics snapshot release. I think James wanted to have a look at it. He will be happy for pull requests for sure ;-)
Excellent - I'll take a look. Been discussing it with James - I've just joined the JBoss Fuse engineering team & looking at the Fabric8 insight stuff. Saw this & thought a lot of my job has been done for me - thanks for that :)
Don't you think "ptrans" a reverse proxy based on java is an overkill ? Shouldn't it be based on c/c++ or go or at-least a python script ?
Your rrds database of gmond gets replicated in influxdb ?
This is something I want to solve now.
Did you have such issue ?
Did you solve it and how ?
Kishorekumar,
this may indeed be overkill. as I wrote in http://pilhuhn.blogspot.de/2014/05/my-first-netty-based-server.html I wanted to try to do something with Netty and had a specific need for something. So I started it like this.
Also my C knowledge is a bit rusty ;-)
It would be awesome if you would contribute a python script for this.
I do not replicated the rrd database, but gmond is sending its data via IP multicast as UDP packets on port 8649, which ptrans receives, translates and forwards to the rhq-metrcis server, where the data is stored.
The Grafana UI is then accessing the rhq-metrics server via the influx "protocol" to retrieve the metrics and display them.
I did not try to directly send those packets to a real influx-db. It should not be too hard though to a add an additional backend to ptrans like https://github.com/rhq-project/rhq-metrics/blob/master/clients/ptranslator/src/main/java/org/rhq/metrics/clients/ptrans/backend/RestForwardingHandler.java that does not send to rhq-metrics, but to influx.
Thanks.
In summary gmond sends to influxdb via ptrans so gmetad doesn't exist and hence no rrds.
Neat.
I have seen https://github.com/prometheus/gmond_exporter which has code to build a gmond - to influxdb sender.
Not exactly - ptrans sends (via rhq-metrics "protocol") to the rhq-metrics db, which is able to understand the influxdb queries to retrieve data.
Thanks for the pointer to the gmond_exporter.
Thanks Heiko Rupp.
Will keep you posted about our work.
To clarify things you don't have a gmetad right ?
Its gmond -> ptrans -> rhq_metrics_db using rhq_metrics protocol.
Cool.
Exactly, no gmetad.
https://github.com/rhq-project/rhq-metrics/blob/master/clients/ptranslator/src/main/java/org/rhq/metrics/clients/ptrans/ganglia/UdpGangliaDecoder.java is decoding the packets from gmond
I should perhaps have added in the blog post that the green-dashed box is rhq-metrics. Will update the post.
Post a Comment