The next graph is a visualization with Grafana of the outcome when this sender was running for 1.5 days in a row:
Graphs of JVM memory usageWildFly memory usage
( It is interesting to see how the JVM is fine tuning its memory requirement over time and using less and less memory for this constant workload ).
The following is a visualization of the setup:
Setup
The sender is running as a subsystem inside WildFly and reading metrics from the WildFly management api. The gathered metrics are then pushed via REST to RHQ-Metrics. Of course it is possible to send them to a RHQ-Metrics server that is running on a separate host.
The configuration of the subsystem looks like this:
<subsystem xmlns="urn:org.rhq.metrics:wildflySender:1.0">
<rhqm-server
name="localhost"
enabled="true"
port="8080"
token="0x-deaf-beef"/>
<metric name="non-heap"
path="/core-service=platform-mbean/type=memory"
attribute="non-heap-memory-usage"/>
<metric name="thread-count"
path="/core-service=platform-mbean/type=threading"
attribute="thread-count"/>
</subsystem>
As you see, the path to the DMR resource and the name of the attribute to be monitored as metrics can be given in the configuration.
The implementation is still basic at the moment - you can find the source code in the RHQ-Metrics repository on GitHub. Contributions are very welcome.
Heiko Braun and Harald Pehl are currently working on optimizing the scheduling with individual intervals and possible batching of requests for managed servers in a domain.
Many thanks go to Emmanuel Hugonnet, Kabir Khan and especially Tom Cerar for their help to get me going with writing a subsystem, which was pretty tricky for me. The parsers, the object model and the XML had a big tendency to disagree with each other :-)
Here's the link to the metric scheduler that we intend to contribute: https://github.com/heiko-braun/metric-scheduler
ReplyDelete>As you see, the path to the DMR resource and the name of the attribute to be monitored as metrics can be given in the configuration.
ReplyDeleteHi,
Is there any plan to make this subsystem accessible from applications deployed on the server?
In other words, expose a way for applications to send custom metrics to the same backend through this subsystem?
This would be fantastic
Thanks for the amazing work
Bruno,
ReplyDeletethat is an interesting idea. Basically have some method/api endpoint/local socket that you send the metrics to and the subsystem would deal with forwarding the data to the RHQ Metrics server.
For now I was thinking the app would do that on its own, but I can definitively see beauty in following your approach.
Heiko