The next graph is a visualization with Grafana of the outcome when this sender was running for 1.5 days in a row:
Graphs of JVM memory usageWildFly memory usage
( It is interesting to see how the JVM is fine tuning its memory requirement over time and using less and less memory for this constant workload ).
The following is a visualization of the setup:
Setup
The sender is running as a subsystem inside WildFly and reading metrics from the WildFly management api. The gathered metrics are then pushed via REST to RHQ-Metrics. Of course it is possible to send them to a RHQ-Metrics server that is running on a separate host.
The configuration of the subsystem looks like this:
<subsystem xmlns="urn:org.rhq.metrics:wildflySender:1.0">
<rhqm-server
name="localhost"
enabled="true"
port="8080"
token="0x-deaf-beef"/>
<metric name="non-heap"
path="/core-service=platform-mbean/type=memory"
attribute="non-heap-memory-usage"/>
<metric name="thread-count"
path="/core-service=platform-mbean/type=threading"
attribute="thread-count"/>
</subsystem>
As you see, the path to the DMR resource and the name of the attribute to be monitored as metrics can be given in the configuration.
The implementation is still basic at the moment - you can find the source code in the RHQ-Metrics repository on GitHub. Contributions are very welcome.
Heiko Braun and Harald Pehl are currently working on optimizing the scheduling with individual intervals and possible batching of requests for managed servers in a domain.
Many thanks go to Emmanuel Hugonnet, Kabir Khan and especially Tom Cerar for their help to get me going with writing a subsystem, which was pretty tricky for me. The parsers, the object model and the XML had a big tendency to disagree with each other :-)
