Tuesday, July 29, 2014

Java Forum Stuttgart 2014

tl;dr: Great as every year.

On July 17th, I attended the Java Forum Stuttgart conference, a one day conference with 49 talks in 7 tracks. This years edition marked a new record with 1600 attendees from all over Germany (and probably also from other countries).

Conference itself was awesome as every year with good talks and good food, 30+ exhibition stands and some free beer at the end :) The conference is organized by the Java User Group Stuttgart e.V. and as such is a non-profit event, which for sure partially explains the success.

I was lucky that my talk "Slim Fast" about a "diet for the application memory footprint" was accepted and I was positively surprised that the room was full despite the fact that the abstract did not contain any buzzwords like IoT or Cloud :-) (German) slides are attached to the above link, JUG Stuttgart now also has posted some pictures.

JUG Stuttgart also posted more pictures of sessions and also some general impression on their G+ page<./p>

Other talks that I've attended were:

"Java FX everywhere" by Gerrit Grunwald, who showed an app that he wrote for the desktop with Java FX and then ported over to devices like Raspberry PI, Android, iPad etc.

"Android ist anders - Android dependency management". This was about dependency management for Android - and for all those that blame Maven hell, check this out and you will like plain Maven afterwards :)

"IBM BlueMix". In this talk a guy from IBM explained a bit what BlueMix is and then demoed creating an app from a simple template and deploying it to a live BlueMix server in the cloud.

"TypeScript - JavaScript für Java Entwickler". Kai Toedter showed the TypeScript language as an alternative to JavaScript. While I was a bit skeptical about "yet another language", the talk totally made me change my mind and I can really see it as good alternative for future UI-work.





Sunday, June 15, 2014

RHQ-Metrics and Grafana (updated)

As you may know I am currently working on RHQ-Metrics, a time series database with some charting extensions that will be embeddable into Angular apps.

One of the goals of the project is also to make the database also available for other consumers that may have other input sources deployed as well as their own graphing.

Over the weekend I looked a bit at Grafana which looks quite nice, so I started to combine RHQ-metrics with Grafana.

I did not (immediately) find a description of the data format between Grafana and Graphite, but saw that InfluxDB is supported, so I fired up WireShark and was able to write an endpoint for RHQ-Metrics that supports listing of metrics and retrieving of data.

The overall setup looks like this:

RHQ metrics grafana setup
Setup used

A Ganglia gmond emits data over IP multicast, which is received with the protocol translator client, ptrans. This translates the data into requests that are pushed into the RHQ-metrics system (shown as the dotted box) and there stored in the Cassandra backend.

Grafana then talks to the RHQ-Metrics server and fetches the data to be displayed.
For this to work, I had to comment out the graphiteUrl in conf.js and use the influx backend like this:

 datasources: {
influx: {
default: true,
type: 'influxdb',
username: 'test',
password: 'test',
url: 'http://localhost:8080/rhq-metrics/influx',
}
},

As you can see, the URL points to the RHQ-Metrics server.

The code for the new handler is still "very rough" and thus not yet in the normal source repository. To use it, you can clone my version of the RHQ-Metrics repository and then run start.sh in the root directory to get the RHQ-Metrics server (with the built-in memory backend) running.

Update

I have now added code (still in my repository) to support some of the aggregating functions of Influx like min, max, mean, count and sum. The following shows a chart that uses those functions:

Wednesday, May 07, 2014

RHQ 4.11 released


I am proud to announce the immediate availability of RHQ 4.11.

As always this release contains new features and bug fixes.

Notable changes are:

  • Plugins can now define plugin specific DynaGroup expressions, which can even be auto-activated
  • Further agent footprint reduction for jmx-clients (especially if the agent runs on jdk 1.6)
  • Much improved agent install via ssh from the RHQ UI.
  • Support for Oracle 12 as backend database
  • New login screen that follows patternfly.org

As always RHQ is available for download in form of a zip archive. If you want to try out RHQ without too much setup, you can also use a pre-created Docker image
from https://index.docker.io/u/gkhachik/rhq-fedora.20/ (the link contains setup information).

Please consult the release notes for further details and a download link.

If you only want to have a quick look, you can also consult a
running RHQ 4.11 instance that we have set up. User/Pass are guest / rhqguest

Special thanks goes to

  • Elias Ross
  • Michael Burman

for their code contributions for this release.

My first Netty-based server

Netty logo
 

For our rhq-metrics project (and actually for RHQ proper) I wanted to be able to receive syslog messages, parse them and in case of content in the form of


type=metric thread.count=11 thread.active=1 heap.permgen.size=20040000

forward the individual key-value pairs (e.g. thread.count=11) to a rest endpoint for further processing.

After I started with the usual plain Java ServerSocket.. code I realized that this may become cumbersome and that this would be the perfect opportunity to check out Netty. Luckily I already had an Early Access copy of Netty in Action lying around and was ready to start.

Writing the first server that listens on TCP port 5140 to receive messages was easy and modifying /etc/syslog.conf to forward messages to that port as well.

Unfortunately I did not get any message.

I turned out that on OS/X syslog message forwarding is only UDP. So I sat down, converted the code to use UDP and was sort of happy. I was also able to create the backend connection to the REST server, but here I was missing the piece on how to "forward" the incoming data to the rest-client connection.

Luckily my colleague Norman Maurer, one of the Netty gurus helped me a lot and so I got that going.

After this worked, I continued and added a TCP server too, so that it is now possible to receive messages over TCP and UDP.

The (current) final setup looks like this:

Overviw of my Netty app

I need two different decoders here, as for TCP the (payload) data received is directly contained in a ByteBuf while for UDP it is inside a DatagramPacker and needs to be pulled out into a ByteBuf for further decoding.

One larger issue I had during development (aside from wrapping my head around "The Netty Way" of doing things) was that I wrote a Decoder like


public class UdpSyslogEventDecoder extends
MessageToMessageDecoder<DatagramPacket>{

protected void decode(ChannelHandlerContext ctx, DatagramPacket msg, List<Object> out) throws Exception {

and it never got called by Netty. I could see that it was instantiated when the pipeline was set up, but for incoming messages it was just bypassed.

After I added a method


public boolean acceptInboundMessage(Object msg) throws Exception {
return true;
}

my decoder was called, but Netty was throwing ClassCastExceptions, but they were pretty helpful, as it turned out that the IDE automatically imported java.net.DatagramPacket, which is not what Netty wants here. As Netty checks for the signature of the decode() and this did not match, Netty just ignored the decoder.

You can find the source of this server on GitHub.

Wednesday, April 30, 2014

Aerogear UPS alert plugin for RHQ


 

As already indicated, Matze Wessendorf and I sat together at Red Hat Summit and have created an alert sender plugin for RHQ that uses the unified push server from Aerogear to send alerts as push notifications to administrator phones.

For this to work you not only need to install the alert-sender plugin on RHQ side, but also have an appropriate application installed on your handset.

After you have installed the plugin in RHQ you need to go to Administration->Server Plugins and select the Alert:UPS entry. In the configuration settings you need to provide the host name where the push server lives, the secret to talk to it and also the id of your deployed handset app:

Bildschirmfoto 2014 04 30 um 09 24 53

After this is done, you can select this alert sender when you set up new notifications as with other senders:

Bildschirmfoto 2014 04 30 um 09 26 08

Right now this does not have any alert-specific settings, but I guess that in the future one could perhaps set the alert sound here for iOS devices or similar.

And after all this is done, the app is deployed and your handset is on, you could receive a push message just like this:

Screenshot

Screenshot of our first alert sent

If you want to have a look at the sender plugin, you can go to the RHQ repository on GitHub.

Saturday, April 26, 2014

RHQ and JBoss ON at Summit 2014

I was very fortunate to be able to attend this years Red Hat Summit in San Francisco.

Right before Summit, the accompanying DevNation conference already started on Sunday with some pretty interesting talks - among others about Fabric8 and Hawt.io.

As both Summit and DevNation happened at Moscone center, it was pretty easy to mix and match sessions. Dev Nation also featured a "hack area": some tables that had huge power strips in the middle to easily connect laptop and smartphone chargers. And usually the table were not as empty as on the next image:

I also met at one of those tables with Matze Wessendorf from the Aerogears team and we developed together an Alert Sender plugin for RHQ, that uses the Unified Push Sender to directly bring alerts from RHQ to the lock screen of the admins phones. The plugin only exists in private git at the moment. We need to do some polishing and remove some passwords before pushing to rhq git.

Summit presentation
Thomas Segismont and I did a presentation "What is new with JBoss ON" and we also had a BOF session later on.
The presentation ran well with around 60 attendees. Most of the audience already knew JBoss ON.
Of those around 2/3 are running 3.1, a few 3.2 and one person was even on 3.0 (unfortunately I was not able to find him later to find out why)

The slides of the presentation are already available online; unfortunately they do not cover the two live demos from Thomas, showing how to add a new Storage Node and how to make use of the new bundle permissions that were introduced in JBoss ON 3.2 (and also RHQ).

IMG 4188

Thomas at work and me doing a bit of talking.

BOF

The BOF/Meet and Greet session later was also quite well attended.
People had good questions and ideas and we were talking a bit about roadmap and also about some vision wrt. JBoss ON 4. Unfortunately we got kicked out the room much too early.

OCSystems
Thomas and I also had the luck to be invited by OCSystems, the makers of RTI that also runs on top of JBoss ON for a dinner at the Waterfront restaurant, from where we had an excellent view on to the bay bridge:

IMG 4163
Front row: Alan Santos, myself, Thomas Segismont

Middle: Tobias Hartwig, Bill Critch, all Red Hat

Behind: Steve North, Georgia Ferretti, both OCSystems

Hackathon

And last but not least at the Dev Nation Hackathon, Team "RHQ" won the 2nd prize with some "home automation":
a RaspberryPI running the agent had an LED blinking and we had also one of the MBed boards (Those are developer boards a little like Arduino but with an ARM cpu and sensors + an LCD on board)
connected to read in sensor data; unfortunately I lost some time at the start of the hackathon,
so we could not really use that info in the plugin; 1/2h more time and ... :-)
For the 2nd half of this, Thomas has built a Cordova application for his Android phone and showed how to receive push messages that were sent when alerts are fired with the brand new aerogear-ups alert sender plugin mentioned above.

I should have taken a photo of all the wiring we had set up, but totally forgot
about it.

I will re-create / finish this demo and blog about it


Thursday, February 13, 2014

Running the RHQ-agent on a Raspberry PI [updated]

I finally got a Raspberry Pi too. After the hurdles of initial installation, got it hooked up to my LAN and of course I had to install an RHQ agent on it.

And it turned out that this was dead simple, as the Pi already has Java 1.7 installed (in the Raspbian Wheezy distro that I am using). Thus it was only a matter of laying down the rhq-agent, and starting it the usual way.

Now there was one caveat: the agent did not find any file systems or network interfaces etc. This is due to the fact that there is no native library for Sigar on arm v6 cpus supplied with the agent.

I cloned Sigar from its git repository, changed into the 1.6 branch and built that library myself.

Now after dropping libsigar-arm-linux.so into the agent's lib/ directory, the native library is available and on agent restart all the native stuff could be found.

Screenshot of platform details in RHQ
(Platform details in RHQ)

If you don't want to compile that library yourself, you can take my version from https://sourceforge.net/projects/rhq/files/rhq/misc/.

I will try to get that library into the upcoming RHQ 4.10 release, but can't promise anything.

Update

If you run the agent from current master (or upcoming RHQ 4.10), you can configure a list of plugins to be enabled, so that the agent only uses these plugins (and thus uses less memory and starts faster).
This property can be found in the file conf/agent-configuration.xml:


<entry key="rhq.agent.plugins.enabled" value="Platforms,JMX,RHQAgent"/>

The entries are a comma separated list of plugin (short) names. To determine those, you can run
plugins info at the agent command prompt:


> plugins info
Details of the plugins that are currently installed:

rhq-agent-plugin-4.10.0-SNAPSHOT.jar
Plugin Name: RHQAgent
Display Name: RHQ Agent
Last Updated: 14. Februar 2014 11:03:35 MEZ
File Size: 51.558 bytes
MD5 Hashcode: f7eb7577af667ee4883437230e4b2d8c
[...]

Summary of installed plugins:
[RHQAgent, Platforms, JMX]

The short names are the ones encoded as "Plugin Name" and which are also shown on the summary line. There has actually been a property to disable unwanted plugins for a longer time, but just enabling the ones needed is probably easier.


The other thing you should do it to remove the -Xms setting in the rhq-agent.sh script- the default of a 64MB minimum heap is just too large here.

With those 3 plugins above and the removed Xms setting, my agent has a committed heap of ~14MB and a used heap of ~11MB. A dump is/was 4MB in size.

P.S.: “Raspberry Pi" is a trademark of the Raspberry Pi Foundation

Saturday, December 21, 2013

Money for nothing and memory for free: Java 7u40

I recently started playing with tools like MAT and also reading up about memory usage, performance (tuning) etc.

One of the interesting blog posts I came by was a comparison of JVM versions, that mentions that in 7u40, the default on how the JVM allocates the backing memory for ArrayList and HashMap has changed.

When you do

List<Foo> bla = new ArrayList<Foo>();

the VM will allocate memory for the base object and in vm versions prior to 7u40 also 10* 4bytes for the references to Foo objects, which (with alignment and so on) accounts to 80 bytes per empty ArrayList.

In 7u40, the array for the references is no longer eagerly allocated so that an empty ArrayList now only occupies the base 24 bytes, which account for 56bytes saving.

You may say so what, that are only 56 bytes, but remember that memory not allocated does not fill the heap, does not need to be garbage collected and also does not require memory bandwidth for the initial nulling out.

And so often an empty list/map does not come alone as in a case like this (output from MAT):

Screenshot 2013 11 15 17 51 25

With 225k empty ArrayLists, 56 bytes matter: 225k*56 bytes are 12 MB that you would save just by switching from a JVM pre 7u40 to 7u40 or later without a code change (of course not allocating those lists at all would save even more).

The situation for HashMap is similar: before 7u40 an empty one uses 136 bytes while in 7u40+ is uses 48, a saving of 88 bytes per empty HashMap or with the 235k empty HashMaps of the above example a saving of 20 MB.

Another (older) change is that in 7u06 the minimum size of String objects also has been reduced by 8 bytes, which matters a lot if you have millions of Strings in your VM.

Of course (as I've mentioned already) if you have access to the source code and can prevent the allocation of those empty objects altogether you would save even more memory.

Saturday, November 02, 2013

Back from OneDayTalk

[ I should have already written this a bit earlier, but I had some trouble with my left knee and had to go through some surgery (which went well). ]

As in the previous years I have been invited to give a talk at the OneDayConference in Munich. This year it was in a new location in a suburb of Munich called Germering. Getting there was easy for me, as there is a S-Bahn stop almost in front of the conference location.

The new location featured more and larger rooms and especially an area to sit down between talks or during lunch time. As in the last years the conference featured three parallel tracks.

As I said before I like that conference as everything is like a big family event with the organizers and also the presenters which featured many JBoss Colleagues; while I wrote that Ray would be there, Andrew Rubinger replaced him. The only talk that I really attended was the Wildfly one from Harald Pehl, which was full house. In the remaining part of the conference I talked to various attendees and colleagues from Jan Wildeboer to Gavin King and Heiko Braun. Heiko gave me an introduction about his (and Harald's) work to generate UIs from descriptors (which they use in the Wildfly console), which looks very interesting and where I think we could use some of that inside of RHQ to create "wizards" for several types of target resources.

In my talk, which was in the last slot, I had around 30 attendees (which was around 1/3 of the attendees still present). To my surprise I found out that the large majority did not yet know or use RHQ, so I had to switch from my original agenda and gave a brief introduction into RHQ first. Next I talked about the recent changes in RHQ and tried to gather feedback for future use case, but that was of course harder with attendees not knowing too much about RHQ. So much for "know your audience".

How do others try to find out their audience when the only thing they know is "This conference is all about JBoss projects" ?

You can find my slides in AsciiDoc format on GitHub that you can render via AsciiDoctor to html presentation.

Friday, November 01, 2013

Beware of the empty tag

I started playing with AngularJS lately and made some progress with RHQ as backend data source (via the REST api). Now I wanted to write a custom directive and use it like this:


<rhq-resource-picker/>
<div id="graph" style="padding: 10px"></div>

Now the very odd thing (for me) was that in the DOM this looked like this:


<rhq-resource-picker>
<div id="graph" style="padding: 10px"></div>
</rhq-resource-picker>

My custom directive wrapped around the <div> tag.

To fix this I had to turn my tag in a series of opening and closing tags instead of the empty one:


<rhq-resource-picker></rhq-resource-picker>
<div id="graph" style="padding: 10px"></div>

And in fact it turns out that a few more tags like the popular <div> tag show the same property of not allowing to write an empty tag, but requiring a tag pair.

Thursday, October 10, 2013

OneDayTalk conference in Munich

I will as in the previous years be at this years JBoss OneDayTalk conference in Munich and talk about recent and future developments in RHQ.

OneDayTalk is a pretty nice little conference organized by the JBoss User Group Munich, that offers three tracks with 6 sessions each, where I usually have the problem that I can't divide myself into three to visit them all. And for 99 Euro you also get some good food and many opportunities to meet myself and other JBossians like Eric Schabell, Gavin King, Emanuel Muckenhuber, Heiko Braun or Ray Ploski. The conference web site has the full listing of speakers as well as the scheduled program.

Friday, September 13, 2013

More clever builds and tests ?

Hey,

when you have already tried to build RHQ and run all the tests you probably have seen that this may take a huge amount of time, which is fine for the first build, but later when you e.g. change a typo in the as4-plugin, you don’t want GWT being compiled again.
Of course as a human developer it is relatively easy to just go into the right folder and only run the build there.

Now on build automation like Jenkins this is less easy, which is why I am writing
this post.

What I want to have is similar to

  • if a class in GWT has changed, only re-build GWT

  • if a class in a plugin has changed, only rebuild and test that plugin
    (and perhaps dependent plugins like hibernate on top of jmx)

  • if a class in enterprise changes, only build and test enterprise

  • if a class in server/jar changes, only rebuild that and run server-itests

  • if a class in core changes, rebuild everything (there may be more fine grained rules as e.g. a change in core/plugin-container does not require compiling GWT again)

This is probably a bit abbreviated, but you get the idea.

What I can imagine is that we compile the whole project (actually we may even do incremental compiles to get build times further down and may also only go into a specific module (and deps) and just build those).
And then instead of running mvn install we run mvn install -DskipTests
and afterwards analyze what has changed, throw the above rules
at it and only run the tests in the respective module(s).

We could perhaps have a little DSL like (which would live in the root of the project tree and be in git)

    rules: {
rule1: {
if :modules/enterprise/,
then: {
compile: modules/enterprise,
skip: modules/enterprise/gui/coregui,
test: ["modules/enterprise"]
}
},
rule2: {
if: modules/plugins/as7-plugin,
then: {
compile:[ modules/plugins/as7-plugin,
modules/integration-tests/as7-plugin],
test: [ modules/plugins/as7-plugin,
modules/integration-tests/as7-plugin]
}
}

And have them evaluated in a clever way by some maven build helper
that parses that file and also the list of changes since the last build to
figure out what needs testing.

We can still run a full build with everything to make sure that we don’t loose coverage
by those abbreviations

There may be build systems like gradle that have this capability built in; I think for maven
this requires some additional tooling

QUESTIONS:

  • Are there any "canned" solutions available?

  • Has anyone already done something like "partial tests" (and how)?

  • Anyone knows of maven plugins that can help here?

  • Anyone interested in getting that going? This is for sure not only interesting for RHQ but for a lot of projects

Having such an infrastructure will also help us in the future to better integrate
external patches, as faster builds / tests can allow to automatically test those and
"pre-approve" them.




Wednesday, September 11, 2013

RHQ 4.9 released

It is a pleasure for me to announce on behalf of the RHQ development team the immediate availability of RHQ 4.9

Features

Some of the new features since RHQ 4.8 are:

Be sure to read the release notes and the installation documents.

Note

For security reasons we have made changes to the installation in the sense that there is no more default password for the rhqadmin super user. Also the default bind address of 0.0.0.0 has been removed. You need to set them before starting the installation.

If you are upgrading from RHQ 4.8 you need to run a script to remove the native components from Cassandrabefore the upgrade. Otherwise RHQ 4.9 will fail to start.

Thanks

Special thanks goes to

  • Elias Ross

  • Jérémie Lagarde

  • Michael Burman

for their code contributions for this release and to Stian Lund for his repeated testing of the new graphs implementation.

Downloads

As usual you can download the release from the RHQ-project downloads on SourceForge




Friday, July 26, 2013

Custom Deserializer in Jackson and validation

tl;dr: it is important to add input validation to custom json deserializers in Jackson.

In RHQ we make use of Json parsing in a few places - be it directly in the as7/Wildfly plugin, be it in the REST-api indirectly via RESTEasy 2.3.5, that already does the heavy lifting.

Now we have a bean Link that looks like

public class Link {
String rel;
String href;
}

The standard way for serializing this is

{ "rel":"edit", "href":"http://acme.org" }

As we need a different format I have written a custom serializer and attached it on the class.

@JsonSerialize(using = LinkSerializer.class)
@JsonDeserialize(using = LinkDeserializer.class)
@Produces({"application/json","application/xml"})
public class Link {

private String rel;
private String href;

This custom format looks like:

{
"edit": {
"href": "http://acme.org"
}
}

As a client can also send links, some custom deserialization needs to happen. A first cut of the deserializer looked like this and worked well:

public class LinkDeserializer extends JsonDeserializer<Link>{

@Override
public Link deserialize(JsonParser jp,
DeserializationContext ctxt) throws IOException
{
String tmp = jp.getText(); // {
jp.nextToken(); // skip over { to the rel
String rel = jp.getText();
jp.nextToken(); // skip over {
[…]

Link link = new Link(rel,href);

return link;
}

Now what happened the other day was that in some tests I was sending data and our server blew up horribly. Memory usage grew, the garbage collector took huge amounts of cpu time and the call eventually terminated with an OutOfMemoryException.

After some investigation I found that the client did not send the Link object in our special format, but in the original format that I showed first. Further investigation showed that in fact the LinkDeserializer was consuming the tokens from the stream as seen above and then also swallowing subsequent tokens from the input. So when it returned, the whole parser was in a bad shape and was then trying to copy large arrays around until we saw the OOME.

After I got this, I changed the implementation to add validation and to bail out early on invalid input, so that the parser won’t get into bad shape on invalid input:

    public Link deserialize(JsonParser jp,
DeserializationContext ctxt) throws IOException
{

String tmp = jp.getText(); // {
validate(jp, tmp,"{");
jp.nextToken(); // skip over { to the rel
String rel = jp.getText();
validateText(jp, rel);
jp.nextToken(); // skip over {
tmp = jp.getText();
validate(jp, tmp,"{");
[…]

Those validate*() then simply compare the token with the passed expected value and throw an Exception on unexpected input:

    private void validate(JsonParser jsonParser, String input,
String expected) throws JsonProcessingException {
if (!input.equals(expected)) {
throw new JsonParseException("Unexpected token: " + input,
jsonParser.getTokenLocation());
}
}

The validation can perhaps be improved more, but you get the idea.




Monday, July 15, 2013

JavaFX UI for the RHQ plugin generator (updated)

In RHQ we have a tool to generate initial plugin skeletons, the plugin generator. This is a standalone tool, that you can use to get going with plugin development.
This is for example described in the "How to write a plugin" article.

Now my colleague Simeon always wanted a graphical frontend (he is very much in favor of graphical frontends, while I am one of those old Unix-neckbeards :). Inspired from this years Java Forum Stuttgart I sat down on the weekend and played a bit with JavaFX. The result is a first cut of UI frontend for the generator:

UI for the plugin generator

On top you see a message bar, that shows field validation issues, the middle is the part where you enter the settings and at the bottom you get a short description for each property.

I've pushed a first cut, that is not yet complete, but you will get the idea. Help is always appreciated.
Code is in RHQ git under modules/helpers/pluginGen.

[update]

I continued working a bit on the generator and have added a possibility to give a directory that contains classes that have been annotated with the annotations from the org.rhq.helpers:rhq-pluginAnnotations module.

Screenshot scan-for-annotations

A class could look like this


public class FooBean {

@Metric(description = "How often was this bean invoked",
displayType = DisplayType.SUMMARY,
measurementType = MeasurementType.DYNAMIC,
units = Units.SECONDS)
int invocationCount;

@Metric(description = "Just a foo", dataType = DataType.TRAIT)
String lastCommand;

@Operation(description = "Increase the invocation count")
public int increaseCounter() {
invocationCount++;
return invocationCount;
}

@Operation(description = "Decrease the counter")
public void decreaseCounter(@Parameter(description
= "How much to decrease?", name = "by") int by) {
invocationCount -= by;
}

And the generator would then create the respective <metric> and <operation> elements in the plugin descriptor - in this case you don't need to select the two "Has Metrics/Operations" flags above.

Now this work is still not finished. And in fact it would be good to find a common set of annotations for a more broader scope of projects.

Wednesday, July 03, 2013

Using Asciidoc with MarsEdit - first cut

With Asciidoc becoming more popular due to Asciidoctor I started authoring documents in Asciidoc and thought it would be a good idea to use that for my blogging as well (in the long run I may set up my blog in Awestruct, but for now Blogger has to do.

Usually I am using
MarsEdit to write my posts, as it is just convenient for me. MarsEdit now allows to write custom text filters since the recently released version 3.6. There are a few such filters provided like for Markdown or Textile, so that you can write blog posts inside MarsEdit in those markup languages and still be able to post to e.g. Blogger, which requires html.

When I saw the filters announced, I thought, that should be possible with Asciidoc as well.

So basically to achieve this, you need to create a directory

~/Library/Application Support/MarsEdit/TextFilters/Asciidoc_0.0.1

and then create a file Asciidoc.rb with the following content:

#!/usr/bin/ruby

require 'rubygems'
require 'asciidoctor'

input = $stdin.read

puts Asciidoctor.render(input)

Make that file executable and install the AsciiDoctor gem.

Then (re-start) MarsEdit and select Asciidoc as Preview Text filter in the connection settings of the blogger account ( see Mars Edit per Blog Settings ). Then click on the "Posting" tab and click on "Apply preview filter before posting".

Unfortunately I have not yet found out how to include (remote) images

NOTE

Not all AsciiDoc markups make sense, as AsciiDoctor usually uses some CSS, that may not be present on the target blog system. It may be not too hard though to create a special "blogger" backend, that uses
blogger css for that or to tell blogger to accept the AsciiDoctor css files

This screenshot shows MarsEdit with source and preview.

I hope this little post makes sense and encourages people to experiment with it in order to make AsciiDoc a real alternative to




Saturday, June 29, 2013

New Android tooling with gradle: Order matters

I am trying to convert RHQPocket over to use the new Android build system with Gradle.
The documentation is comprehensive, but as always does not exactly match what I have in front of me.

After moving the sources around I started generating a build.gradle file. I added my external libs to the dependencies but the build did not succeed with a lot of trial and error.

At the end with the help of googling and StackOverflow I found out that order matters a lot in the build file:

First comes the section about plugins for the build system itself:

buildscript {
repositories {
mavenCentral()
}

dependencies {
classpath 'com.android.tools.build:gradle:0.4.2'
}
}

Settings herein only apply to the build system, but not to the project to be built.

After this we can use those plugins (the android one)

apply plugin: 'android'
apply plugin: 'idea'


And only when those plugins are loaded, we can start talking about the project itself and add dependencies

repositories {
mavenCentral()
}
dependencies {
compile 'org.codehaus.jackson:jackson-core-asl:1.9.12'
compile 'org.codehaus.jackson:jackson-mapper-asl:1.9.12'
}

android {
compileSdkVersion 17
buildToolsVersion "17.0"
}


I guess for someone experienced with Gradle, this is no news, but it took me quite some time to figure out especially as the documentation mentions all the parts but not on a larger example.

The file layout now looks like this:

Bildschirmfoto 2013 06 29 um 22 07 02


One caveat is that the external libraries are not automagically yet loaded into the IDE - one still has to manually add them in the project structure wizard.

The last thing is then to mark build/source/r/debug as source directory to also get completion for the R.* entries.

Tuesday, June 25, 2013

RHQ 4.8 released



RHQ 4.8 has been released with another batch of new features and bug fixes.
Major changes since RHQ 4.7 are
  • New storage backend for metrics based on Cassandra:

    Numerical metric data is now stored in a Cassandra cluster to improve scalability for larger deployments with many metrics.
    Architecture overview cassandra color

    The intention is to move more mass data into Cassandra into the future (events, calltime data etc). We just had to start somewhere. If you are upgrading from an earlier version of RHQ, a utility to migrate metrics gathered is provided.
  • More improvements to the new charts
    Bildschirmfoto 2013 06 25 um 10 57 07

    The selection of the time period has been changed:
    • There is now a button bar at the top to quickly select the timeframe to display
    • If you hover with the mouse between two "bars" you will see a little crosshair, that starts the
      selection mode - drag it over a series of bars to zoom in. Mike Thompson has created a YouTube video that explains this.
  • New installation process:

    With the above change of the backend storage, we have changed the installation process again to reduce the complexity to install RHQ. The new process basically goes:
    • unzip rhq-server-4.8.0.zip
    • cd rhq-server-4.8.0
    • edit bin/rhq-server.properties
    • bin/rhqctl install (--storage-data-root-dir=<some directory>

    For upgrades the process is very similar: you run unzip the new binaries and then run bin/rhqctl upgrade --from-server-dir=<old server>
    Please consult the installation and upgrade guides before starting.
  • REST-api is now (mostly) stable and supports paging on (some) resources. The documentation is also up on SourceForge and the rhq-samples project has a special section on examples for the REST-api.


As always there have been many more smaller improvements and bug fixes - many thanks to everyone who has contributed via bug report, comments, discussions on #rhq and/or via code contributions.

Please check the full release notes for details. They also contain a list of commits.

RHQ is an extensible tool to monitory your infrastructure of machines and applications, alert operators on user defined conditions, configure resources and run operations on them from a central web-based UI. Other ways of communicating with RHQ include a command line interface and a REST-api.

You can download the release from source forge.


As mentioned above, the old installer is gone, so make sure to read
the wiki document describing how to use the new installer.

Maven artifacts are available from the JBoss Nexus repository and should also be available from Central.


Please give us feedback, be it in Bugzilla, Mailing lists or the forum. Or just join us on IRC at irc.freenode.net/#rhq.

And as I have been asked recently: yes we are happy to accept code contributions -- be it for the RHQ core, as well as plugins or for the samples project. Also if you e.g. have written a plugin, share a pointer to it, which we can then share on the wiki etc.

Thursday, May 23, 2013

Support for paging in the RHQ REST-api (updated)

I have just pushed some changes to RHQ master that implement paging for (most) of the REST-endpoints that return collections of data.



If you remember, I was asking the other day if there is a "standard" way to do this. Basically there are two
"big options":
  1. Put paging info in the Http-Header
  2. Put paging info in the body of the message


While I think paging information are meta data that should not "pollute" the body, I understand the arguments from the JavaScript side that says that they don't easily have access to the http headers
within AJAX requests. So what I have now done is to implement both:
  1. Paging in the http header: this is the standard way that you get if you just request the "normal" media types of application/xml or application/json (output from running RestAssured):

    [update]
    My colleague Libor pointed out that the links do not match with format from RFC 5988 Web Linking, which is now fixed.
    [/update]

    Request method: GET
    Request path: http://localhost:7080/rest/resource
    Query params: page=1
    ps=2
    category=service
    Headers: Content-Type=*/*
    Accept=application/json
    HTTP/1.1 200 OK
    Server=Apache-Coyote/1.1
    Pragma=No-cache
    Cache-Control=no-cache
    Expires=Thu, 01 Jan 1970 01:00:00 CET
    Link=<http://localhost:7080/rest/resource?ps=2&category=service&page=2>; rel="next"
    Link=<http://localhost:7080/rest/resource?ps=2&category=service&page=0>; rel="prev"
    Link=<http://localhost:7080/rest/resource?ps=2&category=service&page=152>; rel="last"
    Link=<http://localhost:7080/rest/resource?page=1&ps=2&category=service>; rel="current"
    Content-Encoding=gzip
    X-collection-size=305
    Content-Type=application/json
    Transfer-Encoding=chunked
    Date=Thu, 23 May 2013 07:57:38 GMT

    [
    {
    "resourceName": "/",
    "resourceId": 10041,
    "typeName": "File System",
    "typeId": 10013,
    …..
  2. Paging as part of the body - there the "real collection" is wrapped inside an object that also contains paging meta data as well as the paging links. To request this representation, a media type of application/vnd.rhq.wrapped+json needs to be used (and this is only available with JSON at the moment):

    Request method: GET
    Request path: http://localhost:7080/rest/resource
    Query params: page=1
    ps=2
    category=service
    Headers: Content-Type=*/*
    Accept=application/vnd.rhq.wrapped+json
    Cookies:
    Body:
    HTTP/1.1 200 OK
    Server=Apache-Coyote/1.1
    Pragma=No-cache
    Cache-Control=no-cache
    Expires=Thu, 01 Jan 1970 01:00:00 CET
    Content-Encoding=gzip
    Content-Type=application/vnd.rhq.wrapped+json
    Transfer-Encoding=chunked
    Date=Thu, 23 May 2013 07:57:40 GMT

    {
    "data": [
    {
    "resourceName": "/",
    "resourceId": 10041,
    "typeName": "File System",
    "typeId": 10013,

    ],
    "pageSize": 2,
    "currentPage": 1,
    "lastPage": 152,
    "totalSize": 305,
    "links": [
    {
    "next": {
    "href": "http://localhost:7080/rest/resource?ps=2&category=service&page=2"
    }
    },

    }

Please try this if you can. We want to get that into a "finished" state (as for the whole REST-Api) for RHQ 4.8

Thursday, May 16, 2013

Creating a delegating login module (for JBoss EAP 6.1 )



[ If you only want to see code, just scroll down ]

Motivation


In RHQ we had a need for a security domain that can be used to secure the REST-api and its web-app via container managed security. In the past I had just used the classical DatabaseServerLoginModule to authenticate against the database.

Now does RHQ also allow to have users in LDAP directories, which were not covered by above module. I had two options to start with:
  • Copy the LDAP login modules into the security domain for REST
  • Use the security domain for the REST-api that is already used for UI and CLI


The latter option was of course favorable to prevent code duplication, so I went that route. And failed.

I failed because RHQ was on startup dropping and re-creating the security domain and the server was detecting this and complaining that the security domain referenced from the rhq-rest.war was all of a sudden gone.

So next try: don't re-create the domain on startup and only add/remove the ldap-login modules (I am talking about modules, because we actually have two that we need).

This also did not work as expected:
  • The underlying AS sometimes went into reload needed mode and did not apply the changes
  • When the ldap modules were removed, the principals from them were still cached
  • Flushing the cache did not work and the server went into reload-needed mode


So what I did now is to implement a login module for the rest-security-domain that just delegates to another one for authentication and then adds roles on success.

This way the rhq-rest.war has a fixed reference to that rest-security-domain and the other security domain could just be handled as before.

Implementation



Let's start with the snippet from the standalone.xml file describing the security domain and parametrizing the module


<security-domain name="RHQRESTSecurityDomain" cache-type="default">
<authentication>
<login-module code="org.rhq.enterprise.server.core.jaas.DelegatingLoginModule" flag="sufficient">
<module-option name="delegateTo" value="RHQUserSecurityDomain"/>
<module-option name="roles" value="rest-user"/>
</login-module>
</authentication>
</security-domain>


So this definition sets up a security domain RHQRESTSecurityDomain which uses the DelegatingLoginModule that I will describe in a moment. There are two parameters passed:
  • delegateTo: name of the other domain to authenticate the user
  • roles: a comma separated list of modules to add to the principal (and which are needed in the security-constraint section of web.xml


For the code I don't show the full listing; you can find it in git.

To make our lives easier we don't implement all functionality on our own, but extend
the already existing UsernamePasswordLoginModule and only override
certain methods.


public class DelegatingLoginModule extends UsernamePasswordLoginModule {


First we initialize the module with the passed options and create a new LoginContext with
the domain we delegate to:

@Override
public void initialize(Subject subject, CallbackHandler callbackHandler,
Map<String, ?> sharedState,
Map<String, ?> options)
{
super.initialize(subject, callbackHandler, sharedState, options);

/* This is the login context (=security domain) we want to delegate to */
String delegateTo = (String) options.get("delegateTo");

/* Now create the context for later use */
try {
loginContext = new LoginContext(delegateTo, new DelegateCallbackHandler());
} catch (LoginException e) {
log.warn("Initialize failed : " + e.getMessage());
}



The interesting part is the login() method where we get the username / password and store it for later, then we try to log into the delegate domain and if that succeeded we tell super that we had success, so that it can do its magic.


@Override
public boolean login() throws LoginException {
try {
// Get the username / password the user entred and save if for later use
usernamePassword = super.getUsernameAndPassword();

// Try to log in via the delegate
loginContext.login();

// login was success, so we can continue
identity = createIdentity(usernamePassword[0]);
useFirstPass=true;

// This next flag is important. Without it the principal will not be
// propagated
loginOk = true;

the loginOk flag is needed here so that the superclass will call LoginModule.commit() and pick up the principal along with the roles.
Not setting this to true will result in a successful login() but no principal
is attached.


if (debugEnabled) {
log.debug("Login ok for " + usernamePassword[0]);
}

return true;
} catch (Exception e) {
if (debugEnabled) {
LOG.debug("Login failed for : " + usernamePassword[0] + ": " + e.getMessage());
}
loginOk = false;
return false;
}
}


After success, super will call into the next two methods to obtain the principal and its roles:

@Override
protected Principal getIdentity() {
return identity;
}


@Override
protected Group[] getRoleSets() throws LoginException {

SimpleGroup roles = new SimpleGroup("Roles");

for (String role : rolesList ) {
roles.addMember( new SimplePrincipal(role));
}
Group[] roleSets = { roles };
return roleSets;
}


And now the last part is the Callback handler that the other domain that we delegate to will use to obtain the credentials from us. It is the classical JAAS login callback handler. One thing that first totally confused me was that this handler was called several times during login and I thought it is buggy. But in fact the number of times it is called corresponds to the number of login modules configured in the RHQUserSecurityDomain.


private class DelegateCallbackHandler implements CallbackHandler {
@Override
public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {

for (Callback cb : callbacks) {
if (cb instanceof NameCallback) {
NameCallback nc = (NameCallback) cb;
nc.setName(usernamePassword[0]);
}
else if (cb instanceof PasswordCallback) {
PasswordCallback pc = (PasswordCallback) cb;
pc.setPassword(usernamePassword[1].toCharArray());
}
else {
throw new UnsupportedCallbackException(cb,"Callback " + cb + " not supported");
}
}
}
}



Again, the full code is available in the RHQ git repository.

Debugging (in EAP 6.1 alpha or later )



If you write such a login module and it does not work, you want to debug it. Started with the usual means to find out that my login() method was working as expected, but login just failed. I added print statements etc to find out that the getRoleSets() method was never called. But still everything looked ok. I did some googling and found this good wiki page. It is possible to tell a web app to do audit logging


<jboss-web>
<context-root>rest</context-root>
<security-domain>RHQRESTSecurityDomain</security-domain><disable-audit>false</disable-audit>


This flag alone is not enough, as you also need an appropriate logger set up, which is explained on
the wiki page. After enabling this, I saw entries like


16:33:33,918 TRACE [org.jboss.security.audit] (http-/0.0.0.0:7080-1) [Failure]Source=org.jboss.as.web.security.JBossWebRealm;
principal=null;request=[/rest:….

So it became obvious that the login module did not set the principal. Looking at the code in the superclasses then led me to the loginOk flag mentioned above.

Now with everything correctly set up the autit log looks like


22:48:16,889 TRACE [org.jboss.security.audit] (http-/0.0.0.0:7080-1)
[Success]Source=org.jboss.as.web.security.JBossWebRealm;Step=hasRole;
principal=GenericPrincipal[rhqadmin(rest-user,)];
request=[/rest:cookies=null:headers=authorization=user-agent=curl/7.29.0,host=localhost:7080,accept=*/*,][parameters=][attributes=];

So here you see that the principal rhqadmin has logged in and got the role rest-user assigned, which is the one matching in the security-constraint element in web.xml.

Further viewing



I've presented the above as a Hangout on Air. Unfortunately G+ muted me from time to time when I was typing while explaining :-(

After the video was done I got a few more questions that at the end made me rethink the startup phase for the case that the user has a previous version of RHQ installed with LDAP enabled. In this case, the installer will still install the initial DB-only RHQUserSecurityDomain and then in the startup bean
we check if a) LDAP is enabled in system settings and b) if the login-modules are actually present.
If a) matches and they are not present we install them.

This Bugzilla entry also contains more information about this whole story.