Thursday, May 23, 2013

Support for paging in the RHQ REST-api (updated)

I have just pushed some changes to RHQ master that implement paging for (most) of the REST-endpoints that return collections of data.



If you remember, I was asking the other day if there is a "standard" way to do this. Basically there are two
"big options":
  1. Put paging info in the Http-Header
  2. Put paging info in the body of the message


While I think paging information are meta data that should not "pollute" the body, I understand the arguments from the JavaScript side that says that they don't easily have access to the http headers
within AJAX requests. So what I have now done is to implement both:
  1. Paging in the http header: this is the standard way that you get if you just request the "normal" media types of application/xml or application/json (output from running RestAssured):

    [update]
    My colleague Libor pointed out that the links do not match with format from RFC 5988 Web Linking, which is now fixed.
    [/update]

    Request method: GET
    Request path: http://localhost:7080/rest/resource
    Query params: page=1
    ps=2
    category=service
    Headers: Content-Type=*/*
    Accept=application/json
    HTTP/1.1 200 OK
    Server=Apache-Coyote/1.1
    Pragma=No-cache
    Cache-Control=no-cache
    Expires=Thu, 01 Jan 1970 01:00:00 CET
    Link=<http://localhost:7080/rest/resource?ps=2&category=service&page=2>; rel="next"
    Link=<http://localhost:7080/rest/resource?ps=2&category=service&page=0>; rel="prev"
    Link=<http://localhost:7080/rest/resource?ps=2&category=service&page=152>; rel="last"
    Link=<http://localhost:7080/rest/resource?page=1&ps=2&category=service>; rel="current"
    Content-Encoding=gzip
    X-collection-size=305
    Content-Type=application/json
    Transfer-Encoding=chunked
    Date=Thu, 23 May 2013 07:57:38 GMT

    [
    {
    "resourceName": "/",
    "resourceId": 10041,
    "typeName": "File System",
    "typeId": 10013,
    …..
  2. Paging as part of the body - there the "real collection" is wrapped inside an object that also contains paging meta data as well as the paging links. To request this representation, a media type of application/vnd.rhq.wrapped+json needs to be used (and this is only available with JSON at the moment):

    Request method: GET
    Request path: http://localhost:7080/rest/resource
    Query params: page=1
    ps=2
    category=service
    Headers: Content-Type=*/*
    Accept=application/vnd.rhq.wrapped+json
    Cookies:
    Body:
    HTTP/1.1 200 OK
    Server=Apache-Coyote/1.1
    Pragma=No-cache
    Cache-Control=no-cache
    Expires=Thu, 01 Jan 1970 01:00:00 CET
    Content-Encoding=gzip
    Content-Type=application/vnd.rhq.wrapped+json
    Transfer-Encoding=chunked
    Date=Thu, 23 May 2013 07:57:40 GMT

    {
    "data": [
    {
    "resourceName": "/",
    "resourceId": 10041,
    "typeName": "File System",
    "typeId": 10013,

    ],
    "pageSize": 2,
    "currentPage": 1,
    "lastPage": 152,
    "totalSize": 305,
    "links": [
    {
    "next": {
    "href": "http://localhost:7080/rest/resource?ps=2&category=service&page=2"
    }
    },

    }

Please try this if you can. We want to get that into a "finished" state (as for the whole REST-Api) for RHQ 4.8

Thursday, May 16, 2013

Creating a delegating login module (for JBoss EAP 6.1 )



[ If you only want to see code, just scroll down ]

Motivation


In RHQ we had a need for a security domain that can be used to secure the REST-api and its web-app via container managed security. In the past I had just used the classical DatabaseServerLoginModule to authenticate against the database.

Now does RHQ also allow to have users in LDAP directories, which were not covered by above module. I had two options to start with:
  • Copy the LDAP login modules into the security domain for REST
  • Use the security domain for the REST-api that is already used for UI and CLI


The latter option was of course favorable to prevent code duplication, so I went that route. And failed.

I failed because RHQ was on startup dropping and re-creating the security domain and the server was detecting this and complaining that the security domain referenced from the rhq-rest.war was all of a sudden gone.

So next try: don't re-create the domain on startup and only add/remove the ldap-login modules (I am talking about modules, because we actually have two that we need).

This also did not work as expected:
  • The underlying AS sometimes went into reload needed mode and did not apply the changes
  • When the ldap modules were removed, the principals from them were still cached
  • Flushing the cache did not work and the server went into reload-needed mode


So what I did now is to implement a login module for the rest-security-domain that just delegates to another one for authentication and then adds roles on success.

This way the rhq-rest.war has a fixed reference to that rest-security-domain and the other security domain could just be handled as before.

Implementation



Let's start with the snippet from the standalone.xml file describing the security domain and parametrizing the module


<security-domain name="RHQRESTSecurityDomain" cache-type="default">
<authentication>
<login-module code="org.rhq.enterprise.server.core.jaas.DelegatingLoginModule" flag="sufficient">
<module-option name="delegateTo" value="RHQUserSecurityDomain"/>
<module-option name="roles" value="rest-user"/>
</login-module>
</authentication>
</security-domain>


So this definition sets up a security domain RHQRESTSecurityDomain which uses the DelegatingLoginModule that I will describe in a moment. There are two parameters passed:
  • delegateTo: name of the other domain to authenticate the user
  • roles: a comma separated list of modules to add to the principal (and which are needed in the security-constraint section of web.xml


For the code I don't show the full listing; you can find it in git.

To make our lives easier we don't implement all functionality on our own, but extend
the already existing UsernamePasswordLoginModule and only override
certain methods.


public class DelegatingLoginModule extends UsernamePasswordLoginModule {


First we initialize the module with the passed options and create a new LoginContext with
the domain we delegate to:

@Override
public void initialize(Subject subject, CallbackHandler callbackHandler,
Map<String, ?> sharedState,
Map<String, ?> options)
{
super.initialize(subject, callbackHandler, sharedState, options);

/* This is the login context (=security domain) we want to delegate to */
String delegateTo = (String) options.get("delegateTo");

/* Now create the context for later use */
try {
loginContext = new LoginContext(delegateTo, new DelegateCallbackHandler());
} catch (LoginException e) {
log.warn("Initialize failed : " + e.getMessage());
}



The interesting part is the login() method where we get the username / password and store it for later, then we try to log into the delegate domain and if that succeeded we tell super that we had success, so that it can do its magic.


@Override
public boolean login() throws LoginException {
try {
// Get the username / password the user entred and save if for later use
usernamePassword = super.getUsernameAndPassword();

// Try to log in via the delegate
loginContext.login();

// login was success, so we can continue
identity = createIdentity(usernamePassword[0]);
useFirstPass=true;

// This next flag is important. Without it the principal will not be
// propagated
loginOk = true;

the loginOk flag is needed here so that the superclass will call LoginModule.commit() and pick up the principal along with the roles.
Not setting this to true will result in a successful login() but no principal
is attached.


if (debugEnabled) {
log.debug("Login ok for " + usernamePassword[0]);
}

return true;
} catch (Exception e) {
if (debugEnabled) {
LOG.debug("Login failed for : " + usernamePassword[0] + ": " + e.getMessage());
}
loginOk = false;
return false;
}
}


After success, super will call into the next two methods to obtain the principal and its roles:

@Override
protected Principal getIdentity() {
return identity;
}


@Override
protected Group[] getRoleSets() throws LoginException {

SimpleGroup roles = new SimpleGroup("Roles");

for (String role : rolesList ) {
roles.addMember( new SimplePrincipal(role));
}
Group[] roleSets = { roles };
return roleSets;
}


And now the last part is the Callback handler that the other domain that we delegate to will use to obtain the credentials from us. It is the classical JAAS login callback handler. One thing that first totally confused me was that this handler was called several times during login and I thought it is buggy. But in fact the number of times it is called corresponds to the number of login modules configured in the RHQUserSecurityDomain.


private class DelegateCallbackHandler implements CallbackHandler {
@Override
public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {

for (Callback cb : callbacks) {
if (cb instanceof NameCallback) {
NameCallback nc = (NameCallback) cb;
nc.setName(usernamePassword[0]);
}
else if (cb instanceof PasswordCallback) {
PasswordCallback pc = (PasswordCallback) cb;
pc.setPassword(usernamePassword[1].toCharArray());
}
else {
throw new UnsupportedCallbackException(cb,"Callback " + cb + " not supported");
}
}
}
}



Again, the full code is available in the RHQ git repository.

Debugging (in EAP 6.1 alpha or later )



If you write such a login module and it does not work, you want to debug it. Started with the usual means to find out that my login() method was working as expected, but login just failed. I added print statements etc to find out that the getRoleSets() method was never called. But still everything looked ok. I did some googling and found this good wiki page. It is possible to tell a web app to do audit logging


<jboss-web>
<context-root>rest</context-root>
<security-domain>RHQRESTSecurityDomain</security-domain><disable-audit>false</disable-audit>


This flag alone is not enough, as you also need an appropriate logger set up, which is explained on
the wiki page. After enabling this, I saw entries like


16:33:33,918 TRACE [org.jboss.security.audit] (http-/0.0.0.0:7080-1) [Failure]Source=org.jboss.as.web.security.JBossWebRealm;
principal=null;request=[/rest:….

So it became obvious that the login module did not set the principal. Looking at the code in the superclasses then led me to the loginOk flag mentioned above.

Now with everything correctly set up the autit log looks like


22:48:16,889 TRACE [org.jboss.security.audit] (http-/0.0.0.0:7080-1)
[Success]Source=org.jboss.as.web.security.JBossWebRealm;Step=hasRole;
principal=GenericPrincipal[rhqadmin(rest-user,)];
request=[/rest:cookies=null:headers=authorization=user-agent=curl/7.29.0,host=localhost:7080,accept=*/*,][parameters=][attributes=];

So here you see that the principal rhqadmin has logged in and got the role rest-user assigned, which is the one matching in the security-constraint element in web.xml.

Further viewing



I've presented the above as a Hangout on Air. Unfortunately G+ muted me from time to time when I was typing while explaining :-(

After the video was done I got a few more questions that at the end made me rethink the startup phase for the case that the user has a previous version of RHQ installed with LDAP enabled. In this case, the installer will still install the initial DB-only RHQUserSecurityDomain and then in the startup bean
we check if a) LDAP is enabled in system settings and b) if the login-modules are actually present.
If a) matches and they are not present we install them.

This Bugzilla entry also contains more information about this whole story.

Tuesday, May 07, 2013

RHQ 4.7 released



RHQ 4.7 has been released and one of the two major features in this release are
the new charts that finally have replaced the year old charts that we had since the
start of RHQ project:


Screenshot with new charts


The new charts are implemented on with the awesome D3.js toolkit as I've written before.

The other big change is an upgrade of the underlying app server to JBoss EAP 6.1 alpha 1.

As always there have been many more smaller improvements and bug fixes.

Please check the full release notes for details. They also contain a list of commits.

RHQ is an extensible tool to monitory your infrastructure of machines and applications, alert operators on user defined conditions, configure resources and run operations on them from a central web-based UI. Other ways of communicating with RHQ include a command line interface and a REST-api.

You can download the release from source forge.


As mentioned above, the old installer is gone, so make sure to read
the wiki document describing how to use the new installer.

Maven artifacts are available from the JBoss Nexus repository and should also be available from Central.



Please give us feedback, be it in Bugzilla, Mailing lists or the forum. Or just join us on IRC at irc.freenode.net/#rhq.

Thursday, April 25, 2013

Building WildFly

After Red Hat has renamed TASFKAJB to WildFly, the git repository has also been moved to https://github.com/wildfly/wildfly.

Building WildFly is pretty similar to what it has been before
  • Fork the WildFly repository to your own on GitHub
  • Clone your repository to the local disk (which is in my case)
    git clone git@github.com:pilhuhn/wildfly.git
  • Change into the created wildfly directory
    cd wildfly
  • Run maven to build the server
    mvn install

The server is then created inside the build/target directory

$ pwd
/devel/wildfly/build/target
$ ls
antrun generated-configs wildfly-8.0.0.Alpha1-SNAPSHOT

change into this directory and start the server via bin/standalone.sh

Done :-)

Wednesday, March 13, 2013

Awesome new graphs in RHQ - based on d3.js



Mike Thompson has yesterday presented the latest and greatest version of the new graphs for RHQ in a video on YouTube. Shortly after he has committed the results of his huge work into RHQ master branch.


While this work is not yet finished, it is the result of the work started by Denis Krusko in last years Google Summer of Code. At the moment both the old and new graphs are can be looked at in the UI, so that you can compare them and potentially report non-matches.


Here are some screenshots to foster your appetite:


Popup chart for a single metric
Popup chart for a single metric

Individual metric
Individual metric on the monitoring tab


As the subject already says are those graphs made with the help of the awesome D3.js framework - I let Mike chime in to describe in more details what he and Denis had to do to get this to work inside GWT+SmartGWT.

I've uploaded a snapshot from master as of this morning (my time) of this from our CI server onto SourceForge for you to try. THIS IS NOT FOR PRODUCTION.


There is a known issue where the red bar shows "..global exception.." this is harmless and we will fix that anyway. Also the graph portlets in the dashboard don't honor the column width yet.


Please do not forget to report bugs (if there are any :-)

Thursday, March 07, 2013

Sorry Miss Jackson -- or how I loved to do custom Json serialization in AS7 with RestEasy

Who doesn't remember the awesome "Sorry Miss Jackson" video ?

Actually that doesn't have to do anything with what I am talking here -- except that RestEasy inside JBossAS7 is internally using the Jackson json processing library. But let me start from the beginning.

Inside the RHQ REST api we have exposed links (in json representation) like this:

{ 
"rel":"self",
"href":"http://...
}

which is the natural format when a Pojo

class Link {
String rel;
String href;
}

is serialized (with the Jackson provider in RestEasy).

Now while this is pretty cool, there is the disadvantage that if you need the link for a rel of 'edit', you had to load the list of links, iterate over them and check for each link if its rel is 'edit' and then take the value of the href.

And there is this Bugzilla entry.

I started investigating this and thought, I will use a MessageBodyWriter and all is well. Unfortunately MessageBodyWriters do not work recursively and thus cannot format Link objects as part of another object in a special way.

Luckily I have done custom serialization with Jackson in the as7-plugin before, so I tried this, but the Serializer was not even instantiated. More trying and fiddling and a question on StackOverflow led me to a solution by copying the jackson libraries (core, mapper and jaxrs) into the lib directory of the RHQ ear and then all of a sudden the new serialization worked. The output is now

{ "self":
{ "href" : "http://..." }
}


So far so nice.

Now the final riddle was how to use the jackson libraries that are already in user by the resteasy-jackson-provider. And this was solved by adding respective entries to jboss-deployment-structure.xml, which ends up in the META-INF directory of the rhq.ear directory.

The content in this jboss-deployment-structure.xml then looks like this (abbreviated):


<sub-deployment name="rhq-enterprise-server-ejb3.jar">
<dependencies>
....
<module name="org.codehaus.jackson.jackson-core-asl" export="true"/>
<module name="org.codehaus.jackson.jackson-jaxrs" export="true"/>
<module name="org.codehaus.jackson.jackson-mapper-asl" export="true"/>
</dependencies>
</sub-deployment>


AS7 is still complaining a bit at startup:


16:48:58,375 WARN [org.jboss.as.dependency.private] (MSC service thread 1-3) JBAS018567: Deployment "deployment.rhq.ear.rhq-enterprise-server-ejb3.jar" is using a private module ("org.codehaus.jackson.jackson-core-asl:main") which may be changed or removed in future versions without notice.


but for the moment the result is good enough - and as we do not regularly change the underlying container for RHQ, this is tolerable.

And of course I would be interested in an "official" solution to that -- other than just doubling those jars into my application (and actually I think this may even complicate the situation, as (re-)using the jackson jars that RestEasy inside as7 uses, guarantees that the version is compatible).

Tuesday, February 26, 2013

RHQ 4.6 released


As I've mentioned before, the RHQ team has been very busy since RHQ 4.5.1 (and actually already before that) and has switched the application server it uses to JBoss AS 7.1. Directly after the switchover we have posted a first alpha version.


Now after more work and fixes, we are happy to provide the release 4.6 of RHQ, that has all the issues resolved that arose from the switch. Features of this release are:

  • The internal app server is now JBossAS 7.1.1
  • GWT has been upgraded to version 2.5
  • There is a new installer (this has also changed since the 4.6 alpha release)
  • The REST-Api has been enhanced
  • Korean translations have been added (contributed by SungUk Jeon)
  • Webservices have been removed
  • Building RHQ now requires Java7, but it will still run on Java6
.

See the full release notes for details. They also contain a list of commits.

You can download the release from source forge.


As mentioned above, the old installer is gone, so make sure to read
the wiki document describing how to use the new installer.

Maven artifacts are available from the JBoss Nexus repository and should soon also be available from Central.

We also like to say thank you to our contributors for this release:

  • Jürgen Hoffmann
  • Richard Hensman
  • SungUk Jeon


Please give us feedback, be it in Bugzilla, Mailing lists or the forum. Or just join us on IRC at irc.freenode.net/#rhq.