RHQ and Jopr: new source repository

Yes, we can as well :)

In the past it was often confusing to users that we have two source repositories for RHQ and Jopr. We have recently changed this.

The new RHQ repository lives now at http://git.fedorahosted.org/. There are a few branches defined, but the main one of interest is origin/master.

As you can see, we have not only move the repository location, but also move to the git version control system and even merged Jopr and RHQ together into one repository - so no more riddling where to find what.


See also this post about the change to bug tracking.

RHQ Bugtracker moves to Bugzilla - action required

Project RHQ, which is the foundation for Jopr will soon move all bug tracking from the old JIRA system at jira.rhq-project.org over to a bugzilla instance at Red Hat.

So when you have an account at the jira and reported bugs, you should follow those steps to have your cases linked to you on the new bugzilla site:

1. Go to bugzilla at bugzilla.redhat.com
   
   
   
   1. If you don't yet have an account create one with your preferred email address.
      
   2. If you do have an account, login (or submit a password change request then login). And make a note of the email address you are using.
   
   
2. Go to RHQ jira (http://jira.rhq-project.org/browse/RHQ),
   log in and set your email address through your profile (http://jira.rhq-project.org/secure/EditProfile!default.jspa) to the one you are using in bugzilla above.
   

Cutover will happen within the next few days

For now, the Jopr bugtracker will stay at the JBoss jira, but will probably also merged into bugzilla in the future.

Embedding ruby in java - less than trivial so far

Currently I am trying to get puppet, a configuration management system to be executed in Jruby - and especially embedded within some java code.

Luckily there is the JSR 223 scripting available in Java6, so the start is relatively easy: get jruby from jruby.org and add it to the java programs classpath, java program code looks like this:


// Define path to jruby libraries
System.setProperty("jruby.home","/opt/local/share/java/jruby");

// Get the scripting engine
ScriptEngineManager m = new ScriptEngineManager();
ScriptEngine rubyEngine = m.getEngineByName("jruby");

// Define the puppet script to run as arguments
rubyEngine.put(ScriptEngine.ARGV,new String[]{"example.pp"});

// read the script
File f = new File("puppet.rb");
BufferedReader br = new BufferedReader(new FileReader(f));

// and execute it
rubyEngine.eval(br, context);


Simple - eh?

Unfortunately is it not that simple (yet).

Passing the argument vector to my script does not yet work in jruby 1.4 cr1 (see JRUBY-4090), so I need to do a work around of defining a script, that sets this and then calls puppet


ARGV << 'example.pp'

require 'puppet/application/puppet'

Puppet::Application[:puppet].run



Next, puppet is complaining about missing stuff like OpenSSL (which is a difficult beast in jruby, as the standard ruby implementation needs some C code, which can not directly run in jruby).

Also it is somehow complaining about yecht not being found - I still don't really grok setting and properties up things like jruby.lib, jruby.home etc.

If anyone can shed some light on it, I would appreciate it :-)

Jopr, RHQ and the availability check interval (updated)

Jopr and RHQ check from time to time for each resources availability. This is done in the agent, where a periodic thread calls the getAvailability() method of each ResourceComponent. After this scan the result is sent to the server and the server shows the red and green state on the resource.

This server side processing creates a certain stress in large environements (say: hundreds of agents with ten thousands of resources), so that in Jopr 2.3 we have increased the interval this check is done to 5 minutes.

For many use cases (smaller installs, testing) this is too long. Luckily, this interval is not cast in stone, but configurable in the agent settings file, conf/agent-configuration.xml:

 <!--
  
    _______________________________________________________________
    rhq.agent.plugins.availability-scan.period-secs
 
    Defines how often an availability scan is run.  This type
   of scan is used to determine what resources are up and running
   and what resources have gone down. The value is specified in
   seconds.
-->
   <!--
    <entry key="rhq.agent.plugins.availability-scan.period-secs" value="300"/>
   -->

You see the default is shown as commented value. To change this to a different value, remove the xml comment signs around it and change this back to e.g. 60 seconds (it will probably not make any sense to go lower than that, as the load on the agent, your managed resource and also the Jopr server will increase again):

   <!--
   -->
    <entry key="rhq.agent.plugins.availability-scan.period-secs" value="60"/>

After this is done, you need to restart the agent to have it read the new value (see also next paragraph).

UPDATE

As the agent writes its configuration into the java preferences as backing store, the above change will not directly be honored. This may sound strange at first, but it has the advantage that you can run an agent, remove it, install a newer version of it and have the new version automatically use the saved values of the first agent install.

So you need to tell the agent that it indeed should read the new configuration file. This can be done by starting the agent with the option --clenanconfig or better by supplying -c agent-configuration.xml. This is explained at the top of the agent-configuration.xml file and also in the yellow box in the agent install document.


Of course this applies to JBoss ON 2.3 as well.

Jopr installer bug - careful when testing

The RHQ 1.3 and Jopr 2.3.1 installer has a little issue (see RHQ-2222 jira), where it clears out the database password when you click on "Test connection".

Lets have a look at a screen shot:

When setting a database password (green circle) and then testing the connection (orange circle), the password will be reset and the install will fail later on.

If you are using the embedded H2 database, just re-select it in the drop down (violet circle) to have the database password filled in again by the system. Actually with H2, there is no need to test the connection in the first place :-)

RHQ tip of the day: agent confused?

Darko asked the other day on the Jopr Irc channel:

I have an agent, which isn't collecting my cpu-load data any more. I also have many more problems with the agent. How can I reset the agent to collect all data again? It may have to do with the fact that I had a server crash on the system with the running agent. Since the crash the agent isn't working any more.

It looks like the internal state database of the agent got confused. When the agent is fully configured and working, it will save its inventory of locally managed resources (a part of the global inventory kept on the server) along with their measurement schedules in a local database.

The agent can use this database to start working on the next start even if the server is not reachable. Changes on the server will synchronized when the connection is up again.

Now back to the original question: To get the agent going again, you need to erase the bad inventory and sync with the server again. You can do this by passing option --purgedata to the agent commandline:

$ bin/rhq-agent.sh --purgedata

To see a full list of the agents command line options, you can pass the --help option to the agent start script.

RHQ tip of the day: Help, my resource does not show in the tree!

We see it from time to time that people have correctly set up Jopr or RHQ and want to monitor e.g. a JBoss AS server or other resources. They even have an agent running on the machine with the JBoss AS on, but still it does not show up in the resource tree of this platform.

Most often this is a simple issue, as the resource is just sitting in the queue to be imported into inventory:

Just select the ones you want to import, click on the import button and after a second, the resource will show up in the resource tree below its platform.

Jopr and RHQ require this on purpose, as this way you can ignore e.g. servers that are just up at the moment of import, but which are not to be managed.

As you have seen that resources can be ignored, there is also a way to "bring them back" when you decide later to import them. Either click on "View all..." in the Auto-Discovery portlet seen above or select Auto discovery queue from the Overview menu, then select Both new and Ignored from the drop-down:

This will show ignored resources, that you can check and then click un-ignore to get them back into the new state from where you can finally import them.

RHQ plugins - platform services

There was recently a question on the Jopr devel mailinglist, that Ian Springer answered - I've taken his answers to write this post, as this is of general interest to plugin writers.

Bruno asks:
I have written a plugin which defines a single service like so:

<plugin ...>
  <service
      name="Interdomain Service"
      ...
      supportsManualAdd="true">
  </service>
</plugin>

But during auto-discovery, the resource discovery component does not get called. It only gets called, if I define the Interdomain Service to be a server. What am I missing here?

Lets first have a look at the resource hierarchy within RHQ:

So for <server>s that are standalone within a <plugin> tag, it is obvious that the parent is the <platform>, while <service> can have both other ResourceCategory as parent.

This means that you have to define the service's parent type(s), so RHQ knows where to stick it in the type hierarchy once it's discovered. There are a few ways to do this, depending on your needs.

If it's a platform service (i.e. a service whose parent is the platform
itself), you'd do something like:

<service name="Interdomain Service" ...>
  <runs-inside>
    <parent-resource-type name="Windows"  plugin="Platforms"/>
    <parent-resource-type name="Linux"    plugin="Platforms"/>
    <parent-resource-type name="Solaris"  plugin="Platforms"/>
    <parent-resource-type name="HP-UX"    plugin="Platforms"/>
    <parent-resource-type name="AIX"      plugin="Platforms"/>
    <parent-resource-type name="FreeBSD"  plugin="Platforms"/>
    <parent-resource-type name="Mac OS X" plugin="Platforms"/>
    <parent-resource-type name="Java" plugin="Platforms"/> <!-- any OS that RHQ doesn't have native (i.e. SIGAR) support for -->
  </runs-inside>
      ...
</service>

(If your service will only run on certain OS'es, you could remove the
unsupported platform types from the list of parent types).

Or you could do something similar to say it can be the child of multiple
server or service types, e.g.:

<service name="Interdomain Service" ...>       
  <runs-inside>
    <parent-resource-type name="JMX Server" plugin="JMX"/>
    <parent-resource-type name="JBossAS Server" plugin="JBossAS"/>
    <parent-resource-type name="JBossAS Server" plugin="JBossAS5"/>
  </runs-inside>
      ...
</service>

Finally, if you want it to be a child of a single server or service type
defined in the same plugin, just nest it inside the parent server or
service element, e.g.:

<server name="CoolServer">
   <service name="Interdomain Service" ...>
      ...
   </service>
   ...
</server>

The reason it works if you change it to a server, rather
than a service, is because, when parsing the plugin descriptor, if RHQ
encounters a server element at the top level with no runs-inside types
defined, it assumes the server is a platform server (a server who has
all the platform types as its parents). So:

<server name="Interdomain Service" ...>
    ...
</server>

is essentially equivalent to:

<server name="Interdomain Service" ...>
  <runs-inside>
    <parent-resource-type name="Windows"  plugin="Platforms"/>
    <parent-resource-type name="Linux"    plugin="Platforms"/>
    ...
    <parent-resource-type name="Mac OS X" plugin="Platforms"/>
    <parent-resource-type name="Java" plugin="Platforms"/> <!-- any OS that RHQ doesn't have native (i.e. SIGAR) support for -->
  </runs-inside>
      ...
</server>

Most of the time when writing a plugin you will probably start with a server at the top level of your plugin, so it will just work.

1000 km :-)

Last weekend I crossed the 1000km mark since I started running last year (see here). I took me quite some time since the 800km mark at Stuttgart-Lauf. This had to a big extend to do with the fact that I was not really able to go out much while kids were on vacation from Kindergarden and also needed some rest after Üetliberg. But since start of september I am back on track :-)

So what are my next goals? I don't really know yet. Running Stuttgart-Lauf faster than this year probably. And just improving my overall fitness to be able to run at same pace with a lot lower heart rate.

RHQ / Jopr agent waiting at startup

I have written in the past about RHQ and Jopr "agent waiting at startup. That actually was before high availability has been released, so it is a little outdated.

With high availability (HA) you will see one new type of issues: ip address forward and backward mappings do not match.

Basically you need to make sure that the IP address of your computer can be reverse mapped to the computer name and that this name mapps back to the same ip address.
And this needs to be true along all your hosts.

Suppose you have an IP address of 172.31.7.7, then the results of name resolution should look like the following:

$ dig -x 172.31.7.7
[...]
;; ANSWER SECTION:
7.7.31.172.in-addr.arpa. 86400 IN     PTR     snert
$
$ dig snert
[...]
;; ANSWER SECTION:
snert       74030   IN      A       172.31.7.7

If you had agent-server communication working before and it stopped all of a sudden, go to the Jopr server UI, to the Administration -> High Availability -> Server section and check if the name shown there matches what you expect - same for the agents.


If this all works, and the agent is still hanging, have a look at the older post to rule out other possibilities for misconfiguration.

New api for manual discovery on RHQ and Jopr

The other day I was writing about manual discovery/addition of resources. Now that Jopr 2.3 (and its JBoss ON counterpart is out), we have updated the API for manual discovery to provide its own interface so that you don't need to fiddle around with finding the passed configuration.

The new API lives in the ManualAddFacet, which has one method
DiscoveredResourceDetails discoverResource(Configuration pluginConfiguration, ResourceDiscoveryContext<T> context)
throws InvalidPluginConfigurationException; that you need to implement. This Facet is implemented in addition to the ResourceDiscoveryComponent within the discovery component. Ips has already converted all existing plugins to the new api, so you can have a look at many examples there, so I'll only show a short one here, which is an excerpt from the Twitter plugin:

public class TwitterDiscovery 
   implements ResourceDiscoveryComponent,ManualAddFacet {
 
    // Auto-discovery not supported, so return an empty set
    public Set discoverResources
       (ResourceDiscoveryContext discoveryContext) 
       throws Exception {
 
        // We don't support auto-discovery.
        return Collections.emptySet();
    }
 
    // perform the manual add
    public DiscoveredResourceDetails discoverResource(
       Configuration pluginConfig, 
       ResourceDiscoveryContext discoveryContext)
             throws InvalidPluginConfigurationException {
 
         DiscoveredResourceDetails detail = 
            new DiscoveredResourceDetails(
                discoveryContext.getResourceType(), //ResourceType
                url + "_"+ user, // ResourceKey
                url + " feed for " +user, // ResourceName
                null, // Version
                "One " + url + " user", // Description
                pluginConfig, // Passed plugin config
                null  ); // Process scans
        return detail;
    }
}

I have also updated the plugin skeleton generator code in svn to support this new API and have uploaded the new version to jopr.org

Jopr 2.3.1 released

The Jopr team has released version 2.3.1 of Jopr.

The major difference over Jopr 2.3 is a fix in the dbupgrade scripts.

If you are still on 2.2 and want to upgrade, use 2.3.1, as 2.3 will not correctly update your database schema.

You can download the release from SourceForge.

Jopr talking

I have added an experimental alert sender to Jopr that can call administrators on their phone, tell them the alert conditions and can then take input via DTMF tones. This input can be used to disable or re-enable the alert definition or to clean out this alert from the alert history.

This all has been made possible by via the Mobicents server and SIP servlet.



A special thanks goes to Jean Deruelle from Mobicents, who wrote the integration part on the Mobicents side.

Podcast episode

I have created an episode of the Jopr podcast that shows the setup and some UI demo (the call on the SIP phone has been faked a little bit, as my screen capture program is always crashing at this point, but the message read is real).

As usual, this podcast episode is available on iTunes.

Show me the code...

The code on the Jopr side lives in http://svn.rhq-project.org/repos/rhq/branches/HEIKO-EXP/. There is also code on the Mobicents side, but we still need to clean up things and write some documentation on how to integrate them.

Ping me if you are interested in this integration.

Snow Leopard - mixed feelings?

So far Snow Leopard is somewhat disappointing for me:

• DNS - especially when VPN is on sucks, as SL is sometimes blacklisting very valid hosts all of a sudden. This results in e.g. "The computer is not connected to the internet" messages in Safari for sites like www.google.com (see e.g. here)


• The upgrade broke Aperture - the upgrade process removed a framework needed by Aperture. For me re-installing /Library/Frameworks/Pluginmanager.framework from a TimeMachine backup worked. (see e.g. here)


• Older Software like Parallels 2.5 no longer works


• Nambu 1.2 no longer works (and Nambu 2 is still in closed beta)


• X-Lite 3 has no sound in/out. Dialing works though. And the beta of X-Lite 4 is not even able to show the preferences.

The biggest item for me is currently the DNS one, as this renders the system more or less unusable.


But then there is at least some good thing about it:

• The internal VPN now works against Cisco concentrators, which is very nice


• JDK 1.6 is finally available on 32 bit macs like my 3 years old MacBook

I will update this post, when I see more pros and cons.

Jopr podcast iTunes URL has changed

The Jopr podcast is now available on iTunes at the following new place
http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewPodcast?id=331189266.

Please update your feeds or iTunes subscriptions accordingly. Thanks.

New episode of the Jopr podcast: Manik from Infinispan (updated)

At JBossWorld Chicago I had the pleasure to do a short interview with Manik Surtani, lead of the Infinispan project.

The interview is now available in the Jopr Podcast home page and is also available on iTunes (note, the iTunes url has changed).

Enjoy!

Again: if you like it, please rate it up on iTunes and/or provide Feedback via email to me.

What a weekend

I am currentl staying in New Jersey, as the opportunity to give a talk at JBossWorld in Chicago also gave me the opportunity to go to NJ to meet the biger part of theJopr developers.

I arrived friday evening and as the car rental company ran out of cars, I got the last one remaining: a Chrysler Sebring convertible. On Saturday I went running and also drove around a lot with the roof open and music turned on - very nice :-)

Sunday I spent with Jess and family. Jess and I went climbing - my first time, but this was fun. One one side I can't imagine to do it more often, but on the other side I can :)

Today was Labour Day, so no one in America is working (only joking) and I went to New York. Walked around a lot and visited important landmarks like the Brookly bridge and the Apple store on 5th avenue.

Jopr 2.3 released

Jopr version 2.3 has been released. You can find the binaries http://sourceforge.net/projects/rhq/ on SourceForge.

IMPORTANT UPDATE: 2.3.1 is out with a fix for dbupgrade

Notable changes are:

• Support for JBoss EAP 5


• A new command line interface: you can now talk to Jopr from a Linux or Windows command line. This interface uses the Java 6 script support, so that you can create scripts for re-use.


• Group alerts: it is now possible to fire an alert when any resource in a group runs into an error condition by defining alert templates on the group


• Experimental support for embedded H2 database and MS SQL Server (same as in 2.2.1)


• Plugins can now depend one of several others: if you have e.g. a plugin that uses JMX it is now possible to either depend on JBossAS when the resource types are used within the JBossAS or depend directly (and only) on JMX plugin to discover standalone JVM versions of those resources.


• Support for taking snapshots of resource configurations: it is now possible to gather e.g. all config files from one of your JBossAS servers and have that zipped up and put in a place on the Jopr server


• Dynagroups have been extended to support searching more levels of ancestry


• Postgres 8.4 is now officially supported


• Autodiscovery of JBossAS jnp credentials: Jopr is now looking at the config files to obtain the security credentials for secured JBossAS instances, so that you do not need to type them in by hand in the inventory

And many many more improvements. You can see lists of closed Jiras for
RHQ and Jopr projects.

Matching SVN tags are http://svn.rhq-project.org/repos/rhq/tags/RHQ_1_3_0/ and Jopr_2_3_0.

As so often :) join us on irc.freenode.net #jopr or in the forums or or or :)

Twitter plugin + alert sender for Jopr updated

I've updated the twitter plugin for Jopr:

Manual add instead of auto discovery

The Twitter feed is no longer autodiscovered, but you have to go to the platform and explicitly add it there.

The reason behind it is that most users don't want to have a twitter feed discovered on each agent.

To learn more about auto discovery vs. manuall add, have a look at my post "Jopr plugins: Autodiscovery vs. manual add".

Support for other Twitter-like services

The plugin is now also able to post to other twitter-like services like identi.ca (or your own private one based on the laconi.ca code).
The serverBase is http://identi.ca/api/ for Identi.ca, as well as the search base.
For Twitter, serverBase is http://twitter.com and searchBase is http://search.twitter.com/

If you select the right template, this will automatically be filled in for you.

Alert sender updated too

I've also updated the Alert sending in the HEIKO-EXP branch to be able to send to other networks than Twitter.
This means that in the system configuration, there is a new field for the base url of the services api. See above for possible values.

related...

I've just seen that Twitter is offering a streaming api. This may be a nice addition to the plugin. Today every request to obtain values opens a new connection to Twitter. With the streaming api, this could probably go away.

Jopr plugins: Autodiscovery vs. manual add

One thing that I have neglected so far in the plugin development articles for Jopr that I wrote is the possibility to just manually add a resource instead of having it auto discovered.

Why would you want to manually add a resource?

Some resources are virtual like e.g. Twitter feeds in the twitter plugin. If you have 100 agents, it makes no sense to either take the twitter feeds into inventory on all 100 of them or to have to ignore them all in the auto discovery portlet.

When to use auto discovery then?

Auto discovery is best used for more concrete resources like e.g. processes that can exist on a system (a JBossAS for example) or file systems etc. Those are items you usually want to monitor on each system.

Ok, show me the code

Well, actually the code is not that different from what you already know. Within ResourceDiscoveryComponent.discoverResources() the change is to check for passed resource configurations and use those to create the concrete discovered resource:

for (Configuration config : 
    (Iterable) 
    discoveryContext.getPluginConfigurations()) {
         DiscoveredResourceDetails detail = 
            new DiscoveredResourceDetails(
                discoveryContext.getResourceType(), // ResourceType
                url + "_"+ user, // ResourceKey
                url + " feed for " +user,
                null,
                "One " + url + " user",
                config,
                null  );

          return Collections.singleton(detail);
}
return null;

As the plugin container will only pass us one configuration at a time (even if the call to getPluginConfigurations() is returning a List), we can safely return the constructed detail. And this is already all :-)


One thing you need to change though is in the plugin descriptor:

    <server
            name="Twitter"
            discovery="TwitterDiscovery"
            class="TwitterComponent"
            description="Twitter monitoring subsystem"
            supportsManualAdd="true"

This supportsManualAdd is what lets the resource type show up in its parent (usually platform for a standalone server) "Manually Add" dialog:

For more resources related to plugin development have a look at this post.