Friday, December 05, 2014

RHQ 4.13 released

I am very pleased to announce the release of RHQ 4.13 as an
early gift from St. Nicholas :)

Screenshot of the RHQ UI showing some charts

This release contains a lot of bug fixes and smaller improvements, as well as some new features:

  • Alerts have a new status 'recovered', that can be filtered upon
  • The UI allows to hide elements that are not needed on a per user basis
  • The as7/WildFly plugin now supports runtime queues and topic subscribers
  • Further improvements in the Storage Nodes

As always RHQ is available for download in form of a zip archive. If you want to try out RHQ without too much setup, you can also use a pre-created Docker image
from (the link contains setup information).

Please consult the release notes for further details and a download link.

Maven artifacts will soon be available on Maven Central.

Special thanks goes to

  • Alan Santos
  • Andreas Veithen
  • Elias Ross
  • Jérémie Lagarde

for their code contributions for this release.

Friday, October 10, 2014

WildFly subsystem for RHQ Metrics

For RHQ-Metrics I have started writing a subsystem for WildFly 8 that is able to collect metrics inside WildFly and then send them at regular intervals (currently every minute) to a RHQ-Metrics server.

The next graph is a visualization with Grafana of the outcome when this sender was running for 1.5 days in a row:

Graphs of JVM memory usage
Graphs of JVM memory usageWildFly memory usage

( It is interesting to see how the JVM is fine tuning its memory requirement over time and using less and less memory for this constant workload ).

The following is a visualization of the setup:


The sender is running as a subsystem inside WildFly and reading metrics from the WildFly management api. The gathered metrics are then pushed via REST to RHQ-Metrics. Of course it is possible to send them to a RHQ-Metrics server that is running on a separate host.

The configuration of the subsystem looks like this:

<subsystem xmlns="urn:org.rhq.metrics:wildflySender:1.0">
<metric name="non-heap"
<metric name="thread-count"

As you see, the path to the DMR resource and the name of the attribute to be monitored as metrics can be given in the configuration.

The implementation is still basic at the moment - you can find the source code in the RHQ-Metrics repository on GitHub. Contributions are very welcome.

Heiko Braun and Harald Pehl are currently working on optimizing the scheduling with individual intervals and possible batching of requests for managed servers in a domain.

Many thanks go to Emmanuel Hugonnet, Kabir Khan and especially Tom Cerar for their help to get me going with writing a subsystem, which was pretty tricky for me. The parsers, the object model and the XML had a big tendency to disagree with each other :-)

Sunday, August 24, 2014

Alert definition templates in plugin(descriptor)s ?


his is a more general question and not tied to a specific version of RHQ (but may become part of a future version of RHQ and/or RHQ-alerts).

Do you feel it would make sense to provide alert templates inside the plugin (descriptor) to allow the plugin writer to pre-define alert definitions for certain resource types / metrics? Plugin-writers know best what alert definitions and conditions make sense and should get the power to pre-define them.

This idea would probably work best with some relative metrics like disk is 90% full as opposed to absolute values that probably depend a lot more on concrete customer scenarios (e.g. heap usage over 64MB may be good for small installations, but not for large ones).
In the future with RHQ-alerts, it should also be possible to compare two metrics with each other, which will allow to say "if metric(disk usage) > 90% of metric(disk capacity) then ...".

I've scribbled the idea down in the Wintermute page on the RHQ wiki.

If you think this is useful, please respond here or on the wiki page. Best is if you could add a specific example.

Thursday, August 14, 2014

RHQ-Alerts aka Alerts 2.0 aka Wintermute

The RHQ-team has been thinking about next generation Alerting for RHQ for a while.

I have written my thoughts down in a blog post in the RHQ space of the JBoss
community site in order to start the (design) process:

Thoughts on RHQ-Alerts aka Alerts 2.0 aka Wintermute.

Please feel free to comment here or there :)

Tuesday, July 29, 2014

Java Forum Stuttgart 2014

tl;dr: Great as every year.

On July 17th, I attended the Java Forum Stuttgart conference, a one day conference with 49 talks in 7 tracks. This years edition marked a new record with 1600 attendees from all over Germany (and probably also from other countries).

Conference itself was awesome as every year with good talks and good food, 30+ exhibition stands and some free beer at the end :) The conference is organized by the Java User Group Stuttgart e.V. and as such is a non-profit event, which for sure partially explains the success.

I was lucky that my talk "Slim Fast" about a "diet for the application memory footprint" was accepted and I was positively surprised that the room was full despite the fact that the abstract did not contain any buzzwords like IoT or Cloud :-) (German) slides are attached to the above link, JUG Stuttgart now also has posted some pictures.

JUG Stuttgart also posted more pictures of sessions and also some general impression on their G+ page<./p>

Other talks that I've attended were:

"Java FX everywhere" by Gerrit Grunwald, who showed an app that he wrote for the desktop with Java FX and then ported over to devices like Raspberry PI, Android, iPad etc.

"Android ist anders - Android dependency management". This was about dependency management for Android - and for all those that blame Maven hell, check this out and you will like plain Maven afterwards :)

"IBM BlueMix". In this talk a guy from IBM explained a bit what BlueMix is and then demoed creating an app from a simple template and deploying it to a live BlueMix server in the cloud.

"TypeScript - JavaScript für Java Entwickler". Kai Toedter showed the TypeScript language as an alternative to JavaScript. While I was a bit skeptical about "yet another language", the talk totally made me change my mind and I can really see it as good alternative for future UI-work.

Sunday, June 15, 2014

RHQ-Metrics and Grafana (updated)

As you may know I am currently working on RHQ-Metrics, a time series database with some charting extensions that will be embeddable into Angular apps.

One of the goals of the project is also to make the database also available for other consumers that may have other input sources deployed as well as their own graphing.

Over the weekend I looked a bit at Grafana which looks quite nice, so I started to combine RHQ-metrics with Grafana.

I did not (immediately) find a description of the data format between Grafana and Graphite, but saw that InfluxDB is supported, so I fired up WireShark and was able to write an endpoint for RHQ-Metrics that supports listing of metrics and retrieving of data.

The overall setup looks like this:

RHQ metrics grafana setup
Setup used

A Ganglia gmond emits data over IP multicast, which is received with the protocol translator client, ptrans. This translates the data into requests that are pushed into the RHQ-metrics system (shown as the dotted box) and there stored in the Cassandra backend.

Grafana then talks to the RHQ-Metrics server and fetches the data to be displayed.
For this to work, I had to comment out the graphiteUrl in conf.js and use the influx backend like this:

 datasources: {
influx: {
default: true,
type: 'influxdb',
username: 'test',
password: 'test',
url: 'http://localhost:8080/rhq-metrics/influx',

As you can see, the URL points to the RHQ-Metrics server.

The code for the new handler is still "very rough" and thus not yet in the normal source repository. To use it, you can clone my version of the RHQ-Metrics repository and then run in the root directory to get the RHQ-Metrics server (with the built-in memory backend) running.


I have now added code (still in my repository) to support some of the aggregating functions of Influx like min, max, mean, count and sum. The following shows a chart that uses those functions:

Wednesday, May 07, 2014

RHQ 4.11 released

I am proud to announce the immediate availability of RHQ 4.11.

As always this release contains new features and bug fixes.

Notable changes are:

  • Plugins can now define plugin specific DynaGroup expressions, which can even be auto-activated
  • Further agent footprint reduction for jmx-clients (especially if the agent runs on jdk 1.6)
  • Much improved agent install via ssh from the RHQ UI.
  • Support for Oracle 12 as backend database
  • New login screen that follows

As always RHQ is available for download in form of a zip archive. If you want to try out RHQ without too much setup, you can also use a pre-created Docker image
from (the link contains setup information).

Please consult the release notes for further details and a download link.

If you only want to have a quick look, you can also consult a
running RHQ 4.11 instance that we have set up. User/Pass are guest / rhqguest

Special thanks goes to

  • Elias Ross
  • Michael Burman

for their code contributions for this release.

My first Netty-based server

Netty logo
RHQ logo

For our rhq-metrics project (and actually for RHQ proper) I wanted to be able to receive syslog messages, parse them and in case of content in the form of

type=metric thread.count=11 heap.permgen.size=20040000

forward the individual key-value pairs (e.g. thread.count=11) to a rest endpoint for further processing.

After I started with the usual plain Java ServerSocket.. code I realized that this may become cumbersome and that this would be the perfect opportunity to check out Netty. Luckily I already had an Early Access copy of Netty in Action lying around and was ready to start.

Writing the first server that listens on TCP port 5140 to receive messages was easy and modifying /etc/syslog.conf to forward messages to that port as well.

Unfortunately I did not get any message.

I turned out that on OS/X syslog message forwarding is only UDP. So I sat down, converted the code to use UDP and was sort of happy. I was also able to create the backend connection to the REST server, but here I was missing the piece on how to "forward" the incoming data to the rest-client connection.

Luckily my colleague Norman Maurer, one of the Netty gurus helped me a lot and so I got that going.

After this worked, I continued and added a TCP server too, so that it is now possible to receive messages over TCP and UDP.

The (current) final setup looks like this:

Overviw of my Netty app

I need two different decoders here, as for TCP the (payload) data received is directly contained in a ByteBuf while for UDP it is inside a DatagramPacker and needs to be pulled out into a ByteBuf for further decoding.

One larger issue I had during development (aside from wrapping my head around "The Netty Way" of doing things) was that I wrote a Decoder like

public class UdpSyslogEventDecoder extends

protected void decode(ChannelHandlerContext ctx, DatagramPacket msg, List<Object> out) throws Exception {

and it never got called by Netty. I could see that it was instantiated when the pipeline was set up, but for incoming messages it was just bypassed.

After I added a method

public boolean acceptInboundMessage(Object msg) throws Exception {
return true;

my decoder was called, but Netty was throwing ClassCastExceptions, but they were pretty helpful, as it turned out that the IDE automatically imported, which is not what Netty wants here. As Netty checks for the signature of the decode() and this did not match, Netty just ignored the decoder.

You can find the source of this server on GitHub.

Wednesday, April 30, 2014

Aerogear UPS alert plugin for RHQ


As already indicated, Matze Wessendorf and I sat together at Red Hat Summit and have created an alert sender plugin for RHQ that uses the unified push server from Aerogear to send alerts as push notifications to administrator phones.

For this to work you not only need to install the alert-sender plugin on RHQ side, but also have an appropriate application installed on your handset.

After you have installed the plugin in RHQ you need to go to Administration->Server Plugins and select the Alert:UPS entry. In the configuration settings you need to provide the host name where the push server lives, the secret to talk to it and also the id of your deployed handset app:

Bildschirmfoto 2014 04 30 um 09 24 53

After this is done, you can select this alert sender when you set up new notifications as with other senders:

Bildschirmfoto 2014 04 30 um 09 26 08

Right now this does not have any alert-specific settings, but I guess that in the future one could perhaps set the alert sound here for iOS devices or similar.

And after all this is done, the app is deployed and your handset is on, you could receive a push message just like this:


Screenshot of our first alert sent

If you want to have a look at the sender plugin, you can go to the RHQ repository on GitHub.

Saturday, April 26, 2014

RHQ and JBoss ON at Summit 2014

I was very fortunate to be able to attend this years Red Hat Summit in San Francisco.

Right before Summit, the accompanying DevNation conference already started on Sunday with some pretty interesting talks - among others about Fabric8 and

As both Summit and DevNation happened at Moscone center, it was pretty easy to mix and match sessions. Dev Nation also featured a "hack area": some tables that had huge power strips in the middle to easily connect laptop and smartphone chargers. And usually the table were not as empty as on the next image:

I also met at one of those tables with Matze Wessendorf from the Aerogears team and we developed together an Alert Sender plugin for RHQ, that uses the Unified Push Sender to directly bring alerts from RHQ to the lock screen of the admins phones. The plugin only exists in private git at the moment. We need to do some polishing and remove some passwords before pushing to rhq git.

Summit presentation
Thomas Segismont and I did a presentation "What is new with JBoss ON" and we also had a BOF session later on.
The presentation ran well with around 60 attendees. Most of the audience already knew JBoss ON.
Of those around 2/3 are running 3.1, a few 3.2 and one person was even on 3.0 (unfortunately I was not able to find him later to find out why)

The slides of the presentation are already available online; unfortunately they do not cover the two live demos from Thomas, showing how to add a new Storage Node and how to make use of the new bundle permissions that were introduced in JBoss ON 3.2 (and also RHQ).

IMG 4188

Thomas at work and me doing a bit of talking.


The BOF/Meet and Greet session later was also quite well attended.
People had good questions and ideas and we were talking a bit about roadmap and also about some vision wrt. JBoss ON 4. Unfortunately we got kicked out the room much too early.

Thomas and I also had the luck to be invited by OCSystems, the makers of RTI that also runs on top of JBoss ON for a dinner at the Waterfront restaurant, from where we had an excellent view on to the bay bridge:

IMG 4163
Front row: Alan Santos, myself, Thomas Segismont

Middle: Tobias Hartwig, Bill Critch, all Red Hat

Behind: Steve North, Georgia Ferretti, both OCSystems


And last but not least at the Dev Nation Hackathon, Team "RHQ" won the 2nd prize with some "home automation":
a RaspberryPI running the agent had an LED blinking and we had also one of the MBed boards (Those are developer boards a little like Arduino but with an ARM cpu and sensors + an LCD on board)
connected to read in sensor data; unfortunately I lost some time at the start of the hackathon,
so we could not really use that info in the plugin; 1/2h more time and ... :-)
For the 2nd half of this, Thomas has built a Cordova application for his Android phone and showed how to receive push messages that were sent when alerts are fired with the brand new aerogear-ups alert sender plugin mentioned above.

I should have taken a photo of all the wiring we had set up, but totally forgot
about it.

I will re-create / finish this demo and blog about it

Thursday, February 13, 2014

Running the RHQ-agent on a Raspberry PI [updated]

I finally got a Raspberry Pi too. After the hurdles of initial installation, got it hooked up to my LAN and of course I had to install an RHQ agent on it.

And it turned out that this was dead simple, as the Pi already has Java 1.7 installed (in the Raspbian Wheezy distro that I am using). Thus it was only a matter of laying down the rhq-agent, and starting it the usual way.

Now there was one caveat: the agent did not find any file systems or network interfaces etc. This is due to the fact that there is no native library for Sigar on arm v6 cpus supplied with the agent.

I cloned Sigar from its git repository, changed into the 1.6 branch and built that library myself.

Now after dropping into the agent's lib/ directory, the native library is available and on agent restart all the native stuff could be found.

Screenshot of platform details in RHQ
(Platform details in RHQ)

If you don't want to compile that library yourself, you can take my version from

I will try to get that library into the upcoming RHQ 4.10 release, but can't promise anything.


If you run the agent from current master (or upcoming RHQ 4.10), you can configure a list of plugins to be enabled, so that the agent only uses these plugins (and thus uses less memory and starts faster).
This property can be found in the file conf/agent-configuration.xml:

<entry key="rhq.agent.plugins.enabled" value="Platforms,JMX,RHQAgent"/>

The entries are a comma separated list of plugin (short) names. To determine those, you can run
plugins info at the agent command prompt:

> plugins info
Details of the plugins that are currently installed:

Plugin Name: RHQAgent
Display Name: RHQ Agent
Last Updated: 14. Februar 2014 11:03:35 MEZ
File Size: 51.558 bytes
MD5 Hashcode: f7eb7577af667ee4883437230e4b2d8c

Summary of installed plugins:
[RHQAgent, Platforms, JMX]

The short names are the ones encoded as "Plugin Name" and which are also shown on the summary line. There has actually been a property to disable unwanted plugins for a longer time, but just enabling the ones needed is probably easier.

The other thing you should do it to remove the -Xms setting in the script- the default of a 64MB minimum heap is just too large here.

With those 3 plugins above and the removed Xms setting, my agent has a committed heap of ~14MB and a used heap of ~11MB. A dump is/was 4MB in size.

P.S.: “Raspberry Pi" is a trademark of the Raspberry Pi Foundation