Tuesday, October 18, 2016

A DSL for Alert Trigger Definitions in Hawkular

Hawkular had for a while a UI with the possibility to set up Alert Triggers. As the name suggests are those triggers used to define conditions when an Alert is to be fired.

Since a while there is now the ManageIQ UI that allows to set up such triggers. And also Hawkular APM is now able to forward data into Alerting.

The other day I was doing some testing and a colleague asked me if I had already defined some triggers. I thought that I neither want to log into ManageIQ right now nor pass JSON structures via curl commands.

As I did some DSL work for metrics recently, I thought, why not set up a DSL for trigger definitions. This is work in progress right now and here are two examples

Set up a threshold trigger to fire when the value of _myvalue_ is > 3:

define trigger "MyTrigger"
 enabled
 ( threshold "myvalue" > 3 )
 auto-disable

Set up a trigger on availability when it is reported as DOWN. The trigger is not enabled.

define trigger "MyTrigger"
  ( availability "mymetric" is DOWN )

As with my metric DSL I am implementing this in Ruby with the help of Treetop. And likewise I am integrating this in HawkFX.

Inserting a definition

At the moment it is a very crude integration via entry points in the main menu. And the DSL itself is also far from ready. I consider this an experimentation space. If it turns our successful, it may be possible to take the grammar and directly integrate it into Hawkular-Alerts, so that one can directly POST a document with a DSL fragment, which then gets turned into the internal representation.

If you are looking for code, this is available in the alert_insert branch of HawkFX.

Tuesday, September 27, 2016

Computed metrics for HawkFX (updated)

Computed metrics are something I wanted to do for a very long time, already in RHQ, but never really got around it and sort of forgot about it again.

Lately I found a post that contained a DSL to do exactly this (actually you should read that post not because of the DSL, but because of the idea behind it).

After seeing this, I got the idea on what to do and to include this in HawkFX, my pet project, which is an explorer for Hawkular.

HawkFX screen shot
HawkFx with the input window for formulae, that shows a formula and also a parser error.
The orange chart shows Non-Heap used, the redish one the heap usage of a JVM.

Formulae

Formulas are in a DSL that looks a bit like UPN, e.g. as in the following (I've shortened the metric ID for readability, more on them below):

(+ metric( "MI~...Heap Used" , "max")
   metric( "MI~...NonHeap Used", "max"))

to sum up two metrics (see also screenshot below). The 'metric' element gets two parameters, the metric id and also which of the aggregates that the server sends should be taken (in this case the max value) - this comes from the fact that we request the values to be put into 120 buckets by the server.

Or if you have the total amount of memory you could also subtract the used memory to get a graph of the remaining:

(- 1000000 metric( "MI~...NonHeap Used", "max"))

You could also get the total wait time for responses at a point in time when you multiply the average wait time with the number of visitors:

(* metric("MI~..ResponseTime","avg")
   metric("MI~..NumberVisitors","sum"))

Computed total memory usage

Summing up the metrics for 'Heap Used' and 'NonHeap Used' as shown above would then give you a nice graph of the total memory consumption of a JVM:

Chart with computed metric
The green chart now shows to combined memory usage of Heap and Non-Heap, which is computed from the other two series. Orange and red are as above.

On metric IDs

Metric IDs are the IDs under which a metric is stored inside of Hawkular. The example here comes from an installation of Hawkular-services in Docker. If you just feed your metrics into Hawkular metrics, the IDs will looks like the ones you are using.

ID and path field
ID (upper) and path fields (lower) for a selected item in the tree

I have just pushed an update to HawkFX that provides the ID and path in their own fields at the bottom of the main window, so you can copy&paste them.

Future

I will talk more about the parser in an upcoming article. For now it is a personal playground to also better understand what is doable here. If this turns out to be successful I can imagine that the DSL could directly be incorporated into Hawkular-metrics so that the rules are available to all metrics clients.

It would of course be cool to have an editor for the formulas that allows to interactively pick metric IDs etc, but I doubt that I will get to this any time soon.

Thursday, September 15, 2016

Gespalten

Mein Beitrag zum Comic-Collab von Schlogger zum Thema Gespalten.

Thema für den 15. Oktober: Egoistisch
Mit dabei im September:
Schlogger
Skizzenblog
Isla Volante
Mrs. Frollein
Brausecomics
Rainer Unsinn
BuddelfischOnlinecomics
Karrakula
Mal-Gries
Jane War
Dramatized Depiction
Schaf und Verstand
Schoolpeppers
Schisslaweng
Badham

Friday, July 15, 2016

Sabotage

Mein Beitrag zum Comic-Collab von Schlogger zum Thema Sabotage.

Ich denke Katzenhalter kennen die Situation wenn dringend etwas erledigt werden muss :-)

Thema für den 15. August: „Nichts Neues“
Mit dabei im Juli:
Schlogger
Skizzenblog
Badham
Isla Volante
Mic At Six
OnlineComics
Demystifikation
Karrakula
Dramatized Depiction
Jane War
Catnipsflavour
Schisslaweng
GoboPictures Rainer Unsinn
Schoolpeppers
Mal-Gries Rainking
Lisa Frühbeis

Monday, June 27, 2016

Using Hawkular-services via Docker [updated x3]

As you may know, we have started to create the Hawkular-Services distribution, which we try to build weekly. This distribution comes without embedded Cassandra, with no default user and also without a UI (we plan on re-adding a basic UI).

But you must not fear.

Running Hawkular-services is pretty easy via Docker.

The scenario

Let me start with a drawing of what I want to do here:

Hawk service via docker
Setup of Hawkular-services via Docker

In this scenario I run a Docker daemon (which is extremely easy these days on a Mac thanks to DockerForMac (Beta)). On the daemon I run a Hawkular-services container, which talks to a Cassandra container over the Docker-internal network. On top of that I have two WildFly10 containers running ("HawkFly"), which have been instrumented with the Hawkular-agent.

Running

For the purpose to setup linking and data volumes I am using docker-compose. The following is the docker-compose.yml file used (for the moment all images are on my personal account):

# set up the wildfly with embedded hawkular-agent
hawkfly:
  image: "pilhuhn/hawkfly:latest"
  ports:
    - "8081:8080"
  links:
    - hawkular
# The hawkular-server
hawkular:
  image: "pilhuhn/hawkular-services:latest"
  ports:
    - "8080:8080"
    - "8443:8443"
    - "9990:9990"
  volumes:
    - /tmp/opt/data:/opt/data
  links:
    - myCassandra
  environment:
    - HAWKULAR_BACKEND=remote
    - CASSANDRA_NODES=myCassandra
# The used Cassandra container
myCassandra:
  image: cassandra:3.7
  environment:
    - CASSANDRA_START_RPC=true
  volumes:
    - /tmp/opt/data:/opt/data

To get started save the file as docker-compose.yml and then run:

$ docker-compose up hawkular
This starts first the Cassandra container and then the Hawkular one. If they do not yet exist on the system, they are pulled from DockerHub.

After Hawkular has started you can also start the HawkFly:

$ docker-compose up hawkfly

Update

Right now if you would directly do docker-compose up hawkfly the agent would not work as the hawkular server is not yet up and the agent would just stop. We will add some re-try logic to the agent pretty soon.

I have pushed a new version 0.19.2 of HawkFly that has the retry mechanism. Now it is possible to get the full combo going by only running

$ docker-compose up hawkfly

Running without docker-compose

On my RHEL 7 box, there is Docker support, but no docker-compose available. Luckily docker-compose is more or less a wrapper around individual docker commands. The following would be a sequence that gets me going (you have to be root to do this):

mkdir -p  /var/run/hawkular/cassandra
mkdir -p  /var/run/hawkular/hawkular
chcon -Rt svirt_sandbox_file_t /var/run/hawkular

docker run --detach --name myCassandra -e CASSANDRA_START_RPC=true \
    -v /var/run/hawkular/cassandra:/var/lib/cassandra cassandra:3.7

sleep 10

docker run --detach -v /var/run/hawkular/hawkular:/opt/data \
  -e HAWKULAR_BACKEND=remote -e CASSANDRA_NODES=myCassandra \
  -p 8080:8080 -p 8443:8443 --link myCassandra:myCassandra \
  pilhuhn/hawkular-services:latest

Looking forward

There is an open Pull-Request to the Hawkular-Services Docker build as a part of a release and make it available via DockerHub on the official Hawkular account.

With this PR you can do

$ mvn install
$ cd docker-dist
$ mvn docker:build docker:start

to get your own container built and run together with the C* one.

Open questions

Right now I put in the default user/password and if the agent inside the hawkular-container should be enabled at image build time. Going forward we need to find a way to pass those at the time of the first start. The same applies (probably even more) to SSL-Certificates.

Storing them inside the container itself does not work going forward, as this way they are lost when a newer version of the image is pulled and a new container is constructed from the newer image.

Monday, May 09, 2016

Introducing HawkFX

As I said before, I started playing with JRubyFX. And for me learning something new best works with a use case, so I started creating an inventory browser for Hawkular.

Why JRubyFX?

Let's first start with "What is JRubyFX" anyway? JRubyFX is JavaFX brought to the Ruby world by the means of JRuby. This means that you can implement UIs with the help of the JavaFX framework and use its components and tools to build the UI. The difference to plain JavaFX is though that all the implementation code is written in Ruby and run by JRuby on the JVM.

I was doing a bit of JavaFX in the past and I wanted to generate a standalone inventory browser for Hawkular. Now that I have been working with Ruby lately and we already have the Hawkular client gem, I thought I'd give JRubyFX a try.

And I have to say this is pretty cool.

Some screenshots

login screen
Login screen
Main screen with chart
Main screen with inventory browser (left) and metric chart

The main screen shows a tree view on the left that displays the feeds as top level elements. Opening a feed will show recursively the resources and metrics. Clicking on a metric gets it charted on the right side.

Alerts and Events
Alert and Event list

A menu item in the main screen opens the alerts browser that allows to peek at alerts and events in the system.

Like in the main screen, there is a context menu that will allow to view the raw object as shown below:

Raw event display
Raw display of an event

Custom components

The time range picker on the main screen and alert screen is a custom component, that was implemented once with a .fxml file and some Ruby code:

class TimePicker < Java::javafx::scene::layout::HBox
  include JRubyFX::Controller

  fxml 'TimePicker.fxml'

  def initialize(caller, callback)
[..]
end

Including it is pretty simple too:

    box = find '#alertEventTopBox'
    box.children.add time_picker(self, :set_time_range)

In the first line we find the HBox to add the picker and then just add it to the children of the box. Done.

Running and code

HawkFX is available on my GitHub account at https://github.com/pilhuhn/hawkfx. To run the tool you need JRuby 9

If you are using rvm you can select it via

rvm use jruby-9.0.5.0

install and use bundler to install the required gems

gem install bundler bundle install

then run

jruby hawkfx.rb

Enjoy! :-)

Tuesday, May 03, 2016

Presenting at ManageIQ Design Summit 2016

I have the luck to go to the ManageIQ Design Summit 2016 in Mahwah, New Jersey at the beginning of June. And not only that, but also to be able to present there (more about this in a moment).

You may be looking at the web site of ManageIQ or the Design summit and think "why the heck is the Java and JBoss guy talking at a Ruby conference"? And the answer is simple:

Hawkular and ManageIQ are collaborating on the future of Middleware management.

And so I will talk about "Adding middleware to the game", describing the current state of Red Hat middleware and monitoring with RHQ and Hawkular, the integration with ManageIQ that we are already working on. And also the path and vision going forward.

ManageIQ has all the knowledge about the operating system and infrastructure the Middleware servers are running on, where Hawkular only provides some basic information. ManageIQ on the other hand has no notion of applications yet, wich is the domain of Hawkular. ManageIQ can also be used to provision new VMs and containers with Middleware in them, which can then be monitored and managed by Hawkular.

If you are close by, join me :)

NB: while I am there I will try find some of the nearby Geocaches, as New Jersey is still missing on my list (as so many other states in he US :)