Hawkular at GrafanaCon

I had the pleasure to be invited to present the Hawkular eco-system at GrafanaCon 2016.

The venue for this 2-day conference was certainly not your everyday conference venue in a hotel ballroom or cinema, but a lot more heavy-metal:

USS Intrepid

And thus the sessions of the first day all happened in the on-board theatre which used to be one of the elevators that moved the aircrafts from the hangar to the flight deck.

Former aircraft elevator

The sessions were kicked off by Torkel Odegaard, the creator of Grafana by giving some numbers about the growth and versions of Grafana and the community of users and contributors.

In the next session Kevin from Fermilab talked about lasers and how they are connected to Grafana (hint: monitoring of the huge infrastructure that monitors the collider experiments).

And then next was something that probably most have been waiting for: the official launch of Grafana 4 by Torkel. This included a demonstration of some new features like Alerting, where you can define alerts directly on a graph including a visual value-picker and a simulation mode. The announcements continued later by announcing the renaming of Raintank to GrafanaLabs and the completion of the Stack with Intel Snap, which was announcing its version 1.0 followed by Graphite doing the same

I am not going through all the sessions here, but still want to mention the presentation of Kyle Brandt, creator of Bosun alert engine. His talk was less of a product presentation, but rather a philosophical one on getting the communication right. If one engineer sets up an alert trigger and another one has to work on the fired alert, it is important that the later gets enough context to be able to quickly react to the alert.

The afterparty in the evening happened in a nearby bowling place. Those balls are a lot larger and heavier than those we use in Germany for Kegeln.

The second day had a format with 2 parallel sessions, which was a lot more "how-to" like, which included a good presentation by Brian Brazil of Promtheus fame and a nice (hi)story of monitoring at Sony PlayStation. My talk took place on the afternoon at 3pm. It went well and I got some good questions and feedback.

As this was the first day with better weather I also toured the flight deck and the bridge of the Intrepid

Bridge of the Intrepid

Lockheed A1

Space Shuttle Enterprise

View from the bridge

All talks were recorded and will be pushed online once the video team has edited in the slides. My Slides are available in the meantime from http://www.pilhuhn.de/GrafanaCon2016.pdf. I will update this post once the recordings are online.

A DSL for Alert Trigger Definitions in Hawkular

Hawkular had for a while a UI with the possibility to set up Alert Triggers. As the name suggests are those triggers used to define conditions when an Alert is to be fired.

Since a while there is now the ManageIQ UI that allows to set up such triggers. And also Hawkular APM is now able to forward data into Alerting.

The other day I was doing some testing and a colleague asked me if I had already defined some triggers. I thought that I neither want to log into ManageIQ right now nor pass JSON structures via curl commands.

As I did some DSL work for metrics recently, I thought, why not set up a DSL for trigger definitions. This is work in progress right now and here are two examples

Set up a threshold trigger to fire when the value of _myvalue_ is > 3:

define trigger "MyTrigger"
 enabled
 ( threshold "myvalue" > 3 )
 auto-disable

Set up a trigger on availability when it is reported as DOWN. The trigger is not enabled.

define trigger "MyTrigger"
  ( availability "mymetric" is DOWN )

As with my metric DSL I am implementing this in Ruby with the help of Treetop. And likewise I am integrating this in HawkFX.

At the moment it is a very crude integration via entry points in the main menu. And the DSL itself is also far from ready. I consider this an experimentation space. If it turns our successful, it may be possible to take the grammar and directly integrate it into Hawkular-Alerts, so that one can directly POST a document with a DSL fragment, which then gets turned into the internal representation.

If you are looking for code, this is available in the alert_insert branch of HawkFX.

Computed metrics for HawkFX (updated)

Computed metrics are something I wanted to do for a very long time, already in RHQ, but never really got around it and sort of forgot about it again.

Lately I found a post that contained a DSL to do exactly this (actually you should read that post not because of the DSL, but because of the idea behind it).

After seeing this, I got the idea on what to do and to include this in HawkFX, my pet project, which is an explorer for Hawkular.

HawkFx with the input window for formulae, that shows a formula and also a parser error.
The orange chart shows Non-Heap used, the redish one the heap usage of a JVM.

Formulae

Formulas are in a DSL that looks a bit like UPN, e.g. as in the following (I've shortened the metric ID for readability, more on them below):

(+ metric( "MI~...Heap Used" , "max")
   metric( "MI~...NonHeap Used", "max"))

to sum up two metrics (see also screenshot below). The 'metric' element gets two parameters, the metric id and also which of the aggregates that the server sends should be taken (in this case the max value) - this comes from the fact that we request the values to be put into 120 buckets by the server.

Or if you have the total amount of memory you could also subtract the used memory to get a graph of the remaining:

(- 1000000 metric( "MI~...NonHeap Used", "max"))

You could also get the total wait time for responses at a point in time when you multiply the average wait time with the number of visitors:

(* metric("MI~..ResponseTime","avg")
   metric("MI~..NumberVisitors","sum"))

Computed total memory usage

Summing up the metrics for 'Heap Used' and 'NonHeap Used' as shown above would then give you a nice graph of the total memory consumption of a JVM:

The green chart now shows to combined memory usage of Heap and Non-Heap, which is computed from the other two series. Orange and red are as above.

On metric IDs

Metric IDs are the IDs under which a metric is stored inside of Hawkular. The example here comes from an installation of Hawkular-services in Docker. If you just feed your metrics into Hawkular metrics, the IDs will looks like the ones you are using.

ID (upper) and path fields (lower) for a selected item in the tree

I have just pushed an update to HawkFX that provides the ID and path in their own fields at the bottom of the main window, so you can copy&paste them.

Future

I will talk more about the parser in an upcoming article. For now it is a personal playground to also better understand what is doable here. If this turns out to be successful I can imagine that the DSL could directly be incorporated into Hawkular-metrics so that the rules are available to all metrics clients.

It would of course be cool to have an editor for the formulas that allows to interactively pick metric IDs etc, but I doubt that I will get to this any time soon.

Gespalten

Mein Beitrag zum Comic-Collab von Schlogger zum Thema Gespalten.

(Vergrößern)

Thema für den 15. Oktober: Egoistisch
Mit dabei im September:
Schlogger
Skizzenblog
Isla Volante
Mrs. Frollein
Brausecomics
Rainer Unsinn
BuddelfischOnlinecomics
Karrakula
Mal-Gries
Jane War
Dramatized Depiction
Schaf und Verstand
Schoolpeppers
Schisslaweng
Badham

Sabotage

Mein Beitrag zum Comic-Collab von Schlogger zum Thema Sabotage.

(Vergrößern)

Ich denke Katzenhalter kennen die Situation wenn dringend etwas erledigt werden muss :-)

Thema für den 15. August: „Nichts Neues“
Mit dabei im Juli:
Schlogger
Skizzenblog
Badham
Isla Volante
Mic At Six
OnlineComics
Demystifikation
Karrakula
Dramatized Depiction
Jane War
Catnipsflavour
Schisslaweng
GoboPictures Rainer Unsinn
Schoolpeppers
Mal-Gries Rainking
Lisa Frühbeis

Using Hawkular-services via Docker [updated x3]

As you may know, we have started to create the Hawkular-Services distribution, which we try to build weekly. This distribution comes without embedded Cassandra, with no default user and also without a UI (we plan on re-adding a basic UI).

But you must not fear.

Running Hawkular-services is pretty easy via Docker.

The scenario

Let me start with a drawing of what I want to do here:

Setup of Hawkular-services via Docker

In this scenario I run a Docker daemon (which is extremely easy these days on a Mac thanks to DockerForMac (Beta)). On the daemon I run a Hawkular-services container, which talks to a Cassandra container over the Docker-internal network. On top of that I have two WildFly10 containers running ("HawkFly"), which have been instrumented with the Hawkular-agent.

Running

For the purpose to setup linking and data volumes I am using docker-compose. The following is the docker-compose.yml file used (for the moment all images are on my personal account):

# set up the wildfly with embedded hawkular-agent
hawkfly:
  image: "pilhuhn/hawkfly:latest"
  ports:
    - "8081:8080"
  links:
    - hawkular
# The hawkular-server
hawkular:
  image: "pilhuhn/hawkular-services:latest"
  ports:
    - "8080:8080"
    - "8443:8443"
    - "9990:9990"
  volumes:
    - /tmp/opt/data:/opt/data
  links:
    - myCassandra
  environment:
    - HAWKULAR_BACKEND=remote
    - CASSANDRA_NODES=myCassandra
# The used Cassandra container
myCassandra:
  image: cassandra:3.7
  environment:
    - CASSANDRA_START_RPC=true
  volumes:
    - /tmp/opt/data:/opt/data

To get started save the file as docker-compose.yml and then run:

$ docker-compose up hawkular

This starts first the Cassandra container and then the Hawkular one. If they do not yet exist on the system, they are pulled from DockerHub.

After Hawkular has started you can also start the HawkFly:

$ docker-compose up hawkfly

Update

Right now if you would directly do docker-compose up hawkfly the agent would not work as the hawkular server is not yet up and the agent would just stop. We will add some re-try logic to the agent pretty soon.

I have pushed a new version 0.19.2 of HawkFly that has the retry mechanism. Now it is possible to get the full combo going by only running

$ docker-compose up hawkfly

Running without docker-compose

On my RHEL 7 box, there is Docker support, but no docker-compose available. Luckily docker-compose is more or less a wrapper around individual docker commands. The following would be a sequence that gets me going (you have to be root to do this):

mkdir -p  /var/run/hawkular/cassandra
mkdir -p  /var/run/hawkular/hawkular
chcon -Rt svirt_sandbox_file_t /var/run/hawkular

docker run --detach --name myCassandra -e CASSANDRA_START_RPC=true \
    -v /var/run/hawkular/cassandra:/var/lib/cassandra cassandra:3.7

sleep 10

docker run --detach -v /var/run/hawkular/hawkular:/opt/data \
  -e HAWKULAR_BACKEND=remote -e CASSANDRA_NODES=myCassandra \
  -p 8080:8080 -p 8443:8443 --link myCassandra:myCassandra \
  pilhuhn/hawkular-services:latest

Looking forward

There is an open Pull-Request to the Hawkular-Services Docker build as a part of a release and make it available via DockerHub on the official Hawkular account.

With this PR you can do

$ mvn install
$ cd docker-dist
$ mvn docker:build docker:start

to get your own container built and run together with the C* one.

Open questions

Right now I put in the default user/password and if the agent inside the hawkular-container should be enabled at image build time. Going forward we need to find a way to pass those at the time of the first start. The same applies (probably even more) to SSL-Certificates.

Storing them inside the container itself does not work going forward, as this way they are lost when a newer version of the image is pulled and a new container is constructed from the newer image.

Introducing HawkFX

As I said before, I started playing with JRubyFX. And for me learning something new best works with a use case, so I started creating an inventory browser for Hawkular.

Why JRubyFX?

Let's first start with "What is JRubyFX" anyway? JRubyFX is JavaFX brought to the Ruby world by the means of JRuby. This means that you can implement UIs with the help of the JavaFX framework and use its components and tools to build the UI. The difference to plain JavaFX is though that all the implementation code is written in Ruby and run by JRuby on the JVM.

I was doing a bit of JavaFX in the past and I wanted to generate a standalone inventory browser for Hawkular. Now that I have been working with Ruby lately and we already have the Hawkular client gem, I thought I'd give JRubyFX a try.

And I have to say this is pretty cool.

Some screenshots

Login screen

Main screen with inventory browser (left) and metric chart

The main screen shows a tree view on the left that displays the feeds as top level elements. Opening a feed will show recursively the resources and metrics. Clicking on a metric gets it charted on the right side.

Alert and Event list

A menu item in the main screen opens the alerts browser that allows to peek at alerts and events in the system.

Like in the main screen, there is a context menu that will allow to view the raw object as shown below:

Raw display of an event

Custom components

The time range picker on the main screen and alert screen is a custom component, that was implemented once with a .fxml file and some Ruby code:

class TimePicker < Java::javafx::scene::layout::HBox
  include JRubyFX::Controller

  fxml 'TimePicker.fxml'

  def initialize(caller, callback)
[..]
end

Including it is pretty simple too:

    box = find '#alertEventTopBox'
    box.children.add time_picker(self, :set_time_range)

In the first line we find the HBox to add the picker and then just add it to the children of the box. Done.

Running and code

HawkFX is available on my GitHub account at https://github.com/pilhuhn/hawkfx. To run the tool you need JRuby 9

If you are using rvm you can select it via

rvm use jruby-9.0.5.0

install and use bundler to install the required gems

gem install bundler bundle install

then run

jruby hawkfx.rb

Enjoy! :-)

Presenting at ManageIQ Design Summit 2016

I have the luck to go to the ManageIQ Design Summit 2016 in Mahwah, New Jersey at the beginning of June. And not only that, but also to be able to present there (more about this in a moment).

You may be looking at the web site of ManageIQ or the Design summit and think "why the heck is the Java and JBoss guy talking at a Ruby conference"? And the answer is simple:

Hawkular and ManageIQ are collaborating on the future of Middleware management.

And so I will talk about "Adding middleware to the game", describing the current state of Red Hat middleware and monitoring with RHQ and Hawkular, the integration with ManageIQ that we are already working on. And also the path and vision going forward.

ManageIQ has all the knowledge about the operating system and infrastructure the Middleware servers are running on, where Hawkular only provides some basic information. ManageIQ on the other hand has no notion of applications yet, wich is the domain of Hawkular. ManageIQ can also be used to provision new VMs and containers with Middleware in them, which can then be monitored and managed by Hawkular.

If you are close by, join me :)

NB: while I am there I will try find some of the nearby Geocaches, as New Jersey is still missing on my list (as so many other states in he US :)

Welcome GSoC 2016 Students to JBoss

JBoss.Org has the luck of being selected as one of the mentoring organizations for this years issue of Google Summer of Code.

On Friday April 22nd, Google has announced the 10 students that will all work with JBoss. Those students with their project are:

• Idel Pivnitskiy: AeroGear WebPush and UnifiedPush Server integration
• rohitmohan96: Ceylon Markdown
• Lucas Werkmeister: Ceylon TypeScript Loader
• Samuel Richardson: Drools Rules in Minecraft
• Anton Gabov: Smart HTTP/2-based protocol for Infinispan
• Austin Ko: Hawkular-agent For Vert.x
• mincongh: Hibernate Search: JSR 352 batch job for re-indexing entities
• Anuj Garg: Improve existing Android client of Hawkular
• Tugba: Teiid HDFS Translator/Connector
• dimcho: Test scheduling for large test suites

We also want to thank all the other students for their in total over 70 proposals that they have submitted.

See also the GSoC project page for more details and the announcement post from Google for further information.

Playing with JRubyFX

I started looking at JRubyFX as a combination of using a proven UI solution with Ruby as the programming language and I have to admit that I am impressed with how little it takes to get a UI done -- especially when it is driven by FXML.

Unfortunately the learning curve is steep, as one has to learn JavaFX and the special way JRubyFX calls the JavaFX classes and methods. And unfortunately, Google is not returning too much results here.

Anyway: this little code already creates GUI and populates a TreeView with elements it pulls from a remote server.

require 'jrubyfx'
require 'hawkular_all'

fxml_root File.dirname(__FILE__)

class HawkFx < JRubyFX::Application

  def start(stage)
    with(stage, title: 'Hello World!', width: 800, height: 600) do
      fxml HawkFxController
      show
    end

    stage.show()
  end
end

class HawkFxController
  include JRubyFX::Controller
  fxml 'fxmain.fxml'

  def login # callback from the login button
    creds = { :username => 'jdoe' ,
              :password => 'password' }
    url = 'http://localhost:8080/hawkular/inventory'
    @inventory_client = Hawkular::Inventory::InventoryClient.new(url, creds)
    @tenant = @inventory_client.get_tenant
    @FXMLtextArea.text = "Tenant: #{@tenant}"
    feeds = @inventory_client.list_feeds

    show_initial_tree(feeds)

  end

  def show_initial_tree(feeds)
    tree_root = tree_item('Feeds') do
      feeds.each do |feed|
        item = tree_item(feed) # this already adds the item to the root
      end
    end
    # bind to the view from fxml
    @FXMLtreeView.setRoot(tree_root)

    tree_root.setExpanded true
  end
end

HawkFx.launch

Reacting on IoT data with Hawkular

In the first post I have been talking about how send IoT sensor data to the metrics subsystem of Hawkular and then how to register the metric in Hawkular so that it can be graphed in the console.

In this article I will talk about how the Hawkular alerts component (that is already available in Hawkular-full) can be used to react on incoming data and make an LED on an Arduino blink.

Arduino Uno with Ethernet shield and yellow LED

For the Arduino I have added a cheap Enc28j60 based Ethernet shield. There is a standard library available, that allows to easily set up a web server, which we are using (in fact the code is mostly from that example).

The new (full) setup is like this:

Setup with Arduino as actor

Alerting

Hawkular already has an alerting component built in, that allows to compare incoming values with thresholds and then invoke plugins to forward the fired alert via email, to irc channels and many more way of communication. The plugin we are going to use here is the webhook one. As in the standard Hawkular distribution only the email plugin is present we will need to install the webhook plugin first:

Check out Hawkular-alerts

git clone https://github.com/hawkular/hawkular-alerts.git
cd hawkular-alerts

Build hawkular alerts :

mvn -Pdev -DskipTests install

Now you can copy the plugin to the Hawkular-server

cd hawkular-alerts-actions-plugins
cd hawkular-alerts-actions-webhook
cp hawkular-alerts-actions-webhook.war \
   $HAWKULAR/modules/system/layers/hawkular/org/hawkular/nest/main/deployments/

As Hawkular does not pick this up automatically you need to restart the Hawkular server after copying the webhook.war over.

Set up alerting

I have modified the ruby script to pull in a configuration file (in YAML format):

---
16617927:40.176.91.120.5.0.0.125:
  :name: Living room
  :alert:
    :comparator: :gt
    :value: 25

The first line is the id of the metric to which the following lines apply to. Second line is the display name for the UI. The next section then sets up alerting.

Let's have a look how this looks in code (you can see the full code on GitHub)

Register the webhook to be used below:

@webhook_props = { url: 'http://172.31.7.177/',   # target server
                   method: 'POST' }               # http verb
@alerts_client.create_action 'webhook', "send-via-webhook-#{metric_id}", @webhook_props

Set up a trigger and a condition, trigger first

  t = Hawkular::Alerts::Trigger.new({})
  t.enabled = true
  t.id = "trigger_#{metric_id}"
  t.tags = { :resourceId => "#{feed}/#{res_id}" }

The tags tell the UI on which resource the Trigger applies and thus enables showing the alert in the Hawkular UI

Next the condition:

  c = Hawkular::Alerts::Trigger::Condition.new({})
  c.trigger_mode = :FIRING
  c.type = :THRESHOLD
  c.data_id = metric_id
  c.operator = alert[:comparator].to_s.upcase
  c.threshold = alert[:value]

And then finally as we do not only want the alert to show in the UI, but also be forwarded to our Arduino with the webhook, we need to attach the webhook to the Trigger. Plugin id is webhook and the action_id is what we have set up above.

  # Reference an action definition
  a = Hawkular::Alerts::Trigger::Action.new({})
  a.action_plugin = 'webhook'
  a.action_id = "send-via-webhook-#{metric_id}"
  t.actions.push a

And then we can create the trigger, which will be active immediately.

  @alerts_client.create_trigger t, [c], nil

Fire away

Now when a value above the threshold comes in, the webhook will be triggered, it will open a http connection to the Arduino and the LED will blink.

In addition the alert will also be shown in the Alert Center in the Hawkular UI:

List of high temperature alerts in the alert center.

More

Note that the above trigger definition is pretty simple and only has one comparator. Also the trigger fires each time a value above the threshold comes in. In a more real world scenario one would add some dampening that the trigger only fires when the measurement is a few times over the threshold for some interval. Or if you start your air-conditioning, you would set the trigger to auto-disable and then have a comparator to switch it back on after the temperature has fallen below a certain level.

There is even a lot more you can do with Hawkular Alerts past the above, including standalone deployment for embedding it in your own application.

Some further reading:

Hawkular Alerts for Developers -- pretty comprehensive documentation of Hawkular Alerts.
Introduction to AutoResolve triggers
Using Hawkular Alerts as a standalone engine

Send IoT data to Hawkular-full

In a previous blog post, I was talking about how to send IoT sensor data to Hawkular-metrics.

While this already works quite well, it also lacks the integration with other parts of Hawkular, namely Inventory and Alerting.

In this blog post I will talk about integration with Inventory and how to view the data in the Hawkular-UI. An upcoming article will then talk about Alerting.

I have modified the setup from the last post a bit:

New setup with a Ruby client

Instead of PTrans I've written a small Ruby client, that makes use of the Hawkular-Client-Ruby ruby gem (it needs version 0.2.1, that has not yet been published to RubyGems.org. In addition it uses a MQTT gem, which makes the code pretty short.

The Ruby client MQTT-bridge now listens on /hawkular/+ topics. Metric arriving on /hawkular/metrics are forwarded as such and registration messages on /hawkular/register are used to register the external resource like the ESP8622 micro controller in Hawkular inventory.

The following is an example registration message:

{"feed": "esp16617927",
  "rt":"esp8266",
  "r":"mcu16617927",
  "mt":{
    "id":"thermo",
    "unit":"NONE",
    "type":"GAUGE",
    "collectionInterval":0
  },
  "m":{
    "id":"16617927:40.176.91.120.5.0.0.125",
    "mt":"thermo",
    "na":"thermo_40.176.91.120.5.0.0.125",
  }
}

You can get the client from https://github.com/pilhuhn/hawkular-mqtt-bridge and then easily run it via

$ ruby lib/hawk.rb

If you are not using a Hawkular-Server on localhost with default user, then you first need to modify the ruby code at the top.

The client will connect to the Hawkular server and then wait for messages on the MQTT topics.

The client code is still rather simplistic and does not spool data when the target Hawkular server is down.

UI

Since Hawkular 1 Alpha11 we have an Explorer (as easter egg :-) that allows to browse through inventory and to show resources and their metrics. The following shows the explorer with a chart from the thermo sensor for the last 12h.

Hawkular UI with chart of sensor data from last 12h

The chart shows a peak at 1am - I have no idea why it is there. Possibly one of my cats examined the sensor :)

ESP Sample code

I have also provided a sample Lua script, that can be used on a ESP8266 like the Adafruit Huzzah shown in the previous post.

As the stock Huzzah comes with NodeMCU Lua 0.9.5 which was not working well for me, I have flashed it with a newer version of NodeMCU Lua. This is now running a firmware with the following modules:

NodeMCU custom build by frightanic.com
	branch: master
	commit: c8037568571edb5c568c2f8231e4f8ce0683b883
	SSL: false
	modules: cjson,file,gpio,i2c,mqtt,net,node,ow,rtcmem,rtctime,sntp,tmr,uart,wifi,ws2812
 build 	built on: 2016-02-18 08:33

Alerting?

In an upcoming post I will talk about alerting in Hawkular to act on unusually high or low temperatures.

Flashing new NodeMCU Lua firmware to Adafruit Huzzah ESP

Adafruit delivers the Huzzah ESP with NodeMCU 0.9.5, which is pretty old.

So I was thinking of a newer version. The process is pretty well described on nodemcu.readthedocs.org. Unfortunately it also says

The address for esp_init_data_default.bin depends on the size of your module's flash. ESP-01, -03, -07 etc. with 512 kByte flash require 0x7c000. Init data goes to 0x3fc000 on an ESP-12E with 4 MByte flash.

After digging through Adafruit docs (they do not explicitly mention the type of ESP) and also looking at the output of node.info() I figured out that the Huzzah ESP has an ESP12 with 4MB

I got my self a new image from the awesome on demand build service and then flashed it like this:

$ python esptool.py --port /dev/cu.usbserial-AI02CSDU \
   write_flash 0x00 ../nodemcu-master-15-modules-2016-02-18-08-33-47-integer.bin \
               0x3fc000 ../esp_iot_sdk_v1.4.0/bin/esp_init_data_default.bin
Connecting...
Erasing flash...
Took 2.12s to erase flash block
Wrote 423936 bytes at 0x00000000 in 45.8 seconds (74.1 kbit/s)...
Erasing flash...
Took 0.10s to erase flash block
Wrote 1024 bytes at 0x003fc000 in 0.1 seconds (85.4 kbit/s)...

Future updates of the firmware to newer versions no longer need to flash the esp_init_data_default.bin file.

Sending IoT sensor data to Hawkular-Metrics via MQTT

The other day I was writing about 'RHQ-Metrics and Grafana' and was describing how you can incorporate data from other system management agents.

Fast forward a bit and Raider is now called Twix and RHQ-Metrics has morphed to Hawkular-Metrics under the Hawkular.org umbrella.

Recently I have also been playing with Arduino and Co. and got myself also an Adafruit Huzzah ESP8266 board. This is a breakout board with the ESP8266 microprocessor on, that has a bunch of IO pins and built-in WiFi. With the default firmware it is programmed in Lua.

Huzzah on breakout board

While one can program the ESP from the Arduino IDE, I thought to give Lua a try (also to get a feel for the difference to the WiPy, that also comes with Python as a high level language). What is nice with the ESP and the NodeMCU firmware is that it already comes with support for networking, 1-wire, MQTT and more out of the box.

To get started I took the hello-world of IoT-sensors and hooked up a DS18B20 OneWire thermo sensor (for those old enough, I did that in the past with RHQ) up to the ESP and then have this communicate to a MQTT Broker (mosquitto).

Setup diagram

As said before we have with Ptrans a universal protocol translator that can be used to feed data from collected, ganglia and others into Hawkular-Metrics. I've taken that and added support for MQTT (in my personal repo for now). Ptrans will now connect to a broker and listen on the '/hawkular/metrics' topic for data that needs to be in graphite format like

path value [timestamp_in_s]

The timestamp is optional, as in my case I was not able to get any real time clock data from the micro controller (there seem to be variants that have a clock on board).

To see the data that is coming from the device I can just run

$ mosquitto_sub  -t /hawkular/+
16617927:40.176.91.120.5.0.0.125 24.625
16617927:40.176.91.120.5.0.0.125 23.9375
16617927:40.176.91.120.5.0.0.125 23.8750

So here NodMCU with ID 16617927 and thermo sensor 40.176.91.120.5.0.0.125 is reporting around 24 deg Celsius.

I will post more on the topic in a laster posting.

References:

• A simple connected object with NodeMCU and MQTT
• NodeMCU documentation
• MQTT: Building an Open Internet of Things (in Arduino lang)
• WiFi Temperature Sensor with NodeMCU/ESP8266 (in Arduino lang)
• MQTT Ruby Gem

Arduino powered X-Mas tree

In 2014 we got two cats and decided not to have a classic christmas tree. I also started playing with Arduino and other gadgets, so this made me think to have an Arduino powered tree in 2015. In November my daughter and I started to build the tree.

The tree has 14 RGB LEDs (basically WS2812 ones) from Sparkfun (or EXP-Tech).
Pinout of the LEDs is as follows:

LED-Pinout (click to enlarge)

.
With that, the tree wiring got relatively easy by bending the LED pins in the four different directions and then connecting all 5 Volt pins with the red wire, GND with the black and chaining data in and out either by directly soldering the pins or via the yellow wire:

Detail view:

The overall schematic then looks like this (I am using an Arduino at the moment, but will use a TrinketPro in the future):

Schematic (click to enlarge)

I've added a photo resistor and a white diode to provide some lighting of the tree front, as at night, the RGB LEDs are so light, that the tree itself can't be seen. The resistor value also serves to dim the RGB LEDs at night.
The software driving the LEDs is basically the Strandtest one that Adafruit delivers as example for their Neopixel Library. I've modified the code a bit to also show other patterns.
In the future we will add a stand for the tree that will also house the electronics and will also host some additional surprise that I will talk about in a future post :)

[ UPDATE 2016-12-11 ]

I've added a stand and the dedicated TrinketPRO: