tag:blogger.com,1999:blog-60800822024-03-19T04:14:33.027+01:00Some things to rememberRandom thoughts about random things.Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.comBlogger563125tag:blogger.com,1999:blog-6080082.post-51494417928127368342018-11-09T09:34:00.001+01:002018-11-10T12:34:07.996+01:00L2tp IPSec VPN with Mikrotik<p>I recently discovered the VPN feature on my Android phone and though let me try this to set up a VPN with my home gateway, which is a MikroTik router.</p>
<p>I followed a few guides on the internet, but this just did not work, Android just gave me a short "does not work" message and that's it. So I dropped the ball again.</p>
<p>Now I am going to talk at a conference and I thought, I could set up the scenario on my BigBox at home instead of the Laptop. But how do I expose that? Well, perhaps I should use a VPN. So I googled "MikroTik macOS VPN" and found a few <a href="http://www.geordy.nl/use-a-macios-to-vpn-to-a-mikrotik-router/">pages that also repeated what I had set up</a> before. I tried again, no luck. Did some <a href="https://wiki.mikrotik.com/wiki/Manual:Tools/Packet_Sniffer">packet sniffing on the router</a>. No luck.</p>
<p>And then I got a thought: what if my provider filters udp port 500? Googled around, does not seem so. But it brought me to another idea:</p>
<div style="text-align:center;"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5qnc59FFxa9TG1gG2vKMurVqOHRNYEe3-AvcAjbtPXvYMyjbKQjqGACvQCRsCeo9_pzbdC22PH2yJ66Yt8m9elm9auJPmpw5f5Q91HzTwZX6vlkxF5C9pYQY_FUvlvu6ThfTl/s0/mySetup.png" alt="MySetup" border="0" /></div>
<p>If you look at the above setup, you can see that my MikroTik is not directly connected to the internet, but there is a provider router in-between. And this guy also has a firewall and is filtering out ports. In the past I had poked some holes into this (but I log forgot since), but udp port 500 and others needed for IPsec were not forwarded to the MikroTik.</p>
<p>Once I enabled forwarding of these ports, both VPN on macOS and Android started to work like a charm. :-)</p>
<p>One thing I still had to enable was proxy-arp as described in the above guide to be able to reach not only the router, but also my real target host on my LAN.</p>
<p>I have after the good setup played a bit with the configuration (and not put it under version control and everything failed with "no IKEv1 peer config for".
Solution was that I had a local address in the <code>/ip ipsec peer</code> and this meant that MikroTik would only allow for connections to that specific IP, which failed.</p>
<p>Another good guides on the topic:<br/>
<ul><li><a href="https://itimagination.com/mikrotik-l2tp-ipsec-dedicated-vpn-appliance-setup/">https://itimagination.com/mikrotik-l2tp-ipsec-dedicated-vpn-appliance-setup/</a></li>
<li><a href="https://manuth.life/l2tpipsec-vpn-server-mikrotik-routeros/">https://manuth.life/l2tpipsec-vpn-server-mikrotik-routeros/</a></li>
<li><a href="https://wiki.mikrotik.com/wiki/Manual:IP/IPsec">MikroTik docs</a>
<a href="https://jcutrer.com/howto/networking/mikrotik/ios-ikev2-vpn-mikrotik">https://jcutrer.com/howto/networking/mikrotik/ios-ikev2-vpn-mikrotik</a></li>
<li><a href="https://jcutrer.com/howto/networking/mikrotik/l2tp-over-ipsec-troubleshooting">Troubleshooting guide</a></li> </ul>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-37655378227967340672018-10-11T12:01:00.001+02:002018-10-11T12:01:29.576+02:00Solution for: "macOS, Dovecot and setgroups() failed: Too many extra groups"<p>I upgraded my home server to macOS 10.14 after the upgrade on my laptop went very smooth. Also here the upgrade went smooth, but in the evening I discovered that I did not get any new email.</p>
<p>I started investigating. First I saw that <a href="http://dovecot.org/">Dovecot</a> did not correctly start because an very old pid file said it has pid 104 and indeed there was a process wit pid 104, so dovecot though that is itself and stopped.</p>
<p>After removing the pid file and restarting Dovecot, it started again and correctly listened on port 993 (imaps). But the imap client still could not login.</p>
<p>Dovecot.log showed a lot of messages</p>
<pre>Oct 11 08:10:27 imap(hwr)<12659><YpC..>: Fatal: setgroups(501) failed: Too many extra groups</pre>
<p>Reading the manpage for <code>setgroups()</code> revealed that this call fails when more than <code>NGROUP_MAX</code> (=16) groups are passed. And indeed the user has more than 16 groups:</p>
<pre>$ id -G hwr
20 6 12 61 80 98 500 701 702 30 33 100 204 250 395 103 104</pre>
<p>Looking at them shows a lot of groups that macOS creates and assigns internally and which one can't (easily) delete</p>
<pre>$ id hwr
...
701(com.apple.sharepoint.group.1),702(com.apple.sharepoint.group.2),30(_keytabusers),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),103(com.apple.access_screensharing-disabled),104(com.apple.access_ssh-disabled)</pre>
<p>I <a href="https://dovecot.org/pipermail/dovecot/2018-October/113218.html">wrote to the dovecot mailing list</a> and after an email exchange it turned out that there is a config option already present that can help here to reduce the number of groups passed to <code>setgroups()</code>.</p>
<p>So the solution for me is to <code>last_valid_gid = 100</code> in <code>/usr/local/etc/dovecot/conf.d/10-mail.conf</code> and restart Dovecot.</p>
<p>And now it is again humming along nicely :)</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-39141189068917538032018-10-03T12:26:00.001+02:002018-10-03T12:26:48.210+02:00MikroTik: access control helpers<p>I have a MikroTik router and am overall pretty happy with it. Especially the Caps-Man feature for central administration of Wifi access points is great.</p>
<p>One thing I am a bit fighting with is restricting my kids from accessing the internet at certain times of the day.
The firewall offers quite some features, but I can't get it to cut existing connections when the rule is triggered by time (and this is also true for the dynamic rules of the kid control feature). But then perhaps I just don't understand routeros well enough.</p>
<p>What I have done is to set up access-lists for caps-man. When they are enabled, WiFi access is no longer possible. They kick in well at the right time, so all is good.</p>
<p>But: sometimes my kids get legitimate exceptions and when I then forget to re-enable the rule, they won't tell me :)</p>
<p>What I have done is created the following script, which is nightly triggered from the scheduler:</p>
<pre>:local maccaps
:foreach i in=[/caps-man access-list find ] do={
:set maccaps ( [/caps-man access-list get value-name=comment number=$i])
:if ($maccaps~ "^RESET .*\$") do={
/caps-man access-list enable numbers=$i
}
}</pre>
<p>This looks for access list entries with a comment, where the comment starts with "RESET ". If found the script just re-enables that access list rule.</p>
<p>Permissions needed are read, write and policy.</p>
<p>Please let me know if there are better options.</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-10625895616732693442018-03-29T10:32:00.001+02:002018-03-29T13:22:08.793+02:00Small helper for Mac OS/X to open Jira items<p>In our <a href="https://github.com/kiali">Kiali project</a> we develop at GitHub, but have the <a href="https://issues.jboss.org/projects/KIALI">issue tracker on a Jira instance</a>.</p>
<p>Developers put the jira ticket number into commit comments and pull-request headers. To see the Jira item I had so far to go to the Safari url bar and type in the issue link. Sometimes there was a different item already in the history, so I only had to change the key. But all in all that was tedious.</p>
<p>Luckily OS/X has tools like Automator, that allow to create your own services.</p>
<p>I launched Automator and selected to create a service. For details about this <a href="http://www.macosxautomation.com/services/learn/tut01/index.html">see this article</a>.</p>
<p>I then put a "Execute AppleScript" action into the Automator workflow</p>
<div style="text-align:center;"><img src="https:/lh3.googleusercontent.com/-aQLg5Y02AT8/WryjdadB9oI/AAAAAAAAGno/EHU1_52u9HwWbmixWEzBa2dQGJFczRMuQCHMYCw/s0/Bildschirmfoto%2B2018-03-29%2Bum%2B10.04.31.png" alt="automator workflow" border="0" width="399" /></div>
<p>Saving this whole workflow under "Open in JBoss Jira" then makes the service immediately available</p>
<p>So I can now select a Jira-key and do right-click to get the service</p>
<div style="text-align:center;"><img src="https:/lh3.googleusercontent.com/-v2HXqSHzip4/WrykPCdBvqI/AAAAAAAAGn0/Yy43FCxHag0-5Xeykjs08AvYXAGrgKCrACHMYCw/s0/Bildschirmfoto%2B2018-03-29%2Bum%2B10.22.17.png" alt="My service in the service context menu" border="0" width="399" /></div>
<p>Obviously the service is very much tailored to my need, so I'll put the AppleScript here so you can copy & modify it to your needs.</p>
<pre>on run {input, parameters}
set target to "https://issues.jboss.org/browse/" & (input as string)
tell application "Safari"
activate
tell window 1
set current tab to (make new tab with properties {URL:target})
end tell
end tell
end run
</pre>
<p>
<b>[ UPDATE ]</b>
<br/>
My colleague <a href="https://twitter.com/maxandersen">Max Andersen</a> pointed out an even better way of doing this by having some <a href="https://github.com/maxandersen/autolinkjboss">helpers turn those keys into links</a> that can directly be clicked.
</ü>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-17507030726146751172018-01-03T12:16:00.001+01:002018-01-14T11:22:52.244+01:00Homegrown XY plotter with Arduino<p>[ <i>This is work in progress and only for myself. If you are looking for ready-made kit, then have e.g. a look at the <a href="https://www.heise.de/make/artikel/MaXYposi-Projektseite-zum-universellen-XY-Portalroboter-von-Make-3676050.html">MaXYposi from German Make Magazine</a></i> ]</p>
<p>Back in school we had a HP plotter well hidden in some closet in the science department. I was able to play with it for a while and always had the desire to have one on my own (but why, where dot-matrix printers are so much versatile and useful).</p>
<div style="text-align:center;"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi159d52xmLJ3Q7Qi9MkoO2e43tZAIe-dwlq06oKCHWixkk-cqCkjuh4XM6Qo36s-7cwSkDBzcM3J_1CZwPvrqlAGn5spXOEVh72M1Gz5m5VYpz8Ro1VtrFr2KNtcCXGnAM06xn/?imgmax=1600" alt="Bildschirmfoto 2018 01 03 um 12 11 22" border="0" width="399" /><br/>
Photo from Plotter in action. <a href="https://www.dropbox.com/s/nkk9bnghxywijzf/TRIM_20180102_154921.mp4?dl=0">Video here</a>.</div>
<p>Fast forward many many years. Stepper motors are easily available, I am back into doing stuff with electronics and micro controllers and recently saw someone creating some displays with engraved acrylic. This triggered me to finally get started with the plotter (and later engraver etc).</p>
<p>To get going I got some parts and as I am an old-school 5V guy, I totally like the original Arduino UNO. In the list, I put in links for reference. I am not affiliated with those companies</p>
<ul>
<li><a href="http://www.watterott.com/de/Arduino-FabScan-Shield">FabScan shield</a> - physically hosts the Stepper motor drivers</li>
<li><a href="http://www.watterott.com/de/SilentStepStick-TMC2130">SilentStepSticks</a> - motor drivers, as the Arduino itself can't handle the voltage and current that a stepper motor needs. I am using the ones with a Trinamic TMC2130 chip, but in standalone mode for now. Those are replacements for the Pololu 4988, but allow for much quieter operation.</li>
<li><a href="http://www.watterott.com/de/SilentStepStick-Protector">StepStick protectors</a> - diodes that prevent the turning motor to fry your motor drivers (you want them, believe me)</li>
<li>Stepper motors - I took NEMA 17 ones with 12V (<a href="http://www.watterott.com/de/Schrittmotor-incl-Anschlusskabel">Watterott</a>, <a href="https://www.sparkfun.com/products/9238">Sparkfun</a>)</li>
<li><a href="https://www.ebay.de/itm/CNC-Set-12x-600mm-Linearfuhrung-Linear-Guide-Rail-Stage-3D-/322917927292?hash=item4b2f68897c">Linear guard rails</a></li>
<li>Wooden base plate</li>
<li>Wood, screws, ..</li>
<li>GT2 belt, <a href="http://www.watterott.com/de/OpenBuilds-GT2-2mm-Aluminium-Timing-Pulley">GT2 timing pulley</a></li>
</ul>
<p>As you can see in the next picture I started out much too big - I can't really have the plotter sit on my desk comfortably, but anyway :) I did it for learning purposes (and as I have to re-do some things, I'll also use smaller beams).</p>
<h2>Hardware side</h2>
<div style="text-align:center;"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi0Kl2XLSILf_g3wiYkurMJexKBzVDENHZxiYBI7JKja6FMN1QNIwV6rP5YOjZqsU4eYzpOrzRsHXOEAdARSeD8KWblnKn2pcffWWMYn7pnVeA6y0bpvHiu2ERCJ1Z43YdMzoP/?imgmax=1600" alt="IMG 20180101 185406" border="0" width="399" /><br/>
View of the plotter base plate with X-axis and Y-axis rails</div>
<p>The belt is mounted on both sides of the rail and then slung around the motor with some helper wheels:
<div style="text-align:center;">
<img src="https://photos-4.dropbox.com/t/2/AACiGH_dG0C4a2nhr1tZuCTrFa-h-g_xSInBZ3Z6oAI84Q/12/21627069/jpeg/32x32/3/1514991600/0/2/IMG_20180103_110206.jpg/ENGZnRAYrxcgAigC/e2JCYDTjaCR2OKfF-hdIDV8EihDdQqJ8fjAR7bU83QI?dl=0&size=1280x960&size_mode=3" alt="" border="0" width="400" height="" />
<br/>
Detail of the belt routing on the motor<br/>
</div></p>
<p>On the Arduino side there is a stack of things. On the bottom, the Arduino, then the FabScan shield, then on motor slots 1+2 a StepStick protector and then on top of them the SilentStepSticks. Note that the SCK and SDI pins are not connected</p>
<div style="text-align:center;"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKUiz6AXmUmiv4yswOqIwE4qk3o6yLRIdJa0vFB41Mctrf6yyZI73muv77L095OTIQUA5dEwGHm_SpCyTNlOT6VTXzl7toQLqtd_HpzqtINhC0v8Rbb9q-vaMhERWbPbL6pCyq/?imgmax=1600" alt="IMG 20180103 110111" border="0" width="400" /><br/>
Setup on Arduino (<a href="https://www.dropbox.com/s/7bp3bo5g2ujser8/IMG_20180103_110111.jpg?dl=0">see here for a larger image</a>)</div>
<p>Btw be careful with the wires to the motor to correctly attach them. When in doubt have a look at the data sheet or a ohm-meter to figure out which wires belong together.</p>
<h2>Software side</h2>
<p>I know that there is software like <a href="https://github.com/gnea/grbl">Grbl</a>, that can interpret so called G-codes for tool movement and other things and I could have just flashed that to the Arduino, but I am too curious for that and wanted to better understand things myself.</p>
<h3>Basics</h3>
<p>To drive a stepper motor with the StepStick driver (or compatibles) one basically needs to send a high and then a low signal to the respective pin. Or in Arduino-terms</p>
<code>
digitalWrite(stepPin, HIGH);<br/>
delayMicroseconds(30);<br/>
digitalWrite(stepPin, LOW);<br/>
</code>
<p>Where <code>stepPin</code> is the pin number for the stepper - 3 for motor 1 and 6 for motor 2.</p>
<p>Before the stepper does any work, it needs to be enabled.</p>
<code>
digitalWrite(enPin, LOW);
</code>
<p>Actually the StepStick knows three states for the pin here
<ul>
<li>LOW: motor is enabled</li>
<li>HIGH: motor is disabled</lI>
<li>pin not connected: motor is enabled but goes into some energy saving after a while</li>
</ul></p>
<p>When a motor is enabled, its coils are powered and it keeps its position. It is almost impossible to manually turn its axis. This is good for precision purposes, but also means that both motors and driver chips are "flooded" with power and will thus warm up.</p>
<p>And not last but not least, we need a way to determine the direction</p>
<code>
digitalWrite(dirPin, direction);
</code>
<p>The following table lists the functions and the pins</p>
<table>
<tr><th>Function</th><th>Motor1</th><th>Motor2</th></tr>
</th>
<tr><td>enable</td><td>2</td><td>5</td></tr>
<tr><td>direction</td><td>4</td><td>7</td></tr>
<tr><td>step</td><td>3</td><td>6</td></tr>
</table>
<p>Before we can use the pins, we need to set them to OUTPUT mode, best in the <code>setup()</code> section of the code</p>
<code>
pinMode(enPin1, OUTPUT);<br/>
pinMode(stepPin1, OUTPUT);<br/>
pinMode(dirPin1, OUTPUT);<br/>
digitalWrite(enPin1, LOW);
</code>
<p>With that knowledge we can easily get the stepper to move around:</p>
<pre>
totalRounds = ...
for (int rounds =0 ; rounds < 2*totalRounds; rounds++) {
if (dir==0){ // set direction
digitalWrite(dirPin2, LOW);
} else {
digitalWrite(dirPin2, HIGH);
}
delay(1); // give motors some breathing time
dir = 1-dir; // reverse direction
for (int i=0; i < 6400; i++) {
int t = abs(3200-i) / 200;
digitalWrite(stepPin2, HIGH);
delayMicroseconds(70 + t);
digitalWrite(stepPin2, LOW);
delayMicroseconds(70 + t);
}
}
</pre>
<p>This will make the slider move left and right.
When you execute that code, you can also see that this only deals with one stepper, but for a XY-plotter we have two axis to cater for.</p>
<h3>Command interpreter</h3>
<p>I started to implement a simple command interpreter that allows to use path specifications like</p>
<p><pre>"X30|Y30|X-30 Y-30|X-20|Y-20|X20|Y20|X-40|Y-25|X40 Y25</pre> to describe
relative movements in millimetres (internally 1mm equals 80 steps).</p>
<p>The plotter software implements a <i>continuous mode</i> which allows to feed large paths (in chunks) from a connected PC to the plotter. This how the <a href="https://twitter.com/pilhuhn/status/949737734654124032"">Hilbert-Curve in this tweet</a> has been plotted.</p>
<h2>Better pen holder</h2>
<p>In the first image above the pen was only tied to the y-Axis with some metal string. This was not precise and did also not allow to have the software raise and lower the hand (this explains the big black dots).</p>
<p>I have meanwhile created a better, more precise pen holder that is using a servo to raise and lower the pen. The original form of this new holder can be seen in the Hilbert-Curve video.</p>
<div style="text-align:center;"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7oNMb8jSuuOXx_Hskjh0XRIQqvKeXe64a312-k2F2fX8WbwjKGnLAzZod3ajW_jqK8hT2H5Bv7XXfAiwc2wVXqeJuu7J5bwvlgTzuhG9Ci_sUJsGwtypVZd4Qox2xtm883VGl/?imgmax=1600" alt="Servo to raise/lower the pen" border="0" width="399" /><br/>
Close view of the servo arm in the upper position raising the pen.
</div>
<p>The Pen gets a little clamp attached (the one shown is a size 8 one to attach cables to walls). The servo arm can then raise the pen. When the arm goes down, gravity will also lower the pen.</p>
<h3>Driving the servo</h3>
<p>Driving the servo is relatively straightforward. One has to only provide the position and the servo does all the work.</p>
<pre>
#include <Servo.h>
// Servo pin
#define servoData PIN_A1
// Positions
#define PEN_UP 10
#define PEN_DOWN 50
Servo penServo;
void setup() {
// Attach to servo and raise pen
penServo.attach(servoData);
penServo.write(PEN_UP);
}
</pre>
<p>I am using the servo headers on the Motor 4 place of the FabScan shield, so the pin to use is Analog pin number one.</p>
<p>Lowering the pen is then as easy as</p>
<pre>
penServo.write(PEN_DOWN);
</pre>
<h2>Next</h2>
<p>One of the next steps is to add some end-detectors (but I may skip them and use the StallGuard mode of the TMC2130).
Those detectors can then also be used to implement a home command</p>
<p>And then perhaps in the future I add a real z-Axis than can hold an engraver to do wood milling or PCB drilling or engraving of acrylic or .... :-) (A laser comes to mind as well).</p>
<h2>More ...</h2>
<p>That's it for now. The software is available at <a href="https://github.com/pilhuhn/xy-plotter">https://github.com/pilhuhn/xy-plotter</a> and comes without any warranty.</p>
<p>My original <a href="https://twitter.com/pilhuhn/status/948205323726344193">announcement tweet</a></p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-31204597998463126502017-09-29T16:04:00.001+02:002017-09-29T16:31:36.769+02:00Find the hidden exception with the IntelliJ debugger<p>I was hacking some code and running tests and got:</p>
<pre> testA(org.acme.ATest) Time elapsed: 0.114 sec <<< ERROR!
java.lang.RuntimeException: Could not inject members
Caused by: java.lang.NullPointerException</pre>
<p>No stack trace, no line number, nothing :-(</p>
<p>Attaching a debugger and making it stop on NPE also did not help, as this was slowing execution down to no avail.</p>
<p>Luckily IntelliJ IDEA has a nice feature to just log the exception when it happens.
Just go to the breakpoint view in the Debugger and tick the respective checkbox</p>
<div style="text-align:center;"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgC3YwgYBw1_uXtecX9LdtJVz9LcLGHwtwJ31Kg1uaqAPIPmhly2FnG88f9R43H_beNkT20S-IgFu8zu4Y2hSqW3qmviEVXdv4HgCmHTjSxc_E4a2nONOpx2xpA2k8CGm08l_s_/?imgmax=1600" alt="Bildschirmfoto 2017 09 29 um 16 16 46" border="0" width="493" /></div>
<p>You should also unmark the suspend checkbox as you don't want your porgram being interrupted all the time in places you don't want to know about (I had some hundreds of lines of NPEs within the JRE itself).</p>
<p>When you then attach the debugger to your process, you will see all NPEs along with the place they were thrown. And if the place is within your code, you can navigate to it via the log entry with one click.</p>
<p>I am sure other IDEs have a similar functionality.</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com1tag:blogger.com,1999:blog-6080082.post-64575386231640104122017-06-14T20:49:00.001+02:002017-06-15T10:23:37.237+02:00Was tun ohne Wald?<p>Mein Beitrag zum <a href="http://schlogger.de/comiccollab-069-wald/">Comic-Collab</a> von Schlogger zum Thema <em>Wald</em>.</p>
<div style="text-align:center;"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9Nnlue2pdfIhB1u61GnRPIvuuGEpqlGx82RqXA7y0rxjSrN-81pvS2EGDprFUv_-hYgWH_oqbLHzVcOCt9L19kSnZrf0d991hA6EcQZBzrURK6M7CLnELq7881_vfnMJSsMm9/?imgmax=1600" alt="Comic collab wald" border="0" width="600" /></div>
<p> </p>
<p><strong>Thema für den 15. Juli: „Schmerzen“</strong></p>
<p>Mit dabei im Juni 2017: (wird noch aktualisiert)</p>
<a href="http://schlogger.de/comiccollab-069-wald/">– Schlogger </a><br />
– <a href="http://nadiabader.blogspot.de/2017/06/comic-collab-069-wald.html" target="_blank">Skizzenblog</a><br />
– <a href="http://pilhuhn.blogspot.de/2017/06/was-tun-ohne-wald.html" target="_blank">pilhuhn</a><br />
– <a href="http://rainerunsinn.blogspot.de/2017/06/wald-vor-lauter-baumen.html" target="_blank">Rainer Unsinn</a><br />
– <a href="http://strichweise-diesig.blogspot.de/2017/06/strichweise-diesig-247-wald-beitrag-zu.html" target="_blank">Strichweise Diesig</a><br />
– <a href="https://demystifikation.wordpress.com/2017/06/15/comic-collab-69-wald/" target="_blank">Demystifikation</a><br />
– <a href="https://onlinecomics.wordpress.com/2017/06/15/comic-collab-69-thema-wald/" target="_blank">OnlineComics</a><br />
– <a href="http://pepperworth.blogspot.de/2017/06/comiccollab-wald.html" target="_blank">Pepperworth</a><br />
– <a href="http://www.dramatized.de/comic/wald/" target="_blank">Dramatized Depiction</a><br />
– <a href="https://www.schisslaweng.net/maenner-im-wald/" target="_blank">Schisslaweng</a></p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-82617610964042074272017-04-02T12:05:00.001+02:002017-04-02T12:05:41.849+02:00A beginners guide to DSL writing in Ruby<p>I wrote an article on <a href="https://developers.redhat.com/blog/2017/03/29/a-beginners-guide-to-dsl-writing-in-ruby/">writing (external) Domain Specific Languages in Ruby</a> over at the Red Hat Developers Blog.</p>
<p>This is based on my two posts on <a href="http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html">Computed Metrics for HawkFX</a> and <a href="http://pilhuhn.blogspot.com/2016/10/a-dsl-for-alert-trigger-definitions-in.html">A DSL for alert trigger definitions in Hawkular</a>.</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-37981501269311216372016-12-05T17:28:00.001+01:002016-12-05T20:36:18.653+01:00Hawkular at GrafanaCon<p><img align="right" src="https://lh3.googleusercontent.com/-u0O_fr-8enI/WEXAteSI5kI/AAAAAAAAFmY/JNdLMsHiyUI/grafanacon_logo_final.png?imgmax=1600" alt="Grafanacon logo final" border="0" width="200" />I had the pleasure to be invited to present the Hawkular eco-system at <a href="http://grafanacon.org">GrafanaCon 2016</a>.</p>
<p>The venue for this 2-day conference was certainly not your everyday conference venue in a hotel ballroom or cinema, but a lot more heavy-metal:</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-4EYy5drYy24/WEQFDTLQ2nI/AAAAAAAAFlM/eoyb7HigFTc/IMG_20161130_135818.jpg?imgmax=1600" alt="USS Intrepid" border="0" width="400" /><br/>
<a href="http://www.intrepidmuseum.org/AircraftCarrierIntrepid">USS Intrepid</a></div>
<p>And thus the sessions of the first day all happened in the on-board theatre which used to be one of the elevators that moved the aircrafts from the hangar to the flight deck.</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-SlSt88N-QSI/WEQJHlP5b7I/AAAAAAAAFlY/D9I5TtX82Q0/IMG_20161130_092133.jpg?imgmax=1600" alt="IMG 20161130 092133" border="0" width="400" />
<br/>
Former aircraft elevator</div>
<p>The sessions were kicked off by Torkel Odegaard, the creator of Grafana by giving some numbers about the growth and versions of Grafana and the community of users and contributors.</p>
<p>In the next session Kevin from Fermilab talked about lasers and how they are connected to Grafana (hint: monitoring of the huge infrastructure that monitors the collider experiments).</p>
<p>And then next was something that probably most have been waiting for: the official launch of Grafana 4 by Torkel. This included a demonstration of some new features like Alerting, where you can define alerts directly on a graph including a visual value-picker and a simulation mode. The announcements continued later by announcing the renaming of Raintank to GrafanaLabs and the completion of the Stack with Intel Snap, which was announcing its version 1.0 followed by Graphite doing the same</p>
<p>I am not going through all the sessions here, but still want to mention the presentation of Kyle Brandt, creator of Bosun alert engine. His talk was less of a product presentation, but rather a philosophical one on getting the communication right. If one engineer sets up an alert trigger and another one has to work on the fired alert, it is important that the later gets enough context to be able to quickly react to the alert.</p>
<p>The afterparty in the evening happened in a nearby bowling place. Those balls are a lot larger and heavier than those we use in Germany for Kegeln.</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-zPJAa7y0QYw/WERjXZ4NzHI/AAAAAAAAFls/XRHMVBN7Lxc/DSC_0226.JPG?imgmax=1600" alt="Intrepid by night" border="0" width="400" /></div>
<p>The second day had a format with 2 parallel sessions, which was a lot more "how-to" like, which included a good presentation by Brian Brazil of Promtheus fame and a nice (hi)story of monitoring at Sony PlayStation. My talk took place on the afternoon at 3pm. It went well and I got some good questions and feedback.</p>
<p>As this was the first day with better weather I also toured the flight deck and the bridge of the Intrepid</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-UCY7yp1hVd8/WERmrYB7WOI/AAAAAAAAFl0/TU-wCZcsthE/IMG_20161201_121049.jpg?imgmax=1600" alt="IMG 20161201 121049" border="0" width="337" /><br/>
Bridge of the Intrepid<p/></div>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-XEquQ4AAmwc/WERmxcgVnnI/AAAAAAAAFl4/ajmY-EdspAM/IMG_20161201_121905.jpg?imgmax=1600" alt="IMG 20161201 121905" border="0" width="400" />
<br/>Lockheed A1<p/></div>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-2du7kl2SC0M/WERm3NgoKpI/AAAAAAAAFl8/kE9LBlZIS6s/IMG_20161201_123151.jpg?imgmax=1600" alt="IMG 20161201 123151" border="0" width="400" />
<br/>Space Shuttle Enterprise<p/></div>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-5m8GCBp5js4/WERm6KvQjVI/AAAAAAAAFmA/QE0Q7KjPAig/IMG_20161201_125541.jpg?imgmax=1600" alt="IMG 20161201 125541" border="0" width="400" />
<br/>View from the bridge<p/></div>
<p>All talks were recorded and will be pushed online once the video team has edited in the slides. My Slides are available in the meantime from <a href="http://www.pilhuhn.de/GrafanaCon2016.pdf">http://www.pilhuhn.de/GrafanaCon2016.pdf</a>. I will update this post once the recordings are online.</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-26841363200447957692016-10-18T21:05:00.001+02:002017-04-02T12:05:57.405+02:00A DSL for Alert Trigger Definitions in Hawkular<p>Hawkular had for a while a UI with the possibility to set up Alert Triggers. As the name suggests are those triggers used to define conditions when an Alert is to be fired.</p>
<p>Since a while there is now the <a href="http://manageiq.org">ManageIQ</a> UI that allows to set up such triggers. And also <a href="https://hawkular.gitbooks.io/hawkular-apm-user-guide/content/">Hawkular APM</a> is <a href="http://www.hawkular.org/blog/2016/10/17/hawkular-apm-opentracing-and-alerts.html">now able to forward data into Alerting</a>.</p>
<p>The other day I was doing some testing and a colleague asked me if I had already defined some triggers. I thought that I neither want to log into ManageIQ right now nor pass JSON structures via curl commands.</p>
<p>As I did some <a href="http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html">DSL work for metrics</a> recently, I thought, why not set up a DSL for trigger definitions. This is work in progress right now and here are two examples</p>
<p>Set up a threshold trigger to fire when the value of _myvalue_ is > 3:
<pre>define trigger "MyTrigger"
enabled
( threshold "myvalue" > 3 )
auto-disable</pre></p>
<p>Set up a trigger on availability when it is reported as DOWN. The trigger is not enabled.
<pre>define trigger "MyTrigger"
( availability "mymetric" is DOWN )</pre></p>
<p>As with my metric DSL I am implementing this in Ruby with the help of <a href="http://cjheath.github.io/treetop/">Treetop</a>.
And likewise I am integrating this in <a href="http://pilhuhn.blogspot.com/2016/05/introducing-hawkfx.html">HawkFX</a>.</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-gx25QacP89A/WAZx36xOcDI/AAAAAAAAFic/oBERdsTFtuM/Bildschirmfoto%2525202016-10-18%252520um%25252021.01.54.png?imgmax=1600" alt="Inserting a definition" border="0" width="398" /></div>
<p>At the moment it is a very crude integration via entry points in the main menu. And the DSL itself is also far from ready. I consider this an experimentation space. If it turns our successful, it may be possible to take the grammar and directly integrate it into Hawkular-Alerts, so that one can directly POST a document with a DSL fragment, which then gets turned into the internal representation.</p>
<p>If you are looking for code, this is available in the <a href="https://github.com/pilhuhn/hawkfx/tree/alert_insert">alert_insert branch of HawkFX</a>.</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-54221414944587096832016-09-27T20:07:00.001+02:002017-04-02T12:06:12.599+02:00Computed metrics for HawkFX (updated)<p>Computed metrics are something I wanted to do for a very long time, already in RHQ, but never really got around it and sort of forgot about it again.</p>
<p>Lately I found a <a href="http://www.circonus.com/percentages-arent-people/">post that contained a DSL to do exactly this</a> (actually you should read that post not because of the DSL, but because of the idea behind it).</p>
<p>After seeing this, I got the idea on what to do and to include this in <a href="https://github.com/pilhuhn/hawkfx">HawkFX</a>, my pet project, which is an explorer for Hawkular.</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-jnnarwKMxss/V-po2qJoo7I/AAAAAAAAFfE/3-AwZRd8nK8/Bildschirmfoto%2525202016-09-27%252520um%25252014.27.25.png?imgmax=1600" alt="HawkFX screen shot" border="0" width="399" /><br/>
HawkFx with the input window for formulae, that shows a formula and also a parser error.<br/>
The orange chart shows Non-Heap used, the redish one the heap usage of a JVM.
</div>
<h2>Formulae</h2>
<p>Formulas are in a DSL that looks a bit like UPN, e.g. as in the following (I've shortened the metric ID for readability, more on them below):</p>
<pre>
(+ metric( "MI~...Heap Used" , "max")
metric( "MI~...NonHeap Used", "max"))
</pre>
<p>to sum up two metrics (see also screenshot below). The 'metric' element gets two parameters, the metric id and also which of the aggregates that the server sends should be taken (in this case the max value) - this comes from the fact that we request the values to be put into 120 buckets by the server.</p>
<p>Or if you have the total amount of memory you could also subtract the used memory to get a graph of the remaining:
<pre>
(- 1000000 metric( "MI~...NonHeap Used", "max"))
</pre></p>
<p>You could also get the total wait time for responses at a point in time when you multiply the average wait time with the number of visitors:
<pre>
(* metric("MI~..ResponseTime","avg")
metric("MI~..NumberVisitors","sum"))
</pre>
<p/></p>
<h2>Computed total memory usage</h2>
<p>Summing up the metrics for 'Heap Used' and 'NonHeap Used' as shown above would then give you a nice graph of the total memory consumption of a JVM:</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-It_uFDz4TCY/V-pp5WrvIAI/AAAAAAAAFfQ/6VeirFBB8t4/Bildschirmfoto%2525202016-09-27%252520um%25252014.27.42.png?imgmax=1600" alt="Chart with computed metric" border="0" width="398" /><br/>
The green chart now shows to combined memory usage of Heap and Non-Heap, which is computed from the other two series. Orange and red are as above.
</div>
<h2>On metric IDs</h2>
<p>Metric IDs are the IDs under which a metric is stored inside of Hawkular. The example here comes from an <a href="http://pilhuhn.blogspot.de/2016/06/using-hawkular-services-via-docker.html">installation of Hawkular-services in Docker</a>. If you just feed your metrics into Hawkular metrics, the IDs will looks like the ones you are using.</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-22X-39gBrXM/V-toxTNkV_I/AAAAAAAAFfk/F9G1LCHez5c/Bildschirmfoto%2525202016-09-28%252520um%25252008.48.48.png?imgmax=1600" alt="ID and path field" border="0" width="399" />
<br/>
ID (upper) and path fields (lower) for a selected item in the tree
</div>
<p>I have just pushed an update to HawkFX that provides the ID and path in their own fields at the bottom of the main window, so you can copy&paste them.</p>
<h2>Future</h2>
<p>I will talk more about the parser in an upcoming article. For now it is a personal playground to also better understand what is doable here. If this turns out to be successful I can imagine that the DSL could directly be incorporated into Hawkular-metrics so that the rules are available to all metrics clients.</p>
<p>
It would of course be cool to have an editor for the formulas that allows to interactively pick metric IDs etc, but I doubt that I will get to this any time soon.
</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com2tag:blogger.com,1999:blog-6080082.post-46264273168451504192016-09-15T08:06:00.001+02:002016-09-15T14:29:04.167+02:00Gespalten<p>Mein Beitrag zum Comic-Collab von <a href="http://schlogger.de/">Schlogger</a> zum Thema <em>Gespalten</em>.</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-C9wmedSyulI/V9mjXrVASqI/AAAAAAAAFZs/v2LHQc7Fx_g/Gespalten.jpg?imgmax=1600" alt="Gespalten" border="0" width="380" />
<br/>
(<a href="https://lh3.googleusercontent.com/-C9wmedSyulI/V9mjXrVASqI/AAAAAAAAFZs/v2LHQc7Fx_g/Gespalten.jpg">Vergrößern</a>)
</div>
<p/>
<strong>Thema für den 15. Oktober: Egoistisch</strong></br>
Mit dabei im September:<br/>
<a href="http://schlogger.de/comic-collab-60-gespalten/">Schlogger</a><br/>
<a href="http://nadiabader.blogspot.de/2016/09/comic-collab-060-gespalten.html">Skizzenblog</a><br />
<a href="http://www.isla-volante.ch/comic-collab-60-gespalten/">Isla Volante</a><br />
<a href="http://www.mrs-frollein.com/comic-collab/gespalten/">Mrs. Frollein</a><br />
<a href="http://www.brausecomics.de/2016/09/comic-collab-gespalten/">Brausecomics</a><br />
<a href="http://rainerunsinn.blogspot.de/2016/09/durch-den-kopf-gehen-lassen.html">Rainer Unsinn</a><br />
<a href="http://www.buddelfisch.de/2016/09/comic-collab-060-gespalten/">Buddelfisch</a><a href="https://onlinecomics.wordpress.com/2016/09/15/comic-collab-60-thema-gespalten/">Onlinecomics</a><br />
<a href="http://karrakula.de/blog/2016-09-15_salz-lolly-entscheidungen.html">Karrakula</a><br />
<a href="http://mal-gries.blogspot.de/2016/09/comic-collab-gespalten.html">Mal-Gries</a><br />
<a href="http://rainerunsinn.blogspot.de/2016/09/durch-den-kopf-gehen-lassen.html">Jane War</a><br/>
<a href="http://www.dramatized.de/comic/gespalten/">Dramatized Depiction</a><br/>
<a href="http://schaf-und-verstand.de/comic-collab-von-schlogger/">Schaf und Verstand</a></br>
<a href="http://schoolpeppers.de/comic/schoolpeppers-16-701/">Schoolpeppers</a><br/>
<a href="http://www.schisslaweng.net/genug/">Schisslaweng</a><br/>
<a href="http://badham.de/schnell-schnell-comic-collab-no-60-gespalten/">Badham</a></br>
</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com1tag:blogger.com,1999:blog-6080082.post-69573551012235992722016-07-15T08:32:00.001+02:002016-07-18T10:18:15.262+02:00Sabotage<p>Mein Beitrag zum Comic-Collab von <a href="http://schlogger.de">Schlogger</a> zum Thema Sabotage.</p>
<p><div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-9AnmdecofAA/V4fmzKMByMI/AAAAAAAAFVI/5Z4Co0VCo-8/Bildschirmfoto%2525202016-07-14%252520um%25252021.22.37.png?imgmax=1600" alt="Bildschirmfoto 2016 07 14 um 21 22 37" border="0" width="400" /><br/>
<a href="https://lh3.googleusercontent.com/-9AnmdecofAA/V4fmzKMByMI/AAAAAAAAFVI/5Z4Co0VCo-8/Bildschirmfoto%2525202016-07-14%252520um%25252021.22.37.png">(Vergrößern)</a></div>
<p/>
Ich denke Katzenhalter kennen die Situation wenn dringend etwas erledigt werden muss :-)</p>
<strong>Thema für den 15. August: „Nichts Neues“</strong><br />
Mit dabei im Juli:<br />
<a href="http://schlogger.de/comic-collab-058-selbst-sabotage" target="_blank">Schlogger</a><br />
<a href="http://nadiabader.blogspot.de/2016/07/comic-collab-058-sabotage.html" target="_blank">Skizzenblog</a><br />
<a href="http://www.badham.de/comic-collab-58-listen-all-of-yall-its-a-sabotage/" target="_blank">Badham</a><br />
<a href="http://www.isla-volante.ch/comic-collab-58-sabotage/" target="_blank">Isla Volante</a><br />
<a href="http://www.michael-tewiele.de/micatsix-comiccollab58/" target="_blank">Mic At Six</a><br />
<a href="https://onlinecomics.wordpress.com/2016/07/15/comic-collab-58-thema-sabotage/" target="_blank">OnlineComics</a><br />
<a href="https://demystifikation.wordpress.com/2016/07/15/alle-tage-sabotage/" target="_blank">Demystifikation</a><br />
<a href="http://karrakula.de/blog/2016-07-15_salz-lolly-sabotage.html" target="_blank">Karrakula</a><br />
<a href="http://www.dramatized.de/comic/sabotage/" target="_blank">Dramatized Depiction</a><br />
<a href="http://jane-war.tumblr.com/post/147433103123/comic-collab-58-july-2016-sabotage" target="_blank">Jane War</a><br />
<a href="http://catnipsflavour.blogspot.de/2016/07/comiccollab-sabotage.html" target="_blank">Catnipsflavour</a><br />
<a href="http://www.schisslaweng.net/erholung/" target="_blank">Schisslaweng</a></br/>
<a href="https://gobopictures.wordpress.com/2016/07/15/comic-collab-sabotage/"target="_blank">GoboPictures</a>
<a href="http://rainerunsinn.blogspot.de/2016/07/snowman-suicides-titanic.html" target="_blank">Rainer Unsinn</a><br />
<a href="http://schoolpeppers.de/comic/schoolpeppers-16-665/" target="_blank">Schoolpeppers</a><br />
<a href="http://mal-gries.blogspot.de/2016/07/comic-collab-sabotage.html" target="_blank">Mal-Gries</a>
<a href="https://rainking.de/2016/07/15/15-07-2016-selbst-sabotage-comic-collab/" target="_blank">Rainking</a><br />
<a href="http://lisafruehbeis.tumblr.com/post/147536058465/day-018-the-day-i-went-into-the-wild" target="_blank">Lisa Frühbeis</a></p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-10677580964676327102016-06-27T16:19:00.001+02:002016-09-15T16:48:59.215+02:00Using Hawkular-services via Docker [updated x3]<p>As you may know, we have started to create the <a href="http://www.hawkular.org/blog/2016/04/28/new-packaging.html">Hawkular-Services</a> distribution, which we try to build weekly. This distribution comes without embedded Cassandra, with <a href="http://www.hawkular.org/blog/2016/06/14/hawkular-services-0.0.2.Final.html">no default user and also without a UI</a> (we plan on re-adding a basic UI).</p>
<p>But you must not fear.</p>
<p>Running Hawkular-services is pretty easy via Docker.</p>
<h3>The scenario</h3>
<p>Let me start with a drawing of what I want to do here:</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-BVVKSuzYUvo/V3EnEslVbfI/AAAAAAAAFTg/dwNwTJD0jk0/hawk-service-via-docker.png?imgmax=1600" alt="Hawk service via docker" border="0" width="399" /><br/>
Setup of Hawkular-services via Docker</div>
<p>In this scenario I run a Docker daemon (which is extremely easy these days on a Mac thanks to DockerForMac (Beta)). On the daemon I run a Hawkular-services container, which talks to a Cassandra container over the Docker-internal network. On top of that I have two WildFly10 containers running ("<em>HawkFly</em>"), which have been instrumented with the Hawkular-agent.</p>
<h3>Running</h3>
<p>For the purpose to setup linking and data volumes I am using docker-compose.
The following is the docker-compose.yml file used (for the moment all images are on <a href="https://hub.docker.com/u/pilhuhn/">my personal account</a>):</p>
<pre>
# set up the wildfly with embedded hawkular-agent
hawkfly:
image: "pilhuhn/hawkfly:latest"
ports:
- "8081:8080"
links:
- hawkular
# The hawkular-server
hawkular:
image: "pilhuhn/hawkular-services:latest"
ports:
- "8080:8080"
- "8443:8443"
- "9990:9990"
volumes:
- /tmp/opt/data:/opt/data
links:
- myCassandra
environment:
- HAWKULAR_BACKEND=remote
- CASSANDRA_NODES=myCassandra
# The used Cassandra container
myCassandra:
image: cassandra:3.7
environment:
- CASSANDRA_START_RPC=true
volumes:
- /tmp/opt/data:/opt/data
</pre>
<p>To get started save the file as <code>docker-compose.yml</code> and then run:
<pre>
$ docker-compose up hawkular
</pre>
This starts first the Cassandra container and then the Hawkular one. If they do not yet exist on the system, they are pulled from DockerHub.</p>
<p>After Hawkular has started you can also start the <em>HawkFly</em>:
<pre>
$ docker-compose up hawkfly
</pre></p>
<h4>Update</h4>
<p><strike>Right now if you would directly do <code>docker-compose up hawkfly</code> the agent would not work as the hawkular server is not yet up and the agent would just stop. We will add some re-try logic to the agent pretty soon.</strike></p>
<p>I have pushed a new version 0.19.2 of HawkFly that has the retry mechanism.
Now it is possible to get the full combo going by only running</p>
<pre>
$ docker-compose up hawkfly
</pre></p>
<h3>Running without docker-compose</h3>
<p>On my RHEL 7 box, there is Docker support, but no
docker-compose available. Luckily docker-compose is more or less a wrapper around individual docker commands. The following would be a sequence that gets me going (you have to be root to do this):</p>
<pre>
mkdir -p /var/run/hawkular/cassandra
mkdir -p /var/run/hawkular/hawkular
chcon -Rt svirt_sandbox_file_t /var/run/hawkular
docker run --detach --name myCassandra -e CASSANDRA_START_RPC=true \
-v /var/run/hawkular/cassandra:/var/lib/cassandra cassandra:3.7
sleep 10
docker run --detach -v /var/run/hawkular/hawkular:/opt/data \
-e HAWKULAR_BACKEND=remote -e CASSANDRA_NODES=myCassandra \
-p 8080:8080 -p 8443:8443 --link myCassandra:myCassandra \
pilhuhn/hawkular-services:latest
</pre>
<h3>Looking forward</h3>
<p>There is <a href="https://github.com/hawkular/hawkular-services/pull/25">an open Pull-Request</a> to the Hawkular-Services Docker build as a part of a release and make it available via DockerHub on the official Hawkular account.</p>
<p>With this PR you can do</p>
<pre>
$ mvn install
$ cd docker-dist
$ mvn docker:build docker:start
</pre>
<p>to get your own container built and run together with the C* one.</p>
<h4>Open questions</h4>
<p>Right now I put in the default user/password and if the agent inside the hawkular-container should be enabled at image build time. Going forward we need to find a way to pass those at the time of the first start. The same applies (probably even more) to SSL-Certificates.</p>
<p>Storing them inside the container itself does not work going forward, as this way they are lost when a newer version of the image is pulled and a new container is constructed from the newer image.</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com3tag:blogger.com,1999:blog-6080082.post-67983766922945648062016-05-09T20:54:00.001+02:002017-04-02T12:06:30.031+02:00Introducing HawkFX<p>As I said before, <a href="http://pilhuhn.blogspot.com/2016/03/playing-with-jrubyfx.html">I started playing with JRubyFX</a>. And for me learning something new best works with a use case, so I started creating an inventory browser for <a href="http://hawkular.org/">Hawkular</a>.</p>
<h2>Why JRubyFX?</h2>
<p>Let's first start with "What is JRubyFX" anyway? <a href="https://github.com/jruby/jrubyfx">JRubyFX</a> is JavaFX brought to the Ruby world by the means of <a href="https://github.com/jruby/jruby">JRuby</a>. This means that you can implement UIs with the help of the JavaFX framework and use its components and tools to build the UI. The difference to plain JavaFX is though that all the implementation code is written in Ruby and run by JRuby on the JVM.</p>
<p>I was doing a bit of JavaFX in the past and I wanted to generate a standalone inventory browser for Hawkular. Now that I have been working with Ruby lately and we already have the <a href="https://github.com/hawkular/hawkular-client-ruby/issues">Hawkular client gem</a>, I thought I'd give JRubyFX a try.</p>
<p>And I have to say this is pretty cool.</p>
<h2>Some screenshots</h2>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-iqtTTwRSp4U/VzCI2q_21XI/AAAAAAAAFOc/eBX_prFenz8/Bildschirmfoto%2525202016-05-09%252520um%25252014.52.39.png?imgmax=1600" alt="login screen" border="0" width="398" /><br/>
Login screen</div>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-Fiiu-AS_nk0/VzCI-Z0LGlI/AAAAAAAAFOg/5HkMn1G-7ZQ/Bildschirmfoto%2525202016-05-09%252520um%25252014.53.18.png?imgmax=1600" alt="Main screen with chart" border="0" width="398" /><br/>
Main screen with inventory browser (left) and metric chart</div>
<p>The main screen shows a tree view on the left that displays the feeds as top level elements. Opening a feed will show recursively the resources and metrics. Clicking on a metric gets it charted on the right side.</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-12ZZVcMjisM/VzCJF1g1gPI/AAAAAAAAFOk/n8kCwTFC6tM/Bildschirmfoto%2525202016-05-09%252520um%25252014.53.51.png?imgmax=1600" alt="Alerts and Events" border="0" width="399" /><br/>
Alert and Event list</div>
<p>A menu item in the main screen opens the alerts browser that allows to peek at alerts and events in the system.</p>
<p>Like in the main screen, there is a context menu that will allow to view the raw object as shown below:</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-U8vtPQ8OaG4/VzCJJ1DB6SI/AAAAAAAAFOo/PRPa2ymG9QA/Bildschirmfoto%2525202016-05-09%252520um%25252014.54.18.png?imgmax=1600" alt="Raw event display" border="0" width="398" /><br/>
Raw display of an event</div>
<h2>Custom components</h2>
<p>The time range picker on the main screen and alert screen is a custom component, that was implemented once with a <code>.fxml</code> file and some Ruby code:</p>
<pre>
class TimePicker < Java::javafx::scene::layout::HBox
include JRubyFX::Controller
fxml 'TimePicker.fxml'
def initialize(caller, callback)
[..]
end
</pre>
<p>Including it is pretty simple too:</p>
<pre>
box = find '#alertEventTopBox'
box.children.add time_picker(self, :set_time_range)
</pre>
<p>In the first line we find the <code>HBox</code> to add the picker and then just
add it to the children of the box. Done.</p>
<h2>Running and code</h2>
<p>HawkFX is available on my GitHub account at <a href="https://github.com/pilhuhn/hawkfx">https://github.com/pilhuhn/hawkfx</a>. To run the tool you need JRuby 9</p>
<p>If you are using rvm you can select it via</p>
<code>rvm use jruby-9.0.5.0</code>
<p>install and use bundler to install the required gems</p>
<code>gem install bundler</code>
<code>bundle install</code>
<p>then run</p>
<code>jruby hawkfx.rb</code>
<p>Enjoy! :-)</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-92094042410839361382016-05-03T13:29:00.001+02:002016-05-03T13:29:04.290+02:00Presenting at ManageIQ Design Summit 2016<img src="http://manageiq.org/images/manageiq-summit-logo.svg" align="right" width="200"/>
<p>I have the luck to go to the <a href="http://manageiq.org/summit/">ManageIQ Design Summit 2016</a> in <a href="https://www.google.com/maps?ll=41.104091,-74.163007&z=14&t=m&hl=de-DE&gl=FI&mapclient=embed&cid=521946126656790">Mahwah, New Jersey</a> at the beginning of June. And not only that, but also to be able to present there (more about this in a moment).</p>
<p>You may be looking at the web site of ManageIQ or the Design summit and think "why the heck is the Java and JBoss guy talking at a Ruby conference"? And the answer is simple:</p>
<p><a href="http://www.hawkular.org/blog/2016/02/22/hawkular-manage-iq.html">Hawkular and ManageIQ are collaborating</a> on the future of Middleware management.</p>
<img src="https://avatars1.githubusercontent.com/u/10179627?v=3&s=100" align="left" style="border-style:hidden"; border-width="10px"/>
<p>And so I will talk about "<em>Adding middleware to the game</em>", describing the current state of Red Hat middleware and monitoring with RHQ and Hawkular, the integration with ManageIQ that we are already working on. And also the path and vision going forward.</p>
<p>ManageIQ has all the knowledge about the operating system and infrastructure the Middleware servers are running on, where Hawkular only provides some basic information. ManageIQ on the other hand has no notion of applications yet, wich is the domain of Hawkular. ManageIQ can also be used to provision new VMs and containers with Middleware in them, which can then be monitored and managed by Hawkular.</p>
<p>If you are close by, join me :)</p>
<p>NB: while I am there I will try <a href="https://www.geocaching.com/map/default.aspx?lat=41.08871&lng=-74.14376&asq=QD00MS4wODg3MSwtNzQuMTQzNzYmb3JpZ2luPU1haHdhaCwgTmV3IEplcnNleSZvdD0zJmc9MzIyNTU2JnJhZGl1cz0xNmtt#?ll=41.10025,-74.15731&z=15">find some of the nearby Geocaches</a>, as New Jersey is still missing on my list (as so many other states in he US :)</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com1tag:blogger.com,1999:blog-6080082.post-30121542011309381132016-04-23T14:52:00.001+02:002016-04-23T14:54:45.411+02:00Welcome GSoC 2016 Students to JBoss<img src="https://1.bp.blogspot.com/-qCw01EeaHng/VxLKQqtu41I/AAAAAAAAAEU/hftzkYHzePwlnYEQmodWdnn_CL4Lc_DIwCK4B/s640/soc_horizontal1000.png" width="400"/>
<p>JBoss.Org has the <a href="http://developers.redhat.com/blog/2016/03/08/jboss-participates-in-google-summer-of-code-2016/">luck of being selected</a> as one of the mentoring organizations for this years issue of Google Summer of Code.</p>
<p>On Friday April 22nd, Google has announced the 10 students that will all work with JBoss. Those students with their project are:</p>
<ul>
<li>Idel Pivnitskiy: AeroGear WebPush and UnifiedPush Server integration
<li>rohitmohan96: Ceylon Markdown
<li>Lucas Werkmeister: Ceylon TypeScript Loader
<li>Samuel Richardson: Drools Rules in Minecraft
<li>Anton Gabov: Smart HTTP/2-based protocol for Infinispan
<li>Austin Ko: Hawkular-agent For Vert.x
<li>mincongh: Hibernate Search: JSR 352 batch job for re-indexing entities
<li>Anuj Garg: Improve existing Android client of Hawkular
<li>Tugba: Teiid HDFS Translator/Connector
<li>dimcho: Test scheduling for large test suites
</ul>
<p>We also want to thank all the other students for their in total over 70 proposals that they have submitted.</p>
<p>See also the <a href="https://summerofcode.withgoogle.com/organizations/5944037739593728/#projects">GSoC project page for more details</a> and the <a href="http://google-opensource.blogspot.de/2016/04/students-announced-for-google-summer-of.html">announcement post from Google</a> for further information.</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-543769038180690132016-03-20T16:32:00.001+01:002016-03-20T16:32:59.758+01:00Playing with JRubyFX<p>I started looking at JRubyFX as a combination of using a proven UI solution with Ruby as the programming language and I have to admit that I am impressed with how little it takes to get a UI done -- especially when it is driven by FXML.</p>
<p>Unfortunately the learning curve is steep, as one has to learn JavaFX and the special way JRubyFX calls the JavaFX classes and methods. And unfortunately, Google is not returning too much results here.</p>
<p>Anyway: this little code already creates GUI and populates a TreeView with elements it pulls from a remote server.</p>
<pre>require 'jrubyfx'
require 'hawkular_all'
fxml_root File.dirname(__FILE__)
class HawkFx < JRubyFX::Application
def start(stage)
with(stage, title: 'Hello World!', width: 800, height: 600) do
fxml HawkFxController
show
end
stage.show()
end
end
class HawkFxController
include JRubyFX::Controller
fxml 'fxmain.fxml'
def login # callback from the login button
creds = { :username => 'jdoe' ,
:password => 'password' }
url = 'http://localhost:8080/hawkular/inventory'
@inventory_client = Hawkular::Inventory::InventoryClient.new(url, creds)
@tenant = @inventory_client.get_tenant
@FXMLtextArea.text = "Tenant: #{@tenant}"
feeds = @inventory_client.list_feeds
show_initial_tree(feeds)
end
def show_initial_tree(feeds)
tree_root = tree_item('Feeds') do
feeds.each do |feed|
item = tree_item(feed) # this already adds the item to the root
end
end
# bind to the view from fxml
@FXMLtreeView.setRoot(tree_root)
tree_root.setExpanded true
end
end
HawkFx.launch</pre>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com3tag:blogger.com,1999:blog-6080082.post-25775488699673571672016-03-17T15:08:00.001+01:002016-03-17T17:37:23.032+01:00Reacting on IoT data with Hawkular<p>In the first post I have been talking about how <a href="http://pilhuhn.blogspot.com/2016/02/sending-iot-sensor-data-to-hawkular.html">send IoT sensor data to the metrics subsystem</a> of Hawkular and then <a href="http://pilhuhn.blogspot.com/2016/02/send-iot-data-to-hawkular-full.html">how to register the metric in Hawkular</a> so that it can be graphed in the console.</p>
<p>In this article I will talk about how the Hawkular alerts component (that is already available in Hawkular-full) can be used to react on incoming data and make an LED on an Arduino blink.</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/--oqSUP7Enuw/VuqXgXn7LrI/AAAAAAAAE3U/tUPkrveXXd4/DSC_0160.JPG?imgmax=1600" alt="DSC 0160" border="0" width="400" /><br/>Arduino Uno with Ethernet shield and yellow LED</div>
<p>For the Arduino I have added a cheap Enc28j60 based Ethernet shield. There is a <a href="https://www.arduino.cc/en/Reference/Ethernet">standard library available</a>, that allows to easily set up a web server, which we are using (in fact the code is mostly from that example).</p>
<p>The new (full) setup is like this:</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-MMLxx7QGTtg/Vt2ydhg-CRI/AAAAAAAAEzg/LCgN7IVrdZA/hawk-alert.png?imgmax=800" alt="Hawk alert" border="0" width="398" /><br/>Setup with Arduino as actor</div>
<h2>Alerting</h2>
<p>Hawkular already has an alerting component built in, that allows to compare incoming values with thresholds and then invoke plugins to forward the fired alert via email, to irc channels and many more way of communication. The plugin we are going to use here is the <em>webhook</em> one. As in the standard Hawkular distribution only the email plugin is present we will need to install the webhook plugin first:</p>
<p>Check out Hawkular-alerts</p>
<pre>git clone https://github.com/hawkular/hawkular-alerts.git
cd hawkular-alerts</pre>
<p>Build hawkular alerts :</p>
<pre>mvn -Pdev -DskipTests install</pre>
<p>Now you can copy the plugin to the Hawkular-server</p>
<pre>cd hawkular-alerts-actions-plugins
cd hawkular-alerts-actions-webhook
cp hawkular-alerts-actions-webhook.war \
$HAWKULAR/modules/system/layers/hawkular/org/hawkular/nest/main/deployments/</pre>
<p>As Hawkular does not pick this up automatically you need to restart the Hawkular server after copying the webhook.war over.</p>
<h2>Set up alerting</h2>
<p>I have modified the ruby script to pull in a configuration file (in YAML format):</p>
<p><pre>---
16617927:40.176.91.120.5.0.0.125:
:name: Living room
:alert:
:comparator: :gt
:value: 25</pre>
The first line is the id of the metric to which the following lines apply to. Second line is the display name for the UI. The next section then sets up alerting.</p>
<p>Let's have a look how this looks in code (you can see the <a href="https://github.com/pilhuhn/hawkular-mqtt-bridge/blob/master/lib/hawk.rb">full code on GitHub</a>)</p>
<p>Register the webhook to be used below:
<pre>@webhook_props = { url: 'http://172.31.7.177/', # target server
method: 'POST' } # http verb
@alerts_client.create_action 'webhook', "send-via-webhook-#{metric_id}", @webhook_props
</pre></p>
<p>Set up a trigger and a condition, trigger first
<pre> t = Hawkular::Alerts::Trigger.new({})
t.enabled = true
t.id = "trigger_#{metric_id}"
t.tags = { :resourceId => "#{feed}/#{res_id}" }
</pre>
The <em>tags</em> tell the UI on which resource the Trigger applies and thus enables showing the alert in the Hawkular UI</p>
<p>Next the condition:
<pre> c = Hawkular::Alerts::Trigger::Condition.new({})
c.trigger_mode = :FIRING
c.type = :THRESHOLD
c.data_id = metric_id
c.operator = alert[:comparator].to_s.upcase
c.threshold = alert[:value]
</pre></p>
<p>And then finally as we do not only want the alert to show in the UI, but also be forwarded to our Arduino with the webhook, we need to attach the webhook to the Trigger. Plugin id is <em>webhook</em> and the action_id is what we have set up above.</p>
<p><pre> # Reference an action definition
a = Hawkular::Alerts::Trigger::Action.new({})
a.action_plugin = 'webhook'
a.action_id = "send-via-webhook-#{metric_id}"
t.actions.push a
</pre>
And then we can create the trigger, which will be active immediately.</p>
<pre> @alerts_client.create_trigger t, [c], nil
</pre>
<h3>Fire away</h3>
<p>Now when a value above the threshold comes in, the webhook will be triggered, it will open a http connection to the Arduino and the LED will blink.</p>
<p>In addition the alert will also be shown in the Alert Center in the Hawkular UI:</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-whrIm9fqjU4/VurOOnigPtI/AAAAAAAAE34/QhwX2i45TmA/Bildschirmfoto%2525202016-03-17%252520um%25252016.32.37.png?imgmax=1600" alt="Bildschirmfoto 2016 03 17 um 16 32 37" border="0" width="399" /><br/>
List of high temperature alerts in the alert center.
</div>
<h2>More</h2>
<p>Note that the above trigger definition is pretty simple and only has one comparator. Also the trigger fires each time a value above the threshold comes in. In a more real world scenario one would add some dampening that the trigger only fires when the measurement is a few times over the threshold for some interval.
Or if you start your air-conditioning, you would set the trigger to auto-disable and then have a comparator to switch it back on after the temperature has fallen below a certain level.</p>
<p>There is even a lot more you can do with Hawkular Alerts past the above, including standalone deployment for embedding it in your own application.</p>
<h3>Some further reading:</h3>
<a href="http://www.hawkular.org/docs/dev/alerts.html">Hawkular Alerts for Developers</a> -- pretty comprehensive documentation of Hawkular Alerts.<br/>
<a href="http://www.hawkular.org/blog/2015/08/25/hawkular-alerts-autoresolve.html">Introduction to AutoResolve triggers</a><br/>
<a href="http://www.hawkular.org/blog/2015/08/19/hawkular-alerts-standalone.html">Using Hawkular Alerts as a standalone engine</a><br/>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-13667409271744604962016-03-02T09:22:00.000+01:002016-03-02T09:22:22.810+01:00Send IoT data to Hawkular-full<p>In a previous blog post, I was talking about <a href="http://pilhuhn.blogspot.com/2016/02/sending-iot-sensor-data-to-hawkular.html">how to send IoT sensor data to Hawkular-metrics</a>.</p>
<p>While this already works quite well, it also lacks the integration with other parts of Hawkular, namely Inventory and Alerting.</p>
<p>In this blog post I will talk about integration with Inventory and how to view the data in the Hawkular-UI. An upcoming article will then talk about Alerting.</p>
<p>I have modified the setup from the last post a bit:</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-DRTbO7d3_BI/Vsrrgyl-4nI/AAAAAAAAEtM/BrJ2-H5z_g8/bridge_arch.png?imgmax=800" alt="Bridge arch" border="0" width="340" /><br/>New setup with a Ruby client</div>
<p>Instead of <em>PTrans</em> I've written a small Ruby client, that makes use of the <a href="https://github.com/hawkular/hawkular-client-ruby">Hawkular-Client-Ruby</a> ruby gem (it needs version 0.2.1, that has not yet been published to <a href="https://rubygems.org/gems/hawkular-client">RubyGems.org</a>. In addition it uses a MQTT gem, which makes the code pretty short.</p>
<p>The Ruby client MQTT-bridge now listens on <code>/hawkular/+</code> topics. Metric arriving on <code>/hawkular/metrics</code> are forwarded as such and registration messages on <code>/hawkular/register</code> are used to register the external resource like the ESP8622 micro controller in Hawkular inventory.</p>
<p>The following is an example registration message:
<pre>{"feed": "esp16617927",
"rt":"esp8266",
"r":"mcu16617927",
"mt":{
"id":"thermo",
"unit":"NONE",
"type":"GAUGE",
"collectionInterval":0
},
"m":{
"id":"16617927:40.176.91.120.5.0.0.125",
"mt":"thermo",
"na":"thermo_40.176.91.120.5.0.0.125",
}
}
</pre></p>
<p>You can get the client from <a href="https://github.com/pilhuhn/hawkular-mqtt-bridge">https://github.com/pilhuhn/hawkular-mqtt-bridge</a> and then easily run it via</p>
<code>$ ruby lib/hawk.rb</code>
<p>If you are not using a Hawkular-Server on localhost with default user, then you first need to modify the ruby code at the top.</p>
<p>The client will connect to the Hawkular server and then wait for messages on the MQTT topics.</p>
<p>The client code is still rather simplistic and does not spool data when the target Hawkular server is down.</p>
<h2>UI</h2>
<p>Since Hawkular 1 Alpha11 we have an Explorer (as easter egg :-) that allows to browse through inventory and to show resources and their metrics. The following shows the explorer with a chart from the thermo sensor for the last 12h.</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-50iWAfKqqwY/VtafD7d6h1I/AAAAAAAAEwA/0bYMMEqHvQ8/Bildschirmfoto%2525202016-03-02%252520um%25252008.53.24.png?imgmax=800" alt="Hawkular screenshot with chart data from thermo sensor" border="0" width="399" /><br/>
Hawkular UI with chart of sensor data from last 12h</div>
<p>The chart shows a peak at 1am - I have no idea why it is there. Possibly one of my cats examined the sensor :)</p>
<h2>ESP Sample code</h2>
<p>I have also provided a <a href="https://github.com/pilhuhn/hawkular-mqtt-bridge/blob/master/examples/esp8266/lua/script1.lua">sample Lua script</a>, that can be used on a ESP8266 like the Adafruit Huzzah shown in the previous post.</p>
<p>As the stock Huzzah comes with NodeMCU Lua 0.9.5 which was not working well for me,
I have <a href="http://pilhuhn.blogspot.com/2016/02/flashing-new-nodemcu-lua-firmware-to.html">flashed it with a newer version of NodeMCU Lua</a>. This is now running a firmware with the following modules:</p>
<pre>NodeMCU custom build by frightanic.com
branch: master
commit: c8037568571edb5c568c2f8231e4f8ce0683b883
SSL: false
modules: cjson,file,gpio,i2c,mqtt,net,node,ow,rtcmem,rtctime,sntp,tmr,uart,wifi,ws2812
build built on: 2016-02-18 08:33</pre>
<h2>Alerting?</h2>
<p>In an upcoming post I will talk about alerting in Hawkular to act on unusually high or low temperatures.</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-8546525330790183842016-02-22T12:57:00.001+01:002016-02-22T12:57:58.994+01:00Flashing new NodeMCU Lua firmware to Adafruit Huzzah ESP<p>Adafruit delivers the Huzzah ESP with NodeMCU 0.9.5, which is pretty old.</p>
<p>So I was thinking of a newer version. The process is pretty well described on <a href="http://nodemcu.readthedocs.org/en/dev/en/flash/">nodemcu.readthedocs.org</a>.
Unfortunately it also says</p>
<blockquote>The address for esp_init_data_default.bin depends on the size of your module's flash. ESP-01, -03, -07 etc. with 512 kByte flash require 0x7c000. Init data goes to 0x3fc000 on an ESP-12E with 4 MByte flash.
</blockquote>
<p>After digging through Adafruit docs (they do not explicitly mention the type of ESP) and also looking at the output of <a href="http://nodemcu.readthedocs.org/en/dev/en/modules/node/#nodeinfo"><code>node.info()</code></a> I figured out that the Huzzah ESP has an ESP12 with 4MB</p>
<p>I got my self a new image from the <a href="http://nodemcu.readthedocs.org/en/dev/en/build/#cloud-build-service">awesome on demand build service</a> and then flashed it like this:</p>
<pre>$ python esptool.py --port /dev/cu.usbserial-AI02CSDU \
write_flash 0x00 ../nodemcu-master-15-modules-2016-02-18-08-33-47-integer.bin \
0x3fc000 ../esp_iot_sdk_v1.4.0/bin/esp_init_data_default.bin
Connecting...
Erasing flash...
Took 2.12s to erase flash block
Wrote 423936 bytes at 0x00000000 in 45.8 seconds (74.1 kbit/s)...
Erasing flash...
Took 0.10s to erase flash block
Wrote 1024 bytes at 0x003fc000 in 0.1 seconds (85.4 kbit/s)...</pre>
<p>Future updates of the firmware to newer versions no longer need to flash the <code>esp_init_data_default.bin</code> file.</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-69243404060849443522016-02-11T08:49:00.001+01:002016-02-15T15:04:11.394+01:00Sending IoT sensor data to Hawkular-Metrics via MQTT<p>The other day I was writing about '<a href="http://pilhuhn.blogspot.com/2014/06/rhq-metrics-and-grafana.html">RHQ-Metrics and Grafana</a>' and was describing how you can incorporate data from other system management agents.</p>
<p>Fast forward a bit and Raider is now called Twix and RHQ-Metrics has morphed to Hawkular-Metrics under the <a href="http://hawkular.org/">Hawkular.org</a> umbrella.</p>
<p>Recently I have also been playing with Arduino and Co. and got myself also an <a href="https://learn.adafruit.com/adafruit-huzzah-esp8266-breakout/overview">Adafruit Huzzah ESP8266</a> board. This is a breakout board with the ESP8266 microprocessor on, that has a bunch of IO pins and built-in WiFi. With the default firmware it is programmed in Lua.</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-Fi16c4TWlpI/VruoHSj8BDI/AAAAAAAAErE/FHD3m6o6FN4/IMG_20160210_221228.jpg?imgmax=800" alt="Huzzah on Breakout board" border="0" width="399" /><br>
Huzzah on breakout board</div>
<p>While one can program the ESP from the Arduino IDE, I thought to give Lua a try (also to get a feel for the difference to the <a href="http://pilhuhn.blogspot.com/2015/10/wipy-on-home-network.html">WiPy</a>, that also comes with Python as a high level language). What is nice with the ESP and the NodeMCU firmware is that it already comes with support for networking, 1-wire, MQTT and more out of the box.</p>
<p>To get started I took the hello-world of IoT-sensors and hooked up a DS18B20 OneWire thermo sensor (for those old enough, I did that <a href="http://pilhuhn.blogspot.de/2008/09/driving-digital-thermometer-with-rhq_03.html">in the past with RHQ</a>) up to the ESP and then have this communicate to a MQTT Broker (mosquitto).</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-t-uyic_x5v0/Vrw3G2J7_GI/AAAAAAAAEro/uTPopuBhTbQ/Hawkular-Metrics-IOT.png?imgmax=800" alt="Hawkular Metrics IOT" border="0" width="359" />
<br/>Setup diagram</div>
<p>As said before we have with <i>Ptrans</i> a universal protocol translator that can be used to feed data from collected, ganglia and others into Hawkular-Metrics. I've taken that and added support for MQTT (<a href="https://github.com/pilhuhn/rhq-metrics/tree/mqtt">in my personal repo</a> for now). Ptrans will now connect to a broker and listen on the '/hawkular/metrics' topic for data that needs to be in graphite format like</p>
<pre>
path value [timestamp_in_s]
</pre>
<p>The timestamp is optional, as in my case I was not able to get any real time clock data from the micro controller (there seem to be variants that have a clock on board).</p>
<p>To see the data that is coming from the device I can just run</p>
<pre>
$ mosquitto_sub -t /hawkular/+
16617927:40.176.91.120.5.0.0.125 24.625
16617927:40.176.91.120.5.0.0.125 23.9375
16617927:40.176.91.120.5.0.0.125 23.8750
</pre>
<p>So here NodMCU with ID <code>16617927</code> and thermo sensor <code>40.176.91.120.5.0.0.125</code> is reporting around 24 deg Celsius.</p>
<p>I will post more on the topic in a laster posting.</p>
<p>References:</p>
<ul>
<li><a href="http://www.foobarflies.io/a-simple-connected-object-with-nodemcu-and-mqtt/">A simple connected object with NodeMCU and MQTT</a>
<li><a href="http://nodemcu.readthedocs.org/en/dev/">NodeMCU documentation</a>
<li><a href="http://www.linux.com/news/embedded-mobile/mobile-linux/881847-mqtt-building-an-open-internet-of-things">MQTT: Building an Open Internet of Things</a> (in Arduino lang)
<li><a href="http://www.jerome-bernard.com/blog/2015/10/04/wifi-temperature-sensor-with-nodemcu-esp8266/">WiFi Temperature Sensor with NodeMCU/ESP8266</a> (in Arduino lang)
<li><a href="https://github.com/njh/ruby-mqtt">MQTT Ruby Gem</a>
</ul>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-83264745799844030622016-01-05T11:21:00.001+01:002016-12-11T11:45:49.350+01:00Arduino powered X-Mas treeIn 2014 we got two cats and decided not to have a classic christmas tree. I also started playing with Arduino and other gadgets, so this made me think to have an Arduino powered tree in 2015. In November my daughter and I started to build the tree.<br />
<div style="text-align: center;">
<img alt="The tree" border="0" src="https://lh3.googleusercontent.com/-QaWBxdZZMlk/VouTgGca4oI/AAAAAAAAEgY/aVbK3h4CMyM/IMG_20160105_102955.jpg?imgmax=800" width="399" /></div>
The tree has 14 RGB LEDs (basically WS2812 ones) from <a href="https://www.sparkfun.com/products/12999">Sparkfun</a> (or <a href="http://www.exp-tech.de/led-rgb-addressable-pth-5mm-clear-5-pack?___SID=U">EXP-Tech</a>).<br />
Pinout of the LEDs is as follows:
<br />
<div style="text-align: center;">
<a href="https://lh3.googleusercontent.com/-HIGHVURJjJI/VouW-osgn0I/AAAAAAAAEgk/glzv4dtJags/DSC_0139.JPG"><img alt="LED Pinout" border="0" src="https://lh3.googleusercontent.com/-HIGHVURJjJI/VouW-osgn0I/AAAAAAAAEgk/glzv4dtJags/DSC_0139.JPG" width="163" /></a><br />
LED-Pinout (click to enlarge)</div>
.<br />
With that, the tree wiring got relatively easy by bending the LED pins in the four different directions and then connecting all 5 Volt pins with the red wire, GND with the black and chaining data in and out either by directly soldering the pins or via the yellow wire:<br />
<div style="text-align: center;">
<img alt="Backside of the tree" border="0" src="https://lh3.googleusercontent.com/-LGPz_X4SFug/VouZFfJS7wI/AAAAAAAAEgw/IJhvS7gyjeA/IMG_20160105_103034.jpg?imgmax=800" width="399" /></div>
Detail view:<br />
<div style="text-align: center;">
<img alt="Wiring detail" border="0" src="https://lh3.googleusercontent.com/-7wTINyfCsM4/VouZGNP3-NI/AAAAAAAAEg4/i7GBdMl-2fk/IMG_20160105_103058.jpg?imgmax=800" width="399" /></div>
The overall schematic then looks like this (I am using an Arduino at the moment, but will use a TrinketPro in the future):<br />
<div style="text-align: center;">
<a href="https://lh3.googleusercontent.com/-UFeFO5OCzN8/VouZG3xpb6I/AAAAAAAAEhA/AM_xQl55evE/Weihnachtsbaum_Schaltplan.png?imgmax=800"><img alt="Weihnachtsbaum Schaltplan" border="0" src="https://lh3.googleusercontent.com/-UFeFO5OCzN8/VouZG3xpb6I/AAAAAAAAEhA/AM_xQl55evE/Weihnachtsbaum_Schaltplan.png?imgmax=800" width="399" /></a><br />
Schematic (click to enlarge)</div>
I've added a photo resistor and a white diode to provide some lighting of the tree front, as at night, the RGB LEDs are so light, that the tree itself can't be seen. The resistor value also serves to dim the RGB LEDs at night.<br />
The software driving the LEDs is basically the Strandtest one that Adafruit delivers as example for their Neopixel Library. I've modified the code a bit to also show other patterns.<br />
In the future we will add a stand for the tree that will also house the electronics and will also host some additional surprise that I will talk about in a future post :)<br />
<br />
<b>[ UPDATE 2016-12-11 ]</b><br />
<br />
I've added a stand and the dedicated TrinketPRO:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmIhnjH0_UiblXjf5v3ENEX3GsH1QEtjs_fg5_jeSxZcGgUIM0Q7MdkT1Y7Dtfbw7tvPOYCqXAdYKvp2e6w3wdHkgPTIy7R1ke881sINaABvu6KcOJHz8X02AdcIVAghOvlgbT/s1600/IMG_20161211_113149.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmIhnjH0_UiblXjf5v3ENEX3GsH1QEtjs_fg5_jeSxZcGgUIM0Q7MdkT1Y7Dtfbw7tvPOYCqXAdYKvp2e6w3wdHkgPTIy7R1ke881sINaABvu6KcOJHz8X02AdcIVAghOvlgbT/s640/IMG_20161211_113149.jpg" width="360" /></a></div>
<br />
<br />Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com2tag:blogger.com,1999:blog-6080082.post-27803042130580915222015-10-27T09:35:00.001+01:002015-10-27T09:35:19.591+01:00WiPy on the home network<p>Recently there was a Kickstarter about <a href="https://www.kickstarter.com/projects/wipy/the-wipy-the-internet-of-things-taken-to-the-next/description">"an Arduino that runs Python", the WiPy</a>. This is a small IoT board with WiFi on it (which is not available on stock Arduinos) for an attractive price. And being able to use a high level language along with a lot of existing libraries makes it attractive too.</p>
<p>A few days ago I got the <a href="http://wipy.io">WiPy</a>s that I backed and the first obstacle to get the running was the power supply as they do not have a (micro) USB connector on board (and I also did not back the extension boards they offered).</p>
<p>Luckily I still had an old defunct USB hub from which I could solder out the connector and put it on a small PCB for this purpose (while doing that exercise I also found out that the 5V of the USB port are allowed to be 4.75-5.25V and on
the end of a hub even being as low as 4.4V).</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-K5MkISEF6F4/Vi8wd9Gcc1I/AAAAAAAAD8A/vtCYn3D3Ihw/DSC_0111.JPG?imgmax=800" alt="WiPy on BreadBoard" border="0" width="600" /><br/>WiPy on Breadboard with USB power supply connector.</div>
<p>Now that the WiPy is on the breadboard, I set up my computer to scan for the WLAN of the WiPy and then telneted into the device. I poked around and tried to inspect the WLAN settings as described in the manual:</p>
<pre>
>>> from network import WLAN
>>> wlan = WLAN() # we call the constructor without params
</pre>
<p>This immediately made the connection drop. It turned out I needed to first <a href="http://micropython.org/resources/docs/en/latest/wipy/wipy/general.html#wipy-firmware-upgrade">update the firmware</a> to <a href="https://github.com/wipy/wipy/releases">the latest version</a> (v1.1 at time of the writing), which was painless (but a bit confusing, as they also supply a <code>bootloader.bin</code>, that is not needed).</p>
<p>After the firmware upgrade the above worked and I tried to change the device to talk to my home network. The caveat here is that at the moment you issue</p>
<pre>
wlan.init(WLAN.STA)
</pre>
<p>to put the machine into station mode (default is being an access point), the connection drops. Some users have solved that by connecting via UART and serial line, but I did not really want to go that route.</p>
<p>Instead I edited <code>boot.py</code> on my local computer and then uploaded it via ftp into <code>/flash</code>. I found out, that if you don't terminate the ftp client and have a telnet connection open as well, I could easily (syntax) check the uploaded file by pressing Ctrl-D in the terminal:</p>
<pre>
MicroPython v1.5-1-ge954604 on 2015-10-21; WiPy with CC3200
Type "help()" for more information.
>>> <i><control-D></i> <b><-- here</b>
PYB: soft reboot
Traceback (most recent call last):
File "boot.py", line 17, in <module>
NameError: name not defined
</pre>
<p>The WiPy tells me that there is an error in my file, so I edit it locally and upload it again via the ftp connection. And only at the end when the WiPy is happy, I press the hard-reset button.</p>
<p>Now for reference my boot.by that worked for me:</p>
<p><pre>
from network import WLAN
SSID = 'home_SSID' # SSID of your home network
AUTH = (WLAN.WPA2, 'very_secret') # WPA secret as 2nd param
IP = '10.1.2.42' # Fixed IP the device should get
ROUTER = '10.1.2.3' # IP of your router
DNS = '10.1.2.3' # IP of your DNS server
NETMASK = '255.255.255.0' # Netmask to use
import machine
wlan = WLAN()
if machine.reset_cause() != machine.SOFT_RESET:
wlan.init(WLAN.STA)
# configuration below MUST match your home router settings!!
wlan.ifconfig(config=(IP, NETMASK, ROUTER, DNS))
if not wlan.isconnected():
wlan.connect(ssid=SSID, auth=AUTH, timeout=5000)
while not wlan.isconnected():
machine.idle() # save power while waiting
print('WLAN connection succeeded!')
</pre>
Parts of that script were taken from the <a href="http://micropython.org/resources/docs/en/latest/wipy/wipy/tutorial/wlan.html">WiPy WLAN tutorial</a> and <a href="http://forum.micropython.org/viewtopic.php?f=11&t=992#p5653">this WiPy Forum post</a>.</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0tag:blogger.com,1999:blog-6080082.post-80874094371803178852015-10-06T20:48:00.001+02:002016-01-05T10:48:14.867+01:00Driving a Servo on Arduino from a remote Arduino over secured radio<p>In Stuttgart, there is now a <a href="http://www.meetup.com/de/Hackergarten-Stuttgart/">Hackergarden Meetup group</a> that tinkers with whatever the people showing up want to tinker with.</p>
<p>In the 1st edition I was doing some hacking on Arduino where two Arduinos were transmitting data over an encrypted radio via a RFM69 radio chip. This setup is described <a href="https://blog.codecentric.de/2015/08/diy-leichtgewichtige-verschluesselte-funkuebertragung-2/">on the Codecentric blog</a>.</p>
<p>Now in the 2nd edition we wanted to build on this and control a servo motor on one Arduino remote from the other one. This video shows the end result:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/qSD2HQq_nHc" frameborder="0" allowfullscreen></iframe>
<p>On the left you see a potentiometer, that is read out and the value is then shown on the Neopixel ring. The value is also transmitted via RFM69 chip to the other Arduino that has the receiver and which then drives the servo.</p>
<p>The setup on the server side looks like this:</p>
<div style="text-align:center;"><img src="https://lh3.googleusercontent.com/-5MABJI8OzZk/VhQTR9rUC6I/AAAAAAAAD5c/XrsTQM6-Lqw/poti-data-sender_Steckplatine.png?imgmax=800" alt="Poti data sender Steckplatine" border="0" width="399" /></div>
<p>On the receiver side we used a "normal" setup like the one described on the <a href="https://blog.codecentric.de/2015/08/diy-leichtgewichtige-verschluesselte-funkuebertragung-2/">before mentioned report</a>. We had an issue for a while, as the servo was on a port that was also used by the RFM69 code, but once we fixed that, it worked.</p>
<p>The <a href="https://github.com/pilhuhn/EWHA/blob/master/arduino-rfm69/client-with-poti.ino">client side code is in my GitHub fork</a> of the <a href="https://github.com/hackergarten-stuttgart/EWHA">Hackergarden repository</a> - I hope it will be merged soon :)</p>Heikohttp://www.blogger.com/profile/04989356067273734299noreply@blogger.com0