Linux 5.0

Yeah, I am running a recent self-made build of Linux 5.0 🙂

Despite the major version number change, there's nothing more special about this version, that with other releases. Still, I find this pretty cool!

Now, back to work ...

ssh config for IPv6

This is just a short note to remind me, how to configure a link-local IPv6 address in the ssh-config

Remember to put two precent signs, otherwise you might get errors similar to the following

IPv6 for the win!

Sourdough bread

Another weekend, another bread.

This time I was trying myself on a very simple sourdough recipe, for a tasty bread without any yeast

The recipe is really simple, and split into two parts: The sourdough and the actual bread:

Sourdough

100g flour, 100g water, 10g sourdough starter

I fed an took care of the sourdough started for about 4 days, until it was active enough to get to the bread. Then take out 10g and mix all ingredients together. Let them stand for about 12-16h at 25 degree.

Bread

Sourdough, 400g flour, 200g water, 10g salt

Mix together, and let stand for about 30 Minutes. Then knead it by hand for some minutes. Let stand for about 6 hours at at 25 degrees.

Bake at 200 degrees for about 40 minutes.


The result. The form is not so round because it was to gluey for the bread form, resulting in half of the bread sticking there and consecutively ending up in the sink 🙁

Initially I declared the other half on the baking tray as failure, but because the oven was hot and I wanted to try the taste I was anyways going to bake it. Turns out, that the bread was kind-of self-healing. Lesson learned: Bread can excuse small (and big) mistakes pretty well. At least this one 🙂

The inner part of the bread has very nice pores and tastes fluffy. Pretty much impressive for a bread without any baker's yeast! Nature gives you all you need, you only have to know how to use it. And it takes it's time. You can't hurry a good bread.

Taste-wise I am a bit disappointment. The bread has many aroma, especially in the crust, but the interior is a bit too sour. The reason is probably, that the sourdough started had too many sour-bacteria and too less yeast cultures, but something that I can make better the next time by selectively feeding the yeasts before starting the sourdough (so called "Hefeführung"). And let the bread-dough stand for not so long, not even to peak height.

Well, it was a try, and for now I am happy with the result. Especially after I wanted already to declare it failed after half of the dough sticking to the bread basket.

And for now: Happy baking!

Gridengine and CentOS 7

... there's life in the old dog yet!

We are still using the Gridengine on some of our high performance clusters and getting that thing running isn't really a piece of cake. Since Oracle bought Sun, things have changed a little bit: First of all, the good old (TM) Sun Grid engine doesn't exist anymore. There are some clones of it, with the most promising candidate being probably the Son of Grid Engine project. This is also what I will refer as gridengine henceforth. Noticeworthy, but not covered is the OpenGrid Scheduler and the commercial Univa Grid Engine (I'm not linking there), which is just the old Sun Grid engine, but sold to Univa and commercially distributed

In the Debian world, there is a gridengine deb packet, which just works nicely as it should do. There was a el6 port for CentOS 6, but there is nothing official for CentOS 7 (yet?). I've build them myself and everyone is free to use them as they are. They are provided as-they-are, so no support or warranty of any kind are provided. Also, they should work just fine as they are

Building the Son of Grid Engine

The process was difficult enough to make me fork the repository and setup my own GitHub project. My fork contains two bugfixes, which prevented the original source from building.
The project contains also build instructions in the README.md for OpenSuSE 15 and CentOS 7 and pre-compiled rpms in the releases section.

Short notes about building

The Gridengine comes with it's own build tool, called aimk. One can say a lot about it, but if treated correctly it works okayish. The list of requirements is long and listed in the README.md for CentOS 7 and OpenSuSE 15. It hopefully also works for any other versions.

SGE uses a lot of different libraries. Mixing architectures for a single cluster environment is in general a bad idea and SGE might work, but you really don't want to hassle with the inevitable white hairs that come with all of the unpredictable and sometimes not-easy-to-understand voodoo errors that occur. Just ... Don't do that!

I never used the Hadoop build, so all binaries and everything is tested with -no-herd.

For the impatient (not commented)

Notes about installing the Gridengine

I've tried to automate the install with ansible, but the install_execd -auto script proves to be quiet unreliable. After several failed attempts, I decided to install the Gridengine manually from a shared NFS directory.

This is in general a good idea, as the spool directory anyways needs to be in a NFS share. To prevent trouble I have separated the binaries (read-only NFS) from the spool directory (read-write access to all nodes).

I've tried to mix CentOS and OpenSuSE. The Gridengine works with each other, but you will run into other problems as the execution environment is different. Don't do that!

Running the SGE over NFS is the way I recommend. Be aware of the hassle, when the master node becomes unresponsive. In that case, don't do magic tricks, just reboot the nodes. Everything else is fishy.

Known problems with Son of Grid engine

This section is dedicated to document two bugs and make them appear on google, so that other unfortunate beings, who encounter the same problems can find a solution. I've encountered two errors, when trying to build the original 8.1.9 version

This problem was the reason for me to fork it. Comment out line 51 in sge-8.1.9/source/3rdparty/qtcsh/sh.proc.c


I encountered this error when building as root. Try building as unprivileged user (which you should do anyways!)

Mirrors

I am mirroring the current version of Son of Grid engine on my ftp-server. My own fork is in the GitHub repository gridengine.

iphex

I wrote a small bash script, that transforms IP addresses into HEX format. The tool consists of 10 lines of bash script

I needed the tool to match IP-addresses to HEX files for PXE boot. Normally PXE boot fetches first the MAC-address, and then iteratively for the HEX representation of the IP address, with reducing the number of matching characters. Oracle documents the behavior very nicely for the IP address "192.0.2.91"which matches "C000025B" and the imaginary MAC-address "88:99:AA:BB:CC:DD". Then the PXE client probes for the following files (in the given order)

Now, with iphex I can easily convert the more used numerical representation of IP-addresses like 192.168.2.91 into the IMHO not directly visible HEX representation.

zfs - CKSUM errors (and a bad SATA cable)

Just some days ago, I run a zpool scrub on my ZFS RAID-z array on my home NAS. This is actually a piece of beauty - Powered by a low-energy Celeron and using FreeBSD with ZFS, this is our main storage system for photos, pictures, data, disk images, ecc. It just powers the whole home infrastructure. So during a normal routine zpool scrub, I noticed CKSUM errors popping up.

zfs CKSUM errors

Now, this is not good as CKSUM indicates the number of uncorrectable checksum errors. So, rapid action is required!
As the affected disk is a rather old disk, my first guess was, that this is an indicator of a disk going bad. So, first I run smartctl to check the smart status. So, first I checked the glabel to find the underlying physical device

use glabel status to map the output of zpool status to the underlying hardware
Output of smartctl -a /dev/ada4

The smart status looked good, but I still decided to replace the disk, as the CKSUM errors made me nervous. I put into the new disk and replaced it with the default zfs tools via

The zpool replace command immediately exploded into my face. In dmesg I could trace back some weird messages with something like "No such pool or dataset"
Now, that's weird. So apparently the issue was not the disk itself, but what else? Probably it is rather just a faulty connection. Could be a failing SATA controller or just a bad SATA cable. The cable was working just fine for years, I didn't changed anything so I was wondering. Normally I always suspect moving or mechanical parts first to fail, so a plain cable is way below a failing HDD when CKSUM errors appear in my priority list.
Yeah, I was wrong. Just replaced the SATA cable (because I had one and it was orders of magnitude easier than replacing a SATA controller) healed the whole system. Now zpool scrub runs happily through for the second time (the first time it had still to recover some errors, probably from faulty writes in the first place) and now the NAS is running smoothly as it was doing always.

No errors, everything is happy again 🙂

So, in a nutshell

CKSUM errors in ZFS without any READ or WRITE errors are sometimes also just triggered by a faulty connection (a bad SATA cable) and do not necessarily indicate a failing hard disk.

Lucky me, now I have a sparse disk on stock in case one day one disk really goes bad 🙂

Zsh and Home/End/Delete buttons

I've notices that in Zsh under Mate, the HOME/END/DELETE buttons are for some reasons not working as I expected them to work. I use vim keybindings, and am still accustomed to sometimes hit the end button to reach the end of the line. So far it has never been a problem, but zsh just reacts weirdly here. Before triggering a rage quit, I found a solution of how to deal with it. Put the following lines in your .zshrc and you're good

I found this solution here and mirror it on my blog, in case the original solution gets lost or something.

I also link oh-my-zsh here, in case someone just hopped on zsh as well and wants to make it as fancy as possible 🙂

Iceland: Reynisdrangar and Jökulsárlón

The track goes from Vic to Höfn, a but further than Jökulsárlón because it's the next bigger town on the Ringroad. This was actually the longest route because the number of camping spots available to the Camping Card was the lowest here.
After our rainy trip to the Seljalandsfoss and Vestmannaeyjar, like typically Icelandic weather, the next day was clear and sunny. There's a saying "if you don't like the weather in Iceland, just wait for 5 minutes". On that day we had a very good view at the Reynisdrangar - a spectacular volcanic rock formation in the South of Iceland.
View from Vik to Reynisdrangar
After a nice breakfast we headed to our next goal, but before, I want to notice something peculiar for the Icelandic roads: Most bridges (would say almost all outside of the bigger towns) are single lane bridges. Although it sounds dangerous, in practice it works very well. Somehow as a human you seem to be able to communicate who goes first, if you encounter another car on the road. Probably it works so well because there aren't so many cars.
Most of the bridges in Iceland are single-lane
So, expect those bridges and don't be afraid. It works pretty well!

Jökulsárlón

The next goal of our trip was the Jökulsárlón - the blue glacier lagoon located directly at the Ringroad in the South. The place and the lake is absolutely brilliant. Funny part - Everything in Iceland is breeding while summer. Very close to the touristic areas there are lots of bird nests who are frequently attacking people who come too close. As a human you are tolerated here, but not more 😉 Some birds are more clever and breed on ice isles in the middle of the lake
Bird breeding on a floating iceberg
The lake is somehow an Oxymoron, because the sweet taste of the beauty also carries a little bit of a bitter taste: Fed by an abnormal high melting of the nearby glacier, the blue glacier lake is a witness of human made climate change. So, when you walk to the beach, and see all the glacier ice head into the ocean, keep in mind that they are causing the sea levels to rise. And that this phenomenon, as clear as the ice might be, should not occur in nature. But apart from that, the blue ice bergs are actually really beautiful. If you are very lucky you might also encounter a seal. They are rather shy, but sometimes you see them hunting in the icy waters. Swimming is absolutely not possible - In those waters only the experts of nature are home. A human would instantly freeze, the dangers of colliding ice bergs not to mention. We did a guided tour on the lake with a pretty coll amphibian vehicle. It was not cheap, but if you are interested in nice landscaped and a little bit of glaciology you might enjoy it. Doing some day-hikes near the like should be possible, but since we didn't do that I won't comment on that one. And be prepared that this place is crowded with tourists. Sill: The place is absolutely brilliant! Bring your camera 🙂

Playing around with Infragram

[Quicklink to my tool on GitHub]

Yesterday I started to work on my Infragram camera that I bought some years ago from Adafruit. I never got it really running and at some point I just put it away. Until yesterday.

Infragram by Public Labs is a citizen science project that allows you to visualize plant health. It is based on a multi-spectral satellite imaging technique from NASA - the NDVI analysis.
In simple words: It takes the near infrared and the visible light spectrum and runs some (pretty simple) math on them. We come to that in a second.

Although infragram.org provides a online analysis tool, I found that pretty much unsatisfactory. At the end of the day I want to run those things on my own devices. Since I couldn't find useful tools, at some point I decided to write my own.
Luckily there is enough documentation on the Infragram Wiki, that I was able to do that.

Let's start with the basics.

NDVI analysis

The informations of this section are based on the NDVI wiki page on Public Lab. NDVI stands for "Normalized Difference Vegetation Index" and is a technique that monitors how much photosynthesis is happening [5].

See the example picture, taken from [6]

 

In a nutshell: NDVI is a technique to monitor the photosynthesis by differentiating between infrared and visible light pictures. The math behind it is pretty simple

The output is a value between -1 and 1, where higher values indicate high photosynthesis happening.
Of course, this is only valid for light being reflected by plants, reflective surfaces like lakes or the sky do not match the assumption.
The following picture shows a example output of the NDVI technique

Example picture of the NDVI technique [https://publiclab.org/wiki/ndvi]

Infragram Tool

Since I couldn't find any native tool that directly supports Infragram on Linux, I decided to write my own tool. The aim was a easy-to-use zero-knowledge tool to just start. At the end of the day all I want to do, is to connect the Infragram camera to my laptop/raspberry pi/NanoPi Neo or whatever and take NDVI pictures. So I designed it to work in the following way

Results

I wrote the tool and played a little bit around with it

First picture in daylight. Left: Camera, Right: NDVI

In the daylight, the camera is wayyyy to oversaturated. The resulting picture is not really meaningful. So let's take the plant indoors, where it still gets indirect sunlight.

Left: Camera, Right: ndvi

Ok, looks already pretty good! The chosen colormap is not really good, that's something I have to work on. Now, let's see if the output is actually reflecting how good the plant is in health. I put a piece of paper in front of the plant to cover the IR emitted light.

I'm covering the observed plant with a piece of paper to have a look at the NDVI technique. The paper should be clearly visible as non-healthy part of the plant.

So, how does it look in the end?

Plant with piece of paper to cover IR light. Left: Camera, Right: NDVI

Ok, I can see the piece of paper but on the places where the leaves are pretty close the IR emission seems to penetrate. This is actually a good sign, because it reflects the nature of the IR emission. Next, I want to try how it looks if I thicken the paper

With the folded paper as emission blocker, let's have a look at the picture of the Infragram camera

Two pieces of paper, folded two times in front of the plant. Left: Camera, Right: NDVI

Hmm, I still can see the paper, but the expected behaviour (sharpt edges and cut-out paper) did not happen. Maybe the paper itself is a good in IR emission of the scattered ambient light. I don't know for sure, but yet, the results are OK for the first tries.

I now have a working prototype that runs on my Laptop and can start to further dig deeper into the topic. At least that's something very nice! 🙂

Further work

The provided tool is a working prototype, but the colormap and the NDVI technique are not yet fully utilized. As of now, this might be a good starting point for some other people, but the software is not yet as mature as it can be used for anything but playing around.

Weblinks

Ring Road: Seljalandsfoss and Vestmannaeyjar

This is the second blog post of our Ring Road journey in Iceland. The first one was about the Golden Circle.

In the first part there is a missing segment, which is from the trip from the Skjól camping place, where we slept the first night. In fact we split the Golden Circle into two days, so the trip from Skjól to the Brautarhold camping place is done via a deviation. Anyways, here are the corresponding tracks: [GPX: Golden Circle - Brautarhold] and [GPX: Vestmannaeyjar and Seljalandsfoss]. Both files together complete the track, this post is about the second one.

 

Vestmannaeyjar

This was a day with a lot of driving around. And one of the days, where you discover another side of the true nature of Icelandic weather: Rain and wind. As said before: Take into account that you will encounter at least one full rain day. At least. But it was fun!
The temperature drops to about 10 degrees (we are in high summer!), so be prepared to take adequate clothing with you. We spend the full day in car, so it wasn't very much of a problem, but if you are on the go for the day you're gonna need some warm stuff and good rain protection. Did I mentioned the wind? Yes, it's windy.

Vestmannaeyjar is a nice island in the south of Iceland. It's reachable via ferry (and probably via plane) from the very near end of Landeyjahöfn for a reasonable fee. Vestmannaeyjar is known for their Puffins colony, very adorable little creatures that you should definitely visit! We had the chance on the North-East end, so don't worry, they are also elsewhere. And just because they are so adorable, here's a picture of one of them:

I am a Puffin!

In fact, I was so excited about them, that I dedicated a own photo album on my online photobook. But enough for now, Puffins will definitely get their own blog post.

But there is something nice about Puffins that is only in the to Vestmannaeyjar. First: Puffin chicks are called pysja. Keep that in mind for now.
In fall the Puffins leave the breeding nest for the open sea, where they spend the winter (yes, those little fluffy things are in the freezing cold water in winter!). Until then, the pysja get fed by their parents, mostly Raitt’s sand eels (known as lesser sand eel). See here:

Puffin with those little eels they feed to their children (Puffin chicks are called "pysja")

Ok, now when the chicks are big enough, the parents stop feeding them so that they get out of the nest. Sounds harsh, but sometimes you need a kick in order to prevail. That's just how it works for them ...
So, hungry pysja wants to have food! And so the journey to the sea begins. Just like humans, the birds get attracted by shiny lights, so every year there are a lot of unfortunate and confused pysjas invading the nearby town. Frightend by the noise and the unusual environment, they crawl under cars and wander around plan-less and frightened. That'ts where the inhabitants of Vestmannaeyjar come for the rescue - every fall the children go out and collect the pysjas and bring them to the sea the next morning. Some of the more exhausted pysjas even get nursed before being released. An act of kindness - I find this absolutely brilliant! Maybe one day I have the chance to go there and somehow participate in this event.

For more informations about the pysjas rescue, I refer to the iceland magazine page here.

So, we wanted to take the ferry to Vestmannaeyjar, but once we stood at the shore and got a full load of the bad weather and practically no visibility we decided not to do that. We anyways got another chance to visit those little adorable creatures, but that will be another post in the future 🙂

Seljalandsfoss

On the way back from Landeyjahöfn we encounter another beautiful and well-known waterfall of Iceland: The Seljalandsfoss. Usually it's quiet crowded there during the day, but still it's worth going there and enjoy the beauty that nature can serve you. I didn't managed to really take a breathtaking picture, but other did and posted a lot of very nice pictures on unsplash. I'm leaving you here with my stuporous tries of creating something artistic.

Although some picture have been turned out to not to be too bad

This one I like a lot. I also pushed it to unsplash

After that we drove to the Vík í Mýrdal camping place. There was another camping spot in between: Skógar Camping, that was also included in the camping card, but due to the bad weather we had the time to skip that and therefore have more time on the upcoming destinations. In the end there are so many things to do, that you anyways won't be able to do all of them at once.

If you can, you should also go to Dyrhólaey, it's just on the way.

Dyrhólaey, it was a very rainy day

You will also see the Reynisdrangar, three volcanic rocks in the sea, but that is gonna be part of the next post 🙂