Rye-wheat bread

Another new bread recipe, that I'm right now working on. From the same book as the last experiments, and because of copyright reasons, I won't put the recipe here (Get the book!)

I am very happy that the bread didn't glued too much. In the meantime the kitchen smells amazing, and I can't wait for tomorrow morning to try out this beauty.

For now, good night.

Ginger Sirup

Refreshingly simple recipe with a strong taste. Consider adding fresh peppermint and a lime

Recipe

  • 200g Ginger
  • 2 Lemons
  • 250g Sugar
  • 750ml Water

Peel and cut the Ginger into stripes. Put the Ginger into the water, add the juice of the lemons and cook for 5 minutes. Still in the hot pan, blend it using a hand blender. Then add the sugar and cook it for another 5 minutes.
Put into bottles. Recommended to use within half of a year.

Getting kodi to run nicely on Raspbian

This blog post is about getting Kodi up and running with Netflix on Raspbian. This is not a tutorial, more a collection of notes for myself in order to reproduce the setup.

Basic install

Get a recent version of Raspbian from the Raspberry Pi website. (Or my ftp mirror). Extract it to a fresh micro-SD card and get the system ready. Follow this guide, if you need help. Boot into the system and run a update and install some handy tools

Next, we are gonna harden the system. For that do the following

  • Set root password
  • Rename user pi to something else
  • Remove the NOPASSWD for user pi
  • Permit root-ssh login only via public key

Install Kodi

First add the following repository

Now install kodi

Autostart Kodi

Ok, kodi is installed, now we need to start kodi. For that we create a systemd service by putting the following to /etc/systemd/system/kodi.service

Install the Netflix Plugin

The Netflix Plugin is hosted on GitHub. Download the plugin and install the zip-file. Best option seems to be to put it on a USB-Stick and install it from there. This plugin needs parted or fdisk, you install them on your kodi and then let the plugin install the widevine library, necessary for the DRM. The DRM is by the way also the thing, that was in the way of getting Netflix work in the first place. And it's still a bit of a mess, since we are using here a extraction of the libwidevine from the chromecast.

Well, as long as it works, but it's not nice and probably creating causing one or two times headache until it finally works.

Allow Kodi to reboot and shutdown

This is now more a quick fix, based on the suggestion from here. The original clue was posted long time ago on the kodi forums, but those posts were only of limited help. So, create the following file

Finetuning

I encountered the problem, that if Kodi runs for too long (multiple days), the Netflix plugin stopped working. No errors given, it just won't play a movie again. A quick fix is to schedule a reboot every night using a cronjob

Further work

This is of course just the basic installation. You will need to configure Kodi to your needs (Skins, Addons, Timezone, connect your NAS, ...)

Also, I might create an ansible-playbook to setup this procedure. This looks like a fun project to do on a rainy Sunday.

For now, I'm off, watching some Netflix 🙂

btrfs being notable slow in HDD RAID6

Disclaimer: I don't want to blame anyone. I love btrfs and would very much like to get it running in a good manner. This is a documentation about a use case, where I have some issues

I am setting up a new server for our Telescope systems. The server acts as supervisor for a handful of virtual machines, that control the telescope, instruments and provide some small services such as a file server. More or less just very basic services, nothing fancy, nothing exotic.

The server has in total 8 HDD, configured as RAID6 connected via a Symbios Logic MegaRAID SAS-3 and I would like to setup btrfs on the disks, which turns out to be horribly slow. The decision for btrfs came from resilience considerations, especially regarding the possibility of creating snapshots and conveniently transferring them to another storage system. I am using OpenSuSE 15.0 LEAP, because I wanted a not too old Kernel and because OpenSuSE and btrfs should work nicely together (A small sidekick in the direction of RedHat to rethink their strategy of abandoning btrfs in the first place)

The whole project runs now into problems, because in comparison to xfs or ext4, btrfs performs horribly in this environment. And with horribly I mean a factor of 10 and more!

The problem

The results as text are listed below. Scroll to the end of the section

I use the tool dbench to measure IO throughput on the RAID. When comparing btrfs to ext4 or xfs I notice, the overall throughput is about a factor of 10 (!!) lower, as can be seen in the following figure:

Throughput measurement using dbench - xfs's throughput is 11.60 times as high as btrfs !!!

sda1 and sda2 are obviosly the same disk and created with default parameters, as the following figure demonstrates

My first suspicion was, that maybe cow (copy-on-write) could be the reason for the performance issue. So I tried to disable cow by setting chattr +C and by remounting the filesystem with nodatacow. Both without any noticeable differences as shown in the next two figures

Same dbench run as before, with with chattr +C. No notice-worthy difference
Same dbench run as in the beginning, but with nodatacow mount option. Still, negligible difference

Hmmmm, looks like something's fishy with the filesystem. For completion I also wanted to swap /dev/sda1 and /dev/sda2 (in case something is wrong with the partition alignment) and for comparison also include ext4. So I reformatted /dev/sda1 with the btrfs system (was previously the xfs filesystem) and /dev/sda2 with ext4 (was previously the btrfs partition). The results stayed the same, although there are some minor differences between ext4 and xfs (not discussed here). The order of magnitude difference between btrfs and ext4/xfs remained

Swapping /dev/sda1 and /dev/sda2 and testing dbench on ext4 yields the same results: btrfs performs badly on this configuration

So, here are the results from the figures.

/dev/sda1 - xfsThroughput 7235.15 MB/sec 10 procs
/dev/sda2 - btrfsThroughput 623.654 MB/sec 10 procs
/dev/sda2 - btrfs (chattr +C)Throughput 609.842 MB/sec 10 procs
/dev/sda2 - btrfs (nodatacow)Throughput 606.169 MB/sec 10 procs
/dev/sda1 - btrfsThroughput 636.441 MB/sec 10 procs
/dev/sda2 - ext4Throughput 7130.93 MB/sec 10 procs

And here is the also the text file of the detailed initial test series. In the beginning I said something like a factor of 10 and more. This relates to the results in the text field. When setting dbench to use synchronous IO operations, btrfs gets even worse.

Other benchmarks

Phoronix did a comparison between btrfs, ext4, f2fs and xfs on Linux 4.12, Linux 4.13 and Linux 4.14 that also revealed differences between the filesystems of varying magnitude.

As far as I can interpret their results, they stay rather inconclusive. Sequential reads perform fastest on btrfs, while sequential writes apperently are horrible (Factor 4.3 difference).

For fuck sake

(Sorry, it's in German) (Sorry again, ich bin wütend)

In Anbetracht des aktuellen politischen Klimas: Woher kommt denn nochmals die ganze Politikverdrossenheit? Warum nochmals fühlen sich vor allem jüngere Wähler nicht oder nicht mehr repräsentiert?

PS. Das Outcome der Entscheidung zum Uploadfilter ist hier vollkommen belanglos. Wir können einen Kampf führen und wir können auch verlieren. No hard feelings about that.

Aber WIE in den letzten Wochen mit Kritik umgegangen, wie von seitens einiger EU Parlamentarier bewusst diffamiert und beschwichtigt wurde, ist inakzeptabel.

Das werden wir nicht vergessen. Wir sehen uns bei den nächsten Wahlen.

Landbrot

I just found a new book to work with: "Brotbackbuch Nr. 1" from Lutz Geißler (in German) and love it from the beginning. Every recipe is listed with the effective and absolute time it needs, and apart from the recipies it also incorporates a lot of background informations and craftsmanship knowledge. I came to this book via the (also German) podcast Chaosradio Express, or more specifically, episode 213 - "Brot".

But enough waffle.

My latest creation is the most basic recipe from the book: The "Landbrot"

The recipe is very basic but because I cannot find it publicly at brotbackbuch.de I have to assume it's under copyright. Still some notes about making this bread

  • Plan it well ahead, on the baking day you will need up to 4 hours. Don't start at 8, otherwise you will bring your bread out of the oven at 12. Accelerating bread is not possible or at least not recommendable.
  • I didn't spend the recommended 1 and half hours of the "Stückgare". The result is still a very good bread, but could be a bit more fluffy. Again: don't try to haste bread. That will not work
  • The result is amazing. Very easy to make bread with potential for everyday usage. But I'm still at the beginning
  • I ate a full quarter of it in one morning. It was really good!

Linux 5.0

Yeah, I am running a recent self-made build of Linux 5.0 🙂

Despite the major version number change, there's nothing more special about this version, that with other releases. Still, I find this pretty cool!

Now, back to work ...

ssh config for IPv6

This is just a short note to remind me, how to configure a link-local IPv6 address in the ssh-config

Remember to put two precent signs, otherwise you might get errors similar to the following

IPv6 for the win!

Sourdough bread

Another weekend, another bread.

This time I was trying myself on a very simple sourdough recipe, for a tasty bread without any yeast

The recipe is really simple, and split into two parts: The sourdough and the actual bread:

Sourdough

100g flour, 100g water, 10g sourdough starter

I fed an took care of the sourdough started for about 4 days, until it was active enough to get to the bread. Then take out 10g and mix all ingredients together. Let them stand for about 12-16h at 25 degree.

Bread

Sourdough, 400g flour, 200g water, 10g salt

Mix together, and let stand for about 30 Minutes. Then knead it by hand for some minutes. Let stand for about 6 hours at at 25 degrees.

Bake at 200 degrees for about 40 minutes.

The result. The form is not so round because it was to gluey for the bread form, resulting in half of the bread sticking there and consecutively ending up in the sink 🙁

Initially I declared the other half on the baking tray as failure, but because the oven was hot and I wanted to try the taste I was anyways going to bake it. Turns out, that the bread was kind-of self-healing. Lesson learned: Bread can excuse small (and big) mistakes pretty well. At least this one 🙂

The inner part of the bread has very nice pores and tastes fluffy. Pretty much impressive for a bread without any baker's yeast! Nature gives you all you need, you only have to know how to use it. And it takes it's time. You can't hurry a good bread.

Taste-wise I am a bit disappointment. The bread has many aroma, especially in the crust, but the interior is a bit too sour. The reason is probably, that the sourdough started had too many sour-bacteria and too less yeast cultures, but something that I can make better the next time by selectively feeding the yeasts before starting the sourdough (so called "Hefeführung"). And let the bread-dough stand for not so long, not even to peak height.

Well, it was a try, and for now I am happy with the result. Especially after I wanted already to declare it failed after half of the dough sticking to the bread basket.

And for now: Happy baking!

Gridengine and CentOS 7

... there's life in the old dog yet!

We are still using the Gridengine on some of our high performance clusters and getting that thing running isn't really a piece of cake. Since Oracle bought Sun, things have changed a little bit: First of all, the good old (TM) Sun Grid engine doesn't exist anymore. There are some clones of it, with the most promising candidate being probably the Son of Grid Engine project. This is also what I will refer as gridengine henceforth. Noticeworthy, but not covered is the OpenGrid Scheduler and the commercial Univa Grid Engine (I'm not linking there), which is just the old Sun Grid engine, but sold to Univa and commercially distributed

In the Debian world, there is a gridengine deb packet, which just works nicely as it should do. There was a el6 port for CentOS 6, but there is nothing official for CentOS 7 (yet?). I've build them myself and everyone is free to use them as they are. They are provided as-they-are, so no support or warranty of any kind are provided. Also, they should work just fine as they are

Building the Son of Grid Engine

The process was difficult enough to make me fork the repository and setup my own GitHub project. My fork contains two bugfixes, which prevented the original source from building.
The project contains also build instructions in the README.md for OpenSuSE 15 and CentOS 7 and pre-compiled rpms in the releases section.

Short notes about building

The Gridengine comes with it's own build tool, called aimk. One can say a lot about it, but if treated correctly it works okayish. The list of requirements is long and listed in the README.md for CentOS 7 and OpenSuSE 15. It hopefully also works for any other versions.

SGE uses a lot of different libraries. Mixing architectures for a single cluster environment is in general a bad idea and SGE might work, but you really don't want to hassle with the inevitable white hairs that come with all of the unpredictable and sometimes not-easy-to-understand voodoo errors that occur. Just ... Don't do that!

I never used the Hadoop build, so all binaries and everything is tested with -no-herd.

For the impatient (not commented)

Notes about installing the Gridengine

I've tried to automate the install with ansible, but the install_execd -auto script proves to be quiet unreliable. After several failed attempts, I decided to install the Gridengine manually from a shared NFS directory.

This is in general a good idea, as the spool directory anyways needs to be in a NFS share. To prevent trouble I have separated the binaries (read-only NFS) from the spool directory (read-write access to all nodes).

I've tried to mix CentOS and OpenSuSE. The Gridengine works with each other, but you will run into other problems as the execution environment is different. Don't do that!

Running the SGE over NFS is the way I recommend. Be aware of the hassle, when the master node becomes unresponsive. In that case, don't do magic tricks, just reboot the nodes. Everything else is fishy.

Known problems with Son of Grid engine

This section is dedicated to document two bugs and make them appear on google, so that other unfortunate beings, who encounter the same problems can find a solution. I've encountered two errors, when trying to build the original 8.1.9 version

This problem was the reason for me to fork it. Comment out line 51 in sge-8.1.9/source/3rdparty/qtcsh/sh.proc.c

I encountered this error when building as root. Try building as unprivileged user (which you should do anyways!)

Mirrors

I am mirroring the current version of Son of Grid engine on my ftp-server. My own fork is in the GitHub repository gridengine.