zfs - CKSUM errors (and a bad SATA cable)

Just some days ago, I run a zpool scrub on my ZFS RAID-z array on my home NAS. This is actually a piece of beauty - Powered by a low-energy Celeron and using FreeBSD with ZFS, this is our main storage system for photos, pictures, data, disk images, ecc. It just powers the whole home infrastructure. So during a normal routine zpool scrub, I noticed CKSUM errors popping up.

zfs CKSUM errors

Now, this is not good as CKSUM indicates the number of uncorrectable checksum errors. So, rapid action is required!
As the affected disk is a rather old disk, my first guess was, that this is an indicator of a disk going bad. So, first I run smartctl to check the smart status. So, first I checked the glabel to find the underlying physical device

use glabel status to map the output of zpool status to the underlying hardware
Output of smartctl -a /dev/ada4

The smart status looked good, but I still decided to replace the disk, as the CKSUM errors made me nervous. I put into the new disk and replaced it with the default zfs tools via

The zpool replace command immediately exploded into my face. In dmesg I could trace back some weird messages with something like "No such pool or dataset"
Now, that's weird. So apparently the issue was not the disk itself, but what else? Probably it is rather just a faulty connection. Could be a failing SATA controller or just a bad SATA cable. The cable was working just fine for years, I didn't changed anything so I was wondering. Normally I always suspect moving or mechanical parts first to fail, so a plain cable is way below a failing HDD when CKSUM errors appear in my priority list.
Yeah, I was wrong. Just replaced the SATA cable (because I had one and it was orders of magnitude easier than replacing a SATA controller) healed the whole system. Now zpool scrub runs happily through for the second time (the first time it had still to recover some errors, probably from faulty writes in the first place) and now the NAS is running smoothly as it was doing always.

No errors, everything is happy again 🙂

So, in a nutshell

CKSUM errors in ZFS without any READ or WRITE errors are sometimes also just triggered by a faulty connection (a bad SATA cable) and do not necessarily indicate a failing hard disk.

Lucky me, now I have a sparse disk on stock in case one day one disk really goes bad 🙂

Zsh and Home/End/Delete buttons

I've notices that in Zsh under Mate, the HOME/END/DELETE buttons are for some reasons not working as I expected them to work. I use vim keybindings, and am still accustomed to sometimes hit the end button to reach the end of the line. So far it has never been a problem, but zsh just reacts weirdly here. Before triggering a rage quit, I found a solution of how to deal with it. Put the following lines in your .zshrc and you're good

I found this solution here and mirror it on my blog, in case the original solution gets lost or something.

I also link oh-my-zsh here, in case someone just hopped on zsh as well and wants to make it as fancy as possible 🙂

Iceland: Reynisdrangar and Jökulsárlón

The track goes from Vic to Höfn, a but further than Jökulsárlón because it's the next bigger town on the Ringroad. This was actually the longest route because the number of camping spots available to the Camping Card was the lowest here.
After our rainy trip to the Seljalandsfoss and Vestmannaeyjar, like typically Icelandic weather, the next day was clear and sunny. There's a saying "if you don't like the weather in Iceland, just wait for 5 minutes". On that day we had a very good view at the Reynisdrangar - a spectacular volcanic rock formation in the South of Iceland.
View from Vik to Reynisdrangar
After a nice breakfast we headed to our next goal, but before, I want to notice something peculiar for the Icelandic roads: Most bridges (would say almost all outside of the bigger towns) are single lane bridges. Although it sounds dangerous, in practice it works very well. Somehow as a human you seem to be able to communicate who goes first, if you encounter another car on the road. Probably it works so well because there aren't so many cars.
Most of the bridges in Iceland are single-lane
So, expect those bridges and don't be afraid. It works pretty well!

Jökulsárlón

The next goal of our trip was the Jökulsárlón - the blue glacier lagoon located directly at the Ringroad in the South. The place and the lake is absolutely brilliant. Funny part - Everything in Iceland is breeding while summer. Very close to the touristic areas there are lots of bird nests who are frequently attacking people who come too close. As a human you are tolerated here, but not more 😉 Some birds are more clever and breed on ice isles in the middle of the lake
Bird breeding on a floating iceberg
The lake is somehow an Oxymoron, because the sweet taste of the beauty also carries a little bit of a bitter taste: Fed by an abnormal high melting of the nearby glacier, the blue glacier lake is a witness of human made climate change. So, when you walk to the beach, and see all the glacier ice head into the ocean, keep in mind that they are causing the sea levels to rise. And that this phenomenon, as clear as the ice might be, should not occur in nature. But apart from that, the blue ice bergs are actually really beautiful. If you are very lucky you might also encounter a seal. They are rather shy, but sometimes you see them hunting in the icy waters. Swimming is absolutely not possible - In those waters only the experts of nature are home. A human would instantly freeze, the dangers of colliding ice bergs not to mention. We did a guided tour on the lake with a pretty coll amphibian vehicle. It was not cheap, but if you are interested in nice landscaped and a little bit of glaciology you might enjoy it. Doing some day-hikes near the like should be possible, but since we didn't do that I won't comment on that one. And be prepared that this place is crowded with tourists. Sill: The place is absolutely brilliant! Bring your camera 🙂

Playing around with Infragram

[Quicklink to my tool on GitHub]

Yesterday I started to work on my Infragram camera that I bought some years ago from Adafruit. I never got it really running and at some point I just put it away. Until yesterday.

Infragram by Public Labs is a citizen science project that allows you to visualize plant health. It is based on a multi-spectral satellite imaging technique from NASA - the NDVI analysis.
In simple words: It takes the near infrared and the visible light spectrum and runs some (pretty simple) math on them. We come to that in a second.

Although infragram.org provides a online analysis tool, I found that pretty much unsatisfactory. At the end of the day I want to run those things on my own devices. Since I couldn't find useful tools, at some point I decided to write my own.
Luckily there is enough documentation on the Infragram Wiki, that I was able to do that.

Let's start with the basics.

NDVI analysis

The informations of this section are based on the NDVI wiki page on Public Lab. NDVI stands for "Normalized Difference Vegetation Index" and is a technique that monitors how much photosynthesis is happening [5].

See the example picture, taken from [6]

 

In a nutshell: NDVI is a technique to monitor the photosynthesis by differentiating between infrared and visible light pictures. The math behind it is pretty simple

The output is a value between -1 and 1, where higher values indicate high photosynthesis happening.
Of course, this is only valid for light being reflected by plants, reflective surfaces like lakes or the sky do not match the assumption.
The following picture shows a example output of the NDVI technique

Example picture of the NDVI technique [https://publiclab.org/wiki/ndvi]

Infragram Tool

Since I couldn't find any native tool that directly supports Infragram on Linux, I decided to write my own tool. The aim was a easy-to-use zero-knowledge tool to just start. At the end of the day all I want to do, is to connect the Infragram camera to my laptop/raspberry pi/NanoPi Neo or whatever and take NDVI pictures. So I designed it to work in the following way

Results

I wrote the tool and played a little bit around with it

First picture in daylight. Left: Camera, Right: NDVI

In the daylight, the camera is wayyyy to oversaturated. The resulting picture is not really meaningful. So let's take the plant indoors, where it still gets indirect sunlight.

Left: Camera, Right: ndvi

Ok, looks already pretty good! The chosen colormap is not really good, that's something I have to work on. Now, let's see if the output is actually reflecting how good the plant is in health. I put a piece of paper in front of the plant to cover the IR emitted light.

I'm covering the observed plant with a piece of paper to have a look at the NDVI technique. The paper should be clearly visible as non-healthy part of the plant.

So, how does it look in the end?

Plant with piece of paper to cover IR light. Left: Camera, Right: NDVI

Ok, I can see the piece of paper but on the places where the leaves are pretty close the IR emission seems to penetrate. This is actually a good sign, because it reflects the nature of the IR emission. Next, I want to try how it looks if I thicken the paper

With the folded paper as emission blocker, let's have a look at the picture of the Infragram camera

Two pieces of paper, folded two times in front of the plant. Left: Camera, Right: NDVI

Hmm, I still can see the paper, but the expected behaviour (sharpt edges and cut-out paper) did not happen. Maybe the paper itself is a good in IR emission of the scattered ambient light. I don't know for sure, but yet, the results are OK for the first tries.

I now have a working prototype that runs on my Laptop and can start to further dig deeper into the topic. At least that's something very nice! 🙂

Further work

The provided tool is a working prototype, but the colormap and the NDVI technique are not yet fully utilized. As of now, this might be a good starting point for some other people, but the software is not yet as mature as it can be used for anything but playing around.

Weblinks

Ring Road: Seljalandsfoss and Vestmannaeyjar

This is the second blog post of our Ring Road journey in Iceland. The first one was about the Golden Circle.

In the first part there is a missing segment, which is from the trip from the Skjól camping place, where we slept the first night. In fact we split the Golden Circle into two days, so the trip from Skjól to the Brautarhold camping place is done via a deviation. Anyways, here are the corresponding tracks: [GPX: Golden Circle - Brautarhold] and [GPX: Vestmannaeyjar and Seljalandsfoss]. Both files together complete the track, this post is about the second one.

 

Vestmannaeyjar

This was a day with a lot of driving around. And one of the days, where you discover another side of the true nature of Icelandic weather: Rain and wind. As said before: Take into account that you will encounter at least one full rain day. At least. But it was fun!
The temperature drops to about 10 degrees (we are in high summer!), so be prepared to take adequate clothing with you. We spend the full day in car, so it wasn't very much of a problem, but if you are on the go for the day you're gonna need some warm stuff and good rain protection. Did I mentioned the wind? Yes, it's windy.

Vestmannaeyjar is a nice island in the south of Iceland. It's reachable via ferry (and probably via plane) from the very near end of Landeyjahöfn for a reasonable fee. Vestmannaeyjar is known for their Puffins colony, very adorable little creatures that you should definitely visit! We had the chance on the North-East end, so don't worry, they are also elsewhere. And just because they are so adorable, here's a picture of one of them:

I am a Puffin!

In fact, I was so excited about them, that I dedicated a own photo album on my online photobook. But enough for now, Puffins will definitely get their own blog post.

But there is something nice about Puffins that is only in the to Vestmannaeyjar. First: Puffin chicks are called pysja. Keep that in mind for now.
In fall the Puffins leave the breeding nest for the open sea, where they spend the winter (yes, those little fluffy things are in the freezing cold water in winter!). Until then, the pysja get fed by their parents, mostly Raitt’s sand eels (known as lesser sand eel). See here:

Puffin with those little eels they feed to their children (Puffin chicks are called "pysja")

Ok, now when the chicks are big enough, the parents stop feeding them so that they get out of the nest. Sounds harsh, but sometimes you need a kick in order to prevail. That's just how it works for them ...
So, hungry pysja wants to have food! And so the journey to the sea begins. Just like humans, the birds get attracted by shiny lights, so every year there are a lot of unfortunate and confused pysjas invading the nearby town. Frightend by the noise and the unusual environment, they crawl under cars and wander around plan-less and frightened. That'ts where the inhabitants of Vestmannaeyjar come for the rescue - every fall the children go out and collect the pysjas and bring them to the sea the next morning. Some of the more exhausted pysjas even get nursed before being released. An act of kindness - I find this absolutely brilliant! Maybe one day I have the chance to go there and somehow participate in this event.

For more informations about the pysjas rescue, I refer to the iceland magazine page here.

So, we wanted to take the ferry to Vestmannaeyjar, but once we stood at the shore and got a full load of the bad weather and practically no visibility we decided not to do that. We anyways got another chance to visit those little adorable creatures, but that will be another post in the future 🙂

Seljalandsfoss

On the way back from Landeyjahöfn we encounter another beautiful and well-known waterfall of Iceland: The Seljalandsfoss. Usually it's quiet crowded there during the day, but still it's worth going there and enjoy the beauty that nature can serve you. I didn't managed to really take a breathtaking picture, but other did and posted a lot of very nice pictures on unsplash. I'm leaving you here with my stuporous tries of creating something artistic.

Although some picture have been turned out to not to be too bad

This one I like a lot. I also pushed it to unsplash

After that we drove to the Vík í Mýrdal camping place. There was another camping spot in between: Skógar Camping, that was also included in the camping card, but due to the bad weather we had the time to skip that and therefore have more time on the upcoming destinations. In the end there are so many things to do, that you anyways won't be able to do all of them at once.

If you can, you should also go to Dyrhólaey, it's just on the way.

Dyrhólaey, it was a very rainy day

You will also see the Reynisdrangar, three volcanic rocks in the sea, but that is gonna be part of the next post 🙂

Kernel build bug - KVM_AMD and CRYPTO_DEV_CPP

About a week ago, I failed to build a Kernel for my new Ryzen 2700X working machine. After some time of configuring my kernel I run into some weird problems

The problem

I wanted to have a Kernel with KVM_AMD support enabled. The build was going on fine, until some weird linker errors appeared.

(Full output [Pastebin])

Since I'm a Kernel rookie, it took me some time to realize what was going on. A google search didn't revealed a solution, other than something similar on Unix Stackexchange, that was not directly applicable for my case.

The problem persisted and is reproducible in linux-4.17.1 and linux-4.16.15, using this config file. Building linux-4.14.49 was doing fine. For any options that were not defined by the config file I chose the default suggestion.


Workaround

The problem arises, if CONFIG_CRYPTO_DEV_CCP_DD is compiled as module [=m], also if the SEV is not used. Enabling CONFIG_CRYPTO_DEV_CCP_DD to be compiled in the kernel [=y] is a workaround for the issue.

This commit already revealed the issue.

I had to Include the "Secure Processor device driver", that is found in Cryptographic API > Hardware crypto devices

Weirdly, the suggested solution from Unix Stackexchange was not solving the problem for me, neither was it causing problems. I could build the Kernel (4.17.1) with "Kernel-based Virtual Machine Support" set as module. But those are just my two cents, it might have been an issue some versions ago ...

Unluckily I cannot contribute to Unix Stackexchange yet (not enough reputation *sigh*), so I cannot improve the answer there.

Thanks to Richard!

Many thanks to Richard, who provided me with support, regarding nailing it down to a bug in the Kernel build system.

Ringroad - Golden circle

This is the first part of my series about the Ringroad. For general suggestions, go to the introduction page. Here I'm going to tell you about the first day or days in the middle of the beautiful Þingvellir national park. You can download the GPX track here.

Click title to show track
Golden Circle

(For some weird reson, the names and descriptions of the waypoints are not working) [Download GPX]

Nearby Reykjavik there are a lot of astonishing and breathtaking natural wonders: The Þingvellir national park itself (pronounced Thingvellier), the Geysir (such famous it is the eponym to the geysers)  and the Gullfoss waterfall with it's peculiar tectonic form. All three of them are absolutely worth going there, you don't want to miss one. They are also reachable via car, and, if you are fast, you can visit almost all of them in a single day.
I suggested already in the Ringroad introduction that you might want to take a day or two at least as buffer, and the Golden Circle is a place where you could consider spending a little bit more time, especially if you are lucky and have a couple of sunny days. In this story I follow our route and start with the Þingvellir national park.

Þingvellir national park

(Pronounced more like "Think-vellir" and not like a "P", but I'm not a native speaker, so who am I to teach you?)

You're approaching the park from the West, and after a nice ride you will see it's lake. The pictures are only covering part of it's beauty, most of the time I've been driving and didn't had much more time to make even more pictures.
The park has a inviting landscape and I can totally see it as a very nice place for some day-hikes. If I would live nearby (including Reykjavik) I could totally see myself going for a lot of hikes there.

As you might notice, there are not much woods in Iceland. This is true, Iceland's woods were razed by the vikings. The forest service says, in the 9th century, when the settlement begun, about 25-40% of Icelands area contained birch forests and woodland. Around mid of the 20th century the wood reached a minimum, with only 1% of the area being the typical Icelandic birchwoods. Nowadays there are re-forestation programs established and the people try to re-establish different types of forests, with birch being the governing type. [Forest Service]

Anyways, Þingvellir is absolutely brilliant to visit from the landscape point of view. It's between two tectonic plates and in the middle of the national park you can see a valley, formed by the plates drifting away. Because there is a nearby lake, between the tectonic plates, there is a (not too deep) void, where you can go diving: in Silfra. We decided already in Reykjavik to go on a snorkeling tour there. It's costy but it was worth it: The water is crystal clear, and you're diving in a world filled with blue and green. The colors are amazing and it's an experience you won't regret. Silfra advertises the experience with the "clearest water on earth" (not commenting on that) Also, it's cold, but you will get a dry-suit.

 

This place is historical. Þingvellir was the place for the historical parlament of Iceland, called the Althing. They held session in the park until 1793 [Wikipedia], and you can still see the place, where the people met. It's in the middle of the national park, and it's open for visitors. That is one of the reason, why Þingvellir is also a UNESCO World heritage site.

Oh, and geese are doubtful the governing species in the national park. Don't mess with them! 🙂

Goose are the dominant species in the national park

Geysir Geothermal Area

This is the second place we visited in the Golden Circle.

One of the hot tubs in the Geysir park

Geysir is a active geothermal region with boiling mud pits, hot water tubs and the active Strokkur geysir, that spits water up every few (6-10) minutes. The features picture of this post is an eruption of Strokkur.

Experiencing an eruption was something I haven't seen anywhere else before. It was overwhelming and kind of magic. It teaches you everything you need to know about how small you are as a human being. And Strokkur is rather a small geyser. Geysir (it's the name of the big one in this location) was the name-giver of geysers and erupted in 1845 with a peak altitude of about 170m. Wow!

Geysers are by the way also a very interesting physical phenomena. The water heats up under high pressure until it starts boiling. As it boils, water evaporates, relieving some of the pressure, thus causing a run-away process of evaporation and condensation that blows finally up as the natural phenomena.
But that's maybe stuff for an extra post, if I find time for going into the physics of geysers 😉

... And you can buy canned Icelandic Fresh air in the souvenir shop ... For whatever reasons you would want to buy that ...

Canned fresh mountain air. Who buys such things?!?

Concluding: You would definitely miss out something, if you are not going to visit the Geysir area!

Gullfoss Waterfall

The Gullfoss waterfall was our final stop of the day. It was shaped by two colliding tectonic plates forming a fissue canyon in a unique way The waterfall has it's own website and is a touristic attraction. You might find it a bit crowded, but the waterfall is worth going there and you still will get your peaceful moments.
The Gullfoss is beautiful, but, because you might encounter a lot of other waterfalls (like the not-too-close Öxarárfoss) on the way, this is the only stop that you might consider skipping. But then you have not completed the full Golden Circle, but it's up to you to decide 😉

Flying your drones is prohibited, by the way. For valid reasons.


That's for now, the Golden Circle is definitely worth visiting. It's close enough to Reykjavik to be also a stand-alone day trip with a rental car and a good kickoff for a very nice road trip around Iceland.

I can also imagine, that a helicopter or airplane ride around the Golden Circle can be absolutely breathtaking, but leave it to the ecological side and the financial situation of the individual traveler to actually consider it.

Iceland - Ring Road

There are some upcoming blog posts about our amazing trip on the Ring Road. Since it was a long trip and there were a lot of different sites on the way, I'm going to split that up to a couple of individual blog posts. This series is about our absolutely brilliant roadtrip on the ...

... Ring Road!

The Ring Road - It's the capital road in Iceland that goes around the whole island. Except for a small part it is paved, and worth a road-trip. Going by bike is clearly not recommended - although you will encounter some poor souls, you are constantly on the main road next to busses and tourists in cars. Also the weather conditions are not favorable for biking: Constant wind and frequent rain showers make your days on the bike hard.

A road trip on the other hand is nice. And there are a lot of places you should go on the trip.

Getting started

Best starting point is Reykjavik. Most of the travelers will anyways start from there, because Keflavík the only international airport in Iceland. If you come with your own car then Seyðisfjörður might also be a starting point, because of it's ferry connection with Denmark and the Faroe Islands. This story is about our starting point Reykjavik, you can adapt it to your needs if you're starting from Seyðisfjörður (and there's nothing wrong with that!)

We have been renting a car for about 10 days which was nice but not cheap. Also if possible book your car well before you go there, that saves you lots of trouble and precious travel time. In general I have the following recommendations

  • Plan your trip well ahead
  • Book your car well ahead (3 months is safe, 1 month is possible everything later could become tricky*)
  • If you stay at the Ringroad and it's main branches, you won't need a 4-wheel drive
  • Drive peacefully

* a friend of mine was able to book the car on the spot. I call it a lucky shot, although it was proof that it is possible. Decide for yourself if you want to have that or rather be safe 🙂

Planning your trip

Plan your trip well ahead of time, that safes you trouble. Book your car before you go there.

Although the ring road is well marked, you might consider getting your own GPS or navigational system with you. Since most cars have a power-plug, a not-too-old smartphone with offline maps (I recommend OsmAnd, but you can take whatever you like) works fine. Don't pay the additional fee for a navigational system but get a backup that doesn't depend on electricity (classical maps are awesome). Also: Driving around in Iceland will probably not challenge your navigational skills if you stay on the Ringroad 😉

The Ringroad is mostly paved - In the east there was (or is?) a small section of gravel road, but all of it is easily accessible with a normal 2-wheel drive. They will explicitly tell you to avoid the F-Roads (which are more adventurous roads) where you will need 4-wheel drive but I'm not going to cover that. If you're want to go on a Ringroad-road-trip, then avoid F-roads at all costs.

When going to pick up your car, take your time: Check your car before you go and make pictures of small damages. Because some people go on gravel roads, small bunks can not be excluded, and you want to document them. Do it properly, it safes you trouble. Also ask for gas discounts cards, as some car rental services have such cards for you.

On the road, drive peacefully. First because there are a lot of animals on and besides the roads. Farmers will let you pay if you hit their animals. Second: The roads are in good shape but the surroundings are not car-friendly. Also: Enjoy your trip. You're not on a race 🙂

So, in a nutshell, those are my recommendations based on our experience

  • Plan your trip and book the car at least 3 months before
    • Plan the places you want to visit. We used the the Lonely Planet book about Iceland, but there is plenty of good literature
    • Plan the accommodations. We used the Camping Card. I wrote a blog post about getting the locations of the Camping Card for your navigation/smartphone
  • Have a look at other travel blogs, what they write about Iceland
    I especially recommend I heart Reykjavik, a blog that I still follow
  • Going to restaurants is in general very expensive, so you might consider camping food style
    • Bring your camping cooking equipment, but buy the gas on the Island itself. Most gas stations sell camping gas
  • Bring offline maps or your smartphone but don't pay the navigational system offered by some car rental services
    • I recommend OsmAnd but any other OpenStreetMap based offline app will do the job
  • Plan a day or two buffer. You might want to enjoy some places a bit more 🙂
    • Or you cannot visit a place because of shitty weather
  • The weather changes A LOT
  • Gas stops might be long away from each other. In peripheral areas pump gas ahead of time
  • Road bridges are in general one lane only, but it works just fine
  • Drive peacefully, there are animals all over the place
  • Part of the Ringroad is gravel. Not much, not a problem, but be aware of it
  • The trip is gonna be amazing!

Stay in touch, more posts about our trip with recommendations and know-abouts are on the way.

Resizing a btrfs parition

This is a simple note to myself, in case I need to re-do this again. How to resize a btrfs partition to maximum size (or full capacity)

  1. Resize the partition using parted
  2. Resize the btrfs filesystem using

    Would it make sense to create an alias, so that also btrfs filesystem resize 100% or other percentages would work?

In a nutshell example

Detailed example

First resize the partition (I use parted for that purpose)

Done.