Update - Just use FreeBSD 15 - it just works!
I’m in the process to modernize our very much aging backup machine. Instead of spending about 300 bucks on new hardware, I want to get it running on hardware that I already have - and what I have is a Turing Pi and a Raspberry Pi Compute Module 4 (8 GiB). The Turing Pi has two SATA ports for Node 3, which are just about perfect for what I intent to do. And for the backup machine I want to use FreeBSD, because my main NAS runs openSUSE and in such a configuration I have a different operating system as backup machine. In this configuration I hope to be more resilient also against (very very very) unlike system corruption issues. Knock on wood.
Also: FreeBSD is an awesome system, and that alone is already an argument for doing this.
TLDR;
You need to fix two issues:
- Limit the system memory as described here
# /boot/config.txt
[pi4]
total_mem=4015
- Do not boot the CM4 with an attached ASMedia SATA Controller, e.g. Node 3 on a Turing Pi board
My story
FreeBSD provides pre-build Raspberry Pi images and several posts suggests that the CM4 should also work with it. So let’s try to run and uh no …
pci1: <OFW PCI bus> on pcib1
...
far: 0x0000000000000000
esr: 0x00000000bf000002
panic: Unhandled System Error
cpuid = 0
time = 1
KDB: stack backtrace:
#0 0xffff00000053427c at kdb_backtrace+0x58
#1 0xffff000004de48c at vpanic+0x1d4
#2 0xffff0000004de2b4 at panic+0x44
#3 0xffff0000008cc4a8 at do_serror+0x3c
#4 0xffff0000008a8af8 at handle_serror+0x3c
Uptime: 1s
Resetting system ...
Well, that didn’t went well. Several Google hits later I discovered some older issues with some boards where one would need to add devmatch_enable="NO" to /etc/rc.conf. This was for FreeBSD 13.2 but it kept popping up, so I gave it a try.
panic: Unhandled System Error
Hmpf.
Several unsuccessful attempts later, including adding some more Raspberry Pi Firmware blobs to the image, I finally stumbled across an important section in the FreeBSD Wiki page of the Raspberry Pi 4, which I should definitely have checked out earlier:
A crash occurs with RPI4 devices that have 4GB or more of memory. … This is a known memory issue in these models with FreeBSD 14.2 due to overlapping address spaces. Perform the following workaround to get FreeBSD 14.2 to boot […]
I did not see the mentioned error message, but what the heck, if this is document I can give it a try. It just adding total_mem=4015 to the [pi4] section in /boot/config.txt. Drummroll please and ………
panic: Unhandled System Error
&@#!%!
Okay okay okay, maybe I just screwed this up, so let’s re-do everything very carefully. This is in general a sane thing to do, especially when frustration rises as I tend to make more mistakes then as I become distracted.
Several failed attempts later, I finally stumbled across a different angle to approach the problem, namely this bug: FreeBSD 14-Release crashes on the RPI CM4 with PCIE to PCI bridge. From the bug report:
pci1: <OFW PCI bus> on pcib1
x0: 0x000000000000dead
x1: 0xffff000000f24840
x2: 0x0000000000000000
....
panic: Unhandled System Error
Looks very familiar, so the issue is with the ASMedia ASM1083 - not sure if it’s exactly this model on the Turing Pi but the error report and panic crash looks very similar to my issue. So I plugged the CM4 out of Node 3 (where the SATA Controller is) and into Node 4 and
panic: Unhandled System Error
WHAT THE HECK!?! Oh wait, there’s still the Memory issue, so let’s try to combine and the two and ….
Bingo! That worked! Finally. So, in order to get FreeBSD 14.3 work on a CM4 with 8 GiB RAM, I needed to limit the memory to 4 GiB in the /boot/config.txt AND also not run it on the Node port with the attached ASMedia SATA Controller.
The devmatch_enable=NO was not needed on my system, but I also have a CM4 without Wifi, so your mileage might vary.
The next steps is to update the system and see if I can make it then boot and work together with the SATA controller. That’s my hope, because if this works, this would be a neat little backup system and FreeBSD would be a really good fit for what I want to do here.
Addendum
Edit the FreeBSD image
Note: This works on FreeBSD, this will not work in Linux!
# mdconfig -a -t vnode -f FreeBSD-14.3-RELEASE-arm64-aarch64-RPI.img -u 0
# mkdir /mnt/boot /mnt/image
# mount -t msdosfs /dev/md0s1 /mnt/boot
# mount -t ufs /dev/md0s2a /mnt/image
# vim /mnt/boot/config.txt # boot config
# vim /mnt/image/etc/rc.conf # rc.conf
# umount /mnt/boot
# umount /mnt/image
# mdconfig -d -u 0
# sync
Boot config
This is what I needed to add to /boot/config.txt to make it boot:
# /boot/config.txt
[pi4]
total_mem=4015