Gentoo linux will boot into a ZFS rootfilesystem on a raspberrypi 4 and CM4. Initial impression is that it is stable and some operations are extremely quick. The ZFS root is compressed with lz4 and I currently get a 1.45x compression ratio. The hdparm reported buffered read speed is 356Mb/s which is that maximum I could obtain on my CM4 developer board using a PCIe connected SSD.

Key steps include recompiling the RaspberryPi kernel 5.15 with the ZFS modules built in (currently 2.19) in Gentoo. Then using genkernel to create a ZFS compatible initramfs.

I got lost trying to get UEFI, U-Boot and GRUB to work together, but they are not necessary. If you copy the kernel image with ZFS built-in and the initramfs to the boot partition and alter the config.txt and cmdline appropriately the GPU bootloader works great.

I am going to try and adding multiple devices to see if that breaks things! ZFS advantages include compression and integrity and the performance on the Pi4 and CM4 is excellent.

Multiple devices works too with the expectant speed increase. 270Mb/s over USB with zstd-9 stripped RAID 0 (ZFS). LZ4 compression seems to be much quicker than zstd on the Pi but as expected the compression ration with zstd-9 is better 1.72x

My current route is FreeBSD with a pre-built image avoiding the fragile compiling hassle: The ZFS filesystem via FreeBSD on a Raspberry Pi: A private backup server

There is a recommendation not to use ZFS on Linux (ZoL):

redvli wrote: ↑

Sun Mar 12, 2023 5:51 am

I know from the good experience with some raidz test array with FreeBSD for Intel that I don't want to waste my time with Ubuntu or any other ZoL solution.

However, I don't have details on that.

rasbpi64 wrote: ↑

Sun Mar 12, 2023 4:27 pm

There is a recommendation not to use ZFS on Linux (ZoL):

redvli wrote: ↑

Sun Mar 12, 2023 5:51 am

I know from the good experience with some raidz test array with FreeBSD for Intel that I don't want to waste my time with Ubuntu or any other ZoL solution.

However, I don't have details on that.

That is a quote about running ZFS on legacy x86 systems, not ARM or even Raspberry Pi. I don't understand why ZFS working well on x86 versions of FreeBSD in any way recommends against using ZFS in Linux on a Pi.

ejolson wrote: ↑

Sun Mar 12, 2023 4:52 pm

That is a quote about running ZFS on legacy x86 systems, not ARM or even Raspberry Pi. I don't understand why ZFS working well on x86 versions of FreeBSD in any way recommends against using ZFS in Linux on a Pi.

It's about software, not hardware, see the update here: viewtopic.php?p=2089553#p2089553

Thanks for the info.

I have tried ZFS root with FreeBSD on the Pi and it's excellent.

The problem I had with FreeBSD was getting the GPU working with MESA and Wayland - how did you accomplish this?

With Gentoo/ZoL I have a completely customised OS with the advantages of ZFS and native GPU support with Wayland and no screen tearing. It's odd but I always find the response time and interface much better than with Raspberry Pi OS and that's on a 4K display!

timmawhinney wrote: ↑

Sat Apr 01, 2023 8:24 am

I have tried ZFS root with FreeBSD on the Pi and it's excellent.

The problem I had with FreeBSD was getting the GPU working with MESA and Wayland - how did you accomplish this?

I was using my Raspberry Pi 3 Model B+ as a headless server only, so an SSH access was sufficient for me. No need to get the GPU working.
However, due to USB issues on FreeBSD I gave up for now: viewtopic.php?p=2093171#p2093171

I have used NVMe drives attached via PCIe on the CM4 developer board and NVMe drives attached via USB enclosures with no problem. ZFS also works with USB memory sticks - I don't think there is a need for ECC memory any more.

timmawhinney wrote: ↑

Tue Apr 11, 2023 12:03 pm

I have used NVMe drives attached via PCIe on the CM4 developer board and NVMe drives attached via USB enclosures with no problem. ZFS also works with USB memory sticks - I don't think there is a need for ECC memory any more.

My understanding is that the decision whether or not to use ECC memory with ZFS is based on the idea that the error rate during the scrub should be less then the error rate of the drives themselves.

Seen another way, if you have installed ZFS to detect silent data corruption on the drive, then you are likely interested in detecting silent data corruption in RAM.

Correcting such errors becomes a daily event with petabytes of storage but not so much for a Pi with few USB drives.

ejolson wrote: ↑

Tue Apr 11, 2023 1:45 pm

My understanding is that the decision whether or not to use ECC memory with ZFS is based on the idea that the error rate during the scrub should be less then the error rate of the drives themselves.

There was a case where metadata was corrupted in memory before computing the checksum, such that the data wasn't accessible anymore (but scrub didn't return/correct any errors since the checksum made over the corrupted metadata block was correct):

"That said, bitflips can happen anywhere. One place that is interesting would be a buffer for a write before the checksum is done. That would cause bad data to be written with a good checksum and if it is metadata, then in an extreme case, a pool could become unimportable. This happened to one of the developers on his personal machine. He then debugged it, found the bitflip and wrote a custom tool to fix it."
https://github.com/openzfs/zfs/discussi ... nt-5279415