[gentoo-user] Simple installation on BTRFS

[Michael]

Although I've been using btrfs for the best part of 10 years I have not really
done justice to it, because I have neither explored nor used enough most of
its features. I am now thinking of installing Gentoo on btrfs again, but this
time I want to optimise the structure of btrfs subvolumes, to simplify
snapshots and backups.

I see Ubuntu and derivates install the OS root fs under btrfs subvolume "@"
and /home under subvolume "@home". This makes storing snapshots of the two
subvolumes under the btrfs top-volume, which remains unmounted, cleaner and
reduces the chance of mixing up the fs you may end up in and operate on (live,
or snapshot).

I have 3 partitions for /boot(ESP), / and /home, but have not yet created
additional partitions for general data storage and backups.

What's your recommended approach and subvolume structure for the deployment of
btrfs on Gentoo for a personal PC, if the primary objective is simplicity in
maintenance, combined with ease of fs recovery?

Any gotchas I should be mindful of?

Your favoured snapshot/backup strategy?

NOTES:

The Gentoo wiki pages[1],[2] cover btrfs and RAID configurations, but I found
some of it confusing; e.g.:

The impact of autodefrag on VM performance is noted, but then the example
given proceeds to mount a subvolume for VM storage with 'autodefrag'. :-/

Encryption is mentioned for VMs "... if the VM uses drive encryption, the
whole compression strategy gets blown out of the water" but doesn't mention
what type of encryption, or why/how this presents a problem.

Given btrfs does not offer fs level encryption, what could/would work to
encrypt a subvolume, *without* requiring an initrd, or the introduction of
encryption becoming orthogonal with snapshots and backups? I am not clear on
the best strategy and components to achieve this. I'm also concerned of
introducing an additional complexity layer in trying to recover btrfs when/if
fs corruption creeps in.

PS. Please keep answers specific to btrfs, as comparisons with zfs are well
covered in the interwebs.

[1] https://wiki.gentoo.org/wiki/Btrfs
[2] https://wiki.gentoo.org/wiki/Btrfs/Native_System_Root_Guide

[Neil Bothwick]

Post by Michael
Although I've been using btrfs for the best part of 10 years I have not
really done justice to it, because I have neither explored nor used
enough most of its features. I am now thinking of installing Gentoo on
btrfs again, but this time I want to optimise the structure of btrfs
subvolumes, to simplify snapshots and backups.
I see Ubuntu and derivates install the OS root fs under btrfs subvolume
the two subvolumes under the btrfs top-volume, which remains unmounted,
cleaner and reduces the chance of mixing up the fs you may end up in
and operate on (live, or snapshot).
I have 3 partitions for /boot(ESP), / and /home, but have not yet
created additional partitions for general data storage and backups.
What's your recommended approach and subvolume structure for the
deployment of btrfs on Gentoo for a personal PC, if the primary
objective is simplicity in maintenance, combined with ease of fs
recovery?

I too put everything on subvolumes, and set the one containing / to be
the default when mounted without a subvolid.

Post by Michael
Any gotchas I should be mindful of?
Your favoured snapshot/backup strategy?

I have a script, I can share it with you if you don't criticise my
coding, that creates and destroys snapshots from cron. Based in principle
on zfs-snapshot but written from scratch.

Post by Michael
The impact of autodefrag on VM performance is noted, but then the
example given proceeds to mount a subvolume for VM storage with
'autodefrag'. :-/

I disable COW on the subvolume containing my VM disk volumes.

Post by Michael
Encryption is mentioned for VMs "... if the VM uses drive encryption,
the whole compression strategy gets blown out of the water" but doesn't
mention what type of encryption, or why/how this presents a problem.
Given btrfs does not offer fs level encryption, what could/would work
to encrypt a subvolume, *without* requiring an initrd, or the
introduction of encryption becoming orthogonal with snapshots and
backups? I am not clear on the best strategy and components to achieve
this. I'm also concerned of introducing an additional complexity layer
in trying to recover btrfs when/if fs corruption creeps in.

The lack of encryption is a problem. You have to encrypt the block
device(s) containing btrfs, which means you will need an initrd. It also
means each component of a RAID is encrypted separately. so I only use
encryption on laptops. The alternative is to use ecryptfs for individual
subvolumes or directories.

[Michael]

Post by Neil Bothwick

Post by Michael
Although I've been using btrfs for the best part of 10 years I have not
really done justice to it, because I have neither explored nor used
enough most of its features. I am now thinking of installing Gentoo on
btrfs again, but this time I want to optimise the structure of btrfs
subvolumes, to simplify snapshots and backups.
I see Ubuntu and derivates install the OS root fs under btrfs subvolume
the two subvolumes under the btrfs top-volume, which remains unmounted,
cleaner and reduces the chance of mixing up the fs you may end up in
and operate on (live, or snapshot).
I have 3 partitions for /boot(ESP), / and /home, but have not yet
created additional partitions for general data storage and backups.
What's your recommended approach and subvolume structure for the
deployment of btrfs on Gentoo for a personal PC, if the primary
objective is simplicity in maintenance, combined with ease of fs
recovery?

I too put everything on subvolumes, and set the one containing / to be
the default when mounted without a subvolid.

When you say "everything", do you include temporary and virtual filesystems
too (e.g. /sys, /proc/ /tmp, /run), or do you place these in hierarchically
lower subvolumes so they are not backed up?

Also, how do you treat /var/db and /var/cache/distfiles?

How much space do you allocate for snapshots and at what point you start
moving/deleting older snapshots?

Post by Neil Bothwick

Post by Michael
Any gotchas I should be mindful of?
Your favoured snapshot/backup strategy?

I have a script, I can share it with you if you don't criticise my
coding, that creates and destroys snapshots from cron. Based in principle
on zfs-snapshot but written from scratch.

Me, criticise anyone's coding?! I couldn't code my way out of a paper bag.
:-( Please send over what you have and I'll try to adjust it for my use, or
at least I can see what you've written and learn from it.

Post by Neil Bothwick

Post by Michael
The impact of autodefrag on VM performance is noted, but then the
example given proceeds to mount a subvolume for VM storage with
'autodefrag'. :-/

I disable COW on the subvolume containing my VM disk volumes.

Yes, I've run 'chattr +C' on directories where I store VMs.

Post by Neil Bothwick

Post by Michael
Encryption is mentioned for VMs "... if the VM uses drive encryption,
the whole compression strategy gets blown out of the water" but doesn't
mention what type of encryption, or why/how this presents a problem.
Given btrfs does not offer fs level encryption, what could/would work
to encrypt a subvolume, *without* requiring an initrd, or the
introduction of encryption becoming orthogonal with snapshots and
backups? I am not clear on the best strategy and components to achieve
this. I'm also concerned of introducing an additional complexity layer
in trying to recover btrfs when/if fs corruption creeps in.

The lack of encryption is a problem. You have to encrypt the block
device(s) containing btrfs, which means you will need an initrd. It also
means each component of a RAID is encrypted separately. so I only use
encryption on laptops. The alternative is to use ecryptfs for individual
subvolumes or directories.

I have one SSD and a larger spinning disk. I have a separate partition on the
SSD for /home, so I could put dm-crypt on this partition alone and afford some
basic security for personal data against opportunistic theft. No RAID on this
box, unless you suggest to create a RAID 1 with two partitions, in case the
SSD cells go wrong on one of them?

Without RAID things should be simpler with block device level encryption for /
home. But, ... will this work without an initrd? The unencrypted rootfs will
be mounted before /home.

I am also not clear on steps I would need to follow in recovery operation
scenarios and what I must have available to achieve this. It is not as simple
as booting with any ol' liveUSB to try to access an unecrypted drive/
partition. I'll need dm-crypt and cryptsetup, or ecryptfs-utils and some
familiarity with these tools, if I'm not reading off my exceptionally well
structured notes I had the premonition to put together BEFORE the drive went
south. ;-)

[Neil Bothwick]

Post by Michael

Post by Neil Bothwick
I too put everything on subvolumes, and set the one containing / to be
the default when mounted without a subvolid.

When you say "everything", do you include temporary and virtual
filesystems too (e.g. /sys, /proc/ /tmp, /run), or do you place these
in hierarchically lower subvolumes so they are not backed up?

Everything but virtual filesystems, they are still virtual.

Post by Michael
Also, how do you treat /var/db and /var/cache/distfiles?

/var/db is just a directory on /var. I have $DISTDIR on an NFS mount, so
I can share it with all clients.

Post by Michael
How much space do you allocate for snapshots and at what point you
start moving/deleting older snapshots?

You don't allocate space for them, at least I don't.

I take snapshots every 15 minutes, keeping 5. Then hourly snapshots,
keeping 25, daily snapshots keeping 8, weekly keeping 6 and monthly
snapshots that I clean manually as space requires.

Post by Michael
I have one SSD and a larger spinning disk. I have a separate partition
on the SSD for /home, so I could put dm-crypt on this partition alone
and afford some basic security for personal data against opportunistic
theft. No RAID on this box, unless you suggest to create a RAID 1 with
two partitions, in case the SSD cells go wrong on one of them?
Without RAID things should be simpler with block device level
encryption for / home. But, ... will this work without an initrd? The
unencrypted rootfs will be mounted before /home.

You should only need an initrd if / is encrypted. I encrypt everything
except /boot and dracut handles decrypting via an initrd easily.

Post by Michael
I am also not clear on steps I would need to follow in recovery
operation scenarios and what I must have available to achieve this. It
is not as simple as booting with any ol' liveUSB to try to access an
unecrypted drive/ partition. I'll need dm-crypt and cryptsetup, or
ecryptfs-utils and some familiarity with these tools, if I'm not
reading off my exceptionally well structured notes I had the
premonition to put together BEFORE the drive went south. ;-)

systemrescue has all those tools, plus a web browser so you can work out
how to use them :)

[Michael]

Post by Neil Bothwick
I take snapshots every 15 minutes, keeping 5. Then hourly snapshots,
keeping 25, daily snapshots keeping 8, weekly keeping 6 and monthly
snapshots that I clean manually as space requires.

I doubt I will need anything so frequent, these days my data does not change
often enough. Daily snapshots should do the trick and I could keep more of
them.

[Neil Bothwick]

Post by Michael

Post by Neil Bothwick
I take snapshots every 15 minutes, keeping 5. Then hourly snapshots,
keeping 25, daily snapshots keeping 8, weekly keeping 6 and monthly
snapshots that I clean manually as space requires.

I doubt I will need anything so frequent, these days my data does not
change often enough. Daily snapshots should do the trick and I could
keep more of them.

Snapshots don't take up any space if the data does not change, so
frequent snapshots don't hurt. The biggest impact on snapshot sizes on a
Gentoo system is kernel source package updates.

[Michael]

-----Original Message-----
Sent: Saturday, July 29, 2023 5:43 AM
Subject: Re: [gentoo-user] Simple installation on BTRFS

Post by Michael
I doubt I will need anything so frequent, these days my data does not
change often enough. Daily snapshots should do the trick and I could
keep more of them.

Snapshots don't take up any space if the data does not change, so frequent
snapshots don't hurt. The biggest impact on snapshot sizes on a Gentoo
system is kernel source package updates.

For informational purposes, I once fat-fingered my cron job that takes
hourly snapshots of my workspace during work hours and was taking minutely
snapshots instead.
I didn't notice any adverse effects until I had well over 60,000 snapshots.
The main adverse effect at that point was that doing anything with snapshots
started to get... sluggish.
The second adverse effect was that listing the content of the snapshots
directory took several minutes, and I had to write a script to delete them
because it was too big for my kernel's process command line length limit.
(I also found out how to raise that limit if anyone wants to know, but
decided it wasn't worth the effort.)
It all cleaned up fine in a few hours though.
LMP

You'd probably choose NILFS2 if you wanted that many snapshots!

I haven't setup snapshots yet. I'm waiting for Neil to send me something,
while I am occupied endlessly configuring the kernel from scratch ;-).

[Igor Semyonov]

I simply have @ and @home subvolumes.
I use btrbk for automatic snapshotting and backups to an external drive and
via ssh to another machine.

For full disk encryption, I am not sure and would love some input on it
myself.
I am planning to install gentoo with btrfs over luks on another machine.
My plan is to use luks to encrypt a drive or partition, then put btrfs over
it and do things as before with btrfs.

One thing I am wondering about is: what if I want my btrfs to span multiple
drives?
Would I luks encrypt each drive, then open each one before mounting the
btrfs pool that spans all the luks containers?
Would this be possible to do with grub and the initramfs?
If not, my fallback idea is to make a relatively small root partition on 1
drive with the normal btrfs over luks approach. Then have luks containers
for the rest of the main drive and all other drives all added to a btrfs
pool that would be used for /home and any other partitions not necessary
for booting I want to add in there.

Post by Michael
Although I've been using btrfs for the best part of 10 years I have not really
done justice to it, because I have neither explored nor used enough most of
its features. I am now thinking of installing Gentoo on btrfs again, but this
time I want to optimise the structure of btrfs subvolumes, to simplify
snapshots and backups.
subvolumes under the btrfs top-volume, which remains unmounted, cleaner and
reduces the chance of mixing up the fs you may end up in and operate on (live,
or snapshot).
I have 3 partitions for /boot(ESP), / and /home, but have not yet created
additional partitions for general data storage and backups.
What's your recommended approach and subvolume structure for the deployment of
btrfs on Gentoo for a personal PC, if the primary objective is simplicity in
maintenance, combined with ease of fs recovery?
Any gotchas I should be mindful of?
Your favoured snapshot/backup strategy?
The Gentoo wiki pages[1],[2] cover btrfs and RAID configurations, but I found
The impact of autodefrag on VM performance is noted, but then the example
given proceeds to mount a subvolume for VM storage with 'autodefrag'. :-/
Encryption is mentioned for VMs "... if the VM uses drive encryption, the
whole compression strategy gets blown out of the water" but doesn't mention
what type of encryption, or why/how this presents a problem.
Given btrfs does not offer fs level encryption, what could/would work to
encrypt a subvolume, *without* requiring an initrd, or the introduction of
encryption becoming orthogonal with snapshots and backups? I am not clear on
the best strategy and components to achieve this. I'm also concerned of
introducing an additional complexity layer in trying to recover btrfs when/if
fs corruption creeps in.
PS. Please keep answers specific to btrfs, as comparisons with zfs are well
covered in the interwebs.
[1] https://wiki.gentoo.org/wiki/Btrfs
[2] https://wiki.gentoo.org/wiki/Btrfs/Native_System_Root_Guide

[Neil Bothwick]

Post by Igor Semyonov
I am planning to install gentoo with btrfs over luks on another machine.
My plan is to use luks to encrypt a drive or partition, then put btrfs
over it and do things as before with btrfs.
One thing I am wondering about is: what if I want my btrfs to span
multiple drives?
Would I luks encrypt each drive, then open each one before mounting the
btrfs pool that spans all the luks containers?

Yes, but see my reply to Michael regarding the overhead.

Post by Igor Semyonov
Would this be possible to do with grub and the initramfs?

Yes.

[Wols Lists]

Post by Michael
Although I've been using btrfs for the best part of 10 years I have not really
done justice to it, because I have neither explored nor used enough most of
its features. I am now thinking of installing Gentoo on btrfs again, but this
time I want to optimise the structure of btrfs subvolumes, to simplify
snapshots and backups.

Okay, I've chosen to run ext over lvm over raid, so my only experience
of btrfs is SUSE using it as default, but ...

Post by Michael
subvolumes under the btrfs top-volume, which remains unmounted, cleaner and
reduces the chance of mixing up the fs you may end up in and operate on (live,
or snapshot).
I have 3 partitions for /boot(ESP), / and /home, but have not yet created
additional partitions for general data storage and backups.
What's your recommended approach and subvolume structure for the deployment of
btrfs on Gentoo for a personal PC, if the primary objective is simplicity in
maintenance, combined with ease of fs recovery?

I did investigate btrfs ...

Post by Michael
Any gotchas I should be mindful of?

If you can run two disks and raid, that's always a good idea. SMART is
supposed to catch disk problems, but they still do die without warning.

btrfs raid is (still) full of gotchas, as far as I know.

Don't use anything higher than raid-1. Parity raid isn't reliable last I
knew ...

Make sure you know what you're getting. FS people seem to concentrate on
protecting the file system, not the data. I believe btrfs will raid-1
the metadata without asking if it can, you need to actively ask for the
data to be raided, and that's caught people out not realising btrfs
treats data and metadata differently. (cf the ext3/4 debacle)

Post by Michael
Your favoured snapshot/backup strategy?

Manual ... probably shouldn't be. I snapshot / every friday before I do
an emerge on Saturday. /home I ought to snapshot more than I do.

WATCH YOUR FREE DISK. I think it's all sorted now, but whatever you're
using it was always a good idea not to go over 90% full. For a very long
time, a combination of snapshots and a full disk would wedgie the
system, such that the only way to free up space was to reformat the
entire disk! As I say, I think it's now fixed so you can delete
snapshots, but >90% ain't a good idea anyway

Cheers,
Wol

[Neil Bothwick]

Post by Wols Lists
WATCH YOUR FREE DISK. I think it's all sorted now, but whatever you're
using it was always a good idea not to go over 90% full. For a very
long time, a combination of snapshots and a full disk would wedgie the
system, such that the only way to free up space was to reformat the
entire disk! As I say, I think it's now fixed so you can delete
snapshots, but >90% ain't a good idea anyway

That's been fixed for a long time, but you can still run out of space
when old metadata clogs things up. The fix is simple if a little time
consuming and covered in detail in the wiki. As you say though, not
filling up a filesystem is generally the best idea.

[Michael]

Post by Wols Lists

Post by Michael
Any gotchas I should be mindful of?

If you can run two disks and raid, that's always a good idea. SMART is
supposed to catch disk problems, but they still do die without warning.

Yes, esp. SSDs.

Post by Wols Lists
btrfs raid is (still) full of gotchas, as far as I know.
Don't use anything higher than raid-1. Parity raid isn't reliable last I
knew ...

I've read some horror stories, but I don't know if the problem was one of bad
implementation/recovery, or entirely the fault of the fs. This guy did some
comparisons on RAID 5 which worked OK - at least during his experiments:

https://unixsheikh.com/articles/battle-testing-zfs-btrfs-and-mdadm-dm.html#btrfs-raid-5

The btrfs docs advice caution:

https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid56-status-and-recommended-practices

Post by Wols Lists
Make sure you know what you're getting. FS people seem to concentrate on
protecting the file system, not the data. I believe btrfs will raid-1
the metadata without asking if it can, you need to actively ask for the
data to be raided, and that's caught people out not realising btrfs
treats data and metadata differently. (cf the ext3/4 debacle)

Post by Michael
Your favoured snapshot/backup strategy?

Manual ... probably shouldn't be. I snapshot / every friday before I do
an emerge on Saturday. /home I ought to snapshot more than I do.

I'm also rather reactive and ad hoc with backups. Hence I'm now looking to
implement something more systematic.

Post by Wols Lists
WATCH YOUR FREE DISK. I think it's all sorted now, but whatever you're
using it was always a good idea not to go over 90% full. For a very long
time, a combination of snapshots and a full disk would wedgie the
system, such that the only way to free up space was to reformat the
entire disk! As I say, I think it's now fixed so you can delete
snapshots, but >90% ain't a good idea anyway
Cheers,
Wol

I've had a /var/ partition on btrfs fill up, because I had forgotten to set up
logrotate! No subvolumes or snapshots in there, so deleting some old logs
allowed a full recovery with no loss of data. The problem with snapshots is
they creep gradually on you as data starts to differ over time and suddenly
you could find yourself out of space and potentially out of luck.