[gentoo-user] How to set up drive with many Linux distros?

[Grant Edwards]

For many years, I've used a hard drive on which I have 8-10 Linux
distros installed -- each in a separate (single) partition.

There is also a single swap partition (used by all of the different
Linux installations).

There is also a small partition devoted only to the "master" instance
of Grub that lives in the MBR and the space between the MBR and the
first partition (the drive uses a DOS disklabel). That master instance
of Grub has a menu which contains entries which "chainload" each of
the other partitions.

For many years, this worked great. All of the various distro
installers offered the option of installing the bootloader in the MBR
(e.g. /dev/sda) or in a partition (e.g /dev/sdaN). I would tell the
installer to install the bootloader in the root partition, and
everything "just worked".

However, the choice to install bootloaders in partitions instead of
the MBR has been removed from most (all?) of the common installers.
This forces me to jump through hoops when installing a new Linux
distro:

1. Back up the MBR and gap between the MBR and the first partition.

2. Let the installer install it's bootloader (seems it's always grub
these days) in the MBR.

3. Boot into the newly installed Linux.

4. Manually install grub in the root partition (e.g. /dev/sdaN) using
the --force option to tell grub to use blocklists to find it's
files.

5. Find those grub files and lock them so they can't be moved.

6. Restore the MBR/gap backup from step 1.

It seems like there should be a better way to do this. One might hope
that UEFI offers a solution to this problem. Google has found me
others asking the same question but no real answers.

Is there an easier way to do this?

--
Grant

[Wol]

Post by Grant Edwards
However, the choice to install bootloaders in partitions instead of
the MBR has been removed from most (all?) of the common installers.
This forces me to jump through hoops when installing a new Linux

File a bug!

If that's true, it basically borks any sort of dual boot, unusual disk
layout, whatever.

Last time I installed SUSE, it trashed my boot totally because it didn't
recognise my disk stack, failed to load necessary drivers, and worse
trashed my gentoo boot too ...

Cue one big rescue job to get the system up and working again. At least
it was only the boot that was trashed.

Cheers,
Wol

[Grant Edwards]

Post by Wol

Post by Grant Edwards
However, the choice to install bootloaders in partitions instead of
the MBR has been removed from most (all?) of the common installers.
This forces me to jump through hoops when installing a new Linux

File a bug!

LOL, good one! As if a normal person filing a bug with RedHat or
Ubuntu actually accomplishes anything. I'll tell them to make systemd
optional while I'm at it.

Post by Wol
If that's true, it basically borks any sort of dual boot, unusual disk
layout, whatever.

Yep, it does. The answer from <distro> is:

You should really just (shut up and) install <distro>'s (obviously
superior) bootloader in the MBR. It will auto-detect (some of) the
other already installed (obviously inferior) OSes, and will add
(some subset of) them to the boot menu that will (sometimes) allow
you to boot them (maybe -- if you kneel, bow your head and ask
nicely).

Post by Wol
Last time I installed SUSE, it trashed my boot totally because it
didn't recognise my disk stack, failed to load necessary drivers,
and worse trashed my gentoo boot too...
Cue one big rescue job to get the system up and working again. At
least it was only the boot that was trashed.

I've been reading up on UEFI, and it doesn't seem to be any
better. People complain about distro's stomping on each other's files
in the ESP partiton and multiple distro's using the same name in the
boot slots stored in NVM. And then the boot choice order changes
(though it may not be apparent to the naked eye) when one of the
distros decides to update/reinstall its boot stuff.

--
Grant

[Wol]

Post by Grant Edwards
I've been reading up on UEFI, and it doesn't seem to be any
better. People complain about distro's stomping on each other's files
in the ESP partiton and multiple distro's using the same name in the
boot slots stored in NVM. And then the boot choice order changes
(though it may not be apparent to the naked eye) when one of the
distros decides to update/reinstall its boot stuff.

At least if you use UEFI *as* your bootloader, then that won't happen.
That assumes you're using UEFI, though!

In which case, <distro>'s bootloader doesn't get a look-in.

As for "<distro>'s obviously superior bootloader", well <other distro>
is using the exact same boot-loader, and when IT installs, how is it
going to be able to boot <distro> if it can't call <distro>'s boot
loader because it's just trashed it by overwriting it?

To me, you seem to be describing the *default* installer setup, that's
been there for ever. Last I installed SUSE, iirc I had to specify
"advanced bootloader installation", most of who's options I didn't even
understand!, but it did do what I told it to (apart from not recognising
my weird disk stack!).

If you can find, and understand!, that advanced options, I think you'll
find you can do what you want.

Cheers,
Wol

[Grant Edwards]

Post by Wol

Post by Grant Edwards
I've been reading up on UEFI, and it doesn't seem to be any
better. People complain about distro's stomping on each other's files
in the ESP partiton and multiple distro's using the same name in the
boot slots stored in NVM. And then the boot choice order changes
(though it may not be apparent to the naked eye) when one of the
distros decides to update/reinstall its boot stuff.

At least if you use UEFI *as* your bootloader, then that won't
happen. That assumes you're using UEFI, though!

According to what I've read UEFI isn't a bootloader. It's a boot
manager which can load and run EFI bootloaders (of which there can be
multiple installed).

Post by Wol
In which case, <distro>'s bootloader doesn't get a look-in.

Yes, AFAICT, it does (sometimes?). When you install <distro> under
UEFI it installs EFI bootloader files (either kernels wrapped in EFI
bootloader executables or the grub EFI bootloader) in the EFI Systgem
Partition (ESP), and then adds one or more entries in the EFI NVM that
points to those files (or something like that). The Linux UEFI
systems I have all still use grub2 (which gets written to the ESP).

It's entirely possible for one distro to overwrite files in the ESP
that belong to other distros. I've read multiple complaints about
exactly that when trying to do multi-boot with UEFI. In practice it's
just like the fight over who owns the MBR and the DOS disklable gap.

One recipe I read about for doing what I wanted to do with UEFI
involved installing a Linux distro (didn't really matter which), then
installing rEFInd. After that, some manual renaming and deleting of
the files in the ESP was required. Then he started installing various
distros. After each distro installation, the author had to re-install
rEFInd, and after many of them he had to manually remove or rename
files in the ESP (or adjust the rEFInd config file).

And in the end, he ended up with multiple menu entries (for different
installations) that had identical names.

It was more complicated and difficult than my current scheme.

Post by Wol
As for "<distro>'s obviously superior bootloader", well <other distro>
is using the exact same boot-loader, and when IT installs, how is it
going to be able to boot <distro> if it can't call <distro>'s boot
loader because it's just trashed it by overwriting it?

In my experience, <distro>'s bootloader does not boot other
installations by calling other bootloaders. It does so by rummaging
through all of the other partitions looking for kernel images, intird
files, grub.cfg files, etc. It then adds menu entries to the config
file for <distro>'s bootloader which, when selected, directly load the
kernel image and initrd from those other partitions. Sometimes, it
works -- at least until those other installations get updated without
the knowlege of the distro that currently "owns" the MBR's bootloader
config. Then it stops working until you tell that bootloader to re-do
it's rummaging about routine.

Post by Wol
To me, you seem to be describing the *default* installer setup, that's
been there for ever. Last I installed SUSE, iirc I had to specify
"advanced bootloader installation", most of who's options I didn't even
understand!, but it did do what I told it to (apart from not recognising
my weird disk stack!).

So SuSe still allows you to install grub to a partition instead of
MBR? That's encouraging. RH and Ubuntu used to allow that, but AFAIK,
now they do not.

Post by Wol
If you can find, and understand!, that advanced options, I think
you'll find you can do what you want.

I'd welcome pointers to where those advanced options are in the RH and
Ubunutu installers -- I've searched everywhere I can think of. Various
things Google has found lead me to believe that they no longer support
installing grub in a partition.

I guess I'll stick with my current setup.

Or perhaps I'll switch from a DOS disklabel to a GPT disklabel.
Instead of backing up and restoring the MBR and the gap, I would
backup and restore the MBR and the BIOS boot partition. And I could
use UUIDs and partition labels.

[Wojciech Kuzyszyn]

Hello!

I guess most (all) of the distro's you are talking about use GRUB (or
at least they allow to do it). If that's true, I'm pretty sure you can
happily let them overwrite the GRUB in MBR as many times as they want,
since it's the same (or just probably minor version differences)
bootloader. Just make a copy of /boot/grub/grub.cfg and make sure it's
the same on every partition. Or, even better, if that's possible right
now, make a common /boot partition and after installing the new distro
just merge the (probably new) /boot/grub/grub.cfg with your old one.

I really think that *should* work!

Take care,
ks. Wojciech Kuzyszyn

[Grant Edwards]

Post by Wojciech Kuzyszyn
I guess most (all) of the distro's you are talking about use GRUB (or
at least they allow to do it).

Yes, I belive that they are all now using Grub2.

Post by Wojciech Kuzyszyn
If that's true, I'm pretty sure you can happily let them overwrite
the GRUB in MBR as many times as they want, since it's the same (or
just probably minor version differences) bootloader. Just make a
copy of /boot/grub/grub.cfg and make sure it's the same on every
partition.

That means I have to update all of those grub.cfg files manually every
time I install or update a distro. That's a lot of work.

Post by Wojciech Kuzyszyn
Or, even better, if that's possible right now, make a common /boot
partition and after installing the new distro just merge the
(probably new) /boot/grub/grub.cfg with your old one.

"just merge the new grub.cfg file with the old one" is the problem.

I tried that for a while: every time a distro got installed, I would
add it to the "main" grub config file. Every time a distro got a
kernel update, I'd modify the main grub config file. It was a lot more
work that my current scheme (and a lot more error prone).

Post by Wojciech Kuzyszyn
I really think that *should* work!

It would work, but maintaining the grub.cfg files is a lot of
work. The scheme I'm using now doens't require me to mess with any of
the grub.cfg files when distros get updated or installed.

--
Grant

[Michael]

Post by Grant Edwards

Post by Wol

Post by Grant Edwards
I've been reading up on UEFI, and it doesn't seem to be any
better. People complain about distro's stomping on each other's files
in the ESP partiton and multiple distro's using the same name in the
boot slots stored in NVM. And then the boot choice order changes
(though it may not be apparent to the naked eye) when one of the
distros decides to update/reinstall its boot stuff.

At least if you use UEFI *as* your bootloader, then that won't
happen. That assumes you're using UEFI, though!

According to what I've read UEFI isn't a bootloader. It's a boot
manager which can load and run EFI bootloaders (of which there can be
multiple installed).

The UEFI firmware of the MoBo contains its own bootloader and a boot manager
with its own boot menu, initialised and running from the MoBo's EEPROM. Unlike
conventional/legacy bootloaders stored in the first sector of a disk (MBR),
the UEFI firmware is stored in the MoBo's EEPROM, a larger equivalent to the
old CMOS.

However, this comes with the caveat the UEFI 'bootloader' can only load .efi
executables which have already been placed in the FAT32 formatted EFI System
Partition (ESP). Unlike GRUB's MBR Stage 1 bootloader code, the UEFI firmware
is large enough to contain its own fs driver to be able to read the ESP fs
content and identify and run all .efi applications.

Kernel images which have been built with the EFI stub and therefore masquerade
as .efi compatible files can be loaded directly by the UEFI firmware without
the need of an additional bootloader.

Post by Grant Edwards

Post by Wol
In which case, <distro>'s bootloader doesn't get a look-in.

Yes, AFAICT, it does (sometimes?). When you install <distro> under
UEFI it installs EFI bootloader files (either kernels wrapped in EFI
bootloader executables or the grub EFI bootloader) in the EFI Systgem
Partition (ESP), and then adds one or more entries in the EFI NVM that
points to those files (or something like that). The Linux UEFI
systems I have all still use grub2 (which gets written to the ESP).

Legacy GRUB on an MBR partitioned disk, writes its Stage 1 bootloader code in
the first sector and Stage 1.5 with the fs drivers in sector 34, before the
first partition.

GRUB2 on a UEFI system installs the file /efi/<distro>/grubx64.efi in the ESP,
an equivalent of the Stage 1 and Stage 1.5 of legacy GRUB. The Stage 2 /boot/
grub/ files can be installed either in the ESP, or on a different partition.

Post by Grant Edwards
It's entirely possible for one distro to overwrite files in the ESP
that belong to other distros. I've read multiple complaints about
exactly that when trying to do multi-boot with UEFI. In practice it's
just like the fight over who owns the MBR and the DOS disklable gap.

It doesn't really matter if the grubx64.efi executable is overwritten, as long
as the OS_PROBER is not disabled in /etc/default/grub. Re-running grub-
mkconfig will re-scan the ESP and any drive/partitions, pick up any OS kernel
images known by GRUB and add them to GRUB's boot menu.

The problem starts if/when kernel images are overwritten by successive Linux
OS distros. This is likely when derivatives of the same main distros e.g.
Ubuntu all create a directory called /EFI/ubuntu/ in the ESP and drop their
kernels & initrd images in there potentially overwriting other distro's files.

Post by Grant Edwards
One recipe I read about for doing what I wanted to do with UEFI
involved installing a Linux distro (didn't really matter which), then
installing rEFInd. After that, some manual renaming and deleting of
the files in the ESP was required. Then he started installing various
distros. After each distro installation, the author had to re-install
rEFInd, and after many of them he had to manually remove or rename
files in the ESP (or adjust the rEFInd config file).
And in the end, he ended up with multiple menu entries (for different
installations) that had identical names.

The concept of one bootloader/manager ruling them all is broadly the same
whether you use rEFInd or GRUB, as are the hoops you have to jump through to
accommodate distros' automated/hardcoded installation scripts.

When using a distro's installer menu on a legacy BIOS MoBo you can select a
partition (PBR) to install GRUB, but GRUB will complain and suggest you could
use blocklists but it is unreliable. Last time I received an error like this,
I installed grub in a PBR manually with the '--force' option, without using
the installer GUI. After that, whenever I updated GRUB it complained again
about blocklists, but it worked fine.

Post by Grant Edwards
It was more complicated and difficult than my current scheme.

Post by Wol
As for "<distro>'s obviously superior bootloader", well <other distro>
is using the exact same boot-loader, and when IT installs, how is it
going to be able to boot <distro> if it can't call <distro>'s boot
loader because it's just trashed it by overwriting it?

In my experience, <distro>'s bootloader does not boot other
installations by calling other bootloaders. It does so by rummaging
through all of the other partitions looking for kernel images, intird
files, grub.cfg files, etc. It then adds menu entries to the config
file for <distro>'s bootloader which, when selected, directly load the
kernel image and initrd from those other partitions. Sometimes, it
works -- at least until those other installations get updated without
the knowlege of the distro that currently "owns" the MBR's bootloader
config. Then it stops working until you tell that bootloader to re-do
it's rummaging about routine.

Post by Wol
To me, you seem to be describing the *default* installer setup, that's
been there for ever. Last I installed SUSE, iirc I had to specify
"advanced bootloader installation", most of who's options I didn't even
understand!, but it did do what I told it to (apart from not recognising
my weird disk stack!).

So SuSe still allows you to install grub to a partition instead of
MBR? That's encouraging. RH and Ubuntu used to allow that, but AFAIK,
now they do not.

Post by Wol
If you can find, and understand!, that advanced options, I think
you'll find you can do what you want.

I'd welcome pointers to where those advanced options are in the RH and
Ubunutu installers -- I've searched everywhere I can think of. Various
things Google has found lead me to believe that they no longer support
installing grub in a partition.

Try using '--force' to make GRUB install its image in some distro's boot/root
partition PBR instead of the disk MBR, but you'll probably have to perform
this outside the installer script. I've done this with VMs.

Post by Grant Edwards
I guess I'll stick with my current setup.
Or perhaps I'll switch from a DOS disklabel to a GPT disklabel.
Instead of backing up and restoring the MBR and the gap, I would
backup and restore the MBR and the BIOS boot partition. And I could
use UUIDs and partition labels.

These days I use disks with GPT even on MoBos with legacy BIOS. Instead of
backing up and restoring the MBR/BIOS Boot Partition you could just reinstall
grub and run grub-mkconfig, as long as the latter involves fewer key-presses.
;-)

It goes without saying a backup before messing up with GRUB will save the day
in case files are inadvertently overwritten.

[Grant Edwards]

Post by Michael
The problem starts if/when kernel images are overwritten by
successive Linux OS distros. This is likely when derivatives of the
same main distros e.g. Ubuntu all create a directory called
/EFI/ubuntu/ in the ESP and drop their kernels & initrd images in
there potentially overwriting other distro's files.

Yes, that's the problem that I've read about when trying to multi-boot
with UEFI. I usually have 3-4 different Ubuntu installations and 3-4
different RedHat installations. Ubuntu in particular causes a log of
complaints about overwriting ESP files belonging to other Ubuntus.

Post by Michael
When using a distro's installer menu on a legacy BIOS MoBo you can
select a partition (PBR) to install GRUB,

You used to be able to. I can no longer find the option to do that in
Ubuntu or RedHat. I've been told that Suse still allows it.

Post by Michael
but GRUB will complain and suggest you could use blocklists but it
is unreliable. Last time I received an error like this, I installed
grub in a PBR manually with the '--force' option, without using the
installer GUI. After that, whenever I updated GRUB it complained
again about blocklists, but it worked fine.

Using --force will work fine as long as the grub 1.5 files don't get
moved afterwards. That's why I lock them in place. Locking them will
cause future upgrades to Grub for that distro to fail, but that
doens't happen very often. When it does, you unlock them, updated
grub, re-install using --force, and lock them again.

Post by Michael

Post by Grant Edwards
I'd welcome pointers to where those advanced options are in the RH
and Ubunutu installers -- I've searched everywhere I can think
of. Various things Google has found lead me to believe that they no
longer support installing grub in a partition.

Try using '--force' to make GRUB install its image in some distro's
boot/root partition PBR instead of the disk MBR, but you'll probably
have to perform this outside the installer script. I've done this
with VMs.

Yes, in my OP describing what I'm doing now it explains that's what I
do. Then I lock the Grub files that are located using the blocklists
created by the --force option.

Post by Michael

Post by Grant Edwards
I guess I'll stick with my current setup.
Or perhaps I'll switch from a DOS disklabel to a GPT disklabel.
Instead of backing up and restoring the MBR and the gap, I would
backup and restore the MBR and the BIOS boot partition. And I could
use UUIDs and partition labels.

These days I use disks with GPT even on MoBos with legacy BIOS.

Same here -- except for this one machine. I think I'll switch it over
soon.

Post by Michael
Instead of backing up and restoring the MBR/BIOS Boot Partition you
could just reinstall grub and run grub-mkconfig, as long as the
latter involves fewer key-presses. ;-)

I don't use grub-mkconfig for the "main" grub. It has a fixed grub.cfg
file that does nothing but chainload the user-selected partition.

Currently, backing up MBR+gap only happens once when I install/setup
the main grub. Restoring BMR+gap is one command (which is actually in
a shell script) that's run after any new distro is installed.

MBR+Bios-boot-partition would work pretty much the same way.

--
Grant

[Wols Lists]

Post by Grant Edwards
In my experience, <distro>'s bootloader does not boot other
installations by calling other bootloaders. It does so by rummaging
through all of the other partitions looking for kernel images, intird
files, grub.cfg files, etc. It then adds menu entries to the config
file for <distro>'s bootloader which, when selected, directly load the
kernel image and initrd from those other partitions. Sometimes, it
works -- at least until those other installations get updated without
the knowlege of the distro that currently "owns" the MBR's bootloader
config. Then it stops working until you tell that bootloader to re-do
it's rummaging about routine.

IME distros that try that (SUSE, anyone!) generally get confused as to
which kernel belongs to which root partition.

Hence needing to boot with a live distro to edit the resulting mess and
get the system to actually come up without crashing ...

Cheers,
Wol

[Grant Edwards]

Post by Wols Lists

Post by Grant Edwards
In my experience, <distro>'s bootloader does not boot other
installations by calling other bootloaders. It does so by rummaging
through all of the other partitions looking for kernel images,
intird files, grub.cfg files, etc. It then adds menu entries to
the config file for <distro>'s bootloader which, when selected,
directly load the kernel image and initrd from those other
partitions. Sometimes, it works -- at least until those other
installations get updated without the knowlege of the distro that
currently "owns" the MBR's bootloader config. Then it stops working
until you tell that bootloader to re-do it's rummaging about
routine.

IME distros that try that (SUSE, anyone!) generally get confused as
to which kernel belongs to which root partition.
Hence needing to boot with a live distro to edit the resulting mess
and get the system to actually come up without crashing ...

IIRC, all of the big distros used to do that. It didn't work very
well, but at least it took a really long time.

However, I read recently that Ubuntu had disabled the os-prober by
default in 22.04. Disabling it was always one of the first things I
did after installing a new distro.

The simple solution is to give up on multi-booting a dozen different
distros on a single disk and buy a pocketful of USB 3 thumb drives.

--
Grant

[Mark Knecht]

Post by Grant Edwards
The simple solution is to give up on multi-booting a dozen different
distros on a single disk and buy a pocketful of USB 3 thumb drives.

Given performance does drop a bit and there can be issues with allocating
hardware, why not use Virtualbox which allows you to run both your base
distro and then as many of the others as your hardware can handle as VMs?

This is how I go about investigating a new distro before committing to it.

[Grant Edwards]

Post by Mark Knecht

Post by Grant Edwards
The simple solution is to give up on multi-booting a dozen different
distros on a single disk and buy a pocketful of USB 3 thumb drives.

Given performance does drop a bit and there can be issues with allocating
hardware, why not use Virtualbox which allows you to run both your base
distro and then as many of the others as your hardware can handle as VMs?

The machine is used for testing PCI and PCI-express boards and their
drivers. I don't really trust PCI passthrough (especially when it
comes to interrupt handling) enought to do such testing in a
VM. IFAICT, it's very rare for customers to use VMs. If customers do
start to use VMs, then we'll have to test with VMs also.

--
Grant

[Mark Knecht]

Post by Grant Edwards

On Fri, Feb 23, 2024 at 11:59 AM Grant Edwards <

Post by Grant Edwards
The simple solution is to give up on multi-booting a dozen different
distros on a single disk and buy a pocketful of USB 3 thumb drives.

Given performance does drop a bit and there can be issues with allocating
hardware, why not use Virtualbox which allows you to run both your base
distro and then as many of the others as your hardware can handle as VMs?

The machine is used for testing PCI and PCI-express boards and their
drivers. I don't really trust PCI passthrough (especially when it
comes to interrupt handling) enought to do such testing in a
VM. IFAICT, it's very rare for customers to use VMs. If customers do
start to use VMs, then we'll have to test with VMs also.

In that specific use case I completely agree.

The only other idea I had was to install <new distro> to a different
disk and then use something like Clonezilla to move it to the partition
you want it in on your system.

While I suspect you were being sarcastic I do not think any solution
that involves a 'pocketful of USB 3 thumb drives' will be satisfying.

Wishing you luck,
Mark

[Grant Edwards]

Post by Mark Knecht
The only other idea I had was to install <new distro> to a different
disk and then use something like Clonezilla to move it to the partition
you want it in on your system.
While I suspect you were being sarcastic I do not think any solution
that involves a 'pocketful of USB 3 thumb drives' will be satisfying.

Actually (assuming the thumb drives are relaible) it probably would
work fine. That's more-or-less the typical solution that my colleagues
use -- though it's a drawer full of hard drives in caddies instead of
USB thumb drives. Back in the day when we supported a couple versions
of SCO, various BSDs, Novell Netware and Solaris, it required a fait
bit of drawer space.

For now, I guess I'll stick with the scheme I'm using but switch from
DOS disklabel and gap to GPT disklabel and BIOS boot partition. It
seems ugly, but it's managable with the help of a few shell scripts
that are stored in the parition that belongs to the master copy of
grub.

[g***@krasauskas.dev]

Since this is a fairly custom task, I would approach it with a custom
solution.

- GPT
- systemd-boot
- One /boot partition
- One BTRFS-on-LUKS partition (formatted using the distro with the
oldest kernel)
- {@root,@home,@var,@srv,@opt}-{distro1,distro2,distro3} subvolumes
- Potentially {@distro1,distro2,distro3}-{downloads,documents,pictures}
subvolumes, if there's a usecase for that
- Bootstrap all the distros manually. Arch and Gentoo do that by
default, debian land has debootstrap, not too familiar with rhel
universe, but at the end of the day everything's a file :)

You could also write a script that keeps all the distros up to date
from within whichever one you're currently booted by mounting
subvolumes to /mnt or wherever, chrooting in and running the update.

You could also replace systemd-boot with efibootmgr to use UEFI boot
directly, but I would advise doing that after the everything is working
correctly as it's much easier to experiment using a good old bootloader
edit function.

BTRFS requires _some_ maintenance, but imho it's reasonable - run a
deduplication and defrag job on a scheduling tool of your choice and
you're good to go!

$0.02

[eric]

Post by g***@krasauskas.dev
You could also write a script that keeps all the distros up to date
from within whichever one you're currently booted by mounting
subvolumes to /mnt or wherever, chrooting in and running the update.

To avoid grub not being able to point to a newly updated kernel on one
of the OS's installed, I use a "custom.cfg" file in all my /boot/grub/
directories for each OS where the "linix" and "initrd" point to the
symbolic links of the kernel and init files which point to the newly
updated files on most major distributions like ubuntu, arch, suse, and
debian. The name of the symbolic links stay the same over upgrades. It
works great when using UUID to identify the partition that has root and
I can always boot into any of the OS's installed no matter which one
hijacked the MBR.


https://forums.linuxmint.com/viewtopic.php?t=315584

[Grant Edwards]

Post by eric

Post by g***@krasauskas.dev
You could also write a script that keeps all the distros up to date
from within whichever one you're currently booted by mounting
subvolumes to /mnt or wherever, chrooting in and running the update.

To avoid grub not being able to point to a newly updated kernel on one
of the OS's installed, I use a "custom.cfg" file in all my /boot/grub/
directories for each OS where the "linix" and "initrd" point to the
symbolic links of the kernel and init files which point to the newly
updated files on most major distributions like ubuntu, arch, suse, and
debian. The name of the symbolic links stay the same over upgrades. It
works great when using UUID to identify the partition that has root and
I can always boot into any of the OS's installed no matter which one
hijacked the MBR.

Except I generally have multiple kernels installed for each of the
distros, and need to be able to choose which kernel to boot. There are
also various other boot options (e.g. "safe mode") offered by some
distros that I occasionally need to use.

Post by eric
https://forums.linuxmint.com/viewtopic.php?t=315584

Interesting article, thanks.

After reading up more on UEFI, it looks like that would be even more
work and more mess. So, there seem to be two options:

1) Stick to the dual-stage chainloading scheme I'm using now (though
I'll probably switch from DOS to GTP disklabel). That way after
selecting which parition (distro) to boot, I get all the boot
options normally offered by that distro's install. Installing a
distro involves letting it install to MBR and BIOS-boot,
installing grub manually to the root partition, then restoring MBR
and BIOS-boot.

2) Use a single master grub to boot any distro.

I think I'd need to write my own OS-prober. All of the distros I
care about seem to now be using grub2 now. Instead of looking for
kernels and initrd images and adding them to the master grub.cfg,
I would probe for grub.cfg files, and for each one found
incorporate the entire set of choices in that .cfg file as a
submenu in the main grub.cfg menu.

I think that in order to generate the distro's grub.cfg files, I
still have to allow the distros to install grub to the MBR and
BIOS-boot (or to a second disk that I don't care about), then
restoring MRB/BIOS-boot.

[eric]

Post by Grant Edwards

Post by eric

Post by g***@krasauskas.dev
You could also write a script that keeps all the distros up to date
from within whichever one you're currently booted by mounting
subvolumes to /mnt or wherever, chrooting in and running the update.

To avoid grub not being able to point to a newly updated kernel on one
of the OS's installed, I use a "custom.cfg" file in all my /boot/grub/
directories for each OS where the "linix" and "initrd" point to the
symbolic links of the kernel and init files which point to the newly
updated files on most major distributions like ubuntu, arch, suse, and
debian. The name of the symbolic links stay the same over upgrades. It
works great when using UUID to identify the partition that has root and
I can always boot into any of the OS's installed no matter which one
hijacked the MBR.

Except I generally have multiple kernels installed for each of the
distros, and need to be able to choose which kernel to boot. There are
also various other boot options (e.g. "safe mode") offered by some
distros that I occasionally need to use.

I agree, using the custom.cfg file would not work if needing to boot
different kernels of the same OS and those kernels were being updated.

Regards,
Eric

[Grant Edwards]

Post by eric
I agree, using the custom.cfg file would not work if needing to boot
different kernels of the same OS and those kernels were being updated.

The simple answer is to quit wasting time trying to multi-boot like
that and just buy a dozen USB flash drives.

--
Grant

[Wol]

Post by Grant Edwards

Post by eric
I agree, using the custom.cfg file would not work if needing to boot
different kernels of the same OS and those kernels were being updated.

The simple answer is to quit wasting time trying to multi-boot like
that and just buy a dozen USB flash drives.

And then, if USB isn't the default boot media, he might as well sort out
UEFI boot, and multi-boot that way.

Cheers,
Wol

[Grant Edwards]

Post by Wol

Post by Grant Edwards
The simple answer is to quit wasting time trying to multi-boot like
that and just buy a dozen USB flash drives.

And then, if USB isn't the default boot media, he might as well sort out
UEFI boot, and multi-boot that way.

Except that every time I've found a write-up about multibooting a lot
of Linux distros with UEFI, it turns out that it doesn't actually work
very well and is more work to maintain than what I'm doing now.

--
Grant

[Mark Knecht]

Post by Grant Edwards

Post by Wol

Post by Grant Edwards
The simple answer is to quit wasting time trying to multi-boot like
that and just buy a dozen USB flash drives.

And then, if USB isn't the default boot media, he might as well sort out
UEFI boot, and multi-boot that way.

Except that every time I've found a write-up about multibooting a lot
of Linux distros with UEFI, it turns out that it doesn't actually work
very well and is more work to maintain than what I'm doing now.
--
Grant

I have no experience beyond three operating systems on a single machine
but if you grabbed just 2 or 3 USB flash drives then I would think you
could test it pretty easily. I believe the UEFI boot procedures are
storing a unique ID for the disk or the partition you are requesting. If you
have a unique ID that's different for each flash drive it would (hopefully)
find the one you're looking for which should be relatively simple.

I would suggest you use the boot ordering feature and make the
system hard drive last in the list. If no USB devices are plugged in
it would default to your system drive. If a flash drive is plugged in
it should find its ID and boot that first.

I do not know if, for instance, you had 20 different drives listed in
your BIOS whether it would be a lot slower to boot but you could
test that yourself.

Good luck,
Mark

[Peter Humphrey]

Post by Mark Knecht
I have no experience beyond three operating systems on a single machine
but if you grabbed just 2 or 3 USB flash drives then I would think you
could test it pretty easily. I believe the UEFI boot procedures are
storing a unique ID for the disk or the partition you are requesting. If you
have a unique ID that's different for each flash drive it would (hopefully)
find the one you're looking for which should be relatively simple.
I would suggest you use the boot ordering feature and make the
system hard drive last in the list. If no USB devices are plugged in
it would default to your system drive. If a flash drive is plugged in
it should find its ID and boot that first.
I do not know if, for instance, you had 20 different drives listed in
your BIOS whether it would be a lot slower to boot but you could
test that yourself.

My experience is that the BIOS will discard any boot devices that are not
present at the time of booting. I don't know whether that's typical or just
what I've found.

[Grant Edwards]

Post by Grant Edwards
For many years, I've used a hard drive on which I have 8-10 Linux
distros installed -- each in a separate (single) partition.
[...]
Is there an easier way to do this?

After some additional studying of UEFI and boot managers like rEFInd,
I decided that my current approach was still the easiest method. I did
switch from DOS to GPT disklabel (I bricked my old drive tring to
update the firmware, so I had to start over anyway).

In case anybody is interested in the gory details, I documented my
scheme and the helper shell scripts at

https://github.com/GrantEdwards/Linux-Multiboot

--
Grant

[Michael]

Post by Grant Edwards

Post by Grant Edwards
For many years, I've used a hard drive on which I have 8-10 Linux
distros installed -- each in a separate (single) partition.
[...]
Is there an easier way to do this?

After some additional studying of UEFI and boot managers like rEFInd,
I decided that my current approach was still the easiest method. I did
switch from DOS to GPT disklabel (I bricked my old drive tring to
update the firmware, so I had to start over anyway).
In case anybody is interested in the gory details, I documented my
scheme and the helper shell scripts at
https://github.com/GrantEdwards/Linux-Multiboot

Thank you Grant for taking time to share your clear and well structured write
up, what with helpful scripts and all! Its easy to follow and should help
others, hopefully before they discover belatedly many distro installers can
mess up a multiboot scheme, if they don't step in to keep things in check.

Perhaps I'm picking up on semantics, but shouldn't this sentence:

"... The gap between the DOS disklabel and the first partition"

read:

"The gap between the MBR and the first partition"?

I'm saying this because the MBR in sector 0 contains the bootstrap code (446
bytes), the partition table (a.k.a. DOS disklabel 4x16 bytes) and the boot
sector signature (2 bytes). On MBR partitioned disks the core.img is stored
after the MBR, in sector 1 onward.

Your next paragraph pointed out something which I hadn't considered at any
length. Namely, the installation of GRUB's boot.img in a MBR or VBR also
hardcodes in a block list format the location of the first sector where the
core.img is stored and more importantly, the physical position of this sector
can be altered both by COW fs (and by the wear levelling firmware of flash
storage devices).

I had assumed both the COW fs and/or the flash controller will in both cases
translate any physical data position to the logical layer and presented this
to inquiring software. Have you actually tried using btrfs as a distro's root
fs to see if the VBR installed GRUB boot.img will ever lose access to the
core.img?

[Grant Edwards]

Post by Michael
"... The gap between the DOS disklabel and the first partition"
"The gap between the MBR and the first partition"?

Yes, thanks -- MBR is more accurate, I've changed that sentence.

Post by Michael
Your next paragraph pointed out something which I hadn't considered at any
length. Namely, the installation of GRUB's boot.img in a MBR or VBR also
hardcodes in a block list format the location of the first sector where the
core.img is stored and more importantly, the physical position of this sector
can be altered both by COW fs (and by the wear levelling firmware of flash
storage devices).
I had assumed both the COW fs and/or the flash controller will in
both cases translate any physical data position to the logical layer
and presented this to inquiring software. Have you actually tried
using btrfs as a distro's root fs to see if the VBR installed GRUB
boot.img will ever lose access to the core.img?

No, I haven't. I agree that the flash controller can't change the
logical address of a filesystem data block without the knowledge of the
filesystem, so I don't think controller layer wear-leveling would be
a problem. But, the filesystem layer is allowed to move data blocks
around, so flash-aware filesystems that attempt to do wear-leveling
or defragmentation could move data blocks.

Some of the descriptions I've read of "fancier" filesystem internals
have also implied implied that does happen under certain conditions,
but I may have misunderstood or the descriptions may have been wrong.

My use of these multi-boot installs have no need for anything beyond
exnN, so I've never tried using block lists with anything other than
extN filesystems. Since I am confident extN filesystems won't cause
problems, I've always stuck with that.

--
Grant