[gentoo-user] How to move ext4 partition

[Grant Edwards]

I've got a Gentoo install using a GPT partition table and Legacy boot
using Grub2. There is a single /root parition and a single swap
partition on the drive.

I did not create a bios-boot partition at the start of the disk, so I
had to force grub2 to install using block-lists. I'd like to fix that
now. This requires that I move the ext4 root partition towards the end
of the drive to create 2MB of free space at the start of the drive for
a new bios-boot partition.

I see that Gnu parted no longer has a move command.

However, GParted apparently does. Can GParted move an ext4 filesystem
to a destination location that overlaps its starting location?

For example, I have a 500GB partition containing an ext4 filesystem
starting at sector 2048 (1MiB). I want to move that filesystem so that
it starts at sector 3*2048 (3MiB).

Can that be done in-place?

Or should I just back up the filesystem to a second drive and start
from scratch?

[Neil Bothwick]

Post by Grant Edwards
However, GParted apparently does. Can GParted move an ext4 filesystem
to a destination location that overlaps its starting location?
For example, I have a 500GB partition containing an ext4 filesystem
starting at sector 2048 (1MiB). I want to move that filesystem so that
it starts at sector 3*2048 (3MiB).
Can that be done in-place?
Or should I just back up the filesystem to a second drive and start
from scratch?

Given that you'd want to backup before such an operation anyway, you may
as well then restore from that backup. I'm sure it will be a lot quicker
than GParted's moving all the data around.

[Grant Edwards]

Post by Neil Bothwick

Post by Grant Edwards
For example, I have a 500GB partition containing an ext4 filesystem
starting at sector 2048 (1MiB). I want to move that filesystem so that
it starts at sector 3*2048 (3MiB).
Can that be done in-place?
Or should I just back up the filesystem to a second drive and start
from scratch?

Given that you'd want to backup before such an operation anyway,

It's a machine with very limited uses, so I'd probably only back up
/etc and /root. Reinstalling probably wouldn't take too much longer
than backing up and restoring /.

Post by Neil Bothwick
you may as well then restore from that backup. I'm sure it will be a
lot quicker than GParted's moving all the data around.

That depends on how long it takes me to decide on tar vs. rsync and
what the appropriate options are. After 40 years using Unix, you'd
think I'd know that (or have it written down somewhere). :)

That said, I think I will go with the backup, repartition, restore
method. It's been many, many years since I used GParted, and I can
probably have the whole job done from the command-line before I can
figure out how the GParted GUI works.

[Victor Ivanov]

Post by Grant Edwards
That depends on how long it takes me to decide on tar vs. rsync and
what the appropriate options are.

I've done this a number of times for various reasons over the last 1-2
years, most recently a few months ago due to hard drive swap, and I
find tar works just fine:

$ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .

Likewise to extract, but make sure "--xattrs" is present

Provided backup space isn't an issue, I wouldn't bother with
compression. It could be a lot quicker too depending on the size of
your root partition.

Post by Grant Edwards
That said, I think I will go with the backup, repartition, restore
method.

Sounds like a sound course of action to me, reinstalling is just too
much faff and can often be avoided. I have a long standing distrust of
partition management utilities that claim to move filesystem data too.
I'm sure they work juuuuust fine :)

Just make sure you update /etc/fstab and bootloader config file with
the new filesystem UUID or partition indices.

Regards,
V

[Frank Steinmetzger]

Post by Victor Ivanov

Post by Grant Edwards
That depends on how long it takes me to decide on tar vs. rsync and
what the appropriate options are.

I've done this a number of times for various reasons over the last 1-2
years, most recently a few months ago due to hard drive swap, and I
$ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .

Does that stop at file system boundaries (because you tar up '/')? I think
it must be, otherwise you wouldn’t use it that way.
But when copying a root file system, out of habit I first bind-mount it in a
subdirectory and tar/rsync from there instead. This will also make files
visible which might be hidden under an active mount.

This is not necessary if you do it from a live system, but then you wouldn’t
tar up / in the first place.

Post by Victor Ivanov
Likewise to extract, but make sure "--xattrs" is present
Provided backup space isn't an issue, I wouldn't bother with
compression. It could be a lot quicker too depending on the size of
your root partition.

Or not, depending on the speed of the backup device. ;-)
LZO compression (or zstd with a low setting) has negligible CPU cost, but
can lower the file size quite nicely, specially with large binaries or debug
files.

[Grant Edwards]

Post by Frank Steinmetzger

Post by Victor Ivanov

Post by Grant Edwards
That depends on how long it takes me to decide on tar vs. rsync and
what the appropriate options are.

I've done this a number of times for various reasons over the last 1-2
years, most recently a few months ago due to hard drive swap, and I
$ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .

Does that stop at file system boundaries (because you tar up '/')? I think
it must be, otherwise you wouldn’t use it that way.
But when copying a root file system, out of habit I first bind-mount it in a
subdirectory and tar/rsync from there instead. This will also make files
visible which might be hidden under an active mount.

The partition/fs being backed up isn't live (it's mounted, but it's
not the root partition of the host doing the backup), so nothing is
mounted within it and there aren't any /proc or /sys entries in
it. So, in this case there's no need to worry about crossing
filesystem boundaries.

--
Grant

[Victor Ivanov]

Post by Frank Steinmetzger

Post by Victor Ivanov
$ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .

Does that stop at file system boundaries (because you tar up '/')? I think
it must be, otherwise you wouldn’t use it that way.

No, it doesn't. It will archive everything recursively including
mounted directories, so fair point for raising this. It's something
that I do not normally consider, as I tend not to do full root backup
on the running system itself. But how cool is using bind mount for
this situation as you suggest? Simple and effective, I like it.

Actually, the presence of -C in the above example is purely out of
habit. I use it when pointing to a full path, e.g. "/path/to/dir" so
it doesn't end up creating the "path/to/dir" path prefix inside the
archive and only archives the contents of the path.

It's effectively changing to that directory and archiving everything
there, but saves you from doing the "cd".

Naturally, for "/" this is superfluous and "-C / ." can be replaced
with just "/".

Post by Frank Steinmetzger

Post by Victor Ivanov
Provided backup space isn't an issue, I wouldn't bother with
compression. It could be a lot quicker too depending on the size of
your root partition.

Or not, depending on the speed of the backup device. ;-)
LZO compression (or zstd with a low setting) has negligible CPU cost, but
can lower the file size quite nicely, specially with large binaries or debug
files.

That's true :) I had somehow forgotten of tar's support for LZO and
zstd as my default finger memory approach is to use -J for xz. Good
memory nudge here!

Regards,
V

[Grant Edwards]

Post by Victor Ivanov

Post by Grant Edwards
That depends on how long it takes me to decide on tar vs. rsync and
what the appropriate options are.

I've done this a number of times for various reasons over the last 1-2
years, most recently a few months ago due to hard drive swap, and I
$ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .
Likewise to extract, but make sure "--xattrs" is present

Yep, that's pretty much what I decided on based on the tar command
shown at

https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Stage

Interestingly, the Arch Linux Wiki recommends using bsdtar because
"GNU tar with --xattrs will not preserve extended attributes".

Post by Victor Ivanov
Provided backup space isn't an issue, I wouldn't bother with
compression. It could be a lot quicker too depending on the size of
your root partition.

Both the drive being "fixed" and the backup drive are in a USB3
attached dual slot drive dock, so I'm thinking compression might be
worthwhile.

Post by Victor Ivanov
Just make sure you update /etc/fstab and bootloader config file with
the new filesystem UUID or partition indices.

I always forget one or the other until after I try to boot the first
time. That's why I keep systemrescuecd and Gentoo minimal install
USB drives on hand.

--
Grant

[Victor Ivanov]

Post by Grant Edwards
Yep, that's pretty much what I decided on based on the tar command
shown at
https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Stage
Interestingly, the Arch Linux Wiki recommends using bsdtar because
"GNU tar with --xattrs will not preserve extended attributes".

I remember coming across this too as I've previously had issues
preserving some extended attributes, notably on files under my home
directory. However, I found that using "--xattrs-include='*.*'" in
addition to "--xattrs" works pretty well and does record attributes
that would otherwise be excluded with just "--xattrs".

I cannot comment, however, if it truly includes "everything" in every
possible scenario.

Post by Grant Edwards
Both the drive being "fixed" and the backup drive are in a USB3
attached dual slot drive dock, so I'm thinking compression might be
worthwhile.

Then LZO or zstd might indeed be a better approach as suggested by Frank.

Post by Grant Edwards

Post by Victor Ivanov
Just make sure you update /etc/fstab and bootloader config file with
the new filesystem UUID or partition indices.

I always forget one or the other until after I try to boot the first
time. That's why I keep systemrescuecd and Gentoo minimal install
USB drives on hand.

Me too, even just recently when I migrated my OS to another build I
decided to do a few partition touch ups and fell once more into this
trap. I updated fstab but not the bootloader. Luckily, Gentoo minimal
install image is so tiny a bootable medium can literally be created in
minutes.

Good luck!

Regards,
V

[Grant Edwards]

Post by Victor Ivanov

Post by Grant Edwards

Post by Victor Ivanov
Just make sure you update /etc/fstab and bootloader config file
with the new filesystem UUID or partition indices.

I always forget one or the other until after I try to boot the
first time. That's why I keep systemrescuecd and Gentoo minimal
install USB drives on hand.

Me too, even just recently when I migrated my OS to another build I
decided to do a few partition touch ups and fell once more into this
trap. I updated fstab but not the bootloader. Luckily, Gentoo
minimal install image is so tiny a bootable medium can literally be
created in minutes.

The tar backup restore worked just fine (and didn't take long, even
though both drives were connected via USB). I've since fixed a second
machine by adding a bios-boot partition. I should have started using
them when I switched from MBR to GPT, but I think I got bios-boot
partitions confused with UEFI boot partitions. :/

I'm also working on switching to using either labels or uuids in fstab
and grub configs so that changes in partition numbers don't cause
problems. Of course I've discovered for the Nth time in the past
10-15 years, that for the root= command line argument, the kernel
doesn't grok LABEL or UUID values -- it only understands device names
and PARTUUID.

--
Grant

[Jack]

Post by Grant Edwards

Post by Victor Ivanov

Post by Grant Edwards

Post by Victor Ivanov
Just make sure you update /etc/fstab and bootloader config file
with the new filesystem UUID or partition indices.

I always forget one or the other until after I try to boot the
first time. That's why I keep systemrescuecd and Gentoo minimal
install USB drives on hand.

Me too, even just recently when I migrated my OS to another build I
decided to do a few partition touch ups and fell once more into this
trap. I updated fstab but not the bootloader. Luckily, Gentoo
minimal install image is so tiny a bootable medium can literally be
created in minutes.

The tar backup restore worked just fine (and didn't take long, even
though both drives were connected via USB). I've since fixed a second
machine by adding a bios-boot partition. I should have started using
them when I switched from MBR to GPT, but I think I got bios-boot
partitions confused with UEFI boot partitions. :/
I'm also working on switching to using either labels or uuids in fstab
and grub configs so that changes in partition numbers don't cause
problems. Of course I've discovered for the Nth time in the past
10-15 years, that for the root= command line argument, the kernel
doesn't grok LABEL or UUID values -- it only understands device names
and PARTUUID.

while my Gentoo grub.cfg has root=PARTUUID=, my Artix Linux install
(using openrc) has root=UUID=.  I wasn't aware they had mucked with grub
(2.12-rc1) nor do I know if it's a recent change in grub.

[Grant Edwards]

Post by Jack

[...] Of course I've discovered for the Nth time in the past 10-15
years, that for the root= command line argument, the kernel doesn't
grok LABEL or UUID values -- it only understands device names and
PARTUUID.

while my Gentoo grub.cfg has root=PARTUUID=, my Artix Linux install
(using openrc) has root=UUID=.  I wasn't aware they had mucked with
grub (2.12-rc1) nor do I know if it's a recent change in grub.

AFAIK, it's not grub (grub does know how to handle LABEL and UUID when
setting grub's root). For the kernel, it's something in the initrd's
'init' executable that parses the root=UUID= or root=LABEL=, searches
the available filesystems to find a match, then mounts the matching
filesystem and does a chroot to it (or someting like that).

If you don't have an initrd, then the kernel itself has to handle the
root=<whatever> and that code only knows about device names and
partition UUIDs. It doesn't know anything about filesystems (which
doesn't make much sense, since the next step is to mount the specified
partition, and it obviously knows about filesystems at that point).

At least that's what I've read...

[Neil Bothwick]

Post by Grant Edwards

Post by Neil Bothwick
you may as well then restore from that backup. I'm sure it will be a
lot quicker than GParted's moving all the data around.

That depends on how long it takes me to decide on tar vs. rsync and
what the appropriate options are. After 40 years using Unix, you'd
think I'd know that (or have it written down somewhere). :)

If you written everything down, after 40 years you'd have so many bits of
paper you wouldn't be able to find anything ;-)

Post by Grant Edwards
That said, I think I will go with the backup, repartition, restore
method. It's been many, many years since I used GParted, and I can
probably have the whole job done from the command-line before I can
figure out how the GParted GUI works.

:)

[Wol]

Post by Neil Bothwick

Post by Grant Edwards
However, GParted apparently does. Can GParted move an ext4 filesystem
to a destination location that overlaps its starting location?
For example, I have a 500GB partition containing an ext4 filesystem
starting at sector 2048 (1MiB). I want to move that filesystem so that
it starts at sector 3*2048 (3MiB).
Can that be done in-place?
Or should I just back up the filesystem to a second drive and start
from scratch?

Given that you'd want to backup before such an operation anyway, you may
as well then restore from that backup. I'm sure it will be a lot quicker
than GParted's moving all the data around.

Or, assuming the people who wrote gparted have two brain cells to rub
together, I'm pretty sure they use the same technique as memmove.

"If the regions overlap, make sure you start from whichever end won't
overwrite the source, otherwise start at whichever end you like".

Barring screw-ups (a very unsafe assumption :-), I'm pretty certain you
don't even need a backup!

I suspect the man-page even confirms this behaviour.

Cheers,
Wol

[Grant Edwards]

Post by Wol
Or, assuming the people who wrote gparted have two brain cells to rub
together, I'm pretty sure they use the same technique as memmove.
"If the regions overlap, make sure you start from whichever end won't
overwrite the source, otherwise start at whichever end you like".
Barring screw-ups (a very unsafe assumption :-), I'm pretty certain you
don't even need a backup!
I suspect the man-page even confirms this behaviour.

Not that I could see. The only mention of "move" on the man page is
in this list of features.

With gparted you can accomplish the following tasks:
- Create a partition table on a disk device.
- Enable and disable partition flags such as boot and hidden.
- Perform actions with partitions such as create, delete,
resize, move, check, label, copy, and paste.

Assuming GParted is smart enough to do overlapping moves, is it smart
enough to only copy filesystem data and not copy "empty" sectors?
According to various forum posts, it is not: moving a partion copies
every sector. [That's certainly the obvious, safe thing to do.]

The partition in question is 200GB, but only 7GB is used, so I think
backup/restore is the way to go...

--
Grant

[Wols Lists]

Post by Grant Edwards
Assuming GParted is smart enough to do overlapping moves, is it smart
enough to only copy filesystem data and not copy "empty" sectors?
According to various forum posts, it is not: moving a partion copies
every sector. [That's certainly the obvious, safe thing to do.]

Seeing as it knows nothing about filesystems, and everything about
partitions, it will treat the partition as an opaque blob and move it as
a single object ...

Post by Grant Edwards
The partition in question is 200GB, but only 7GB is used, so I think
backup/restore is the way to go...

You would think so :-)

I use ext4, and make heavy use of hard links. Last time I tried a
straight copy (not backup/restore) I think the copied partition would
have been three times the size of the original - that is if it hadn't
run out of space first :-)

But it sounds like that would work well for you.

Cheers,
Wol

[Håkon Alstadheim]

Post by Wols Lists

Post by Grant Edwards
Assuming GParted is smart enough to do overlapping moves, is it smart
enough to only copy filesystem data and not copy "empty" sectors?
According to various forum posts, it is not: moving a partion copies
every sector. [That's certainly the obvious, safe thing to do.]

Seeing as it knows nothing about filesystems, and everything about
partitions, it will treat the partition as an opaque blob and move it
as a single object ...

Post by Grant Edwards
The partition in question is 200GB, but only 7GB is used, so I think
backup/restore is the way to go...

You would think so :-)
I use ext4, and make heavy use of hard links. Last time I tried a
straight copy (not backup/restore) I think the copied partition would
have been three times the size of the original - that is if it hadn't
run out of space first :-)

Just for completeness, you should check out the e2image command, though
NOT for direct in-place overlapping move. Something like "e2image -ra
$sourcevol $targetvol" has worked very well for me. That is obviously
NOT for overlapping move, but for a backup it is quite fast. If you are
moving to a bigger volume, I'd do resize as a separate step after the move.