[gentoo-user] Something not right with LVM, I think.

[Dale]

Howdy,

I have several external hard drives. I have one that is, weird.  Others
work fine.  When I first power up the drive, cryptsetup can't open it
because it doesn't exist yet.  This is the relevant part of lsblk -f
when it doesn't work:


NAME               FSTYPE      FSVER    LABEL      
UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sdr                                                                                                      

└─sdr1             LVM2_member LVM2 001            
SLbMcr-d3M9-XONQ-jPtM-Meyd-H6tY-hjSc5S


Then I restart lvm, which I don't like doing when things are running.  :/ 


NAME               FSTYPE      FSVER    LABEL      
UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sdr                                                                                                      

└─sdr1             LVM2_member LVM2 001            
SLbMcr-d3M9-XONQ-jPtM-Meyd-H6tY-hjSc5S               
  └─10tb-10tb--lv  crypto_LUKS 2                   
eab6268e-721b-4c72-aef3-0ecc319f18d4


As you can see, when I restart the LVM service, it adds that last little
bit which is what cryptsetup needs to open.  My question is this, why do
I need to restart lvm for that?  Also, everything for that drive shows
up just fine in pvs, vgs and lvs.  They all recognize the drive as being
connected and ready.  I might add, it also shows up under /dev/mapper as
well, before I restart LVM that is. 

This seems to have started when I added some encryption options to the
new kernel.  Thing is, I also started using this external case about the
same time.  Also, it seems to work fine on the 770T.  I tested it on
that too.  Here is some additional info:



***@fireball / # emerge -pv sys-fs/lvm2 sys-fs/cryptsetup

These are the packages that would be merged, in order:

Calculating dependencies... done!
Dependency resolution took 63.13 s (backtrack: 0/50).

[ebuild   R    ] sys-fs/lvm2-2.03.21-r1::gentoo  USE="lvm readline udev
-sanlock (-selinux) -static -static-libs -systemd -thin -valgrind" 0 KiB
[ebuild   R    ] sys-fs/cryptsetup-2.6.1:0/12::gentoo  USE="argon2 nls
openssl udev -fips -gcrypt -kernel -nettle -pwquality -ssh -static
-static-libs -test -urandom" 0 KiB

Total: 2 packages (2 reinstalls), Size of downloads: 0 KiB
***@fireball / #



Anyone have a idea on why I have to restart LVM to get this one drive to
work?  Once I restart LVM, everything works fine.  Is it LVM or
something else?  It seems LVM is working, it is seeing the drives. 
However, restarting LVM fixes it.  Is it something to do with cryptsetup
and restarting LVM fixes it?  Open to ideas.  Need more info, let me
know, maybe how to get it as well. 

Thanks.

Dale

:-)  :-) 

[Neil Bothwick]

I have several external hard drives. I have one that is, weird.  Others
work fine.  When I first power up the drive, cryptsetup can't open it
because it doesn't exist yet.

Is it the drive/enclosure taking too long to power up? There are kernel
options to delay the boot to allow for USB devices to become available,
such as usb-storage.delay_use. Try setting a long delay and, if that
fixes it, reducing the delay until you find a suitable setting.

[Michael]

Post by Neil Bothwick

I have several external hard drives. I have one that is, weird. Others
work fine. When I first power up the drive, cryptsetup can't open it
because it doesn't exist yet.

Is it the drive/enclosure taking too long to power up? There are kernel
options to delay the boot to allow for USB devices to become available,
such as usb-storage.delay_use. Try setting a long delay and, if that
fixes it, reducing the delay until you find a suitable setting.

There's also a rootwait kernel cmdline option. I had to use this to be able
to boot a rootfs on a (sluggish) spinning drive on a USB docking station, but
I suspect this does not apply to your setup if the rootfs is not stored on
this drive.

Either way, there may be a case for setting rc_after=lvm in the device_mapper
configuration, to make sure all ducks are in a row as the devices and fs
containers are initialised in the correct order. You can manage rc service
priorities by playing with rc_after and rc_need options.

You mention two things, 770T works fine with this combo of external drives and
the problem started after you enabled crypto hardware acceleration in the FX
kernel. Assuming these observations are reliably repeatable, then this could
be explained as follows:

The 770T performs decryption slowly using its CPU core(s), as well as loading
drivers and initialising any rc services including LVM. This means any
initialisation by the kernel of (sluggish) hardware will give it enough time
to get up and running. With the FX system, everything happens much faster and
at some point it trips over itself, because the sluggish hardware hasn't got
its boots on in a timely fashion.

[Dale]

Post by Michael

Post by Neil Bothwick

I have several external hard drives. I have one that is, weird. Others
work fine. When I first power up the drive, cryptsetup can't open it
because it doesn't exist yet.

Is it the drive/enclosure taking too long to power up? There are kernel
options to delay the boot to allow for USB devices to become available,
such as usb-storage.delay_use. Try setting a long delay and, if that
fixes it, reducing the delay until you find a suitable setting.

There's also a rootwait kernel cmdline option. I had to use this to be able
to boot a rootfs on a (sluggish) spinning drive on a USB docking station, but
I suspect this does not apply to your setup if the rootfs is not stored on
this drive.
Either way, there may be a case for setting rc_after=lvm in the device_mapper
configuration, to make sure all ducks are in a row as the devices and fs
containers are initialised in the correct order. You can manage rc service
priorities by playing with rc_after and rc_need options.
You mention two things, 770T works fine with this combo of external drives and
the problem started after you enabled crypto hardware acceleration in the FX
kernel. Assuming these observations are reliably repeatable, then this could
The 770T performs decryption slowly using its CPU core(s), as well as loading
drivers and initialising any rc services including LVM. This means any
initialisation by the kernel of (sluggish) hardware will give it enough time
to get up and running. With the FX system, everything happens much faster and
at some point it trips over itself, because the sluggish hardware hasn't got
its boots on in a timely fashion.

I may try to boot my old kernel and see if it works as it should there. 
I'm pretty sure I connected the new enclosure a couple times tho to the
old kernel and it worked.  I'm just not 100% sure.  I'm just wondering
if something I enabled might affect how it works somehow.  I mostly
followed the Gentoo dm-crypt wiki except that I enabled a few extra
encryption logarithms.  Thing is, given the 770T has a newer kernel than
my main rig, hard to compare. 

I got my backups moved around so I won't need to hook it back up for a
week or so.  If I get the chance to reboot tho, I can always test it then.

Wol, I used USB a few times on some old enclosures I had.  I bricked a
few hard drives with those things.  I was always getting data errors and
eventually, the drives would die.  Oddly, one of the enclosures worked
fine.  I stopped using them when I bought the ones I use now. 

Dale

:-)  :-) 

[Dale]

Post by Neil Bothwick

Post by Dale
I have several external hard drives. I have one that is, weird.  Others
work fine.  When I first power up the drive, cryptsetup can't open it
because it doesn't exist yet.

Is it the drive/enclosure taking too long to power up? There are kernel
options to delay the boot to allow for USB devices to become available,
such as usb-storage.delay_use. Try setting a long delay and, if that
fixes it, reducing the delay until you find a suitable setting.

I connect by eSATA.  I hate using USB for a hard drive.  This is when I
have my system already booted so I'm hot plugging.  Also, I use
cryptsetup manually.  I should have mentioned that in the original
post.  I did plug up one of my other external drives and watch
/var/log/messages.  Both have the same things listed when I power up the
external drives.  I couldn't tell anything different.  Also, they all
power up in the same time frame.  Usually, 10 seconds or so from power
on to nothing more in messages. 

The enclosure that doesn't work right is a StarTech SAT3510BU2E.  The
ones that work fine, they are Rosewill RX304APU335B.  Both of those
models have fans.  They have both eSATA and USB but I've never used the
USB ports. 

All the external drives appear the same in every way except that one
enclosure requires me to restart LVM to get that last bit cryptsetup
needs.  I can't make any sense of it.  I have two of the StarTech ones. 
I may try the other one.  See if it does the same thing.  What's odd, it
seems to work fine on the 770T system with a fresh install and new
kernel.  

I don't see anything different between the two types of enclosures
except I have to restart LVM with one of them.  Weird for sure. 

Any ideas?

Dale

:-)  :-) 

[Wol]

Post by Dale
I don't see anything different between the two types of enclosures
except I have to restart LVM with one of them.  Weird for sure.

Well, on the linux raid wiki I explicitly tell people not to use USB
drives for raid. "We don't know why, but it seems guaranteed to cause
problems".

Cheers,
Wol

[Mark Knecht]

On Sun, Oct 29, 2023 at 7:28 AM Dale <***@gmail.com> wrote:
<SNIP>.

The enclosure that doesn't work right is a StarTech SAT3510BU2E. The
ones that work fine, they are Rosewill RX304APU335B. Both of those
models have fans. They have both eSATA and USB but I've never used the
USB ports.
All the external drives appear the same in every way except that one
enclosure requires me to restart LVM to get that last bit cryptsetup
needs. I can't make any sense of it. I have two of the StarTech ones.
I may try the other one. See if it does the same thing. What's odd, it
seems to work fine on the 770T system with a fresh install and new
kernel.
I don't see anything different between the two types of enclosures
except I have to restart LVM with one of them. Weird for sure.
Any ideas?

First, Hi Dale...

I'm hesitant to take you in the wrong direction. Just jotting down
a couple of thoughts.

1) I know you don't like rebooting but if it was me I would mod my
/etc/fstab to not mount these drives automatically. (noauto vs auto).
I would then reboot and do some experimenting mounting by hand,
or in a bash script, to see if they reliably mount and unmount after
the machine is up.

2) You suggest these enclosures are identical, but in a quick search
for specs the Rosewill appears to support drives up to 6TB but the
StarTech only supports up to 4TB. What size drives are you using?

https://media.startech.com/cms/pdfs/sat3510bu2e_datasheet.pdf

https://www.newegg.com/rosewill-rx304-apu3-35b/p/N82E16817182316

If those specs are the right ones the enclosures are not identical.

Good luck,
Mark

[Dale]

Post by Mark Knecht
<SNIP>.

The enclosure that doesn't work right is a StarTech SAT3510BU2E.  The
ones that work fine, they are Rosewill RX304APU335B.  Both of those
models have fans.  They have both eSATA and USB but I've never used the
USB ports.
All the external drives appear the same in every way except that one
enclosure requires me to restart LVM to get that last bit cryptsetup
needs.  I can't make any sense of it.  I have two of the StarTech ones.
I may try the other one.  See if it does the same thing.  What's odd, it
seems to work fine on the 770T system with a fresh install and new
kernel.  
I don't see anything different between the two types of enclosures
except I have to restart LVM with one of them.  Weird for sure.
Any ideas?

First, Hi Dale...
I'm hesitant to take you in the wrong direction. Just jotting down
a couple of thoughts.
1) I know you don't like rebooting but if it was me I would mod my
/etc/fstab to not mount these drives automatically. (noauto vs auto).
I would then reboot and do some experimenting mounting by hand,
or in a bash script, to see if they reliably mount and unmount after 
the machine is up.

That is already done.  They are never connected when I boot up anyway. 
I did the same on the NAS box as well. 

Post by Mark Knecht
2) You suggest these enclosures are identical, but in a quick search
for specs the Rosewill appears to support drives up to 6TB but the
StarTech only supports up to 4TB. What size drives are you using?
https://media.startech.com/cms/pdfs/sat3510bu2e_datasheet.pdf
https://www.newegg.com/rosewill-rx304-apu3-35b/p/N82E16817182316
If those specs are the right ones the enclosures are not identical. 
Good luck,
Mark

I meant the info in the messages file when powering up the drives were
the same.  The enclosures are a bit different but they are all eSATA.  I
did swap out the enclosure and the other did the same. I may put that
drive in a Rosewill and see if it works as it should then, eliminate the
drive as a problem.  I have three Rosewill and two Startech. 

I've seen those limited to smaller sizes in the past.  Generally tho,
they always work regardless of size unless they really small like older
IDE drives or something.  I read once what causes that and don't buy
those.  Usually if I read it in the description, my bell goes off.  I
haven't seen one that is actually limited in ages. 

Good idea tho. 

Dale

:-)  :-) 

[Dale]

Post by Dale

Post by Mark Knecht
2) You suggest these enclosures are identical, but in a quick search
for specs the Rosewill appears to support drives up to 6TB but the
StarTech only supports up to 4TB. What size drives are you using?
https://media.startech.com/cms/pdfs/sat3510bu2e_datasheet.pdf
https://www.newegg.com/rosewill-rx304-apu3-35b/p/N82E16817182316
If those specs are the right ones the enclosures are not identical. 
Good luck,
Mark

I meant the info in the messages file when powering up the drives were
the same.  The enclosures are a bit different but they are all eSATA. 
I did swap out the enclosure and the other did the same. I may put
that drive in a Rosewill and see if it works as it should then,
eliminate the drive as a problem.  I have three Rosewill and two
Startech. 
I've seen those limited to smaller sizes in the past.  Generally tho,
they always work regardless of size unless they really small like
older IDE drives or something.  I read once what causes that and don't
buy those.  Usually if I read it in the description, my bell goes
off.  I haven't seen one that is actually limited in ages. 
Good idea tho. 
Dale
:-)  :-) 

Here's a update.  I really don't like restarting lvm when I'm logged
into my desktop and everything.  It has never hurt anything in the past
but still, it could mess up something.  So, I removed the 10TB hard
drive from the Startech enclosure and put it in a spare Rosewill
enclosure.  I powered it up, it sees the drive itself but does not add
the LVM part I need to decrypt just like before.  I restarted LVM and
there it is, the LVM part I need.  So, it isn't the enclosure, it
appears to be the drive itself which is really confusing.  I've never
had any hard drive to do this even when I'm starting with a fresh drive
and setting it up.  Everything gets added as I set it up.  Everything
from partition table to the encrypted file system. 

I may just redo this drive from scratch.  Use dd to erase the partition
table and even some data, after all it is encrypted already, and start
from scratch.  Maybe I did something wrong and didn't notice it.  I dunno. 

I just wanted to update with this info just in case someone else runs
into this issue.  No real solution yet but will post back if restarting
from scratch fixes it. 

Dale

:-)  :-) 

[Dale]

Post by Dale
Here's a update.  I really don't like restarting lvm when I'm logged
into my desktop and everything.  It has never hurt anything in the past
but still, it could mess up something.  So, I removed the 10TB hard
drive from the Startech enclosure and put it in a spare Rosewill
enclosure.  I powered it up, it sees the drive itself but does not add
the LVM part I need to decrypt just like before.  I restarted LVM and
there it is, the LVM part I need.  So, it isn't the enclosure, it
appears to be the drive itself which is really confusing.  I've never
had any hard drive to do this even when I'm starting with a fresh drive
and setting it up.  Everything gets added as I set it up.  Everything
from partition table to the encrypted file system. 
I may just redo this drive from scratch.  Use dd to erase the partition
table and even some data, after all it is encrypted already, and start
from scratch.  Maybe I did something wrong and didn't notice it.  I dunno. 
I just wanted to update with this info just in case someone else runs
into this issue.  No real solution yet but will post back if restarting
from scratch fixes it. 
Dale
:-)  :-) 

Final update I guess.  I used dd to erase partition table etc from the
drive and started over from scratch.  The drive still does the same
thing.  I have to restart LVM for it to allow me to have the LVM part I
need to make cryptsetup work.  It is really weird but this is the only
drive that does this.  I ran smartctl -l selftest and it passed.  I'm
not sure what to think about this.  As soon as I can, I'll replace the
drive just in case something is wrong that smart isn't picking up on. 

Dale

:-)  :-)