[gentoo-user] 6.1.53-gentoo-r1 kernel not booting

Post by Valmor F. de Almeida
Hello,
For a while now (3 weeks or so) I have been upgrading the linux kernel
on a Dell XPS laptop starting from 6.1.41-gentoo (which is my current
working kernel) to 6.1.53-gentoo-r1. No kernel I have built since is
make oldconfig, etc...
[snip]
* INIT: Entering runlevel: 3
[snip]
* Starting cronie ...
* Starting DHCP Client Daemon ...
* Starting laptop_mode ...
* Mounting network filesystems ...
/etc/init.d/netmount: line 45 /lib/rc/bin/ewend: Input/output error
/lib/rc/sh/rc-cgroup.sh: line 184: rmdir: command not found
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"

Can you show /etc/fstab and the console-log for the entire boot? Seems
/sbin is not readable. You sure you have the kernel modules loaded? Are
you using an initramfs? If so, does that build without errors ?

Valmor F. de Almeida

2023-10-01 05:00:01 UTC

Here is fstab:

/dev/nvme0n1p2 /boot ext2 defaults 0 2
/dev/nvme0n1p3 none swap sw 0 0
/dev/nvme0n1p4 / ext4 noatime,discard
0 1
/dev/cdrom /mnt/cdrom auto noauto,user 0 0

I have not changed anything from 6.1.41-gentoo (which compiles and
boots) except updating the config file for compiling the new kernel.
Then I do: make && modules_install. Which runs without errors. After
that: grub-mkconfig -o /boot/grub/grub.cfg.
I am not using initramfs.

I don't know how to save the boot messages; it seems that if I turn on
the logger in openrc, it will log the openrc messages but not sysinit
runlevels?

Thanks,
--
Valmor

Michael

2023-10-01 11:40:01 UTC

/dev/nvme0n1p2 /boot ext2 defaults 0 2
/dev/nvme0n1p3 none swap sw 0 0
/dev/nvme0n1p4 / ext4 noatime,discard
0 1
/dev/cdrom /mnt/cdrom auto noauto,user 0 0
I have not changed anything from 6.1.41-gentoo (which compiles and
boots) except updating the config file for compiling the new kernel.
Then I do: make && modules_install. Which runs without errors. After
that: grub-mkconfig -o /boot/grub/grub.cfg.
I am not using initramfs.
I don't know how to save the boot messages; it seems that if I turn on
the logger in openrc, it will log the openrc messages but not sysinit
runlevels?
Thanks,
--
Valmor

I think the error messages you're getting indicate inability to access your
rootfs. Have you perhaps changed the fs drivers in the latest kernels, from
built in to modules?

Have you diff'ed your 6.1.41-gentoo .config file against the latest kernels'
.config files to see what might have changed/missing?

You can set up a netconsole to check boot time messages:

https://www.kernel.org/doc/Documentation/networking/netconsole.txt
https://wiki.ubuntu.com/Kernel/Netconsole

In summary, build netconsole in the new kernel, or as a module, change GRUB's
default CMDLINE from "quiet splash" to "debug" and add netconsole in the
kernel command line:

netconsole=@/,[PORT]@[DEST_IP]

Then at the destination PC launch netcat/socat/telnet; e.g.

nc -u -l -p [PORT]

and reboot the PC you want to debug with the latest kernel, while watching the
output on the destination PC's terminal with nc.

Valmor F. de Almeida

2023-10-01 17:50:01 UTC

Post by Michael

I think the error messages you're getting indicate inability to access your
rootfs. Have you perhaps changed the fs drivers in the latest kernels, from
built in to modules?
Have you diff'ed your 6.1.41-gentoo .config file against the latest kernels'
.config files to see what might have changed/missing?
https://www.kernel.org/doc/Documentation/networking/netconsole.txt
https://wiki.ubuntu.com/Kernel/Netconsole
In summary, build netconsole in the new kernel, or as a module, change GRUB's
default CMDLINE from "quiet splash" to "debug" and add netconsole in the
Then at the destination PC launch netcat/socat/telnet; e.g.
nc -u -l -p [PORT]
and reboot the PC you want to debug with the latest kernel, while watching the
output on the destination PC's terminal with nc.

Here is the diff for the config files:

-> diff config /boot/config-6.1.41-gentoo
3c3
< # Linux/x86 6.1.53-gentoo-r1 Kernel Configuration
---

Post by Michael
# Linux/x86 6.1.41-gentoo Kernel Configuration

5c5
< CONFIG_CC_VERSION_TEXT="gcc (Gentoo 13.2.1_p20230826 p7) 13.2.1 20230826"
---

Post by Michael
CONFIG_CC_VERSION_TEXT="gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526"

7c7
< CONFIG_GCC_VERSION=130201
---

Post by Michael
CONFIG_GCC_VERSION=120301

455d454
< CONFIG_CPU_SRSO=y
457d455
< # CONFIG_GDS_FORCE_MITIGATION is not set
646d643
< CONFIG_ARCH_HAS_CPU_FINALIZE_INIT=y
3136d3132
< CONFIG_VIDEO_V4L2_SUBDEV_API=y
3139,3140d3134
< CONFIG_V4L2_FWNODE=m
< CONFIG_V4L2_ASYNC=m
3236c3230,3233
< CONFIG_VIDEO_CAMERA_SENSOR=y
---

Post by Michael
#
# Camera sensor devices
#

3295a3293

Post by Michael
# end of Camera sensor devices

I guess I could explore some of this difference.
I have not changed the way I compile the kernel and I typically try to
have everything built in (only a few modules).
Thanks for the netconsole info.
--
Valmor

Håkon Alstadheim

2023-10-01 17:40:01 UTC

Post by Valmor F. de Almeida
Hello,
For a while now (3 weeks or so) I have been upgrading the linux
kernel on a Dell XPS laptop starting from 6.1.41-gentoo (which is my
current working kernel) to 6.1.53-gentoo-r1. No kernel I have built
since is able to boot. I have been following the same method for
many years: make oldconfig, etc...
[snip]
* INIT: Entering runlevel: 3
[snip]
* Starting cronie ...
* Starting DHCP Client Daemon ...
* Starting laptop_mode ...
* Mounting network filesystems ...
/etc/init.d/netmount: line 45 /lib/rc/bin/ewend: Input/output error
/lib/rc/sh/rc-cgroup.sh: line 184: rmdir: command not found
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"

Can you show /etc/fstab and the console-log for the entire boot?
Seems /sbin is not readable. You sure you have the kernel modules
loaded? Are you using an initramfs? If so, does that build without
errors ?

/dev/nvme0n1p2          /boot           ext2 defaults        0 2
/dev/nvme0n1p3          none            swap sw              0 0
/dev/nvme0n1p4          /               ext4 noatime,discard        0 1
/dev/cdrom              /mnt/cdrom      auto noauto,user     0 0
I have not changed anything from 6.1.41-gentoo (which compiles and
boots) except updating the config file for compiling the new kernel.
Then I do: make && modules_install. Which runs without errors. After
that: grub-mkconfig -o /boot/grub/grub.cfg.
I am not using initramfs.

I can't see anything fishy in fstab, no network-mounts that might have
failed. From the original message it looks like your network interface
is not being set up correctly, but you should still be able to boot.
Also "rmdir: command not found" probably means your PATH does not get
set, or else you don't have execute permissions on rmdir. Odd that this
should be dependent on kernel version. You should go over and verify
that the transfer of your linux .config went OK. Then verify that the
new config does in fact enable modules for your network card, and enable
the necessary settings for your init system (looks like you are using
openrc?)

Frank Steinmetzger

2023-10-01 19:40:01 UTC

Post by Valmor F. de Almeida
Hello,
For a while now (3 weeks or so) I have been upgrading the linux kernel
on a Dell XPS laptop starting from 6.1.41-gentoo (which is my current
working kernel) to 6.1.53-gentoo-r1. No kernel I have built since is
able to boot. I have been following the same method for many years: make
oldconfig, etc...
[snip]
* INIT: Entering runlevel: 3
[snip]
* Starting cronie ...
* Starting DHCP Client Daemon ...
* Starting laptop_mode ...
* Mounting network filesystems ...
/etc/init.d/netmount: line 45 /lib/rc/bin/ewend: Input/output error
/lib/rc/sh/rc-cgroup.sh: line 184: rmdir: command not found
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"

Can you show /etc/fstab and the console-log for the entire boot? Seems /sbin
is not readable. You sure you have the kernel modules loaded? Are you using
an initramfs? If so, does that build without errors ?

The input/output error â to me â indicates a hardware problem. When you
mounted the FS by hand, can you read ewend? For instance with md5sum.

--
GrÃŒÃe | Greetings | Salut | Qaplaâ
Please do not share anything from, with or about me on any social network.

âIf wishes were horses weâd all be eating steak.â â Jayne, Firefly

Håkon Alstadheim

2023-10-02 00:30:01 UTC

Post by Valmor F. de Almeida
Hello,
For a while now (3 weeks or so) I have been upgrading the linux kernel
on a Dell XPS laptop starting from 6.1.41-gentoo (which is my current
working kernel) to 6.1.53-gentoo-r1. No kernel I have built since is
able to boot. I have been following the same method for many years: make
oldconfig, etc...
[snip]
* INIT: Entering runlevel: 3
[snip]
* Starting cronie ...
* Starting DHCP Client Daemon ...
* Starting laptop_mode ...
* Mounting network filesystems ...
/etc/init.d/netmount: line 45 /lib/rc/bin/ewend: Input/output error
/lib/rc/sh/rc-cgroup.sh: line 184: rmdir: command not found
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"

Can you show /etc/fstab and the console-log for the entire boot? Seems /sbin
is not readable. You sure you have the kernel modules loaded? Are you using
an initramfs? If so, does that build without errors ?

The input/output error – to me – indicates a hardware problem. When you
mounted the FS by hand, can you read ewend? For instance with md5sum.

except it boots ok with older kernels. When you've eliminated the
impossible, whatever remains, however improbable has to be a kernel
config change (missing&necessary or erroneous and unintended) , or
initramfs failing to build/install correctly. Check error output from
your kernel&initramfs build.

Lee K

2023-10-02 03:40:01 UTC

Post by Valmor F. de Almeida
Hello,
For a while now (3 weeks or so) I have been upgrading the linux kernel
on a Dell XPS laptop starting from 6.1.41-gentoo (which is my current
working kernel) to 6.1.53-gentoo-r1. No kernel I have built since is
able to boot. I have been following the same method for many years: make
oldconfig, etc...
[snip]
* INIT: Entering runlevel: 3
[snip]
* Starting cronie ...
* Starting DHCP Client Daemon ...
* Starting laptop_mode ...
* Mounting network filesystems ...
/etc/init.d/netmount: line 45 /lib/rc/bin/ewend: Input/output error
/lib/rc/sh/rc-cgroup.sh: line 184: rmdir: command not found
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"

Can you show /etc/fstab and the console-log for the entire boot? Seems /sbin
is not readable. You sure you have the kernel modules loaded? Are you using
an initramfs? If so, does that build without errors ?

The input/output error – to me – indicates a hardware problem. When you
mounted the FS by hand, can you read ewend? For instance with md5sum.

I found this:

https://www.reddit.com/r/Gentoo/comments/168qrbx/gentoosources_6146_kernel_reports_io_error/

--
Lee

Lee

2023-10-02 03:50:01 UTC

I found this:

https://www.reddit.com/r/Gentoo/comments/168qrbx/gentoosources_6146_kernel_reports_io_error/

Lee ð

make

Post by Valmor F. de Almeida
oldconfig, etc...
[snip]
* INIT: Entering runlevel: 3
[snip]
* Starting cronie ...
* Starting DHCP Client Daemon ...
* Starting laptop_mode ...
* Mounting network filesystems ...
/etc/init.d/netmount: line 45 /lib/rc/bin/ewend: Input/output error
/lib/rc/sh/rc-cgroup.sh: line 184: rmdir: command not found
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"

Can you show /etc/fstab and the console-log for the entire boot? Seems

/sbin

Post by HÃ¥kon Alstadheim
is not readable. You sure you have the kernel modules loaded? Are you

using

Post by HÃ¥kon Alstadheim
an initramfs? If so, does that build without errors ?

The input/output error â to me â indicates a hardware problem. When you
mounted the FS by hand, can you read ewend? For instance with md5sum.

Peter Böhm

2023-10-02 09:40:01 UTC

If your system is working with your old kernel I dont think it could be a
hardware problem. If your kernel config is the same then I dont think it is a
problem with the kernel.

My questions would be:

Which CPU do you use ? Have you emerged a new microcode for CPU ?

Example: Intel has disabled avx in some CPUs via microcode update. Now some
applications may throw an illegal instruction ... and your problem description
look like that only some programs have a problem. You could test it by
compiling your old kernel again ... you should have then the same problem with
it (of course you backup your old kernel before).

Regards,
Peter

Valmor F. de Almeida

2023-10-04 03:50:01 UTC

Post by Peter BÃ¶hm
If your system is working with your old kernel I dont think it could be a
hardware problem. If your kernel config is the same then I dont think it is a
problem with the kernel.
Which CPU do you use ? Have you emerged a new microcode for CPU ?
Example: Intel has disabled avx in some CPUs via microcode update. Now some
applications may throw an illegal instruction ... and your problem description
look like that only some programs have a problem. You could test it by
compiling your old kernel again ... you should have then the same problem with
it (of course you backup your old kernel before).
Regards,
Peter

The config file is not the same but I have played with what has changed
and I don't seem to be able to fix the problem; no boot.

-> cat /proc/cpuinfo

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 158
model name : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
stepping : 9
microcode : 0x84
cpu MHz : 3680.490
cache size : 6144 KB

[snip]
There are 8 cores.

There may have been a microcode emerge from dependencies; I don't emerge
directly. I do agree with you: if I try to recompile the working
kernel it will not boot. However I am not willing to try just right now
and having to troubleshoot a broken system.
Thanks,
--
Valmor

Neil Bothwick

2023-10-04 07:30:01 UTC

Post by Valmor F. de Almeida
There may have been a microcode emerge from dependencies; I don't
emerge directly. I do agree with you: if I try to recompile the working
kernel it will not boot. However I am not willing to try just right now
and having to troubleshoot a broken system.

Microcode is usually applied as an initramfs, and I think you said you
are not using any.

If you set LOCALVERSION in the kernel config, your existing kernel will
be installed with a new name, so you'll still have the working kernel to
fall back on.

--
Neil Bothwick

Taglines are like cars - You get a good one, then someone nicks it.

Peter Böhm

2023-10-04 11:30:01 UTC

Post by Valmor F. de Almeida
-> cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 158
stepping : 9
microcode : 0x84

At the moment this is our actual microcode for Intel CPUs:

sys-firmware/intel-microcode-20230808_p20230804

If you look into:

https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/blob/main/
releasenote.md

You will see that version f4 is actual for your CPU (dec 6-158-9 is hex
6-9e-9) instead your version 84.

KBL-G/H/S/X/E3 B0 06-9e-09/2a 000000f2 000000f4 Core
Gen7; Xeon E3 v6

What you can always do:

1. Boot with our GentooMinimalCD (so you have the new/actual microcode active
on your CPU)
2. Chroot into your system (like you do with a new installation)
3. Re-emerge all packages with "emerge -e ..."

After this, if you boot your kernel-6.1.53-r1 you should have no problem.

Valmor de Almeida

2023-10-07 06:30:02 UTC

Post by Valmor F. de Almeida
-> cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 158
stepping : 9
microcode : 0x84

sys-firmware/intel-microcode-20230808_p20230804
https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/blob/main/
releasenote.md
You will see that version f4 is actual for your CPU (dec 6-158-9 is hex
6-9e-9) instead your version 84.
KBL-G/H/S/X/E3 B0 06-9e-09/2a 000000f2 000000f4 Core
Gen7; Xeon E3 v6
1. Boot with our GentooMinimalCD (so you have the new/actual microcode active
on your CPU)
2. Chroot into your system (like you do with a new installation)
3. Re-emerge all packages with "emerge -e ..."
After this, if you boot your kernel-6.1.53-r1 you should have no problem.

I tried to chroot in the installation using the latest GentooMinimalCD
(it does run the linux-6.1.53-gentoo-r1 kernel) but I am unable to
mount the root directory.

-> lsblk
[snip]
nvme0n1 259:0 0 9453.9G 0 disk
|--nvme0n1p1 259.0 0 2M 0 part
|--nvme0n1p2. 259.0 0 128M 0 part
|--nvme0n1p3 259.0 0 1.6G 0 part
|--nvme0n1p4 259.0 0 952.2G 0 part

-> swapon /dev/nvme0n1p3
-> mount /dev/nvme0n1p4 /mnt/gentoo
mout: /mnt/gentoo: can't read superblock on /dev/nvme0n1p4.
dmesg(1) may have more information after failed mount system call.

-> dmesg

[snip]
nvme nvme0: controller is down; will reset: ....
nvme nvme0: Does your device have a faulty power saving mode enabled?
[snip]
EXT4-fs (nvme0n1p4): unable to read superblock

I am still able to boot with the old kernel 6-1-41-gentoo.
Does this mean I may have a hardware problem with the NVMe drive?

Thanks for inputs.
--
Valmor

Peter Böhm

2023-10-07 12:10:05 UTC

Post by Valmor de Almeida
-> mount /dev/nvme0n1p4 /mnt/gentoo
mout: /mnt/gentoo: can't read superblock on /dev/nvme0n1p4.

Maybe we need to dig a little deeper. Maybe you are using LVM (or have an
encrypted root partition; or both).

Post by Valmor de Almeida
-> dmesg
[snip]
nvme nvme0: controller is down; will reset: ....
nvme nvme0: Does your device have a faulty power saving mode enabled?
[snip]
EXT4-fs (nvme0n1p4): unable to read superblock
I am still able to boot with the old kernel 6-1-41-gentoo.
Does this mean I may have a hardware problem with the NVMe drive?

I dont think you have a hardware problem because with 6.41 you have no
problems.

For me the following questions would arise:

a) How did you install the kernel 6.1.41 ? (genkernel; dist-kernel; manual
approach)
b) Did you make any kernel configuration changes in your 6.1.41 ?
c) Do you use a bootmanager ? (which ?; grub, refind ?)
d) How did you update to 6.1.53 ? (every step)

To better understand your system, I would look at the output of
"parted -l" (complete),
"dmesg" (complete),
"lspci -k" and
"emerge --info"
(after you have booted 6.1.41).

To clarify all this here via the mailing list might be difficult; therefore I
would suggest to create a thread in our Gentoo support forum:

https://forums.gentoo.org/

(I am there also; but we have also a lot of great experts there)

If you create a thread in our forum, then please use wgetpaste for big files:
https://wiki.gentoo.org/wiki/Wgetpaste

Greetings,
Peter

Nuno Silva

2023-10-07 12:40:01 UTC

Post by Valmor de Almeida
-> mount /dev/nvme0n1p4 /mnt/gentoo
mout: /mnt/gentoo: can't read superblock on /dev/nvme0n1p4.

Maybe we need to dig a little deeper. Maybe you are using LVM (or have an
encrypted root partition; or both).

I dont think you have a hardware problem because with 6.41 you have no
problems.
a) How did you install the kernel 6.1.41 ? (genkernel; dist-kernel; manual
approach)
b) Did you make any kernel configuration changes in your 6.1.41 ?
c) Do you use a bootmanager ? (which ?; grub, refind ?)
d) How did you update to 6.1.53 ? (every step)
To better understand your system, I would look at the output of
"parted -l" (complete),
"dmesg" (complete),
"lspci -k" and
"emerge --info"
(after you have booted 6.1.41).
To clarify all this here via the mailing list might be difficult; therefore I
https://forums.gentoo.org/
(I am there also; but we have also a lot of great experts there)
https://wiki.gentoo.org/wiki/Wgetpaste

I'm not sure it's fair to say it'll be easier in the webforum. For the
files, if you go the wgetpaste/"pastebin" route, you can add addresses
to e-mail messages too (although that creates the inconvenience of not
being in the message and requiring a separate fetch).

Does the gentoo webforum actually support *threads*, and not just
*topics*? A lot of webforum software packages seem to support only the
concept of topics, which would mean conversations are actually easier to
navigate via mailing list or network news (i.e. gmane).

It is worth it mentioning the issue there, though, as somebody there who
is not reading this list might know what is going on.

--
Nuno Silva

Peter Böhm

2023-10-07 13:00:01 UTC

I'm not sure it's fair to say it'll be easier in the webforum. [...]

I think everything has its advantages and disadvantages. I suggested the forum
because there are also some Gentoo developers active there. If a problem
requires several different experts (because I don't know everything either)
it's certainly faster and easier there than here. I would also like to
emphasize that it was only a suggestion.

It is worth it mentioning the issue there, though, as somebody there who
is not reading this list might know what is going on.

I am one of the moderators of the forum and therefore read every new thread

;-)

Dale

2023-10-07 16:10:01 UTC

I'm not sure it's fair to say it'll be easier in the webforum. [...]

It is worth it mentioning the issue there, though, as somebody there who
is not reading this list might know what is going on.

I am one of the moderators of the forum and therefore read every new thread
;-)

I haven't been following all of this thread but I recently built a
kernel 6.1.55 and it boots fine but no GUI. I made sure the symlink is
pointing to the right kernel, re-emerged nvidia-drivers and such but
still no joy. If a thread is started there, could someone post a link
here? I'd like to follow this to see if anything helps me with my
issue, although it is different, it may have a similar solution.

Dale

:-) :-)

Mark Knecht

2023-10-07 16:10:01 UTC

I'm not sure it's fair to say it'll be easier in the webforum. [...]

It is worth it mentioning the issue there, though, as somebody there who
is not reading this list might know what is going on.

I am one of the moderators of the forum and therefore read every new thread
;-)

Have you triple checked the NVidia site to make sure your
new kernel is actually supported by the driver you're building.

Sometimes they are delayed by a bit.

HTH,
Mark

Dale

2023-10-07 16:20:02 UTC

I'm not sure it's fair to say it'll be easier in the webforum. [...]

I think everything has its advantages and disadvantages. I

suggested the forum

Post by Peter BÃ¶hm
because there are also some Gentoo developers active there. If a

problem

Post by Peter BÃ¶hm
requires several different experts (because I don't know

everything either)

Post by Peter BÃ¶hm
it's certainly faster and easier there than here. I would also like to
emphasize that it was only a suggestion.

It is worth it mentioning the issue there, though, as somebody

there who

is not reading this list might know what is going on.

I am one of the moderators of the forum and therefore read every

new thread

Post by Peter BÃ¶hm
;-)

I haven't been following all of this thread but I recently built a
kernel 6.1.55 and it boots fine but no GUI.Â I made sure the symlink is
pointing to the right kernel, re-emerged nvidia-drivers and such but
still no joy.Â If a thread is started there, could someone post a link
here?Â I'd like to follow this to see if anything helps me with my
issue, although it is different, it may have a similar solution.
Dale
:-) Â :-)

Have you triple checked the NVidia site to make sure your
new kernel is actually supported by the driver you're building.
Sometimes they are delayed by a bit.
HTH,
Mark

My first kernel, emerge prechecks warned that it wasn't and might or
might not work.Â So, I went back to a version that it claims is
supported.Â I have to say tho, the nvidia site wasn't very helpful.Â The
messages when emerging nvidia-drivers was a lot more help.Â That said,
it does load the drivers.Â They are listed with lsmod.Â Given the kernel
is on my main rig, it's harder to test since I don't reboot often.Â At
some point, I may start a thread about it if a update or something
doesn't fix it.Â After all, current kernel is working just fine.Â

I just thought it worthwhile to follow this problem since it might help
or give clues on things to look into.Â Might be related, might not.Â

Dale

:-)Â :-)Â

Mark Knecht

2023-10-07 18:50:02 UTC

On Sat, Oct 7, 2023 at 9:18â¯AM Dale <***@gmail.com> wrote:
<SNIP>

Post by Dale
My first kernel, emerge prechecks warned that it wasn't and might or

might not work. So, I went back to a version that it claims is supported.
I have to say tho, the nvidia site wasn't very helpful. The messages when
emerging nvidia-drivers was a lot more help. That said, it does load the
drivers. They are listed with lsmod. Given the kernel is on my main rig,
it's harder to test since I don't reboot often. At some point, I may start
a thread about it if a update or something doesn't fix it. After all,
current kernel is working just fine.

Post by Dale
I just thought it worthwhile to follow this problem since it might help

or give clues on things to look into. Might be related, might not.
Hi Dale,

So assuming I understand your situation, you built nvidia-drivers to work
with
6.1.55 - but got no GUI. The build told you it might not work and it didn't.

You went back to an earlier kernel and it does work.

I don't think you shared what NVidia card is in this machine, and
I don't think you shared what version of NVidia drivers worked
with the old kernel and what version didn't work with the new kernel?

- Mark

Dale

2023-10-07 19:00:01 UTC

Post by Mark Knecht
<SNIP>

Post by Dale
My first kernel, emerge prechecks warned that it wasn't and might or

might not work.Â So, I went back to a version that it claims is
supported.Â I have to say tho, the nvidia site wasn't very helpful.Â
The messages when emerging nvidia-drivers was a lot more help.Â That
said, it does load the drivers.Â They are listed with lsmod.Â Given
the kernel is on my main rig, it's harder to test since I don't reboot
often.Â At some point, I may start a thread about it if a update or
something doesn't fix it.Â After all, current kernel is working just
fine.

Post by Dale
I just thought it worthwhile to follow this problem since it might

help or give clues on things to look into.Â Might be related, might not.
Hi Dale,
So assuming I understand your situation, you built nvidia-drivers to
work withÂ
6.1.55 - but got no GUI. The build told you it might not work and it didn't.
You went back to an earlier kernel and it does work.
I don't think you shared what NVidia card is in this machine, and
I don't think you shared what version of NVidia drivers worked
with the old kernel and what version didn't work with the new kernel?
- Mark

Actually, I went to the latest version in the tree, 6.5, and got the
message it likely won't work.Â I went back to a version the message said
should work, 6.1.Â It still doesn't work.Â It may be the driver, it may
not be.Â I could have left something new out in the kernel that it needs
but I didn't know it.Â It could be any number of things.Â At some point
when I got time to mess with it, I'm going to boot into the kernel that
doesn't work and save the log files, sddm, Xorg etc.Â Then maybe it will
have a clue as to the problem.Â

Until then I just want to sort of follow this to see if it might help
with my problem.Â If not, oh well.Â

Dale

:-)Â :-)Â

P. S.Â My pack of drives arrived early.Â Running SMART test on them
now.Â :-D

Neil Bothwick

2023-10-08 10:40:01 UTC

Post by Dale
Actually, I went to the latest version in the tree, 6.5, and got the
message it likely won't work.Â I went back to a version the message said
should work, 6.1.Â It still doesn't work.Â It may be the driver, it may
not be.Â

What does /var/log/Xorg.0.log tell you?

--
Neil Bothwick

What do you have when you have six lawyers buried up to their necks in
sand? Not enough sand.

Valmor F. de Almeida

2023-10-07 16:20:01 UTC

Post by Valmor de Almeida
-> mount /dev/nvme0n1p4 /mnt/gentoo
mout: /mnt/gentoo: can't read superblock on /dev/nvme0n1p4.

Maybe we need to dig a little deeper. Maybe you are using LVM (or have an
encrypted root partition; or both).

No LVM, and no encryption.

I dont think you have a hardware problem because with 6.41 you have no
problems.

It seems like it:

-> smartctl --health /dev/nvme0n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.41-gentoo] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Post by Peter BÃ¶hm
a) How did you install the kernel 6.1.41 ? (genkernel; dist-kernel; manual
approach)

manual:
1) portage sync
2) make oldconfig in the new kernel directory for linux-6.1.53-gentoo-r1

• Mitigations for speculative execution vulnerabilities
*
Mitigations for speculative execution vulnerabilities
(SPECULATION_MITIGATIONS) [Y/n/?] y
Remove the kernel mapping in user mode (PAGE_TABLE_ISOLATION) [Y/n/?] y
Avoid speculative indirect branches in kernel (RETPOLINE) [Y/n/?] y
Enable return-thunks (RETHUNK) [Y/n/?] y
Enable UNRET on kernel entry (CPU_UNRET_ENTRY) [Y/n/?] y
Enable IBPB on kernel entry (CPU_IBPB_ENTRY) [Y/n/?] y
Enable IBRS on kernel entry (CPU_IBRS_ENTRY) [Y/n/?] y

Up to here default is picked automatically; I don't have a choice.
Next, I have tried with and without accepting the NEW features.

Mitigate speculative RAS overflow on AMD (CPU_SRSO) [Y/n/?] (NEW)
Mitigate Straight-Line-Speculation (SLS) [N/y/?] n
Force GDS Mitigation (GDS_FORCE_MITIGATION) [N/y/?] (NEW)

3) make menuconfig
4) make && make modules_install
5) make install
6) grub-mkconfig -o /boot/grub/grub.cfg
7) reboot

Post by Peter BÃ¶hm
b) Did you make any kernel configuration changes in your 6.1.41 ?

no; I use what is in /boot

-> ls /boot/
config-6.1.41-gentoo grub/ System.map-6.1.41-gentoo
vmlinuz-6.1.41-gentoo
config-6.1.53-gentoo-r1 lost+found/ System.map-6.1.53-gentoo-r1
vmlinuz-6.1.53-gentoo-r1

Post by Peter BÃ¶hm
c) Do you use a bootmanager ? (which ?; grub, refind ?)

I use grub-2

* sys-boot/grub
Latest version available: 2.06-r9
Latest version installed: 2.06-r9

Post by Peter BÃ¶hm
d) How did you update to 6.1.53 ? (every step)

-> revdep-rebuild --ignore
-> emerge --depclean
-> eclean distfiles
-> eclean packages
-> emerge @preserved-rebuild
-> emerge --sync
-> etc-update
-> env-update
(run a script to source profile)
-> emerge -vp --update --newuse --deep --tree --with-bdeps=y @world
check conflicts etc., fetch files, then run without -vp

Post by Peter BÃ¶hm
To better understand your system, I would look at the output of
"parted -l" (complete),
"dmesg" (complete),
"lspci -k" and
"emerge --info"
(after you have booted 6.1.41).
To clarify all this here via the mailing list might be difficult; therefore I

I will give it shot.

Post by Peter BÃ¶hm
https://forums.gentoo.org/
(I am there also; but we have also a lot of great experts there)
https://wiki.gentoo.org/wiki/Wgetpaste
Greetings,
Peter

Thanks,
--
Valmor

Valmor F. de Almeida

2023-10-07 17:20:02 UTC

[snip]

Post by Peter BÃ¶hm
a) How did you install the kernel 6.1.41 ? (genkernel; dist-kernel; manual
approach)

Sorry I described the install for the 6.1.53-gentoo-r1 kernel. The
install for 6.1.41 was similar but I used the config file from the
earlier kernel which is 6.1.38 to start make oldconfig.

--
Valmor

Post by Valmor F. de Almeida
1) portage sync
2) make oldconfig in the new kernel directory for linux-6.1.53-gentoo-r1
• Mitigations for speculative execution vulnerabilities
*
Mitigations for speculative execution vulnerabilities
(SPECULATION_MITIGATIONS) [Y/n/?] y
Remove the kernel mapping in user mode (PAGE_TABLE_ISOLATION) [Y/n/?] y
Avoid speculative indirect branches in kernel (RETPOLINE) [Y/n/?] y
Enable return-thunks (RETHUNK) [Y/n/?] y
Enable UNRET on kernel entry (CPU_UNRET_ENTRY) [Y/n/?] y
Enable IBPB on kernel entry (CPU_IBPB_ENTRY) [Y/n/?] y
Enable IBRS on kernel entry (CPU_IBRS_ENTRY) [Y/n/?] y
Up to here default is picked automatically; I don't have a choice.
Next, I have tried with and without accepting the NEW features.
Mitigate speculative RAS overflow on AMD (CPU_SRSO) [Y/n/?] (NEW)
Mitigate Straight-Line-Speculation (SLS) [N/y/?] n
Force GDS Mitigation (GDS_FORCE_MITIGATION) [N/y/?] (NEW)
3) make menuconfig
4) make && make modules_install
5) make install
6) grub-mkconfig -o /boot/grub/grub.cfg
7) reboot

Post by Peter BÃ¶hm
b) Did you make any kernel configuration changes in your 6.1.41 ?

no; I use what is in /boot
-> ls /boot/
config-6.1.41-gentoo grub/ System.map-6.1.41-gentoo
vmlinuz-6.1.41-gentoo
config-6.1.53-gentoo-r1 lost+found/ System.map-6.1.53-gentoo-r1
vmlinuz-6.1.53-gentoo-r1

Post by Peter BÃ¶hm
c) Do you use a bootmanager ? (which ?; grub, refind ?)

I use grub-2
* sys-boot/grub
Latest version available: 2.06-r9
Latest version installed: 2.06-r9

Post by Peter BÃ¶hm
d) How did you update to 6.1.53 ? (every step)

-> revdep-rebuild --ignore
-> emerge --depclean
-> eclean distfiles
-> eclean packages
-> emerge --sync
-> etc-update
-> env-update
(run a script to source profile)
check conflicts etc., fetch files, then run without -vp

I will give it shot.

Post by Peter BÃ¶hm
https://forums.gentoo.org/
(I am there also; but we have also a lot of great experts there)
https://wiki.gentoo.org/wiki/Wgetpaste
Greetings,
Peter

Thanks,
--
Valmor

Neil Bothwick

2023-10-08 10:40:01 UTC

Post by Valmor F. de Almeida
2) make oldconfig in the new kernel directory for linux-6.1.53-gentoo-r1
â¢ Mitigations for speculative execution vulnerabilities
*
Mitigations for speculative execution vulnerabilities
(SPECULATION_MITIGATIONS) [Y/n/?] y
Remove the kernel mapping in user mode (PAGE_TABLE_ISOLATION)
[Y/n/?] y Avoid speculative indirect branches in kernel (RETPOLINE)
[Y/n/?] y Enable return-thunks (RETHUNK) [Y/n/?] y
Enable UNRET on kernel entry (CPU_UNRET_ENTRY) [Y/n/?] y
Enable IBPB on kernel entry (CPU_IBPB_ENTRY) [Y/n/?] y
Enable IBRS on kernel entry (CPU_IBRS_ENTRY) [Y/n/?] y
Up to here default is picked automatically; I don't have a choice.

Those are settings you already have in your existing confid, shown to
give context to the following NEW settings.

You can change them, but with make config, not oldconfig.

--
Neil Bothwick

Love is grand. Divorce is a few grand more.

Valmor F. de Almeida

2023-11-11 04:20:01 UTC

Post by Neil Bothwick

Post by Valmor F. de Almeida
2) make oldconfig in the new kernel directory for linux-6.1.53-gentoo-r1
• Mitigations for speculative execution vulnerabilities
*
Mitigations for speculative execution vulnerabilities
(SPECULATION_MITIGATIONS) [Y/n/?] y
Remove the kernel mapping in user mode (PAGE_TABLE_ISOLATION)
[Y/n/?] y Avoid speculative indirect branches in kernel (RETPOLINE)
[Y/n/?] y Enable return-thunks (RETHUNK) [Y/n/?] y
Enable UNRET on kernel entry (CPU_UNRET_ENTRY) [Y/n/?] y
Enable IBPB on kernel entry (CPU_IBPB_ENTRY) [Y/n/?] y
Enable IBRS on kernel entry (CPU_IBRS_ENTRY) [Y/n/?] y
Up to here default is picked automatically; I don't have a choice.

Those are settings you already have in your existing confid, shown to
give context to the following NEW settings.
You can change them, but with make config, not oldconfig.

After some distractions...
For the record. This problem solved itself with the 6.1.57-gentoo
kernel. Just used the config from 6.1.41-gentoo in make oldconfig.
Thanks for the help.
--
Valmor

Valmor F. de Almeida

2023-10-02 07:20:02 UTC

Post by Valmor F. de Almeida
Hello,
For a while now (3 weeks or so) I have been upgrading the linux kernel
on a Dell XPS laptop starting from 6.1.41-gentoo (which is my current
working kernel) to 6.1.53-gentoo-r1. No kernel I have built since is
able to boot. I have been following the same method for many years: make
oldconfig, etc...
[snip]
* INIT: Entering runlevel: 3
[snip]
* Starting cronie ...
* Starting DHCP Client Daemon ...
* Starting laptop_mode ...
* Mounting network filesystems ...
/etc/init.d/netmount: line 45 /lib/rc/bin/ewend: Input/output error
/lib/rc/sh/rc-cgroup.sh: line 184: rmdir: command not found
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"

Can you show /etc/fstab and the console-log for the entire boot? Seems /sbin
is not readable. You sure you have the kernel modules loaded? Are you using
an initramfs? If so, does that build without errors ?

The input/output error – to me – indicates a hardware problem. When you
mounted the FS by hand, can you read ewend? For instance with md5sum.

Right it boots with 6.1.41. I started again from the config of 6.1.41
and unset SRSO:

-> diff linux/.config /boot/config-6.1.41-gentoo
3c3
< # Linux/x86 6.1.53-gentoo-r1 Kernel Configuration
---

Post by HÃ¥kon Alstadheim
# Linux/x86 6.1.41-gentoo Kernel Configuration

5c5
< CONFIG_CC_VERSION_TEXT="gcc (Gentoo 13.2.1_p20230826 p7) 13.2.1 20230826"
---

Post by HÃ¥kon Alstadheim
CONFIG_CC_VERSION_TEXT="gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526"

7c7
< CONFIG_GCC_VERSION=130201
---

Post by HÃ¥kon Alstadheim
CONFIG_GCC_VERSION=120301

455d454
< # CONFIG_CPU_SRSO is not set
457d455
< # CONFIG_GDS_FORCE_MITIGATION is not set
646d643
< CONFIG_ARCH_HAS_CPU_FINALIZE_INIT=y
3136d3132
< CONFIG_VIDEO_V4L2_SUBDEV_API=y
3139,3140d3134
< CONFIG_V4L2_FWNODE=m
< CONFIG_V4L2_ASYNC=m
3236c3230,3233
< CONFIG_VIDEO_CAMERA_SENSOR=y
---

Post by HÃ¥kon Alstadheim
#
# Camera sensor devices
#

3295a3293

Post by HÃ¥kon Alstadheim
# end of Camera sensor devices

Still no luck; kernel build has no errors. Boot hangs.
Thanks for the replies. Maybe 6.2 will not have this problem for my system.
--
Valmor

Lee

2023-10-02 08:50:01 UTC

I found this:

https://www.reddit.com/r/Gentoo/comments/168qrbx/gentoosources_6146_kernel_reports_io_error/

Lee ð

Post by Valmor F. de Almeida
Hello,
For a while now (3 weeks or so) I have been upgrading the linux kernel
on a Dell XPS laptop starting from 6.1.41-gentoo (which is my current
working kernel) to 6.1.53-gentoo-r1. No kernel I have built since is
able to boot. I have been following the same method for many years: make
oldconfig, etc...
[snip]
* INIT: Entering runlevel: 3
[snip]
* Starting cronie ...
* Starting DHCP Client Daemon ...
* Starting laptop_mode ...
* Mounting network filesystems ...
/etc/init.d/netmount: line 45 /lib/rc/bin/ewend: Input/output error
/lib/rc/sh/rc-cgroup.sh: line 184: rmdir: command not found
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"

Can you show /etc/fstab and the console-log for the entire boot? Seems /sbin
is not readable. You sure you have the kernel modules loaded? Are you using
an initramfs? If so, does that build without errors ?

The input/output error â to me â indicates a hardware problem. When you
mounted the FS by hand, can you read ewend? For instance with md5sum.

Right it boots with 6.1.41. I started again from the config of 6.1.41
-> diff linux/.config /boot/config-6.1.41-gentoo
3c3
< # Linux/x86 6.1.53-gentoo-r1 Kernel Configuration
---

Post by HÃ¥kon Alstadheim
# Linux/x86 6.1.41-gentoo Kernel Configuration

5c5
< CONFIG_CC_VERSION_TEXT="gcc (Gentoo 13.2.1_p20230826 p7) 13.2.1 20230826"
---

Post by HÃ¥kon Alstadheim
CONFIG_CC_VERSION_TEXT="gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1

20230526"
7c7
< CONFIG_GCC_VERSION=130201
---

Post by HÃ¥kon Alstadheim
CONFIG_GCC_VERSION=120301

Post by HÃ¥kon Alstadheim
#
# Camera sensor devices
#

3295a3293

Post by HÃ¥kon Alstadheim
# end of Camera sensor devices

Still no luck; kernel build has no errors. Boot hangs.
Thanks for the replies. Maybe 6.2 will not have this problem for my system.
--
Valmor

Valmor F. de Almeida

2023-10-04 03:40:01 UTC

https://www.reddit.com/r/Gentoo/comments/168qrbx/gentoosources_6146_kernel_reports_io_error/ <https://www.reddit.com/r/Gentoo/comments/168qrbx/gentoosources_6146_kernel_reports_io_error/>
Lee 😎

This is probably it.
Thanks.
--
Valmor

Post by Valmor F. de Almeida
Hello,
For a while now (3 weeks or so) I have been upgrading the

linux kernel

Post by Valmor F. de Almeida
on a Dell XPS laptop starting from 6.1.41-gentoo (which is my

current

Post by Valmor F. de Almeida
working kernel) to 6.1.53-gentoo-r1. No kernel I have built

since is

Post by Valmor F. de Almeida
able to boot. I have been following the same method for many
make
oldconfig, etc...
[snip]
* INIT: Entering runlevel: 3
[snip]
* Starting cronie ...
* Starting DHCP Client Daemon ...
* Starting laptop_mode ...
* Mounting network filesystems ...
/etc/init.d/netmount: line 45 /lib/rc/bin/ewend: Input/output

error

Post by Valmor F. de Almeida
/lib/rc/sh/rc-cgroup.sh: line 184: rmdir: command not found
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"
INIT: cannot execute "/sbin/agetty"

Can you show /etc/fstab and the console-log for the entire boot?
Seems /sbin
is not readable. You sure you have the kernel modules loaded?

Are you

Post by HÃ¥kon Alstadheim
using
an initramfs? If so, does that build without errors ?

The input/output error – to me – indicates a hardware problem.

When you