[gentoo-user] emerge times blown out

[Adam Carter]

I have three systems (all ~arch) and the emerge times have blown out on all
of them across all packages. Worst example appears to be;

Fri Dec 23 13:11:44 2022 >>> net-libs/webkit-gtk-2.38.3-r410
merge time: 37 minutes and 8 seconds.

Fri Dec 23 13:43:08 2022 >>> net-libs/webkit-gtk-2.38.3
merge time: 31 minutes and 24 seconds.

Sat Feb 4 21:16:40 2023 >>> net-libs/webkit-gtk-2.38.4-r410
merge time: 6 hours, 53 minutes and 28 seconds.

Sun Feb 5 04:17:12 2023 >>> net-libs/webkit-gtk-2.38.4
merge time: 7 hours and 32 seconds.

Is anyone else seeing this?

[Dale]

Post by Adam Carter
I have three systems (all ~arch) and the emerge times have blown out
on all of them across all packages. Worst example appears to be;
    Fri Dec 23 13:11:44 2022 >>> net-libs/webkit-gtk-2.38.3-r410
       merge time: 37 minutes and 8 seconds.
     Fri Dec 23 13:43:08 2022 >>> net-libs/webkit-gtk-2.38.3
       merge time: 31 minutes and 24 seconds.
     Sat Feb  4 21:16:40 2023 >>> net-libs/webkit-gtk-2.38.4-r410
       merge time: 6 hours, 53 minutes and 28 seconds.
     Sun Feb  5 04:17:12 2023 >>> net-libs/webkit-gtk-2.38.4
       merge time: 7 hours and 32 seconds.
Is anyone else seeing this?

A lot of this will obviously depend on CPU speed, number of
cores/threads, amount of memory and also, is it compiling on its own or
are other packages also being compiled and taking up CPU time or other
things using CPU time as well.  Here, AMD CPU at 4GHz with 8 cores. 
32GBs of memory and most packages built on tmpfs, except large
packages.  This is a recent few examples. 


     Sat Aug  6 06:34:31 2022 >>> net-libs/webkit-gtk-2.36.5-r1
       merge time: 1 hour, 39 minutes and 13 seconds.

     Sun Aug 21 16:04:23 2022 >>> net-libs/webkit-gtk-2.36.6
       merge time: 3 hours, 41 minutes and 44 seconds.

     Sat Sep  3 22:29:36 2022 >>> net-libs/webkit-gtk-2.36.7
       merge time: 2 hours, 41 minutes and 7 seconds.



I left out binary emerges and such.  Oldest at top, newest at bottom.  I
suspect for the middle and most likely the bottom one, there was another
package compiling as well.  It's possible the top one was either on its
own part or all of the time. 

It's really hard to compare these things.  Even if our rigs are close in
speed/power/whatever, there can still be a lot of things that affect the
compile time.  I run KDE, watch TV from computer, have torrent software
that runs basically 24/7 plus other stuff running.  If you for example
use Fluxbox and have little else running, even with the same CPU and
memory, compile times will vary. 

Does that info help? 

Dale

:-)  :-) 

[Adam Carter]

Post by Dale
Does that info help?

My reason for asking is that i'm seeing this across multiple systems, 2
AMD, 1 Intel, who's configuration hasn't really changed and while there is
some variance there has been a step change late December / early January.
Another example

Sat Nov 26 14:34:50 2022 >>> sys-apps/systemd-252.2
merge time: 2 minutes and 19 seconds.

Sat Dec 10 20:59:29 2022 >>> sys-apps/systemd-252.3
merge time: 1 minute and 54 seconds.

Wed Dec 14 13:56:52 2022 >>> sys-apps/systemd-252.3
merge time: 2 minutes and 56 seconds.

Wed Dec 21 20:08:36 2022 >>> sys-apps/systemd-252.4
merge time: 3 minutes and 7 seconds.

Tue Jan 3 22:29:43 2023 >>> sys-apps/systemd-252.4
merge time: 12 minutes and 42 seconds.

Thu Jan 12 14:56:32 2023 >>> sys-apps/systemd-252.4-r1
merge time: 22 minutes and 12 seconds.

Sat Jan 21 12:00:06 2023 >>> sys-apps/systemd-252.4-r1
merge time: 12 minutes and 3 seconds.

Mon Jan 30 15:41:44 2023 >>> sys-apps/systemd-252.5
merge time: 21 minutes and 45 seconds.

Fri Feb 17 21:18:21 2023 >>> sys-apps/systemd-252.6
merge time: 22 minutes and 18 seconds.

[Dale]

Does that info help? 
My reason for asking is that i'm seeing this across multiple systems,
2 AMD, 1 Intel, who's configuration hasn't really changed and while
there is some variance there has been a step change late December /
early January. Another example
    Sat Nov 26 14:34:50 2022 >>> sys-apps/systemd-252.2
       merge time: 2 minutes and 19 seconds.
     Sat Dec 10 20:59:29 2022 >>> sys-apps/systemd-252.3
       merge time: 1 minute and 54 seconds.
     Wed Dec 14 13:56:52 2022 >>> sys-apps/systemd-252.3
       merge time: 2 minutes and 56 seconds.
     Wed Dec 21 20:08:36 2022 >>> sys-apps/systemd-252.4
       merge time: 3 minutes and 7 seconds.
     Tue Jan  3 22:29:43 2023 >>> sys-apps/systemd-252.4
       merge time: 12 minutes and 42 seconds.
     Thu Jan 12 14:56:32 2023 >>> sys-apps/systemd-252.4-r1
       merge time: 22 minutes and 12 seconds.
     Sat Jan 21 12:00:06 2023 >>> sys-apps/systemd-252.4-r1
       merge time: 12 minutes and 3 seconds.
     Mon Jan 30 15:41:44 2023 >>> sys-apps/systemd-252.5
       merge time: 21 minutes and 45 seconds.
     Fri Feb 17 21:18:21 2023 >>> sys-apps/systemd-252.6
       merge time: 22 minutes and 18 seconds.

The ones I listed before also jumped in compile times.  As I said tho, I
don't know if other things compiling affected that time.  Still, it does
seem to have increased but I remember when I was on my old single core
rig with just a few GBs of memory.  As time goes by, software gets
bigger therefore takes longer to compile.  Yours from the 4th to the 6th
in the list sure does increase.  That's 8 to 10 times longer roughly. 
That's a large difference.  A true test, emerge something interesting
all by itself.  See what it comes closest to, the old times or the newer
and longer times. 

I suspect this is changes in features of software and could even be
related to gcc or some other tool compiling uses.  It is a interesting
jump.  I don't think you are alone in this.  Maybe someone else will
post their info.  For those interested, genlop -t <package name> is how
to get this info. 

BTW, I don't use systemd so I can't list mine.  ;-)

Dale

:-)  :-)

[Alan J. Wylie]

Post by Adam Carter
My reason for asking is that i'm seeing this across multiple systems, 2
AMD, 1 Intel, who's configuration hasn't really changed and while there is
some variance there has been a step change late December / early January.
Another example
Sat Nov 26 14:34:50 2022 >>> sys-apps/systemd-252.2
merge time: 2 minutes and 19 seconds.
Sat Dec 10 20:59:29 2022 >>> sys-apps/systemd-252.3
merge time: 1 minute and 54 seconds.
Wed Dec 14 13:56:52 2022 >>> sys-apps/systemd-252.3
merge time: 2 minutes and 56 seconds.
Wed Dec 21 20:08:36 2022 >>> sys-apps/systemd-252.4
merge time: 3 minutes and 7 seconds.
Tue Jan 3 22:29:43 2023 >>> sys-apps/systemd-252.4
merge time: 12 minutes and 42 seconds.
Thu Jan 12 14:56:32 2023 >>> sys-apps/systemd-252.4-r1
merge time: 22 minutes and 12 seconds.
Sat Jan 21 12:00:06 2023 >>> sys-apps/systemd-252.4-r1
merge time: 12 minutes and 3 seconds.
Mon Jan 30 15:41:44 2023 >>> sys-apps/systemd-252.5
merge time: 21 minutes and 45 seconds.
Fri Feb 17 21:18:21 2023 >>> sys-apps/systemd-252.6
merge time: 22 minutes and 18 seconds.

All my three systems are fine

AMD FX(tm)-8350 Eight-Core Processor, 16GiB

Sun Oct 2 13:07:33 2022 >>> sys-apps/systemd-251.4
merge time: 3 minutes and 35 seconds.

Fri Nov 25 08:06:10 2022 >>> sys-apps/systemd-251.7
merge time: 3 minutes and 24 seconds.

Sat Dec 10 10:11:16 2022 >>> sys-apps/systemd-251.8
merge time: 3 minutes and 19 seconds.

Tue Jan 3 08:43:49 2023 >>> sys-apps/systemd-252.4
merge time: 3 minutes and 22 seconds.

Sat Jan 28 06:28:16 2023 >>> sys-apps/systemd-252.4-r1
merge time: 5 minutes and 24 seconds.

AMD FX(tm)-4300 Quad-Core Processor, 16GiB

Sun Oct 2 13:42:48 2022 >>> sys-apps/systemd-251.4
merge time: 5 minutes and 32 seconds.

Fri Nov 25 08:11:27 2022 >>> sys-apps/systemd-251.7
merge time: 5 minutes and 40 seconds.

Sat Dec 10 12:31:38 2022 >>> sys-apps/systemd-251.8
merge time: 5 minutes and 36 seconds.

Tue Jan 3 08:47:13 2023 >>> sys-apps/systemd-252.4
merge time: 5 minutes and 36 seconds.

Sat Jan 28 06:28:27 2023 >>> sys-apps/systemd-252.4-r1
merge time: 6 minutes and 17 seconds.

Intel(R) Atom(TM) CPU N2800 @ 1.86GHz, 4GiB, not systemd

Wed Sep 28 08:32:50 2022 >>> net-dns/bind-9.16.33
merge time: 15 minutes and 28 seconds.

Fri Jan 20 07:50:41 2023 >>> net-dns/bind-9.16.36
merge time: 16 minutes and 23 seconds.

Thu Feb 16 07:33:46 2023 >>> net-dns/bind-9.16.37
merge time: 17 minutes and 9 seconds.

[Stefan Schmiedl]

Samstag, 18. Februar 2023 01:49:
 

I have three systems (all ~arch) and the emerge times have blown out on all of them across all packages. Worst example appears to be;
    Fri Dec 23 13:11:44 2022 >>> net-libs/webkit-gtk-2.38.3-r410
       merge time: 37 minutes and 8 seconds.
     Fri Dec 23 13:43:08 2022 >>> net-libs/webkit-gtk-2.38.3
       merge time: 31 minutes and 24 seconds.
     Sat Feb  4 21:16:40 2023 >>> net-libs/webkit-gtk-2.38.4-r410
       merge time: 6 hours, 53 minutes and 28 seconds.
     Sun Feb  5 04:17:12 2023 >>> net-libs/webkit-gtk-2.38.4
       merge time: 7 hours and 32 seconds.
Is anyone else seeing this?

When I had something like this about two years ago, the culprit was me
setting the cpufreq thingie in the kernel config to powersave or whatever
it's called ... running emerge with 800 MHz instead of 3.2 GHz produced
effects similar to yours.

s. 
 

[Grant Edwards]

Post by Stefan Schmiedl
 

I have three systems (all ~arch) and the emerge times have blown out on all of them across all packages. Worst example appears to be;
    Fri Dec 23 13:11:44 2022 >>> net-libs/webkit-gtk-2.38.3-r410
       merge time: 37 minutes and 8 seconds.
     Fri Dec 23 13:43:08 2022 >>> net-libs/webkit-gtk-2.38.3
       merge time: 31 minutes and 24 seconds.
     Sat Feb  4 21:16:40 2023 >>> net-libs/webkit-gtk-2.38.4-r410
       merge time: 6 hours, 53 minutes and 28 seconds.
     Sun Feb  5 04:17:12 2023 >>> net-libs/webkit-gtk-2.38.4
       merge time: 7 hours and 32 seconds.
Is anyone else seeing this?

When I had something like this about two years ago, the culprit was me
setting the cpufreq thingie in the kernel config to powersave or whatever
it's called ... running emerge with 800 MHz instead of 3.2 GHz produced
effects similar to yours.

The same thing happened to me recently. Somehow, I broke the load/temp
based CPU clock scaling on one of my machines, and it suddenly took 4X
as long to build things as my other machines. It was running at the
lowest clock speed possible all the time. Unfortunately, I don't
remember exactly what I did to fix it...

--
Grant

[Stefan Schmiedl]

Samstag, 18. Februar 2023 19:05:

 

Post by Grant Edwards

Post by Stefan Schmiedl

I have three systems (all ~arch) and the emerge times have blown out on all of them across all packages.

When I had something like this about two years ago, the culprit was me
setting the cpufreq thingie in the kernel config to powersave or whatever
it's called ... running emerge with 800 MHz instead of 3.2 GHz produced
effects similar to yours.

The same thing happened to me recently. Somehow, I broke the load/temp
based CPU clock scaling on one of my machines, and it suddenly took 4X
as long to build things as my other machines. It was running at the
lowest clock speed possible all the time. Unfortunately, I don't
remember exactly what I did to fix it...

CONFIG_CPU_FREQ_GOV_PERFORMANCE
CONFIG_CPU_FREQ_GOV_POWERSAVE
CONFIG_CPU_FREQ_GOV_USERSPACE
CONFIG_CPU_FREQ_GOV_ONDEMAND
CONFIG_CPU_FREQ_GOV_CONSERVATIVE
CONFIG_CPU_FREQ_GOV_SCHEDUTIL

Back when I finally found it, all of these were set to "n" but for POWERSAVE which was set to "y".
I rebuilt the kernel to use ONDEMAND and things were back to normal.

s.

[Adam Carter]

Thanks everyone for your suggestions. I've checked the frequencies of the
cores and they are scaling properly:
cpu MHz : 4024.653
cpu MHz : 4024.678
cpu MHz : 4024.639
cpu MHz : 4024.605
cpu MHz : 4024.643
etc

Will continue to pursue these lines of thought.

[Alexander Puchmayr]

Post by Adam Carter
I have three systems (all ~arch) and the emerge times have blown out on all
of them across all packages. Worst example appears to be;
Fri Dec 23 13:11:44 2022 >>> net-libs/webkit-gtk-2.38.3-r410
merge time: 37 minutes and 8 seconds.
Fri Dec 23 13:43:08 2022 >>> net-libs/webkit-gtk-2.38.3
merge time: 31 minutes and 24 seconds.
Sat Feb 4 21:16:40 2023 >>> net-libs/webkit-gtk-2.38.4-r410
merge time: 6 hours, 53 minutes and 28 seconds.
Sun Feb 5 04:17:12 2023 >>> net-libs/webkit-gtk-2.38.4
merge time: 7 hours and 32 seconds.
Is anyone else seeing this?

Could it be that some kind of Spectre mitigation is active? I just read about
some massive performance problems in Kernel 5.19+ on Skylake CPUs. Stable
gentoo kernel was upgraded to 6.1 recently, which could also be affected by
this problem.
See https://lkml.iu.edu/hypermail/linux/kernel/2209.1/02248.html

Try turning the mitigations off or revert to a 5.15.x kernel.

Alex

[Adam Carter]

Post by Alexander Puchmayr
Could it be that some kind of Spectre mitigation is active? I just read about
some massive performance problems in Kernel 5.19+ on Skylake CPUs. Stable
gentoo kernel was upgraded to 6.1 recently, which could also be affected by
this problem.
See https://lkml.iu.edu/hypermail/linux/kernel/2209.1/02248.html
Try turning the mitigations off or revert to a 5.15.x kernel.

I have one skylake and its now running 6.2 with retbleed=stuff. The others
are AMD.

I will try 5.15.x. I have .configs from that time so it will rule kernel
stuff in/out.