[gentoo-user] Emerge order not deterministic !?

[Walter Dnes]

Ongoing installation. I looked at 2 instances of
"emerge -pv x11-base/xorg-server" and the order was somewhat different.
Here are a couple of outputs, just a few seconds apart. Is this a bug
or a feature? See attachments.

[Alan McKinnon]

Post by Walter Dnes
Ongoing installation. I looked at 2 instances of
"emerge -pv x11-base/xorg-server" and the order was somewhat different.
Here are a couple of outputs, just a few seconds apart. Is this a bug
or a feature? See attachments.

Emerge order is not deterministic, especially with parallel builds. The
reason is that it does not need to be according to the dep graph - if
two packages are at the same level and do not depend on each other, then
the order they are built in does not affect the final result.
Practically all parallel processing works this way.

What is deterministic, is that if you build the same set of packages
twice and even if portage does them in different order, the binaries
produced are functionally identical

[Jörg Schaible]

Post by Alan McKinnon

Post by Walter Dnes
Ongoing installation. I looked at 2 instances of
"emerge -pv x11-base/xorg-server" and the order was somewhat different.
Here are a couple of outputs, just a few seconds apart. Is this a bug
or a feature? See attachments.

Emerge order is not deterministic, especially with parallel builds. The
reason is that it does not need to be according to the dep graph - if
two packages are at the same level and do not depend on each other, then
the order they are built in does not affect the final result.
Practically all parallel processing works this way.
What is deterministic, is that if you build the same set of packages
twice and even if portage does them in different order, the binaries
produced are functionally identical

...

Hmmm. And how can you then ever use

emerge --resume --skip-fist

if not even the first build is deterministic? I skip the first package
anyway only if the problematic package is the first one to build after
resume, but if I cannot even rely on that?

Cheers,
Jörg

[Alan McKinnon]

Post by Jörg Schaible

Post by Alan McKinnon

Post by Walter Dnes
Ongoing installation. I looked at 2 instances of
"emerge -pv x11-base/xorg-server" and the order was somewhat different.
Here are a couple of outputs, just a few seconds apart. Is this a bug
or a feature? See attachments.

Emerge order is not deterministic, especially with parallel builds. The
reason is that it does not need to be according to the dep graph - if
two packages are at the same level and do not depend on each other, then
the order they are built in does not affect the final result.
Practically all parallel processing works this way.
What is deterministic, is that if you build the same set of packages
twice and even if portage does them in different order, the binaries
produced are functionally identical

Hmmm. And how can you then ever use
emerge --resume --skip-fist
if not even the first build is deterministic? I skip the first package
anyway only if the problematic package is the first one to build after
resume, but if I cannot even rely on that?

...

Because it re-uses the previous build order, not re-generate a new one.

[Jörg Schaible]

Post by Alan McKinnon

Post by Jörg Schaible

Post by Alan McKinnon

Post by Walter Dnes
Ongoing installation. I looked at 2 instances of
"emerge -pv x11-base/xorg-server" and the order was somewhat different.
Here are a couple of outputs, just a few seconds apart. Is this a bug
or a feature? See attachments.

Emerge order is not deterministic, especially with parallel builds. The
reason is that it does not need to be according to the dep graph - if
two packages are at the same level and do not depend on each other, then
the order they are built in does not affect the final result.
Practically all parallel processing works this way.
What is deterministic, is that if you build the same set of packages
twice and even if portage does them in different order, the binaries
produced are functionally identical

Hmmm. And how can you then ever use
emerge --resume --skip-fist
if not even the first build is deterministic? I skip the first package
anyway only if the problematic package is the first one to build after
resume, but if I cannot even rely on that?

Because it re-uses the previous build order, not re-generate a new one.

...

That's simply not true. Emerge resume calculates the order again and for me
it starts often with a different package.

Cheers,
Jörg

[Neil Bothwick]

Post by Jörg Schaible

Post by Alan McKinnon

Post by Jörg Schaible
Hmmm. And how can you then ever use
emerge --resume --skip-fist
if not even the first build is deterministic? I skip the first
package anyway only if the problematic package is the first one to
build after resume, but if I cannot even rely on that?

Because it re-uses the previous build order, not re-generate a new one.

That's simply not true. Emerge resume calculates the order again and
for me it starts often with a different package.

Then use emerge --keep-going and portage will take care of skipping
failing merges for you.

[Jörg Schaible]

Post by Neil Bothwick

Post by Jörg Schaible

Post by Alan McKinnon

Post by Jörg Schaible
Hmmm. And how can you then ever use
emerge --resume --skip-fist
if not even the first build is deterministic? I skip the first
package anyway only if the problematic package is the first one to
build after resume, but if I cannot even rely on that?

Because it re-uses the previous build order, not re-generate a new one.

That's simply not true. Emerge resume calculates the order again and
for me it starts often with a different package.

Then use emerge --keep-going and portage will take care of skipping
failing merges for you.

...

Ah, no, that's not an option. It breaks for a reason. Sometimes I can ignore
that and look for it later and in this case I skip it, but normally I fix
the problem first. However, you have to take care, which package you're
actually skipping. Especially if the build order is different with resume.

Cheers,
Jörg

[Neil Bothwick]

Post by Jörg Schaible

Post by Neil Bothwick
Then use emerge --keep-going and portage will take care of skipping
failing merges for you.

Ah, no, that's not an option. It breaks for a reason. Sometimes I can
ignore that and look for it later and in this case I skip it, but
normally I fix the problem first. However, you have to take care, which
package you're actually skipping. Especially if the build order is
different with resume.

--keep-going will emerge all unaffected packages, meaning you are then
working with a much smaller list when you try to fix the problem. At
least, that's the approach that normally works for me.

--keep-going is intelligent enough to skip any packages that depend on
the failed package. That means you often end up with a package list that
is a single branch dependency tree, so the order is unlikely to change.

[Hans]

Post by Neil Bothwick

Post by Jörg Schaible

Post by Neil Bothwick
Then use emerge --keep-going and portage will take care of skipping
failing merges for you.

Ah, no, that's not an option. It breaks for a reason. Sometimes I can
ignore that and look for it later and in this case I skip it, but
normally I fix the problem first. However, you have to take care, which
package you're actually skipping. Especially if the build order is
different with resume.

--keep-going will emerge all unaffected packages, meaning you are then
working with a much smaller list when you try to fix the problem. At
least, that's the approach that normally works for me.
--keep-going is intelligent enough to skip any packages that depend on
the failed package. That means you often end up with a package list that
is a single branch dependency tree, so the order is unlikely to change.

...

I use the following commands to upgrade my Gentoo boxes:
emerge --sync
emerge --update --deep --with-bdeps=y --newuse --backtrack=300
--changed-deps=y --keep-going=y @world -va

When necessary adding, deleting or changing use flags, keywords, masks.

Followed by:
emerge --depclean
revdep-rebuild

No more problems since using this sequence unless there is a bug in a
ebuild, like the one last one in busybox ebuild.

[Alan McKinnon]

Post by Jörg Schaible

Post by Alan McKinnon

Post by Jörg Schaible

Post by Alan McKinnon

Post by Walter Dnes
Ongoing installation. I looked at 2 instances of
"emerge -pv x11-base/xorg-server" and the order was somewhat different.
Here are a couple of outputs, just a few seconds apart. Is this a bug
or a feature? See attachments.

Emerge order is not deterministic, especially with parallel builds. The
reason is that it does not need to be according to the dep graph - if
two packages are at the same level and do not depend on each other, then
the order they are built in does not affect the final result.
Practically all parallel processing works this way.
What is deterministic, is that if you build the same set of packages
twice and even if portage does them in different order, the binaries
produced are functionally identical

Hmmm. And how can you then ever use
emerge --resume --skip-fist
if not even the first build is deterministic? I skip the first package
anyway only if the problematic package is the first one to build after
resume, but if I cannot even rely on that?

Because it re-uses the previous build order, not re-generate a new one.

That's simply not true. Emerge resume calculates the order again and for me
it starts often with a different package.

...

I've never noticed that. For me --skip-first has always skipped the
correct first package (the one that previously failed).

As long as a known build failure is not in the --resume list, I don't
care what the build order is because it is irrelevant. The only time it
becomes relevant is when an ebuild has a bug such as a missing dep. But
that's a bug in the ebuild and is fixed there.

[Jörg Schaible]

[snip]

Post by Alan McKinnon

Post by Jörg Schaible

Post by Alan McKinnon

Post by Jörg Schaible
Hmmm. And how can you then ever use
emerge --resume --skip-fist
if not even the first build is deterministic? I skip the first package
anyway only if the problematic package is the first one to build after
resume, but if I cannot even rely on that?

Because it re-uses the previous build order, not re-generate a new one.

That's simply not true. Emerge resume calculates the order again and for
me it starts often with a different package.

I've never noticed that. For me --skip-first has always skipped the
correct first package (the one that previously failed).

...

That's what I always did originally also, until my build suddenly broke at
the same package again and I had to notice that it skipped a completely
different.

Post by Alan McKinnon
As long as a known build failure is not in the --resume list, I don't
care what the build order is because it is irrelevant. The only time it
becomes relevant is when an ebuild has a bug such as a missing dep. But
that's a bug in the ebuild and is fixed there.

Well, normally I don't care about the sequence either, except when skipping
the first ;-)

Cheers,
Jörg