Discussion:
[gentoo-user] how to prevent ebuild from checking for available space
(too old to reply)
Jack
2024-03-22 17:00:01 UTC
Permalink
In this case, the offending package is dev-lang/rust, but this has
happened to me previously with other packages that require a lot of
space and time to build.

The build fails for any reason.  Since it has already progressed over
half way through the build, I would like to continue the build instead
of starting from scratch.  I fix whatever caused the issue.  In this
particular case, it failed for out of space, so I added another G to the
ramdisk mounted at /var/tmp/portageg.  I issue "ebuild
path/to/package.ebuild compile" but it fails immediately because there
isn't enough free space.  The problem is that it's checking for enough
free space to do the entire build, not just to continue from where it
left off.

I've done some searching, and found one old forum post which suggests to
just temporarily remove the space check from the ebuild.

In this case, that seems to be line 236 of
/usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is
"CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last
line withing pre_build_checks().  However, if I comment out that line,
and run "ebuild path/to/ebuild manifest" another attempt to compile
still gives the same error about not enough space.

Am I commenting the wrong line?  Have I missed something about where
this check is atually done?  Is it actually possible to do what I'm
trying to do?

Thanks for any pointers or suggestions.

Jack
Neil Bothwick
2024-03-22 20:30:01 UTC
Permalink
Post by Jack
In this case, the offending package is dev-lang/rust, but this has
happened to me previously with other packages that require a lot of
space and time to build.
The build fails for any reason.  Since it has already progressed over
half way through the build, I would like to continue the build instead
of starting from scratch.  I fix whatever caused the issue.  In this
particular case, it failed for out of space, so I added another G to
the ramdisk mounted at /var/tmp/portageg.  I issue "ebuild
path/to/package.ebuild compile" but it fails immediately because there
isn't enough free space.  The problem is that it's checking for enough
free space to do the entire build, not just to continue from where it
left off.
Why not add more to the ramdisk, assuming it is a tmpfs. If it needs more
than your physical memory, it will use swap, but that won't happen
because you only need the extra space.

I generally avoid this problem by having large packages use a
PORTAGE_TMPDIR on a spinning disk. It may be slightly slower, but that
doesn't really matter on a long compile, not nearly as much as having to
restart it!
Post by Jack
I've done some searching, and found one old forum post which suggests
to just temporarily remove the space check from the ebuild.
In this case, that seems to be line 236 of
/usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is
"CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last
line withing pre_build_checks().  However, if I comment out that line,
and run "ebuild path/to/ebuild manifest" another attempt to compile
still gives the same error about not enough space.
Have you tried setting M to a smaller value immediately before that line?
--
Neil Bothwick

Religious error: (A)tone, (R)epent, (I)mmolate?
Jack
2024-03-22 20:40:01 UTC
Permalink
Post by Jack
Post by Jack
In this case, the offending package is dev-lang/rust, but this has
happened to me previously with other packages that require a lot of
space and time to build.
The build fails for any reason.  Since it has already progressed
over
Post by Jack
half way through the build, I would like to continue the build
instead
Post by Jack
of starting from scratch.  I fix whatever caused the issue.  In this
particular case, it failed for out of space, so I added another G to
the ramdisk mounted at /var/tmp/portageg.  I issue "ebuild
path/to/package.ebuild compile" but it fails immediately because
there
Post by Jack
isn't enough free space.  The problem is that it's checking for
enough
Post by Jack
free space to do the entire build, not just to continue from where
it
Post by Jack
left off.
Why not add more to the ramdisk, assuming it is a tmpfs. If it needs more
than your physical memory, it will use swap, but that won't happen
because you only need the extra space.
That's actually what I did. The problem is not how to get enough
space, it's how to resume the emerge instead of starting over, once I
have added the space.

It was initially set to 14G (out of 32G RAM) and I added 2G. I suppose
I can add another 14G, but that would only leave 4G for the system
itself. I'm not sure how well that would work, but I suppose it's
worth a try. The worst that happens is I reboot and start the emerge
from scratch.
Post by Jack
I generally avoid this problem by having large packages use a
PORTAGE_TMPDIR on a spinning disk. It may be slightly slower, but that
doesn't really matter on a long compile, not nearly as much as having to
restart it!
Post by Jack
I've done some searching, and found one old forum post which
suggests
Post by Jack
to just temporarily remove the space check from the ebuild.
In this case, that seems to be line 236 of
/usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is
"CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last
line withing pre_build_checks().  However, if I comment out that
line,
Post by Jack
and run "ebuild path/to/ebuild manifest" another attempt to compile
still gives the same error about not enough space.
Have you tried setting M to a smaller value immediately before that line?
I tried that first, and when that didn't work, I tried commenting out
the line I thought was calling the check. I'll try again, to be sure
it was a reasonable value for M.
Post by Jack
--
Neil Bothwick
Religious error: (A)tone, (R)epent, (I)mmolate?
Neil Bothwick
2024-03-23 09:30:02 UTC
Permalink
Post by Jack
Post by Neil Bothwick
Why not add more to the ramdisk, assuming it is a tmpfs. If it needs more
than your physical memory, it will use swap, but that won't happen
because you only need the extra space.
That's actually what I did. The problem is not how to get enough
space, it's how to resume the emerge instead of starting over, once I
have added the space.
It was initially set to 14G (out of 32G RAM) and I added 2G. I suppose
I can add another 14G, but that would only leave 4G for the system
itself. I'm not sure how well that would work, but I suppose it's
worth a try.
tmpfs only uses the space it needs, so it would appear to the ebuild that
there is plenty of space, but it would only use and extra gig or two of
your RAM.

For me, avoiding tmpfs for big ebuilds is the least hassle, using
package.env.

% cat /etc/portage/package.env/chromium
www-client/chromium disk-tmpdir.conf ...

% cat /etc/portage/env/disk-tmpdir.conf
PORTAGE_TMPDIR="/mnt/scratch"
--
Neil Bothwick

Linux like wigwam. No windows, no gates, Apache inside.
Wol
2024-03-25 11:40:02 UTC
Permalink
Post by Neil Bothwick
tmpfs only uses the space it needs, so it would appear to the ebuild that
there is plenty of space, but it would only use and extra gig or two of
your RAM.
For me, avoiding tmpfs for big ebuilds is the least hassle, using
package.env.
I've yet to set my current system up with a tmpfs, but that's my
preferred setup.

Iirc, my current mobo takes 4x16GB chips, so "double ram as swap" makes
128GB - I put that on every disk (I have 3 or 4), so I've probably got
not far short of 1/2TB swap!

So nearly all of my build spaces, I would typically have a ramdisk of
maybe 128GB.

As previously mentioned, it only uses what space it needs, so I guess
with 32GB of real RAM, with most stuff my system wouldn't touch swap
even with huge ramdisks being declared.

Cheers,
Wol

Jack
2024-03-23 00:20:01 UTC
Permalink
Post by Jack
Post by Jack
In this case, the offending package is dev-lang/rust, but this has
happened to me previously with other packages that require a lot of
space and time to build.
The build fails for any reason.  Since it has already progressed
over
Post by Jack
half way through the build, I would like to continue the build
instead
Post by Jack
of starting from scratch.  I fix whatever caused the issue.  In this
particular case, it failed for out of space, so I added another G to
the ramdisk mounted at /var/tmp/portageg.  I issue "ebuild
path/to/package.ebuild compile" but it fails immediately because
there
Post by Jack
isn't enough free space.  The problem is that it's checking for
enough
Post by Jack
free space to do the entire build, not just to continue from where
it
Post by Jack
left off.
Why not add more to the ramdisk, assuming it is a tmpfs. If it needs more
than your physical memory, it will use swap, but that won't happen
because you only need the extra space.
I generally avoid this problem by having large packages use a
PORTAGE_TMPDIR on a spinning disk. It may be slightly slower, but that
doesn't really matter on a long compile, not nearly as much as having to
restart it!
Post by Jack
I've done some searching, and found one old forum post which
suggests
Post by Jack
to just temporarily remove the space check from the ebuild.
In this case, that seems to be line 236 of
/usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is
"CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last
line withing pre_build_checks().  However, if I comment out that
line,
Post by Jack
and run "ebuild path/to/ebuild manifest" another attempt to compile
still gives the same error about not enough space.
Have you tried setting M to a smaller value immediately before that line?
It seems the problem is that the enviroment file in the temp dir of the
build area is sourced when you run ebuild/emerge. (It's among the
first output when you run ebuild.) Since that file was created based
on the state of the ebuild when the first emerge/ebuild was run,
editing the ebuild file doesn't affect it. I ended up finding a place
to unset CHECKREQS_FAILED, and the compile is now continuing.
Bryan Gardiner
2024-03-24 01:10:01 UTC
Permalink
On Fri, 22 Mar 2024 20:11:57 -0400
Post by Jack
It seems the problem is that the enviroment file in the temp dir of
the build area is sourced when you run ebuild/emerge. (It's among
the first output when you run ebuild.) Since that file was created
based on the state of the ebuild when the first emerge/ebuild was
run, editing the ebuild file doesn't affect it. I ended up finding a
place to unset CHECKREQS_FAILED, and the compile is now continuing.
According to check-reqs.eclass, you can also bypass the check by
exporting CHECKREQS_DONOTHING=1. I assume this would work for a
resumed build too.

Cheers,
Bryan
Loading...