[gentoo-user] Question about emerge sync and where it all goes.

[Dale]

Howdy,

I've mentioned before that I build my packages in a chroot.  I have a OS
copy on a separate drive.  I do this because of the long compile times
of some packages.  On occasion tho, I catch the tree in a bad place. 
Some conflict or other happens and I need to sync again to get fixes
etc.  Given my process tho, I don't want to sync the chroot without also
syncing my main system because the two will not be in sync and I'll lose
my update since it deletes files as well including any updates I did
emerge successfully.  I need the two to be identical.  So, I'd like to
sync my main system and then copy the new data over to the chroot
without copying the rest of the OS.  I did this with the following
command earlier. 

rsync -av --progress --delete  /var/cache/portage/tree/*
/backup/gentoo-build/var/cache/portage/tree/

As you can see, my tree location is not the default.  It is located at
/var/cache/portage/tree/.  It's the same on the chroot obviously.  Would
the above command copy all the needed files/directories over that it
needs after a emerge sync or is there more to it?  The reason I think it
needs more, when I tried to do the usual emerge -aukDN world on my main
system, it wanted to emerge more packages than the chroot did.  I
suspect there is more to it than just the tree directory. 

Anyone have the details on this?  Anyone know what other
files/directories need to be copied over as well?

Thanks.

Dale

:-)  :-) 

[Michael]

Post by Dale
Howdy,
I've mentioned before that I build my packages in a chroot. I have a OS
copy on a separate drive. I do this because of the long compile times
of some packages. On occasion tho, I catch the tree in a bad place.
Some conflict or other happens and I need to sync again to get fixes
etc. Given my process tho, I don't want to sync the chroot without also
syncing my main system because the two will not be in sync and I'll lose
my update since it deletes files as well including any updates I did
emerge successfully. I need the two to be identical. So, I'd like to
sync my main system and then copy the new data over to the chroot
without copying the rest of the OS.

What you do is duplicating effort and storage space with questionable benefit,
besides warming up your room. I think it would be more efficient if you used
the same /var/cache/distfiles and /var/db/repos/ filesystems on both
installations. This way you'd sync once and then download any source files
once only.

However, an even more optimised solution would be to compile packages once in
your chrooted fs with the '--buildpkg y' option and then emerge these as
binary packages from /var/cache/binpkgs/ in your production OS. An overnight
emerge in the chrooted OS will take only a few minutes on the production
system.

Post by Dale
I did this with the following command earlier.
rsync -av --progress --delete /var/cache/portage/tree/*
/backup/gentoo-build/var/cache/portage/tree/

Your syntax for the source tree will fail to copy a couple files immediately
under /var/cache/portage/tree/ which start with "."

It would be better if you used:

/var/cache/portage/tree /backup/gentoo-build/var/cache/portage/tree

Post by Dale
As you can see, my tree location is not the default. It is located at
/var/cache/portage/tree/. It's the same on the chroot obviously. Would
the above command copy all the needed files/directories over that it
needs after a emerge sync or is there more to it? The reason I think it
needs more, when I tried to do the usual emerge -aukDN world on my main
system, it wanted to emerge more packages than the chroot did. I
suspect there is more to it than just the tree directory.
Anyone have the details on this? Anyone know what other
files/directories need to be copied over as well?
Thanks.
Dale
:-) :-)

Did you diff /var/lib/portage/world between the two systems to see what
differences may be there?

I can think of a hypothetical case where the chrooted fs may have already
emerged build time dependencies, e.g. if some emerge failed and then you
fixed/restarted it a number of times, but the production system would not yet
have such build time dependencies installed.

[Dale]

Post by Michael

Post by Dale
Howdy,
I've mentioned before that I build my packages in a chroot. I have a OS
copy on a separate drive. I do this because of the long compile times
of some packages. On occasion tho, I catch the tree in a bad place.
Some conflict or other happens and I need to sync again to get fixes
etc. Given my process tho, I don't want to sync the chroot without also
syncing my main system because the two will not be in sync and I'll lose
my update since it deletes files as well including any updates I did
emerge successfully. I need the two to be identical. So, I'd like to
sync my main system and then copy the new data over to the chroot
without copying the rest of the OS.

What you do is duplicating effort and storage space with questionable benefit,
besides warming up your room. I think it would be more efficient if you used
the same /var/cache/distfiles and /var/db/repos/ filesystems on both
installations. This way you'd sync once and then download any source files
once only.
However, an even more optimised solution would be to compile packages once in
your chrooted fs with the '--buildpkg y' option and then emerge these as
binary packages from /var/cache/binpkgs/ in your production OS. An overnight
emerge in the chrooted OS will take only a few minutes on the production
system.

One reason for doing it this way, I have a backup of my OS as well. 
Both are generally identical.  If my main hard drive were to fail, I
could boot rescue media and copy what I usually chroot into over to a
new drive, reinstall grub and such then boot back up.  Should be back up
and running in likely less than a hour, if that.  I took the idea from a
stage4 thing I found.  I just don't compress it since I use it.  After
all, that 750GB hard drive is only used for hourly backups of my email. 
May as well put the rest of the thing to some use. 

The buildpkg option is kinda what happens except I let it also install
and that gives me a backup OS as well.  The main drive with the OS on it
is a bit old.  It is a drive designed for 24/7 use and years of
service.  Still, it could blow smoke or start making noises at any time,
or run for many more years until a OS won't fit on it anymore.  It's
already getting pretty full.  LOL  Someone on this list mailed it to me
years ago with my previous build I think. 

The updates from binary packages only takes a few minutes unless the
number of packages is huge.  Usually a HUGE KDE update and LOo or
something added on top of that can take a hour or so.  Still, much
faster and more stable than trying to compile on my running OS.  Things
get out of sync and some programs start acting weird or won't open due
to a version mismatch.

Post by Michael

Post by Dale
I did this with the following command earlier.
rsync -av --progress --delete /var/cache/portage/tree/*
/backup/gentoo-build/var/cache/portage/tree/

Your syntax for the source tree will fail to copy a couple files immediately
under /var/cache/portage/tree/ which start with "."
/var/cache/portage/tree /backup/gentoo-build/var/cache/portage/tree

I edited that in my little script.  The locations with rsync can be
tricky.  Each way that one can use gives a slightly different execution. 

Post by Michael

Post by Dale
As you can see, my tree location is not the default. It is located at
/var/cache/portage/tree/. It's the same on the chroot obviously. Would
the above command copy all the needed files/directories over that it
needs after a emerge sync or is there more to it? The reason I think it
needs more, when I tried to do the usual emerge -aukDN world on my main
system, it wanted to emerge more packages than the chroot did. I
suspect there is more to it than just the tree directory.
Anyone have the details on this? Anyone know what other
files/directories need to be copied over as well?
Thanks.
Dale
:-) :-)

Did you diff /var/lib/portage/world between the two systems to see what
differences may be there?
I can think of a hypothetical case where the chrooted fs may have already
emerged build time dependencies, e.g. if some emerge failed and then you
fixed/restarted it a number of times, but the production system would not yet
have such build time dependencies installed.

The world file is also copied over each time.  Sometimes I install
something on the running OS and I also want it on the other.  I keep the
two pretty well synced. 

It could be the path change you listed above will change things.  I'll
know the next time I have to do this.  I was even wondering if after
copying the tree over, I needed to run a command to update something to
make sure emerge would see the changes.  I knew it was doing something
different between the two but only did so after I copied the tree over. 
I figured I missed something, be it a file that didn't copy over or a
command I needed to run. 

I'll know when this happens again.  I may even test this when I do the
next update.  I've got most of the kinks worked out on this but things
do pop up.  ;-) 

Thanks much. 

Dale

:-)  :-) 

[Michael]

Post by Michael

Post by Dale
I did this with the following command earlier.
rsync -av --progress --delete /var/cache/portage/tree/*
/backup/gentoo-build/var/cache/portage/tree/

Your syntax for the source tree will fail to copy a couple files
immediately under /var/cache/portage/tree/ which start with "."
/var/cache/portage/tree /backup/gentoo-build/var/cache/portage/tree

I edited that in my little script. The locations with rsync can be
tricky. Each way that one can use gives a slightly different execution.

Oops! Scratch that - I was wrong. Sorry!

This is what will work:

/foo/ /backup/foo

will copy the contents of source /foo into destination /backup/foo, without
copying the top level directory /foo itself. This way both /foo at source and
/backup/foo at destination will have identical content.

If you omit a trailing slash you'll copy the source top level directory itself
and will get:

/backup/foo/foo/...

which NOT what you're after.

Hope this helps.