[gentoo-user] Is distfile partial mirror with failover possible?

[Walter Dnes]

I may be misunderstanding, but it seems to me that local mirrors are
all-or-nothing. In the interests of saving bandwidth, I'd like to have
a client first check the host's /var/cache/distfiles directory for a
source tarball file. If not found, then fail over to another mirror as
per GENTOO_MIRRORS in /etc/portage/make.conf. This would require emerge
doing the lookup and potential failover for each file. Is this possible?

[Michael]

Post by Walter Dnes
I may be misunderstanding, but it seems to me that local mirrors are
all-or-nothing. In the interests of saving bandwidth, I'd like to have
a client first check the host's /var/cache/distfiles directory for a
source tarball file. If not found, then fail over to another mirror as
per GENTOO_MIRRORS in /etc/portage/make.conf. This would require emerge
doing the lookup and potential failover for each file. Is this possible?

Unless I misunderstand what you're asking, isn't this what takes place anyway?

Emerge will look at /var/cache/distfiles and if the source tarball is missing,
will try to fetch it from the list of GENTOO_MIRRORS you have configured. For
example, if you have:

GENTOO_MIRRORS="http://192.168.2.35/my_gentoo_distfiles/ https://
www.mirrorservice.org/sites/distfiles.gentoo.org/"

any host configured as above will first look at its own /var/cache/distfiles,
then at the first configured mirror which happens to be in the same LAN, then
whatever other external mirrors you have added. You could of course export
your local mirror's LAN ${DISTDIR} via NFS for your hosts to use.

PS. Is http_replicator still available/maintained? I see the wiki mentions
apt-cacher-ng for local distfiles cache.

[Neil Bothwick]

Post by Michael

Post by Walter Dnes
I may be misunderstanding, but it seems to me that local mirrors are
all-or-nothing. In the interests of saving bandwidth, I'd like to
have a client first check the host's /var/cache/distfiles directory
for a source tarball file. If not found, then fail over to another
mirror as per GENTOO_MIRRORS in /etc/portage/make.conf. This would
require emerge doing the lookup and potential failover for each file.
Is this possible?

Unless I misunderstand what you're asking, isn't this what takes place anyway?

The reference to a host makes me think Walter wants one machne to hold
the distfiles for all on the network.

Post by Michael
PS. Is http_replicator still available/maintained? I see the wiki
mentions apt-cacher-ng for local distfiles cache.

I use apt-cacher-ng for this and it does what you are looking for.

[Alan McKinnon]

Quick n dirty solution:

put all distfiles on a central server
FS mount that remote dir to /var/cache/distfiles on all hosts

Alan

Post by Neil Bothwick

Post by Michael

Post by Walter Dnes
I may be misunderstanding, but it seems to me that local mirrors are
all-or-nothing. In the interests of saving bandwidth, I'd like to
have a client first check the host's /var/cache/distfiles directory
for a source tarball file. If not found, then fail over to another
mirror as per GENTOO_MIRRORS in /etc/portage/make.conf. This would
require emerge doing the lookup and potential failover for each file.
Is this possible?

Unless I misunderstand what you're asking, isn't this what takes place anyway?

The reference to a host makes me think Walter wants one machne to hold
the distfiles for all on the network.

Post by Michael
PS. Is http_replicator still available/maintained? I see the wiki
mentions apt-cacher-ng for local distfiles cache.

I use apt-cacher-ng for this and it does what you are looking for.
--
Neil Bothwick
The facts, although interesting, are usually irrelevant.

[Neil Bothwick]

Post by Alan McKinnon
put all distfiles on a central server
FS mount that remote dir to /var/cache/distfiles on all hosts

That's what I used to do, but had problems when downloading the same file
from to clients at once.

BTW Welcome back Alan, but leave your dirty top-posting in Archland :P

Post by Alan McKinnon
Alan

Post by Neil Bothwick

Post by Michael

Post by Walter Dnes
I may be misunderstanding, but it seems to me that local
mirrors are all-or-nothing. In the interests of saving
bandwidth, I'd like to have a client first check the host's
/var/cache/distfiles directory for a source tarball file. If not
found, then fail over to another mirror as per GENTOO_MIRRORS in
/etc/portage/make.conf. This would require emerge doing the
lookup and potential failover for each file. Is this possible?

Unless I misunderstand what you're asking, isn't this what takes place anyway?

The reference to a host makes me think Walter wants one machne to hold
the distfiles for all on the network.

Post by Michael
PS. Is http_replicator still available/maintained? I see the wiki
mentions apt-cacher-ng for local distfiles cache.

I use apt-cacher-ng for this and it does what you are looking for.
--
Neil Bothwick
The facts, although interesting, are usually irrelevant.

[Alan McKinnon]

Post by Neil Bothwick

Post by Alan McKinnon
put all distfiles on a central server
FS mount that remote dir to /var/cache/distfiles on all hosts

That's what I used to do, but had problems when downloading the same file
from to clients at once.
BTW Welcome back Alan, but leave your dirty top-posting in Archland :P

Eh, I use gmail in the browser ..... the blerry thing is built to top post,
like Outlook


Alan

[Mark Knecht]

Post by Alan McKinnon

Post by Neil Bothwick

Post by Alan McKinnon
put all distfiles on a central server
FS mount that remote dir to /var/cache/distfiles on all hosts

That's what I used to do, but had problems when downloading the same file
from to clients at once.
BTW Welcome back Alan, but leave your dirty top-posting in Archland :P

Eh, I use gmail in the browser ..... the blerry thing is built to top

post, like Outlook

Post by Alan McKinnon
Alan

I'm a GMail user also. Sadly you'll want to not only bottom post, but also
select all text you're responding to, remove formatting (Ctrl-V) and then
type your response or you'll be down voted for responding in HTML.

I hate it also, but this list is easily one of my favorites and I'm no
longer
a Gentoo user.

Best wishes,
Mark

[Arve Barsnes]

Post by Mark Knecht
I'm a GMail user also. Sadly you'll want to not only bottom post, but also
select all text you're responding to, remove formatting (Ctrl-V) and then
type your response or you'll be down voted for responding in HTML.
I hate it also, but this list is easily one of my favorites and I'm no longer
a Gentoo user.

I also use both GMail and Outlook, and you can set up GMail to be
plain text by default. It's not too difficult to click the three dots
and type your reply where it belongs. Not so easy in Outlook which is
actively hating its users.

Regards,
Arve

[Walter Dnes]

Post by Neil Bothwick
The reference to a host makes me think Walter wants one machne to hold
the distfiles for all on the network.

Correct.

Post by Neil Bothwick
I use apt-cacher-ng for this and it does what you are looking for.

https://wiki.gentoo.org/wiki/Local_distfiles_cache#Open_issues says...

Post by Neil Bothwick
Open issues
* Apt-cacher-ng installs a cron job to delete unreferenced files
from the cache. Since the Gentoo cache contains no index files,
this probably deletes either everything or nothing from it.

Have you run into ptoblems with this?

It looks like remote-mounting /var/cache/distfiles might be the
quick-n-dirty solution like Alan suggested. And I never have a need to
have emerge run simultaneously on more than one machine.

[Neil Bothwick]

Post by Walter Dnes

Post by Neil Bothwick
I use apt-cacher-ng for this and it does what you are looking for.

https://wiki.gentoo.org/wiki/Local_distfiles_cache#Open_issues says...

Post by Neil Bothwick
Open issues
* Apt-cacher-ng installs a cron job to delete unreferenced files
from the cache. Since the Gentoo cache contains no index files,
this probably deletes either everything or nothing from it.

Have you run into ptoblems with this?

Not that I've noticed.

Post by Walter Dnes
It looks like remote-mounting /var/cache/distfiles might be the
quick-n-dirty solution like Alan suggested. And I never have a need to
have emerge run simultaneously on more than one machine.

It seems the simplest suitable solution for your needs.

[Walter Dnes]

Post by Neil Bothwick

Post by Walter Dnes
It looks like remote-mounting /var/cache/distfiles might be the
quick-n-dirty solution like Alan suggested. And I never have a need to
have emerge run simultaneously on more than one machine.

It seems the simplest suitable solution for your needs.

And while we're at it, howsabout "emerge --sync" on one host, and then
remote-mounting /var/db/repos/gentoo ?

[Michael]

Post by Walter Dnes

Post by Neil Bothwick

Post by Walter Dnes
It looks like remote-mounting /var/cache/distfiles might be the
quick-n-dirty solution like Alan suggested. And I never have a need to
have emerge run simultaneously on more than one machine.

It seems the simplest suitable solution for your needs.

And while we're at it, howsabout "emerge --sync" on one host, and then
remote-mounting /var/db/repos/gentoo ?

Take a look here:

https://wiki.gentoo.org/wiki/Local_Mirror

[Neil Bothwick]

Post by Michael

Post by Walter Dnes
And while we're at it, howsabout "emerge --sync" on one host, and
then remote-mounting /var/db/repos/gentoo ?

https://wiki.gentoo.org/wiki/Local_Mirror

I used to do that, the server syncing with Gentoo then everything else
syncing with that. However, when I switched to git syncing, it was so
fast that there was no point in running a local mirror - everything syncs
directly from Github now.

[Walter Dnes]

Post by Michael

Post by Walter Dnes
And while we're at it, howsabout "emerge --sync" on one host, and then
remote-mounting /var/db/repos/gentoo ?

https://wiki.gentoo.org/wiki/Local_Mirror

That's what I do right now. I figure that remote-mounting might make
things easier.

[Michael]

Post by Walter Dnes

Post by Michael

Post by Walter Dnes
And while we're at it, howsabout "emerge --sync" on one host, and then
remote-mounting /var/db/repos/gentoo ?

https://wiki.gentoo.org/wiki/Local_Mirror

That's what I do right now. I figure that remote-mounting might make
things easier.

There's more work required to set it up than just rsyncd and NFS requires you
run the local mirror all the time if the clients expect it to be up & running
when they boot.

[William Kenworthy]

Post by Walter Dnes

Post by Neil Bothwick

Post by Walter Dnes
It looks like remote-mounting /var/cache/distfiles might be the
quick-n-dirty solution like Alan suggested. And I never have a need to
have emerge run simultaneously on more than one machine.

It seems the simplest suitable solution for your needs.

And while we're at it, howsabout "emerge --sync" on one host, and then
remote-mounting /var/db/repos/gentoo ?

Similarly - I was mounting a single location on moosefs (network
filesystem) for ~15 hosts.  One host updates using --sync.  One
buildhost for each architecture building packages.  Distfiles are
handled the same way. Then "emerge -k" on the remainder checking for
packages that needed an actual build.  If none, parallel merge (via
ansible) or else single builds.  Resulting packages are also stored
remotely per arch (various amd64, arm32, arm64 etc.)

Working well for a few years, but I am in the process of converting to
nfs with a similar structure as I have decommissioned the moosefs system
and many machines that are no longer needed.  A single machine will
handle built packages, distfiles and repos which are mounted on the
other systems via nfs. I will look into apt-cacher-ng as I was a fan of
http-replicator until it was deprecated.

BillK

[hitachi303]

Post by Michael
PS. Is http_replicator still available/maintained? I see the wiki mentions
apt-cacher-ng for local distfiles cache.

IIRC it died when python or perl 2.x was no longer available.

[Michael]

Post by hitachi303

Post by Michael
PS. Is http_replicator still available/maintained? I see the wiki mentions
apt-cacher-ng for local distfiles cache.

IIRC it died when python or perl 2.x was no longer available.

It's worth mentioning another option, not involving a full caching proxy as
such. It may suit Walter's workflow so I mention it here for completeness. A
lightweight browser can serve files directly from the local mirror's own /var/
cache/distfiles.

Unlike a caching proxy, files not already downloaded/needed by the local
mirror won't be available locally, so the clients will download these from
external mirrors. The local server will need to be updated first, to fetch
all the latest source file versions. If there is a large difference in
package requirements between different clients and the server, then only a
partial bandwidth efficiency will be achieved this way. Also if all the
clients require the same packages not available on the server, then these will
be downloaded by each client separately from external mirrors, further eroding
any savings in overall bandwidth.

Setting up a small footprint browser like gatling, or just a couple of lines
of python http.server, will allow any light on resources gentoo machine to
become a useful local mirror with minimal configuration.