[gentoo-user] Tailing compressed build logs

[Bryan Gardiner]

Hi folks,

How can I follow Portage's compressed build logs in real time as they
are generated?

I keep build logs and use FEATURES=compress-build-logs so that they
don't get too large. I can peek at how a build is going with zless on
build.log.gz, which doesn't update (understandably), but I would
really like to be able to watch a log with some "tail -f" equivalent.
I get streaming output with

tail -c +1 -f build.log.gz | od -t x1

but the following hangs with no output:

tail -c +1 -f build.log.gz | gunzip

even with a build log that is 72KB compressed (2.4MB uncompressed),
which should be larger than any pipe buffers... Any idea why gunzip
can't handle this, or what I should I should be doing instead?

Thanks,
Bryan

[Mickaël Bucas]

Post by Bryan Gardiner
Hi folks,
How can I follow Portage's compressed build logs in real time as they
are generated?
I keep build logs and use FEATURES=compress-build-logs so that they
don't get too large. I can peek at how a build is going with zless on
build.log.gz, which doesn't update (understandably), but I would
really like to be able to watch a log with some "tail -f" equivalent.
I get streaming output with
tail -c +1 -f build.log.gz | od -t x1
tail -c +1 -f build.log.gz | gunzip
even with a build log that is 72KB compressed (2.4MB uncompressed),
which should be larger than any pipe buffers... Any idea why gunzip
can't handle this, or what I should I should be doing instead?
Thanks,
Bryan

Hi

Reading the man page, "zless" is just a wrapper around "less".
You can check with:
$ file $(which zless)
/usr/bin/zless: POSIX shell script, ASCII text executable
$ less $(which zless)

So it should support the same options including typing "F" at the end
of a file to keep trying to read when the end of file is reached.

I made a small test, but it didn't work:
# Create a growing file
$ yes | nl | gzip > zless-test.gz &
# Try to follow at the end
$ zless zless-test.gz

With ">" to go to the end and "F" to continue, I didn't get the
expected behavior, it stood still at the point I was viewing.
I don't know if it's really a bug or if I made a mistake...
(Don't forget to stop the growing file :) )

Best regards

Mickaël Bucas

[Michael]

Post by Mickaël Bucas

Post by Bryan Gardiner
Hi folks,
How can I follow Portage's compressed build logs in real time as they
are generated?
I keep build logs and use FEATURES=compress-build-logs so that they
don't get too large. I can peek at how a build is going with zless on
build.log.gz, which doesn't update (understandably), but I would
really like to be able to watch a log with some "tail -f" equivalent.
I get streaming output with
tail -c +1 -f build.log.gz | od -t x1
tail -c +1 -f build.log.gz | gunzip
even with a build log that is 72KB compressed (2.4MB uncompressed),
which should be larger than any pipe buffers... Any idea why gunzip
can't handle this, or what I should I should be doing instead?
Thanks,
Bryan

Hi
Reading the man page, "zless" is just a wrapper around "less".
$ file $(which zless)
/usr/bin/zless: POSIX shell script, ASCII text executable
$ less $(which zless)
So it should support the same options including typing "F" at the end
of a file to keep trying to read when the end of file is reached.
# Create a growing file
$ yes | nl | gzip > zless-test.gz &
# Try to follow at the end
$ zless zless-test.gz
With ">" to go to the end and "F" to continue, I didn't get the
expected behavior, it stood still at the point I was viewing.
I don't know if it's really a bug or if I made a mistake...
(Don't forget to stop the growing file :) )
Best regards
Mickaël Bucas

You could try:

tail -c +1 -f build.log.gz | gunzip | less

I think it should work, but I haven't tried it.

[Bryan Gardiner]

Hi,

On Tue, 07 Mar 2023 09:01:34 +0000

Post by Michael

Post by Mickaël Bucas

Post by Bryan Gardiner
Hi folks,
How can I follow Portage's compressed build logs in real time as
they are generated?
...
tail -c +1 -f build.log.gz | od -t x1
tail -c +1 -f build.log.gz | gunzip

Reading the man page, "zless" is just a wrapper around "less".

Neat, zless is a lot simpler than I thought.

Post by Michael

Post by Mickaël Bucas
So it should support the same options including typing "F" at the
end of a file to keep trying to read when the end of file is
reached.
# Create a growing file
$ yes | nl | gzip > zless-test.gz &
# Try to follow at the end
$ zless zless-test.gz
With ">" to go to the end and "F" to continue, I didn't get the
expected behavior, it stood still at the point I was viewing.
I don't know if it's really a bug or if I made a mistake...
(Don't forget to stop the growing file :) )

I suppose it makes sense that a simple "gunzip -c file.gz | less"
doesn't work, assuming zless is equivalent to that. gunzip sees the
end of the file and exits rather than waiting for additional content.
Then less sees its pipe close.

So I think the "tail -c +1 -f build.log.gz" bit is needed, and appears
to work since od shows data appended every few seconds.

Post by Michael
tail -c +1 -f build.log.gz | gunzip | less
I think it should work, but I haven't tried it.

Thanks, but I suspect that less is still at the whim of gunzip here,
and gunzip isn't producing anything.

Maybe gunzip just has a massive output buffer. If I do something a bit
more complicated:

(N=0; while true; do
(( N++ ))
echo "$N" | gzip
if (( N % 4000 == 0 )); then sleep 1; fi
done) | gunzip | od -A d -t x1

then nothing is shown for 15 seconds or so, then od quickly dumps
51952 bytes of hex output, then another long pause before dumping
another 60480 bytes of hex. If I try this with "pigz -cd" instead of
"gunzip" then output starts flowing after just ~400 bytes, but sadly
that still doesn't work for build.log.gz.

Anyway, I've had another thought, that since portage is flushing lines
to the gzipped log fairly frequently, it's probably not getting the
best compression ratio. So I went and tested recompressing some logs,
and got a decent improvement:

- nodejs-18.9.1: build.log.gz is 112KB, recompressed it's 68.9KB
(38.5% savings).

- qtwebengine-5.15.8_p20230112: build.log.gz is 1.44MB, recompressed
it's 1.01MB (29.9% savings).

- gtk+-3.24.37: build.log.gz is 157KB, recompressed it's 120KB
(23.6% savings).

So maybe I'm better off abandoning this feature, and setting something
up to compress build logs after the fact. Then good ol' tail -f will
work just fine.

Thanks, cheers,
Bryan

[Andreas Stiasny]

Post by Bryan Gardiner
tail -c +1 -f build.log.gz | gunzip

I think you should either use gunzip -f --stdout instead of just gunzip
or use zcat.


Andreas

[Bryan Gardiner]

On Wed, 8 Mar 2023 14:13:35 +0100

Post by Andreas Stiasny

Post by Bryan Gardiner
tail -c +1 -f build.log.gz | gunzip

I think you should either use gunzip -f --stdout instead of just
gunzip or use zcat.
Andreas

Hi, thanks! --force and --stdout don't seem to help in this case.

- Bryan

[Andreas Stiasny]

Post by Bryan Gardiner
Hi, thanks! --force and --stdout don't seem to help in this case.

Sorry, I was wrong. The problem is not the actual use of gunzip but the
fact that you are trying to decompress a file without reading it from
the beginning. Usually this is not possible.

zcat build.log.gz | tail -n 30

would show you the last 30 lines but no new content because zcat or
gunzip is not waiting for new input. You could try gztool

https://github.com/circulosmeos/gztool/

"gztool -T" should work like "tail -f".


Andreas

[Peter Humphrey]

Post by Andreas Stiasny
You could try gztool
https://github.com/circulosmeos/gztool/
"gztool -T" should work like "tail -f".

That looks very handy. But:

Compiling it from the github source fails because I haven't found all the
prerequisites.

Fetching a Debian compiled version and running deb2targz on it ends up with a
.so file, which where to put?

I haven't found it in any Gentoo overlays.

How can I get hold of a version for Gentoo?

[Matt Connell]

Post by Peter Humphrey
Fetching a Debian compiled version and running deb2targz on it ends up with a
.so file, which where to put?

app-arch/deb2targz exists. Would probably satisfy the need.

[Dale]

Post by Peter Humphrey

Post by Andreas Stiasny
You could try gztool
https://github.com/circulosmeos/gztool/
"gztool -T" should work like "tail -f".

Compiling it from the github source fails because I haven't found all the
prerequisites.
Fetching a Debian compiled version and running deb2targz on it ends up with a
.so file, which where to put?
I haven't found it in any Gentoo overlays.
How can I get hold of a version for Gentoo?

I found it in a overlay.  At least google and github page says it is
there. 

https://github.com/gentoo-mirror/slonko

It appears to be called gztool. 

Hope that helps.

Dale

:-)  :-) 

[Peter Humphrey]

Post by Peter Humphrey

Post by Andreas Stiasny
You could try gztool
https://github.com/circulosmeos/gztool/
"gztool -T" should work like "tail -f".

Compiling it from the github source fails because I haven't found all the
prerequisites.
Fetching a Debian compiled version and running deb2targz on it ends up
with a .so file, which where to put?
I haven't found it in any Gentoo overlays.
How can I get hold of a version for Gentoo?

I found it in a overlay. At least google and github page says it is
there.
https://github.com/gentoo-mirror/slonko
It appears to be called gztool.
Hope that helps.

It does - thanks!

[Bryan Gardiner]

On Thu, 9 Mar 2023 10:01:16 -0600

Post by Dale

Post by Peter Humphrey

Post by Andreas Stiasny
You could try gztool
https://github.com/circulosmeos/gztool/
"gztool -T" should work like "tail -f".

Compiling it from the github source fails because I haven't found
all the prerequisites.
Fetching a Debian compiled version and running deb2targz on it ends
up with a .so file, which where to put?
I haven't found it in any Gentoo overlays.
How can I get hold of a version for Gentoo?

I found it in a overlay.  At least google and github page says it is
there. 
https://github.com/gentoo-mirror/slonko
It appears to be called gztool. 
Hope that helps.

Ah, this is great. "gztool -TW" does exactly what it should. Nice
find, thanks a bunch!

- Bryan