[gentoo-user] Console scrollback is back again!

[Alan Mackenzie]

Hello, Gentoo.

Yes, console soft scrolling is back! That essential feature that was
stripped out of the kernel at around 5.4.x has returned!

Only this time, it's even better! Instead of one scrollback buffer
shared between all tty's, there's now a buffer for each tty.

How to get it working:
(i) Extract the enclosed patch file to your /usr/src/linux-5.10.27-gentoo
directory (or probably any 5.10 kernel's).
(ii) From that directory run $ patch -p0 < diff.20210405.diff.
(iii)(a) If you've already got a working 5.10.27, do a $ make oldconfig.
That should bring up the new configuration items, for which you can
accept the defaults. These two items are a flag to enable the
scrollback, and a buffer size defaulting to 128kB.
(iii)(b) If you haven't yet got a 5.10.27, just configure your kernel in
the usual way. The two new items are under Device Drivers/Graphics
support/Console display driver support.
(iv) Build the kernel.
(v) Put the new kernel into your usual boot manager.
(vi) Reboot and enjoy!

Admittedly, the exercise isn't quite finished - the patched source files
still have my personal change markings in them, to make debugging easier.
But the problems I reported here a few days ago are now solved. One or
two features haven't (yet) been implemented - having a single scroll
buffer shared amongst all tty's isn't there, and there're no kernel
command line parameters to control the feature.

Also, you may wonder about how safe the patch is. All I can say is that
there is nothing malicious in it, and I am known on this list, but of
course if it breaks I won't be to blame.

Bug reports and other comments are welcome, of course.

If anybody would like the corresponding patch which works on 5.4.n, for n

= 80, that is available, too.

[antlists]

Post by Alan Mackenzie
Bug reports and other comments are welcome, of course.
If anybody would like the corresponding patch which works on 5.4.n, for n

= 80, that is available, too.

Why did it get removed from the kernel?

If you want to get it back in, couldn't you ping Greg KH and get some
help - or even just advice - getting it accepted?

Cheers,
Wol

[Alan Mackenzie]

Hello, Wol.

Post by antlists

Post by Alan Mackenzie
Bug reports and other comments are welcome, of course.
If anybody would like the corresponding patch which works on 5.4.n, for n

= 80, that is available, too.

Why did it get removed from the kernel?

Allegedly, there were security problems with it. I think the kernel
people had a program which fired random inputs at lots of components,
and noted when things went wrong; and things went wrong in the scroll
back component. And they couldn't find anybody to look into these
problems.

That's the official position. I'm a bit more cynical about it.

Post by antlists
If you want to get it back in, couldn't you ping Greg KH and get some
help - or even just advice - getting it accepted?

I have no reason to believe all the "security problems" will have been
resolved by my hacking. I took the algorithmic bits from 4.19.97
basically unchanged.

Maybe there's not much enthusiasm for this feature, in which case I will
keep it working for myself. Linux is basically unusable to me without
it. There are probably quite a few patches to restore the scrollback
floating around the web by now: the changes were not particularly
difficult to anybody who knows the code, but getting to know the code
was unusually difficult.

We'll see how people react to it here, first.

Post by antlists
Cheers,
Wol

[Michael Jones]

Post by Alan Mackenzie
Hello, Wol.

Post by antlists

Post by Alan Mackenzie
Bug reports and other comments are welcome, of course.
If anybody would like the corresponding patch which works on 5.4.n,

for n

Post by antlists

Post by Alan Mackenzie

= 80, that is available, too.

Why did it get removed from the kernel?

Allegedly, there were security problems with it. I think the kernel
people had a program which fired random inputs at lots of components,
and noted when things went wrong; and things went wrong in the scroll
back component. And they couldn't find anybody to look into these
problems.
That's the official position. I'm a bit more cynical about it.

Post by antlists
If you want to get it back in, couldn't you ping Greg KH and get some
help - or even just advice - getting it accepted?

I have no reason to believe all the "security problems" will have been
resolved by my hacking. I took the algorithmic bits from 4.19.97
basically unchanged.
Maybe there's not much enthusiasm for this feature, in which case I will
keep it working for myself. Linux is basically unusable to me without
it. There are probably quite a few patches to restore the scrollback
floating around the web by now: the changes were not particularly
difficult to anybody who knows the code, but getting to know the code
was unusually difficult.
We'll see how people react to it here, first.

Post by antlists
Cheers,
Wol

--
Alan Mackenzie (Nuremberg, Germany).

Without looking at the patch itself:

Have you considered something like kmscon as a userland alternative?

I installed it on my laptop, and aside from not working with gpmd or
consolation for mouse support, it's a perfectly functional local tty.

[Alan Mackenzie]

Hello, Michael.

On Mon, Apr 05, 2021 at 18:08:58 -0500, Michael Jones wrote:

[ .... ]

Post by Michael Jones
Have you considered something like kmscon as a userland alternative?

To be honest, no. I wasn't really aware of it when I starting working on
the Linux console.

According to Wikipedia, the last release was in 2013, and development
stopped altogether in 2015. The Gentoo package for it is masked.

Post by Michael Jones
I installed it on my laptop, and aside from not working with gpmd or
consolation for mouse support, it's a perfectly functional local tty.

I suppose it's one of these programs that once working, pretty much
carries on working for ever.

[Michael Jones]

Post by Alan Mackenzie
Hello, Michael.
[ .... ]

Post by Michael Jones
Have you considered something like kmscon as a userland alternative?

To be honest, no. I wasn't really aware of it when I starting working on
the Linux console.
According to Wikipedia, the last release was in 2013, and development
stopped altogether in 2015. The Gentoo package for it is masked.

Good points, and I can't speak to the code cleanliness of kmscon, or
potential security problems.

That being said, even though I was annoyed when the scrollback was removed
from the kernel, I do understand the reason for it being removed. It does
seem like the kind of functionality that would be better provided by
userspace.

Post by Alan Mackenzie

Post by Michael Jones
I installed it on my laptop, and aside from not working with gpmd or
consolation for mouse support, it's a perfectly functional local tty.

I suppose it's one of these programs that once working, pretty much
carries on working for ever.

That's entirely possible :-)
Maybe someone (not necessarily you) will find the removal of the scrollback
buffer to be enough motivation to bring kmscon back to life as a project.

[Peter Humphrey]

Post by Alan Mackenzie
We'll see how people react to it here, first.

You're my hero!

[John Covici]

On Tue, 06 Apr 2021 09:14:23 -0400,

Post by Peter Humphrey

Post by Alan Mackenzie
We'll see how people react to it here, first.

You're my hero!

Would this patch work on 5.4.96 and following?

[Alan Mackenzie]

Hello, John

Post by John Covici
On Tue, 06 Apr 2021 09:14:23 -0400,

Post by Peter Humphrey

Post by Alan Mackenzie
We'll see how people react to it here, first.

You're my hero!

Would this patch work on 5.4.96 and following?

No, there is a slight difference (two struct fields moved and renamed)
between 5.4 and 5.10.

For 5.4, please use the attached patch instead. It has been tested on
5.4.80-r1 and 5.4.97.

Post by John Covici
--
How do
you spend it?
John Covici wb2una

[Jorge Almeida]

Post by Alan Mackenzie
Hello, Gentoo.
Yes, console soft scrolling is back! That essential feature that was
stripped out of the kernel at around 5.4.x has returned!
Only this time, it's even better! Instead of one scrollback buffer
shared between all tty's, there's now a buffer for each tty.

I just tested it with kernel 5.11.11. Seems fine.

Jorge Almeida

[Alan Mackenzie]

Hello, Jorge.

Post by Jorge Almeida

Post by Alan Mackenzie
Hello, Gentoo.
Yes, console soft scrolling is back! That essential feature that was
stripped out of the kernel at around 5.4.x has returned!
Only this time, it's even better! Instead of one scrollback buffer
shared between all tty's, there's now a buffer for each tty.

I just tested it with kernel 5.11.11. Seems fine.

Thanks!

Post by Jorge Almeida
Jorge Almeida

[Jorge Almeida]

Post by Alan Mackenzie

Post by Jorge Almeida
I just tested it with kernel 5.11.11. Seems fine.

Thanks!

Thank _you_!

It is regrettable that console scrolling is out of the kernel. Even if
one doesn't use the console that much, it is used once in a while, and
those occasions tend to be precisely the ones back scrolling will be
most missed.

Jorge Almeida

[k***@aspodata.se]

Post by Alan Mackenzie
Yes, console soft scrolling is back! That essential feature that was
stripped out of the kernel at around 5.4.x has returned!

It is commit 087b6cb17df5834d395ab72da3f937380470ba15,
between v5.4.65 and 66.
Perhaps also 0d123a8c64fde433a, cf5a7ded53652c3d63d72, and possible
other stuff.

Post by Alan Mackenzie
Only this time, it's even better! Instead of one scrollback buffer
shared between all tty's, there's now a buffer for each tty.

Great, no more missing data when switching vts.

Btw, wasn't that added in commit aabd31c421ddc730edf6d89c4ed3885e4fca5e30
but turned off by default.

...

Post by Alan Mackenzie
One ord two features haven't (yet) been implemented - having a
single scroll buffer shared amongst all tty's isn't there,

...

Is there any reason to share the buffer ?

Post by Alan Mackenzie
Bug reports and other comments are welcome, of course.

git log drivers/video/console/vgacon.c etc. can give some ideas
what one is up against.

Regards,
/Karl Hammar

[Alan Mackenzie]

Hello, Karl.

Post by k***@aspodata.se

Post by Alan Mackenzie
Yes, console soft scrolling is back! That essential feature that was
stripped out of the kernel at around 5.4.x has returned!

It is commit 087b6cb17df5834d395ab72da3f937380470ba15,
between v5.4.65 and 66.
Perhaps also 0d123a8c64fde433a, cf5a7ded53652c3d63d72, and possible
other stuff.

I will confess that I don't actually have a git repository of Linux on
my machine. I'd have to back it up, somehow. It would be a fair amount
of work to write an efficient back-up procedure - the one I have at the
moment for git repos backs up the entire repository, which would
probably be excessive for Linux.

Currently, I've changed a mere five files, for which I'm getting by with
symbolic links in the kernel tree pointing into my home directory.

Post by k***@aspodata.se

Post by Alan Mackenzie
Only this time, it's even better! Instead of one scrollback buffer
shared between all tty's, there's now a buffer for each tty.

Great, no more missing data when switching vts.
Btw, wasn't that added in commit aabd31c421ddc730edf6d89c4ed3885e4fca5e30
but turned off by default.

There was no capability at all for one buffer per tty in framebuffer
consoles. It was there for VGA consoles, and, I think, one or two
others. The configuration page for it was confusing - the option for
"persistent scrollback" was purely for VGAcon, though that wasn't
obvious. I spent time in the past trying to get multiple buffers
working, and was even considering reporting a bug.

Post by k***@aspodata.se
...

Post by Alan Mackenzie
One or two features haven't (yet) been implemented - having a
single scroll buffer shared amongst all tty's isn't there,

...
Is there any reason to share the buffer ?

Not that I can think of. The only reason might be if somebody wanted a
very large scrollback buffer, but didn't have a lot of RAM. But even
so, the current default takes a total of less than 1 MB of RAM with the
standard 7 tty's. Even increasing the buffer size to 10 MB, that would
only come to 70 MB. Such a size of buffer would really need more
sophisticated handling, with search facilities, some marking of
positions, and so on. That would be a large project.

Post by k***@aspodata.se

Post by Alan Mackenzie
Bug reports and other comments are welcome, of course.

git log drivers/video/console/vgacon.c etc. can give some ideas
what one is up against.

Once I find some way to read it. ;-(

Anyhow, the patch I have is working software, and is not going away any
time soon. It means I can carry on using Linux.

Just in case you're interested, the problem I had with no scrolling on
tty1 was due to the way the console was initialised early on in the boot
process. Its structures were initialised in a separate place from the
normal init function, thus bypassing the scrollbuffer's init. When it
came to re-initialising it for scrollbuffer, this partial initialisation
acted as a flag not to carry on with the init. At least it caused me to
learn about printk. ;-)

Post by k***@aspodata.se
Regards,
/Karl Hammar

[konsolebox]

Post by Alan Mackenzie
Hello, Gentoo.
Yes, console soft scrolling is back! That essential feature that was
stripped out of the kernel at around 5.4.x has returned!

Thank you. I'll try to make this work in 5.10.28 once it's released.

[J. Roeleveld]

Post by Alan Mackenzie
Hello, Gentoo.
Yes, console soft scrolling is back! That essential feature that was
stripped out of the kernel at around 5.4.x has returned!
Only this time, it's even better! Instead of one scrollback buffer
shared between all tty's, there's now a buffer for each tty.
(i) Extract the enclosed patch file to your /usr/src/linux-5.10.27-gentoo
directory (or probably any 5.10 kernel's).
(ii) From that directory run $ patch -p0 < diff.20210405.diff.
(iii)(a) If you've already got a working 5.10.27, do a $ make oldconfig.
That should bring up the new configuration items, for which you can
accept the defaults. These two items are a flag to enable the
scrollback, and a buffer size defaulting to 128kB.
(iii)(b) If you haven't yet got a 5.10.27, just configure your kernel in
the usual way. The two new items are under Device Drivers/Graphics
support/Console display driver support.
(iv) Build the kernel.
(v) Put the new kernel into your usual boot manager.
(vi) Reboot and enjoy!

Thank you!

I will add this to my "/etc/portage/patches/sys-kernel/gentoo-sources"
directory. Right next to the gpl-defang patch.

[konsolebox]

Post by Alan Mackenzie
(i) Extract the enclosed patch file to your /usr/src/linux-5.10.27-gentoo
directory (or probably any 5.10 kernel's).

Works with 5.10.28-gentoo. Thanks again.

I finally can uninstall my 5.8.6 which still has the old scrollback.

[Alan Mackenzie]

Hello, Gentoo.

A new kernel, gentoo-sources-5.10.49-r1 has become current.

For those who wish to have console soft scrolling, the patch which worked
with 5.10.27 actually still works in 5.10.49-r1. But I've tidied that
patch up somewhat for 5.10.49-r1 and have attached the patch file to this
post.

The instructions for using it are basically unchanged, apart from file
names, etc. Updating those instructions gives us:
(i) Extract the attached patch file to your
/usr/src/linux-5.10.49-gentoo-r1 directory (or probably any 5.10
kernel's).
(ii) From that directory run $ patch -p0 < 5.10.49-scroll.20210715.diff.
(iii)(a) If you've already got a working 5.10.49-r1, do a $ make oldconfig.
That should bring up the new configuration items, for which you can
accept the defaults. These two items are a flag to enable the
scrollback, and a buffer size defaulting to 128kB.
(iii)(b) If you haven't yet got a 5.10.49-r1, just configure your kernel
in the usual way. The two new items are under Device Drivers/Graphics
support/Console display driver support.
(iv) Build the kernel.
(v) Put the new kernel into your usual boot manager.
(vi) Reboot and enjoy!

As before, it works for me, and there's nothing malicious in it, but if
it breaks for you, I'm sorry but I won't be responsible.
--
Alan Mackenzie (Nuremberg, Germany).

Post by Alan Mackenzie
Hello, Gentoo.
Yes, console soft scrolling is back! That essential feature that was
stripped out of the kernel at around 5.4.x has returned!
Only this time, it's even better! Instead of one scrollback buffer
shared between all tty's, there's now a buffer for each tty.
(i) Extract the enclosed patch file to your /usr/src/linux-5.10.27-gentoo
directory (or probably any 5.10 kernel's).
(ii) From that directory run $ patch -p0 < diff.20210405.diff.
(iii)(a) If you've already got a working 5.10.27, do a $ make oldconfig.
That should bring up the new configuration items, for which you can
accept the defaults. These two items are a flag to enable the
scrollback, and a buffer size defaulting to 128kB.
(iii)(b) If you haven't yet got a 5.10.27, just configure your kernel in
the usual way. The two new items are under Device Drivers/Graphics
support/Console display driver support.
(iv) Build the kernel.
(v) Put the new kernel into your usual boot manager.
(vi) Reboot and enjoy!
Admittedly, the exercise isn't quite finished - the patched source files
still have my personal change markings in them, to make debugging easier.
But the problems I reported here a few days ago are now solved. One or
two features haven't (yet) been implemented - having a single scroll
buffer shared amongst all tty's isn't there, and there're no kernel
command line parameters to control the feature.
Also, you may wonder about how safe the patch is. All I can say is that
there is nothing malicious in it, and I am known on this list, but of
course if it breaks I won't be to blame.
Bug reports and other comments are welcome, of course.
If anybody would like the corresponding patch which works on 5.4.n, for n

= 80, that is available, too.

--
Alan Mackenzie (Nuremberg, Germany).