[gentoo-user] Openrc service crash

[Michael]

I noticed 'www-servers/gatling' now crashes at boot. When I check with
netstat there is no listening port for connections. rc-service reports
gatling has crashed, there is a PID for it, but it is not listed under ps:

# rc-service gatling status
* status: crashed
# cat /run/gatling.pid
4021
# ps axfu | grep gatling
root 4165 0.0 0.0 6572 2224 pts/1 S+ 13:13 0:00
\_ grep --colour=auto gatling


If I try to start it I get a service already started complain. All I see in
the logs is:

* ip link set dev enp2s0 up
* Configuring enp2s0 for MAC address 00:AA:BB:CC:DD:FF ...
[ ok ]
* Starting ifplugd on enp2s0 ...
* start-stop-daemon: fopen `/run/ifplugd.enp2s0.pid': No such file or
directory
* Detaching to start `/usr/sbin/ifplugd' ...
[ ok ]
* Backgrounding ...
* WARNING: net.enp2s0 has started, but is inactive
* WARNING: gatling will start when net.enp2s0 has started


I think this problem started when I changed this PC's LAN network and it now
has IPv6 as well as IPv4, but this could be a coincidence. The init.d script
contains net as a startup dependency:

depend() {
need net
}

Could it be negotiating IP addresses now takes too long and gatling fails to
initialize and crashes for this reason? It is configured to only listen to an
IPv4 address, so I assume the addition of IPv6 should not really affect it.

Restarting it manually works and the service stays up, until the next reboot.

Any ideas what might be causing this?

[Michael]

Post by Michael
I noticed 'www-servers/gatling' now crashes at boot. When I check with
netstat there is no listening port for connections. rc-service reports
# rc-service gatling status
* status: crashed
# cat /run/gatling.pid
4021
# ps axfu | grep gatling
root 4165 0.0 0.0 6572 2224 pts/1 S+ 13:13 0:00
\_ grep --colour=auto gatling
If I try to start it I get a service already started complain. All I see in
* ip link set dev enp2s0 up
* Configuring enp2s0 for MAC address 00:AA:BB:CC:DD:FF ...
[ ok ]
* Starting ifplugd on enp2s0 ...
* start-stop-daemon: fopen `/run/ifplugd.enp2s0.pid': No such file or
directory
* Detaching to start `/usr/sbin/ifplugd' ...
[ ok ]
* Backgrounding ...
* WARNING: net.enp2s0 has started, but is inactive
* WARNING: gatling will start when net.enp2s0 has started
I think this problem started when I changed this PC's LAN network and it now
has IPv6 as well as IPv4, but this could be a coincidence. The init.d
depend() {
need net
}
Could it be negotiating IP addresses now takes too long and gatling fails to
initialize and crashes for this reason? It is configured to only listen to
an IPv4 address, so I assume the addition of IPv6 should not really affect
it.
Restarting it manually works and the service stays up, until the next reboot.
Any ideas what might be causing this?

It was probably the addition of IPv6 - as a workaround I added 'sleep 5s'
before the gatling init.d exec command and it now starts normally.

[Arve Barsnes]

Post by Michael
It was probably the addition of IPv6 - as a workaround I added 'sleep 5s'
before the gatling init.d exec command and it now starts normally.

If it was indeed the network that was missing, you could try changing
the service from "need" to "after" starting the net service.

Add to /etc/rc.conf:
rc_gatling_after="net"

Regards,
Arve

[Michael]

Post by Arve Barsnes

Post by Michael
It was probably the addition of IPv6 - as a workaround I added 'sleep 5s'
before the gatling init.d exec command and it now starts normally.

If it was indeed the network that was missing, you could try changing
the service from "need" to "after" starting the net service.
rc_gatling_after="net"
Regards,
Arve

Thanks Arve, I will try it. Should this replace the "need net" string here,
or added somewhere else?

depend() {
need net
}

[Michael]

Post by Michael

Post by Arve Barsnes

Post by Michael
It was probably the addition of IPv6 - as a workaround I added 'sleep 5s'
before the gatling init.d exec command and it now starts normally.

If it was indeed the network that was missing, you could try changing
the service from "need" to "after" starting the net service.
rc_gatling_after="net"
Regards,
Arve

Thanks Arve, I will try it. Should this replace the "need net" string here,
or added somewhere else?
depend() {
need net
}

Oops! Please ignore my question ... I didn't have enough coffee yet. :-)

[Michael]

Post by Michael

Post by Michael

Post by Arve Barsnes

Post by Michael
It was probably the addition of IPv6 - as a workaround I added 'sleep 5s'
before the gatling init.d exec command and it now starts normally.

If it was indeed the network that was missing, you could try changing
the service from "need" to "after" starting the net service.
rc_gatling_after="net"
Regards,
Arve

Thanks Arve, I will try it. Should this replace the "need net" string
here, or added somewhere else?
depend() {
need net
}

Oops! Please ignore my question ... I didn't have enough coffee yet. :-)

OK, I added rc_gatling_after="net" in rc.conf, but it didn't work. gatling
crashed. However, adding 'sleep 5s' in its init.d startup script works. What
might be causing this discrepancy?

[Arve Barsnes]

Post by Michael
OK, I added rc_gatling_after="net" in rc.conf, but it didn't work. gatling
crashed. However, adding 'sleep 5s' in its init.d startup script works. What
might be causing this discrepancy?

I see now in your first message that the output says that gatling will
wait and start after net.enp2s0 has started, I guess there must be a
crash while waiting. Is there no logs from the daemon? There might be
a way to add a delay in rc.conf as well, so you don't directly edit
the init script.

Regards,
Arve

[Michael]

Post by Arve Barsnes
I see now in your first message that the output says that gatling will
wait and start after net.enp2s0 has started, I guess there must be a
crash while waiting. Is there no logs from the daemon? There might be
a way to add a delay in rc.conf as well, so you don't directly edit
the init script.
Regards,
Arve

The gatling webserver is rather simple with no logging or debugging options
that I have found. The rc.conf log does not capture gatling's internal code
crash, so I'm none the wiser. gatling works nicely once up & running with a
minimal footprint, which is why I chose it when boa fell off the tree. I
suppose commensurate with this is its lack of more sophisticated features,
like backgrounding at launch without crashing. ;-p

The *BSD's rc.conf has a netwait option, but I don't think Gentoo's rc uses
this. Hacking the init script with a simple 'sleep' addition is not the best
approach, but I'm happy it works for now. :-)

[Holger Hoffstätte]

Post by Michael

Post by Michael

Post by Michael

Post by Arve Barsnes

Post by Michael
It was probably the addition of IPv6 - as a workaround I added 'sleep 5s'
before the gatling init.d exec command and it now starts normally.

If it was indeed the network that was missing, you could try changing
the service from "need" to "after" starting the net service.
rc_gatling_after="net"
Regards,
Arve

Thanks Arve, I will try it. Should this replace the "need net" string
here, or added somewhere else?
depend() {
need net
}

Oops! Please ignore my question ... I didn't have enough coffee yet. :-)

OK, I added rc_gatling_after="net" in rc.conf, but it didn't work. gatling
crashed. However, adding 'sleep 5s' in its init.d startup script works. What
might be causing this discrepancy?

Without having more information, I wouldn't be surprised if it's the notorious
IPv6 DAD delay:

https://bugs.gentoo.org/831743

cheers
Holger

[Michael]

Post by Holger Hoffstätte

Post by Michael
OK, I added rc_gatling_after="net" in rc.conf, but it didn't work.
gatling
crashed. However, adding 'sleep 5s' in its init.d startup script works.
What might be causing this discrepancy?

Without having more information, I wouldn't be surprised if it's the
https://bugs.gentoo.org/831743
cheers
Holger

Yes, quite likely. Although this application is configured to listen only on
IPv4, the problem manifested after I moved this system on a new network port
which also has IPv6.

I've come across posts complaining about services crashing at startup, but
causal attribution was somewhat conflated between 1Gb switch gear and IPv6.
Since this PC was on the same 1Gbps switch before and only the addition of
IPv6 coincided with this problem, I'd think the bug you mention is a more
probable cause.