[gentoo-user] Network throughput from main Gentoo rig to NAS box.

[Mark Knecht]

On Sat, Sep 23, 2023 at 5:05 AM Dale <***@gmail.com> wrote:
<SNIP>

If you need more info, let me know. If you know the command, that might
help too. Just in case it is a command I'm not familiar with.
Thanks.
Dale
:-) :-)

You can use the iperf command to do simple raw speed testing.

For instance, on your server open a terminal through ssh and run

iperf -s

It should tell you the server is listening.

On your desktop machine run

iperf -c 192.168.86.119

(replace with the IP of your server)

It runs for 5-10 seconds and then reports what it sees
as throughput.

Remember to Ctrl-C the server side when you're done.

HTH,
Mark

[Dale]

Post by Mark Knecht
<SNIP>

If you need more info, let me know.  If you know the command, that might
help too.  Just in case it is a command I'm not familiar with.
Thanks.
Dale
:-)  :-)

You can use the iperf command to do simple raw speed testing.
For instance, on your server open a terminal through ssh and run
iperf -s
It should tell you the server is listening.
On your desktop machine run
iperf -c 192.168.86.119
(replace with the IP of your server)
It runs for 5-10 seconds and then reports what it sees 
as throughput.
Remember to Ctrl-C the server side when you're done.
HTH,
Mark

I had to install those.  On Gentoo it's called iperf3 but it works. 
Anyway, this is what I get from running the command on the NAS box to my
main rig. 


***@nas:~# iperf -c 10.0.0.4
tcp connect failed: Connection refused
------------------------------------------------------------
Client connecting to 10.0.0.4, TCP port 5001
TCP window size: -1.00 Byte (default)
------------------------------------------------------------
[  1] local 0.0.0.0 port 0 connected with 10.0.0.4 port 5001
***@nas:~#


This is when I try to run from my main rig to the NAS box. 


***@fireball / # iperf3 -c 10.0.0.7
iperf3: error - unable to connect to server - server may have stopped
running or use a different port, firewall issue, etc.: Connection refused
***@fireball / #


I took what you said to mean to run from the NAS box.  I tried both just
in case I misunderstood your meaning by server.  ;-)

Ideas?

Dale

:-)  :-) 

That pepper sauce is getting loud.  '_O

[Mark Knecht]

Post by Mark Knecht
<SNIP>

If you need more info, let me know. If you know the command, that might
help too. Just in case it is a command I'm not familiar with.
Thanks.
Dale
:-) :-)

You can use the iperf command to do simple raw speed testing.
For instance, on your server open a terminal through ssh and run
iperf -s
It should tell you the server is listening.
On your desktop machine run
iperf -c 192.168.86.119
(replace with the IP of your server)
It runs for 5-10 seconds and then reports what it sees
as throughput.
Remember to Ctrl-C the server side when you're done.
HTH,
Mark
I had to install those. On Gentoo it's called iperf3 but it works.

Anyway, this is what I get from running the command on the NAS box to my
main rig.

Post by Mark Knecht
tcp connect failed: Connection refused
------------------------------------------------------------
Client connecting to 10.0.0.4, TCP port 5001
TCP window size: -1.00 Byte (default)
------------------------------------------------------------
[ 1] local 0.0.0.0 port 0 connected with 10.0.0.4 port 5001
This is when I try to run from my main rig to the NAS box.
iperf3: error - unable to connect to server - server may have stopped

running or use a different port, firewall issue, etc.: Connection refused

Post by Mark Knecht
I took what you said to mean to run from the NAS box. I tried both just

in case I misunderstood your meaning by server. ;-)

Post by Mark Knecht
Ideas?
Dale

I thought the instructions were clear but let's try again.

When using iperf YOU have to set up BOTH ends of the path, so:

1) On one end - let's say it's your NAS server - open a terminal. In that
terminal type

***@plex:~$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------

2) Then, on your desktop machine that wants to talk to the NAS server type
this command,
replacing my service IP with your NAS server IP

***@science2:~$ iperf -c 192.168.86.119
------------------------------------------------------------
Client connecting to 192.168.86.119, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.86.43 port 40320 connected with 192.168.86.119 port
5001
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-10.0808 sec 426 MBytes 354 Mbits/sec
***@science2:~$

In this case, over my wireless network, I'm getting about 354Mb/S. Last time
I checked it I hooked a cable between the 2 rooms I got about 900Mb/s.

[Dale]

Post by Dale

Post by Mark Knecht
<SNIP>

If you need more info, let me know.  If you know the command, that

might

Post by Mark Knecht

help too.  Just in case it is a command I'm not familiar with.
Thanks.
Dale
:-)  :-)

You can use the iperf command to do simple raw speed testing.
For instance, on your server open a terminal through ssh and run
iperf -s
It should tell you the server is listening.
On your desktop machine run
iperf -c 192.168.86.119
(replace with the IP of your server)
It runs for 5-10 seconds and then reports what it sees
as throughput.
Remember to Ctrl-C the server side when you're done.
HTH,
Mark
I had to install those.  On Gentoo it's called iperf3 but it works. 

Anyway, this is what I get from running the command on the NAS box to
my main rig.

Post by Mark Knecht
tcp connect failed: Connection refused
------------------------------------------------------------
Client connecting to 10.0.0.4, TCP port 5001
TCP window size: -1.00 Byte (default)
------------------------------------------------------------
[  1] local 0.0.0.0 port 0 connected with 10.0.0.4 port 5001
This is when I try to run from my main rig to the NAS box.
iperf3: error - unable to connect to server - server may have

Connection refused

Post by Mark Knecht
I took what you said to mean to run from the NAS box.  I tried both

just in case I misunderstood your meaning by server.  ;-)

Post by Mark Knecht
Ideas?
Dale

I thought the instructions were clear but let's try again.
1) On one end - let's say it's your NAS server - open a terminal. In
that terminal type
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
2) Then, on your desktop machine that wants to talk to the NAS server
type this command,
replacing my service IP with your NAS server IP
------------------------------------------------------------
Client connecting to 192.168.86.119, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.86.43 port 40320 connected with 192.168.86.119
port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0808 sec   426 MBytes   354 Mbits/sec
In this case, over my wireless network, I'm getting about 354Mb/S. Last time
I checked it I hooked a cable between the 2 rooms I got about 900Mb/s.

Oh.  My pepper sauce was getting loud and my eyes were watery.  Now that
I got that done, I can see better after opening the doors a few
minutes.  This is what I get now.  My NAS box, running it first: 

***@nas:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------


From main rig, running NAS box command first and it appeared to be waiting.


***@fireball / # iperf3 -c 10.0.0.7
iperf3: error - unable to connect to server - server may have stopped
running or use a different port, firewall issue, etc.: Connection refused
***@fireball / #


So, it appears to be waiting but my main rig isn't getting it.  Then it
occurred my VPN might be affecting this somehow.  I stopped it just in
case.  OK, same thing.  I did run the one on the NAS box first, since I
assume it needs to be listening when I run the command on my main rig. 
After stopping the VPN, I ran both again. 

Just so you know the machine is reachable, I am ssh'd into the NAS box
and I also have it mounted and copying files over with rsync.  Could my
router be blocking this connection?  I kinda leave it at the default
settings.  Read somewhere those are fairly secure. 

I'm working in garden a bit so may come and go at times.  I'm sure you
doing other things too.  :-D 

Dale

:-)  :-)

[Mark Knecht]

<SNIP>

Oh. My pepper sauce was getting loud and my eyes were watery. Now that

I got that done, I can see better after opening the doors a few minutes.

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
From main rig, running NAS box command first and it appeared to be waiting.
iperf3: error - unable to connect to server - server may have stopped

running or use a different port, firewall issue, etc.: Connection refused

So, it appears to be waiting but my main rig isn't getting it. Then it

occurred my VPN might be affecting this somehow. I stopped it just in
case. OK, same thing. I did run the one on the NAS box first, since I
assume it needs to be listening when I run the command on my main rig.
After stopping the VPN, I ran both again.

Just so you know the machine is reachable, I am ssh'd into the NAS box

and I also have it mounted and copying files over with rsync. Could my
router be blocking this connection? I kinda leave it at the default
settings. Read somewhere those are fairly secure.

I'm working in garden a bit so may come and go at times. I'm sure you

doing other things too. :-D

Dale
:-) :-)

If you're running a VPN then you'll need someone at a higher pay grade than
me, but packetizing TCP packets into and out of VPN security is certainly
going to use CPU cycles and slow things down, at least a little. No idea if
that's what caused your gkrellm pictures. Also, any network heavy apps,
like playing video from the NAS or from the Internet is also going to slow
file transfers down.

Your error message is telling you that something is in the way.

Can you ping the NAS box?

ping 10.0.0.7

Can you tracepath the NAS box?

tracepath 10.0.0.7

Are you sure that 10.0.0.7 is the address of the NAS box?

Do you have a /etc/hosts file to keep the names straight?

HTH,
Mark

[Frank Steinmetzger]

I read the other replies and I think it is caching the data, the drives
writes and catches up and then it asks for more data again.

Tool tip: dstat

It puts out one line of values every x seconds (x == 1 by default).
With arguments you can tell it what to show. To see disks in action, I like
to run the following during upgrades that involve volumous packages:

dstat --time --cpu --disk -D <comma-list of disks you want to monitor> --net --mem-adv --swap'

The cpu column includes IO wait.

The disk columns show read and write volume. If you omit the -D option, you
will only see a total over all disks, which might still be enough for your
use case.

The mem-adv shows how much data is in the file system write cache (the --mem
option does not).

[ralfconn]

Howdy,
As most everyone knows, I redone my NAS box.  Before I had Truenas on it
but switched to Ubuntu server thingy called Jimmy.  Kinda like the
name.  lol  Anyway, Ubuntu has the same odd transfer pattern as the
Truenas box had.  I'm not sure if the problem is on the Gentoo end or
the Ubuntu end or something else.  I'm attaching a picture of Gkrellm so
you can see what I'm talking about.  It transfers a bit, then seems to
stop for some reason, then start up again and this repeats over and
over.  I'm expecting more of a consistent throughput instead of all the
idle time.  The final throughput is only around 29.32MB/s according to
info from rsync.  If it was not stopping all the time and passing data
through all the time, I think that would improve.  Might even double.
...
Has anyone ever seen something like this and know why it is idle for so
much of the time?  Anyone know if this can be fixed so that it is more
consistent, and hopefully faster?

I found a similar pattern when I checked some time ago, while
transferring big (several Gb) files from one desktop to the other. I
concluded the cause of the gaps was the destination PC's SATA spinning
disk that needed to empty its cache before accepting more data. In
theory the network is 1Gb/s (measured with iperf, it is really close to
that) and the SATA is 6Gb/s so it should not be the limit, but I have
strong doubts as how this speed is measured by the manufacturer. As
indirect confirmation, when I switched the destination disk to a flash
one the overall network transfer speed improved a lot (a lot lot!).

raffaele

[Frank Steinmetzger]

Howdy,
As most everyone knows, I redone my NAS box.  Before I had Truenas on it
but switched to Ubuntu server thingy called Jimmy.  Kinda like the
name.  lol  Anyway, Ubuntu has the same odd transfer pattern as the
Truenas box had.  I'm not sure if the problem is on the Gentoo end or
the Ubuntu end or something else.  I'm attaching a picture of Gkrellm so
you can see what I'm talking about.  It transfers a bit, then seems to
stop for some reason, then start up again and this repeats over and
over.  I'm expecting more of a consistent throughput instead of all the
idle time.  The final throughput is only around 29.32MB/s according to
info from rsync.  If it was not stopping all the time and passing data
through all the time, I think that would improve.  Might even double.
...
Has anyone ever seen something like this and know why it is idle for so
much of the time?  Anyone know if this can be fixed so that it is more
consistent, and hopefully faster?

I found a similar pattern when I checked some time ago, while transferring
big (several Gb) files from one desktop to the other. I concluded the cause
of the gaps was the destination PC's SATA spinning disk that needed to empty
its cache before accepting more data. In theory the network is 1Gb/s
(measured with iperf, it is really close to that) and the SATA is 6Gb/s so
it should not be the limit, but I have strong doubts as how this speed is
measured by the manufacturer.

Please be aware there is a difference between Gb and GB: one is gigabit, the
other gigabyte. 1 Gb/s is theoretically 125 MB/s, and after deducting
network overhead you get around 117 MB/s net bandwidth. Modern 3.5″ HDDs
read more than 200 MB/s in their fastest areas, 2.5″ not so much. In their
slowest region, that can go down to 50..70 MB/s.

[Jack]

Howdy,
As most everyone knows, I redone my NAS box.  Before I had Truenas on it
but switched to Ubuntu server thingy called Jimmy.  Kinda like the
name.  lol  Anyway, Ubuntu has the same odd transfer pattern as the
Truenas box had.  I'm not sure if the problem is on the Gentoo end or
the Ubuntu end or something else.  I'm attaching a picture of Gkrellm so
you can see what I'm talking about.  It transfers a bit, then seems to
stop for some reason, then start up again and this repeats over and
over.  I'm expecting more of a consistent throughput instead of all the
idle time.  The final throughput is only around 29.32MB/s according to
info from rsync.  If it was not stopping all the time and passing data
through all the time, I think that would improve.  Might even double.
A little more info.  The set of drives this is being copied from use LVM
and are encrypted with dm-crypt.  They are also set up the same way on
the NAS box.  I also notice that on the NAS box, using htop, the CPUs
sit at idle for a bit then show heavy use, on Ubuntu about 60 or 70%,
then go back to idle.  This seems to be the same thing I'm seeing on
htop with the data throughput.  One may have something to do with the
other but I don't know what.  I got so much stuff running on my main rig
that I can't really tell if that CPU has the same changes or not.  By
the way, it showed the same when Truenas was on there.  These things are
mounted using nfs.  I don't know if that matters or not.  In case this
is a routing issue, I have a Netgear router with 1GB ports.  This is the
mount -t nfs -o nolock
Has anyone ever seen something like this and know why it is idle for so
much of the time?  Anyone know if this can be fixed so that it is more
consistent, and hopefully faster?
If you need more info, let me know.  If you know the command, that might
help too.  Just in case it is a command I'm not familiar with.

I can't add to what others have suggested to improve the throughput, but
the gkrellm pic you posted tells me something is handling the data in
batches.  enp3s0 (your network interface) gets a peak of activity then
stops while crypt (the disk) has a peak of activity. Rinse and repeat. 
I don't know if this is caused by the program invoked by the command you
issued or by some interaction of different pieces that get called to do
the work.  My guess is that it is reading until it fills some buffer,
then writes it out. (Note, it doesn't matter which device is reading and
which is writing, the two just don't overlap)  If encryption is
involved, it might be that there is actually a encrypt/decrypt which
takes place in between the disk and network flows.  I don't know of any
way to change this, but it might explain why the network transfer rate
is as fast as it can get, but the overall throughput is lower.

[Rich Freeman]

I'm expecting more of a consistent throughput instead of all the
idle time. The final throughput is only around 29.32MB/s according to
info from rsync. If it was not stopping all the time and passing data
through all the time, I think that would improve. Might even double.

Is anything else reading data from the NAS at the same time? The
performance is going to depend on a lot of details you haven't
provided, but anything that reads from a hard disk is going to
significantly drop throughput - probably to levels around what you're
seeing.

That seems like the most likely explanation, assuming you don't have
some older CPU or a 100Mbps network port, or something else like WiFi
in the mix. The bursty behavior is likely due to caching.

--
Rich

[Mark Knecht]

Post by Rich Freeman

I'm expecting more of a consistent throughput instead of all the
idle time. The final throughput is only around 29.32MB/s according to
info from rsync. If it was not stopping all the time and passing data
through all the time, I think that would improve. Might even double.

Is anything else reading data from the NAS at the same time? The
performance is going to depend on a lot of details you haven't
provided, but anything that reads from a hard disk is going to
significantly drop throughput - probably to levels around what you're
seeing.
That seems like the most likely explanation, assuming you don't have
some older CPU or a 100Mbps network port, or something else like WiFi
in the mix. The bursty behavior is likely due to caching.
--
Rich

Let's not forget that Dale also likes to put layers of things on his
drives, LVM & encryption at a minimum. We also don't know anything
about his block sizes at either end of this pipe.

I would think maybe running iotop AND btop on both ends would give some
clues on timing. Is the time when gkrellm is idle due to the host disk not
responding or the target getting flooded with too much data?

- Mark

[Dale]

Post by Mark Knecht

I'm expecting more of a consistent throughput instead of all the
idle time.  The final throughput is only around 29.32MB/s according to
info from rsync.  If it was not stopping all the time and passing data
through all the time, I think that would improve.  Might even double.

Is anything else reading data from the NAS at the same time?  The
performance is going to depend on a lot of details you haven't
provided, but anything that reads from a hard disk is going to
significantly drop throughput - probably to levels around what you're
seeing.
That seems like the most likely explanation, assuming you don't have
some older CPU or a 100Mbps network port, or something else like WiFi
in the mix.  The bursty behavior is likely due to caching.
--
Rich

Let's not forget that Dale also likes to put layers of things on his
drives, LVM & encryption at a minimum. We also don't know anything
about his block sizes at either end of this pipe.
I would think maybe running iotop AND btop on both ends would give some 
clues on timing. Is the time when gkrellm is idle due to the host disk
not 
responding or the target getting flooded with too much data?
- Mark

This is true.  I have LVM on the bottom layer with dm-crypt, cryptsetup,
above that and then the file system.  It's the only way I can have more
than one drive and encrypt it.  Well, there's ZFS but I already been
down that path.  ;-) 

I posted in another reply a picture that shows this same thing happening
even when the copy process is on the same rig and even on the same LV. 
That should rule out network issues.  I think as was pointed out, it is
transferring the data until the cache fills up and then it has to wait
for it to catch up then repeat.  It could be encryption slows that
process down a good bit, it could be LVM, or both.  I do know when I did
a test copy to the NAS box when the drives on the NAS box were not
encrypted, it was a good bit faster, about double or more.  Once I got
LVM and the encryption set up, it was slow again.  Also just like when
using Truenas. 

Anyway, I think the network is ruled out at least.  I'm not sure what
can be done at this point.  If it is a cache or drive can't keep up
issue, we can't fix that.  At least it does work, even if it takes a
while to get there.  :-D  Only took almost two weeks to copy over to my
backups.  ROFL 

Thanks to all for the ideas, help, suggestions etc etc.

Dale

:-)  :-)