[gentoo-user] qbittorrent crashes after to many files open error

[Dale]

Howdy,

As some know, I discovered torrentting a while back.  It has caused
issues ever since.  LOL  I recently upgraded qbittorrent.  Other than
having to limit some speed settings since it would make my desktop
response slow, it has worked OK, better than ktorrent at least.  Then a
couple days ago, perhaps related to a upgrade, it would crash.  In the
notifications I would find a error like below.  I have changed the names
to protect the innocent.  ;-)

An I/O error occurred for torrent 'ABCDEF'.
Reason: ABCDEF file_open (/home/dale/Desktop/Videos/ABCDEF error: Too
many open files


I did a google search and found out more info which lead me to this
eventually: 


https://www.tecmint.com/increase-set-open-file-limits-in-linux/

Then this:

***@fireball / # sysctl fs.file-max
fs.file-max = 3289952
***@fireball / # sysctl -w fs.file-max=32899520
fs.file-max = 32899520
***@fireball / # cat /proc/sys/fs/file-max
32899520


So, I increased the limit on open files by a factor of ten, I added a
zero on the end.  It was easy enough and didn't require a calculator or
other fancy maths.  Question is, is this a better fix or could it just
be qbittorrent itself having issues?  Could there be more to this or
something else causing this error?

I was using qbittorrent-4.4.5 but downgraded to qbittorrent-4.4.4.  I'm
hoping one or the other will fix this crashing issue.  It did only start
after the upgrade but could be a coincidence to I guess. 


By the way, the 10TB drive I bought a couple weeks or so ago, well, this
is it now. 

/dev/mapper/10tb           9.1T  8.7T  345G  97% /mnt/10tb


The 14TB is supposed to be here this weekend.  I have really got to
create a solution to this.  My current plan, make the 14TB my backup
drive.  Put the 10TB in my rig, for now.  It's getting full too.  ROFL 
I could cut off the internet I guess.  ROFLMBO


Thoughts on the files open error?  Qbittorrent crashing?  Anyone else
ran into this before?  Proper solution?  Better solution? 


Dale

:-)  :-) 

[Mateusz Okulus]

Post by Dale
fs.file-max = 3289952
fs.file-max = 32899520
32899520

On my pc the limit is 2^63. But this is max files for kernel, which is
different for max files per user and max files per process.

For example if I run:

$ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
Limit Soft Limit Hard Limit Units
Max open files 524288 524288 files

524288 = 2^19

QBittorrent increases it's process limit to hard limit. With 10 TB of
torrents it's hard to tell if it's a bug or just 500k files is not
enough, I'd say it's probably the latter.

It seems you can change the default hard limit in
/etc/security/limits.conf

For example:

# 2^24, increased from 2^19
* hard nofile 16777216

Then after reloging:

$ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
Limit Soft Limit Hard Limit Units
Max open files 16777216 16777216 files

So this works, at least for me.

Check also limits.conf man page.

Regards,
mmokulus

[Dale]

Post by Mateusz Okulus

Post by Dale
fs.file-max = 3289952
fs.file-max = 32899520
32899520

On my pc the limit is 2^63. But this is max files for kernel, which is
different for max files per user and max files per process.
$ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
Limit Soft Limit Hard Limit Units
Max open files 524288 524288 files
524288 = 2^19
QBittorrent increases it's process limit to hard limit. With 10 TB of
torrents it's hard to tell if it's a bug or just 500k files is not
enough, I'd say it's probably the latter.
It seems you can change the default hard limit in
/etc/security/limits.conf
# 2^24, increased from 2^19
* hard nofile 16777216
$ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
Limit Soft Limit Hard Limit Units
Max open files 16777216 16777216 files
So this works, at least for me.
Check also limits.conf man page.
Regards,
mmokulus
.

It appears the change I made wasn't what was needed.  I took a nap,
after doing today what I used to do on Friday, which involves going to
Doctor, getting shots, then shopping around town etc etc.  Anyway, I
woke up to Qbittorrent being gone.  It crashed at some point.  In the
notification thingy, it had a few of the file open errors again.  So, I
ran some of the commands you posted, my hard limit was set to 4096.  I
whipped out my calculator and doubled it.  I set it in the limits.conf
file like so: 

***@fireball / # cat /etc/security/limits.conf | grep nofile
#        - nofile - max number of open file descriptors
*               hard    nofile          8192
***@fireball / #


I logged out and back in.  I ran the command you shared and it was
indeed set to the new amount.  I'm going to upgrade qbittorrent to the
new version again and test it some more. 

Thanks much for the help.  Maybe this will fix it.

Dale

:-)  :-) 

[m***@tutanota.com]

I'd suggest looking at the "advanced" section of the preferences.  Lots of way to limit/expand memory use etc.  You really should also check all the other Qbit preferences, there are a lot.  For example you can limit number of I/O threads, block multiple connections from the same host (this prevents some known abuses related to your' client.

--"Fascism begins the moment a ruling class, fearing the people may use their political democracy to gain economic democracy, begins to destroy political democracy in order to retain its power of exploitation and special privilege." Tommy Douglas

Post by Dale

Post by Mateusz Okulus

Post by Dale
fs.file-max = 3289952
fs.file-max = 32899520
32899520

On my pc the limit is 2^63. But this is max files for kernel, which is
different for max files per user and max files per process.
$ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
Limit Soft Limit Hard Limit Units
Max open files 524288 524288 files
524288 = 2^19
QBittorrent increases it's process limit to hard limit. With 10 TB of
torrents it's hard to tell if it's a bug or just 500k files is not
enough, I'd say it's probably the latter.
It seems you can change the default hard limit in
/etc/security/limits.conf
# 2^24, increased from 2^19
* hard nofile 16777216
$ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
Limit Soft Limit Hard Limit Units
Max open files 16777216 16777216 files
So this works, at least for me.
Check also limits.conf man page.
Regards,
mmokulus
.

It appears the change I made wasn't what was needed.  I took a nap,
after doing today what I used to do on Friday, which involves going to
Doctor, getting shots, then shopping around town etc etc.  Anyway, I
woke up to Qbittorrent being gone.  It crashed at some point.  In the
notification thingy, it had a few of the file open errors again.  So, I
ran some of the commands you posted, my hard limit was set to 4096.  I
whipped out my calculator and doubled it.  I set it in the limits.conf
file like so: 
#        - nofile - max number of open file descriptors
*               hard    nofile          8192
I logged out and back in.  I ran the command you shared and it was
indeed set to the new amount.  I'm going to upgrade qbittorrent to the
new version again and test it some more. 
Thanks much for the help.  Maybe this will fix it.
Dale
:-)  :-) 

[Dale]

Post by m***@tutanota.com
I'd suggest looking at the "advanced" section of the preferences.  Lots of way to limit/expand memory use etc.  You really should also check all the other Qbit preferences, there are a lot.  For example you can limit number of I/O threads, block multiple connections from the same host (this prevents some known abuses related to your' client.
--"Fascism begins the moment a ruling class, fearing the people may use their political democracy to gain economic democracy, begins to destroy political democracy in order to retain its power of exploitation and special privilege." Tommy Douglas

I've looked in there and even looked at the help pages it links to.  I
think everything there is set as it should, for what I understand
anyway.  I did change a couple really early on but those were changes
others made to make it work better.  I'm not sure they would even be
related to open file things. 

I wish I could recall what I changed now.  :/

Dale

:-)  :-) 

[Mateusz Okulus]

Post by Dale
#        - nofile - max number of open file descriptors
*               hard    nofile          8192
I logged out and back in.  I ran the command you shared and it was
indeed set to the new amount.  I'm going to upgrade qbittorrent to the
new version again and test it some more. 

I don't think 8k opened files will be enough for 10TB of torrents. Keep
in mind this is more for multi user purpose, in case someone writes and
uses a buggy program, so the whole system won't crash for other users.
If you just have single user computer it's safe to just set it to the
max limit.

I'm certain it's not a bug in qbittorrent, but rather this limit is just
very low by default for you, as I've mentioned for me it was 2^19, and
I'm on 512GB laptop.

Regards,
mmokulus

[Dale]

Post by Mateusz Okulus

Post by Dale
#        - nofile - max number of open file descriptors
*               hard    nofile          8192
I logged out and back in.  I ran the command you shared and it was
indeed set to the new amount.  I'm going to upgrade qbittorrent to the
new version again and test it some more. 

I don't think 8k opened files will be enough for 10TB of torrents. Keep
in mind this is more for multi user purpose, in case someone writes and
uses a buggy program, so the whole system won't crash for other users.
If you just have single user computer it's safe to just set it to the
max limit.
I'm certain it's not a bug in qbittorrent, but rather this limit is just
very low by default for you, as I've mentioned for me it was 2^19, and
I'm on 512GB laptop.
Regards,
mmokulus

You make a excellent point.  While I have not had this error since the
change I made, I do have large storage media here.  My crypt mount point
is two 8TB hard drives on LVM.  The drive that qbit uses to store files
on is a 6TB drive.  Not including the drive for the OS itself, I have
20TBs of drive space.  Those will be changing soon as I grow things
here.  Since that setting likely affects things not even related to
qbittorrent, these larger storage drives can result in reducing what
file qbit can have open. 

I'm going to up that setting a bit shortly.  I don't think it would hurt
anything given I have lots of memory and a pretty fast CPU so it
shouldn't cause anything to fail due to lack of memory or CPU power.  I
don't see a downside to this change. 

Thanks much for the help. 

Dale

:-)  :-) 

[Lee]

Divine punishment perhaps?

Lee 😎

Post by Dale
Howdy,
As some know, I discovered torrentting a while back. It has caused
issues ever since. LOL I recently upgraded qbittorrent. Other than
having to limit some speed settings since it would make my desktop
response slow, it has worked OK, better than ktorrent at least. Then a
couple days ago, perhaps related to a upgrade, it would crash. In the
notifications I would find a error like below. I have changed the names
to protect the innocent. ;-)
An I/O error occurred for torrent 'ABCDEF'.
Reason: ABCDEF file_open (/home/dale/Desktop/Videos/ABCDEF error: Too
many open files
I did a google search and found out more info which lead me to this
https://www.tecmint.com/increase-set-open-file-limits-in-linux/
fs.file-max = 3289952
fs.file-max = 32899520
32899520
So, I increased the limit on open files by a factor of ten, I added a
zero on the end. It was easy enough and didn't require a calculator or
other fancy maths. Question is, is this a better fix or could it just
be qbittorrent itself having issues? Could there be more to this or
something else causing this error?
I was using qbittorrent-4.4.5 but downgraded to qbittorrent-4.4.4. I'm
hoping one or the other will fix this crashing issue. It did only start
after the upgrade but could be a coincidence to I guess.
By the way, the 10TB drive I bought a couple weeks or so ago, well, this
is it now.
/dev/mapper/10tb 9.1T 8.7T 345G 97% /mnt/10tb
The 14TB is supposed to be here this weekend. I have really got to
create a solution to this. My current plan, make the 14TB my backup
drive. Put the 10TB in my rig, for now. It's getting full too. ROFL
I could cut off the internet I guess. ROFLMBO
Thoughts on the files open error? Qbittorrent crashing? Anyone else
ran into this before? Proper solution? Better solution?
Dale
:-) :-)