Data Backup and Recovery

Too many open files with java agent

clilescapario
3,424 Views

Now that scAgent has changed to being java based we now have the ability to launch multiple backups for different configs at the same time. No more running multiple agents on different ports and staggering your configs! This is great since it cuts down on the amount of memory needed to run the agent(s) and simplifies the SC_AGENT setting across your configs.

One thing you might run into on a linux system is that when running many jobs at the same time through a single agent, you could exhaust the number of file descriptors available and have your backups fail. You can end up with errors that look like

ERROR: SCF-00028: Auto Discovery for plug-in [vibe] failed with error [java.net.SocketException: SocketException invoking https://myschost:9090/SnapCreator/operations/status/blah-blah-blah/blah: Too many open files] and exit code [-1], Exiting!

If you are not very familiar with fd's you can use this simple process to increase your available fd's and try again.

  1. Determine the existing hard limit that scAgent init script will use when starting
    ~ # ulimit -Hn
    1024
    
  2. Determine the system maximum number of open files
    ~ # cat /proc/sys/fs/file-max
    308213
    
  3. Set a new limit somewhere lower than the maximum but greater that the current hard limit by adding the following lines to /etc/security/limits.conf, replace the snapcreator username with the name of the user you are running the agent under (could be root).
    snapcreator          soft    nofile  65536
    snapcreator           hard    nofile  65536
    
  4. Log out and back in so the new limit is picked up.
  5. Restart the scAgent.
  6. Check scAgent log to ensure that there was no warning trying to set the max descriptors on startup. The warning would look like
    Could not set maximum file descriptor limit: $your_max_number_here
    
1 REPLY 1

Clemens_Siebler
3,424 Views

Hey,

thanks for reporting this issue. Would it be possible to check the number of files that the agent currently has open, plus the connection it currently holds? This would be valuable data for further investigation from our side.

Thanks,

Clemens

Public