Thursday, August 16, 2012

Configure for high load Linux system

Suggest kernel parameters:

sysctl -w fs.file-max=360000
net.ipv4.ip_local_port_range="1024 65000"
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 262144
net.core.wmem_max = 262144

**//10GB NIC

# increase TCP max buffer size setable using setsockopt()
# 16 MB with a few parallel streams is recommended for most 10G paths
# 32 MB might be needed for some very long end-to-end 10G or 40G paths
net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216 
# increase Linux autotuning TCP buffer limits 
# min, default, and max number of bytes to use
# (only change the 3rd value, and make it 16 MB or more)
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# recommended to increase this for 10G NICS
net.core.netdev_max_backlog = 30000

To change this, you have to edit the file /etc/security/limits.conf as root and make the following changes or add the following lines, respectively:

# To increase the shell limits for <user>
<user> soft nproc 2047
<user> hard nproc 16384
<user> soft nofile 1024
<user> hard nofile 65536

The «soft limit» in the first line defines the number of file handles or open files that the user will have after login. If the user gets error messages about running out of file handles, then the user can increase the number of file handles like in this example up to 63536 («hard limit») by running the following command:

ulimit -n 63536

Note that we do not recommend to set the «hard limit» for nofile for the user equal to/proc/sys/fs/file-max. If you do that and the user uses up all the file handles, then the system would run out of file handles. This could mean that you won't be able to initiate new remote logins any more since the system won't be able to open any PAM modules which are required for performing a login. That's why we set the hard limit to 63536 and not to 65536.

You also need to make sure that pam_limits is configured in the file /etc/pam.d/system-auth. This is the PAM module that will read the /etc/security/limits.conf file. The entry should read like:

session required
session required