Tuning Linux hosts for increased TCP performance

I wrote this for work, but since it's of wider interest...

Linux's TCP behaviour is tuned by editing /etc/sysctl.conf. After altering this file give the command:

# sysctl -p

Set the TCP buffers to be the worst-case bandwidth-delay product. The worst case for a Gigabit ethernet host on AARNet is Perth-Siberia, that is

1Gbps × 500ms =
1×10⁹bps ÷ 8 bits-per-byte × 0.5s =
60MB.

This amount of memory can be reduced, use the ping command to estimate the actual round-trip delay to the sites of interest. Add about 10% to the BDP calculated from ping to allow for the headers attached to kernel buffers.

When calculating the TCP buffers for hosts connected to other ISPs the bottleneck speed is usually the speed of the network's tail, not the speed of the host's interface. For example, to calculate the bandwidth-delay product — and thus the amount of TCP buffering required for full performance — for a 24Mbps ADSL2+ circuit use

24Mbps × 500ms =
24×10⁶bps ÷ 8 bits-per-byte × 0.5s =
1.43MB

The Linux default value for maximum is 4MB, so no setting of the TCP buffer needs to be done for hosts using ADSL2+. Significant performance benefits can still be gained by activating all of the kernel's TCP performance features.

tcp_rmem and tcp_wmem are a vector of <minimum, initial, maximum> bytes of TCP buffer. maximum is the largest amount of memory which will be used for all TCP buffers, set it to the bandwidth-delay product. minimum is the smallest amount memory which will be used for a TCP buffer, set it to one page of memory. inital is the amount of TCP buffer assigned to a new connection, set it to the default value. Large values of initial can use a huge amount of memory for trivial local connections, allowing denial of service.

net.ipv4.tcp_rmem=4096 87380 62500000
net.ipv4.tcp_wmem=4096 16384 62500000

Programs can request a TCP buffer. This isn't recommended in the first instance as is disables Linux's TCP autotuning. These requests are trimmed by rmem_max and by wmem_max. If applications on the host are trusted then set these to be able to use all of the available TCP buffers.

net.core.rmem_max=62500000
net.core.wmem_max=62500000

Ensure basic TCP performance is switched on.

net.ipv4.tcp_moderate_rcvbuf=1
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_window_scaling=1

Many distributions de-tune their TCP performance to work around historical bugs in firewalls. Re-tune these parameters, as for reasonable TCP performance faulty devices will need to be upgraded anyway.

# Cisco PIX: Bug CSCse14419, fix in 7.0(7), 7.1(2.34), 7.2(2.2), 8.0(0.141)
net.ipv4.tcp_sack=1
# Cisco PIX: Bug CSCds23698, fix in 5.1(2.206), 5.1(2.207), 5.2(1.200)
net.ipv4.tcp_ecn=1
# Linux-based firewalls: Bug lwn.net/Articles/92727
# fix in 1.9 of tcp-window-tracking in Netfilter patch-o-matic
net.ipv4.tcp_adv_win_scale=7

Syncookies are a feature to prevent Syn attacks on servers. Unfortunately it also disables TCP timestamps and these are vital for performance. For a full discussion see lwn.net/Articles/277146.

net.ipv4.tcp_syncookies=0

This tuning requires a recent Linux kernel. We recommend recent software for hosts which require high TCP performance. For example, Fedora in preference to RHEL (as RHEL freezes the kernel at the date of release). This recommendation has system administration challenges and may result in a different system design.

Note the large amount of memory needed for TCP buffers. This is a substantial portion of the 512MB of memory available to Linux for all kernel buffers under Intel/AMD 32-bit computers. If substantial disk I/O is also happening then this 512MB will fragment like a crushed eggshell (although less so in more recent kernels). 64-bit CPUs and operating systems have much more memory available for kernel buffers and are recommended for hosts which require high TCP performance and high disk performance.

Unfortunately, there is a wide availability of hardware which is ill-designed for high TCP performance (faulty ethernet chipsets, slow RAID controllers, and so on). Early acceptance testing of hardware is wise.

AARNet are happy to assist computing staff in diagnosing end-to-end network performance issues. In the first instance contact our Network Operations Centre.