FreeBSD Network Tuning

From Hyades
Jump to: navigation, search

The default installation of FreeBSD is pretty good and will work well for typical system. However, the default settings appear to be selected as to achieve good performance on Gigabit or slower network hardware. To achieve optimal performance on 10GE or faster network cards, one must tune, tweak and optimize FreeBSD's network stack.

There are a few excellent guides on tuning FreeBSD on the internet[1][2][3][4]. In this article, we document our tweaks of the 2 FreeBSD systems, Ambrosia & Coronis. Each system is equipped with a dual-port Intel X520-DA2 10 Gigabit Network Adapter and a dual-port Intel I350 Gigabit Ethernet Controller.

defaults

Here we collect and tabulate the default values of selected kernel parameters that are relevant to network performance. One can immediately see that many default values have been significantly increased in FreeBSD 10.x.

parameter FreeBSD 9.2 FreeBSD 10.1
kern.maxvnodes 1095802 3339551
kern.maxproc 6164 201940
kern.maxfiles 12328 6289573
kern.maxusers 384 12620
kern.ipc.nmbclusters 25600 12255534
kern.ipc.nmbjumbop 12800 6127766
kern.ipc.nmbjumbo9 6400 5446902
kern.ipc.nmbjumbo16 3200 4085176
kern.ipc.maxsockets 25600 6289575
kern.ipc.maxsockbuf 2097152 2097152
kern.ipc.somaxconn 128 2048
net.inet.tcp.sendbuf_auto 1 1
net.inet.tcp.recvbuf_auto 1 1
net.inet.tcp.sendspace 32768 32768
net.inet.tcp.recvspace 65536 65536
net.inet.tcp.sendbuf_max 2097152 2097152
net.inet.tcp.recvbuf_max 2097152 2097152
net.inet.tcp.sendbuf_inc 8192 8192
net.inet.tcp.recvbuf_inc 16384 16384
net.inet.tcp.tcbhashsize 32768 2097152
net.inet.ip.intr_queue_maxlen 256 256
net.route.netisr_maxqlen 256 256
hw.igb.max_interrupt_rate 8000 8000
hw.ix.max_interrupt_rate 31250 31250
hw.igb.rxd 1024 1024
hw.igb.txd 1024 1024
hw.ix.txd 2048 2048
hw.ix.rxd 2048 2048
hw.igb.num_queues 0 0
hw.ix.num_queues 8 8

/boot/loader.conf

The file loader.conf contains descriptive information on bootstrapping the system. Through it one can specify the kernel to be booted, parameters to be passed to it, and additional modules to be loaded. The defaults are specified in /boot/defaults/loader.conf.

/boot/loader.conf for Ambrosia (running FreeBSD 9.3):

kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbop=262144
kern.ipc.nmbjumbo9=65536
kern.ipc.nmbjumbo16=32768
kern.maxusers=1024
# number of hash table buckets to handle incoming TCP connections
net.inet.tcp.tcbhashsize=65536
# H-TCP optimized congestion control algorithm for high speed networks with high latency
cc_htcp_load="YES"
zfs_load="YES"

/boot/loader.conf for Coronis (running FreeBSD 10.1):

# H-TCP optimized congestion control algorithm for high speed networks with high latency
cc_htcp_load="YES"
zfs_load="YES"

Note:

  1. The default values for parameters controlling the memory buffers mbufs and mbuf clusters in FreeBSD 9.x are too low and must be adjusted
  2. One can run netstat -m to show statistics on memory buffers
  3. The default TCP congestion-avoidance algorithm in FreeBSD is TCP New Reno.
  4. CUBIC and HTCP are better congestion control algorithms. We'll use HTCP, so we load the kernel module cc_htcp.ko on system startup.

/etc/sysctl.conf

The file sysctl.conf is read in when the system goes into multi-user mode to set default settings for the kernel. It is the primary tuning & optimization file for FreeBSD.

/etc/sysctl.conf for Ambrosia (running FreeBSD 9.3):

# $FreeBSD: releng/9.3/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $
#
kern.maxfiles=51200
kern.ipc.somaxconn=2048
# set to at least 16MB for 10GE hosts
kern.ipc.maxsockbuf=16777216
# socket buffers
net.inet.tcp.recvspace=4194304
net.inet.tcp.sendspace=2097152
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
# security
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
# drop UDP packets destined for closed sockets
net.inet.udp.blackhole=1
# drop TCP packets destined for closed sockets
net.inet.tcp.blackhole=2
# ipfw
net.inet.ip.fw.verbose_limit=3
# H-TCP congestion control algorithm
net.inet.tcp.cc.algorithm=htcp
# maximum incoming and outgoing IPv4 network queue sizes
net.inet.ip.intr_queue_maxlen=2048
net.route.netisr_maxqlen=2048
# net.inet.tcp.tso=0
# net.bpf.optimize_writers=1

/etc/sysctl.conf for Coronis (running FreeBSD 10.1):

# $FreeBSD: releng/10.1/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $
#
# Ensure ZFS uses 4k block size
vfs.zfs.min_auto_ashift=12
kern.ipc.somaxconn=2048
# set to at least 16MB for 10GE hosts
kern.ipc.maxsockbuf=16777216
# socket buffers
net.inet.tcp.recvspace=4194304
net.inet.tcp.sendspace=2097152
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
# security
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
# drop UDP packets destined for closed sockets
net.inet.udp.blackhole=1
# drop TCP packets destined for closed sockets
net.inet.tcp.blackhole=2
# ipfw
net.inet.ip.fw.verbose_limit=3
# H-TCP congestion control algorithm
net.inet.tcp.cc.algorithm=htcp
# maximum incoming and outgoing IPv4 network queue sizes
net.inet.ip.intr_queue_maxlen=2048
net.route.netisr_maxqlen=2048

Note the default values for parameters controlling socket buffers are too low for 10GE network cards, and must be adjusted.

References

  1. FreeBSD wiki - What need to be done to tune networking stack (WIP)
  2. ESnet - FreeBSD Tuning
  3. Calomel - FreeBSD Tuning and Optimization
  4. PSC - Enabling High Performance Data Transfers