Sysadmin miniconf -- TCP tuning

During the Q&A after my talk someone asked what I think of applications accelerators. My reply dissed their lack of operational readiness (a common fault in many middleboxes).

Now that I've had a while to think about TCP acceleration here is a more considered answer. I'll note that I've never used one -- the bandwidth of the networks I use has always exceeded the throughput of middleboxes.

Firstly, a lot of the low-end application accelerators are snake oil. They simply provide another TCP endpoint -- a well implemented and tuned TCP, but not better than you can do by tuning Linux. These make sense if the host's TCP stack is totally borked (Windows<Vista for ≥1Gbps), but an application proxy server would be just as useful.

Secondly, a lot of application accelerators are firewall traversal devices. You put the accelerator outside the firewall and file transfers suddenly work well. But if you had put the host outside the firewall the effect would be the same.

That leaves us with the useful devices. Current TCP algorithms have poor performance at high bandwidth delay-products so inserting a few "TCP proxies" on the path to reduce the delay between TCP endpoints makes some sense. It does change the semantics of TCP though -- an Ack is no longer a solid receipt of delivery, but Acks have been becoming less certain in that regard for a long time (eg, TOEs).

Which brings us back to operational issues. Those TCP proxies hold hard state, so every packet in the connection needs to take the same path. That can be a difficult thing to arrange as the time taken to transfer the data rises. Something, somewhere will break and the data will drop to a fallback path.

We know tracking TCP state opens the device to DoS attacks. This is why we protect firewalls with hardware-based routers which limit floods of Syn packets. Arranging for that sort of protection mid-span is annoying, but possible (generally core routers just route, all the access control to the forwarding plane is done by the edge routers).

Hard state middleboxes also raise tricky network design issues. Implementing network and device redundancy usually requires them to participate in the interior routing, but implementing OSPF or IS-IS isn't commonly seen by middlebox vendors as an important feature. Vendors usually see High Availability keep-alives as adequate, but middleboxes using only this mechanism are difficult to deploy across redundant paths, since the routers can't know if a failing middlebox has compromised the path and will keep forwarding packets into the doomed device until alerted by some slower and difficult to configure mechanism (such as running an iBGP neighbouring bolted down to the path).

There are also financial issues. Adding additional drops to an undersea SDH circuit is expensive. Backhaul out of a landing station is also expensive, but within landing stations only NEBS gear is acceptable. At 10Gbps these environment constraints imply an expensive ASIC-based design for the accelerator, even at 1Gbps a PC-based design would use too much power.

There might be room for a TCP proxy function on the forwarding-plane CPUs of routers. That would be good for a few Gbps, and the backhaul is already paid for. Of course that implies routers on long spans rather than optical solutions to add-drop of intermediate destinations. Bringing us back to finance.

You can see from this discussion that TCP accelerators are unlikely to be widely deployed in the medium term in ISP backbone networks (ie, the networks with the worst BDP issues). If they turn out to be necessary in the longer term (eg, because there can be no TCP-NG algorithm) then ISPs would look to the function being implemented in routers.

Where TCP accelerators make some sense is at satellite ground stations. A geosync satellite has a RTT of 500ms, so an TCP proxy at each ground station would help the average Windows Xp user a lot. A typical satellite channel is 45Mbps so unlike other high BGP paths there isn't a huge bandwidth stressing the design. Ground stations also have huge amounts of available power (unlike cable landing stations they are a lot less geographically constrained and can be placed at a convenient nexus of power and comms).