The Bandwidth Delay Product

I’ve actually needed to perform this calculation in the past, but never knew it had a proper name! The value produced by this simple calculation will tell you how much of your network pipe you can actually fill at any given point in time. To figure this out, you need two values: the available bandwidth, and the latency (or delay) between the two communicating hosts.

In this example, I’ll figure out my BDP between my server in the basement, and this blog.

First, I’ll use ‘ping’ to figure out the “delay” value:

jonesy@canvas:~$ ping www.protocolostomy.com
PING www.protocolostomy.com (74.53.92.66) 56(84) bytes of data
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=1 ttl=252 time=57.2 ms
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=2 ttl=252 time=57.4 ms
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=3 ttl=252 time=57.3 ms
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=4 ttl=252 time=57.3 ms
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=5 ttl=252 time=57.2 ms
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=6 ttl=252 time=57.1 ms
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=7 ttl=252 time=57.0 ms
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=8 ttl=252 time=57.0 ms
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=9 ttl=252 time=56.9 ms
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=10 ttl=252 time=56.8 ms
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=11 ttl=252 time=56.7 ms
64 bytes from web36.webfaction.com (74.53.92.66): icmp_seq=12 ttl=252 time=56.7 ms

This is a nice set of values, all hovering around 57ms. I’ll use 57 as my RTT. For my available bandwidth, I guess I could just use my ISPs advertised speed of 6Mbps upload and 3Mbps download. Let’s see what my BDP looks like in that scenario.

(6,000,000 b/s * 0.57s)  = ~342kb/s

(342 kb/s) / 8 = 42KiB/s

So, if there were no other components involved, I could potentially move 42 kilobytes per second across my connection. However, there are other components involved, most notably the send and receive buffers in the Linux kernel. In kernels prior to 2.6.7, these buffers were really configured by default to strike a balance between good performance for local network connections and low system overhead (e.g. CPU and memory used to process connections and packets). They were not optimized for moving large data sets over long-haul paths. However, more recent kernels now automatically tune the values so that you should receive excellent performance in machines with over 1GB RAM and a BDP of under 4MB. I only just learned that this is the case – I had been searching around for my notes from 2001 about echoing values into files under /proc/sys/net, and using the sysctl variables…. no more!

If you want to see if your system has autotuning enabled, check to see if /proc/sys/net/ipv4/tcp_moderate_rcvbuf is set to “1”, by cat’ing the file. The corresponding ‘sndbuf’ file is essentially irrelevant since sender-side autotuning has been enabled since the early 2.4 kernels.

Of course, I don’t actually see this kind of performance. This was a quick and dirty test using the worst possible tool for the job: scp. Tools like ssh and scp add lots and lots of overhead at various levels due to protocol overhead and (especially) encryption. So given that, my performance really isn’t bad. I’ll see how high I can get it and post my results next week, just for giggles. :-)

  • http://www.standalone-sysadmin.com Matt Simmons

    I had some issues with GbE lan speed a while back. You might find my posts interesting:

    http://standalone-sysadmin.blogspot.com/2008/10/answer-to-slow-speeds-from-last-week.html

    When I took ssh out of the equation, my speeds more than tripled. I now transfer database images with FTP (over a private network, of course and do so in a fraction of the time my rsyncs took.

  • m0j0

    Right. I hope I was clear there about SSH being the worst possible tool for moving data where performance is important at all. I wish I could tell you that the actual project I’m working on is as simple as upgrading the NIC and using FTP. Nope, the machines in question are doing long-haul, multi-terabyte transfers, and there’s already 10GbE capability between the two (geographically dispersed) sites. While I *am* going to be mucking with some network performance tuning, the real bottleneck at this point appears to be disk — or at least the client is resigned to the fact that they’ll need new hardware (which might be a good thing) — but it’s early on in the project and I won’t know more until I return from another project in a couple of weeks. I’ll post as things progress, though, because for sure this will be the most data I’ve ever had to move, over the biggest pipe I’ve ever had access to (well, over the ‘net anyway).

    Thanks for the links, btw!

  • http://www.standalone-sysadmin.com Matt Simmons

    @m0j0

    That is an impressive amount of data / bandwidth to be dealing with. I’m really looking forward to hearing about the rest of it :-) And you’re very welcome!

  • Jonathan Billings

    If you’re concerned about network performance and SSH, and you have control over the SSH server and client, you might want to take a look at the SSH HPN patches: http://www.psc.edu/networking/projects/hpn-ssh/

  • Ray Qadri

    where does your latency factors in you mentioned 57 msec’s
    should it not be total available bandwidth x latency value = ????? / 8 ?

  • Ray Qadri

    Your equation is wrong
    bandwidth-delay product: capacity = bandwidth * RTT

  • bkjones

    Oh my. You’re totally right. I’ve corrected my mistake. Thanks for letting me know. Scary how long this post was here without anyone mentioning it :(