Re: [openib-general] Re: TSO and IPoIB performance degradation

On Tue, 2006-03-07 at 13:49 -0800, Stephen Hemminger wrote:
On Tue, 07 Mar 2006 13:44:51 -0800
Matt Leininger <mlleinin@xxxxxxxxxxxxxxxxxx> wrote:

On Mon, 2006-03-06 at 19:13 -0800, Shirley Ma wrote:

More likely you are getting hit by the fact that TSO prevents the
congestion
window from increasing properly. This was fixed in 2.6.15 (around mid
of Nov 2005).

Yep, I noticed the same problem. After updating to the new kernel, the
performance are much better, but it's still lower than before.

Here is an updated version of OpenIB IPoIB performance for various
kernels with and without one of the TSO patches. The netperf
performance for the latest kernels has not improved the TSO performance
drop.

Any comments or suggestions would be appreciated.

- Matt

Configuration information? like did you increase the tcp_rmem, tcp_wmem?
Tcpdump traces of what is being sent and available window?
Is IB using NAPI or just doing netif_rx()?

I used the standard setting for tcp_rmem and tcp_wmem. Here are a
few other runs that change those variables. I was able to improve
performance by ~30MB/s to 403 MB/s, but this is still a ways from the
474 MB/s before the TSO patches.

Thanks,

- Matt

All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0
dual EM64T 3.2 GHz PCIe IB HCA (memfull)
patch 1 - remove changeset 314324121f9b94b2ca657a494cf2b9cb0e4a28cc
msi_x=1 for all tests

Kernel OpenIB netperf (MB/s)
2.6.16-rc5 in-kernel 403
tcp_wmem 4096 87380 16777216 tcp_rmem 4096 87380 16777216

2.6.16-rc5 in-kernel 395
tcp_wmem 4096 102400 16777216 tcp_rmem 4096 102400 16777216

2.6.16-rc5 in-kernel 392
tcp_wmem 4096 65536 16777216 tcp_rmem 4096 87380 16777216

2.6.16-rc5 in-kernel 394
tcp_wmem 4096 131072 16777216 tcp_rmem 4096 102400 16777216

2.6.16-rc5 in-kernel 377
tcp_wmem 4096 131072 16777216 tcp_rmem 4096 153600 16777216

2.6.16-rc5 in-kernel 377
tcp_wmem 4096 131072 16777216 tcp_rmem 4096 131072 16777216

2.6.16-rc5 in-kernel 353
tcp_wmem 4096 262144 16777216 tcp_rmem 4096 262144 16777216

2.6.16-rc5 in-kernel 305
tcp_wmem 4096 262144 16777216 tcp_rmem 4096 524288 16777216

2.6.16-rc5 in-kernel 303
tcp_wmem 4096 131072 16777216 tcp_rmem 4096 524288 16777216

2.6.16-rc5 in-kernel 290
tcp_wmem 4096 524288 16777216 tcp_rmem 4096 524288 16777216

2.6.16-rc5 in-kernel 367 default tcp values

--------------------
All with standard tcp settings
Kernel OpenIB netperf (MB/s)
2.6.16-rc5 in-kernel 367
2.6.15 in-kernel 382
2.6.14-rc4 patch 12 in-kernel 436
2.6.14-rc4 patch 1 in-kernel 434
2.6.14-rc4 in-kernel 385
2.6.14-rc3 in-kernel 374
2.6.13.2 svn3627 386
2.6.13.2 patch 1 svn3627 446
2.6.13.2 in-kernel 394
2.6.13-rc3 patch 12 in-kernel 442
2.6.13-rc3 patch 1 in-kernel 450
2.6.13-rc3 in-kernel 395
2.6.12.5-lustre in-kernel 399
2.6.12.5 patch 1 in-kernel 464
2.6.12.5 in-kernel 402
2.6.12 in-kernel 406
2.6.12-rc6 patch 1 in-kernel 470
2.6.12-rc6 in-kernel 407
2.6.12-rc5 in-kernel 405
2.6.12-rc5 patch 1 in-kernel 474
2.6.12-rc4 in-kernel 470
2.6.12-rc3 in-kernel 466
2.6.12-rc2 in-kernel 469
2.6.12-rc1 in-kernel 466
2.6.11 in-kernel 464
2.6.11 svn3687 464
2.6.9-11.ELsmp svn3513 425 (Woody's results, 3.6Ghz EM64T)


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages