Bandwidth and Latency Testing (BLT) Protocol
Latency, packet loss, and low bandwidth related bugs are a major problem in Second Life. One huge barrier to fixing these bugs is a lack of testing protocol to reproduce these bugs that will rarely show up for someone who lives in San Francisco, or even in America. The BLT Protocol aims to create a standardized set of test conditions that can be referred to in reproductions, and tested against during QA.
This meta-bug contains all bugs that might be related to latency, bandwidth, or packet loss:
Reproducing a bug
If you confirm a bug under one of the test scenarios listed below, the following format should be used in the reproduction in a Jira comment:
Incoming Latency: Outgoing Latency: Packet Loss Parameters:
Followed by the step-by-step reproduction of the bug, under the BLT conditions listed.
- A partial fix should be noted as such in the Jira comments, along with which scenarios still fail.
- High percentage reproductions of a bug justify placing that bug on the upcoming triage schedule, if it is not already imported into private Jira.
Testing is done with the linux kernel traffic shaping and WAN emulation modules.
Note that you will be missing the "ifb" module on Ubuntu boxes, unless you have 7.10. They just screwed up and forgot to compile it for several versions, it's a standard kernel module that should be compiled on most distros.
For scanarios that require incoming packet adjustments:
modprobe ifb ip link set dev ifb0 up tc qdisc add dev eth0 ingress tc filter add dev eth0 parent ffff: \ protocol ip u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb0
Replace eth0 with your interface. This is required for any scenario that mentions "ifb0"
NOTE: wherever it says "add dev" replace that with "change dev" if you are already running one scenario.
Alternate Test Setups
These alternate test setups may be used to reproduce bugs also, but the results may or may not correspond exactly to netem setups:
One alternative to consider for those not inclined to set up a Linux box is m0n0wall, which is a nice distribution which includes web administration of DummyNet. The nifty thing about m0n0wall is that it can be run in two configurations that are potentially easier for Windows developers:
- As a VMWare image, which can be run with the free (as in beer) VMWare player
- with a cheap (~$250), low-power/profile Soekris box for those that are going to do a lot of testing and don't want to bog down their main machine. Not to mention, the Soekris boxes are fun toys for many other uses.
NOTE: wherever it says "change dev" below replace that with "add dev" if it is your first scenario. Change eth0 to reflect whatever interface you use.
- Asymmetric Latency
tc qdisc change dev eth0 root netem delay 300ms
This adds 300ms of latency on top of whatever was already there. This is an asymmetric latency, it is only on outgoing packets.
- Symmetric latency.
tc qdisc change dev eth0 root netem delay 500ms tc qdisc change dev ifb0 root netem delay 500ms
This is good for reproducing a bug when all else fails. Try this one first, then try the lesser scenarios. If the lesser scenarios still repro it, mention them instead.
150ms symmetric is similar to some far-flung international user's connections. Note that due to bandwidth*delay product, this scenario also constrains TCP bandwidth in nearly all cases. MS Vista, with dynamic window sizing up to 8megs, and linux, with generally better window settings, may perform better than other OS (especially old windows, which had terrible window settings) on this scenario, so beware.
Symmetric 75ms latency is similar to europe (or SF to East Coast), symmetric 40ms latency is similar to East Coast to TX.
- Variable Latency
tc qdisc change dev eth0 root netem delay 200ms 200ms 50% tc qdisc change dev ifb0 root netem delay 200ms 200ms 50%
This adds variable round trip latency from 0-400ms.
- Emulated Satellite internet
tc qdisc change dev eth0 root netem delay 350ms 50ms 50% tc qdisc change dev ifb0 root netem delay 350ms 50ms 50% tc qdisc change dev eth0 root netem loss 0.3% 33.33% tc qdisc change dev ifb0 root netem loss 0.3% 33.33%
In reality, satellite internet isn't THIS bad, because they do use TCP acceleration which converts TCP into a sort of UDP-esqe stream to avoid bandwidth-delay product problems that severely constrain bandwidth in this scenario. The latency and loss figures, are however accurate.
- Packet Loss
Low bursty packet loss
tc qdisc change dev eth0 root netem loss 0.3% 33.33% tc qdisc change dev ifb0 root netem loss 0.3% 33.33%
High bursty packet loss
tc qdisc change dev eth0 root netem loss 3.0% 33.33% tc qdisc change dev ifb0 root netem loss 3.0% 33.33%
Insane packet loss
tc qdisc change dev eth0 root netem loss 15.0% 33.33% tc qdisc change dev ifb0 root netem loss 15.0% 33.33%