Bandwidth and Latency Testing (BLT) Protocol

Latency, packet loss, and low bandwidth related bugs are a major problem in Second Life. One huge barrier to fixing these bugs is a lack of testing protocol to reproduce these bugs that will rarely show up for someone who lives in San Francisco, or even in America. The BLT Protocol aims to create a standardized set of test conditions that can be referred to in reproductions, and tested against during QA.

This meta-bug contains all bugs that might be related to latency, bandwidth, or packet loss: MISC-506

Reproducing a bug

If you confirm a bug under one of the test scenarios listed below, the following format should be used in the reproduction in a Jira comment:

BLT# Percentage reproduction ##%

Followed by the step-by-step reproduction of the bug, under the BLT conditions listed.

Rules

If the reproduction percentage is less than 100%, you should try other BLT scenarios. If none are 100%, list all scenarios that reproduced the bug with the percentage for each.

When testing a patch for the bug, every scenario listed on the bug as a reproduction scenario must be tested, and all should be 0% reproduction.

A partial fix should be noted as such in the Jira comments, along with which scenarios still fail.

If you can create a new scenario to reproduce the bug 100%, add that scenario to this page and assign it a BLT number.

High percentage reproductions of a bug justify placing that bug on the upcoming triage schedule, if it is not already imported into private Jira.

Test Setup

Testing is done with the linux kernel traffic shaping and WAN emulation modules.

http://linux-net.osdl.org/index.php/Netem

Note that you will be missing the "tc" module on Ubuntu boxes, unless you get the Gutsy beta release. They just screwed up and forgot to compile it for several versions, it's a standard kernel module that should be compiled on most distros.

For scanarios that require incoming packet adjustments:

modprobe ifb
ip link set dev ifb0 up
tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: \ 
  protocol ip u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb0

Replace eth0 with your interface. This is required for any scenario that mentions "ifb0"

NOTE: wherever it says "add dev" replace that with "change dev" if you are already running one scenario.

BLT Scenarios

NOTE: wherever it says "change dev" below replace that with "add dev" if it is your first scenario.

BLT0

Insane unrealistic latency.

tc qdisc change dev eth0 root netem delay 500ms
tc qdisc change dev ifb0 root netem delay 500ms

This is good for reproducing a bug when all else fails. Try this one first, then try the lesser scenarios. If the lesser scenarios still repro it, use them instead.

BLT1

300ms asymmetric latency

tc qdisc change dev eth0 root netem delay 300ms

Change eth0 to reflect whatever interface you use. This adds 300ms of latency on top of whatever was already there. This is an asymmetric latency, it is only on outgoing packets.

BLT2

Symmetric 150ms latency. (300ms ping)

tc qdisc change dev eth0 root netem delay 150ms
tc qdisc change dev ifb0 root netem delay 150ms

This scenario is similar to some far-flung international user's connections. Note that due to bandwidth*delay product, this scenario also constrains TCP bandwidth in nearly all cases. MS Vista, with dynamic window sizing up to 8megs, and linux, with generally better window settings, may perform better than other OS (especially old windows, which had terrible window settings) on this scenario, so beware.

BLT3

Symmetric 75ms latency. (150ms ping)

tc qdisc change dev eth0 root netem delay 75ms
tc qdisc change dev ifb0 root netem delay 75ms

Watered down version of BLT2. Simulates most of Europe.

BLT4

Symmetric 40ms latency. (80ms ping)

tc qdisc change dev eth0 root netem delay 40ms
tc qdisc change dev ifb0 root netem delay 40ms

Watered down version of BLT3. Simulates East Coast if you are on the West Coast.

BLT5

Variable Latency

tc qdisc change dev eth0 root netem delay 200ms 200ms 50%
tc qdisc change dev ifb0 root netem delay 200ms 200ms 50%

This adds variable round trip latency from 0-400ms.

BLT6

Satellite internet

tc qdisc change dev eth0 root netem delay 350ms 50ms 50%
tc qdisc change dev ifb0 root netem delay 350ms 50ms 50%
tc qdisc change dev eth0 root netem loss 0.3% 33.33%
tc qdisc change dev ifb0 root netem loss 0.3% 33.33%

In reality, satellite internet isn't THIS bad, because they do use TCP acceleration which converts TCP into a sort of UDP-esqe stream to avoid bandwidth-delay product problems that severely constrain bandwidth in this scenario. The latency and loss figures, are however accurate.

BLT7

Low bursty packet loss

tc qdisc change dev eth0 root netem loss 0.3% 33.33%
tc qdisc change dev ifb0 root netem loss 0.3% 33.33%

BLT8

High bursty packet loss

tc qdisc change dev eth0 root netem loss 3.0% 33.33%
tc qdisc change dev ifb0 root netem loss 3.0% 33.33%

BLT9

Insane packet loss

tc qdisc change dev eth0 root netem loss 15.0% 33.33%
tc qdisc change dev ifb0 root netem loss 15.0% 33.33%

BLT

Contents