User:Strife Onizuka/Projects/TestHost

Time Table

• 2 Days Understanding requirements and drawing charts
• 2 Days Interface Design
• 18 Days Scripting
• 8 Days Debugging & Finishing

Milestones

• Day	3	–	Rough Script Layout Test <-> Controller Communications Designed Controller <-> Coordinator Communications Designed
• Day	7	–	Coordinator Interface scripted
• Day	12	–	Test <-> Controller Communications 3 Different test scripts written to prove the interface
• Day	15	–	Controller <-> User Interface
• Day	19	–	Controller <-> Coordinator communications
• Day	22	–	Coordinator guts (patching the comms to the interface)
• Day	22	–	Start debugging
• Day	25	–	Finish debugging test scripts
• Day	27	–	Finish debugging controller
• Day	30	–	Finish debugging coordinator
• Day	30	–	Debugging finished

Command Style

Comma separated lists starting with target

Automated LSL Test Protocol

We need to have a simple protocol for performing automated tests of lsl functions. We want to be able to rez objects which will reset their state, be able to register their existence to a coordinator, and start them all, individually or in groups.

The exact protocol below is not a specific requirement -- any protocol which provides an unambiguous way to easily communicate with the coordinator and the test units is sufficient. The messages need to be easy enough for a human to type to control the controller or a single test unit.

Units are always MKS -- meters, kilograms, seconds unless otherwise specified.

Coordinator

The coordinator holds testing sessions by registering test units and commanding them to start, run as individuals or groups, and collect their results.

Requirements

• There should be no need to link any new objects, including test units, to the coordinator.
• There is no need for secure communication between test objects and the coordinator.
• The coordinator does not need to be a single lsl script.
• The coordinator will announce the ability for nearby test units to register.
• Once a test session has begun, the coordinator will not accept any more registrations.
• The coordinator can start a group of tests and collect results.
• The coordinator can start a single test unit and collect results.
• The coordinator can start all known tests.
• The coordinator will have a variable test broadcast channel.
• The coordinator will accept commands from its controller and broadcast appropriate commands to all, groups, or individual test units.
• Collect reported results from registered test units.
  • All pass/fail messages by group and test unit.
  • All late registrations.
• It should be possible for an agent or another script to control the

coordinator.

• Every test session will record a start time, test location,
• The collection of all tests which failed the last test session is an ad-hoc test group known only to the controller. For example we run all tests in 'quiet mode' and then all failures are treated as a new group which allows 'verbose mode' testing individually or all at once.
• If results are filling memory, the coordinator should output information prematurely, but needs to keep statistical totals as well as the known failed group.
• Every time summary or transient results are output, the session id, and current time are included in the output.
• The coordinator must know by the end of a test session and be able to report on:
  • Which test units passed
  • Which test units failed
  • Which test units failed to report before timeout.
• At the end of every test session it is possible to get a summary from the coordinator.
• The summary reports:
  • Number of test units in the last session
  • Number of passed test units.
  • Number of failed test units.
  • Number of test units which failed to report before timeout.

Communication

All commands should be said in a chat channel. The channel has to be controllable before the tests start.

The coordinator can send results and summaries to different channels. For spoken results, it simply says the results as needed and when a summary is requested. For email, the same data is emailed to the destination address. For http endpoints, the same data is sent via POST in the content body.

States

There are five primary states for the coordinator.

Initialization

On startup, the coordinator is pristine. It only needs to collect who will be processing the results and how.

• The coordinator will listen on it's control channel.
• if the control channel command is issued, the controller will change

Commands

• "coordinator, control-channel, 1234"
• "coordinator, report-destination, say, 0"
• "coordinator, report-destination, email, test@secondlife.com"
• "coordinator, report-destination, http, http://example.com/"
• "coordinator, registration-timeout, 10"
• "coordinator, set-up"

Registration

• During this phase, the coordinator will announce registration is available.
  • llSay(test_broadcast_channel, "testunits, register");
• Test units will register their name and group
  • llSay(test_broadcast_channel, "testunit, group_math, integer_limits");

Set Up

This phase is to control which tests to run, what channel to use for communication, and how verbose the reporting should be.

Requirements

• Able to change broadcast channel for test unit and coordinator communication.
• Able to
• Able to start tests.
• Record start time of tests

Commands

• "coordinator, broadcast-channel, 1234"
  • change the broadcast channel internally
  • tell all registered test units to use the new channel
• "coordinator, test-timeout, 100"
• "coordinator, start, quiet, all"
  • broadcast start command to every test unit or group with instructions to perform minimal reporting (summary only)
• "coordinator, start, normal, test_group, math"
  • broadcast start command to specified group with instructions to perform normal reporting (failures and summary)
• "coordinator, start, verbose, test_unit, multiplication"
  • broadcast start command to specified test unit with instructions to perform verbose reporting

Collect results

• Keep track of total known registrants running in this session.
• Track number of test units that report all tests passed.
• Track number of test units that report failure.
• Track number of test units that failed to report within the timeout.

Tests Complete

Commands

• "coordinator, summary, say, 0"
• "coordinator, summary, email, test@secondlife.com"
• "coordinator, summary, http, http://example.com/"
• "coordinator, set-up"
  • return to setup
• "coordinator, reset"
  • tell every registered test unit to return to the default broadcast channel.
  • reset internal state, and go to the initialization phase.

Controller

We will also need a simple default controller which understands the coordinator protocol and provides a touch and dialog menu controls so that an avatar can approach the controller and run every test unit or test group.

Requirements

• The controller must be separate from the coordinator -- not the same script, nor required to be in the same linked set.
• The controller does not need to be a single lsl script.
• The controller must provide avatar controller interface.
  • An avatar can control the tests through a series of dialog menus or prompted typed commands.
  • The controller should activate when an avatar touches it.
  • There should be a shiny red button inviting the touch. :)
• The controller knows enough of the coordinator protocol to initiate a set of tests.
• The controller must be able to initiate:
  • A single test unit
  • A test group
  • All tests
• The controller can assume that only one coordinator is available to run tests
• The controller should allow specification of test results endpoint:
  • llSay on channel 0
  • email to a specified address
  • http

Test Units

Each test unit will have a small amount of boiler plate code to talk to the controller and understand the communication enough to change to another channel, start tests, and report test results.

Requirements

• All test units reset to a default test broadcast channel of 0.
• All test units will be in a test group and the unit knows the group name.
• All test units have a name.
• Register to a coordinator on announcement of registration.
• Change communication channel on request of the coordinator
• Run its tests on the request of the coordinator
• Report test results with multiple levels of output
  • verbose: everything
  • normal: failures and summary information
  • quiet: summary information only