User:Lum Pfohl/LSL Goodies/HI-WA Occupancy Counter System/Collection Server

Collection Server

Background

The Collection Server receives the inbound data from the Sensors, and stores them in an internal memory structure, updating previous counts with new counts it has just received from a particular sensor. It can discern which region a particular piece of information originated, and store it accordingly. That is, it keeps OI info together, HI info together and WA/IH info together. The collection server produces, upon request, a dump of the stored information to the Relay Servers.

Looking at the Specific Requirements Section 2 of the Business Requirements document, we see the Collection Server must satisfy 4 requirements.

Performance

REQT-2.01 states that the server must be able to accommodate all inbound data and process them at a reasonable rate. In order to satisfy this requirement, we select a low-latency, lag-free simulator for this purpose. Currently (as of 1 September 2008), the Collection Server is housed on the Mouse Point sim (in violation of REQT-1.05, soon to be rectified).

If processing for all OI's HI's and Welcome Area/Infohubs prove to be too stressful on the Collection Server, then the Collection Server can be split to handle only OI's, only HI's and only WA/IH's. Their outputs can be concatenated to produce the required payload structure (REQT-2.02).

The Collection Server checks for new email every 2.5 seconds. If no email has been received, it will continue checking every 2.5 seconds. If it DOES receive mail, it switches to 0.25 seconds checking interval and processes mail as fast as it can until 10 consecutive checks at 0.25 seconds yields no new mail.

Communication

REQT-2.02 states that the information that has been collected must be made available to the device that requests it. It must be in a format that it can transmit the entire payload in one transmission.

Incoming messages from the sensors are in the form

Object-Name: ResCtr HI 217
Region: Help Island 217 (261888, 268800)
Local-Position: (143, 153, 22) 

NOTIFY~HI 217~10~0

From this, we strip off the header until we are left with

NOTIFY~HI 217~10~0

This instructs the server that we are sending a NOTIFY email from a sensor located on Help Island 217, with 10 Residents and 0 Mentors.

Incoming messages from the Relay Servers in the form

Object-Name: Lum's OI/HI Relay 1
Region: SLVEC (230400, 229888)
Local-Position: (98, 107, 25)

QUERY ALL EXT~5233b8b2-2eb4-622b-30e8-eef4377bb7a6

Again we strip off the header until we are left with:

QUERY ALL EXT~5233b8b2-2eb4-622b-30e8-eef4377bb7a6

This is a "Query All Extended" request (REQT-2.04) from my Relay Server in SLVEC. The UUID is the key for the Relay Server to which the Collection Server will send an email.

Outbound messages to the Relay Servers are in the form:

<OI CSV LIST> ! <HI CSV LIST> ! <WA CSV LIST>

Here is a clipping of what it might look like when received by the Relay Server:

Object-Name: ResCtr Server
Region: Mouse Point (259840, 230912)
Local-Position: (134, 174, 76)

0, 9~0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0!0~0, 0, 0~0, 0~0, 
0~0, 0~0, 0~0, 0~0, 0~0, 0~0, 0~0, 0~0, 0~0, 0, 0~0, 0~0, 23~2, 11~4, 0, 0, 
11~0, 0~0, 0~0, 1~0, 0~0, 12~0, 8~0, 9~1, 1~0, 2~0, 2~0, 12~0, 0~0, 11~1, 0~0, 
0~0, 0~0, 10~0, 2~0, 0~0, 0~0, 3~0, 4~0, 12~0, 12~0, 11~1, 0~0, 2~0, 2~0, 13~0, 
0~0, 0~0, 0~0, 0~0, 0~0, 0~0, 0~0, 1~0, 0~0, 0~0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
0, 0, 0, 0, 0, 0, 0, 0, 0, 0!0, 10~0, 0, 0, 3~0, 13~0, 0, 29~0, 0, 0, 7~0, 5~0, 
0, 0, 9~0, 1~0, 3~0, 8~0, 3~0, 0, 2~0, 42~1, 0, 0, 0, 0, 0, 0, 0, 0

Again we strip off the header and we parse the long list of numbers and commas. The values are actually comma-separated and also '!' (exclamation point) separated. Using llParseString2List( ) and llCSV2List( ), we can easily manipulate these lists in a manageable fashion.

The order in which we find a number is relevant. For OI's, each position represents an OI number. The first one is OI 1, the next is OI 2, etc. up to OI 100. Since there is no OI 2 and no OI 19, we can "stuff" other OI's in there. For example OIP is stored in the OI 2 slot, OI 22b is stored in the OI 19 slot. A similar thing happens for the HI's. The first in the series is HI, followed by HI 2, HI 3, ... up to HI 16. There is room for up to HI 20 - so we use HI 17 to store HIP data, and HI 18 to store HIP2. Then we make room for HI 210 - 260. The Welcome Areas are stored according to an array of Sim Names (see code).

Outbound messages are sent using email slaves. It is a well-documented fact that the llEmail( ) (email send) function causes whichever script that executed it, to pause for 20 seconds. That means that script cannot do anything for 20 seconds even if there are data from Sensors or requests from Relay Servers to handle. We can split off the task of emailing to slave scripts whose sole purpose is to email, leaving the main script to continue working (REQT-2.01)

Reliability

REQT-2.03 states that if the Collection Server goes offline or the SIM goes offline, the rest of the system should not halt. This is accomplished by the use of Relay Servers and the Email system. Incoming email from the sensors will queue up while the Collection Server is offline. Relay Servers will continue to retransmit cached data until such time the Collection Server comes back online.

If the Collection Server code is reset and all retained data is lost, the dataset can be rebuilt within 3 or 4 minutes as the sensors transmit their data on that cycle.

Quality of Service

REQT-2.04 states that the Collection Server must be able to somehow accommodate different versions of sensors and clients talking to it. This is important because we must recognize that we cannot upgrade the entire system in a single event (a 'commit' operation known in the Database world). In reality, if we must upgrade all sensors then we must be able to handle the "old" versions pumping data while they are gradually replaced by "new" new versions.

We must program the Collection Server to specifically to accommodate mismatched versions of data. The same for downstream components (such as the Relay Servers). If an update to the HUD is mandatory that it requires the Relay Servers to get different data, then the Collection Servers must correctly respond to potentially different versions of the Relay Server while the upgrades to the rest of the system is completed.