Mono Scheduler testing

A simulator which incorporates a revised Mono Scheduler was deployed to Aditi, the preview grid, on 19 March 2009. For the next two weeks Residents will be able to go to these regions and test their scripts on this scheduler. Please return any feedback you have to the JIRA listed below. We'll use that feedback to fine tune the scheduler performance before we merge it into trunk for release with an upcoming server version.

Please provide any feedback as a comment to SVC-3997. If you find any outright bugs, please create a separate JIRA and link it to this one.

Update: 10April09 Based on the wealth of resident testing and analysis presented in SVC-3997 we've decided to hold off on the deploy of the new Mono Scheduler. Instead we will await the next Mono-related project, the effort to transition to 64-bit, and fold the Mono scheduler into that project. Additionally we'll figure out how best to optimize the simple library functions to mitigate the observed slowdowns. The Scheduler Test regions on aditi will be around over the weekend, but will likely be taken down before the 17th of April.

The Mono Scheduler

The Scheduler is responsible for managing time and memory resources for all the scripts running in a simulator. As a busy sim might have thousands of active scripts, keeping them all happy and keeping overall simulator performance up is non-trivial. Changes to the scheduler can affect existing scripts: some might run out of memory and break, others might experience performance differences. Because of these possible behavior changes we want to give Residents the chance to test their scripts and give us feedback. We can fine tune the Scheduler so that content is minimally affected.

Motivation for the reworking

The original Mono Scheduler shipped with server 1.24 -- the version that introduced Mono. Its non-deterministic behavior had two deleterious side effects:

• Some "lucky" scripts were able to acquire more than the 64K memory limit, decreasing available memory for other scripts.
• Other scripts were able to monopolize time resources, increasing their performance at the expense of other scripts and region performance.

A fix for this was included in an early version of server 1.25, however this fix was reverted because it broke considerable content. Babbage Linden and Studio Blighty then reworked the Mono Scheduler so as to make it deterministic, but also to allow us to tune its performance. We now have a version which passes our benchmarks, and we'd like to share that version with the SL scripting community for feedback.

Goal for this revision

Since simulator resources (cpu, memory) are limited the Scheduler's job is essentially making tradeoffs. Too tight a control and scripts run out of memory more frequently, and script performance is unacceptable. Too loose control allows scripts to hog cpu and RAM at the expense of the simulator's other duties (physics, communication) and so region performance becomes unacceptable.

We added some benchmark tests to our internal QA for the scheduler in an attempt to find a sweet spot which delivers optimal script and region performance. The scheduler deployed for testing passes those benchmarks, so now we need Resident feedback.

Who should test

If you're a scripter who uses Mono for LSL and you write scripts which have either or both:

• high memory consumption
• a lot of calculation/processing

then you should consider testing your scripts in the provided sandbox. Scripts which operate on large lists are particularly prone to be in both of those categories and should be tested.

How to test

1. Use any recent Viewer to go to Aditi, the preview grid.
   • from the login screen hit Ctrl-⇧ Shift-G to bring up the grid dropdown menu
   • select "Aditi" from the menu
   • Note: if Ctrl-⇧ Shift-G doesn't work, try clicking on the Name or Password boxes to move focus to the dialog panel, then do Ctrl-⇧ Shift-G again.
   • Note: when you're done testing you can select "Agni" from the grid list to return to Second Life, and again hit Ctrl-⇧ Shift-G to hide the grid select box
2. Teleport to Scheduler Test 2 or Scheduler Test 3 (teen scripters: Scheduler Test TG)
3. You should compare performance between Agni and these test regions. Please let us know if there are significant differences.
4. We want to minimize the amount of content that will break with this new scheduler. Please let us know if you have a script which runs on the main grid but fails (out of memory) on the test regions
5. Add all your feedback to SVC-3997.
6. The Scheduler test regions should remain up through the first week of April

Notes

• A further complication has arisen with script memory. Early builds of the 1.26 server were deployed to large swaths of Aditi for normal beta testing. Residents discovered many scripts which no longer had enough memory to run. These scripts ran fine on Agni (1.25) but broke on Aditi (1.26). This resulted because the 1.26 server was originally built with 64 bit Etch, our first simulator release in 64 bit. The larger word size for references bloated script memory usage, and caused these failures. We have since reverted back to 32 bit builds while we assess the issue. Both the current 1.26 test server, deployed to most of Aditi, and the Scheduler Test sandboxes on select regions now use 32 bit etch.