Mono

From Second Life Wiki
Revision as of 07:08, 19 August 2008 by Periapse Linden (Talk | contribs)

Jump to: navigation, search

"Mono for Second Life" refers to a simulator upgrade which can dramatically speed the running of scripts — especially calculation intensive ones. The Linden Scripting Language ( LSL ) will not change in any way*, so all of your existing scripted objects and attachments continue to function as before, only now they will have the opportunity to run faster. The key to this improvement is an open-sourced virtual machine called Mono.

Mono is slated for the Main Grid with server version 1.24, which should deploy during the week of 17th August. Prior to that you can still give Mono a spin on the preview grid (see below).

You can also see videos of Mono in action.

How LSL scripts work

All your LSL scripts are run by the simulator (Linden Lab server program) that also runs the region you are in. When you teleport, or region cross, your new region's simulator takes on the duty of running all your scripted attachments. But the simulators cannot understand LSL directly -- the language was designed for human readability, not machine. So before the script can be executed, it must be turned into a machine readable format. This process is called compilation, and the resulting machine readable version of the script is called bytecode. LSL scripts are compiled when they are created by resident-programmers. The bytecode itself is stored on the Linden Lab asset servers and never needs to be referred to directly by residents. Instead, when you rez a scripted object, the simulator for the region you are in notes the script(s) in the object, and requests the appropriate bytecode from the asset database. The simulator program has several parts, and the part which runs the script bytecode is called the LSL virtual machine.

In today's Second Life, scripts are everywhere in regions: from simple rotating objects to complicated vehicles, vendors, or attachments that respond to your chat commands. For many regions the LSL virtual machine is kept busy trying to execute hundreds of scripts all at once. As the number and complexity of scripts in a region rises, so do the demands upon the simulator. After a certain point the virtual machine starts taking up so much processing time that the rest of the simulator (particularly the physics engine) bogs down, and server-side lag results. Thus anything that can speed up the execution of scripts can push out the point where server lag starts to occur.

Enter Mono

Mono is another kind of virtual machine. It is fully open-sourced and has a proven record of speed and versatility. For over a year now Mono has been considered by Linden Lab as an alternative to the old LSL virtual machine. But there are difficulties with switching virtual machines. The most fundamental problem is that all bytecodes are different. So the LSL bytecode is just gibberish to Mono, and vice versa. Before we could start using a Mono virtual machine, we need to develop a compiler which can take LSL scripts and turn them into Mono bytecode. This is tricky, because the goal is to make scripts running under Mono behave 'exactly' like scripts running under the current virtual machine. It's painstaking work, and requires an extraordinary amount of testing. The final sprint of coding was completed in the third quarter of 2007, and since November Linden Lab QA has been rigorously pounding on the new virtual machine with an assortment of tests both automated and manual.

The Plan

Thanks to the efforts of the beta test residents, and the Linden development and QA teams, Mono is now ready for the main grid. The viewer and server changes will deploy separately in August.

The Mono viewer changes are checked in for the 1.21 viewer, which should go to release candidate on 20th August. The viewer changes include a checkbox on the script edit dialog (which allows you to make a script compile to Mono), a Tools menu item to allow you to recompile to Mono all the scripts in your selection, and a change to simulator statistics to show events per second instead of instructions per second.

The Mono server changes are checked in for the 1.24 server. This server is currently deployed to nearly all regions of the Preview Grid. The plan is for the Mono server to roll out to the main grid the week of 17th August.

Those who would like to try out Mono before it goes to the main grid can download the preview grid viewer from the downloads page and test Mono on the various preview grid regions.

According to this plan the Mono server will deploy before the 1.21 viewer is available as a release candidate. Thus, although the main grid will be Mono enabled, no one using the regular or release candidate viewer will actually be able to create Mono scripts. During this time before the viewer release enterprising scripters can use the preview grid viewer and force it to connect to the main grid. Note that no special viewer is required to run Mono scripts -- that is automatic and handled by the server. A Mono viewer is only required in order to change the compile target of a script (from LSL2 to Mono, or vice versa).

Once Mono has been live on the main grid for several months we may turn off compilation to LSL2. At that point all new and edited scripts will be Mono. We will, however, keep the LSL2 virtual machine running. This means that old scripts will continue to run as before, but as soon as they are edited they will become Mono scripts.

Mono benefits

We've run some benchmarks to compare the performance of the LSL2 virtual machine and the Mono VM. When we run the tests side by side we found that Mono is up to 220x faster than LSL2. These benchmarks were math intensive scripts usually used to evaluate performance. It should be noted that for ordinary scripts the gains are much more humble.

Mono uses more memory than the typical LSL bytecode. It offsets this by introducing dynamic memory allocation. With LSL2, a full 16K is allocated for all scripts, even simple "Hello, Avatar" ones. Mono allocates only the memory it needs. In tests on typical regions it turns out that the combination of Mono using more memory, but allocating memory better, is about a wash as far as overall memory footprint goes.

In some extreme cases Mono scripts can use up to 4 times the memory as LSL2 scripts. In order to maintain backwards compatibility the script size limit has been increased from 16K to 64K.

Mono tip: Mono can do bytecode sharing. Thus multiple copies of scripts with the same asset id will only take up as much room as one instance. Imagine some script that you use a dozen times on your land. If each of the objects containing the script is separately compiled from text source, you will use up a dozen times the script's size of memory. But if instead you simply drag a copy of the single, already compiled script into each of the dozen objects, then no matter how many copies exist they only take up the size of one script (plus data) in memory.

Mono How-to

Mono is scheduled for deployment to the main grid with the 1.24 server, starting 17th August. After that deployment all regions will be able to run LSL scripts compiled to Mono. The 1.21 viewer, scheduled for first release candidate on 20th August, has the user interface needed to compile scripts to Mono (any viewer will run a previously created Mono script). The following information should help developers start working with Mono during this transition period, until both server and viewer are fully deployed.

  • Download a viewer with the Mono UI if you want to create Mono scripts
    • When the 1.21 viewer is available in release candidate, use it. See the Test Viewers section of the downloads page.
    • Before the 1.21 viewer is available, you can still compile to Mono using the preview grid viewer, available from the Test Viewers section of the downloads page.
  • Log in to SL and go to a Mono-enabled region
    • Mono will deploy to the main grid with the 1.24 server. It will deploy over three days starting 19th August (see the SL blog for a post when it happens). After the deploy all main grid regions will be Mono enabled.
    • If you wish to test Mono before it rolls out to the main grid, you can go to the preview grid. Nearly all regions on the preview grid are now running Mono. Check the server version by doing Help / About Second Life.
    • If you want to use the preview grid viewer to compile mono scripts on the main grid (necessary until the 1.21 viewer is available in release candidate), you will need to force that viewer to log you in to the main grid instead of the preview grid.
      • Open the preview grid viewer
      • At the login page hit Ctrl-Shift-G. This will bring up the grid select drop down menu. (If Ctrl-Shift-G isn't available on your system, you can set ForceShowGrid to TRUE under Debug Settings.)
      • Select "agni" (the main grid) from the dropdown, and log in normally
      • Note: if, later, you want to use that viewer for the preview grid again, select the preview grid (aditi) from the drop-down.
  • Create/edit a script an compile it to Mono
    • Rez your script in an object.
      • Edit the script from the object's contents tab.
      • On the script editing dialog you will see a new checkbox at the bottom "Mono". Check it.
      • Hit Save to recompile this script to Mono.
    • You may now treat this script like any other. It will automatically run in the Mono runtime, regardless of ownership transfers or viewer version.
    • If you fail to see the checkbox, you are not running a Mono viewer.
    • If the checkbox is grayed out for Mono, you are not in a Mono enabled region.
    • To convert a Mono script back to ordinary LSL2, uncheck the checkbox.
  • To convert a number of scripted objects all to Mono at once
    • Rez all the scripted objects
    • Use the select tool to select them all.
    • From the Tools menu select Recompile Selection / Mono.

After you have converted a scripted object to Mono you will need to do a full QA run. Though Mono is compatible with old LSL2, there are timing differences which may cause object behavioral changes. Test your objects thoroughly before releasing Mono versions.

To report Mono issues, use JIRA

    • First check this meta JIRA to see if anyone has reported the issue already
      • This may not be doable unless you've drilled down into your script to locate what is breaking.
    • If a JIRA already exists, and it appears to be the same failure, then simply add your information as a comment and attachment to that issue.
    • If not, open a new SVC ticket for each bug you find. In the ticket summary please use the words "Mono beta" so that we can filter for them.
    • In addition to your description of the problem please attach your script itself, or (ideally) a smaller script which illustrates the difference in behavior between the two virtual machines.
    • Link your JIRA to the meta JIRA.

Testing

During the integration of Mono we have used tests to ensure there are no regressions. For the current version of the regression test see LSL Language Test. This effectively serves as the specification of LSL.

There are tests for library call bindings in LSL Library Call Test 1 and LSL Library Call Test 2. This is split to overcome the memory limitations.

Event test script

There are several benchmarks to test the performance. They are available here:

FAQ

Section for questions.

When will the Mono Beta start? 
The Mono beta is in progress
Do you have more info about the Mono Beta? 
Yes, see Mono/Beta FAQ.
Will the available memory for scripts change? (Currently 16k in LSL2 VM) 
For the same LSL script the Mono bytecode and LSL2 bytecode will be of different size. In order to be compatible with all known scripts, we have expanded the size ceiling for Mono to be 64k. This is ok to do for Mono because unlike LSL2, Mono allocates memory dynamically, whereas all LSL2 scripts occupy 16K. Mono scripts only allocate the memory that they need.
64K? Wow, isn't that going to encourage inefficient scripting? 
We hope that the change will promote more efficient scripting. Currently programmers have to get around the 16K limit by using multiple scripts, and a lot of cycles get spent on passing data between those scripts. With a single script that would not be necessary.
Will I have to manually convert all my objects to using Mono, or is there an automated tool? 
Yes, you must manually invoke compilation to Mono. Though you can make use of the Tools Menu to recompile all scripts in selection to Mono.
Can I keep my scripts running on the old LSL2 VM forever? 
We have no plans to eliminate the LSL2 VM. This will be re-examined after Mono has been live on the Main Grid for a while. But right now it is easier to continue to support LSL2 than to migrate all scripts.
Will I be able to write scripts in languages besides LSL since Mono supports lots of languages? 
Eventually. Right now our goal is to make Mono completely compatible with LSL2 for LSL scripts.
Why didn't you use <favorite_language> instead of Mono? (ie lisp, python, lua, javascript) 
TBA
Will LSL be getting real language features with this change? (ie arrays, references/pointers, includes/imports)
No. The LSL language is not changing with this update.
In LSL2 VM scripts are compiled on the viewer then uploaded, will that change with the Mono VM? 
Yes, Mono compilation is done in a distributed fashion on the sim hosts.
Related to the above, I use a "clever trick" to upload my compiled bytecode in LSL2 with out the correct script text. What will happen to scripts I uploaded in this way when converted to Mono? Will I be able to continue to use my "clever trick" for Mono scripts? 
The Mono compiler looks only at the script text. The Mono VM will only run bytecode which has been compiled by our Mono compiler. You will not be able to run any uploaded Mono bytecode.
What about scripts whose LSL code has been lost, ie scripts that still run, but result in "Script missing from database." when you try to edit them? Is there any possibility of bytecode translation, or are these scripts stuck in the LSL vm forever?
There are currently no plans to allow byte code translation of LSL scripts, only compiling from source. This may be considered depending on resident demand.
The Havok4 Early Adopter program was a great success. When are we going to get a Mono Early Adopter program?
We don't think a Mono Early Adopter program would really work. We have a general fear of the main grid being split into Mono regions and LSL2-only regions. What we foresee are residents doing teleports and region crosses with Mono-compiled attachments. When they go to a non-Mono region their scripts just stop working. Some will understand what has happened, but many will not. They will contact the scripter demanding a "fixed" script, or they will contact LL support. So our strategy is to test Mono thoroughly on the preview grid and then check it in and push it live.
When is Mono going to go out to the main grid?
We've decided that the current Mono feature set is enough to get it out onto the main grid and let everyone start using it. So this will happen in Q2. The current plan is to merge the Mono viewer changes into the 1.21 viewer, which will go into release candidate in June. The server changes will be checked in next, and likely be deployed while the client is still in rc. As soon as the Mono-enabled simulators fill the grid residents wishing to compile Mono scripts can use the rc client to do so. While the release client will lack the Mono UI until 1.21 becomes the default, people running the release client will still be able to use scripts previously compiled to Mono using the rc viewer.
What are the known differences between the LSL2 and Mono compilers and runtimes?
We have not tried to make Mono 100% compatible with LSL2. At least not to the point of duplicating any of the tricks or hacks that LSL2 allowed. Below are listed some known behavioral differences. Add to the list as more are discovered.
  • Unicode support. From Strife Onizuka "In LSO LSL, the entire Unicode range was supported by complying to RFC 2279 (about 2 billion possible characters). Mono supports RFC 3629 which supplants RFC 2279 and limits the Unicode range to the first 1,114,112 character codes. This directly effects these functions: llBase64ToString, llUnescapeURL. Strings being passed from LSO scripts to Mono scripts will become corrupted (in a reliable way) if they contain characters outside the limited Unicode range." -- SVC-1960
After Mono launches why not recompile *all* scripts everywhere on the Grid to Mono?
While it is certainly appealing to have only one compiler and runtime to support, practical concerns make this not feasible. Here's a list of the difficulties that give us pause:
  • We do not have the text asset (the LSL code) for all running scripts. These "bytecode only" scripts would stop working.
  • The automatic recompile would restart all scripts. Many scripts are meant to run continuously without restarting.
  • Mono scripts have a different timing profile than LSL2 (usually faster). This will introduce behavior differences which will lead to some scripts breaking, often in subtle ways.
  • Some scripts take advantage of undocumented "features" of LSL2. We did not strive for 100% compatibility in such situations, but rather made Mono behave as sensibly and predictably as possible.
  • As a result of both reasons above, scripts need QA work after they have been recompiled. Residents who code and sell LSL scripts will have to test and possibly adjust the behavior of Mono versions of their scripts. If conversion were automatic they would not be reimbursed for their QA effort. With manual recompilation the resident scripters can sell Mono versions of their scripts as an upgrade, after they have tested and modified them.
  • Recompiling all scripts would play rather heavy-handedly with the permissions system. If someone has made and sold a script as "no-modify", an automatic recompilation would violate their policy. While some scripters would be ok with this, many would not.