Difference between revisions of "Lua FAQ"

From Second Life Wiki
Jump to navigation Jump to search
Line 436: Line 436:


If you're so inclined, you can go as far as to take the Luau transpiler and hack it up to support <code>!=</code> and <code>//</code> comments and really whatever you want. So long as it can take that source and output it to standard Lua, you're good.
If you're so inclined, you can go as far as to take the Luau transpiler and hack it up to support <code>!=</code> and <code>//</code> comments and really whatever you want. So long as it can take that source and output it to standard Lua, you're good.
=== Are you going to be using Lua 5.1? Why not 5.4? ===
We're using Luau as the base, which is ''compatible'' with Lua 5.1 but not necessarily Lua 5.1. It includes various features of PUC-Rio's Lua implementation up to and including 5.4. The Luau Compatibility reference is a good guide for which features are taken from which versions of PUC-Rio Lua: https://luau-lang.org/compatibility
=== Are there going to be integer types in SL's Lua implementation? ===
We have added 32-bit integers but they are strictly for compatibility with existing LSL APIs where integer type identity is important within a list (<code>llSetPrimitiveParams()</code> and friends.) They don't have a literal form in Luau scripts and in general you should just use regular Lua numbers if you care about performance, since the Luau VM optimizes for their use, but technically you can use them. The Luau project discussions have some explanation as to why integers aren't a first-class user-facing type in Luau https://github.com/luau-lang/luau/discussions/242#discussioncomment-1707387.
==== Are integer operations supported at all, then? (i.e. bitwise operators) ====
Yes, through the <code>bit32</code> library. In fact, ''more'' integer operations than are currently supported in LSL, like logical shift right and rotate operations. See https://luau-lang.org/library#bit32-library .

Revision as of 21:50, 6 August 2024

Can I keep using LSL?

Yep! LSL should work exactly the same as before, and a lot of testing work has been done (and is being done) to ensure that is the case. Even if you’re an LSL pro and rely on things like (some_list != []) for getting the list length, or strange things like ((largestring="") + largestring), you’re fully covered. Those still work, and now have optimized implementations.

In fact, LSL will get better than ever before, running faster, and using less memory! The generated bytecode for an LSL script in Lua is generally half the size of a Mono script, and most strings take up half as much memory by virtue of using UTF-8 instead of UTF-16.

We’ve taken a lot of care in writing the new LSL compiler to make the scripting experience for people who prefer LSL better than it was before.

Are you going to stop adding functions / events to LSL?

In the general case, no. A big part of the work we’ve done is making it easy to share functions between LSL and Lua without much extra effort. For events and functions that can be reasonably exposed to LSL (i.e., they only use simple types that LSL has anyway) then LSL should still get those.

For more advanced features, the idea is that there should still be a path for exposing them to LSL. Obviously, with Lua it’s possible to have object-oriented APIs that would not be possible in LSL, but those objects will generally be written in a way that calling a method on a “special” object we introduce in Lua will really boil down to a function call that can be expressed in LSL.

Not a real example, but some_object:giveInventoryList("folder_name", inventory_list) might really boil down to a llGiveInventoryList(some_object_id, "folder_name", "inventory_list"); call. In that way, most additional Lua functionality will be a fancy “easier” wrapper around functions shared with LSL!

The caveat is if functions are added that use new types that aren't in LSL (like functions that take in a key -> value map) likely won't be exposed to LSL, as there's no way to represent them in LSL anyway.

Wait, LSL can run on Lua now? How? Why would you do that?

A reasonable question! It sounds crazy at first, but as it turns out, LSL maps really well to Lua bytecode, and we already have a compiler that can output both Mono and legacy LSO2 bytecode. Writing a compiler that could output Lua bytecode wasn't terribly difficult, and it allows LSL scripts to run faster, and be more memory-efficient. Most importantly, it allows us to test behavior of the new scripting engine against the behavior of the existing Mono scripting engine. This helps us make sure we're doing an apples-to-apples comparison on our benchmarks, and that all the nice things that Mono currently gets us will also work correctly under the new scripting engine.

Additionally, for better or for worse, LSL is here to stay. People have been writing scripts in it for 20 years, and they're going to be running on the servers for years to come. Performance and memory usage issues with LSL scripts will continue to matter for the foreseeable future. The faster the server can run existing LSL scripts, the more time it can dedicate to running newer Lua scripts. Memory usage has been a particular problem with the existing Mono scripting engine, and this would help us address this.

What about running scripts on the viewer-side if I want viewer-side effects? Does this have any interaction with the work adding Lua addons into the viewer?

No, the work on adding Lua addons to the viewer is a separate effort. Lua scripts on SL will strictly run on the region, and not on the people's viewers. We're focused on providing a high-quality replacement to the existing server-side scripts before we look at anything else.

Note that we are all too aware of the current difficulty of scripting HUD UIs for Second Life (we make a lot of our own) and we're thinking of ways we can make things better, but there are no definite plans, and it wouldn't involve running scripts on the viewer in the near-term. The current UI tooling exposed to Lua viewer addons isn't tooled for being called from untrusted code either way, so lack of "client-side" scripts is unlikely to hinder any effort to create a better API for HUDs for in-world systems.

You said Lua runs LSL faster than Mono? It uses less memory? How? I'm suspicious.

A reasonable suspicion! We were also suspicious and suspected that Lua would be slightly slower, but in reality, yes, Lua can run LSL scripts quite a bit faster than the custom-patched Mono 2.6 we currently use. The best LSL on Mono has performed compared to Luau in our internal tests is 10% slower than Luau. Similarly, most LSL on Luau scripts end up being around half the size of Mono scripts. That means much more memory to play with in your LSL scripts.

The reason why isn't intuitive, after all, Mono is a lot more intricate and has a lot of fancy performance optimizations like a JIT optimizer, but it becomes clear when you look at the bytecode that's generated for LSL on each scripting engine.

Consider the following LSL function as an example: 

float loudSubtract(float val, float sub) {
    // subtract, telling the owner what we did
	float new_val = val - sub;
	llOwnerSay("New val is " + (string)new_val);
	return new_val;
}

This compiles to the following Luau bytecode (subject to further optimization):

Okay, that's pretty decent. Some places where things could be more optimized, but no big deal. The equivalent Mono bytecode is (and bear with us, this is long:)

While this might look unreasonably large, this is also pretty similar to how async C# functions that have awaits get compiled. The code required to allow a function to pause in the middle to let other things run is necessarily very large, and our Mono bytecode instrumentation is essentially a special variant of that which optimizes for cases where the function won't have to pause. Try pasting the following code into sharplab.io (while decompiling to IL) and you'll see something very similar: 

using System.Threading.Tasks;
public class Example {
    public async Task llOwnerSay(string val)
    {
        // stub
    }
    
    public async Task<float> loudSubtract(float val, float sub) {
        float new_val = val - sub;
        await llOwnerSay("New val is " + new_val.ToString());
        return new_val;
    }
}

The Lua bytecode ends up being a lot smaller, because the Lua scripting engine itself knows how to handle this case, and doesn't need to add any special code to your scripts to allow them to be paused and resumed in the middle of a function.

Even though Mono has a JIT optimizer, which should in theory make the script run faster, this sort of code is very unusual, and the JIT optimizer isn't able to optimize it very well. That means that even though Luau itself is in theory slower, it's able to run LSL much faster than Mono can.

You might also notice things like call float64 [LslUserScript]LindenLab.SecondLife.LslUserScript::Subtract(float64, float64) in the Mono code. Those are necessary because of LSL's weird operand evaluation order, but are unnecessary in Luau because of the more efficient form of bytecode that Luau uses.

In general, C# doesn't aim at producing very small bytecode for async functions, which is important for something like Second Life where you might have thousands of different individual scripts running within a region. Luau does generate small bytecode, while still having decent performance, making it a good choice for Second Life. And hey, strings are UTF-8 in Lua instead of UTF-16 as in Mono, so they use much less memory for most data as well.

Are we going to get a higher memory limit in scripts?

Not at first, Luau uses less memory than Mono, so you shouldn't really miss it. You're not going to start hitting script memory limits just because you converted your script to Lua. Using Lua proper should allow you to more precisely control memory usage in your scripts, e.g. by being able to handle memory allocation errors in pcall() and access to the mutable buffer class for byte-packing.

There is some interest in giving specific scripts a little more memory if they're designated "high priority", like a main "game manager" object for an experience, but there are no specific plans for this at the moment.

Are my existing LSL scripts going to run on Lua Automatically?

While we can run LSL in Lua now, and the idea is to eventually phase out Mono, there’s no immediate plans to transparently migrate existing scripts over to Lua. There are ideas about how this could be done safely without breaking existing content, but we want to make sure we have everything right before we start work on this.

How does interoperation with LSL work?

Everything that you expect from LSL should have some equivalent in Lua. There’s keys, rotations and vectors and something like some_vec * some_rot in LSL is the exact same in Lua scripts.

All LSL functions will be available in Lua, and something like llSay(0, “hi!”) is just ll.Say(0, “hi!”) now. What about llSetPos(llGetPos() + <0, 0, 1>);? Just ll.SetPos(ll.GetPos() + vector(0, 0, 1)).

No special magic required here, and you can write whatever fancy wrappers you want around the existing LSL functions. We’ll be looking at how residents actually write these wrappers to inform the design of “official” higher-level Lua APIs, so we’re excited to see your ideas!

Can I mix LSL and Lua in the same script?

No, although most things should be a direct translation. At some point we may make a tool to help you do this, but there are no specific plans at the moment.

If your object has multiple scripts, you can mix LSL and Lua within the same object, though.

Can we use all of Lua? Is anything not there?

Most everything will be there, except for two notable exceptions:

  • getfenv() / setfenv() / _ENV are not accessible because they necessarily make the script harder to optimize. Usually people only use this for mocking in test code, and this can be accomplished by passing in a parameter with a table of function references rather than pretending you have different global variable values for certain functions.
  • loadstring() is not supported since it has a huge impact on performance. For those not familiar, this is equivalent to eval() in other languages. Every single time you call loadstring() the compiler has to go over what you passed in, compile it, and then load the generated bytecode. The Lua compiler isn’t particularly fast, and doing this can cause your script to halt in a way that we can’t interrupt to give other scripts time to run. This is a big part of why eval() and loadstring() are generally considered poor programming practice.

The one real use-case we could think of for loadstring() is a debugger, and we’re looking at implementing a “real” debugger that lets you step through your code. Note that this won’t happen with the initial release, but watch this space!

One thing that is planned, but will likely not be in the initial release is the coroutine library. This is one we’d really like to have in there, so you can do fancy things like have multiple “timers” in a script by just having coroutines that sleep every so often, but multiple running coroutines within the same script doesn’t fit well with the existing scripting architecture the Second Life server uses. Rest assured, this will be in there soon™ after Lua is released, but in the meantime you can still use something like the Promise chaining that JavaScript uses to write asynchronous event handlers.

We’ll have code demonstrating "proper" usage of Lua around the time that the public beta starts.

Why Lua? I wanted C#!

Great question!

The server Lua scripting initiative actually started off as an attempt to get C# working on Second Life, basically as a revival of the work started in the 2000s. The initial thought was “this should be pretty easy, most of the work was already done getting everything onto Mono!” A lot of effort was put in over several months to:

  • Understand the Mono and .NET Core runtimes
  • Observe how other Engines and Virtual Worlds have embedded C# and scripting in general
  • Understand how recent advancements in scripting technologies (like WebAssembly) might make this integration easier
  • Whether unique properties of how scripting / content creation works on Second Life would make this harder to implement

Unfortunately, the reasons we were ultimately unable to do this were very technical, so there’s no side-stepping a whole lot of tech-speak here, but there were several key properties that made integrating C# in Second Life rather difficult to implement:

Sandboxing with reflection is hard

Previous proof-of-concept code for C# in Second Life relied on CoreCLR sandboxing in Mono. This approach has fallen out of favor in recent years, because similar approaches led to many, many security holes in Java’s browser plugin. This sandboxing method is not possible in .NET Core, and has been totally removed. Since this sandboxing is no longer in .NET, we need to rely on the ability to look at code to determine if it looks “dangerous” before running it. Some people have done this, but it can only be done properly by disabling reflection, which is a core part of C# and .NET! Without reflection, you’re unable to use the fancy third-party Newtonsoft.JSON libraries or whatnot that you would have otherwise been able to use.

Big portions of .NET’s standard library use reflection in a way that would be unsafe to expose to residents for scripting facilities, so you essentially have to maintain a list of functions and classes that are “safe” to use, and write alternatives to functions that are “unsafe” to call, but provide useful functionality. At this point, the scripting language that you’re exposing is a pretty bare-bones C# that needs to be scripted fundamentally differently from how you would write C# in Unity, for example.

Allowing the use of modern C# features would require migration off of Mono and onto .NET core, which would be more involved than integrating Lua. Mono is in maintenance mode at the moment, and only supports the legacy .NET framework Ensuring that we’re able to correctly limit each script’s memory usage in a low-impact way is a bit of an unsolved problem in .NET Core. Generally, memory allocation tracking is something that you only do for debugging, but we need to do it for all scripts! What happens if you declare a static global in a C# script? Is the memory usage counted against all scripts of that type? Do all scripts share the data in that static variable?

What about if there’s a function in the standard library that keeps a static cache of data? How do we penalize (or not penalize) scripts for allocations that they didn’t even control? How do we make sure that data remains consistent across script unloads / reloads?

These are problems that most people embedding C# don’t need to solve, and the solutions that do exist are not generally workable into Second Life’s scripting model without multiple months of work. Most other virtual worlds do script sandboxing on a per-parcel or per-region level and put everything into the same logical scripting sandbox, which Second Life does not do. People would be rightly annoyed if someone’s HUD or attachments caused their parcel to go over the memory limit and things stopped working.

Making everything micro-threaded with extreme size constraints is hard

We need to be able to temporarily pause scripts in the middle of running an event handler so we can ensure we fairly schedule code execution. Previously, we wrote a script to do some quite clever binary instrumentation of .NET assemblies to inject yield points. This works quite well for for LSL scripts, but there are issues with C# in particular.

One is that it doesn’t handle the try/finally construct in C#. While not impossible to handle, this would be a difficult undertaking. Something that would be much more difficult is allowing pausing execution inside a constructor or other such special constructs in C#. While the stack induction and bytecode manipulation done by our current system is very cool, it is quite complex to update, relies on outdated tools that don’t work with new .NET versions, and necessarily requires making the bytecode quite a bit larger than it would otherwise be by injecting hooks.

But that’s not a huge deal, our .NET instrumentation library was written when async/await was not a thing in C#, so we should be able to inject async/await into user-provided source code instead to make it possible to transparently pause scripts right? Ehhhh, sort of. There are certain functions that cannot be made async in C# without breaking things. Object constructors are one of those, we have no way to pause and resume inside those. The situation is similar inside ToString() overloads. If we make a user-provided ToString() overload async, it’ll just never get called in cases where ToString() would normally get called! And what about implementing IEnumerable.GetIterator()? That might work… if you did a lot of work of replacing IEnumerable with IAsyncEnumerable., and foreach with await foreach... sometimes.

Suffice to say, that if you go this route, you end up with something that’s sort-of C# but is missing a lot of things people expect of C#, and may not behave the same as the regular C# that it pretends to be.

Saving script state

But let’s not forget one of the most important cases where SL is unique: Scripts can move into your inventory, across region boundaries, etc, without having to care too much that it even happened. Even if your script runs forever in a single function, that’s totally fine! For example, this is a valid LSL script: 

 default {
   state_entry() {
     integer i;
     while(TRUE) {
       llSetText("Run Number " + (string)(i++), <1,1,1>, 1);
     }
   }
 }

All it does is run forever in the same function, incrementing a number in the floating text above an object. It still works if you take the object into your inventory, it still works if you move the object to another region, as a scripter you don't really have to care except in limited circumstances.

That's a pretty contrived example, but there are tons of scripts with a structure like this! Probably a more common example is a math-heavy function that just takes a while to run. Under the current model, you can interrupt what's going on by just picking up the object, then plop it back down later and let it continue on. There are a number of ways you can handle this without serializing the whole runtime state, like raising an exception in the middle of executing the function, but what are the chances that people will handle this correctly? How many command-line scripts have you used that do something really weird when you pressed CTRL+C?

To keep things simple for creators, we wanted to keep this nice property of LSL in whatever language we chose. This seemingly simple feature requires a lot of work because, think about it, how many programming languages let you seamlessly halt a program in the middle of it doing something, move it over to another computer, then resume whatever it was doing? Can you do that without doing a full-memory snapshot that would be gigabytes in size? That's not really a thing that people generally do, but it's required for us to not have sharp edges on our script creation experience!

We have a version of this in our Mono scripting engine, which was all written in-house. It works very well for LSL, but it has trouble with more complex structures. As it turns out, taking all the state of a running program, serializing it, then transparently deserializing it is a difficult problem, even with data that looks simple.

Suppose we have something like Dictionary<SomeObject, int> that we use as a counter for how many times we've seen each SomeObject. What do you do if the GetHashCode() bucketing changes after the script deserializes and the memory increases? Can you prevent that from happening? Will live IEnumerators still work correctly? How can you be sure of this? .NET makes no guarantees about any of this behavior, and good luck digging through the .NET source code to figure out if any of this is even the case.

What about types that aren't marked [Serializable], including types in the standard library? Can people just not use those? Do we have to write special custom serializers for them? Could they just randomly break when the script is deserialized and we accept that as a consequence?

As you can see, there are a lot of hidden sharp edges involved in transparently moving scripts across servers, and the general answer provided when you ask about doing this is "That's very hard, make users write more intricate scripts that explicitly handle their own serialization and save their own state." This is fine if most of your users are professional programmers, not so if you have a mix of professional programmers and regular users who just want an object that counts how many people have clicked on their head.

Wrapping it up

This is a very small subset of the issues we ran into while trying to design a satisfactory C#-based scripting system that fits Second Life's needs. Seriously, there's about 20 pages of notes.

Meanwhile, Lua makes most of these things easy. Want to halt a script mid-execution? Okay, set an interrupt handler. Want to serialize your script while it's running and have it move to another server? Okay, there are libraries that have existed for decades that can do this. Fable's save file system was based on this. Quite a lot of scripting engine integration problems we have that would rise to the level of extremely difficult and relatively novel Computer Science research in .NET (and by extension, C#) are relatively simple to do with Lua.

Okay, but seriously, why Lua specifically? Because it's popular for games? Because <other platform> uses it? Why not <X>?

Actually, no. Neither other users of Lua, or a specific love of the language had anything to do with the decision. If you prefer a different programming language, there's a good chance we considered it. We actually looked at quite a lot of different languages and their associated engines, including JavaScript and WebAssembly, Lua was considered fairly late in the process. The ones below are just a small subset of the ones that we looked at:

Lsl vm comparison.png

As you can see, Luau was the only scripting runtime that fulfilled all of our requirements for a scripting engine within Second Life, and we arrived at it quite late in the decision process. We wanted something that we knew could deliver a high-quality scripting experience to creators, and we believe that Lua can best provide it.

I genuinely hate Lua and refuse to use it

A lot of people feel the same about LSL! Kidding aside, this is a totally defensible position, Lua can be a quirky language. Yes, it's strange that array indices start at 1. Yes, sometimes you can forget that you join strings with .. and not +. It can take some getting used to.

A big part of the reason why we're using Luau is due to the quality of the scripting engine itself, not specifically the language. A lot of issues people had with older versions of Lua have been addressed in Luau, and ones that haven't been for one reason or another can be easily dealt with since we have such a high degree of control over the scripting engine now. Luau is very easy to modify. If we find that there are real quality-of-life issues with Lua that people run into while writing scripts, we can just change how it works, assuming the benefits outweigh the cost of breaking interoperability with Lua scripts from elsewhere.

But hey, maybe you really still hate the idea of using Lua. That's fine! The nice thing about having Lua instead of just LSL is there are a number of languages that can compile down to Lua code that could potentially work in Second Life! People have done work to get TypeScript and C# compiling to Lua, and it works pretty well for them, and there's also more exotic languages like MoonScript that target Lua specifically. See this GitHub repo for a listing: https://github.com/hengestone/lua-languages

If you're so inclined, you can go as far as to take the Luau transpiler and hack it up to support != and // comments and really whatever you want. So long as it can take that source and output it to standard Lua, you're good.

Are you going to be using Lua 5.1? Why not 5.4?

We're using Luau as the base, which is compatible with Lua 5.1 but not necessarily Lua 5.1. It includes various features of PUC-Rio's Lua implementation up to and including 5.4. The Luau Compatibility reference is a good guide for which features are taken from which versions of PUC-Rio Lua: https://luau-lang.org/compatibility

Are there going to be integer types in SL's Lua implementation?

We have added 32-bit integers but they are strictly for compatibility with existing LSL APIs where integer type identity is important within a list (llSetPrimitiveParams() and friends.) They don't have a literal form in Luau scripts and in general you should just use regular Lua numbers if you care about performance, since the Luau VM optimizes for their use, but technically you can use them. The Luau project discussions have some explanation as to why integers aren't a first-class user-facing type in Luau https://github.com/luau-lang/luau/discussions/242#discussioncomment-1707387.

Are integer operations supported at all, then? (i.e. bitwise operators)

Yes, through the bit32 library. In fact, more integer operations than are currently supported in LSL, like logical shift right and rotate operations. See https://luau-lang.org/library#bit32-library .

Retrieved from "https://wiki.secondlife.com/w/index.php?title=Lua_FAQ&oldid=1217019"