LSL HTTP server

Goals

Create an alternative to the XMLRPC server and email gateway for communication with LSL scripts initiated from outside Second Life that is easy to use and scalable. Extra bonus for enabling LSL -> LSL communication at the same time.

Design

(Updated by Kelly Linden 11:28, 13 June 2008 (PDT))

This design is in flux during development. Please comment on the discussion page, as comments here tend to get lost when the design is updated. Thanks.

A teaser image.

URLs / Namespace

public url: https://sim123.agni/cap/f23b4b94-012d-44f2-bd0c-16c328321221
request: curl https://sim123.agni/cap/f23b4b94-012d-44f2-bd0c-16c328321221{untrusted-arg}
{untrusted-arg} examples: /foo/bar, ?foo=bar, /foo/bar?one=1

LSL

• key llRequestPublicURL(string trusted_path)

Request a new LSL Server public URL.

The 'trusted-path' will be available as a header for all http_request events from the public url from this request

An http_request event will be triggered with success or failure and include the returned key

lsl: request_id = llRequestPublicURL("test-case/foo"); 

• llReleasePublicURL(string url)

Clear the specific public url

lsl: llReleasePublicURL("https://sim123.agni/cap/f23b4b94-012d-44f2-bd0c-16c328321221");

• http_request(key id, string method, string body)

Event triggered when an URL is hit:

• id is unique to this request
• Supported methods are GET/POST/PUT/DELETE
• body: The body of the request.

Event also triggered with response to llRequestPublicURL

• id matches the key returned by llRequestPublicURL
• method == URL_REQUEST_GRANTED for success, URL_REQUEST_DENIED for failure to get an URL
• body is the public URL. If unable to get a public URL body will be empty.

• llHTTPResponse(key id, integer status, string body)

Send body to the requester with status code status

• id is the id from http_request that maps to the specific request

• string llHTTPHeader(key id, string header)

Returns the string for the specified header in the specified request

• Potential headers are:

• "x-trusted-path": The portion of the path as specified in the request
• "x-untrusted-argument": A string that is any trailing characters from the external request
• "x-forwarded-for": The host that made the request
• "x-forwarded-for-port": The port from the requester

• system_changed(integer change) (possibly a subset of existing changed event)

Tells scripts about system events, allows script to maintain public urls

• SYSTEM_CAPS_RESET
• SYSTEM_REGION_RESTART

Simulator

What the simulator needs to do:

• Pass these headers into the lsl script:

• 'x-forwarded-for' and 'x-forwarded-for-port': for the ip/port of the requester.
• 'x-trusted-path': A string specified by the user at url request
• 'x-untrusted-argument': A header appended by the cap server containing any path and/or arguments appended to the cap

• Clear/invalidate caps in some situations:

• Caps will automatically be revoked when the region goes down.
• When the script is reset, resaved or deleted.
• Object removed from world
• Object region change
• Object owner change
• Region startup (clear all by region)
• Script request (llReleasePublicURL)

• Scripts should know when they need to re-request urls

Goal: Scripts should be able to know when public urls are lost and recover from that loss

• Existing events: default/state_entry (new script), on_rez, changed(region cross, owner)
• New events: cap server is restarted, region is restarted

• The simulator tracks LSL Cap URLs as a Limited Script Resource.

• This is a first use for a more general Limited Script Resource system that should eventually also handle script memory and cpu cycles.
• Not all requests for an url will succeed, the scripter is expected to handle the failure case.
• The number of available urls will be based on the amount of land owned in the region
• integer llGetFreePublicURLs() returns how many public URLs are available.

Cap Server

• Grants caps per normal that look like:

public url: https://sim123.agni/cap/f23b4b94-012d-44f2-bd0c-16c328321221

• Trailing path after the cap ID is passed to the internal url as part of the header:

request: https://sim123.agni/cap/f23b4b94-012d-44f2-bd0c-16c328321221/foo/bar?one=1
forwarded to: <internal_url>
  w/ header: x-untrusted-path: foo/bar?one=1

• Regions are aware of cap server resets (to notify scripts)

Questions / Issues

• Should these caps time out?
• Define response codes - no script/object found, throttled.
• Define mime-handling

I believe that we should, like the llHTTPRequest() call, be very clear in our handling of bodies and mime-types.  
In particular, accept only text/* mime types, and be sure to do proper charset handling and conversion into the 
UTF-8 that LSL strings use.  (The code should all be cullable from the llHTTPRequest() implementation.)
Zero Linden 14:37, 16 November 2007 (PST)

Interface Requirements

• No GUI components.
• LSL Functions are written in stone, must get them right.

Performance Requirements

This should add no database, assetserver or viewer load.

Simulator:

• In general load should be no higher than existing alternatives (xmlrpc, llemail and llhttprequest) for any single action.
• Connections need to be throttled.

It would be nice if this could happen before the simulator on a per-cap basis, but throttling in the simulator handler would probably work as well.

Capserver:

• TBD: More info on cap server's load characteristics. We believe this will be ok, especially in the world of unified cap server, multi-threaded apache and memcached - but we need to verify.
• Separate apache config and/or resource pools for agent caps vs task/LSL caps would give us opportunities to ensure good behavior for agent critical features at the expense of LSL/Caps performance if needed and allow us to minimize the impact of DOS style attacks against the LSL caps.

TODO:

• Get statistics on # of XMLRPC connections, llHTTPRequests and llEmails per unit of time per region to set expectations for level of usage.

We currently have 2 XMLRPC servers each processing about 30-40 concurrent requests.
I don't know the actual rate of requests, but I do know that we start failing at ~150
concurrent requests (out of a theoretical max of 200) per server, or ~300 total.
Kelly Linden 12:52, 14 May 2008 (PDT)

Security Impact

Creating a server accessible in any way from outside needs to be done with care. The cap server already does this, and security concerns should already be handled here. This isn't something to take for granted though.

• Scripts should not be blocked from using llHTTPRequest to contact the public interface of the cap server.
• Need to be careful about this since these are requests that originate from inside our network.

Limitations

• Size of the body of the requests will probably need to be limited. At least to something that will fit within script memory.

This is needed to keep from overloading scripts and to reduce the overall potential load of handling large sets of data.  
We also need to limit the size of returned data in llHTTPResponse()
Kelly Linden 12:58, 14 May 2008 (PDT)

• There will be a max number of 'in process' requests allowed per script (requests that have been made but not responded to or timed out yet), requests when past this limit will receive an appropriate http status code. This will be roughly on the order of maximum number of queued events in a script, give or take a factor of 2.
• We may wish to throttle at the CAP server level as well, if possible.

Interactions

• Relies on capabilities and the cap server
• Effects LSL development including MONO
• Limited Script Resources

Testing

TODO: Export internal test plan pages.

Previous Design

Previous design and comments