LSL HTTP server/design

Goals

Create an alternative to the XMLRPC server and email gateway for communication with LSL scripts initiated from outside Second Life that is easy to use and scalable. Extra bonus for enabling LSL -> LSL communication at the same time.

Design

(Updated by Kelly Linden 11:19, 2 September 2008 (PDT))

This design is in flux during development. Please comment on the discussion page, as comments here tend to get lost when the design is updated. Thanks.

URLs / Namespace

url: http://sim3015.aditi.lindenlab.com:12046/cap/3ff4f3f2-ea08-76c1-cef6-a22b4a573a7c
request: curl http://sim3015.aditi.lindenlab.com:12046/cap/3ff4f3f2-ea08-76c1-cef6-a22b4a573a7c{untrusted-arg}
{untrusted-arg} examples: /foo/bar, ?foo=bar, /foo/bar?one=1

LSL

• key llRequestURL()

Request a new LSL Server public URL.

An http_request event will be triggered with success or failure and include the returned key

lsl: request_id = llRequestURL(); 

• key llRequestSecureURL()

Similar to llRequestURL except requests an HTTPS / SSL URL.

An http_request event will be triggered with success or failure and include the returned key

lsl: request_id = llRequestSecureURL(); 

• llReleaseURL(string url)

Clear the specific URL, used for both secure and non-secure URLs.

lsl: llReleaseURL("http://sim3015.aditi.lindenlab.com:12046/cap/3ff4f3f2-ea08-76c1-cef6-a22b4a573a7c");

• http_request(key id, string method, string body)

Event triggered when an URL is hit:

• id is unique to this request
• Supported methods are GET/POST/PUT/DELETE
• body: The body of the request.

Event also triggered with response to llRequestURL and llRequestSecureURL

• id matches the key returned by llRequestURL or llRequestSecureURL
• method == URL_REQUEST_GRANTED for success, URL_REQUEST_DENIED for failure to get an URL
• body is the public URL. If unable to get a public URL body will be empty.

• llHTTPResponse(key id, integer status, string body)

Send body to the requester with status code status

• id is the id from http_request that maps to the specific request

• string llGetHTTPHeader(key id, string header)

Returns the string for the specified header in the specified request

• Supported headers are:

• "x-script-url": The base url, as originally recieved from llRequestPublicURL
• "x-path-info": Any trailing path information from the requested url
• "x-query-string": Any query arguments, the text past a ? in the url
• "x-forwarded-for": The host that made the request
• "user-agent": The user-agent header as reported by the requester

requested url: https://sim3015.aditi.lindenlab.com:12043/cap/a7717681-2c04-e4ac-35e3-1f01c9861322/foo/bar?arg=gra
x-script-url: https://sim3015.aditi.lindenlab.com:12043/cap/a7717681-2c04-e4ac-35e3-1f01c9861322
x-path-info: /foo/bar
x-query-string: arg=gra

• changed(integer change)

• CHANGED_REGION_RESTART: New changed() event triggered on region startup.

• integer llGetFreeURLs()

Returns the number of URLs available to this script.

Simulator

What the simulator needs to do:

• Pass the headers mentioned above into the lsl script:
• Clear/invalidate caps in some situations:

• Caps will automatically be revoked when the region goes down.
• When the script is reset, resaved or deleted.
• Object removed from world
• Object region change
• Object owner change
• Region startup (clear all by region)
• Script request (llReleasePublicURL)

• Scripts should know when they need to re-request urls

Goal: Scripts should be able to know when public urls are lost and recover from that loss

• Existing events: default/state_entry (new script), on_rez, changed(region cross, owner)
• New events: cap server is restarted, region is restarted

• The simulator tracks LSL Cap URLs as a Limited Script Resource.

• This is a first use for a more general Limited Script Resource system that should eventually also handle script memory and cpu cycles.
• Not all requests for an url will succeed, the scripter is expected to handle the failure case.
• The number of available urls will be based on the amount of land owned in the region
• integer llGetFreeURLs() returns how many URLs are available.

Cap Server

• Grants caps that that look like:

http://sim3015.aditi.lindenlab.com:12046/cap/3ff4f3f2-ea08-76c1-cef6-a22b4a573a7c

• Optionally grants SSL caps that look like:

https://sim3015.aditi.lindenlab.com:12043/cap/a7717681-2c04-e4ac-35e3-1f01c9861322

• Trailing path after the cap ID is passed to the internal url as part of the header:

request: http://sim3015.aditi.lindenlab.com:12046/cap/3ff4f3f2-ea08-76c1-cef6-a22b4a573a7c/foo/bar?one=1
forwarded to: <internal_url>
  w/ headers: x-script-url: http://sim3015.aditi.lindenlab.com:12046/cap/3ff4f3f2-ea08-76c1-cef6-a22b4a573a7c
              x-path-info: /foo/bar
              x-query-string: one=1

Questions / Issues

• Define response codes - no script/object found, throttled.
• Define mime-handling

  We will transcode incoming data to UTF-8. Per standards if no type is defined we assume the web standard of 'iso-8859-1'.

  Content type of responses will be 'text/plain; charset=utf-8'

Interface Requirements

• No GUI components.
• LSL Functions are written in stone, must get them right.

Performance Requirements

This should add no database, assetserver or viewer load.

Simulator:

• In general load should be no higher than existing alternatives (xmlrpc, llemail and llhttprequest) for any single action.
• Connections need to be throttled.

It would be nice if this could happen before the simulator on a per-cap basis, but throttling in the simulator handler would probably work as well.

Capserver:

• TBD: More info on cap server's load characteristics. We believe this will be ok, especially in the world of unified cap server, multi-threaded apache and memcached - but we need to verify.
• Separate apache config and/or resource pools for agent caps vs task/LSL caps would give us opportunities to ensure good behavior for agent critical features at the expense of LSL/Caps performance if needed and allow us to minimize the impact of DOS style attacks against the LSL caps.

TODO:

• Get statistics on # of XMLRPC connections, llHTTPRequests and llEmails per unit of time per region to set expectations for level of usage.

We currently have 2 XMLRPC servers each processing about 30-40 concurrent requests.
I don't know the actual rate of requests, but I do know that we start failing at ~150
concurrent requests (out of a theoretical max of 200) per server, or ~300 total.
Kelly Linden 12:52, 14 May 2008 (PDT)

Security Impact

Creating a server accessible in any way from outside needs to be done with care. The cap server already does this, and security concerns should already be handled here. This isn't something to take for granted though.

• Scripts should not be blocked from using llHTTPRequest to contact the public interface of the cap server.
• Need to be careful about this since these are requests that originate from inside our network.

Limitations

• Size of the body of the requests will be limited to 2k bytes.
• Size of headers of requests will be limited to 255 bytes.
• The size of responses to requests is not currently limited, but this is subject to review during testing.
• There is a cap of 64 in flight requests per script.
• We may throttle the rate we accept hits at the CAP server level as well. This is possible, but has not yet been decided.

Interactions

• Relies on capabilities and the cap server
• Effects LSL development including MONO
• Limited Script Resources

Testing

• Some example scripts: LSL http server/examples

TODO: Export internal test plan pages.

Previous Design

Previous design and comments