LSL HTTP server

From Second Life Wiki
Jump to navigation Jump to search

Goals

Create an alternative to the XMLRPC server and email gateway for communication with LSL scripts initiated from outside Second Life that is easy to use and scalable. Extra bonus for enabling LSL -> LSL communication at the same time.

Design

URLs / Namespace

public url: https://sim123.agni/cap/f23b4b94-012d-44f2-bd0c-16c328321221
request: curl https://sim123.agni/cap/f23b4b94-012d-44f2-bd0c-16c328321221/{untrusted-path}

LSL

  • llHTTPServerRequestURL(string trusted_path)
Request a new LSL Server public URL.
The 'trusted-path' will be passed into all http_server events from the public url from this request
An http_server event will be triggered with success or failure and include the public url
lsl: llHTTPServerRequest(SERVER_REQUEST_URL,"test-case/foo");
  • llHTTPServerClearURL(string url)
Clear the specific public url
lsl: llHTTPServerRequest(SERVER_CLEAR_URL,"https://sim123.agni/cap/f23b4b94-012d-44f2-bd0c-16c328321221");
  • llHTTPServerClearAllURLs()
  • Clears all URLs associated with this task.
lsl: llHTTPServerRequest(SERVER_CLEAR_ALL_URLS,"");
  • llHTTPServerGetURLs()
Requests a list of all URLs associated with this task
Results are handled by the http_server event
lsl: llHTTPServerRequest(SERVER_GET_URLS,"");
  • http_server(string method, list meta, string body)
Event triggered when an URL is hit or with results of llHTTPServerRequest
  • method is GET/POST/PUT/DELETE for normal hits
  • meta: A list of headers of the format [header1,value1,header2,value2], standard headers:
  • "x-trusted-path": A string as set when the url is requested, essentially trusted meta data associated with a specific cap.
  • "x-untrusted-path": A string that is any trailing characters from the external request
  • "x-forwarded-for": The host that made the request
  • "x-forwarded-for-port": The port from the request (is this the port of the requester?)
  • body: The body of the request.
  • method is one of the following string constants for special case events.
  • URL_REQUEST_GRANTED: The script now has a public url. The url is in the body.
  • URL_REQUEST_FAILED: Unable to get an URL, if possible a reason is given in the body.
  • GET_URLS: meta is a list of urls held by this task
  • URL_LOST: body is the url that was lost.
Is there a case where we won't know what urls were lost, just that they all were?
Kelly 15:30, 14 May 2008 (PDT)

http_server(string method, list meta, string body)
{
   if (method == URL_REQUEST_GRANTED)
   {
      string public_url = body;
      // We now have the url, probably want to let someone else know - llHTTPRequest or similar
   }
   else if (method == URL_REQUEST_FAILED)
   {
      // Retry and/or let owner know we are broken.
   }
   else if (method == "GET")
   {
      integer i = llListFindList(meta,TRUSTED_PATH);
      string trusted = llList2String(meta,i+1);
      if (trusted == "echo")
      {
         llHTTPServerResponse(200,body);
      }
      else if (trusted == "status")
      {
         i = llListFindList(meta,UNTRUSTED_PATH);
         string untrusted = llList2String(meta,i+1);
         llRequestAgentData(AGENT_ONLINE,(key)(untrusted));
         // !! Won't have anything to return via llHTTPServerResponse until we get a dataserver() event!
      }
   }
}
  • integer llHTTPServerResponse(integer status, string body)
Send body to the requester with status code status
Returns TRUE if the response was sent, FALSE if the connection was no longer available to respond on.
Can we do this outside this event?  
We need to time out pending requests anyway if the script takes too long.
Kelly Linden 15:30, 14 May 2008 (PDT)

If we're going to respond outside of the event, every request will need an id. 
Another bad thing about doing it that way is that it allows people to hold 
onto a file descriptor for a long time. 
Phoenix 16:32, 14 May 2008 (PDT)

Any more than if they plunk a while(1) in there? Or just take a really long 
time to do whatever?  We need to handle the case of 'they 
are taking too long to send a response' even if we limit them to sending 
responses from within the event.  Allowing responses to be sent outside the 
event allows for a lot more flexibility - sensors, notecard reading, link messages 
and all the dataserver requests all become possibilities.
Kelly Linden 13:46, 16 May 2008 (PDT)

Simulator

What the simulator needs to do:

  • Passes the method, body and these headers into the lsl script:
  • 'x-forwarded-for' and 'x-forwarded-for-port': for the ip/port of the requester.
  • 'x-trusted-path': The string passed into the url request
  • 'x-untrusted-path': A header appended by the cap server containing any path appended to the cap
  • Clear/invalidate caps in some situations:
Caps will be automatically be revoked when the region goes down.  
The caps can also be granted using the object as the "key" - and revoked when the object is destroyed. 
Zero Linden 14:37, 16 November 2007 (PST)

Need more details about this.
Kelly Linden 11:15, 14 May 2008 (PDT)
  • Object removed from world
  • Object region change
  • Object owner change
  • Region startup (clear all by region)
  • Script request
  • Trigger an lsl http_server event with URL_LOST whenever a cap becomes invalid but the object and script still exist
  • Object first rezed
  • Object changed regions
  • Region restarted
  • Capability was successfully removed by script request
  • Object owner changed
  • This is a first use for a more general Limited Script Resource system that could eventually also handle script memory and cpu cycles.
  • Not all requests for an url will succeed, the scripter is expected to handle the failure case - we must let the scripter know when it fails.
  • The number of available urls will be based on the amount of land owned in the region
  • Need the ability to figure out how many/much of a resource is used and available
Do we need to support transient users or objects (where they own no land)?
Do we have special handling for sandbox regions?
How do we handle rented parcels in private estates?  What controls (if any) do we need to give estate owners?
Kelly Linden 15:58, 14 May 2008 (PDT)

Cap Server

This is the cap server on the same host as the region

  • Grants caps per normal that look like:
public url: https://sim123.agni/cap/f23b4b94-012d-44f2-bd0c-16c328321221
  • Trailing path after the cap ID is passed to the internal url as the 'x-untrusted-path' header:
request: https://sim123.agni/cap/f23b4b94-012d-44f2-bd0c-16c328321221/foo/bar
forwarded to: <internal-url>
  w/ header: x-untrusted-path: foo/bar
  • Caps are cleared when a region shuts down or restarts
  • Caps are cleared by lsl request (via specific lsl function)
  • Caps are cleared by region request (object deleted or moves to another region)
  • Simulator must be able to verify a cap still exists.
  • Separate apache config and/or resource pools for agent and viewer related caps vs task/LSL caps
    • Would give us opportunities to ensure good behavior for agent critical features at the expense of LSL/Caps performance if needed
    • Potentially allow us to minimize the impact of DOS style attacks against the LSL caps.

Questions / Issues

  • Should these caps time out?
  • Define response codes - no script/object found, request throttled.
  • Define mime-handling
I believe that we should, like the llHTTPRequest() call, be very clear in our handling of bodies and mime-types.  
In particular, accept only text/* mime types, and be sure to do proper charset handling and conversion into the 
UTF-8 that LSL strings use.  (The code should all be cullable from the llHTTPRequest() implementation.)
Zero Linden 14:37, 16 November 2007 (PST)

Interface Requirements

  • No GUI components.
  • LSL Functions are written in stone, must get them right.

Performance Requirements

This should add no database, assetserver or viewer load.

Simulator:

  • In general load should be no higher than existing alternatives (xmlrpc, llemail and llhttprequest) for any single action.
  • Connections need to be throttled.
It would be nice if this could happen before the simulator on a per-cap basis, but throttling in the simulator handler would probably work as well.

Capserver:

  • TBD: More info on cap server's load characteristics. We believe this will be ok but we need to verify.
  • Separate apache config and/or resource pools for agent caps vs task/LSL caps would give us opportunities to ensure good behavior for agent critical features at the expense of LSL/Caps performance if needed and allow us to minimize the impact of DOS style attacks against the LSL caps.

TODO:

  • Get statistics on # of XMLRPC connections, llHTTPRequests and llEmails per unit of time per region to set expectations for level of usage.
We currently have 2 XMLRPC servers each processing about 30 concurrent requests.
I don't know the actual rate of requests, but I do know that we start failing at ~150
concurrent requests (out of a theoretical max of 200) per server, or ~300 total.
Kelly Linden 12:52, 14 May 2008 (PDT)

Security Impact

Creating a server accessible in any way from outside needs to be done with care. The cap server already does this, and security concerns should already be handled here. This isn't something to take for granted though.

  • Scripts should not be blocked from using llHTTPRequest to contact the public interface of the cap server.
  • Need to be careful about this since these are requests that originate from inside our network.

Limitations

  • Size of the body of the requests will probably need to be limited. At least to something that will fit within script memory - 2k is probably right.
This is needed to keep from overloading scripts and to reduce the overall potential load of handling large sets of data.  
We probably also need to limit the size of returned data
Kelly Linden 12:58, 14 May 2008 (PDT)
  • Incoming requests will need to be throttled at some rate. Script event queuing acts as a throttle for what connections the script will be able to hanlde, but that probably isn't enough. I would guess that near the rate of llHTTPRequest throttling is probably good - ~1 / second / cap.
Can we throttle before we reach the sim? As part of apache, the cap server or squid?
Kelly Linden 12:58, 14 May 2008 (PDT)
  • Need to limit the size of the 'trusted-path' and 'untrusted-path'

Interactions

Testing

TODO: Export internal test plan pages, once they exist. :)

Previous Design

Previous design and comments

Retrieved from "https://wiki.secondlife.com/w/index.php?title=LSL_HTTP_server&oldid=67704"