Talk:LSL HTTP server/design

I believe one of the worst-case scenarios from XML-RPC today is the InnerLife / Mandala Web Temple architecture. Last I experimented (with an early format) we got stable results at 4 second intervals (this was back in mid-2006); We wanted faster (not enough for a proper biofeedback interface) so most dropped the project. Be aware that we wanted something that could be sustained for long intervals (3-4 hours straight, possibly with multiple avatars contributing toward 'merged' results). I'm not certain what the current request load is today, you can check with 'Carlos Tapioca' in-world as he is still actively perusing this. I'll also direct him here. Alan Kiesler 22:42, 6 April 2008 (PDT)

Parsing variables

There probably needs to be a routine to provide access to preparsed form/CGI/response variables. Even with the URL decoding parsing a set of response variables in LSL is going to require a lot of rather inefficient LSL code duplicated (often badly) across the grid. Having seen people implementing these routines over and over again on the web, often poorly, I think it would help reduce the load a LOT to just have a known good implementation in the server.

Possible APIs:

1. list llHTTPvariableList(method) - returns a list of name-value pairs for the form variables provided by the URL (GET) or body (POST).

2. string llGetHTTPVariable(method, name) - returns a single value from the appropriate collection. The additional pseudo-method "ANY" will look under the body then the URL. -- Argent Stonecutter 14:32, 28 September 2008 (PDT)

URL_REQUEST_TIMEOUT

I think there should be a URL_REQUEST_TIMEOUT method. That way the script has a chance to catch the event and send the user some meaningful information before the connection actually cut. A user could have two scripts, one that handles the HTTP serving while the other gathers data. If the data gathering sends the data as it receives it then the HTTP script can cache the data and when it receives all of it or the timeout is thrown it can send the data it has. If the script misses the window on the connection and the event was queued then it should be dequeued and not triggered. -- Strife Onizuka 16:53, 16 May 2008 (PDT)

- There are no plans for such an event. Requests will time out which will send a timed out error / status (504 Gateway Timeout) to the requester. Events are a scarce resource in LSL, also anything that lets you do 'one last thing' before a time out either creates a time out loophole or is already too late. If you find your requests are frequently timing out you will need to adjust your code to return smaller result sets more frequently. - Kelly Linden 10:55, 13 June 2008 (PDT)

HTTP_STATUS_*

Might be beneficial to have HTTP_STATUS_* code constants. -- Strife Onizuka 20:34, 16 May 2008 (PDT)

- Specifically which codes do you want defined this way? I'd like to keep the number of constants to a minimum, would _OK (200), _MISSING (404) and _ERROR (500) be enough? - Kelly Linden 10:47, 13 June 2008 (PDT)

Those sound good. -- Strife Onizuka 11:42, 13 June 2008 (PDT)

Should have one for that evil Linden error 499 (for use with the llHTTPRequest side of things). -- Strife Onizuka 23:30, 16 June 2008 (PDT)

robots.txt

Do we want google spidering the caps urls? -- Strife Onizuka 20:34, 16 May 2008 (PDT)

- This is a good point, and probably not. - Kelly Linden 10:56, 13 June 2008 (PDT)

Payload Size Limits / Content

Will the data payload size be similar to the current http_response limits? And will we be able to implement something like reverse http with it?

Thanks,

BlueWall <j>

- There will be payload size limits for PUT and POST, and a max lengh for x-untrusted-argument. And no, you will not be able to implement something like reverse http. There is no provision for 'upgrade' and requests time out rather quickly (<25 seconds) which will make any long poll mechanism difficult - Kelly Linden 15:10, 17 June 2008 (PDT)

Content-Type

What about the content-type and more importantly charset? Will it be utf-8? Are URLs in utf-8? Is it "text/plain; charset=utf-8" then? - Thomas Shikami 20:31, 16 June 2008 (PDT)

- Yes, responsees will be "text/plain; charset=utf-8". We will attempt to transcode the body of POST and PUT actions to UTF-8 before they hit the script. URLs will be url encoded - including the 'x-untrusted-argument' which will remain url encoded until (if) the scripter uses llUnescapeURL. - Kelly Linden 15:01, 17 June 2008 (PDT)

I vote for a mechanism to set the response content-type arbitrarily (except the charset, where UTF-8 is fine). If worried about arbitrary response content-types, perhaps the initial implementation could limit the allowable content-types to the set ["text/plain","text/xml","application/json", ...]. Then a future implementation could expand the allowed types without changing the function API.

Consequently, it would also be useful to be able to accept "text/xml", "application/json" and perhaps others.

In order to discourage phishing, the implementation could automatically set the response content-type to text/plain if the user agent sent identifies any of the major web browsers (IE/Mozilla/Webkit..). - Cenji Neutra 08:09, 16 July 2008 (PDT)

While more content types would be useful, they will probably not be implemented until a later date. Kelly Linden

Landowner only usage?

• Not all requests for an url will succeed, the scripter is expected to handle the failure case.
• The number of available urls will be based on the amount of land owned in the region

As the majority of my work is placed on land I don't own, does this mean I'm going to be in "last place" when it comes to getting URLs? And if it fails, am I then left to using XML-RPC and email again? Seems pointless to build this in if I still need to have the script set up the other systems, with their associated timers and failsafes. Also, what kind of limits are we talking about per sim? I'm thinking of a rental system I made with approx 40-50 rental units on a sim. Each one would need to be updated from the web - and http in would be perfect, but only if the limits make it feasible and reliable. ~~ Hippyjim Starbrook‎ 15:37, 15 July 2008 (PDT)

I would say that the vast majority of objects in SL that communicate with other objects and external services are owned by residents that either don't own land or are on parcels not owned by the object owner. In our case we have 100s-of-thousands of vendors, lease devices etc., deployed by our customers, usually on land they don't own.

Would I be correct in assuming the number of available URLs for all the objects on a parcel is limited by the amount of land owned in the region by the parcel owners - independently of who owns the object that contains the script? (i.e. by allowing a scripted object on their land a land owner is giving implicit permission for use of URLs from their quota - shared between all the other objects on the owners land in that region). Correct? - Cenji Neutra 08:17, 16 July 2008 (PDT)

---

Public urls follow very closely the model of prim limits:

• All (unattached) objects will use the resource pool from the parcel they are over.
  • There are 15,000 available urls per region so 1 per prim essentially.
  • These pools work like prim limits - each parcel owner has a max count for urls in the region just like they do for prims in the region.
• Attached objects will use the resource pool from the agent they are attached to.
  • There are 38 available urls per agent, so 1 per attachment point essentially.
  • Vehicles will end up in the agent's pool the first time they cross a parcel boundary.
• As can be seen from the design if you want an url you must request it, if none are available an URL_REQUEST_DENIED event will be triggered instead of an URL_REQUEST_GRANTED event.
• If an object that is already using public URLs attempts to move into a parcel that does not have enough available public urls the movement will be blocked and the object will not move. (NOTE!! Vehicles do not follow this rule, see above)
  • This does not apply to crossing region boundaries since all urls are released and cleared when an object changes region and new urls must be requested.
• The only time any owner priority takes place is when a parcel goes beyond its limit. This can only happen when someone who owns multiple parcels in a region sells or deeds one of them such that their remaining available urls drops below what is being used on their remaining parcels.
  • In this case the same logic is followed as for prim returns, namely:
  • Newest to oldest temp objects, newest to oldest 'other' objects, newest to oldest group objects, newest to oldest owner objects, newest to oldest selected objects.
  • Example: You have 2 parcels that each support 10 urls, parcels A and B. You are using 15 urls on B which is fine since you have a total pool of 20 for the region. If you sell A, then you will only have a total pool of 10 available and the above process will be used to return objects using urls until only 10 are being used.

In short the only time objects are potentially automatically returned due to urls being used is when a parcel is sold.

I hope this makes it clearer?

- Kelly Linden 16:03, 17 July 2008 (PDT)

Yes, so much clearer - I especially like the idea of the limits matching the prim limit - one per (allowed) prim is a nice simple rule to remember. Thanks for clearing that up! I'm a little concerned about the possibility of mobile objects hitting an invisible "parcel wall" if they attempt to cross into (or through) a parcel that has a url-hungry object in it. Will there be some kind of event raised to warn the script that it's being prevented from entering the parcel? Could we pass llGetFreePublicURLs() a location vector to check the free urls in our target location? Hippyjim Starbrook‎ 01:05, 18 July 2008 (PDT)

---

There is a bullet point above about vehicles which I don't think your question takes into account, and I realize my statement about moving objects can be taken to include vehicles (so I added a clarifying comment).

One of the difficulties here is that there is no clear marker on what is or isn't a vehicle. The rule used in the rest of the system is 'anything with someone sitting on it' - which is why selected / sat upon have special object count behaviors. If we just applied this to public urls however we could end up people causing weird behavior by sitting on objects. So for urls we do a lazy conversion. If you sit on an object, that object's urls are still accounted by the parcel the object is over. They stay that way until the object crosses a region boundary while someone is sitting on it. At this point we know it is a vehicle and we don't want it getting returned or you running into walls so the needed urls will count towards the sitting agent's pool count. Technically the first sitting agent with enough available urls in the case of multiple sitting agents. The problem might be if a resident already has attachments that use all their urls, then that resident will not be able to drive vehicles across region boundaries no matter what the parcel url counts look like.

Region crossings are still special here since urls are lost, however there is probably some room for optimization such that new urls requested after you cross a region boundary immediately come from the resident's pool.

This probably makes things less clear. :D Kelly Linden 13:49, 18 July 2008 (PDT)

Kelly, can you clarify "The problem might be if a resident already has attachments that use all their urls, then that resident will not be able to drive vehicles across region boundaries no matter what the parcel url counts look like." I assume that only applies for vehicles with scripts in them that are using this API. -- Argent Stonecutter 15:00, 28 September 2008 (PDT)

---

Thanks for the explanation about vehicles - almost, but not quite, entirely clear as mud ;) I was actually thinking of automatic roaming objects - with nobody sat on them, or deciding where they go. An example is the "greeter bots" that float towards you when you enter a store. Another example might be an object that's scanning the whole sim for new visitors, or simply to rez objects in a particular position. Although vehicles have their riders url pool to draw on, an "unmanned" object wouldn't. As events are at a premium, a way to allow scripters to prepare for being unable to enter a parcel would be to allow checking of another parcels available urls before you enter it - which is where passing a location to llGetFreePublicURLs() would be helpful. Either way, I suspect this is not a situation that'll happen very often, and I'd hate for the whole http server launch to be slowed by dealing with every possible contingency - so please put it on the "nice to have" list! Hippyjim Starbrook‎ 14:35, 18 July 2008 (PDT)