Talk:LLSD

This is a talk page associated with the Open Source Portal. Changes to all pages like this one can be tracked by watching the "Related Changes" for "Category:Open Source Talk Page"
Please sign comments you leave here by putting four tildes (~~~~) at the end of your comment. For more guidelines, see Talk Page Guidelines

LLSD JSON-ish notation

The login sequence appears to use "binary" LLSD for two of the variables, home and look_at. Unfortunately the format doesn't match up exactly with what is documented on this wiki. Sample values:

{'region_handle':[r255232, r256512], 'position':[r33.6, r33.71, r43.13], 'look_at':[r34.6, r33.71, r43.13]}

[r0.99967899999999998428,r-0.025334599999999998787,r0]

No size values, and the map keys are encased in single quotes. Is this normal? -- —The preceding unsigned comment was added by Eddy Stryker

Oh wow, notation format. Ummmmm, I never documented that, did I. LLSD started it's life as a json-like language, with a well defined serialization which is not actually documented. I'll get on that. Phoenix Linden 16:10, 6 April 2007 (PDT)

Documentation has been added. Phoenix Linden 17:34, 12 April 2007 (PDT)

Versioning?

Personally I miss a few things here, especially the versioning/compatibility. Sure you can asume that older request just have less keys in a map, and that might work well for things like statistics results. But as soon as you want to call a method, it is much better to make the version obvious. You can encode that in a key/value pair, but it is much better to have that in a fixed element for routing (you can route requests for inventories to different servers, depending on the interface version)

I see a Version:i1 in the notation example for the teleport, which makes it clear, that this element is used already in LL. Maybe it is hard for you to make it to a llsd envelop, but I think you guys will have great use for it.

<llsd>
  <method version="1" interface="simstatistics" message="result" />
...

or something like that.--Bernd Elswit 10:27, 18 May 2007 (PDT)

Efficient binary support

maybe a more efficient binary transport can be added, too? For example Adobe's ASCII85 allows better compression (however you need to escape less-than and ampersand and CDATA). Very efficient could be precalculated huffman tables for the most common file formats.

Sample code at the end of the page: http://www.javaworld.com/javaworld/javatips/jw-javatip117.html

--Bernd Elswit 10:26, 18 May 2007 (PDT)

Any time that we need to conserve space or cpu resources, we will use zlib routines or binary serialization. Phoenix Linden 12:41, 21 May 2007 (PDT)

hexadecimal ?

does the integer type allow hexadecimal values ? e.g. 0xffcc00 instead of entering 16763904 ?

SignpostMarv Martin 09:54, 31 May 2007 (PDT)

Binary map and array encoding ?

The description is not clear how the delimiters for map and array elements should be encoded. See here:

array      '[' + htonl(array.length()) + ''(''child0'','' child1'','' ...'')'' + ']'

map        '{' + htonl(map.length()) + ''((''key0'',''value0''),(''key1, value1''),'' ...'')'' + '}'

Especially the binary encoding of the

''

marked values is not defined. As I am sitting at an implementation of this, I will try with the supposed reference implementation llsd.py and see what that actually does for this part.

--Leffard Lassard 10:04, 27 March 2008 (PDT)

From my own research it is more like this:

array      '[' + htonl(array.length()) + child0 + child1 + ']'

map        '{' + htonl(map.length()) + 'k' + htonl(key0.length) + key0 + value0 + 'k' + htonl(key1.length) + key1 + value 1 ... + '}'

--Leffard Lassard 11:26, 28 March 2008 (PDT)

XML <-> Notation?

The XML format is, let's face it, not fun to read or write, for humans, in those weird cases where humans have to. Is there a serialization converter, somewhere, or should I just rig my own? --Ky Aska 06:12, 3 June 2008 (PDT)

Motivations Behind LLSD

Just a quick note to say i've been having discussions with peeps about why LLSD was invented in the first place; more discussions than I ever thought I would have on the subject. The best answer I could give was a) we like dynamic data models and b) it helps a bit to avoid the Fragile Binary Interface problem. If we ever get around to updating the intro, we may want to mention why dynamic data models are "nice" from an agile programming perspective and why FBI is bad. And BTW, here's the wikipedia entry on FBI: http://en.wikipedia.org/wiki/Fragile_binary_interface_problem . --Infinity Linden

base85 considered harmful

Please note: base85 as an encoding for binary elements in the XML serialization of LLSD data is not canonical.

about integers and real numbers

In the quote-unquote "real" world there are an infinite number of integers, and real numbers truly go up to infinity. Is there any work being done to support numbers higher than the current limits, and does this have any relevance to the LLSD standard? It seems to me we at least have to make it easily extendible, since it may be around for several (or even many) decades. Tammy Nowotny 06:34, 17 December 2008 (UTC)

base16 example

I'm having trouble with the Base16 example, if it is correct can someone point me to an RFC that explains it? -- Strife ^{(talk|contribs)} 18:05, 3 December 2010 (UTC)

binary header wrong?

The python implementation at https://bitbucket.org/lindenlab/llbase/src/78beb7385d87/llbase/llsd.py (see lines 1263-4) is expecting the header to be "<?llsd/binary?>\n" which differs considerably from the header in this doc: "<? LLSD/Binary ?>\n". I suspect the documentation is wrong but I'm not sure which would be considered correct else I would have changed the docs. Dahlia Trimble 21:23, 1 December 2011 (PST)

I'd say correct the documentation here since the cited file is likely newer than the documentation. -- Strife ^{(talk|contribs)} 22:14, 2 December 2011 (PST)

Why not move to support Protocol Buffers (protobuf), MessagePack, or BSON?

While it's true that, at the time LLSD was invented, there was no uniform, standardised way of serialising data with the flexibility of JSON but without its quirky limitations, this is not the case today, so I wonder why LL still stubbornly maintains LLSD around?

I totally get the notion of a 'platform-independent, long-term-survival', object serialisation format; around the late 1990s/early 2000s, I confess being frustrated at a lack of a general-purpose standard (Microsoft's SOAP came close, at the cost of verbosity...). I don't hate a XML-based format, and the amazing community of volunteers that keep writing libraries in all possible programming languages to support LLSD is truly flabbergasting (and they keep maintaining such libraries!); nevertheless, these days, we have quasi-industry 'standards', such as protobuf, MessagePack, or even Binary JSON (BSON), for which a gazillion libraries, tools, and automatic code generation software has already been written and maintained, on all possible platforms, environments and programming languages, by a gazillion volunteers as well as some industry leaders contributing code & developer time to keep them operational and interchangeable.

I mean, there is simply a limit to the time that a small team of LL & volunteer developers can continue to tinker with LLSD and figure out the best way to use/implement it for all possible requirements (past — for legacy software, present, and future). For example: when a more compact format was required (because wire transmission speed is at a premium, when you have the viewer requesting an astonishing amount of all kinds of data, allegedly in real time, to keep up with those beautiful 50+ FPS that people demand from the viewer running under current-generation graphic cards; also, with the move to Amazon's cloud — which is leased paying for bandwidth, disk space, number of messages and so forth — i.e. it's not a flat fee, like you were used to pay for when you owned your own 'bare metal' servers — the more compact the format, the more LL saves in running costs), you 'switched over' to a proprietary binary encoding, which works well... if you make a lot of assumptions at what CPU/other layers of hardware the endpoints are using. Start using more exotic combinations, and assumptions such as big-endian, or number of bits per word, start to fail — requiring even more code, even more interoperation tests and debugging, even more backporting of libraries to keep up with technological advancements, and so forth.

In other words, in order to freeze a serialisation format, which worked quite well when SL had a few thousands of users and a rather simple infrastructure to maintain, at a time when nobody could agree on a 'standard' serialisation format anyway, it made sense to create Yet Another Serialisation Format — sorry, I meant LLSD.

But it's 2022, and there are more than enough alternatives, backed up by industry giants and tens of thousands of volunteers, which will gladly keep up with any technological advances, and just ship version after version of their libraries — just link them to your code, rebuild it, and Bob's your uncle. You won't need to lift a single finger to worry about future-proofness of all that data.

Granted, you might argue... 'what about the legacy code we've got?' I'd ask back: 'why is it relevant?' I mean, even if we wished, there is simply no way to use an ancient Viewer 1.X to connect to the grid today. Since the server code is not open source, it's not as if someone could grab an ancient version of it and run their own SL-compatible grid. You, Linden Lab, and only you, decide what viewer(s) are allowed to connect to your grid, and under which conditions. You don't have to worry about anyone else. Even the third-party viewers share 99% of their code with your mainline, official SL Viewer. And all they would ever need to worry about was to replace the LLSD library by something else, and that would be it.

Consider the advantages that such a switch to a 'standard' serialisation protocol would bring to your developers. They could use off-the-shelf tools to develop, debug, and maintain that enormous codebase you've got — and easily expand such tools easily enough, instead of having to write specific LLSD-aware tools for that (or adapt existing ones — which may be possible if such tools are open source and easily pluggable/changeable, but not if you're using specific commercial tools, designed to work only with common standards used by the industry). In this decade, you already have a huge choice of frameworks where you basically just specify an API's skeleton (sometimes even visually, by just dragging cute symbols around a canvas!), click a button, and the tool automagically generates everything — from the full source code implementing that API, to a testing battery to validate the battery itself, to automatic documentation for each and every aspect of the API — which can be instantly published on an internal website (or possibly even to an external one — but doing it instantly, not requiring half a decade for someone to reach out to this wiki and change a few lines...) or to a reference card/PDF... and all of that for free.

Think how much wasted developer time could be saved by your team, to do much more productive — and creative — things instead!

Anyway, enough ranting. Actually, I was just wondering if you have a XLST for LLSD somewhere, because I couldn't find one around here (perhaps I didn't specify the right query...). And to validate legitimate XML (which LLSD is), it's pointless without an adequate XLST — that's the whole point of using XML, right? ;-) I mean, there is no real advantage of XML over any other representation format, except that XML can be easily validated using XLST (and other 'competitors' to XML have had to retrofit their serialisation language description to support similar schemas and try to 'emulate' the way XML works). So... is someone kind enough to point me to the XLST? Maybe it's deeply hidden somewhere in the viewer source code — which I really haven't delved into since 2007?

I'm just wondering...

Oh, and there is no rush, really; it's highly likely that I will only see an answer here in, well, half a decade. In the meantime, maybe you moved on from LLSD anyway :)

A girl can only dream...

Cheers,

— Gwyneth Llewelyn (talk) 13:31, 1 August 2022 (PDT)

Replying to myself, after 'merely' a month: there is a DTD for LLSD. — Gwyneth Llewelyn (talk) 23:59, 1 September 2022 (PDT)

And replying to myself once more, a few months past: there is a DTD, sure, but getting it to be accepted by XML+DTD validators, now that's a completely different story!! — Gwyneth Llewelyn (talk) 12:09, 23 February 2023 (PST)