From Second Life Wiki
[8:59] Scouse Linden: babbage is on his way [8:59] You: hello [8:59] Phantom Ninetails: Oh wow there he is [8:59] Entering god mode, level 200 [8:59] Scouse Linden: Its only 8:59 :) [8:59] Phantom Ninetails: Well I was talking with Periapse a little while ago and that's what we figured [9:00] You shout: i've turned off scripts while we hold the office hours [9:00] Phantom Ninetails: The wiki says it starts at 8 [9:00] Scouse Linden: oh damn daylight savings..... [9:00] Scouse Linden: It hasnt changed for us [9:00] Phantom Ninetails: Ah ^_^ [9:01] You: let me ping zero, we wanted to talk to him about httprequest and unicode [9:01] Phantom Ninetails: :) [9:01] Scouse Linden: Since no one else is around, if you've got anything phantom go ahead and ask [9:01] Phantom Ninetails: Nope I've got nothing [9:02] Scouse Linden: I guess others probably turned up and left by now [9:02] You: yep, let me ping the group [9:02] Phantom Ninetails: Actually not many showed up at 8 either ;) [9:02] Scouse Linden: yeh it has slowed down [9:02] Scouse Linden: I bet it wouldnt be the same if we did it on the main grid [9:02] Phantom Ninetails: Only Periapse Linden and Becky Pippen showed up before [9:02] Phantom Ninetails: lol, yeah you might get more there [9:03] You: maybe we should ping the group on aditi too? [9:04] Phantom Ninetails: Couldn't hurt [9:04] Scouse Linden: Go ahead! [9:04] Scouse Linden: What things are the most important to you to get fixed next? [9:05] Phantom Ninetails: Control event. [9:05] Scouse Linden: Fixed it today [9:05] Phantom Ninetails: Yay :) [9:05] Scouse Linden: Infact I was writing tests for events [9:05] Scouse Linden: I'll put them on t he wiki [9:05] Scouse Linden: It will be fixed in the next refresh [9:05] Phantom Ninetails: Alrighty :) [9:06] Joshua Linden is Online [9:07] You: we've uncovered some weird lsl2 behaviour recently [9:07] Phantom Ninetails: Weird? [9:07] You: see the language test 2 [9:07] You: we added tests for while, for and do while [9:07] You: a couple seemed odd in LSL2 [9:08] Scouse Linden: specifically if () is false, if ([1,2] is true [9:08] Phantom Ninetails: That does indeed seem weird [9:08] You: it's one of the cases in which we wonder whether being compatible is the correct approach [9:09] Phantom Ninetails: Heh [9:09] You: we probably should be, but it seems very odd [9:10] Phantom Ninetails: Yeah.. That certainly looks like a difficult choice [9:10] You: the other case is where LSL2 allows you to have some control flows return values, but not others [9:10] Zero Linden is Online [9:10] You: that can be caught by the compiler [9:10] You: and you can always fix it [9:10] You: so it's less bad [9:10] Phantom Ninetails: SVC-1421? [9:11] You: you can't catch the if(list) at compile time easily [9:11] Scouse Linden: https://wiki.secondlife.com/wiki/Event_test_script [9:11] You: hi zero [9:11] Phantom Ninetails: Greetings Zero [9:11] You: welcome [9:11] Zero Linden: oh - we're in text [9:11] You: yes [9:11] You: maybe something about scripting? ;-) [9:11] Periapse Linden is Online [9:11] Zero Linden: scripting - ah yes - I remember scripting while out on the Seregheti! [9:12] Phantom Ninetails: Welcome back Periapse :) [9:12] You: we're communicating in text as it's the medium of code [9:12] You: hi peri [9:12] Periapse Linden: lol -- hello again phantom [9:12] Phantom Ninetails: :) [9:12] You: so, the issue we'd like to discuss with zero [9:12] You: (and we'd welcome input from you too phantom) [9:12] Phantom Ninetails: ^_^ [9:12] You: is a problem with llHTTPRequest specifically [9:13] You: and UTF8 handling in general [9:13] Periapse Linden: zero -- babbage turned off scripts, so the chair will be wonky [9:13] You: Mono uses UTF16 to represent strings internally [9:13] Zero Linden: yes it will! [9:13] You: so, we need to convert all UTF8 strings to UTF16 at some point [9:13] You: if the UTF8 is invalid, that conversion fails [9:14] Zero Linden: well - what do we do with LL strings now? [9:14] You: and by default Mono returns NULL strings [9:14] Zero Linden: Ah - [9:14] You: at the moment LSL will accept borked UTF8 [9:14] You: and just pass it around [9:14] Zero Linden: hmmm.... so the first question to ask is if mono handles the Astral Planes correctly [9:14] Zero Linden: loves Unicode terminology [9:14] You: and render what it can when you try to output it [9:14] Scouse Linden: Some web servers give us back latin 1 despite asking for utf-8 [9:15] You: yes, the bug we found [9:15] You: was a server reporting UTF-8 [9:15] Zero Linden: Scouse - no! really, can you point me to such a site? [9:15] Scouse Linden: Mono refuses to parse latin 1 as utf-8 [9:15] You: but returning latin-1 [9:15] Zero Linden: well, Latin-1 isn't UTF-8 and [9:15] Zero Linden: and can't be parse as such at all [9:15] Zero Linden: it is invalid [9:15] Scouse Linden: exactly [9:15] You: so, the llHTTPRequest gets a null conversion [9:15] You: doesn't touch the latin 1 [9:15] Scouse Linden: (under mono) [9:15] Zero Linden: so, we check, in llHTTPRequest, what the returned charset is and check that [9:16] You: then Mono can't convert it to UTF16 and so fails [9:16] Zero Linden: Wait, - is there a case where the site is giving us UTF-8 , claiming it is UTF-8, but isn't actually UTF-8? [9:16] Scouse Linden: yes [9:16] You: it claims it's giving us UTF8 [9:16] You: but then gives us latin 1 [9:17] Zero Linden: well then - we *shoud* treat that string as empty [9:17] You: but, with LSL2, we currently don't [9:17] Zero Linden: there is nothing you can do [9:17] Zero Linden: really? [9:18] Scouse Linden: Browsers typically replace the text with � [9:18] You: so, we can fix llHTTPRequest [9:18] You: to reject bad UTF8 after conversion [9:18] You: but, there are other ways to give borked UTF8 to LSL [9:18] Zero Linden: I'm checking the source - does it not validate if the result claims to be UTF8? [9:18] You: and we currently just pass it around and then throw it at the screen [9:19] Babbage Linden can't remember [9:19] Zero Linden: which won't render correctly [9:19] You: it was a while ago [9:19] Scouse Linden: http://www.salahzar.info/lsl/feedburner.php?feed=http://www.beppegrillo.it/index.xml the URL, if you change the character set it displays the characters correctly [9:19] Zero Linden: there is no , fallback to Latin-1 [9:19] Scouse Linden: I personally think we should try to do like the browsers and mark invalid characters [9:19] You: so, should we try to work around broken UTF8 [9:20] You: try to fix it so it turns in to some reasonable UTF16 when passed on Mono [9:20] Zero Linden: okay, so first off [9:20] You: or, just sulk and replace strings that return NULL when passed to Mono [9:20] You: with some "Invalid UTF8" string? [9:20] You: (which would be a change in behaviour to LSL2) [9:21] Zero Linden: well, first off, in that case we should not support that feed [9:21] Zero Linden: it is broken acording to at least three RFCs! [9:21] You: ok, so that is easy for the llHTTPRequest case [9:21] You: try to validate the UTF8 [9:22] You: convert the status to 40X [9:22] Zero Linden: Now, when Doug did the UTF8 conversion to LSL - he intended that non-UTF-8 strings would be considered ILLEGAL [9:22] You: that is not currently the case [9:22] Zero Linden: and I don't know if other functions will do bad things [9:22] You: the other one we found that could generate borked UTF8 [9:22] You: was llBase64ToString [9:23] Zero Linden: *THAT* function is EVIL [9:23] Zero Linden: :-) [9:23] Zero Linden: So - let's look at the idea of "fixing" it [9:23] You: that has no out of band error [9:23] You: no status we can set [9:23] Zero Linden: so - you have an LSL string with arbitrary binary data in it, say from llBase64ToString [9:23] You: our only option is to return some "special string" [9:23] Zero Linden: how would we fix it? [9:23] You: or cause a fault in the script [9:23] You: (LSL error semantics are horrible) [9:24] Zero Linden: so that it can become a valid Unicode string, that Mono can deal with as UTF-16? [9:24] Joshua Linden is Offline [9:24] You: so, we ask Mono for the UTF16 string representation [9:24] You: check the return [9:24] You: if it's NULL [9:24] Zero Linden: You can't really, since you don't know the encoding..... [9:24] Zero Linden: BUT [9:24] You: use some error string [9:24] Andrew Linden is Online [9:25] Zero Linden: you could pick some arbitrary 8-bit character set which has all 256 characters legal, and which 'round trips into and out of Unicode [9:25] You: (if it was C# we could throw an exception) [9:25] Zero Linden: and just DECLARE it to be in taht character set [9:25] Andrew Linden is Offline [9:26] Zero Linden: That will work, but will be unreliable.... [9:27] Zero Linden: In otherwords - if the llBase64Decode [9:27] Zero Linden: resutls in a sequence that happens to be UTF-8 valid, then you get the sequence of bytes you decoded [9:27] Zero Linden: if it doesn't, then you get re-interpretation in another charset, which result sin a UTF-8/UTF-16 string that represents [9:27] Zero Linden: a different sequece of bytes [9:28] Zero Linden: so - you could do that, but it would break llBase64Decode anyway [9:28] Zero Linden: So, I don't see a "fall back" position that works for llBase64Decode [9:28] You: no [9:28] Scouse Linden: Its pretty evil [9:28] You: so, in general [9:29] You: the goal was to dissallow invalid UTF8 in LSL anyway [9:29] Zero Linden: Well - we should state (even if we don't) that the Base64 functions ONLY work on Unicode text [9:29] Zero Linden: not arbitary binary data [9:29] You: so if we start converting invalid strings to error strings [9:29] Zero Linden: now [9:29] You: that may actually be better than what we have [9:29] Zero Linden: you could create a class of objects, "binary string" [9:30] You: (although it's different behaviour) [9:30] Zero Linden: well - you could get away with it perhaps, with these semantics [9:30] Zero Linden: all the string functions [9:30] Zero Linden: on binary strings produce binary strings [9:30] You: " Converts a Base 64 string to a conventional string. If the conversion creates any unprintable characters, they are converted to question marks" [9:30] Zero Linden: but as soon as the binary string is mixed with a normal string [9:30] You: from the LSL wiki [9:31] Zero Linden: the bianry string is interpreted as UTF-8 and if invalid is "" [9:31] Zero Linden: well, Babbage, "unprintable" in what charset? what charset are you considering the byte stream of the conversion to be in [9:31] Zero Linden: UTF-8? [9:31] Zero Linden: so say [9:31] You: so, there is a (defacto) specification atm it seems [9:32] You: even if the ? chars are currently coming from trying to display the invalid cahrs [9:32] You: instead of due to the conversion [9:32] Zero Linden: "Converts a Base 64 encoded string into it's decoded bytes, which are then interpreted as a UTF-8 string. Invalid UTF-8 sequences in this byte stream will be replaced with question marks" [9:32] Simil Miles: Hi all [9:32] Phantom Ninetails: Greetings [9:32] Zero Linden: actually, I think Unicode has a "this was a conversion error" character [9:32] You: (most browsers do that) [9:33] You: replace borked chars with "question mark in a diamond" [9:33] Zero Linden: thats the Unicode thingy [9:33] You: emulating that was scouse's favoured approach [9:34] You: given that we don't have good ways of reporting errors, i think it may be the best way forward [9:34] Zero Linden: I think that is reasonable [9:34] You: returning "" or setting an error and stopping the script seem horrible in terms of compatibility [9:35] Zero Linden: and the right thing to do in that HTTP case - the server says explicitly that the result is UTF-8 (I checked with Curl) [9:35] Zero Linden: and so we have no choice but to believe it --- [9:35] Zero Linden: so - if it fails conversion - replace them characters [9:35] Zero Linden: rather, them bytes [9:35] You: great [9:35] You: so now we just need a gadget to do the replacement [9:35] You: we'll have a look in glib [9:35] Zero Linden: what is the mono method that interprets a vector of bytes into a string? [9:36] You: it's just mono_string_new [9:36] You: which calls the glib UTF8 to UTF16 function under the hood [9:37] Zero Linden: ahand that function doesn't have a "what to do with bad encodings" option? [9:37] You: no, it just reports an error if the stream is invalid UTF8 [9:38] You: (so we need to massage the vector of bytes before hand) [9:38] Zero Linden: U+FFFD [9:38] Zero Linden: That's easy - I can write that with you if you need [9:38] Scouse Linden: � [9:38] Zero Linden: I have UTf-8 encoding in my blood [9:38] You: ok, cool [9:38] Zero Linden: � [9:38] You: we'll have a sniff around [9:39] Zero Linden: bizarre that on the Mac that doesn't print correctly [9:39] Scouse Linden: The are glib functions apparently which do the stuff we need [9:39] Scouse Linden: according to the folks in the mono channel [9:39] You: and if we can't find anything suitable we'll take you up on the offer [9:39] You: can we use the httprequest conversion dodad/ [9:39] Zero Linden: that uses APR, which is a wrapper on iconv [9:39] You: "convert potentially borken utf8 in to mostly fixed utf8" [9:39] You: that's the thingy [9:40] You: i'll have a look in to iconv too [9:40] Zero Linden: hmmm... I notice that we as to do the transcode no amtter what the charset [9:40] Zero Linden: so we ask to transcode from utf-8 to utf-8! [9:40] Zero Linden: which is good! [9:41] You: does that behaviour sound reasonable to everyone? [9:41] You: (you should only notice if you do something other than displaying the strings) [9:41] Scouse Linden: yes [9:41] You: (like emailing them or HTTP requesting them) [9:42] You: silence == consent [9:42] Zero Linden: well - it will cause llBase64Encode(llBase64Decode(s)) to not be the identity function [9:43] Zero Linden: but I think this is reasonable [9:43] Zero Linden: the other option, the BinaryString class option [9:43] Zero Linden: would preserve this [9:43] Scouse Linden: We might have to [9:43] Phantom Ninetails: lol, well I've never used any of the functions being discussed here so.. But yeah seems like a good idea [9:45] Babbage Linden doesn't like the idea of defining a new, non standard string class to support that corner case [9:45] Zero Linden: I think it might be okay to break llBase64Decode in that way [9:45] Zero Linden: if you gneerate a non-UTF-8 string that way now, almost nothing else in the LSL library will work right [9:45] You: you probably didn't mean to do it if you do now [9:46] Babbage Linden is tempted to go ahead with this plan and deal with the screaming later [9:47] Phantom Ninetails: lol [9:47] You: ok, so that's enough of that i think [9:47] You: thanks for coming along zero [9:47] Scouse Linden: yep [9:47] You: has anyone got anything else? [9:47] Zero Linden: welcome [9:47] Siann Beck: Any word on the db update? [9:48] You: peri? [9:48] You: do you know about that? [9:48] Joshua Linden is Online [9:49] Scouse Linden: So that means we don't know [9:49] Joshua Linden is Offline [9:49] You: joel was leading the charge on the agni db update [9:49] Siann Beck: Last Friday Peri said he was going to talk to the Havok4 team about it; I just wondered if there was any news. [9:49] You: i'll ask him about that [9:49] Siann Beck: OK [9:50] Periapse Linden: Sorry, I was in IM. [9:50] Scouse Linden: did you get the question? [9:50] Periapse Linden: No, I haven't talked to Sidewinder yet. I'll ping him now. But I dont' think the update is likely, as havok4 is trying hard to get off of aditi and onto the maingrid [9:51] Siann Beck: OK [9:51] Siann Beck: And when can we expect a Mono refresh? [9:51] You: tomorrow hopefully [9:51] Siann Beck: COol [9:51] Scouse Linden: Tuesday at the latest [9:51] You: (us brits are on holiday on friday and monday, so we want to give you some new code before we push off) [9:51] Scouse Linden: (if for some reason tomorrows fails) [9:52] Siann Beck: Ah, I see. What holiday is it? [9:52] Scouse Linden: easter [9:52] You: easter friday and monday [9:52] Siann Beck: Oh, duh! I knew that :) [9:52] Phantom Ninetails: I've got easter coming too :P [9:52] You: so, peri may be running an office hours on friday [9:52] Periapse Linden: I will run the office hour on Friday as normal [9:52] You: from the land of state/religion separation [9:52] Zero Linden: oy - so apr-iconv is a think wrapper around only 3/4 of iconv. [9:53] Zero Linden: and iconv the lib has "translisterate" [9:53] Zero Linden: as an option [9:53] Zero Linden: which we can get by converting to "UTF-8//TRANSLIT" [9:53] Zero Linden: which we can sneak past APR's call I bet [9:53] Zero Linden: BUT [9:53] Zero Linden: the command line iconv has exactly the feature we want [9:54] Zero Linden: the question is - how does it do it... [9:54] Babbage Linden wonders whether we can call iconv directly [9:55] Babbage Linden wonders whether we do elsewhere already [9:56] You: ok, are we all done? [9:57] You: anything else? [9:57] Zero Linden: may not need to [9:57] Siann Beck: Well, I don't know what I missed, so I'll just wait for the transcript. [9:57] Phantom Ninetails: Eerie silence. I guess it's just about over [9:57] Zero Linden: the one non-wrapped call isn't needed, now I see [9:57] Periapse Linden: Siann, I just talked to sidewinder [9:58] Zero Linden is Offline [9:58] Periapse Linden: He canceled the refresh because the beta grid would have to be down for 48 hours [9:58] Siann Beck: I see. [9:58] Periapse Linden: So I think we'll do it when havok4 leaves the grid [9:58] Phantom Ninetails: I find it odd that I became able to come on [9:58] Siann Beck: OK. Well, the object I need to test doesn't really have anything extraordinary, so it's probably fine. [9:59] Periapse Linden: But we can refresh specific people, so as to allow people to log on to aditi [9:59] Phantom Ninetails: Ah [9:59] You: i'll do the chat log peri [9:59] Phantom Ninetails: That'd probably explain it then :) [9:59] Siann Beck: I wondered about that, since I've seen people here who rezzed after the last update. [9:59] Periapse Linden: Thanks, Babbage.