BTLT

BTLT isn't anything special, it's just a combination of a couple very complicated libraries, FUIST-STIUF and TLT (FUIST-STIUF is an extension of FUIS-SIUF which is an extension of FUI-IUF). It encodes lists into strings and back into lists with care to restore the types but not degrade the contents while using a basic compression scheme.

History

Please excuse any slight errors in the timeline. In the beginning there was a scripter who wanted to be able to store floats in a lossless string format. The first attempt at solving this problem was Float to Scientific Notation and was terribly slow. Soon after that the scripter embarked upon a project to write an LSL bytecode interpretor written in LSL. Around this time the scripter discovered by accident that LSL supported HexFloats, and set about writing the encoder Float2Hex in the style of Float2Sci. To store the floats for the interpretor the FUI-IUF functions were written. During the writing of those functions the scripter realized two things, 1) that the process for encoding floats for FUI could be applied to Float2Hex and 2) that writing an interpretor in LSL wasn't worth the work. The process of matching the techniques of each function pushed the thinking forwards to the benefit of both Float2Hex and FUI-IUF. For grins the scripter modified FUI-IUF into FUIS-SIUF and found that it encoded and decoded floats faster then Hex2Float. Time passes and the scripter returns to a mothballed project: Hierarchical linking. This project had a couple of problems, how to store rotations in a tight format in notecards and ugly quaternion math. The problem with notecards is that if a script is going to read them, the lines can be no longer then 255 characters. This is problematic. One solution was FUIST-STIUF, it added trimming to FUIS-SIUF and was designed to be dropped into TLT. Before Float2Sci was even in the works the scripter was looking for a list-string transport functions. Not finding any satisfactory solutions the scripter laid out the TightList standard. It was integral part of the Strife Combat System and the functions were given the names "parse" and "dump". It wasn't until the development of TLTP and TLML did TightList become the official name of the functions. During the development of TLML did there become a need for a version of TL that could restore the type informations. After much thought TLT was created. Because of the memory restrictions of TLML(TLML's extensive feature set and tight syntax resulted in huge amounts of bytecode), TLT had to support drop-in encoders at compile time. Consequently TLT was written with ESL in mind (everything the scripter was writing at that time was written and still is written in ESL). The extension of TLT into BTLT while against the TLT standard is reasonable and to be expected.

Why Use BTLT?

BTLT was designed to be a tight format to be used for mostly float based datatypes. It wasn't intended to be particularly fast but it won't be unreasonably slow.

How It Works

Encoder:

• A TLT encoder outfitted with the FUIST family of functions gets first crack at the values.
  • FUIST family of functions encode floats, vectors and rotations to strings.
    1. The float is converted to an integer with the FUI technique.
       • If you took the memory representation of a float and read it as an integer you would have the same output as FUI.
    2. Then the integer is turned into a Base64 string.
    3. Then the any trailing A's and ='s are trimmed off the end of that string.
    4. For Vectors and Rotations a size field is attached to the front. Vectors use a 1 char field, rotations 2 chars.

Decoder:

• A TLT decoder outfitted with the STIUF family of functions.
  • STIUF family of functions do the inverse of FUIST.

Notes:

• The FUIST-STIUF functions used are revision 2 of the library. R1 was intended as standalone functions from TLT, R2 was optimized to take advantage of TLT, allowing for the stripping of a character from the vector size field.

Code

<lsl>//BTLT is part of the Combined Library. //BTLT Version 0.1 beta

//===================================================// // Combined Library // // "Nov 5 2007", "04:00:34" // // Copyright (C) 2004-2007, Strife Onizuka (cc-by) // // http://creativecommons.org/licenses/by/3.0/ // //===================================================// //{

rotation stiufr(string in) //Mono Unsafe, LSO Safe, Double Unsafe { //--XXXXYYYYZZZZSSSS

   integer     raw = llBase64ToInteger(llGetSubString("AAA" + in + "AAA", 0, 5)) >> 2;
   integer     offset = 1;
   integer     len = raw & 7;
   return <stiuf(llDeleteSubString(in, len + 2, 1)),
       stiuf(llDeleteSubString(in, (len = (7 & (raw >> 3))) - ~(offset += len), offset)),
       stiuf(llDeleteSubString(in, (len = (7 & (raw >> 6))) - ~(offset += len), offset)),
       stiuf(llDeleteSubString(in, 0, offset + len)) >;

}

vector stiufv(string in) //Mono Unsafe, LSO Safe, Double Unsafe { //-XXXXYYYYZZZZ

   integer     raw = llBase64ToInteger(llGetSubString("AAAA" + in + "AA", 0, 5)) >> 2;
   integer     offset = 0;
   integer     len = raw & 7;
   return <stiuf(llDeleteSubString(in, -~len, 0)),
       stiuf(llDeleteSubString(in, (len = (7 & (raw >> 3))) - ~(offset += len), offset)),
       stiuf(llDeleteSubString(in, 0, offset + len)) >;

}

string vfuist(vector in) //Mono Unsafe, LSO Safe, Double Unsafe {

   string      buf = fuist(in.y);
   integer     len = llStringLength(buf);
   return llGetSubString(llIntegerToBase64(((len << 3) + llStringLength(buf)) << 2), 4, 4) + (buf =
                                                                                              fuist(in.x)) + buf + fuist(in.z);

} //-XXXXYYYYZZZZ

string rfuist(rotation in) //Mono Unsafe, LSO Safe, Double Unsafe {

   string      z = fuist(in.z);
   string      buf = fuist(in.y);
   integer     len = (llStringLength(z) << 3) | llStringLength(buf);
   return llGetSubString(llIntegerToBase64(((len << 3) | llStringLength(buf)) << 2), 3, 4) + (buf =
                                                                                              fuist(in.x)) + buf + z +
       fuist(in.s);

} //--XXXXYYYYZZZZSSSS

string ist(integer b) //Mono Safe, LSO Safe, Double Unsafe {

   string      src = llIntegerToBase64(b);
   integer     c = 6;
   do;
   while ("A" == llGetSubString(src, c = ~-c, c));
   return llDeleteSubString(src, -~c, 0x8000);

}

string fuist(float input) //Mono Unsafe, LSO Safe, Double Unsafe { //float union to base64ed integer

   if (input) {	//is it non zero?
       integer     sign = (input < 0) << 31;	//the sign, but later this variable is reused to store the shift
       integer     exp;
       if ((input = llFabs(input)) < 2.3509887016445750159374730744445e-38)	//Denormalized range check & last stirde of normalized range
           sign = sign | (integer) (input / 1.4012984643248170709237295832899e-45);	//the math overlaps; saves cpu time.
       else
           sign = (0x7FFFFF & (integer) (input * (0x1000000 >> sign))) | (((exp + 126 + (sign = ((integer) input - (3 <= (input /= (float) ("0x1p" + (string) (exp -= ((exp = llFloor(llLog(input) / 0.69314718055994530941723212145818)) == 128)))))))) << 23) | sign);	//extremes will error towards extremes. this yuch corrects it.
       return ist(sign);
   }	//for grins, detect the sign on zero. it's not pretty but it works. the previous requires alot of unwinding to understand it.
   if ((string) input == (string) (0.0))
       return "";
   return "g";

}

float stiuf(string input) { //base64ed integer union to float

   integer     buf = llBase64ToInteger(llGetSubString(input + "AAAAAA", 0, 5));
   return ((float) ("0x1p" + (string) ((buf | !buf) - 150))) * ((!!(buf = (0xff & (buf >> 23))) << 23) | ((buf & 0x7fffff))) *
       (1 | (buf >> 31));

} //will crash if the raw exponent == 0xff; reason for crash deviates from float standard; though a crash is warented.

list BTLTParse(string input) {

   list        partial;
   if (llStringLength(input) > 6) {
       string      seperators = llGetSubString(input, (0), 6);
       integer     pos =
           ([] !=
            (partial =
             llList2List(input +
                         llParseStringKeepNulls(llDeleteSubString(input, (0), 5),[],
                                                [input =
                                                 llGetSubString(seperators, (0), (0)), llGetSubString(seperators, 1, 1),
                                                 llGetSubString(seperators, 2, 2), llGetSubString(seperators, 3, 3),
                                                 llGetSubString(seperators, 4, 4), llGetSubString(seperators, 5, 5)]),
                         (llSubStringIndex(seperators, llGetSubString(seperators, 6, 6)) < 6) << 1, -1)));
       integer     type = (0);
       integer     sub_pos = (0);
       do {
           list        current = (list) (input = llList2String(partial, sub_pos = -~pos));	//TYPE_STRING || TYPE_INVALID (though we don't care about invalid)
           if (!(type = llSubStringIndex(seperators, llList2String(partial, pos))))	//TYPE_INTEGER
               current = (list) (llBase64ToInteger(llGetSubString(input + "AAAAAA", 0, 5)));
           else if (type == 1)	//TYPE_FLOAT
               current = (list) (stiuf(input));
           else if (type == 3)	//TYPE_KEY
               current = (list) ((key) (input));
           else if (type == 4)	//TYPE_VECTOR
               current = (list) (stiufv(input));
           else if (type == 5)	//TYPE_ROTATION
               current = (list) (stiufr(input));
           partial = llListReplaceList(partial, current, pos, sub_pos);
       } while ((pos = -~sub_pos) & 0x80000000);
   }
   return partial;

}

string BTLTDump(list input, string seperators) { //This function is dangerous

   seperators += "|/?!@#$%^&*()_=:;~`'<>{}[],.\n\" qQxXzZ\\"; //"//Buggy highlighter fix
   string      cumulator = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/" + (string) (input);
   integer     counter = (0);
   do
       if (~llSubStringIndex(cumulator, llGetSubString(seperators, counter, counter)))
           seperators = llDeleteSubString(seperators, counter, counter);
       else
           counter = -~counter;
   while (counter < 6);
   seperators = llGetSubString(seperators, (0), 5);

   string      buffer = cumulator = "";

   if ((counter = (input !=[]))) {
       do {
           integer     type = ~-llGetListEntryType(input, counter = ~-counter);

           if (type == 4)	//TYPE_VECTOR - 1
               buffer = vfuist(llList2Vector(input, counter));
           else
            if (type == 5)	//TYPE_ROTATION - 1
               buffer = rfuist(llList2Rot(input, counter));
           else
            if (type == 1)	//TYPE_FLOAT - 1
               buffer = fuist(llList2Float(input, counter));
           else
            if (!type)	//TYPE_INTEGER - 1
           {
               buffer = ist(llList2Integer(input, counter));
           } else

               buffer = llList2String(input, counter);

           cumulator = (cumulator = llGetSubString(seperators, type, type)) + buffer + cumulator;
       } while (counter);
   }
   return seperators + cumulator;

}

//} Combined Library </lsl>