User talk:Zero Linden/Office Hours/2007 Mar 06

i found the original transcript a tad hard to read & slapped together this little perl script to reformat into a wiki table:

#!/usr/bin/perl
# Placed into the public domain by IBM Research.

use strict;
use English;

sub eject_header ( );
sub eject_line ( $$$$ );
sub eject_footer ( );

eject_header;
while (<>) {
    if (m/(                         # $1: captures whole lead
           (\[\d{1,2}:\d{2}\]\s+)?  # $2: captures optional timestamp [13:00]
           (\S+\s+[^\s:]+)          # $3: captures Firstname Lastname
           ([\s:])                  # $4: captures (:| )
          )
          \s*                       # gobble up white space
          (.*)                      # $5 captures actual text
         /x) {
	my $timestamp = $2 || " ";
	my $who = $3;
	my $sep = $4;
	my $text = $5;
	eject_line $timestamp, $who, $sep, $text;
	next;
    }
    
    print "??? ", $_;
}
eject_footer;

exit 0;

sub eject_header() {
    printf STDOUT "{|\n";
}

my $last_who = undef;
my $cidx = 0;

sub eject_line($$$$) {
    my $timestamp = shift;
    my $who = shift;
    my $sep = shift;
    my $text = shift;

    my $bg = "";

    if (defined $last_who and $last_who ne $who) {
	$cidx = ($cidx + 1) % 2;
    }
    $last_who = $who;

    if (0 == $cidx) {
	$bg="background-color:#FFFFFF;";
    } else {
	$bg="background-color:#F0F0F0;";
    }

    $text =~ s/%/%%/g;
    $who =~ s/\s+/ /g;

    printf STDOUT "|- style=\"vertical-align:top;$bg\"\n";
    printf STDOUT "| $timestamp\n";
    if ($sep =~ /:/) {
	printf STDOUT "| $who:\n";
	printf STDOUT "| style=\"white-space:normal;\"|" . $text . "\n";
    } else {
	printf STDOUT "| colspan=\"2\"|''$who " . $text . "''\n";
    }
}

sub eject_footer() {
    printf STDOUT "|}";
}

with this as the result:

[13:01]	Maxx Duell accepted your inventory offer.
[13:01]	Zha Ewry:	Hello Zero!
[13:01]	Dale Innis:	Hi hi!
[13:01]	Jason Ayakashi:	hi, zero
[13:01]	Zero Linden:	hello hello
[13:02]	Hiro Market:	hi zero
[13:02]	Dale Innis apologized for suddenly running off in a few minutes. :)
[13:02]	Dale Innis:	(apologizes)
[13:02]	Zero Linden:	Welcome all
[13:02]	Zha Ewry:	Hey Rex, welcome back
[13:02]	Rex Cronon:	hello everybody
[13:03]	Zero Linden:	Everyone had a good weekend?
[13:03]	Dale Innis:	Even slept some. :)
[13:03]	Zero Linden:	I spent far too much of mine thinking about cross-protocol endpoints.
[13:03]	Dale Innis:	lol
[13:03]	Zha Ewry:	eewe
[13:04]	Zha Ewry dislkies m*n problems
[13:04]	Zha Ewry:	Hello Vitis
[13:04]	Rex Cronon:	the things just started here right?
[13:04]	Vitis Obviate:	Heya
[13:04]	Rex Cronon:	hi
[13:04]	Zha Ewry:	Zero has even done his "ground rules" yet
[13:04]	Zero Linden:	Yes - and who would have thought that a simple concept like "the sender of this message" could have such a property?!?!?
[13:04]	Zha Ewry would :-)
[13:04]	Zero Linden:	Yes, things are just started, Rez
[13:05]	Zero Linden:	wow - big turn out today!
[13:05]	flying ball whispers: I am ALIVE!
[13:05]	Sifu Moraga:	Gotta load my cache....
[13:05]	flying ball:	, you can't sit on me![cd]
[13:05]	flying ball whispers: I am ALIVE!
[13:06]	Dale Innis:	:) yeah some of us will be running of momentarily.
[13:06]	Dale Innis:	(off)
[13:06]	Sifu Moraga:	I'll be falling asleep in a moment
[13:06]	Zero Linden:	Well - as most of you know, but protocol dictate I say....
[13:06]	Zero Linden:	This is Zero's Techno-Geek-Fest office hours
[13:06]	Zero Linden:	And it the transcripts will be posted to the SL wiki
[13:07]	Sifu Moraga:	Oops, a lot of IBMers here
[13:07]	Zero Linden:	Unlike last session - I have nothing in particular on the agenda to talk about..... though I'll give a brief update
[13:08]	Zha Ewry:	Good, then I can pester you about in / out blobs
[13:08]	C4 (say /5 boom): Ka-Boom
[13:08]	Zero Linden:	Last Thursday we were walking about a sim where almost all the sim traffice to the viewer went over LLSD, via the event queue, via a capability
[13:09]	Zero Linden:	That was a good milestone. We hope this week to have a grid up where the only traffic that is UDP are some messages like ObjectUpdate and the rest
[13:09]	Zero Linden:	of the servers are all talking LLSD over some TCP based transport, as well as the viewer to sim communications
[13:09]	josserand Jacobus:	Hi very nice place, Zha ! May I sit down ?
[13:09]	Zero Linden:	That's all from me
[13:09]	Zero Linden:	So, I'm happy to talk blobs
[13:10]	Zero Linden:	For those that don't know, there is a Zero Office Hour Discussion page in the wiki
[13:10]	Hiro Market:	can i just request the wiki for the last session is updated with the diagrams
[13:10]	Zero Linden:	https://wiki.secondlife.com/wiki/Discussion_Page_for_Zero_Linden%27s_Office_Hours
[13:10]	Hiro Market:	missing atm
[13:10]	Zha Ewry:	Good. How much pain would it cause to have bigger than 2K objects come in as blobs?
[13:10]	Vitis Obviate:	ahh..images are up...ty
[13:10]	Zero Linden:	Hiro - what? https://wiki.secondlife.com/wiki/ZeroTranscript2007Mar01 has images
[13:10]	Sifu Moraga:	(and then you'd have to decide how to store them)
[13:10]	Zha Ewry:	I'm especially keen on the idea, as that gets us out of the LSL data space problems
[13:11]	Zha Ewry:	As to where Sifu, in inventory, inside the script's prim, is one easy thought
[13:11]	Dale Innis:	Yeah; get everyone to store the big stuff on their own external srevers! I like that. :)
[13:11]	josserand Jacobus:	No answer so I sit down
[13:11]	Zha Ewry:	Because, you don't want them to be live, without some effort on the part of the scripter
[13:11]	Sifu Moraga:	I currently use notecards for blobs
[13:11]	Hiro Market:	no, nothing for me, tried opera and ie
[13:12]	Zero Linden:	So, for others, if you haven't read it, you might want to bop on over to the wiki discussion page and read Zha's proposal
[13:12]	Zero Linden:	Sifu - right, but notecards arent r/w and aren't likely to be.... ever.
[13:12]	Sifu Moraga:	yes, notecards are a real pain
[13:12]	josserand Jacobus:	Zero, what mean LindenVillager please
[13:12]	Sifu Moraga:	but they were all we could get to work
[13:12]	Dale Innis:	Hence the desire for more general blobs. :)
[13:12]	Zha Ewry:	Hmm. Before we dive to far, why aren't notecards writable?
[13:12]	Rex Cronon:	a link to zha proposal is on you page zero?
[13:12]	Sifu Moraga:	exactly
[13:12]	Zero Linden:	Hiro - does this work for you: https://wiki.secondlife.com/wiki/Image:ZeroOfficeHours-20070301-slide1.png ??
[13:13]	Zha Ewry:	https://wiki.secondlife.com/wiki/Discussion_Page_for_Zero_Linden%27s_Office_Hours
[13:13]	Zero Linden:	https://wiki.secondlife.com/wiki/Discussion_Page_for_Zero_Linden%27s_Office_Hours#Binary_Blob_in.2Fout_from_SL
[13:13]	Zha Ewry:	If you use mine, scroll down, Zero's is more precise
[13:13]	Zero Linden:	So
[13:13]	Zero Linden:	there are two aspects about blobs I'd like to discuss
[13:14]	Zero Linden:	First - mechanics of what blobs would look like:
[13:15]	Zero Linden:	Right now, the inventory of an object is essentially a list of assets and access rights information
[13:15]	Zero Linden:	A binary blob, especially a r/w one could not be stored in the asset server becuase the asset server
[13:15]	Zero Linden:	(not a single machine, but logically so), was designed, and is cached, on the assumption that
[13:15]	Zero Linden:	assets are data that never change once written
[13:16]	Zha Ewry:	I'd assume you'd take it into a native SL type inbound
[13:16]	Sifu Moraga:	Hmmm, I see the problem
[13:16]	Zha Ewry:	So, the mime type of the blob would tell the inbound process how to turn it into an SL object, you'ddump it into the asset server
[13:16]	Zero Linden:	but nonetheless, we'd have to extend the concept of inventory to include objects that aren't assets.
[13:16]	Zha Ewry:	and the pass the uuid to the invenotry of the prim running the script
[13:16]	Sifu Moraga:	yes
[13:16]	Zero Linden:	Not impossible, but definitely different
[13:17]	Zero Linden:	And once one wants all the inventory like operations to work on them - you see that we'd be effectively trying to
[13:17]	Zha Ewry:	thus it becomes a fairly normal SL object, before it gets into the asset server
[13:17]	Zero Linden:	create a sub-class of an abstract class (inventory item) that doesn't currently exist
[13:17]	Sifu Moraga:	Since an object is local to a sim, could transitory blobs be stored by the sim server?
[13:17]	Zero Linden:	Sifu - it would have to be -- so
[13:18]	Zero Linden:	right now, when we move an inventory item (or copy it), there is no data movement at all, becuase the assent doesn't change or move
[13:18]	Sifu Moraga:	Ko, so the problem is that an object can reference anything that is not in the asset server
[13:18]	Zha Ewry:	right, you just move a uuid
[13:18]	Zero Linden:	for a blob, we'd have to hook all those and do copying and/or moving and/or reference counting.... etc...
[13:18]	Zero Linden:	none of this is impossible, just more difficult and involves a refactoring of a part of the system
[13:19]	Zha Ewry:	Well, you actually looked at amore general solution... II won't complain if you do that... mind you
[13:19]	Zha Ewry:	I'd be happy to have a mim driven way of getting blobbed assets into the asset server, and into the inventory of a local object
[13:19]	Zero Linden:	So now we have to store that data too... and as Sifu said, it would be on the sim... some of the time
[13:19]	Sifu Moraga thinks out loud, could an object store a reference to a transient object?
[13:19]	Zero Linden:	so - when you take an object into inventory (take or take copy), we are basically saving the state of the object (and scripts), and the inventory list, and writing that into a new asset
[13:20]	Zha Ewry:	I'd asume you'd be reluctant to do to much of that, as you now have state which is in a bad way, if the sim crashes
[13:20]	Zero Linden:	So in this case, we'd have to bundle all the data of the blob up into the inventory entry and store that too
[13:20]	Zha Ewry:	right now, assets either are, or don't exist. Not much inbetween
[13:20]	Zero Linden:	again, not difficult, but more work to do.... and more data to move
[13:20]	Vitis Obviate reflects that we should discuss the "why can't we write notecards" problem also - as Zha suggested - before we dive too deep
[13:20]	Zha Ewry:	In the asset server, with a uuid, or non-existent
[13:21]	Zero Linden:	Internally, if the object is rezz'd we'd have to have the data hanging off the inventory item... again not hard
[13:21]	Zero Linden:	and when the sim state saves, we have to write out all that data with objects in the same way as when sent to the asset server during Take
[13:21]	Zero Linden:	So these changes wouldn't be that hard.. easier than the inventory operation refactoring, actually
[13:22]	Zero Linden:	But - then there is the issue of the data and the cost of storing it and moving it around
[13:22]	Zha Ewry:	Vitis, I think you just got a big chunk of your answer to read/write notecard, btw
[13:22]	Zha Ewry:	Since the same issiues, roughtly obtain
[13:22]	Zha Ewry:	(lots of state, and the question of the UUIDs of the notecard in each state)
[13:22]	Zero Linden:	If... imagine, a region with 1,000 objects that used blobs. And, say, 2 blobs per object (pulling numbers out of thin air)
[13:23]	Rex Cronon:	but what if i am not interested to save the state of the scripts that in an linked set inventory, if i can use scripts to write to notecard, the script can save its onw state
[13:23]	Zero Linden:	and say 20k of data per blob
[13:23]	Zero Linden:	that's 20k x 1k x 2 = 40M
[13:23]	Hiro Market:	somewhere in the forum there's a reference to LL saying they'd open up a database for scripts to read/write data
[13:24]	Hiro Market:	seperatly to other mechanisms
[13:24]	Zero Linden:	and, since we store objects in XML -- you'd have to base 64 this when stored
[13:24]	Zha Ewry:	So, zero,suppose we assume that we use blobs, just to get objects (current set of SL native ones) into/outof the asset server
[13:24]	Zero Linden:	Since a typical region runs in 200M of RAM, and we try to keep it under 512M
[13:25]	Sifu Moraga thinks that the fact that my blobs are already XML doesn't help
[13:25]	Zha Ewry:	Which scopes things down a lot
[13:25]	Zero Linden:	you can see that this would start to put a strain on memory
[13:25]	Sifu Moraga:	But you could still have a local DB, though
[13:25]	Zha Ewry:	I'd love the more general solution, mind you, but I'm looking for ways for use to unload the storage issue (and the assembly issues) from SL and the scripters
[13:26]	Sifu Moraga:	not everything hast too be in memory, does it
[13:26]	Zero Linden:	Hiro - I don't know what forum post you're referring to -- but perhaps you are thinking about silo.php -- which is my script for storing arbitrary amounts of data on your own web server from LSL
[13:26]	Zero Linden:	?
[13:26]	Zha Ewry:	and let us (the developers who own the binary data) get it to the asset server as neatly as possible
[13:26]	Zha Ewry:	(and the converse, to get it out)
[13:27]	Zero Linden:	Zha- if your goal is to be able to add textures / sounds / notecards or even (gasp!) scripts to the asset system under LSL control.....
[13:27]	Hiro Market:	no, not silo, this is a reference to LL agreeing that not writing to notcards was an issue, and supplying a seperate database mechanism accessible by LSL
[13:27]	Zero Linden:	or similarly, to read that data out....
[13:27]	Hiro Market:	quite old, looking for it, caught my eye when I saw it
[13:27]	Zero Linden:	then perhaps we should talk about issuing capabilities to that information so you can just do it directly
[13:27]	Zha Ewry:	(and follow the DRM model, as we do it...)
[13:27]	Sifu Moraga:	Certainly that is not what I need blob transfer for
[13:28]	Zero Linden:	Oh, Hiro, don't know about there - there isn't a current plan internally that I know of
[13:28]	Zha Ewry:	Capabitlies, with the right strucfture, linked ot scripting, would get us most of the way.
[13:28]	Zha Ewry:	to
[13:28]	Sifu Moraga:	you think so, Zha?
[13:28]	Zero Linden:	Mind you - web server space that runs PHP is so cheep - seems like a much better solution to just have people store their own data in data stores they manage and pay for (and backup!)
[13:29]	Zha Ewry:	The question is mostly how to get the data from those servers into SL
[13:29]	Zero Linden:	Rather than the scaling issue of us managing a big writable store for everyeone...
[13:29]	Zero Linden:	Zha - this brings up my second question... which is - what do you want to do with the blobs
[13:29]	Sifu Moraga:	Unfortunatly, I'd rather have a standard-like solution like REST or SOAP (gad, ack)
[13:29]	Zha Ewry:	Now, to an extent, I'd be happy to bypass an asset server, but I'm stuck on how to do that such that it all works when the script moves across sim lines
[13:30]	Zha Ewry:	REST, not SOAP, please :-) and that's a personal not corporate preference :-)
[13:30]	Zero Linden:	and perhaps also notecards and sounds
[13:30]	Zero Linden:	and I'm sensing it is you want to import lots of textures into SL for autmated builds or catalogs or knowledge bases or some such
[13:30]	Zero Linden:	Ditto here, Zha --- personally, I don't like the taste of SOAP...
[13:30]	Sifu Moraga:	I know, I know, Zha, but I don't always decide....
[13:30]	Zha Ewry:	Well also to drive larger data driven scripts.
[13:31]	Zha Ewry:	We have lots of use cases where we don't (can't) do the compute in SL, nor do you want us to
[13:31]	Sifu Moraga:	So, what I;m doing with my notecards is to do some light processing on them and then sending them to an instrumented client
[13:31]	Zha Ewry:	But we need efficient ways of getting the data in, well beyond the 2048 bytes at a time solutions, and with less load on both the scripting workload
[13:31]	Zero Linden:	Right - but driving data driven scripts is easy - because since the scripts can't compute that much anyway, you just have the scripts request the data in chunks it can handle, which are going to be small
[13:32]	Zha Ewry:	and the http requests pounding the sims
[13:32]	Sifu Moraga:	And then, the instrumented client gives a notecard back and gives it to an object via the AV
[13:32]	Zero Linden:	Really? Surely you can process 2k worth of, say geometry instructions, in LSL in the 1/2 second it takes to fetch the next 2k
[13:33]	Zha Ewry:	If I want to texture, as I go, it gets worse, and I can drive a nice class five sim down to a time dialation of oh, .4 or ..3 with just geometry and scripts
[13:33]	Sifu Moraga:	I don't need to do much processing, except a bit of string replacement
[13:33]	Zha Ewry:	not even loading texutres. Just rezzing prims, and moving them into place in large numbers
[13:33]	Zha Ewry:	If I want to load textures, as I rez... its gets painful
[13:34]	Zero Linden:	Right - but I bet the load from the llHTTPRequest isn't really contributing to that problem
[13:34]	Sifu Moraga has seen what Zha is up to :-)
[13:34]	Zero Linden:	We tested doing several 100 llHTTPRequests per second in a sim with no noticable lag at all.
[13:35]	Zero Linden:	So, for driving scripts, I doubt getting the data in bigger chunks would be any better
[13:35]	Rex Cronon:	so zha, u want to build something off grid and upload those objects(or linked sets) into a sim?
[13:35]	Zero Linden:	Now, notecards and textures are two totally other problems
[13:35]	Hiro Market:	slightly tangental, but has support for procedural textures ever been considered?
[13:35]	Zha Ewry:	Hmm. Say, a 5500 prim molecular model? Yeah.
[13:35]	Sifu Moraga:	The problem I have is that I can't transfer the entire notecard in one piece, so I need to do about 4 transfers
[13:36]	Zha Ewry:	And then stich them together, and then run out of LSL memory
[13:36]	Zero Linden:	Righy - but doing the, I dunno, 100 2k data requests for the data for that molecule isn't nearly so much the load on the sim as rezzing and moving 'em
[13:36]	Sifu Moraga thinks the molecure is awsome
[13:36]	Zero Linden:	especially since I bet each of those has a script!
[13:36]	Zha Ewry:	One problem I see, is we end up with lots of scripts to deal with dataspace
[13:36]	Zha Ewry:	Not for long. We kill 'em as fast as we can.
[13:37]	Zha Ewry:	If I leave 5500 scripts listening, the sim is an unhappy place for 2 agents
[13:37]	Zero Linden:	none the less, the cost of starting a script far outweights the cost of doing one llHTTPRequest
[13:37]	Zero Linden:	Let's talk notecards -
[13:37]	Zero Linden:	Sifu - you seem to be wanting to do two things with them....
[13:37]	Zha Ewry:	Hmm. I'm not surprised. Scripts require lots of context
[13:37]	Zero Linden:	One is transmit coded data to a specially instrumented browser
[13:38]	Sifu Moraga nods
[13:38]	Vitis Obviate would use that also
[13:38]	Zero Linden:	You could, of course, use specially coded IM messages -- (what is that crypto product that works over AIM???)
[13:38]	Zha Ewry as well
[13:38]	Zero Linden:	but i imagine that is slow....
[13:39]	Zero Linden:	So - why not put the data via llHTTPRequest onto your server, and have the client retrieve it?
[13:39]	Sifu Moraga:	We thought of that, but still have to get it out to a web service at one point
[13:39]	Zero Linden:	granted, you have to take up the store of it
[13:39]	Zero Linden:	right - but the instrumented data is for the viewer, not the human, right?
[13:39]	Sifu Moraga:	that too
[13:39]	Zero Linden:	is it larger than 2k?
[13:39]	Sifu Moraga:	It also needs a bit or processing in-world
[13:39]	Sifu Moraga:	The data is for a cpu, its all XML, but unreadable
[13:40]	Sifu Moraga:	it is about 16kb in it's least cryptographicaaly secure form
[13:40]	Zha Ewry:	A fair bit of crypto/cert stuff ends up bigger than 2048...
[13:40]	Zero Linden:	We've had requests to have a version fo llHTTPRequest that is issued by the viewer, not the sim
[13:40]	Sifu Moraga:	easily
[13:40]	Zha Ewry:	Ah. There's a nice hard number
[13:41]	Sifu Moraga:	not the same problem, but still of interest for other stuff
[13:41]	Zero Linden:	so a script on the sim makes the request, the request is sent to the viewer, the viewer makes the call, and teh results piped back up to the
[13:41]	Zha Ewry:	Thus inside the client's firewall, and access to the local machine?
[13:41]	Zero Linden:	sim
[13:41]	Sifu Moraga:	yup
[13:41]	Zero Linden:	Right - the desire is to allow HUD objects that control things like MP3 players, and other, er, "add-ons"
[13:41]	Sifu Moraga:	more or less, though the firewall is incidental to what we are doing
[13:41]	Zha Ewry:	That helps although if its 2048, only a little, as you're still in the data stitching game
[13:42]	Zero Linden:	This strikes me as more of what you are after in this case, Sifu?
[13:42]	Sifu Moraga:	important is that we have two completely separate zone we need to access
[13:42]	Zero Linden:	Well, Zha, if the data has to go through LSL, while we can relax the size a bit (we intend to make it configuratble at some point)
[13:42]	Zero Linden:	it still won't be useful beyond, say 6k
[13:43]	Zha Ewry:	Right. Thus the desire to bypass LSL
[13:43]	Zero Linden:	since LSL execution space is going to be limited to 16k for quite some time
[13:43]	Sifu Moraga:	16 kb is the minimum
[13:43]	Zha Ewry:	I'm looking at how I can move bigger chunks without having them have to land in LSL at all
[13:43]	Zero Linden:	But - the only thing I think I hear that you'd entirely bypass LSL for is: Textures, and Notecards for users to read
[13:43]	Sifu Moraga:	It did occure that if we could just send the notecard directly from llHTTPRequest....
[13:43]	Zha Ewry:	Well, notecards for programs as well
[13:43]	Vitis Obviate:	yes
[13:43]	Sifu Moraga:	...that would bypass the LSL memory restrictions
[13:44]	Zero Linden:	Right - but where? If not a texture or a human readable notecard, then all other destinations in SL require LSL, yes?
[13:44]	Sifu Moraga:	I do think the use ofr notecards to be just wrong, somehow
[13:44]	Zha Ewry:	I'm happy to loop across a long notecard, one line at a time, never getting big chunks into the LSL data space
[13:44]	Vitis Obviate nods
[13:44]	Zero Linden:	Then why not read the data from your web server one line at a time? It is almost as fast, if not faster!
[13:44]	Sifu Moraga:	umm, you'l be happy until you have tried it like me
[13:45]	Zha Ewry:	I'm muchg less happoy trying to coordinate that across http gets with all the logic to deal with dropped /oiur of order stuff
[13:45]	Sifu Moraga:	I get some delay that the users don't like
[13:45]	Sifu Moraga:	yes, that is worse
[13:45]	Zero Linden:	Indeed, the notary reads notecards one line at a time (needs to for hashing) and it is a bear
[13:45]	Zha Ewry:	I also see about 50% of my code space go into the parse/manage packet order problem
[13:46]	Zero Linden:	Really? Why not just issue your HTTP requests serially, like you'd do for notecards?
[13:46]	Sifu Moraga:	Also, I can't get llHTTPrequest to add chunking headers
[13:46]	Zha Ewry:	I understand the 16K problem, so I'm looking for reaosnale ways to bypass it
[13:46]	Zha Ewry:	I can issue them serially, but I have to have chunking/retry logic and I do end up exercising ti
[13:46]	Zha Ewry:	it
[13:47]	Zero Linden:	By the way - I'm not trying to be resistant - just trying to really get at what the user needs are here -- alas, with our highly constrained system, and our opaque implementation, it is hard to know what the crux of a user problem is
[13:47]	Sifu Moraga totally understands
[13:47]	Zha Ewry:	Yes, I get that And having the discusssion is the right way to do it
[13:47]	Zha Ewry:	I'm totally happy witrh this process, at the moemnt
[13:48]	Zero Linden:	ah- I think I'm now seeing the bigger issue - here
[13:48]	Zha Ewry:	Point a number of people at the problem, and get to a good technical answer works for me
[13:48]	Zero Linden:	I'm wondering then if instead we supported byte-ranges,
[13:48]	Sifu Moraga:	If I was travelling to SF any time soon, I'd love to have a RL discussion of it
[13:48]	Sifu Moraga:	(unfrotunatly, I'm not)
[13:48]	Zero Linden:	and since our requests are squid cached
[13:48]	Zero Linden:	if that would work - because squid requests the whole thing when you do a byte range request
[13:49]	Zero Linden:	so once the frist range worked
[13:49]	Zero Linden:	you'd be getting the rest out of the cache
[13:49]	Zha Ewry:	Oh, allow us to load a bigger chunk into the squid, and then grab it 2048 at a time?
[13:49]	Sifu Moraga thinks
[13:49]	Zero Linden:	Exactly- if everyone played right by the headers and all....
[13:50]	Sifu Moraga doesn't get it
[13:50]	Zero Linden:	If you do a GET to say http://example.com/foo/bar
[13:50]	Zero Linden:	but include a byte range header asking for bytes 0-2047
[13:50]	Zha Ewry willing to be a very obedient header wreiter
[13:50]	Zero Linden:	Squid cache will instead ask for the whole thing
[13:50]	Zero Linden:	and cache the whole resource at that URL
[13:50]	Zero Linden:	and deliver only the requested bytes
[13:50]	Zero Linden:	when you then make a second request to the same URL
[13:51]	Zero Linden:	but with byte range 2048-4095
[13:51]	Zha Ewry:	with the byterange...
[13:51]	Zha Ewry:	Yes..
[13:51]	Zero Linden:	squid just answers out of the cache
[13:51]	Vitis Obviate will use whatever incontonation is required
[13:51]	Zha Ewry:	How big a chunk could the squid handle?
[13:51]	Zero Linden:	Now, mind you, it is all REST and stateless... so
[13:51]	Zha Ewry likes REST and stateless...
[13:51]	Zero Linden:	in theory, you could have squid drop the resource in the middle and have to refetch
[13:51]	Vitis Obviate:	fine with me
[13:51]	Sifu Moraga thinks REST is good
[13:51]	Zha Ewry:	Yes, just delay on random fetches if it does
[13:52]	Zero Linden:	Squid? Oh, I don't know what our maximum cached object size is set to....
[13:52]	Zha Ewry:	as long as the URL is idempotent, we're ok
[13:52]	Zha Ewry smiles
[13:52]	Zero Linden:	right now it is probably pretty large... 1M or so?
[13:52]	Zero Linden:	Exactly, Zha
[13:52]	Zha Ewry:	I can deal with that :-)
[13:52]	Zha Ewry:	Doesn't help with textures or sounds, but... it would help a lot.
[13:52]	Zha Ewry:	Be harder to do in the other direction, I'd imagine
[13:53]	Zero Linden:	Yes - but there there is the much more natural solution - just let you texture an object with a URL to your own image on your own server...
[13:53]	Zero Linden:	Not that we are doing that right now....
[13:53]	Zha Ewry:	Yes.
[13:53]	Zero Linden:	We do have to be careful about opening the flood gates to textures
[13:53]	Hiro Market:	like the sound of that
[13:54]	Zha Ewry:	That'd be fine, within reason. How exactly we get the proggramitrc control is interesting, but I can see ways.
[13:54]	Zha Ewry:	Yes, I'm wiling to hear lots of stories about throttle on textures
[13:54]	Zero Linden:	Obviously if everyone just uploaded a texture for every surface they needed, under LSL command... we'd see things like
[13:54]	Hiro Market:	from my perspective one of the biggest restrictions on LSL is not being able to programatically manipulate textures
[13:54]	Zero Linden:	billboards that uploaded new textures every 15 seconds
[13:54]	Rex Cronon:	it would be ok to me if instead of html i can use svg
[13:54]	Hiro Market:	at pixel level
[13:54]	Zero Linden:	which would be cool from the SL experience point of view
[13:54]	Zha Ewry:	Yes... throttleing is an issue, of course.
[13:54]	Zero Linden:	but would really kill the asset server
[13:54]	Zha Ewry:	Althoug.. if you can have "transient" textures
[13:55]	Sifu Moraga:	It does sound like you will need the concept of transient blobs stored by the sims
[13:55]	Zha Ewry:	locally cached, I can come up with a story that only pounds the sim
[13:55]	Zha Ewry:	And not even that bad.
[13:55]	Zero Linden:	Hiro - you aren't likely to see things like paramatic textures or pixel level texture control from LSL
[13:55]	Zha Ewry:	The sim has to fetch/cache the tyextures today
[13:55]	Zero Linden:	These are memory and process intensive operations that the sim doesn't have the resoruces to do
[13:55]	Zha Ewry:	Nor should it
[13:55]	Zero Linden:	what is more likely, what scales much better, is
[13:55]	Zha Ewry:	Keep as much rendering on the client as possible
[13:55]	Hiro Market:	again I wonder about procedural textures on the client with shader code on server
[13:56]	Zha Ewry nods at Hiro
[13:56]	Zero Linden:	do the pixel level processing on your own computer or server - and then have a way to put that texture on an object
[13:56]	Zero Linden:	The servers, the sims, have no graphics hardware at all... so no OpenGL going on there
[13:56]	Zha Ewry:	I think mosgt of us who have loooked at the scale issues want a smuch rendering as possible on the client
[13:56]	Zha Ewry:	I don't want the SIMs even thinking about rendering
[13:56]	Hiro Market:	don't need it, just the shader code on server, rendered on client
[13:57]	Zero Linden:	don't expect the sims to be doing graphic manipulation or rendering (which is a hint, by the way, as to why the map is such a pain point for us!)
[13:57]	Zha Ewry:	Among other horrors, you end up with the sim needing to know things like the resolutoin of every client they are talking to
[13:57]	Zero Linden:	Ah - but Hiro, can you imagine what it would be like if an object could down load arbitrary shader code to your viewer?!?!?
[13:57]	Hiro Market:	but procedural shader code is simply code, no rendering on server required at all
[13:57]	Zero Linden:	Talk about griefing!
[13:58]	Hiro Market:	hmm, point taken, but the advantages must be worth exploring surerly?
[13:58]	Zha Ewry:	He. Just because sooner or later we'd have all the pron spam in the shader :-)
[13:58]	Zero Linden:	Alas, OpenGL doesn't have virtualization or sandboxing facilities to make that reasonable
[13:58]	Zero Linden:	Nor does anyone have shader verifiers as far as I know....
[13:58]	Zero Linden:	But - one could imagine in
[13:58]	Zero Linden:	the future
[13:58]	Sifu Moraga thinks there might be a case for somre rendering on the server, but it would probably need a different architecture
[13:59]	Hiro Market:	3dlabs have one
[13:59]	Zha Ewry shudders at what you could get the shader to do.
[13:59]	Hiro Market:	a verifier that is
[13:59]	Zero Linden:	private sims that said "If you come here, I will use spiffy special purpose shaders.... allow?"
[13:59]	Rex Cronon:	talking about griefing, what exactly does the physics engine do to attachements, when the one that wears them is blitzed?
[13:59]	Zha Ewry:	Hm. That'd be interesting
[13:59]	Zha Ewry:	In general, the question of private sim behavioir is interesting
[13:59]	Zero Linden:	Hiro - curious - good to know - though again I wonder if it worries about hostile environments or just makeing sure things won't crash...
[14:00]	Sifu Moraga:	My cranial CPU is shutting down, and my Powerbook is out of juice, so I'm off early
[14:00]	Sifu Moraga:	cu all
[14:00]	Zero Linden:	Yes, it is.... short term, once we start letting estate owners control upgrade times.... we are going to have to deal with
[14:01]	Zero Linden:	viewers and regions with different feature sets....
[14:01]	Zha Ewry:	Yes
[14:01]	Zero Linden:	Well, Sifu - actually it is 2pm so I'm signing off soon too
[14:01]	Zero Linden:	Shorter term -
[14:01]	Zha Ewry:	we talked about tadeoffs, between number of scirpts and data spaces and such
[14:01]	Zha Ewry:	last week.
[14:01]	Zha Ewry:	So lots of things I'd love to be able to contrrol on the sim :-)
[14:02]	Zero Linden:	one could easily imagine custome viewers that have special shaders in 'em and then some way for objects
[14:02]	Zero Linden:	to invoke that shader behavior if running on viewer that has 'em
[14:02]	Zha Ewry:	Wow. This was incredbly useful Zero. Hope it was for you. I'll try and update my blob comments a bit. and think about some of your comments
[14:02]	Zero Linden:	Zha - yes - though realize, like all server tuning, the more parameters you have, the better performance you can achieve, but the more compicated it is for people to manage it
[14:03]	Zha Ewry:	Hey I race saliboats. More strings, more fast. or more slow.
[14:03]	Zha Ewry:	so, yes.
[14:03]	Zero Linden:	and - one has to realize that you can only do so much, even with so many knobs
[14:03]	Zero Linden:	(thinks of tuning squid, or apache, or mysql....)
[14:03]	Zha Ewry:	Yep. 1 cpu, 512M of memory, etc.
[14:03]	Zha Ewry:	But, there are cases where we can play those off well
[14:04]	Zero Linden:	It is great - I'm very happy with my office hours experiences - and have been talking up how good they are inside of LInden
[14:04]	Zha Ewry:	Thanks again Zero, as always. Really focused discussion today
[14:04]	Zero Linden:	Thanks all for coming
[14:04]	Zero Linden:	See you next time.
[14:04]	Kooky Jetaime:	Thank you Zero
[14:04]	Vitis Obviate:	very valuable info - ty
[14:04]	Drewan Keats:	Thanks Zero.
[14:04]	Hiro Market:	ty zero, aoppreciated
[14:04]	Zha Ewry:	Thanks everyone, actually, very good discussion today

--Dr Scofield 01:57, 20 March 2007 (PDT)