User:Zero Linden/Office Hours/2007 Mar 01

Transcript of Zero Linden's office hours from 2007 March 1st:

Special thanks to Zha Ewry who sent me the transcript after my client crashed right near the end!

Rex Cronon: i guess u came to the right place to talk to a linden:)
Zero Linden: Hello hello
Zha Ewry: Hey Zero
Johannes Mulligan: forget it
Vitis Obviate: Hey Zero
Rex Cronon: hello
Zero Linden: Well, Rex, you did if what you wnat is deep tech talk!
Vitis Obviate: I can only stay about 10 min this morning
Vitis Obviate: so appologizes in advance
Zero Linden: Oh dear - and I've got a treat for you all
Zero Linden: so, perhaps we should just get going!
Rex Cronon: actually i was talking to Johannes Mulligan, he said he was 12 years old:)
Xanshin Paz: HI, FOLKS!
Rex Cronon: hi
Zero Linden: Well- actually, the treat is going to take a while, so ....
Soft Noel: Hey hey :)
Zha Ewry: Hey Vitus
Zha Ewry: Still missing a few of the usual suspects, but I've got my RL coffe, and am ready to listen
Vitis Obviate is listening
Zero Linden: yikes - I'm spilling my coffee all over my couch
Zero Linden: All right
Zero Linden: as always - the transcript of this chat will be posted 
Zha Ewry nods
Zero Linden: Can everyone see that new white screen I installed
Zha Ewry: Yes
Rex Cronon: it has a ver nice white color:)
Zero Linden: I want to show you some images of the message system
Xanshin Paz: MHMM
Zero Linden: next

Zha Ewry Smiles. Oooh, Images
Zero Linden: warning - 512x512 images coming!
Zero Linden: okay - this is the message system today
Rex Cronon: this is an good example why html is needed on obj surfaces:)
Zero Linden: There is a binary API for creating and reading messages
Vitis Obviate: will you be posting these images to the blog also?
Zero Linden: by binary, I mean that the API was written on the assumption that the messages were packed a certain way
Zero Linden: we also have a binary encoding of messages, using the fabled message template
Zero Linden: and as you know, we transmit those messages over UDP
Zero Linden: You can find all this code in the open source release
Zero Linden: The advantage of this system is that
Zero Linden: it is very efficient in packing, and network use 
Zero Linden: and, for messages that have UDP like semantics, it poses no overhead
Zero Linden: The downsides, alas, are many:
Zha Ewry: Ah, so you're doubly hammered. both UDP, and you've a very low level formatting world, you're fixing both of these witht he message update
Rex Cronon: i was hoping for a UML diagram, but one can only hope
Zero Linden: fixed message length
Zero Linden: fixed message layout (the template)
Zero Linden: and doing anything with longer data, or reliable data, must be done at a higher level in the protocol stack
Zero Linden: But the biggest impediment right now is
Zero Linden: that you have to change every piece of SW if the message template changes
Vitis Obviate nods
Zha Ewry: (with attendent code to parse/unparse of course. for bigger things, and missing packets, means no message at all in a bigger chunk)
Zero Linden: and this is the reason for so many viewer downloads
Zero Linden: next

Zero Linden: This is phase 1
Rex Cronon: sorry, but what does "sw" mean?
Zero Linden: The goal here is to use LLSD (our own structued data abstration that is extensible) 
Zero Linden: and other transports to move messages
Zero Linden: sw = sowftware
Zha Ewry: Zero, any problem snapshotting these and passing them along to others?
Rex Cronon: oh
Zero Linden: Zha - no - I'll be posting them with the transcript, though...
Vitis Obviate: excellent
Zero Linden: but you can grab 'em now if you want
Zha Ewry nods. I'm impatient :-)
Xanshin Paz: wow
Zero Linden: If the data is in LLSD, then we can extend it, and be tollerant of both forward and backward compatible changes
Zero Linden: BUT
Soft Noel: Capability - that's a general system, or a class of systems?
Zero Linden: there are hundreds of call-sites in the code base that use the existing API
Zero Linden: So, we have to endevour to first make the new encoding and transports compatible with all the old code
Zero Linden: it would be too destabilizing to try to update all the code to some new API
Vitis Obviate nods thanks - and looks forward to reading the transcript. Sends Zha and Rez, and Xan and Soft clarity dust to ensure Zero stays focused on on track - bows out 
Zero Linden: Since the old message API is rather old, and has evolved very closely with the binary encoding and UDP transport,
Zero Linden: it has become rather entangled -- hence this work is the hardest
Zha Ewry nods. Are you going to both beta-grid and first look the updates?
Zero Linden: Capability here refers to capability based transport - where the viewer gets a capabiliy, or an unforgable, unguessable, URL to an HTTP service
Soft Noel: ty
Zero Linden: This code has a live configurable switch where we can send messages either via UDP or via LLSD transports
Zha Ewry nods, echos of the Hydra/MMP work
Hazime Watanabe: stop
Boyoma Bi Plane whispers: Shutting down
Zero Linden: Zha, yes
Zero Linden: Right now, we have a test grid where it is all turned LLSD - and it is almost working
Zero Linden: Then we'll test and deploy this code with it all turned back to UDP
Zero Linden: In this way, we'll ensure that the fallback - should anything go wrong in the future
Zero Linden: will still work
Zero Linden: and we can switch it live on the grid 
Zero Linden: without shutting down
Zha Ewry cringes at phrase "almost" She's spent too much time fixing "almost working" code :-(
Zero Linden: questions?
Zero Linden: on this slide?
Zha Ewry: One. Does ...
Soft Noel: There's an equivalent switch going back upstream, viewer->sim?
Rex Cronon: i guess the the voice chat to be added is using llsd?
Zero Linden: Zha - my studio has been using XP practices, estimating our tasks and tracking our velocity
Zero Linden: we are pretty sure of our time table
Zha Ewry: everything go tot he capabitilty stuff in time, or just stuff you really care about
Zha Ewry: Xtreme programming helps, yes.
Zero Linden: it took some time to get the team to this point - but we've got things running well
Hazime Watanabe: start
Boyoma Bi Plane whispers: Starting engines
Zero Linden: Zha - just things between systems that don't have trust
Zero Linden: between simulators, we use direct HTTP
Zha Ewry: Ah, what determines that now? COlo? or subsystem ownership?
Zero Linden: they are all behind our firewall, so, no need for the capabilities 
Zero Linden: okay
Zero Linden: next

Rex Cronon: np
Zero Linden: The next step is this
Zha Ewry: Ahh...
Zero Linden: we add new APIs for sending and receiving messages
Zero Linden: These APIs use LLSD, and so
Zha Ewry: That's a nce leverage point
Zero Linden: a handler for a message written this way can deal with older versinos of the message as well as newer
Zero Linden: The nice thing about this step is....
Zero Linden: ....it is no code at all!
Zero Linden: all it is doing is exposing the internal APIs we developed for the first step
Zero Linden: !
Zha Ewry: But it does, nicely further decouple stuff
Soft Noel grins.
Zero Linden: At this point, so long as a message is routed over a LLSD transport
Rex Cronon: looks like the llsd doesn't use any udp
Zero Linden: we can have systems of different versions interact
Zha Ewry: Messaging today, so far, tree
Zha Ewry: I assume you want to have no UDP, at all, over time, Zero? 
Zero Linden: Rex - well, it is concievable that we could make a LLSD transport that ran over UDP
Zero Linden: but we don't anticipate a need for that now
Zha Ewry cringes
Zha Ewry: UDP isn't gonig to be happy SFO to Europe or Asia
Zero Linden: Zha - we will probably alwyas have some UDP, with binary encoding for those messages that it makes sense for:
Zero Linden: things like object updates where if you lose one, no big deal -
Zha Ewry: Across colos?
Zero Linden: and where the compatness of the data seems important
Zero Linden: Zha - no between colos, we'll be doing TCP based transports
Zero Linden: but realize that right now - it is ALL UDP
Zha Ewry: Yes, and it shows, alas.
Zero Linden: Yes, so sympathize with our Eurpean and Asian residents!!!
Tree Kyomoon: I have a question about messaging to inventoried objects...will it be possible soon to communicate with objects inside inventories that arent rezzed?
Zha Ewry: Ok, UDP, box to box, inside a colo, I'll believe
Zero Linden: Actually - even there, probably mostly TCP - because 
Zha Ewry is on the east coast. I have sympathy with me!
Zero Linden: most of that traffic needs to be reliable, and can get large
Zero Linden: next

Zha Ewry: OK, so, UDP local, when it makes sense. Fair enough. I'll stop bitching
Zero Linden: Now, for safety
Zero Linden: we've design this pathway
Zero Linden: IF we need, for some reason, to keep a particular message binary/UDP -- then we can still support
Rex Cronon: now makes more sense
Zha Ewry: Heh. Belt and Suspenders, and don't trust peple not to be lazy. Makes sense
Zero Linden: the new APIs by bridging
Zero Linden: Indeed -
Rex Cronon: i gues i might not need the binary builder/reader
Zero Linden: In other software environments, I'm usually much more of a rip-it-all-out-and -redo-it
Zero Linden: kind of guy
Rex Cronon: guess u*
Zero Linden: but here at Linden, I"m very concious that we've got to keep a very complex system up and running
Zha Ewry: I'm especially happy to see this paranoia, given the open source client, as well, Zero
Zero Linden: oh
Zero Linden: prev

Zero Linden: Here - on this one
Zero Linden: At this point we "freeze" the Message Template
Zero Linden: meaning that we only allow new messages, depricating old messages, or extending messages at the end of the message
Zha Ewry: Ahh. So, from there forward, no new binary formats. Excellent
Zero Linden: It is required to allow the binary transport to support differeing versions - at least after the freeze point
Zero Linden: Well, again, we need to be cautious - so I can't say *no* new ones
Zero Linden: There is enough working going on simultaneously in dev
Zha Ewry grins. Ok, best effort. Be tough.
Zero Linden: that the most we can be realistic about is, only wholey new ones, or extensions -
Zero Linden: but no removing data fields, no renumbering,
Zero Linden: we call this point
Zero Linden: Día de la Liberación
Zha Ewry nods. I assume you've been telling people this is comming for a while, so they'll get any last minute changes in now.
Xanshin Paz: hehe
Zero Linden: Oh yes - I've been preaching this internally for several months 
Zha Ewry blnks. Will you be sequenced after the new render pipeline? (or in paralell?)
Zero Linden: And the funny name is partially to drive home the point that this is an important moment in engineering
Zero Linden: after -
Zha Ewry nods. Good.
Zero Linden: we are aiming for end of March internally - meaning we'll have the first liberated test grid internally by end of quarter
Zero Linden: next
Zero Linden: next

Zero Linden: So - this is the future of the message system
Zero Linden: Thought you all might like to see it!
Zha Ewry: Yes, very much
Zero Linden: Any questions?
Soft Noel: You had the switch sim-side for the old/LLSD crossover per message type. The sim will also be telling the client what transport to use on the fly?
Soft Noel: s/client/viewer/
Zero Linden: We'll actually run both transports to the viewer
Zero Linden: Some messages will go binary/UDP, other LLSD
Zero Linden: Initially all going UDP, but we'll be able to switch, message by message, over to LLSD
Zha Ewry: IS that per sim, or across the whole grid? 
Zero Linden: slide 1

Zero Linden: whole grid
Zero Linden: though in the future, once we have the internal management to run different versions of the sim on differnt hosts
Zero Linden: we'll be able to do it version by version
Zha Ewry: Any chance of doing it per sim for beta testing?
Zero Linden: Yes - that is exactly the plan
Zero Linden: Once we can run a hetrogenous grid
Zero Linden: (the day of liberation!)
Zero Linden: We can then do things like bring up some set of regions on the main grid with the next release, in beta
Zero Linden: we can then test moving messages over to LLSD on those regions only
Zero Linden: The viewer will happily accept messages either way
Zero Linden: Notice that we don't talk HTTP directly to the viewer or from it ---
Zero Linden: To the viewer we can't due to firewalls - so we use the event queue - where the viewer comes back an polls
Zero Linden: this sounds awful BUT
Zero Linden: it is a "long poll" meaning that if viewer asks for messages and there aren't any- the sim just stalls replying "none".... 
Zero Linden: it stalls replying at all... until the moment a message comes
Zero Linden: and then it replys with "here"
Zero Linden: From viewer to sim, we use a capability - (which itself is HTTP), but in this way, the capability provides the authenication
Zha Ewry: Hmm. Any way of having us "kick" the long poll from the client side? (I think I asked this Tuesday)
Zero Linden: Meaning? restart it?
Zha Ewry: Yes, pretty much
Zha Ewry: I'm often aware we've lost the pipe, but the server (sim) doesn't know, of course
Zero Linden: Well, the current viewer code doesn't have a UI for that
Zero Linden: but you could easily - add a menu item- the code base, LLEventPoll (I think), could easily be made to handle just dropping it's current request on the floor and restarting a new one...
Zero Linden: However - are you sure it is just the long poll that has dropped?
Zha Ewry: Well, that's a good question.
Zero Linden: usually what kills things is that the UDP channel has lost its connection (meaning our higher level abstraction of circuit) 
Zha Ewry: It *might* be worth a debug option to test/sort that out.
Rex Cronon: how can udp channel loose connection?
Rex Cronon: is never connectd
Rex Cronon: connected*
Tree Kyomoon: could add it to the statistics bar
Zero Linden: slide 4

Zero Linden: Rex - it isn't the UDP layer,(since UDP has no connection), is the connection layer in our message transport built on top of the UDP system
Zero Linden: it relies on a set of pings and other sync. operations
Zha Ewry: Ahh.
Zha Ewry: Sure.
Zero Linden: because the current system tries to send reliable data through UDP!
Zha Ewry: So, when one "ghosts" of the grid, we don't really know what's lost synch
Zha Ewry: off
Zero Linden: Also, realize that since most viewers are behind NATs - if you don't keep the UDP circuit running, you can loose the NAT association, and
Rex Cronon: oh, so you tried to build your own version on TCP:)
Zero Linden: you may get a new external port on the next send.... which kind of breaks thigns
Zero Linden: Rex - cool isn't exactly the term I'd use.... 
Zha Ewry: Heh. Yes. I've looked at my firewall logs at home
Rex Cronon: i didn't say cool
Zha Ewry: That makes perfect sense, now.
Tree Kyomoon: so what can be done about the ghosting?
Zha Ewry: Oh, it's cool, not cool, in the really pretty pair of shoes, sense, but cool, like the dancing bear. It does dance.
Zero Linden: oops - you're right - I misscanned your last name and "oh" into "cool"... suppose I need to use a bigger font!
Soft Noel: If it's going TCP now, there's a connection that dies when you ghost
Zero Linden: Tree - I think that once we have most things moved over to LLSD transports - then we can
Zero Linden: work on adding support for re-establishing the UDP link if it goes...
Zero Linden: right now, that isn't really possible as you have almost no other link to the grid...
Zha Ewry looks thoughtful...
Tree Kyomoon: well for now could you make it visually obvious that you are not on the grid?
Tree Kyomoon: by logging the user off or whatever?
Zha Ewry: Yeah, sure. You're just giong to be miserable, trying to resynch you whole set of things at the moment 
Zha Ewry: I'd rather see the effort go into getting the new stack finished :-)
Zero Linden: Again - the problem with a UDP based transport is that you really don' t know on the sim side what is going on...
Rex Cronon: i have see lots of people that crash, and their av is still there
Zha Ewry: That's UDP for you.
Tree Kyomoon: Ive built some neat stuff while I was crashed...only to find I had built it entirely while offline
Zha Ewry: The sim never gets told that you're gone until you re-log
Zero Linden: Rex - well, the other problem is, if those AVs have lots of attachments, (or big, in the data sense) then the sim has to wait until those attachments are saved
Zero Linden: mind you - I suppose we could add a message to the viewer that says "don't show this ghost"
Tree Kyomoon: that would be good
Zero Linden: though I prefer the message that says "let other av's draw on this ghost with lipstick"
Zha Ewry this the always wrong, but well meaning "don't log on until next thursday" messages
Rex Cronon: i had the same problem that tree had 
Zero Linden: Yes -
Zha Ewry: You come back on, the sim says, "Ah, Ave gone" starts the logout... Until that's done, can't let you back in as the asset server is incoherent
Zero Linden: a different approach to that problme would be if we had a way to know which scripts had any meaningful persistent state
Zero Linden: most scripts in attachements do not - you can reset them at will
Rex Cronon: the thing is as long as u can move aroun u are connected, that seems one way to know when u lost connection
Zero Linden: in those cases, we could, as long as yo uhaven't edited the attachement, just not save it at all
Tree Kyomoon: yes thats what I do now
Zha Ewry: Yes, once you can only turn, but not move/standup, etc. You're ghosted
Zero Linden: Yes - that is a pretty good test - though again, I've seen UDP drop for 30 seconds and then return....
Tree Kyomoon: yes, Ive seen that too
Zero Linden: but admittedly, it is rare
Zero Linden: that it comes back, that is
Tree Kyomoon: yes
Zha Ewry: Hah! Now... Here's a good question? How about having a stronger "Gonig away" message, when you exit from the browser, to start the logout process? 
Soft Noel: I frequently end up ghosted while flying, and warp back the moment I hit a sim boundary.
Tree Kyomoon: lol
Rex Cronon: if u have open a browser and load a page, sometimes u r reconnected
Zero Linden: Rex - that sounds more like a problem with intermediate routers - like perhaps your NAT box....
Zero Linden: but again,SL's heavy UDP use isn't very friendly to most home NAT boxes
Tree Kyomoon: so I would really like to get my question in about messaging to inventorys of objects before we go
Tree Kyomoon: if that is ok
Rex Cronon: migh be possible, i am on wireless:)
Zero Linden: Sure
Zero Linden: Shoot
Zha Ewry: Did you see my notes on the wiki about blobs?
Tree Kyomoon: I need to be able to send messages / scripts to inventoried objects
Zero Linden: I did Zha - and have some thoughts back - mostly about the need for additional API to access that data from LSL
Tree Kyomoon: will that ever be possilbe?
Zero Linden: Tree - no, never
Zha Ewry: Yes, I owe you, some api samples, of that sort, if you'd like. 
Zero Linden: objects in inventory are "freeze dried" - they are templates for objects, not objects themselves
Zha Ewry: I've a pretty good image in my head, of what that might look like
Soft Noel: Tree: What do they need to do in inventory that they can't check for when pulled out of inventory?
Zero Linden: only when "rez'd", the template is instantiated and then you have an object
Tree Kyomoon: basically I need a parent/child relationship
Tree Kyomoon: and that doesnt seem to exist in "objects"
Zero Linden: This does make some semblence of sense - rez'd objects take resources of the simulator: object lists, script state,etc....
Zha Ewry nods. You want a strong distinction, otherwise you end up uable to tell what might need to be live in people's 10,000 object inven tory
Tree Kyomoon: and you cant pass much to a rezzed object
Zero Linden: Inventories only cost asset server data storage
Zero Linden: Tree - well, perhaps that is the bigger issue
Zero Linden: most builders have solved it with rather intracate and awkward messaging between objects 
Rex Cronon: actually u can but is so slow:(
Zha Ewry frowns. Yes. I've been noodling over how to handle it in the blobbing cases
Zero Linden: but a richer API for rezing an object would be nice
Tree Kyomoon: yes, taking days or weeks to do very simple tasks in most oop languages
Tree Kyomoon: yes, is that possible in the future?
Zha Ewry: Something that says "These are the children I've rezzed recently" (in a list) and (this is my parent)
Zha Ewry: With a simpe API to to things similar to LinkedMessageSends to the children/parent
<<zero linden crashes and reconnects>>
Zero Linden: bother
Tree Kyomoon: lol
Zero Linden: see - even Lindens!
Zha Ewry smiels. SL
Zero Linden: So - er, yes
Rex Cronon: your client crashed zero?
Zero Linden: even my own projects make me frustrated
Zero Linden: when working in LSL
Tree Kyomoon: so I was just asking about parent/child relatiohships with objects 
Zero Linden: Rex - yes
Zero Linden: Oooooo that one!
Zero Linden: Short term - no we aren't going to be doing major changes to the object system like that
Tree Kyomoon: aww
Zha Ewry: Yes. Got it all the way back to before spilled coffee
Rex Cronon: i don't know if u know zero, but there is a wepon that can crash the client, it seem it spams it with textures
Zero Linden: longer term (year?) we are trying to get to the point where such things could be done
Rex Cronon: weapon*
Zero Linden: Well, we are aware of all sorts of bad content - and we know that some of it can cause the client to crash -
Zha Ewry: Am I right in guessing that the theme, at least for the first half of '07 is scale not function?
Zero Linden: Zha - without question
Zero Linden: there is a group working on high priority bug fixes and UI problems
Tree Kyomoon: well mabey a libsecondlife project to create parent child object emulation
Zha Ewry: So, one question is, how can we prioritize the functions we do need, so that we have a chance of getting those in the pipelines soonest, once you get the 
Zha Ewry: scaleing under control?
Zero Linden: and there is my studio working on larger architectural changes
Zero Linden: but everyone else is scaling!
Zha Ewry: Heh. You're scaling too, just in the large
Zero Linden: Zha - well, actually, the rate of total fall overs has reduced somewhat in the last few months
Zha Ewry: Yes, it has.
Zero Linden: We have lots of things to do, but seem to be keeping ahead of the tidal wave!
Rex Cronon: can there be a pool asking people what functions do they need?
Zero Linden: But I will admit we feel like we're in a bit of a race!
Zha Ewry: I'd love to know why the worst things hapen 2-8 hours after peak loads
Zha Ewry: (You'll notice that the pattern is very consistent. Peak load around 3:00 pm, bad grid by around 8:00 pm)
Zha Ewry: Especialy when we hit new total highs
Zero Linden: Zha - the interactions of the DBs are difficult to model - but
Zero Linden: things start to fall behind at peak - and get progressivly worse
Zero Linden: that is , the slave dbs are falling behind in logical time 
Zero Linden: when they get very out of whack - stuff breaks
Zero Linden: right at peak, they are just starting to fall behind, so things might not be so bad...
Zha Ewry: What's odd, is that we often seem quite good, from an hour or two after the peak, and then, it falls over. One would *hope* it would catch up
Zha Ewry: Instead, its as if something is harvesting objects, or otherwise adding load well after people log off
Zero Linden: catching up, once behind, seems very diffficult and sluggish to us -
Zero Linden: we have DB folks, internal and external, looking at it, and they poke here and there, but so far no clear solution to that particular problem
Tree Kyomoon: bfore the end....just wanted to ask.....sheepishly....cookies-any further thoughts?
Zha Ewry: Well, of course, if you used DB/2 in full commercial setup... :-)
Zero Linden: so our goal is keep ahead of it by moving things out of the DBs, and splitting DBs.
Zero Linden: Sorry, Tree, not at the moment...
Zha Ewry: (I think I'm required to say that :-) 
Tree Kyomoon: ok just call me sqeaky
Zha Ewry: But yes, scaling out, to more DBs should help
Zero Linden: Okay all - my time is up - got to get back to coding
Zha Ewry: Zero, would you like some API thoghts on how to deal with blobbed objects?
Zha Ewry: I can add them to the wiki over the next week
Zha Ewry: (before next tuesdasy)
Zero Linden: Hope all this was interesting.
Zha Ewry: Very.
Zero Linden: Thank you all for coming
Tree Kyomoon: very
Zha Ewry: I'll e-mail you the transcript montarily
Zero Linden: Yes, Zha - I do read the wiki entries you all post!
Rex Cronon: bye
Wyn Galbraith smiles, "Thank you Zero."
Xanshin Paz: thanks for having us!
Zha Ewry: Thanks, as always
Zero Linden: great, Zha, appreaciate it
Zha Ewry: And the slides were really, really helpful
Tree Kyomoon: thnks again!