User:Zero Linden/Office Hours/2007 Mar 01
< User:Zero Linden/Office Hours
Jump to navigation
Jump to search
Revision as of 00:04, 16 March 2007 by SignpostMarv Martin (talk | contribs) (used template for categorisation)
Transcript of Zero Linden's office hours from 2007 March 1st:
Special thanks to Zha Ewry who sent me the transcript after my client crashed right near the end!
Rex Cronon: i guess u came to the right place to talk to a linden:) Zero Linden: Hello hello Zha Ewry: Hey Zero Johannes Mulligan: forget it Vitis Obviate: Hey Zero Rex Cronon: hello Zero Linden: Well, Rex, you did if what you wnat is deep tech talk! Vitis Obviate: I can only stay about 10 min this morning Vitis Obviate: so appologizes in advance Zero Linden: Oh dear - and I've got a treat for you all Zero Linden: so, perhaps we should just get going! Rex Cronon: actually i was talking to Johannes Mulligan, he said he was 12 years old:) Xanshin Paz: HI, FOLKS! Rex Cronon: hi Zero Linden: Well- actually, the treat is going to take a while, so .... Soft Noel: Hey hey :) Zha Ewry: Hey Vitus Zha Ewry: Still missing a few of the usual suspects, but I've got my RL coffe, and am ready to listen Vitis Obviate is listening Zero Linden: yikes - I'm spilling my coffee all over my couch Zero Linden: All right Zero Linden: as always - the transcript of this chat will be posted Zha Ewry nods Zero Linden: Can everyone see that new white screen I installed Zha Ewry: Yes Rex Cronon: it has a ver nice white color:) Zero Linden: I want to show you some images of the message system Xanshin Paz: MHMM Zero Linden: next
Zha Ewry Smiles. Oooh, Images Zero Linden: warning - 512x512 images coming! Zero Linden: okay - this is the message system today Rex Cronon: this is an good example why html is needed on obj surfaces:) Zero Linden: There is a binary API for creating and reading messages Vitis Obviate: will you be posting these images to the blog also? Zero Linden: by binary, I mean that the API was written on the assumption that the messages were packed a certain way Zero Linden: we also have a binary encoding of messages, using the fabled message template Zero Linden: and as you know, we transmit those messages over UDP Zero Linden: You can find all this code in the open source release Zero Linden: The advantage of this system is that Zero Linden: it is very efficient in packing, and network use Zero Linden: and, for messages that have UDP like semantics, it poses no overhead Zero Linden: The downsides, alas, are many: Zha Ewry: Ah, so you're doubly hammered. both UDP, and you've a very low level formatting world, you're fixing both of these witht he message update Rex Cronon: i was hoping for a UML diagram, but one can only hope Zero Linden: fixed message length Zero Linden: fixed message layout (the template) Zero Linden: and doing anything with longer data, or reliable data, must be done at a higher level in the protocol stack Zero Linden: But the biggest impediment right now is Zero Linden: that you have to change every piece of SW if the message template changes Vitis Obviate nods Zha Ewry: (with attendent code to parse/unparse of course. for bigger things, and missing packets, means no message at all in a bigger chunk) Zero Linden: and this is the reason for so many viewer downloads Zero Linden: next
Zero Linden: This is phase 1 Rex Cronon: sorry, but what does "sw" mean? Zero Linden: The goal here is to use LLSD (our own structued data abstration that is extensible) Zero Linden: and other transports to move messages Zero Linden: sw = sowftware Zha Ewry: Zero, any problem snapshotting these and passing them along to others? Rex Cronon: oh Zero Linden: Zha - no - I'll be posting them with the transcript, though... Vitis Obviate: excellent Zero Linden: but you can grab 'em now if you want Zha Ewry nods. I'm impatient :-) Xanshin Paz: wow Zero Linden: If the data is in LLSD, then we can extend it, and be tollerant of both forward and backward compatible changes Zero Linden: BUT Soft Noel: Capability - that's a general system, or a class of systems? Zero Linden: there are hundreds of call-sites in the code base that use the existing API Zero Linden: So, we have to endevour to first make the new encoding and transports compatible with all the old code Zero Linden: it would be too destabilizing to try to update all the code to some new API Vitis Obviate nods thanks - and looks forward to reading the transcript. Sends Zha and Rez, and Xan and Soft clarity dust to ensure Zero stays focused on on track - bows out Zero Linden: Since the old message API is rather old, and has evolved very closely with the binary encoding and UDP transport, Zero Linden: it has become rather entangled -- hence this work is the hardest Zha Ewry nods. Are you going to both beta-grid and first look the updates? Zero Linden: Capability here refers to capability based transport - where the viewer gets a capabiliy, or an unforgable, unguessable, URL to an HTTP service Soft Noel: ty Zero Linden: This code has a live configurable switch where we can send messages either via UDP or via LLSD transports Zha Ewry nods, echos of the Hydra/MMP work Hazime Watanabe: stop Boyoma Bi Plane whispers: Shutting down Zero Linden: Zha, yes Zero Linden: Right now, we have a test grid where it is all turned LLSD - and it is almost working Zero Linden: Then we'll test and deploy this code with it all turned back to UDP Zero Linden: In this way, we'll ensure that the fallback - should anything go wrong in the future Zero Linden: will still work Zero Linden: and we can switch it live on the grid Zero Linden: without shutting down Zha Ewry cringes at phrase "almost" She's spent too much time fixing "almost working" code :-( Zero Linden: questions? Zero Linden: on this slide? Zha Ewry: One. Does ... Soft Noel: There's an equivalent switch going back upstream, viewer->sim? Rex Cronon: i guess the the voice chat to be added is using llsd? Zero Linden: Zha - my studio has been using XP practices, estimating our tasks and tracking our velocity Zero Linden: we are pretty sure of our time table Zha Ewry: everything go tot he capabitilty stuff in time, or just stuff you really care about Zha Ewry: Xtreme programming helps, yes. Zero Linden: it took some time to get the team to this point - but we've got things running well Hazime Watanabe: start Boyoma Bi Plane whispers: Starting engines Zero Linden: Zha - just things between systems that don't have trust Zero Linden: between simulators, we use direct HTTP Zha Ewry: Ah, what determines that now? COlo? or subsystem ownership? Zero Linden: they are all behind our firewall, so, no need for the capabilities Zero Linden: okay Zero Linden: next
Rex Cronon: np Zero Linden: The next step is this Zha Ewry: Ahh... Zero Linden: we add new APIs for sending and receiving messages Zero Linden: These APIs use LLSD, and so Zha Ewry: That's a nce leverage point Zero Linden: a handler for a message written this way can deal with older versinos of the message as well as newer Zero Linden: The nice thing about this step is.... Zero Linden: ....it is no code at all! Zero Linden: all it is doing is exposing the internal APIs we developed for the first step Zero Linden: ! Zha Ewry: But it does, nicely further decouple stuff Soft Noel grins. Zero Linden: At this point, so long as a message is routed over a LLSD transport Rex Cronon: looks like the llsd doesn't use any udp Zero Linden: we can have systems of different versions interact Zha Ewry: Messaging today, so far, tree Zha Ewry: I assume you want to have no UDP, at all, over time, Zero? Zero Linden: Rex - well, it is concievable that we could make a LLSD transport that ran over UDP Zero Linden: but we don't anticipate a need for that now Zha Ewry cringes Zha Ewry: UDP isn't gonig to be happy SFO to Europe or Asia Zero Linden: Zha - we will probably alwyas have some UDP, with binary encoding for those messages that it makes sense for: Zero Linden: things like object updates where if you lose one, no big deal - Zha Ewry: Across colos? Zero Linden: and where the compatness of the data seems important Zero Linden: Zha - no between colos, we'll be doing TCP based transports Zero Linden: but realize that right now - it is ALL UDP Zha Ewry: Yes, and it shows, alas. Zero Linden: Yes, so sympathize with our Eurpean and Asian residents!!! Tree Kyomoon: I have a question about messaging to inventoried objects...will it be possible soon to communicate with objects inside inventories that arent rezzed? Zha Ewry: Ok, UDP, box to box, inside a colo, I'll believe Zero Linden: Actually - even there, probably mostly TCP - because Zha Ewry is on the east coast. I have sympathy with me! Zero Linden: most of that traffic needs to be reliable, and can get large Zero Linden: next
Zha Ewry: OK, so, UDP local, when it makes sense. Fair enough. I'll stop bitching Zero Linden: Now, for safety Zero Linden: we've design this pathway Zero Linden: IF we need, for some reason, to keep a particular message binary/UDP -- then we can still support Rex Cronon: now makes more sense Zha Ewry: Heh. Belt and Suspenders, and don't trust peple not to be lazy. Makes sense Zero Linden: the new APIs by bridging Zero Linden: Indeed - Rex Cronon: i gues i might not need the binary builder/reader Zero Linden: In other software environments, I'm usually much more of a rip-it-all-out-and -redo-it Zero Linden: kind of guy Rex Cronon: guess u* Zero Linden: but here at Linden, I"m very concious that we've got to keep a very complex system up and running Zha Ewry: I'm especially happy to see this paranoia, given the open source client, as well, Zero Zero Linden: oh Zero Linden: prev
Zero Linden: Here - on this one Zero Linden: At this point we "freeze" the Message Template Zero Linden: meaning that we only allow new messages, depricating old messages, or extending messages at the end of the message Zha Ewry: Ahh. So, from there forward, no new binary formats. Excellent Zero Linden: It is required to allow the binary transport to support differeing versions - at least after the freeze point Zero Linden: Well, again, we need to be cautious - so I can't say *no* new ones Zero Linden: There is enough working going on simultaneously in dev Zha Ewry grins. Ok, best effort. Be tough. Zero Linden: that the most we can be realistic about is, only wholey new ones, or extensions - Zero Linden: but no removing data fields, no renumbering, Zero Linden: we call this point Zero Linden: Día de la Liberación Zha Ewry nods. I assume you've been telling people this is comming for a while, so they'll get any last minute changes in now. Xanshin Paz: hehe Zero Linden: Oh yes - I've been preaching this internally for several months Zha Ewry blnks. Will you be sequenced after the new render pipeline? (or in paralell?) Zero Linden: And the funny name is partially to drive home the point that this is an important moment in engineering Zero Linden: after - Zha Ewry nods. Good. Zero Linden: we are aiming for end of March internally - meaning we'll have the first liberated test grid internally by end of quarter Zero Linden: next Zero Linden: next
Zero Linden: So - this is the future of the message system Zero Linden: Thought you all might like to see it! Zha Ewry: Yes, very much Zero Linden: Any questions? Soft Noel: You had the switch sim-side for the old/LLSD crossover per message type. The sim will also be telling the client what transport to use on the fly? Soft Noel: s/client/viewer/ Zero Linden: We'll actually run both transports to the viewer Zero Linden: Some messages will go binary/UDP, other LLSD Zero Linden: Initially all going UDP, but we'll be able to switch, message by message, over to LLSD Zha Ewry: IS that per sim, or across the whole grid? Zero Linden: slide 1
Zero Linden: whole grid Zero Linden: though in the future, once we have the internal management to run different versions of the sim on differnt hosts Zero Linden: we'll be able to do it version by version Zha Ewry: Any chance of doing it per sim for beta testing? Zero Linden: Yes - that is exactly the plan Zero Linden: Once we can run a hetrogenous grid Zero Linden: (the day of liberation!) Zero Linden: We can then do things like bring up some set of regions on the main grid with the next release, in beta Zero Linden: we can then test moving messages over to LLSD on those regions only Zero Linden: The viewer will happily accept messages either way Zero Linden: Notice that we don't talk HTTP directly to the viewer or from it --- Zero Linden: To the viewer we can't due to firewalls - so we use the event queue - where the viewer comes back an polls Zero Linden: this sounds awful BUT Zero Linden: it is a "long poll" meaning that if viewer asks for messages and there aren't any- the sim just stalls replying "none".... Zero Linden: it stalls replying at all... until the moment a message comes Zero Linden: and then it replys with "here" Zero Linden: From viewer to sim, we use a capability - (which itself is HTTP), but in this way, the capability provides the authenication Zha Ewry: Hmm. Any way of having us "kick" the long poll from the client side? (I think I asked this Tuesday) Zero Linden: Meaning? restart it? Zha Ewry: Yes, pretty much Zha Ewry: I'm often aware we've lost the pipe, but the server (sim) doesn't know, of course Zero Linden: Well, the current viewer code doesn't have a UI for that Zero Linden: but you could easily - add a menu item- the code base, LLEventPoll (I think), could easily be made to handle just dropping it's current request on the floor and restarting a new one... Zero Linden: However - are you sure it is just the long poll that has dropped? Zha Ewry: Well, that's a good question. Zero Linden: usually what kills things is that the UDP channel has lost its connection (meaning our higher level abstraction of circuit) Zha Ewry: It *might* be worth a debug option to test/sort that out. Rex Cronon: how can udp channel loose connection? Rex Cronon: is never connectd Rex Cronon: connected* Tree Kyomoon: could add it to the statistics bar Zero Linden: slide 4
Zero Linden: Rex - it isn't the UDP layer,(since UDP has no connection), is the connection layer in our message transport built on top of the UDP system Zero Linden: it relies on a set of pings and other sync. operations Zha Ewry: Ahh. Zha Ewry: Sure. Zero Linden: because the current system tries to send reliable data through UDP! Zha Ewry: So, when one "ghosts" of the grid, we don't really know what's lost synch Zha Ewry: off Zero Linden: Also, realize that since most viewers are behind NATs - if you don't keep the UDP circuit running, you can loose the NAT association, and Rex Cronon: oh, so you tried to build your own version on TCP:) Zero Linden: you may get a new external port on the next send.... which kind of breaks thigns Zero Linden: Rex - cool isn't exactly the term I'd use.... Zha Ewry: Heh. Yes. I've looked at my firewall logs at home Rex Cronon: i didn't say cool Zha Ewry: That makes perfect sense, now. Tree Kyomoon: so what can be done about the ghosting? Zha Ewry: Oh, it's cool, not cool, in the really pretty pair of shoes, sense, but cool, like the dancing bear. It does dance. Zero Linden: oops - you're right - I misscanned your last name and "oh" into "cool"... suppose I need to use a bigger font! Soft Noel: If it's going TCP now, there's a connection that dies when you ghost Zero Linden: Tree - I think that once we have most things moved over to LLSD transports - then we can Zero Linden: work on adding support for re-establishing the UDP link if it goes... Zero Linden: right now, that isn't really possible as you have almost no other link to the grid... Zha Ewry looks thoughtful... Tree Kyomoon: well for now could you make it visually obvious that you are not on the grid? Tree Kyomoon: by logging the user off or whatever? Zha Ewry: Yeah, sure. You're just giong to be miserable, trying to resynch you whole set of things at the moment Zha Ewry: I'd rather see the effort go into getting the new stack finished :-) Zero Linden: Again - the problem with a UDP based transport is that you really don' t know on the sim side what is going on... Rex Cronon: i have see lots of people that crash, and their av is still there Zha Ewry: That's UDP for you. Tree Kyomoon: Ive built some neat stuff while I was crashed...only to find I had built it entirely while offline Zha Ewry: The sim never gets told that you're gone until you re-log Zero Linden: Rex - well, the other problem is, if those AVs have lots of attachments, (or big, in the data sense) then the sim has to wait until those attachments are saved Zero Linden: mind you - I suppose we could add a message to the viewer that says "don't show this ghost" Tree Kyomoon: that would be good Zero Linden: though I prefer the message that says "let other av's draw on this ghost with lipstick" Zha Ewry this the always wrong, but well meaning "don't log on until next thursday" messages Rex Cronon: i had the same problem that tree had Zero Linden: Yes - Zha Ewry: You come back on, the sim says, "Ah, Ave gone" starts the logout... Until that's done, can't let you back in as the asset server is incoherent Zero Linden: a different approach to that problme would be if we had a way to know which scripts had any meaningful persistent state Zero Linden: most scripts in attachements do not - you can reset them at will Rex Cronon: the thing is as long as u can move aroun u are connected, that seems one way to know when u lost connection Zero Linden: in those cases, we could, as long as yo uhaven't edited the attachement, just not save it at all Tree Kyomoon: yes thats what I do now Zha Ewry: Yes, once you can only turn, but not move/standup, etc. You're ghosted Zero Linden: Yes - that is a pretty good test - though again, I've seen UDP drop for 30 seconds and then return.... Tree Kyomoon: yes, Ive seen that too Zero Linden: but admittedly, it is rare Zero Linden: that it comes back, that is Tree Kyomoon: yes Zha Ewry: Hah! Now... Here's a good question? How about having a stronger "Gonig away" message, when you exit from the browser, to start the logout process? Soft Noel: I frequently end up ghosted while flying, and warp back the moment I hit a sim boundary. Tree Kyomoon: lol Rex Cronon: if u have open a browser and load a page, sometimes u r reconnected Zero Linden: Rex - that sounds more like a problem with intermediate routers - like perhaps your NAT box.... Zero Linden: but again,SL's heavy UDP use isn't very friendly to most home NAT boxes Tree Kyomoon: so I would really like to get my question in about messaging to inventorys of objects before we go Tree Kyomoon: if that is ok Rex Cronon: migh be possible, i am on wireless:) Zero Linden: Sure Zero Linden: Shoot Zha Ewry: Did you see my notes on the wiki about blobs? Tree Kyomoon: I need to be able to send messages / scripts to inventoried objects Zero Linden: I did Zha - and have some thoughts back - mostly about the need for additional API to access that data from LSL Tree Kyomoon: will that ever be possilbe? Zero Linden: Tree - no, never Zha Ewry: Yes, I owe you, some api samples, of that sort, if you'd like. Zero Linden: objects in inventory are "freeze dried" - they are templates for objects, not objects themselves Zha Ewry: I've a pretty good image in my head, of what that might look like Soft Noel: Tree: What do they need to do in inventory that they can't check for when pulled out of inventory? Zero Linden: only when "rez'd", the template is instantiated and then you have an object Tree Kyomoon: basically I need a parent/child relationship Tree Kyomoon: and that doesnt seem to exist in "objects" Zero Linden: This does make some semblence of sense - rez'd objects take resources of the simulator: object lists, script state,etc.... Zha Ewry nods. You want a strong distinction, otherwise you end up uable to tell what might need to be live in people's 10,000 object inven tory Tree Kyomoon: and you cant pass much to a rezzed object Zero Linden: Inventories only cost asset server data storage Zero Linden: Tree - well, perhaps that is the bigger issue Zero Linden: most builders have solved it with rather intracate and awkward messaging between objects Rex Cronon: actually u can but is so slow:( Zha Ewry frowns. Yes. I've been noodling over how to handle it in the blobbing cases Zero Linden: but a richer API for rezing an object would be nice Tree Kyomoon: yes, taking days or weeks to do very simple tasks in most oop languages Tree Kyomoon: yes, is that possible in the future? Zha Ewry: Something that says "These are the children I've rezzed recently" (in a list) and (this is my parent) Zha Ewry: With a simpe API to to things similar to LinkedMessageSends to the children/parent <<zero linden crashes and reconnects>> Zero Linden: bother Tree Kyomoon: lol Zero Linden: see - even Lindens! Zha Ewry smiels. SL Zero Linden: So - er, yes Rex Cronon: your client crashed zero? Zero Linden: even my own projects make me frustrated Zero Linden: when working in LSL Tree Kyomoon: so I was just asking about parent/child relatiohships with objects Zero Linden: Rex - yes Zero Linden: Oooooo that one! Zero Linden: Short term - no we aren't going to be doing major changes to the object system like that Tree Kyomoon: aww Zha Ewry: Yes. Got it all the way back to before spilled coffee Rex Cronon: i don't know if u know zero, but there is a wepon that can crash the client, it seem it spams it with textures Zero Linden: longer term (year?) we are trying to get to the point where such things could be done Rex Cronon: weapon* Zero Linden: Well, we are aware of all sorts of bad content - and we know that some of it can cause the client to crash - Zha Ewry: Am I right in guessing that the theme, at least for the first half of '07 is scale not function? Zero Linden: Zha - without question Zero Linden: there is a group working on high priority bug fixes and UI problems Tree Kyomoon: well mabey a libsecondlife project to create parent child object emulation Zha Ewry: So, one question is, how can we prioritize the functions we do need, so that we have a chance of getting those in the pipelines soonest, once you get the Zha Ewry: scaleing under control? Zero Linden: and there is my studio working on larger architectural changes Zero Linden: but everyone else is scaling! Zha Ewry: Heh. You're scaling too, just in the large Zero Linden: Zha - well, actually, the rate of total fall overs has reduced somewhat in the last few months Zha Ewry: Yes, it has. Zero Linden: We have lots of things to do, but seem to be keeping ahead of the tidal wave! Rex Cronon: can there be a pool asking people what functions do they need? Zero Linden: But I will admit we feel like we're in a bit of a race! Zha Ewry: I'd love to know why the worst things hapen 2-8 hours after peak loads Zha Ewry: (You'll notice that the pattern is very consistent. Peak load around 3:00 pm, bad grid by around 8:00 pm) Zha Ewry: Especialy when we hit new total highs Zero Linden: Zha - the interactions of the DBs are difficult to model - but Zero Linden: things start to fall behind at peak - and get progressivly worse Zero Linden: that is , the slave dbs are falling behind in logical time Zero Linden: when they get very out of whack - stuff breaks Zero Linden: right at peak, they are just starting to fall behind, so things might not be so bad... Zha Ewry: What's odd, is that we often seem quite good, from an hour or two after the peak, and then, it falls over. One would *hope* it would catch up Zha Ewry: Instead, its as if something is harvesting objects, or otherwise adding load well after people log off Zero Linden: catching up, once behind, seems very diffficult and sluggish to us - Zero Linden: we have DB folks, internal and external, looking at it, and they poke here and there, but so far no clear solution to that particular problem Tree Kyomoon: bfore the end....just wanted to ask.....sheepishly....cookies-any further thoughts? Zha Ewry: Well, of course, if you used DB/2 in full commercial setup... :-) Zero Linden: so our goal is keep ahead of it by moving things out of the DBs, and splitting DBs. Zero Linden: Sorry, Tree, not at the moment... Zha Ewry: (I think I'm required to say that :-) Tree Kyomoon: ok just call me sqeaky Zha Ewry: But yes, scaling out, to more DBs should help Zero Linden: Okay all - my time is up - got to get back to coding Zha Ewry: Zero, would you like some API thoghts on how to deal with blobbed objects? Zha Ewry: I can add them to the wiki over the next week Zha Ewry: (before next tuesdasy) Zero Linden: Hope all this was interesting. Zha Ewry: Very. Zero Linden: Thank you all for coming Tree Kyomoon: very Zha Ewry: I'll e-mail you the transcript montarily Zero Linden: Yes, Zha - I do read the wiki entries you all post! Rex Cronon: bye Wyn Galbraith smiles, "Thank you Zero." Xanshin Paz: thanks for having us! Zha Ewry: Thanks, as always Zero Linden: great, Zha, appreaciate it Zha Ewry: And the slides were really, really helpful Tree Kyomoon: thnks again!