User:Which Linden/Office Hours/2009 May 14

• [11:03] Which Linden: Good morning
• [11:03] Raz Welles: 'soup :3
• [11:04] Nexii Malthus: lol... a plant pot
• [11:04] Lodi McMillan: nice avi, Which
• [11:04] Aimee Trescothick: hi Which :)
• [11:04] Raz Welles: Hihi Which :)
• [11:04] Which Linden: Hi Aimee!
• [11:04] Morgaine Dinova: Hiya Which :-)
• [11:04] Which Linden: Good to see you, Morgaine
• [11:04] Simargl Talaj: cryptic coloration
• [11:05] BackHome Soon: I'm impressed
• [11:05] Nexii Malthus: god damn, why are scripts off, you broke my avatar D:
• [11:05] Which Linden: Hah, I'm paranoid
• [11:05] Nexii Malthus: avatar breaker ;<
• [11:06] Raz Welles: aw nuts sound driver broke again, brb
• [11:06] Aimee Trescothick: not surprising considering the neighbours :D
• [11:06] Which Linden: hah
• [11:06] Nexii Malthus: \o/ there we go
• [11:06] Nexii Malthus: perhaps the sim is just lagging...
• [11:06] Which Linden: A script error?
• [11:06] Nexii Malthus: looks like it, 0.9 - 0.8
• [11:06] Nexii Malthus: time dilation
• [11:07] Lodi McMillan: that avi is so Tardis-tic, Which
• [11:07] Nexii Malthus: I have never seen sims that were 1.00 time dilation
• [11:07] Which Linden: Thanks!
• [11:07] Lodi McMillan: blends right in
• [11:07] Which Linden: So.... on to the topic, shall we?
• [11:07] Which Linden: Last time I was giving a general overview of the four different types of chat in SL (yes there's 4 different systems)
• [11:08] BackHome Soon: you know that's my hot topic :-)
• [11:08] Which Linden: https://wiki.secondlife.com/wiki/User:Which_Linden/Office_Hours/2009_May_7 if you want to catch up
• [11:08] Which Linden: heh
• [11:08] Morgaine Dinova: Oh cool. Did you find out about any others?
• [11:08] Nexii Malthus: P2P IM.. Local Chat...Group Chat...Uh Coalesce* Chat?
• [11:08] Which Linden: Group notices is the last one
• [11:09] Simargl Talaj: Will this be a lesson? I am not looking for a general office hour free-for-all discussion. I am looking for an opprtunity to learn how SL is "put together" and to understand what causes so many bugs.
• [11:09] Which Linden: Yep
• [11:09] Which Linden: So here in a nutshell is how group notices work
• [11:09] Which Linden: They are, as I guessed last time, multi-P2P IM
• [11:10] Morgaine Dinova: Simargl: Which's OH is always technical
• [11:11] Which Linden: When you send a group notice, the simulator you're sitting on looks up the members of the group and sits in a big loop delivering an IM over the normal 1:1 channels to it
• [11:11] Which Linden: *to every member individually
• [11:11] Nexii Malthus: Ouch
• [11:11] Morgaine Dinova: Define "looks up"
• [11:12] Simargl Talaj: hhm, not a central location to which each individual viewer will refer.
• [11:12] Which Linden: I think there's just a service that allows it to get back a big list of group members from the database
• [11:12] lufpleh Obstreperous: would it be more effective to send group notices to large groups from very quiet sims
• [11:12] Which Linden: And, of course, it also has to look up presence for each agent to know whether to send and offline email or directly deliver to the sim
• [11:12] Simargl Talaj: INteresting. SO, the region in which you are sitting...that region's server is the one that sends our group IMs.
• [11:13] Which Linden: lufpleh : not necessarily; each time it sends a 1:1 IM, it involves the dataserver on the simhost, so ideally you'd want that to be not-busy
• [11:13] Simargl Talaj: simhost is a server runing 4 regions, yes?
• [11:13] Morgaine Dinova: Which: the worst interpretation of "looks up" is "on each message, queries D/B for group membership" ... do you mean that?
• [11:14] Which Linden: Simargl: yep
• [11:14] Simargl Talaj: Group IMs to large groups fail more often than group IMs to small groups?
• [11:14] Simargl Talaj: If the message can't get to all, it does not get to anyone?
• [11:14] Simargl Talaj: so it just "fails"?
• [11:15] Morgaine Dinova: Simargl: no, the failures are per user, not all or nothing.
• [11:16] Which Linden: Morgaine: it's possible that it looks up the entire membership each time (though I see that it does so in a way that would err on the side of taking a long time rather than crushing the db)
• [11:17] Which Linden: If an individual user doesn't get a notice it can probably be chalked up to one of two things: UDP *shakes fist*, or the dataserver being bad
• [11:17] Which Linden: Presumably 1:1 IM would suffer from the same problems if it had the same usage patterns
• [11:18] Simargl Talaj: Okay. I am sitting in a moderately busy region on a moderately busy simhost. I send a group IM asking "Hey guys is it just me or is all of SL laggy today?" and I get an error message "COuld not send group IM blah blah lbah". The IM message never appears on the group IIM window, as far as I can see. AAre you telling me that some people might actually have gotten it?
• [11:18] Which Linden: But "sending thousands of individual IMs from one host" is not generally the way it works when it's humans driving
• [11:18] Which Linden: Simargl: group IM is not the same thing as group notices
• [11:18] Lodi McMillan: Notices, Simargi, not IMs
• [11:18] Which Linden: Totally different system
• [11:18] Simargl Talaj: oops
• [11:18] Simargl Talaj: sorry, big context error :)
• [11:18] Lodi McMillan: would this also be the same path that "friend conf IMs" take, Which?
• [11:18] Which Linden: s'ok, correcting that misconception is the point of this exercise!
• [11:19] Which Linden: Lodi: group IM and friend conferences use the same infrastructure
• [11:19] Lodi McMillan: kk
• [11:19] Raz Welles: I got a question to queu up (related), why is it that I'll log in and get snippets of conversations I should have gotten (but never did) from a previous session? I've missed a lot of IM replies that way and just assumed the person went afk >_>;
• [11:19] Morgaine Dinova: Which: well it would imply that there is no exploder being built on entry/exit from a group. Instead, group management is done per-message, making it extremely non-scalable for active groups since for them message rate is vastly >> group entry/exit rate.
• [11:20] Which Linden: So.....that's how group notices are different from [group IM, 1:1 IM, local chat]
• [11:20] Which Linden: Morgaine: true
• [11:21] Which Linden: Though I suspect that for large groups entry/exit rate is higher than one would think; I haven't actually gotten any numbers on this
• [11:21] BackHome Soon: so, each of us should rez a cone, hollow it, affix a handle and use it as a megaphone? ;-)
• [11:21] Which Linden: Ha ha ha
• [11:21] Morgaine Dinova: Yes, the balance tip when a group is large but inactive, like the fashion groups.
• [11:21] BackHome Soon: tips which way, Morgaine?
• [11:22] Which Linden: Hard to design a system that's fast in both cases -- almost you'd need a heuristic to shift groups from "high churn but few messages" to "low churn but high messages" buckets
• [11:22] Morgaine Dinova: 10k users in Fashion Consolidated, but no user-posts, only announcements, so there the membership entry/exit from the group explosion exceeds the message rate.
• [11:23] Morgaine Dinova: Yep
• [11:24] Which Linden: So the last chat system that I haven't yet given an architectural overview on is local chat
• [11:24] Which Linden: The type we're using right now
• [11:24] Which Linden: (hi)
• [11:24] Imaze Rhiano: would be nice to have different news delivery method for groups like fashion group - maybe "news channel/news paper" subscription - instead of groups
• [11:24] Which Linden: Imaze: yeah totally, it may be worth devising two completely separate modes of communication for two different use cases
• [11:25] Which Linden: So local chat
• [11:25] Simargl Talaj: Some groups operate mostly offworld on blogs and non-:: message boards.
• [11:25] Simargl Talaj: non-LL
• [11:25] Which Linden: The way it works is, your viewer sends a message to the simulator, and the simulator does some sort of radius calculation based on the "volume" of your message
• [11:26] Which Linden: It has a way of grabbing every agent within a particular range since it has that info in internal data structures
• [11:26] Which Linden: So it just loops over each agent that's "in range" and sends their viewer a packet, directly, that contains the chat
• [11:27] Simargl Talaj: then tailors it depending on whether it is regular chat or whipser or shout.
• [11:27] Morgaine Dinova: I don't think they require different mechanisms: they collapse into a single one if the "send message rarely" use case is implemented as "send message and build exploder list but only if it hasn't changed since last message to group".
• [11:28] Simargl Talaj: Yesterday chat filed for me the way group IMs sometimes do. IT was bizarre. That hasn't happened to me in SL for maybe two years or more.
• [11:28] Simargl Talaj: failed
• [11:28] Morgaine Dinova: LOL, s/hasn't changed/has changed/
• [11:28] Simargl Talaj: has changed, yeah
• [11:28] Morgaine Dinova: Worst kind of typo, makes the opposite assertion of what was intended :P
• [11:29] Simargl Talaj: How can regular chat possibly fail?
• [11:29] Which Linden: If the range overlaps with a neighboring simulator, then it forwards the message to the neighboring sim directly and that sim does the range calculation and forwarding stuff
• [11:29] Simargl Talaj: how can it fail for one person and not for others?
• [11:29] Which Linden: Simargl: I believe that it only fails when the UDP message system fails
• [11:30] Which Linden: Which is not as rare as you'd think; that stuff is rickety
• [11:30] Simargl Talaj: WOuldn't that be accompanied by massive packet loss?
• [11:30] Which Linden: Well you'd be losing at least one packet yes
• [11:30] lufpleh Obstreperous: likes when his own chat lags on his own screen!
• [11:30] Aimee Trescothick: checked under your desk?
• [11:30] Which Linden: The token fell out!
• [11:31] BackHome Soon: i HATE when that happens
• [11:31] BackHome Soon: and you're dating yourself if you know what it means
• [11:31] Raz Welles: I.. kind of remember token ring networks.. :P
• [11:31] Simargl Talaj: looks between back of the desk and the wall. FInds old fragments of conversation, dust bunnies, and a half- empty package of Cheetohs from 2007.
• [11:31] Which Linden: I have heard that the UDP packet system can behave poorly under CPU stress, and can fall behind in weird ways
• [11:33] Which Linden: I think it would be correct to move local chat to the TCP event queue but I don't know if that would introduce new issues or not
• [11:33] Which Linden: It would be a project in itself to figure that out
• [11:33] Morgaine Dinova: The sim really shouldn't be involved with vicinity chat messaging, as that reduces its ability to scale. Each person defines a "vicinity chat group" containing all the people within earshot, so it should be the client that updates a chat server with additions/removals from that virtual group. Nothing to do with sim.
• [11:34] Aimee Trescothick: hmm, but if it's up to the client to determine who's in chat range you have to be careful about gaming it
• [11:34] Nexii Malthus: what about scripts then
• [11:34] Simargl Talaj: That would make the success of chat very dependent upon each perosn's computer power?
• [11:34] Nexii Malthus: high volume script messages..that would be nasty
• [11:34] Morgaine Dinova: Which: don't think so. Message libaration was already done. My understanding of what Zero said is that any particular thing being sent over UDP could easily be switched to use TCP.
• [11:34] Which Linden: Morgaine: that seems like it might be subject to inconsistent states; like, I can't hear you but you can hear me because you're on my list but I'm not on yours
• [11:34] Aimee Trescothick: could eaves drop by telling the server someones in chat range when they're not
• [11:34] Simargl Talaj: And people could tweak their own client to broadcast everything they say over 20 sims or something?
• [11:35] Aimee Trescothick: hmm, maybe I'm think from the wrong end there
• [11:35] Which Linden: Anythign can be easily switched to TCP, but one thing I've learned in my time here is that unintended consequences can arise
• [11:35] Simargl Talaj:  :)
• [11:35] Morgaine Dinova: Which that's correct operation: I may be training my ear to listen to something specific, while someone else in my earshot may want to hear only the people closest to her.
• [11:35] Morgaine Dinova: Vicinity hearing is highly selective
• [11:36] Morgaine Dinova: And per-user choice
• [11:36] Morgaine Dinova: And the key to crowd scalability
• [11:36] Morgaine Dinova: Definitely not symmetric
• [11:37] Morgaine Dinova: That's why each person should have her own chat capture list.
• [11:37] Aimee Trescothick: but then it leaves you uncertain all the time whether someone can hear you
• [11:37] Nexii Malthus: I wonder how the client would be if it had an in-built IRC client
• [11:37] Morgaine Dinova: Correct Aimee, as in RL
• [11:37] Nexii Malthus: I think that'd be cool
• [11:38] Morgaine Dinova: Of course, chat originators have control too: they may be whispering.
• [11:38] Aimee Trescothick: hmm, that would effectively lead to more lost conversation fragments though, people don't expect it to work the same as in RL
• [11:38] Morgaine Dinova: I disagree --- people are 100% used to the idea of needing to be in earshot to hear stuff.
• [11:39] Aimee Trescothick: generally in RL you're often aware someone is saying something although you miss what and can ask them to repeat
• [11:39] Which Linden: Right -- it seems as though server control has kind of won the implementation battle for most purposes (bittorrent notwithstanding)
• [11:39] Nexii Malthus: well thats just parsing errors
• [11:39] Aimee Trescothick: whereas here there are a lot less extra cues to go by
• [11:40] Simargl Talaj: I like SL's servers to do as much heavy lifting as possible because otherwise only people with very powerful computers will have the same experiences in SL and people with weaker systems will be left out even more than they already are. And variablilty in *how* different computers work, too, seems to affect things.
• [11:40] Morgaine Dinova: Simargl: the server has to do as LITTLE as possible, not as much as possible, or else the service scales badly.
• [11:40] Nexii Malthus: Agreed, im on a weak system myself
• [11:41] Simargl Talaj: Morggaine, yes, I see it has to be a balancing act.
• [11:42] Aimee Trescothick: hmm, maybe we should just implement group chat as chinese whispers,that would be more fun
• [11:42] Aimee Trescothick: local chat I mean
• [11:42] Morgaine Dinova: You have to remember that sims are just poor little PCs, yet need to support populations of people. If there are N people in a sim, that inherently means they can only dedicate 1/N of their power to you --- so can't expect them to do heavy lifting for you.l
• [11:42] Which Linden: And then do a chinese fire drill when we shout
• [11:42] Aimee Trescothick: lol
• [11:42] BackHome Soon: Which, how about the other "chat-type" notices -- such as friend's status -- same plumbing as local/group chat?
• [11:42] Imaze Rhiano: how about translating all group chat instantly to Klingon?
• [11:43] Which Linden: BackHome: I don't know how online/offline notices are routed
• [11:43] Which Linden: I assume they work similarly to group notices, but haven't checked
• [11:43] Nexii Malthus: Hmmm
• [11:44] Nexii Malthus: opens llviewermessage.cpp
• [11:44] BackHome Soon: just curious because there are separate categories of them - -and they ALWAYS seem to be delivered
• [11:45] Which Linden: Really? I don't think I'd notice if a friend logged off somewhere off in the world and I didn't get a notice about it
• [11:45] BackHome Soon: I've never seen a friend logoff in front me me and NOT seen the notice
• [11:46] BackHome Soon: hard to prove a negative ;-)
• [11:46] Which Linden: Right but in front of you it's not really an IM, the simulator just short-circuits that
• [11:46] Morgaine Dinova: This will be where scripted testing will come into its own. We can't keep track of such things, test scripts can.
• [11:46] BackHome Soon: kk, that's why I was asking about it's plumbing
• [11:46] Which Linden: I mean it is an IM, but it's all in-memory so it's more reliable
• [11:47] Which Linden: So I guess that implies that if you send a group notice and a lot of the group members are in the same region as you, they should all get it
• [11:47] Which Linden: If that turns out not to be the case, then I guess I'm wrong about short-circuitign
• [11:48] Which Linden: Or .... it could also mean that the UDP message system couldn't reach everyone's viewers
• [11:48] Which Linden: So many failure points, so little time :-)
• [11:48] Morgaine Dinova: I guess it just "evolved" into all these special cases, instead of using a common messaging infrastructure?
• [11:49] Morgaine Dinova: I'm curious why Zero hasn't switched IM delivery to clients over to TCP, for the simple purpose of eliminating one variable.
• [11:50] Morgaine Dinova: Seems a no-brainer.
• [11:50] Which Linden: Changes that are simple to do in code are not necessarily easy to do procedurally
• [11:50] BackHome Soon: brb, sorry
• [11:51] Morgaine Dinova: After all, current sims can't hit TCP scalability limits on per-system streams, so no worries there. And that's exactly what Message Liberation was for, to give you that flexibility.
• [11:52] Which Linden: Yes, things definitely evolved organically over time; when all this code was written there really weren't many general-purpose utilities for doing messaging; still aren't really if you want to solve local chat with a separate server
• [11:52] BackHome Soon: back
• [11:52] Simargl Talaj: Must go. Thank you.
• [11:52] Which Linden: Morgaine: yeah I agree that someone should just do the switch
• [11:52] Which Linden: Thank you Simargl, nice to chat
• [11:54] Which Linden: So.... I guess I laid all those chat systems out today -- other questions?
• [11:54] Morgaine Dinova: Which: it can be done on a per-sim basis, so pretty safe. The main problem is viewers ... they'd have to be able to use both mechanisms, so would require across-the-board updates far ahead of the plan.
• [11:55] BackHome Soon: offtipic, but will you sell me your avi? ;-)
• [11:55] Imaze Rhiano: what are you going to do for them? :P
• [11:56] Which Linden: Morgaine: well I thought Liberation meant that viewers didn't have to be changed to do the TCP thing
• [11:56] Which Linden: BackHome: I'm afraid I don't sell it
• [11:57] BackHome Soon: thought as much, but was worth a try :-)
• [11:57] Morgaine Dinova: Which: oh cool, if that path is already built in!
• [11:57] Morgaine Dinova: I'll try to remember to ask Zero about that. Been a long time since we mentioned Message Liberation
• [11:58] Which Linden: Yeah even though I worked on that project I've totally forgotten the details
• [11:58] Morgaine Dinova: Or even better, maybe I could look inside Opensim and see if it can switch.
• [11:58] Which Linden: And broad architectural goals apparently
• [11:58] Morgaine Dinova: Hehe
• [11:59] Which Linden: Heh, well, I should GTFO; thanks all y'all for being here
• [11:59] Imaze Rhiano: bye
• [11:59] Aimee Trescothick: bye Which :)
• [11:59] BackHome Soon: enjoyed and appreciate the info, Which. Look forward to more
• [11:59] Lodi McMillan: thank you, Which
• [11:59] Which Linden: Take it easy!