User:Which Linden/Office Hours/2009 June 18

• [11:05] Morgaine Dinova: I felt a disturbance in the aether, as if a bamboo were materializing
• [11:05] Baldi McMillan: srry
• [11:05] Which Linden: poof!
• [11:06] Morgaine Dinova: Hehe. Hiya Which!
• [11:06] Aimee Trescothick: shoot!
• [11:06] Which Linden: what's up y'all!
• [11:06] Morgaine Dinova: Had a good holiday?
• [11:06] Soph Oh: hi all
• [11:06] Aimee Trescothick: points, that way
• [11:06] Morgaine Dinova: Hiya Soph
• [11:07] Baldi McMillan: Love the avi Which!
• [11:07] Which Linden: Thanks!
• [11:07] Morgaine Dinova: Which avi?
• [11:07] Morgaine Dinova: :P
• [11:07] Baldi McMillan: O.O
• [11:07] Baldi McMillan: Which avi?
• [11:07] Which Linden: Who?
• [11:07] Aimee Trescothick: what?
• [11:07] Baldi McMillan: hmmm
• [11:07] Goldie Dastardly: Second base
• [11:08] Baldi McMillan: this is technicall
• [11:08] Morgaine Dinova: Moreover, why?
• [11:08] Which Linden: So yeah -- technical topic options for tdaoy: I might be interested in discussing doing huge sums in mysql if you are
• [11:08] Which Linden: alternately, transient key-value stores
• [11:08] Baldi McMillan: LOL
• [11:09] Morgaine Dinova: Jeez no, there's nothing special about RDBMSs
• [11:09] Which Linden: You know, one of the things that I really relish the opportunity to do is to tackle a problem and really solve it 100%
• [11:10] Morgaine Dinova: Haha
• [11:10] Baldi McMillan: group chatlag?
• [11:10] Baldi McMillan: O.o
• [11:10] Morgaine Dinova: That doesn't happen in engineering :P
• [11:10] Baldi McMillan:  ;-)
• [11:10] Baldi McMillan: LOL
• [11:10] Baldi McMillan: LOL
• [11:10] Which Linden: Right, too often you end up only having enough time to do something slapdash and make assumptions
• [11:11] Morgaine Dinova: One thing that arose since your last meeting was Sardonyx's blog post about group IM solutions.
• [11:11] Which Linden: Oh, really? Didn't see that
• [11:11] Morgaine Dinova: https://blogs.secondlife.com/community/technology/blog/2009/06/15/improving-the-quality-of-group-chat
• [11:12] Which Lindenri: Ashton: hello all
• [11:12] Baldi McMillan: hello!
• [11:12] Baldi McMillan:  ;-)
• [11:12] Which Lindenri: Ashton: sorry, kinda late entry, had to do something for RL first :(
• [11:13] Morgaine Dinova: Ignoring the face-saving PR that suggests LL didn't know that there was an IM problem .... :-) But the technical bits are more interesting
• [11:13] Morgaine Dinova: Hi Youri :-)
• [11:13] Which Lindenri: Ashton: hi :)
• [11:13] Baldi McMillan: You mean Maggies post? [1]
• [11:13] Which Linden: Yeah that's a nice deep blog post
• [11:14] Which Linden: Definitely starting to get closer to 100% on that stuff
• [11:15] Morgaine Dinova: Yeah, it's a good post, just wish he hadn't *lied* about not knowing .... ;-) And if it's not a plain lie, then clearly there are communication problems inside the lab :-(
• [11:15] Which Linden: He didn't lie
• [11:15] Morgaine Dinova: Communications troubles then.
• [11:15] Which Linden: One thing I've noticed is that group chat bug reporting has been unusually poor
• [11:15] Which Linden: I mean, not to blame
• [11:15] Morgaine Dinova: makes you wonder why people bother with Jiras
• [11:15] Which Lindenri: Ashton: wb sai
• [11:16] Morgaine Dinova: Hi Sai :-)
• [11:16] Which Linden: The best way to be helpful with the JIRAs about group chat is to be highly specific
• [11:16] Morgaine Dinova: The good news is that *in theory* there will be a major improvement in IM as soon as the fixes come through in the release schedule.
• [11:16] Which Linden: Give a time, a region, an exeact report of the error message that was seen and what you were doing when it happened
• [11:17] Morgaine Dinova: Which: don't go there, I recommend. The zillions of people who were highly specific about IM problems in Jira would not appreciate it. Let's just talk about the tech :-)
• [11:17] Baldi McMillan: LOL
• [11:17] Baldi McMillan:  :-)
• [11:18] Baldi McMillan: gotta point there
• [11:18] Which Linden: Maybe I've just been looking at the JIRAs open since 2006
• [11:18] Morgaine Dinova: I'm more interested in the solutions, PR is PR, we're not gonna change the world on that
• [11:19] Saijanai Kuhn: so we're chatting about group chat?
• [11:19] Morgaine Dinova: I think the best thing about that news is the instrumentation that Sard says is now available for IM.
• [11:19] Which Linden: Well I was hoping not to, Sai
• [11:19] Which Linden: Yes, definitely instrumentation will be very helpful
• [11:19] Morgaine Dinova: Sai, not really, but I pasted this -- https://blogs.secondlife.com/community/technology/blog/2009/06/15/improving-the-quality-of-group-chat
• [11:19] Saijanai Kuhn: makes no neverwhichmind to me which
• [11:20] Saijanai Kuhn: makes no neverwhichmind to me mind which never fear
• [11:20] Which Linden: ha ha ha
• [11:20] Mojito Sorbet: Ok, I got one for you
• [11:20] Mojito Sorbet: The place is here. The time is right now
• [11:21] Which Linden: So... no one's interested in transient key-value stores?
• [11:21] Morgaine Dinova: Sure!
• [11:21] Saijanai Kuhn: not sure what those are...
• [11:21] Morgaine Dinova: Especially if it's distributed hash tables :P
• [11:21] Which Lindenri: Ashton: not sure what you mean by that Which, but ill bite
• [11:21] Baldi McMillan: is here to learn ;-)
• [11:21] Aimee Trescothick: Im an audio engineers, transients is a dirty word
• [11:22] Which Linden: So what would be useful for a system such as ours is a key-value store that is very fast and doesn't worry too much about saving data
• [11:22] Aimee Trescothick: erm, I'm also singular not plural
• [11:22] Which Linden: DHTs would be a good implementation for this
• [11:23] Which Linden: It's ok aimee, the royal we is accepted here
• [11:23] Morgaine Dinova: Aimee: you're mono? ;-)
• [11:23] Aimee Trescothick: LOL
• [11:23] Which Linden: A non-distributed transient key-value store is memcached
• [11:23] Morgaine Dinova: Well there's an unmaintained open DHT that is begging for some new hacks .... [2]
• [11:23] Aimee Trescothick: well, stereo on a good day
• [11:24] Which Linden: Another one is Tokyo Tyrant, I think
• [11:24] Morgaine Dinova: Memchached has a very good reputation
• [11:24] Morgaine Dinova: s/h//
• [11:24] Which Linden: Yeah -- only problem is it's not distributed
• [11:24] Aimee Trescothick: woah, that header does weird things to your eyes
• [11:24] Morgaine Dinova: googles for "distributing memcached"
• [11:25] Which Linden: "I'm announcing today that sometime on July 1, 2009, I will take OpenDHT down."
• [11:25] Which Linden: sad
• [11:25] Baldi McMillan: yup
• [11:26] Mojito Sorbet: I have a reproducer for group IM problems. is there an existing JIRA I should append it to?
• [11:26] Which Linden: Mojito: I'll find that out for ya
• [11:26] Morgaine Dinova: But it's only the service that's going down, the code it runs is on another site
• [11:26] Morgaine Dinova: Bamboo :-)))))
• [11:26] Morgaine Dinova: [3]
• [11:26] Which Lindenri: Ashton: i have to run, looks like someone need my help =\
• [11:26] Morgaine Dinova: Now that's gotta appeal ... :)
• [11:26] Which Lindenri: Ashton: thanks for the time here Which
• [11:27] Morgaine Dinova: Seeya Your
• [11:27] Which Linden: Thanks Youri, have a agreat day!
• [11:27] Morgaine Dinova: Seeya Youri
• [11:27] Baldi McMillan: Take care!
• [11:27] Which Linden: So what's up with OpenDHT then? Was it a global DHT that everyone used?
• [11:27] Morgaine Dinova: Yeah, needed a lot of resources.
• [11:27] Which Linden: Man great logo for that Bamboo project
• [11:28] Aimee Trescothick: relative?
• [11:28] Morgaine Dinova: I came across it a month or two ago, then Bruce Perens wrote a Slashdot article about the site disappearing. Apparently the service is used by Evolution or some other package.
• [11:28] Morgaine Dinova: LOL Aimee :-)
• [11:29] Which Linden: hm.... so is it low-latency?
• [11:30] Morgaine Dinova: Can't see any metrics
• [11:30] Which Linden: hm -- so basically what we'd want to use such a thing for is presence
• [11:31] Morgaine Dinova: Found the Slashdot article, it wasn't Evolution, but Adeona -- [4]
• [11:31] Which Linden: when you log in or TP the sim would stick your location in the store keyed off your agent id, and then services that wanted to know where you were could read from it
• [11:31] Which Linden: oh adeona.... I have that installed
• [11:32] Morgaine Dinova: I wonder if your installation is using OpenDHT or if it's moved to something else
• [11:33] Which Linden: Well it could just be silently broken too
• [11:33] Morgaine Dinova: Yeah
• [11:33] Morgaine Dinova: Found the config file? I assume it's not hardwired.
• [11:34] Which Linden: So, in general such a storage system needs to be extremely low-latency because presence checks happen all teh dang time
• [11:35] Morgaine Dinova: Although it happens all the time because the population is huge, it should never be polling, so in that sense presence is an infrequently exercised service, per user.
• [11:35] Which Linden: But it's not so critical that the data is written to disk, because presence is refreshed all the tiem too so if a node dies at worst we lose 5 minutes
• [11:35] Morgaine Dinova: Yeah, program for the working case, not the broken exception
• [11:35] Which Linden: Morgaine: true, though I'd say once per 5 minutes is actually pretty frequent in aggregate
• [11:36] Which Linden: So it'd be great if there was a scaleable tool out there that made the desired tradeoff to optimize for speed
• [11:37] Morgaine Dinova: No Which, you're suggesting polling for presence. That's broken, Presence should be a reliable state switch which only gets activated when the state changes. No polling.
• [11:37] Which Linden: Presence is pushed every 5 minutes per avatar, not read
• [11:37] Morgaine Dinova: Chuck out UDP, and then you don't have the problem.
• [11:37] Latif Khalifa: lol
• [11:37] Which Linden: It's read for every IM sent
• [11:38] Morgaine Dinova: You don't need to push out presence, except when there has been a change
• [11:38] Which Linden: I think it's pushed specifically because there are situations where it can change without successfully pushing (e.g. sim crash)
• [11:38] Morgaine Dinova: "Code for the working path, not the broken one"
• [11:39] Which Linden: True, but in this case the broken situation is pretty bad -- presence would prevent you from logging in
• [11:39] Which Linden: Also missing al your IMs for <insert time period here> would be bad
• [11:40] Morgaine Dinova: You're coding for failure as standard.
• [11:40] Mojito Sorbet: If a sim crashes it will break all TCP connections it has open. The other ends see it break and behave accordingly. Just one of the benfits of chucking UDP.
• [11:40] Morgaine Dinova: As Mojito says
• [11:40] Latif Khalifa: lmao
• [11:41] Mojito Sorbet: TYou want to use technology that works all the time, not just most of the time if the wind is right.
• [11:41] Morgaine Dinova: When there is fallure, you fix the failure using exception mechanisms. You don't code the entire system on the basis of unreliable nodes.
• [11:41] Which Linden: Yeah that's a good point, if a sim crashes its neighbors could notice and somehow update presence for all agents on the carashed sim
• [11:41] Mojito Sorbet: Assuming you keep a READ pening on the TCP channel.
• [11:42] Mojito Sorbet: *pending
• [11:42] Which Linden: However -- that would mean the second sim would have to have a complete list of the agents on the first sim. Keeping that in sync is not a trivial problem
• [11:42] Which Linden: Morgaine: actually I'm not sure I agree with you there, nearly every reliable system I'm aware of codes for failure cases
• [11:42] Baldi McMillan: RL..gootta run...Love to all!
• [11:42] Baldi McMillan:  ;-)
• [11:42] Latif Khalifa: you have to code for failure cases
• [11:43] Mojito Sorbet: Every *reliable* system.
• [11:43] Which Linden: The code is optimized so that success cases run fastest, but that's not the same as saying it's designed so that failures cannot be detected
• [11:43] Morgaine Dinova: Which: every system has to code for failure cases, or you have a broken system. But you don't code the entire system for breakage. You code it for working components, and provide the components with the relaibility mechanisms.
• [11:44] Mojito Sorbet: try-catch, or other similar constructs.
• [11:44] Morgaine Dinova: If you don't do that then you can never have a working system, because all the possible failure modes combine into an explosion of failure states.
• [11:44] Which Linden: Hm, still don't agree. In a distributed system it's very easy to design things such that failures in one place spiral out of control in other places
• [11:44] Morgaine Dinova: And then you have a compleoity problem, so your system becomes massively non-scalable.
• [11:45] Which Linden: We mya just be violently agreeing
• [11:45] Latif Khalifa: lol
• [11:45] Mojito Sorbet: Yes, I agree it is very easy to design a distributed system incorrectly.
• [11:45] Morgaine Dinova: Well that's why you avoid failure propagation at all costs. If failures spiral out of control by affecting other parts, you have a problem. Failure containment is dead important.
• [11:46] Which Linden: In any case I think that writing code so that it assumes that presence is correct if it's fresh and writing other code that tries its hardest to keep it fresh is coding for the success case
• [11:46] Which Linden: The failure case here is simple: the data isn't fresh so you assume the person is offline
• [11:47] Mojito Sorbet: Let me get this straight - there is no single server on the grid that is reponsible for knowing if I am connected?
• [11:48] Which Linden: Ultimately there's kind of a spread of responsibility. The sim that you're logged into is the ultimate authority
• [11:48] Which Linden: The presence servers know, on an advisory basis, what sim you're on
• [11:48] Saijanai Kuhn: since almost everything is passed around by UDP, I guess its impossible to ensure that the entire system is up to date
• [11:48] Latif Khalifa: lol
• [11:49] Mojito Sorbet: If this presence server knew when a sim went down, it could immediately mark as offline all agents that were reported to be there.
• [11:49] Which Linden: In general it's not really possible to assume an entire system of any size is up to date at any particular moment
• [11:49] Morgaine Dinova: Not the way I'd do it. If presence is a reflection of state and only changes on state change, then its "age" is rather meaningless. If I only log out once a year, I expect my presence state not to change any more often than that. An 11-month old presence is still perfectly valid.
• [11:49] Which Linden: Mojito: well assuming that it had a mapping of sim-> agent, which could be expensive to maintain
• [11:49] Latif Khalifa: morg, presence is not simpley online/offline
• [11:49] Latif Khalifa: its also where you are
• [11:50] Which Linden: True
• [11:50] Latif Khalifa: so i decide to send you an IM
• [11:50] Latif Khalifa: system needs to resolve to which sim to send that message
• [11:50] Mojito Sorbet: No, when a sim sees me arrive it tells the Presence Server "Mojito is here", and that gets recorded along with the name of the sim that rported it.
• [11:50] Which Linden: That's how it work
• [11:50] Morgaine Dinova: Sure, where you are is state change too, but you don't need to poll for it. When you change location, you change your state info./
• [11:50] Which Linden: *works
• [11:50] Mojito Sorbet: If my sim goes down, Presence Server does a quesry on all entries reported form that sim
• [11:51] Mojito Sorbet: Does a query on its database that is
• [11:51] Which Linden: How do you guarantee that the Presence server knows the sim is down?
• [11:51] Morgaine Dinova: A s Mojito says :-)
• [11:51] Mojito Sorbet: BY USING TCP
• [11:51] Which Linden: What if the host loses its NIC?
• [11:51] Morgaine Dinova: Then buy it another one :P
• [11:51] Latif Khalifa: Mojito, you you think presence server should maintain 30k tcp conections to the sims?
• [11:51] Latif Khalifa: lol
• [11:51] Mojito Sorbet: No, you build them in a tree
• [11:52] Mojito Sorbet: Say at the rack level
• [11:52] Which Linden: TCP timeouts are 180 seconds ... polling might be cheaper
• [11:53] Mojito Sorbet: If you keep a read pending on a TCP channel, you find out pretty fast.
• [11:53] Latif Khalifa: and then you have the edge cases, i tp from sim a to sim b, sim a says to the presence system latif left, which sends me an im, presence says latif is not in any sim, im goes to my email, and then sim b reports the presence of me arriving there
• [11:53] Morgaine Dinova: Were's talking about LANs. If the data isn't reaching you in less than a millisecond, you already have an issue. You sure don't wait for TCP timeout, you know long before.
• [11:54] Latif Khalifa: morg, not true, presence servers and sims can be continents away, most certanly in different data centers
• [11:54] Mojito Sorbet: Put one watchdog process per rack that locally keeps an eye on the sims in that rack. Then reports to a central place.
• [11:54] Which Linden: Yeah Latif is right -- there's a lot of race conditions
• [11:54] Mojito Sorbet: That is why it has to be hierarchial
• [11:54] Mojito Sorbet: Rack, Room, Building.
• [11:54] Morgaine Dinova: Latif: you're right, post interop, the rules change. But I don't think LL is concerned about that right now :-)
• [11:55] Latif Khalifa: Morg, its not post interop its right now, presence system and sim hosts can be in different data centers across the country
• [11:55] Mojito Sorbet: 30k servers throwing UDP packets at each other is too complicated.
• [11:55] Which Linden: Another one is : I TP to a new region and crash it immediately -- the presence system might show that I'm logged into the old region stilll
• [11:55] Mojito Sorbet: The old server will report you leaving
• [11:55] Which Linden: And also we have to assume that the presence server never has any failures of its own
• [11:56] Mojito Sorbet: Redundant, High Availability
• [11:56] Morgaine Dinova: Latif: so you set up the timeouts to suit your particular case. Inside a single datacenter, it's still sub-millisecond RTT.
• [11:56] Which Linden: Mojito: but then you have the other race condition which is that the old server reports you're leaving but it takes a while to upload your assets so you spend a lot of tiem "in limbo" -- to whom to IMs go then?
• [11:56] Mojito Sorbet: The handoff has to have multiple states
• [11:57] Mojito Sorbet: There is a period while to destination server is loading your assests, BEFORE the leaving server can really complete the handoff
• [11:57] Mojito Sorbet: You are indeed in limbo duuring that period
• [11:57] Mojito Sorbet: So the IMs go to the old server
• [11:57] Mojito Sorbet: Once handoff is coimplete, the old server passes them over to the new server
• [11:58] Which Linden: What if the sending of "pending leaving Region A" message gets to the presence server after the "Arriving at Region B" message?
• [11:58] Morgaine Dinova: AFK a sec while I log into OSgrid
• [11:58] Mojito Sorbet: There is thins thing in data processing called the "two phase commit".
• [11:58] Which Linden: Two phase commits are extremely time-intensive
• [11:59] Which Linden: Like, really laggy
• [11:59] Mojito Sorbet: As far as the Prsence Server is concerned, it only needs to know when the handoff is complete.
• [11:59] Mojito Sorbet: TPing is already laggy, if the new server cant fetch the assets in time
• [12:00] Mojito Sorbet: Until the destination sim has enough info about you, it is not in any shape to take responsibility for your messages
• [12:00] Mojito Sorbet: A->B Prepare for Mojito
• [12:00] Mojito Sorbet: B->A Ok, Mojito is ready here
• [12:00] Which Linden: Basically though, the system is such that it isn't that big a deal if one agent's presence is wrong for a short time
• [12:01] Mojito Sorbet: A->B Ok, you have her
• [12:01] Which Linden: That's basically what happens now
• [12:01] Which Linden: I think it's called "ushering"
• [12:01] Aimee Trescothick: "unable to start a chat session with 'Friends of Raglan Shire'. The session no longer exists"
• [12:01] Which Linden: Oh right I was finding that jira
• [12:02] Morgaine Dinova: reads back
• [12:02] Which Linden: Aimee: SVC-1509
• [12:02] Mojito Sorbet: I get "Unable to send your message to the chat session with [RECIPIENT]. Error making request, please try later"
• [12:02] Aimee Trescothick:  :)
• [12:02] Mojito Sorbet: I can get that message right now, any time you want
• [12:03] Which Linden: Mojito: I think same JIRA
• [12:03] Which Linden: which group?
• [12:03] Aimee Trescothick: Friends of Raglan Shire
• [12:03] Morgaine Dinova: Aimee: yeah, that's happening a lot now on incoming IMs
• [12:03] Mojito Sorbet: Hippo Technologies User Group, which has 1390 members
• [12:03] Aimee Trescothick: happens all the time
• [12:03] Mojito Sorbet: This happens when I try to SEND a message to that group
• [12:04] Mojito Sorbet: That is normally a very chatty groupl. I have not heard form it in a few days
• [12:04] Thunderclap Morgridge: the mentors group too
• [12:04] Which Linden: ok, cool, so errors on send are in a different category than errors on receipt
• [12:04] Mojito Sorbet: Proabbly
• [12:04] Mojito Sorbet: Or different symptoms of a single cause
• [12:05] Aimee Trescothick: yeah, the one i said happens on every receive from that group, unless I pre-empt it and open the group myself
• [12:05] Morgaine Dinova: Rememeber the group listener is stopped if you click off the IM tab. It's a totally terrible feature.
• [12:05] Which Linden: better for debugging to treat them as separate
• [12:05] Latif Khalifa: in your opinion morg
• [12:05] Mojito Sorbet: Yes, SVC-1509 is the one I get, the variant without the name of the group in it.
• [12:06] Mojito Sorbet: Very reproducible
• [12:06] Which Linden: there's two error messages, then, possibly two bugs there....gross
• [12:06] Morgaine Dinova: Latif: no Latif, not in my opinion. It's objectively bad because there is no way of turning all your 25 group listener back on short of clicking all the tabs or relogging. So it's crap by design for users. Was only done to reduce the IM load for LL.
• [12:07] Mojito Sorbet: Sometimes it says the name, other times [RECIPIENT]. I get the latter
• [12:07] Latif Khalifa: Morg, its objectively bad because you don't like it, its the only critearia you accept
• [12:07] Which Linden: Do not assume malicious intent where design cruft is entirely a sufficent explanation
• [12:07] Morgaine Dinova: Latif: has nothing to do with liking or disliking. Having to click all your tabs doesn't scale. You gonna click 1000 tabs when you have 100 groups?
• [12:07] Which Linden: ok, I should probably run
• [12:08] Morgaine Dinova: You really don't have your head screwed on a lot of the time Latif
• [12:08] Latif Khalifa: morg, still you personal preference any way you want to twist it
• [12:08] Which Linden: Whoa! Let's keep it nice
• [12:08] Thunderclap Morgridge: hey which you were on the master office hours list
• [12:08] Thunderclap Morgridge: this is cool but I stumbled hefre
• [12:08] Which Linden: Hi Thunderclap.
• [12:08] Latif Khalifa: Which, I'm used to get insults from her, doesn't bother me :)
• [12:08] Which Linden: Ha ha ha
• [12:08] Morgaine Dinova: I'm no diplomat. But someone who reckons that clicking 1000 tabs is reasonable is clearly not rational.
• [12:08] Thunderclap Morgridge: and why are you a bamboo plant if you dont mind me asking
• [12:09] Which Linden: For stylistic reasons!
• [12:09] Latif Khalifa: hahaha
• [12:09] Thunderclap Morgridge: awesome
• [12:09] Thunderclap Morgridge: you have a bear or a miniature plant bear
• [12:09] Which Linden: I hope to see you again next time!
• [12:09] Which Linden: I'll hook you up with a bear
• [12:09] Latif Khalifa: i hope space server gets better haha
• [12:09] Morgaine Dinova: See you next time Which :-)
• [12:09] Thunderclap Morgridge: thanks
• [12:09] Latif Khalifa: take care which :)
• [12:09] Aimee Trescothick: bye :)
• [12:09] Which Linden: Thanks everybody!