User:Which Linden/Office Hours/2009 Mar 19

[11:05] Which Linden: hey there!
[11:05] Morgaine Dinova: Hiya Which!
[11:06] Saijanai Kuhn: hey which
[11:06] Which Linden: Sorry I'm late, was frantically trying to get the message queue notes out there on the wiki
[11:06] Which Linden: and here they are: https://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
[11:06] Saijanai Kuhn: heh
[11:07] Morgaine Dinova: Woot!
[11:08] Which Linden: Hi Geo!
[11:09] Geo Meek: wow the weeds are getting high
[11:09] Geo Meek: LL is trying to say something?
[11:09] Which Linden: Plants have feelings too!
[11:09] Geo Meek: roll and smoke?>
[11:09] gift Asp: hey all'
[11:09] Which Linden: Hi there Gift
[11:10] Which Linden: welcome to my office hours
[11:10] gift Asp: thanks :D
[11:10] Geo Meek: voice?
[11:10] Which Linden: we generally don't do voice here -- no transcript
[11:10] Geo Meek: try both
[11:10] Geo Meek: just a idea
[11:11] Which Linden: You know, it's interesting, I'm not aware of any Linden office hours that use voice
[11:11] Which Linden: Do you? It seems natural for the triage type ones
[11:11] Geo Meek: Torley uses voice
[11:12] Saijanai Kuhn: is a f k for a bit
[11:12] Morgaine Dinova: Interesting Which, I read the Summary and the Scaling bit
[11:12] Which Linden: it goes downhill from there
[11:12] Which Linden: :-)
[11:12] Morgaine Dinova: lol
[11:12] Geo Meek: really?
[11:12] Geo Meek: how?
[11:14] Which Linden: Geo: check this page out: https://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
[11:14] Geo Meek: well do
[11:15] Geo Meek: is it just BS ?
[11:15] Which Linden: We had an interesting realization when working on this. And no, it was not horse puckey :-)
[11:15] Morgaine Dinova: I'm surprised that nobody scaled in the direction you need Which. That's pretty odd, huh?
[11:16] Which Linden: I think I know why
[11:16] Which Linden: The way that anything gets done in one of these tools is by consulting the routing state
[11:16] Geo Meek: may i ask who wrote it
[11:16] Which Linden: Geo: I didi
[11:16] Geo Meek: k
[11:16] Which Linden: along with Kartic, one of my teammates; we each wrote about half
[11:16] Geo Meek: text is way to slow and i dont read well
[11:17] Which Linden: So, the routing state is basically just a pair of mappings
[11:17] Geo Meek: and text is a big reson for lag
[11:17] Morgaine Dinova: Text allows you to think.
[11:17] Which Linden: I'm sorry, Geo, I don't think I'm going to swithc to voice just yet
[11:18] Geo Meek: thats ok the smart people well find a way
[11:18] Which Linden: Love will find a way
[11:18] Which Linden: ok, back to the realization about why no one scales along this axis
[11:18] Which Linden: The forward mapping in the routing state maps routing keys to queues
[11:19] Which Linden: I.e. it anwsers the question "I have this message for Bob, what queues are listening for Bob's messages and where are they?"
[11:19] Which Linden: The reverse mapping maps queues to routing keys.
[11:19] Morgaine Dinova: Without exposing that info. The key is opaque
[11:20] Which Linden: That answers the question "I has this queue, what am I listening for, and how do I change that?"
[11:20] Which Linden: The reverse mapping is less important
[11:21] Morgaine Dinova: Have I read correctly that you *expect* the number of possible groups per client top always be small?
[11:22] Which Linden: Morgaine: heh, well that's stepping in it a bit with regard to the cap on group memberships isn't it
[11:22] Which Linden: But it's important to note that this is one technical benefit that accrues from having a cap
[11:23] Which Linden: The way these queue systems work is when they get an incoming message they look up the destination queues in the forward mapping and deliver to them -- they're optimized for this to be very fast, and thus the mapping is always in memory.
[11:23] Morgaine Dinova: Well this is why I mentioned it. Since the destination leaves of the routing tree are kinda unlimited (millions), the idea that group membership would always be a small set wouldn't really fit in that.
[11:24] Which Linden: lemme finish this thought here, one sec I'll get back to you
[11:24] Morgaine Dinova: k
[11:24] Which Linden: The realization we had was that the forward mapping was an ideal candidate for a distributed key-value store; this is ironic because that is another category we were evaluating at the same time
[11:25] Which Linden: another technology categroy
[11:25] Which Linden: So, if you had a perfectly scalable message queue system, you'd also have to have a scalable key-value store
[11:26] Which Linden: And that's why none of those things have it -- because that's another really difficult problem that's not solved very well by anyone
[11:26] Geo Meek: welcome to sl
[11:26] Morgaine Dinova: That's an interesting insight. (Still thinking about it)
[11:27] Morgaine Dinova: Yeah, I think it makes sense, even if your routing paths are multilevel
[11:27] Morgaine Dinova: Ie. one router leads to another
[11:28] Which Linden: Yeah, even if you have federation that knows how to delegate routing decisions you've just sort of pushed the problem out by a bit
[11:29] Which Linden: It only really works if every node has access to the global picture
[11:29] Morgaine Dinova: Which isn't scalable
[11:30] Morgaine Dinova: My last line is ambiguous :P
[11:32] Which Linden: Wait'll I show you my clone army!
[11:32] Which Linden: Then we'll see who's scalable!
[11:32] Morgaine Dinova: Hehe
[11:32] Which Linden: OK, so, group limits
[11:33] Which Linden: What do you mean by "Since the destination leaves of the routing tree are kinda unlimited (millions)"
[11:34] Morgaine Dinova: Total number of "end" endpoints is given by population
[11:34] Morgaine Dinova: So is total number of "start" endpoints, but your max # groups determines the first fanout #
[11:35] Which Linden: Start endpoints meaning groups?
[11:35] Morgaine Dinova: No hang on, the max # groups determintes the "fan-in" to each destination endpoint
[11:35] Morgaine Dinova: Cancel 2nd line @ 11:34
[11:35] Which Linden: kk :-)
[11:36] Which Linden: In the worst case each client has to talk to a server-per-group
[11:36] Morgaine Dinova: Both the incoming and outgoing endpoint <= size of population
[11:37] Which Linden: assuming that the agent hosts are holding the connections open to the viewers, and that we have only a handful of message queue servers, then yeah, it doesn't blow up that bad
[11:37] Morgaine Dinova: The # number of groups just says how many queues are funnelled into a single endpoint, nothing else.
[11:37] Which Linden: and by # number of groups I assume you mean number of groups per Resident
[11:37] Morgaine Dinova: Yes
[11:38] Morgaine Dinova: Since the resident is the endpoint.
[11:38] Which Linden: Right, yeah, so, it's clearly bad for a system to have jillions of open TCP connections
[11:38] Which Linden: Any system where the # of TCP connections goes even linearly proportional to the number of users needs a good hard looking at
[11:39] Morgaine Dinova: Oh, you wouldn't keep the TCP connections open. They'd be persistent for a while, but disappear when unused, and opened on demand.
[11:39] Which Linden: And this system would be K*N^2 roughly
[11:39] Morgaine Dinova: No system that keeps TCP connections open is ever goign to scale :P
[11:39] Which Linden: Morgaine: the "demand" will occur when a message is sent, so the client cannot open a connection then
[11:39] Morgaine Dinova: So they gotta be transient
[11:40] Which Linden: A client who wants to wait for messages will have to always have a connection open
[11:41] Morgaine Dinova: The client's only connected by a single connection, or a small bundle of them. Don't confuse that with transient ones ---- the permenent session or bundle handles the messaging that opens a temporary connection when a message is ready to be sent
[11:41] Morgaine Dinova: No no
[11:41] Which Linden: OK, when I talk about "client" I mean the agent host
[11:42] Morgaine Dinova: Doing it your way, the number of possible connections is badly restricted
[11:42] Which Linden: The viewer only has one connection, but I'm assuming that the agent host does the aggregation for it
[11:42] Morgaine Dinova: The thing is, you have to start from requirements, not with the technical restrictions of a solution you have in mind.
[11:43] Which Linden: Well, that's a good point, and the conclusion is the technical restrictions of the existing solutions don't map very well to our requirements
[11:43] Which Linden: We're just trying to shoehorn them in
[11:44] Morgaine Dinova: And the user requirement is simply for unlimited group membership, just like in RL. In SL for example, every music enthusiast wants to subscribe to each of their favourite bands' groups --- I would be subscribed to some 2-300 for example.
[11:44] Which Linden: Our current group chat system is pretty ok in this regard -- when you send a message, it opens a transient connection to push the message to the recipient
[11:44] Which Linden: I think what we're realizing is that not every usage of groups is also coupled to a requirement for always-on chat
[11:45] Which Linden: Or that, at least, not all those chats need to be latency-free
[11:46] Which Linden: If our group chat system looked and acted like Twitter everyone would love it, because it's got about the same latency properties
[11:46] Morgaine Dinova: Remember that group membership is self-limiting: (i) ordinary users will be limited by what they can read, so they'll stop subscribing if it gets too much; and (ii) power users will be limited by their bandwidth. Either way, the set is not infinite, you don't have to worry about it spiraling out of control, because it has natural negative feedback.
[11:47] Which Linden: I'm not sure that the self-limitation breaks down so easily
[11:47] Which Linden: Usually when you're pushing some physical limit like "how much you can read" or "how much your pipe can handle" you are getting towards catastrophic failure
[11:48] Morgaine Dinova: Responding to your 3 lines back: well people have been asking for the ability to turn off group chat for ages, without unsubscribing from group of course :-)
[11:48] Which Linden: We'll get around to that someday
[11:48] Morgaine Dinova: Haha
[11:49] Which Linden: But I agree it's necessary
[11:49] Which Linden: I'd be interested in splitting groups into different concepts
[11:49] Morgaine Dinova: And for a disable button on group IM tabs too, to prevent accidental writing to wrong group. Infinity would love that :P
[11:50] Which Linden: Because people use groups for different things -- notifications, chat, permissions, companies
[11:50] Which Linden: It seems that having one conceptual entity do all those activities simultaneously is asking too much
[11:51] Morgaine Dinova: Yes and no. For example, current notiofications suffers from NOT being integrated into IM system.
[11:51] Morgaine Dinova: So we have annoying immoveable blue boxes
[11:52] Which Linden: well the annoyingness of the blue boxes is unrelated to the core concept which is that notifications are one-way communication
[11:53] Which Linden: If we had, say, "notifications-only" groups, then we could use a high-latency but scalable delivery mechanism to get all the group notifications out, and we'd not have any reason to restrict memberships in them
[11:53] Which Linden: I should note that I'm speaking personally, I have no idea if this something LL will ever do
[11:53] Morgaine Dinova: It's related to the fact that they're not integrated into IM, and therefore don't appear in moveable IM windows naturally. Announce-only groups would of course be good, without requiring notification() to be a new verb --- just make it a property on an IM group.
[11:54] Which Linden: Agreed about the UI -- I believe that there is actually a UI project to make the blue boxes less annoying
[11:54] Morgaine Dinova: Less annoying but still not integrated <sigh> :-))))
[11:54] Which Linden: Oh, yeah, I see what you mean
[11:55] Which Linden: I....think that's a UI thing whether it's integrated or separate; agreed that it's conceptually simpler to have it integrated in some ways, but then you have the challenge of explaining "why can't I reply to this?"
[11:56] Morgaine Dinova: "Because if you look at the Channel Properties boxes, you'll see that Owner-Write-Only is ticked." (Or whatever :P)
[11:57] Saijanai Kuhn: Which, just catching up. One thing that should be of intgerest to you is that we're getting the preliminaries setup for pyogp GUI, which will be much more nimble at testing this stuff than the GPL client
[11:57] Morgaine Dinova: And for user-friendliness, attempting to write to it might give a "This channel is configured as ....blah blah".
[11:57] Which Linden: Cool sai, I guess that would be an opportunity to try out various crazy ideas
[11:58] Morgaine Dinova: Your latest works fine from here, Sai.
[11:58] Saijanai Kuhn: well, ENus' use-case is for having 10K bots per script. WOuld definitely test group IM to have 10K bots from the same script talking to each other
[11:58] Morgaine Dinova: Which: Sai is having to use pastebin as a "code distribution mechanism" .... not ideal.
[11:59] Which Linden: Do we agree that it might not be terrible to assume that a user will only be in a finite, small, set of group chats at any given time?
[11:59] Saijanai Kuhn: well, that's partly because I don't recall how to use google code as set up by Tao
[11:59] Which Linden: Sai: get an account on bitbucket
[11:59] Which Linden: Mercurial is the light!
[11:59] Morgaine Dinova: Which: depends what you mean by "small". Like I said, the requirement in SL Live Music is hundreds.
[12:00] Morgaine Dinova: That doesn't mean they'll have the group IM turned on for all of them of course.
[12:00] Saijanai Kuhn: well, I'm actually starting to think my joke solution is viable: a Squeak/Croquet client embedded in every client to facilitate ad hoc P2P setups
[12:01] Morgaine Dinova: They'll turn on whichever groups currently interest them.
[12:01] Saijanai Kuhn: doesnt have to be fullblown croquet, just a TObject server
[12:01] Which Linden: Morgaine: ok, but only so many active at any given time
[12:02] Morgaine Dinova: I think it's pretty user-limited. If you need to limit it for technology reasons, I think you may have the design wrong.
[12:02] Which Linden: I invite you to propose a design that does not have such limits
[12:03] Which Linden: I mean, seriously, that would be worth a lot -- I'm not just shooting you down
[12:03] Morgaine Dinova: Well we've already mentioned that the actual connections to active routers can be transient, persisting only while active for a while.
[12:03] Which Linden: Well we already know that such a system has latency problems
[12:04] Morgaine Dinova: The problem seems to be that you've preallocated paths through your routers. Those don't really need to be static at all.
[12:04] Which Linden: Where have we preallocated them? They're always dynamic
[12:05] Morgaine Dinova: TCP only have a small latency problem on the initial opening. Keeping them persistent while in use resolves that.
[12:05] Which Linden: The problem is not in the TCP connection open -- the problem is in deciding what host to connect to
[12:06] Morgaine Dinova: Well if you haven't preallocated them in your routers than you have no inherent small number of groups limit :-)))))
[12:06] Morgaine Dinova: We may be mixing old and new here.
[12:06] Which Linden: We do have such a limit because no matter how good a job we do of dynamically moving them around we'll still have N^2 connections
[12:07] Morgaine Dinova: Not N^2 simultaneously active those. Those N^2 are just the *potential* endpoint set.
[12:07] Which Linden: Heh
[12:07] Morgaine Dinova: s/those/though/
[12:08] Which Linden: Well, here's the thing; if the client is not keeping the connection open to the server, when a message is ready to be delivered the server needs to find the client and get it to open a connection back
[12:08] Saijanai Kuhn: how does the Agent Domain affect this discussion? Or does it in any substantial way?
[12:08] Which Linden: Sai: it just reduces the number of clients, theoretically
[12:09] Morgaine Dinova: I'll continue examining what you wrote, see if I can understand the vey hard limit given by group size. Don't see it currently though
[12:09] Which Linden: From "all simhosts" to "agent hosts only"
[12:09] Which Linden: There's no hard limit, Morgaine
[12:09] Which Linden: We'd just have to draw a line somewhere
[12:09] Morgaine Dinova: I mean "hard" as in "hard to manage" :-)
[12:09] Which Linden: Oh, that kind of hard
[12:09] Morgaine Dinova: Hehe
[12:10] Morgaine Dinova: You know, where the numbers head off into the stratosphere.
[12:10] Which Linden: When you say group size, I understand that to mean "# of members in a group"
[12:10] Which Linden: Did you actually mean the converse (# of groups a resident can be in)?
[12:10] Saijanai Kuhn: is it possible to apply a SETI-like solution to the problem, where clients collaborate to forward messages?
[12:10] Which Linden: Sai: I'm sure there is!
[12:10] Saijanai Kuhn: a VW phone tree, so to speak
[12:11] Morgaine Dinova: Oh darn, I see what you mean, I've been imprecise, sorry. No, I've been talking about number of groups per person.
[12:11] Which Linden: I've been assuming you meant that most of the time, I just wanted to check
[12:12] Which Linden: In any case, let's come back to this next week! I should run off
[12:12] Saijanai Kuhn: Which, you're familiar with TObjects in croquet, right?
[12:12] Which Linden: Sai: I'm afraid I'm not
[12:12] Morgaine Dinova: KK Which --- great doc, thanks for that, going to read more thoroughyl and think about it.
[12:13] Saijanai Kuhn: ah, OK. A TObject is a nano server that distributes copies of itself to other participating croquet clients
[12:13] Which Linden: Thanks for a very interesting discussion about it, Morgaine
[12:14] Saijanai Kuhn: just a hint of a glimmer of a thought: could a group IM message be encapsulated in a TObject-type thing for forwarding to other clients in the "phone tree"
[12:14] Morgaine Dinova: Very intrigued by this subject.
[12:14] Which Linden: Again, I think the problem is locating the people to whom you're talking
[12:15] Which Linden: I believe that solutions currently fall into one of these categories: can deliver with low latency if some axis is limited (either size or memberships), can deliver with high latency without any limits
[12:16] Which Linden: Maybe someone has something that is both low-latency and unlimited-size, but, that person should be making billions
[12:16] Saijanai Kuhn: Seems that the problem boils down to trackign the subscription list
[12:16] Which Linden: yup, that's the routing table
[12:16] Which Linden: ok, chew on it, time for my watering!
[12:17] Saijanai Kuhn: can that be distributed in any reasonable way?
[12:17] Which Linden: Sai: read back for our "realization"
[12:17] Which Linden: :-)
[12:17] Saijanai Kuhn: KK
[12:17] Which Linden: Will catch you later

User:Which Linden/Office Hours/2009 Mar 19

Navigation menu

Search