User:Which Linden/Office Hours/2009 Feb 12

• [11:09] Which Linden: hey imaze
• [11:09] Imaze Rhiano: hi!
• [11:09] Imaze Rhiano: no one else here? :(
• [11:09] Which Linden: no, I should send out the aw groupies announcement
• [11:12] Imaze Rhiano: sighs...
• [11:13] Which Linden: any topic you'd like to discuss, Imaze?
• [11:14] Imaze Rhiano: what have you been doing this week?
• [11:14] Imaze Rhiano: hi!
• [11:14] Which Linden: Hey sucker, nice group title
• [11:14] Imaze Rhiano: :P
• [11:15] Enus Linden: gotta tell it like it is
• [11:15] Which Linden: Well, Imaze, over the past week we've been working on finding out how many queues rabbitmq can comfortably handle
• [11:16] Which Linden: The answer it turns out is around 100,000
• [11:16] Which Linden: We obviously need that to go higher so we'll see what we can do
• [11:16] Enus Linden: where do queues come into play in LL's future use of rabbitmq?
• [11:17] Which Linden: Well we'd want to have a queue per concurrent user
• [11:17] Imaze Rhiano: enterprise messaging system...
• [11:17] Which Linden: If we were to use it as an IM system
• [11:18] Imaze Rhiano: oh..
• [11:18] Which Linden: A queue is a primitive in the world of messaging systems
• [11:18] Enus Linden: we wouldn't have one single rabbitmq instance would we?
• [11:18] Which Linden: You have your queue, and you take messages off of it
• [11:19] Which Linden: No, we'd have a cluster, but we're having trouble making our cluster handle more than 100,000 queues too
• [11:19] Enus Linden: ew
• [11:19] Which Linden: it's possible it's just us doing it rong
• [11:20] Enus Linden: how does the cluster work?
• [11:20] Which Linden: setting it up is actually really simple -- you just tell both machines to cluster with each other
• [11:20] Which Linden: like, the command is basically "cluster otherhostname"
• [11:20] Which Linden: at that point they share mnesia databases
• [11:21] Enus Linden: nice name lol
• [11:21] Which Linden: the mnesia database contains the routing tables
• [11:21] Which Linden: yeah
• [11:21] Which Linden: mnesia is a database that's bundled with erlang
• [11:21] Enus Linden: ok
• [11:21] Enus Linden: do you have a notion yet of where the bottleneck might be?
• [11:22] Which Linden: no, I don't know how it works beyond that -- this is why we gotta get in touch with an expert
• [11:23] Enus Linden: k.
• [11:23] Which Linden: but really, the bottom line is that it doesn't work right now, so some work is required to make it happen
• [11:23] Which Linden: I guess it was optimistic of us to assume that we'd be able to find something that works out of the box
• [11:23] Enus Linden: what uses other than IM could this have?
• [11:24] Imaze Rhiano: sending group notifications?
• [11:24] Which Linden: Yup, anything that shows up in a viewer
• [11:24] Enus Linden: so: move functionality out of the simulator and onto a different queue based service/
• [11:25] Which Linden: We could use it for some things like balance updates -- i.e. when someone does a transaction with you your viewer gets a message saying that your balance got updated over a message queue
• [11:25] Which Linden: Yup, the benefit would mostly be that we'd have less code to maintain
• [11:26] Which Linden: Also, I get the feeling that we'd invent more applications to use message queues, if we had them
• [11:27] Imaze Rhiano: what kind speed your are expecting it to have?
• [11:27] Which Linden: Tens of milliseconds message latency
• [11:28] Which Linden: Well, I guess for group IM, half a second would be a huge improvement over current affairs
• [11:28] Enus Linden:  :)
• [11:28] Imaze Rhiano: ya
• [11:29] Which Linden: What use cases can you think of?
• [11:29] Enus Linden: i presume the plan is to move to a framework thaat scales forver given (hardware/cleverness/cookies)?
• [11:29] Which Linden: That would be nice, yes
• [11:30] Enus Linden: use cases = anythin not pertaining to simulation really
• [11:30] Which Linden: ha ha ha that's a broad stroke
• [11:31] Which Linden: I guess you could use it for such purposes as sim-sim communication
• [11:31] Which Linden: replacing the entire UDP message system
• [11:31] Imaze Rhiano: faster teleportation?
• [11:32] Which Linden: WellI think to get faster teleportation we need a system that doesn't copy all avatar data on teleport
• [11:32] Enus Linden: yeah, agent handoff could be improved by a ton
• [11:32] Which Linden: until we do that we're still bandwidth-limited
• [11:32] Enus Linden: avatar data is too heavy for the queue system?
• [11:33] Enus Linden: i wear this ant b/c it is effin heavy, so much to transfer from sim to sim
• [11:33] Enus Linden: reminds me that we have a long way to go in some ways
• [11:33] Tray Guisse: is the limited bandwidth been the cause of some of the lag or is that somthing diffrent altogeather
• [11:33] Which Linden: well basically if you have a complex avatar, that's like several megabytes that has to be copied from one sim to the next
• [11:34] Imaze Rhiano: like one of my hair with 255 prims and 255 scripts....
• [11:35] Which Linden: there yeah -- scripts are tricky cause they have to 1) stop running and 2) their state has to be copied over and restarted
• [11:36] Which Linden: so no matter whether you transport the avatar data over http or message queue it will still be burdensome
• [11:36] Enus Linden: i'm about to crash, i can smell it. gonna bail. catch y'all laters...
• [11:36] Enus Linden: <3
• [11:37] Imaze Rhiano: bye
• [11:37] Which Linden: laters! thanks for popping by!
• [11:37] Tray Guisse: ttyl
• [11:37] Which Linden: that's odd -- I got a message that he was offline, but his av persisted here for a few seconds afterwards
• [11:38] Tray Guisse: i have seen that happening alot latly
• [11:38] Which Linden: I guess maybe the sim was serializing his avatar state to teh asset server
• [11:38] Which Linden: hm maybe the code used to hide the avatar while it did that, but now it's not
• [11:40] Tray Guisse: what about the lag issues that is popping up here and there any idea on what cusing it?
• [11:40] Imaze Rhiano: have you been following MMOX message list lately?
• [11:40] Tray Guisse: some what
• [11:40] Which Linden: Tray: probably, the central database, is my guess
• [11:40] Tray Guisse: ack
• [11:41] Which Linden: Imaze has anything of interest been transpiring there?
• [11:41] Imaze Rhiano: no... just semantics are arguing should there be int64, int8, int16 etc... in LLSD
• [11:42] Which Linden: ha ha ha, standards geeks will be standards geeks
• [11:42] Tray Guisse: lol
• [11:42] Imaze Rhiano: I would rather talk about application objects and interactions between them
• [11:43] Which Linden: application objects meaning what?
• [11:43] Imaze Rhiano: idefying components like assets server etc...
• [11:43] Imaze Rhiano: identifying...
• [11:44] Which Linden: oh yeah -- well that's more of an OGP thing
• [11:44] Which Linden: I really respect the effort to get a consistent data representation out there
• [11:44] Which Linden: that makes everything built on top of it so much more useful
• [11:45] Imaze Rhiano: I am just wondering why it is so hard to add those int16, int64, etc to that damn LLSD :P
• [11:45] Imaze Rhiano: let's add them there and get moving
• [11:46] Tray Guisse: yes moving ahead is good
• [11:47] Which Linden: ah but see, who knows if adding them would cause problems down the line
• [11:47] Which Linden: would make it harder for implementations or something
• [11:47] Tray Guisse: has testing even been started to see if their might be an issue
• [11:47] Imaze Rhiano: maybe I should make my own shadow draft from LLSD ... let's say "kitten SD" and add fish, milk and catnap types there...
• [11:48] Which Linden: du et
• [11:48] Imaze Rhiano: oh well..
• [11:49] Imaze Rhiano: no one doesn't seem to come here tonight :(
• [11:49] Imaze Rhiano: I am going to watch some tv... or maybe I will think bit about these b-trees that I have been working on :P
• [11:49] Which Linden: whatcha working on them for?
• [11:50] Tray Guisse: kk got to get back to work on my build ttyl ^^
• [11:50] Imaze Rhiano: I am working with database engine that is specialized to store/restore unstructed data - like images, videos etc...
• [11:51] Imaze Rhiano: bye bye!
• [11:51] Which Linden: cya tray!
• [11:51] Tray Guisse: shure we shale see oneanother againe
• [11:51] Tray Guisse: Well Met all
• [11:52] Which Linden: so Imaze your database is along the lines of such unstructured dbs as couchdb?
• [11:52] Imaze Rhiano: pretty
• [11:52] Imaze Rhiano: it is for .NET
• [11:53] Which Linden: .NET is the backend, or you have a client written in .NET?
• [11:53] Imaze Rhiano: oriented mostly for storing streams and object graphs
• [11:53] Imaze Rhiano: it is embedded database
• [11:53] Imaze Rhiano: or going to be ... I hope
• [11:53] Imaze Rhiano: it is not yet ready
• [11:54] Imaze Rhiano: I am working with b-tree currently
• [11:54] Which Linden: b-trees are cool!
• [11:54] Imaze Rhiano: basic storing functionality is there...
• [11:54] Which Linden: btrees are one of those data structures that you don't find too many library implementations of -- everyone just implements their own
• [11:55] Imaze Rhiano: they are pretty much core of every database
• [11:55] Which Linden: for the indices?
• [11:56] Imaze Rhiano: yes - if you want to have them
• [11:56] Which Linden: bah, indices are for the weak
• [11:56] Imaze Rhiano: how so?
• [11:56] Which Linden: real men upload their rows to an ftp server and let an army of followers memorize them
• [11:56] Imaze Rhiano: LOL
• [11:57] Imaze Rhiano: another big part is transactions
• [11:57] Which Linden: what's the technique for implementing those?
• [11:58] Which Linden: locks galore?
• [11:58] Imaze Rhiano: and then ofcourse concurrency and self repairing if somethign goes wrong...
• [11:58] Imaze Rhiano: I am using "shadow pages"
• [11:58] Imaze Rhiano: basically if transaction replaces stream in database by another stream
• [11:59] Imaze Rhiano: then old stream stays in database until all transactions running same time are terminated
• [11:59] Which Linden: hm, yes
• [12:00] Imaze Rhiano: my implementation is currently one bad drawback...
• [12:00] Which Linden: what's that?
• [12:00] Imaze Rhiano: it is using memory to store indice changes... so when memory runs out....
• [12:00] Imaze Rhiano: but that shouldn't happend unless user is stupid enough to create massive transactions
• [12:01] Imaze Rhiano: I guess I need to set somekind limit for transactions lifetime and add functionality for bulk operations
• [12:01] Which Linden: yeah hm
• [12:01] Imaze Rhiano: or ... I just need too start using locks in b-trees
• [12:02] Which Linden: well both have their uses -- I personally hate locks
• [12:02] Imaze Rhiano: ya
• [12:02] Which Linden: well I'm looking forward to your database! I wish you the best of luck on it
• [12:03] Imaze Rhiano: thank you :)
• [12:03] Which Linden: I'll catch you next week!
• [12:03] Imaze Rhiano: bye!
• [12:03] Imaze Rhiano: have fun!