User:Which Linden/Office Hours/2009 Feb 12

From Second Life Wiki
Jump to: navigation, search
  • [11:09] Which Linden: hey imaze
  • [11:09] Imaze Rhiano: hi!
  • [11:09] Imaze Rhiano: no one else here? :(
  • [11:09] Which Linden: no, I should send out the aw groupies announcement
  • [11:12] Imaze Rhiano: sighs...
  • [11:13] Which Linden: any topic you'd like to discuss, Imaze?
  • [11:14] Imaze Rhiano: what have you been doing this week?
  • [11:14] Imaze Rhiano: hi!
  • [11:14] Which Linden: Hey sucker, nice group title
  • [11:14] Imaze Rhiano:  :P
  • [11:15] Enus Linden: gotta tell it like it is
  • [11:15] Which Linden: Well, Imaze, over the past week we've been working on finding out how many queues rabbitmq can comfortably handle
  • [11:16] Which Linden: The answer it turns out is around 100,000
  • [11:16] Which Linden: We obviously need that to go higher so we'll see what we can do
  • [11:16] Enus Linden: where do queues come into play in LL's future use of rabbitmq?
  • [11:17] Which Linden: Well we'd want to have a queue per concurrent user
  • [11:17] Imaze Rhiano: enterprise messaging system...
  • [11:17] Which Linden: If we were to use it as an IM system
  • [11:18] Imaze Rhiano: oh..
  • [11:18] Which Linden: A queue is a primitive in the world of messaging systems
  • [11:18] Enus Linden: we wouldn't have one single rabbitmq instance would we?
  • [11:18] Which Linden: You have your queue, and you take messages off of it
  • [11:19] Which Linden: No, we'd have a cluster, but we're having trouble making our cluster handle more than 100,000 queues too
  • [11:19] Enus Linden: ew
  • [11:19] Which Linden: it's possible it's just us doing it rong
  • [11:20] Enus Linden: how does the cluster work?
  • [11:20] Which Linden: setting it up is actually really simple -- you just tell both machines to cluster with each other
  • [11:20] Which Linden: like, the command is basically "cluster otherhostname"
  • [11:20] Which Linden: at that point they share mnesia databases
  • [11:21] Enus Linden: nice name lol
  • [11:21] Which Linden: the mnesia database contains the routing tables
  • [11:21] Which Linden: yeah
  • [11:21] Which Linden: mnesia is a database that's bundled with erlang
  • [11:21] Enus Linden: ok
  • [11:21] Enus Linden: do you have a notion yet of where the bottleneck might be?
  • [11:22] Which Linden: no, I don't know how it works beyond that -- this is why we gotta get in touch with an expert
  • [11:23] Enus Linden: k.
  • [11:23] Which Linden: but really, the bottom line is that it doesn't work right now, so some work is required to make it happen
  • [11:23] Which Linden: I guess it was optimistic of us to assume that we'd be able to find something that works out of the box
  • [11:23] Enus Linden: what uses other than IM could this have?
  • [11:24] Imaze Rhiano: sending group notifications?
  • [11:24] Which Linden: Yup, anything that shows up in a viewer
  • [11:24] Enus Linden: so: move functionality out of the simulator and onto a different queue based service/
  • [11:25] Which Linden: We could use it for some things like balance updates -- i.e. when someone does a transaction with you your viewer gets a message saying that your balance got updated over a message queue
  • [11:25] Which Linden: Yup, the benefit would mostly be that we'd have less code to maintain
  • [11:26] Which Linden: Also, I get the feeling that we'd invent more applications to use message queues, if we had them
  • [11:27] Imaze Rhiano: what kind speed your are expecting it to have?
  • [11:27] Which Linden: Tens of milliseconds message latency
  • [11:28] Which Linden: Well, I guess for group IM, half a second would be a huge improvement over current affairs
  • [11:28] Enus Linden:  :)
  • [11:28] Imaze Rhiano: ya
  • [11:29] Which Linden: What use cases can you think of?
  • [11:29] Enus Linden: i presume the plan is to move to a framework thaat scales forver given (hardware/cleverness/cookies)?
  • [11:29] Which Linden: That would be nice, yes
  • [11:30] Enus Linden: use cases = anythin not pertaining to simulation really
  • [11:30] Which Linden: ha ha ha that's a broad stroke
  • [11:31] Which Linden: I guess you could use it for such purposes as sim-sim communication
  • [11:31] Which Linden: replacing the entire UDP message system
  • [11:31] Imaze Rhiano: faster teleportation?
  • [11:32] Which Linden: WellI think to get faster teleportation we need a system that doesn't copy all avatar data on teleport
  • [11:32] Enus Linden: yeah, agent handoff could be improved by a ton
  • [11:32] Which Linden: until we do that we're still bandwidth-limited
  • [11:32] Enus Linden: avatar data is too heavy for the queue system?
  • [11:33] Enus Linden: i wear this ant b/c it is effin heavy, so much to transfer from sim to sim
  • [11:33] Enus Linden: reminds me that we have a long way to go in some ways
  • [11:33] Tray Guisse: is the limited bandwidth been the cause of some of the lag or is that somthing diffrent altogeather
  • [11:33] Which Linden: well basically if you have a complex avatar, that's like several megabytes that has to be copied from one sim to the next
  • [11:34] Imaze Rhiano: like one of my hair with 255 prims and 255 scripts....
  • [11:35] Which Linden: there yeah -- scripts are tricky cause they have to 1) stop running and 2) their state has to be copied over and restarted
  • [11:36] Which Linden: so no matter whether you transport the avatar data over http or message queue it will still be burdensome
  • [11:36] Enus Linden: i'm about to crash, i can smell it. gonna bail. catch y'all laters...
  • [11:36] Enus Linden: <3
  • [11:37] Imaze Rhiano: bye
  • [11:37] Which Linden: laters! thanks for popping by!
  • [11:37] Tray Guisse: ttyl
  • [11:37] Which Linden: that's odd -- I got a message that he was offline, but his av persisted here for a few seconds afterwards
  • [11:38] Tray Guisse: i have seen that happening alot latly
  • [11:38] Which Linden: I guess maybe the sim was serializing his avatar state to teh asset server
  • [11:38] Which Linden: hm maybe the code used to hide the avatar while it did that, but now it's not
  • [11:40] Tray Guisse: what about the lag issues that is popping up here and there any idea on what cusing it?
  • [11:40] Imaze Rhiano: have you been following MMOX message list lately?
  • [11:40] Tray Guisse: some what
  • [11:40] Which Linden: Tray: probably, the central database, is my guess
  • [11:40] Tray Guisse: ack
  • [11:41] Which Linden: Imaze has anything of interest been transpiring there?
  • [11:41] Imaze Rhiano: no... just semantics are arguing should there be int64, int8, int16 etc... in LLSD
  • [11:42] Which Linden: ha ha ha, standards geeks will be standards geeks
  • [11:42] Tray Guisse: lol
  • [11:42] Imaze Rhiano: I would rather talk about application objects and interactions between them
  • [11:43] Which Linden: application objects meaning what?
  • [11:43] Imaze Rhiano: idefying components like assets server etc...
  • [11:43] Imaze Rhiano: identifying...
  • [11:44] Which Linden: oh yeah -- well that's more of an OGP thing
  • [11:44] Which Linden: I really respect the effort to get a consistent data representation out there
  • [11:44] Which Linden: that makes everything built on top of it so much more useful
  • [11:45] Imaze Rhiano: I am just wondering why it is so hard to add those int16, int64, etc to that damn LLSD :P
  • [11:45] Imaze Rhiano: let's add them there and get moving
  • [11:46] Tray Guisse: yes moving ahead is good
  • [11:47] Which Linden: ah but see, who knows if adding them would cause problems down the line
  • [11:47] Which Linden: would make it harder for implementations or something
  • [11:47] Tray Guisse: has testing even been started to see if their might be an issue
  • [11:47] Imaze Rhiano: maybe I should make my own shadow draft from LLSD ... let's say "kitten SD" and add fish, milk and catnap types there...
  • [11:48] Which Linden: du et
  • [11:48] Imaze Rhiano: oh well..
  • [11:49] Imaze Rhiano: no one doesn't seem to come here tonight :(
  • [11:49] Imaze Rhiano: I am going to watch some tv... or maybe I will think bit about these b-trees that I have been working on :P
  • [11:49] Which Linden: whatcha working on them for?
  • [11:50] Tray Guisse: kk got to get back to work on my build ttyl ^^
  • [11:50] Imaze Rhiano: I am working with database engine that is specialized to store/restore unstructed data - like images, videos etc...
  • [11:51] Imaze Rhiano: bye bye!
  • [11:51] Which Linden: cya tray!
  • [11:51] Tray Guisse: shure we shale see oneanother againe
  • [11:51] Tray Guisse: Well Met all
  • [11:52] Which Linden: so Imaze your database is along the lines of such unstructured dbs as couchdb?
  • [11:52] Imaze Rhiano: pretty
  • [11:52] Imaze Rhiano: it is for .NET
  • [11:53] Which Linden: .NET is the backend, or you have a client written in .NET?
  • [11:53] Imaze Rhiano: oriented mostly for storing streams and object graphs
  • [11:53] Imaze Rhiano: it is embedded database
  • [11:53] Imaze Rhiano: or going to be ... I hope
  • [11:53] Imaze Rhiano: it is not yet ready
  • [11:54] Imaze Rhiano: I am working with b-tree currently
  • [11:54] Which Linden: b-trees are cool!
  • [11:54] Imaze Rhiano: basic storing functionality is there...
  • [11:54] Which Linden: btrees are one of those data structures that you don't find too many library implementations of -- everyone just implements their own
  • [11:55] Imaze Rhiano: they are pretty much core of every database
  • [11:55] Which Linden: for the indices?
  • [11:56] Imaze Rhiano: yes - if you want to have them
  • [11:56] Which Linden: bah, indices are for the weak
  • [11:56] Imaze Rhiano: how so?
  • [11:56] Which Linden: real men upload their rows to an ftp server and let an army of followers memorize them
  • [11:56] Imaze Rhiano: LOL
  • [11:57] Imaze Rhiano: another big part is transactions
  • [11:57] Which Linden: what's the technique for implementing those?
  • [11:58] Which Linden: locks galore?
  • [11:58] Imaze Rhiano: and then ofcourse concurrency and self repairing if somethign goes wrong...
  • [11:58] Imaze Rhiano: I am using "shadow pages"
  • [11:58] Imaze Rhiano: basically if transaction replaces stream in database by another stream
  • [11:59] Imaze Rhiano: then old stream stays in database until all transactions running same time are terminated
  • [11:59] Which Linden: hm, yes
  • [12:00] Imaze Rhiano: my implementation is currently one bad drawback...
  • [12:00] Which Linden: what's that?
  • [12:00] Imaze Rhiano: it is using memory to store indice changes... so when memory runs out....
  • [12:00] Imaze Rhiano: but that shouldn't happend unless user is stupid enough to create massive transactions
  • [12:01] Imaze Rhiano: I guess I need to set somekind limit for transactions lifetime and add functionality for bulk operations
  • [12:01] Which Linden: yeah hm
  • [12:01] Imaze Rhiano: or ... I just need too start using locks in b-trees
  • [12:02] Which Linden: well both have their uses -- I personally hate locks
  • [12:02] Imaze Rhiano: ya
  • [12:02] Which Linden: well I'm looking forward to your database! I wish you the best of luck on it
  • [12:03] Imaze Rhiano: thank you :)
  • [12:03] Which Linden: I'll catch you next week!
  • [12:03] Imaze Rhiano: bye!
  • [12:03] Imaze Rhiano: have fun!