User:Ama Omega/archive/Office Hours/2007-04-09

Transcript

Khamon Fate: hi kelly
You: howdy
Gareth Ellison: hi
Khamon Fate: are these your first office hours?
You: Yeah
Khamon Fate: we must have a pizza throwing contest
Drewan Keats: He Kelly, Gareth Khamon
Khamon Fate: it's mandatory
You: Grab a blue seat.
Gareth Ellison: i stand
You: sure
Gareth Ellison: (or rather not move too much due to lag)
Khamon Fate: what fabulous topic shall we discuss today?
You: Heh! I don't have an agenda. I'm actually between projects right now, just got off gridmonkey duty so ......
You: I'm rather unprofessional about the whole office hours thing.
Khamon Fate: I have a question then
Dzonatas Sol: i got an idea
Dzonatas Sol takes a number
Khamon Fate: How do y'all assign someone to bugs reported in jira? I have three on tap and nobody assigned to them. It's making GIR cry.
Gareth Ellison: question about grid infrastructure - is sim content loaded over NFS? or assets or local disk?
You: I'm not sure about the external one. We have a studio dedicated to it - Blacklight - and they follow the external jira and internal jira bugs closely.
Khamon Fate: assets are all served by web servers
Gareth Ellison: but the sim content is loaded from where?
Gareth Ellison: i.e the looocation of prims etc
You: I'm not sure about how they decide what gets assigned to whom. I know robla also does some passes at moving some of the issues from the external jira to the internal one
Khamon Fate: from the sim
You: The simulator process keeps all of that info in memory Gareth
Khamon Fate: do you know if we see the internal information after they're migrated?
Gareth Ellison: at startup where does it load from?
You: Khamon, know you don't.
Dzonatas Sol: Grid stuff -- would it be in the future of SL to allow servers not to be so unified to control one region at a time?
Khamon Fate: k
You: er no, you don't see any internal comments or changes. All thought all the imported ones include a link back to the external issue
Braisin Coakes shouts: ahhhH!
Gareth Ellison: kelly - at startup wwhere do the sims load from?
You: I don't konw if I understand that question Dzonatas
Gareth Ellison: local disk, asset server or NFS?
Khamon Fate: are we about to have more city sims?
You: Gareth: Simstates are saved to either the central asset servers durring 'normal' operation or to the local sim's disk during updates (rolling or otherwise)
Khamon Fate: sorry, that's not a public question
You: We don't use NFS for any grid infrastructure stuff. The only thing we use NFS for is user home dirs.
Gareth Ellison: are they routinely backed up while running?
You: Simulators save simstate once / hour.
Gareth Ellison: how much data?
Dzonatas Sol: right now there is one server per 65Ksqm per 15k prims. I was wondering what the dynamics were like to allow guest servers to control subregions within a host server (for example)
You: Once it is compressed? up to 20mb. Uncompressed up to a couple hundred mb
Gareth Ellison: i'm wondering if that could be contributing to some of the problems with missing objects some people have
You: dzonatas: First it is 1 region / cpu on a server. We run dual cpu, dual core servers (4 cores) that run 4 regions. We also run some 'void' regions that are 4-1 (4 regions per cpu), so 16 regions on a box
You: Gareth, possible, unlikely, and would depend on the actual case you are talking about.
You: Somone should totally sit in that blue chair.
Dzonatas Sol waiting for rez why background compilation occurs
Dzonatas Sol: *while
Gareth Ellison: i've noticed at imes after grid restarrrrts or individuaaaal sim crashes, objects go missing
Gareth Ellison: sory fort he typossss
You: Yes, that would be why, especially in the crash case. Up to 1 hour of content may go missing.
You: Even in the crash case though the sim does attempt to save it's state before crashing. It succeeds most of the time.
Gareth Ellison: do LL have plans for scaling the grid?
Dzonatas Sol: Kelly, I understand the 1 region per sim as you explained - i'm just wonder how hard or if the dynamics allows for guest servers to work with host servers. If the data can only be kept in one box at a time, that is understandable. If there was a way to cluster boxes further, I was curious.
Gareth Ellison: i.e multiple grid/user servers
You: I guess I'm not sure what you mean by a guest server?
You: Do you mean having multiple servers host the same region for improved performance?
Dzonatas Sol: oh, I use host/ghost in virtualization terms... the host has the main control of the hardware
Dzonatas Sol: yes
You: Gareth: We are focused on replacing non-scalable systems with scalable systems - the Userserver will essentially be gone in an upcoming release for example. We are moving to partitioning data across multiple hardware to increase scaling
You: We aren't really looking into creating multiple grids for scaling.
You: dzonatas we don't do any virtualization right now, although we have thought some about it, because there are some potential benefits to it. I don't know a lot of details about that though.
Macaroni Kidd: Hi Kelly, I have an easy question.
You: sure macaroni
Macaroni Kidd: I have two adjacent properties, and so my object quota is combined for the two properties. Which is great. I can put the full quota of objects on one property, and none on the other if I want.
Macaroni Kidd: If I buy another property in the same region that is not adjacent to my property, will I be able to add the object quota for that property to my overall count? So, if buying another property gives me another 200 objects to my quota, can I use all that quota on my first property, and leave the new property bare?
You: yes macaroni, as long as all the properties are in the same region
Macaroni Kidd: Great. Thanks very much!
You: welcome
Dzonatas Sol: there are benefits, yes. I'm not sure if I confused the question with virtualization terms. hehe... what I'm getting at is if there is a future to SL with borderless regions
You: We have borderless regions ..... although region crossing needs to be improved some. I'm not trying to be dense, really.
Gareth Ellison: i was refering to multiplllle grid servers on one grid
Gareth Ellison: you guys cal it the space serverrrr ((((the one that hannnndles sim datttta)
Gareth Ellison: ewwww, lag
You: gareth we are already moving in that direction. The space server too will be replaced by a system that partitions the data across multiple hosts. We have user databases that host your inventory, saved IMs etc - around 12 of those now I think, each person on a different one.
Gareth Ellison: heh,, and i thought i was ahead of you in just spec'ing tthe protocols
You: The space server really doesn't contribute to lag, it has a very minimal job and your client never talks to it.
Gareth Ellison: /meee has desssigned a set of ppprotocols for dddistributing all the main baaackend servers acros physical hosts
Dzonatas Sol: not saying you are! lol... just trying to get a feel for it without getting picking apart internals. To me, sim crossing is kinda like a border. I'm thinking of a way to transpose sim together.
Dzonatas Sol: *sims
Gareth Ellison: so there's no extra load on any part of the whole grid as new users andget aded
Gareth Ellison: so long as new grid/useeer/aset srrrevers come onllline to
You: Gareth that is the grand scheme idea. We aren't there yet but yes! we are definitely devoting substantial development power to getting there.
Gareth Ellison: my thoughts werrre something along the lines of DNS
You: dzonatas, historically the simulator has done a LOT of stuff, and it still does a LOT of stuff. It is a middle man in sending textures to viewers, it used to be a middle man in uploading textures, it is a middle man in every request that hits a database (such as Searching etc)
You: dzonatas: What we are doing is moving as many of the pieces that are NOT simulation (simulation is physics, scripts, interest list, objects in world) out of the simulator and over to webservices.
Summer Seale: ao off
otakup0pe Neumann: oh hi.
Gareth Ellison: teh poppe!
You: Uploading assets is already moved to this, fetching assets / textures is *very* soon going to be this way. All dataserver requests will also be this way soon
otakup0pe Neumann: CAPS ?
You: What this will mean is that the simulator will become leaner, meaner and better at handling the core things - including region crossing
Summer Seale: sorry i'm late and i don't mean to interrupt, but what's today's topic?
You: otakup0pe, yes sim caps enables the moving of services out of the simulator.
otakup0pe Neumann: Which is a Good Thing(tm)
Summer Seale: lol
otakup0pe Neumann: I noticed how there are some new CAPS enabled on the main grid....
otakup0pe Neumann: You guys just kinda sneaked them in
You: One reason the simulator did everything is because we trusted that the simulator knew who you were and that you were a valid connection, or at least the simulator could verify it to some degree. We didn't want to recreate that for a whole bunch of services - but caps means we don't have to.
Gareth Ellison: does the viewer suport more CAPS than the servers right now?
otakup0pe Neumann: They can disable / enable caps on the fly, right ?
You: Sometimes new caps are added disabled to verify that the system will work that way, incase they are ever needed to be disabled in the future.
otakup0pe Neumann: I don't remember seeing a server update before notecards adn animations were moved to CAPS
You: yes, caps can be enabled or disabled on the fly
Dzonatas Sol: the caps is a good move. I think what Gareth mentioned was in the likeness of my question.
Gareth Ellison: so, viewer CAPS?
Gareth Ellison: got texture ddownloads therE?
You: The servers don't yet support that, at least not in any released code.
otakup0pe Neumann: How is it looking on the non-released code ?
Summer Seale rated you positively: nice, cute, *and* a Mac user. can't go wrong with that kinda dreaminess <3 =)
You: That cap will move texture downloads out of the normal udp stream and over to http
You: which should be a huge benefit
Gareth Ellison: iis the veiwer capabble of doin textures over CAPS as son as the server does it?
Gareth Ellison: i.e any client-sidee updates neded?
You: I'm not sure.
Dzonatas Sol: caps give sims the extra bandwidth that was used as being the middle-man
Gareth Ellison: if i was to pput textures over CAPSS in my own sim woould the curent viewer usei t?
You: well, the extra cpu cycles really in not managing image lists, not routing messages.
otakup0pe Neumann: We'd have to implement CAPS that same way Gareth.
otakup0pe Neumann: Does any opensim stuff use CAPS ?
Gareth Ellison: yes
otakup0pe Neumann: I know libSL supports CAPS, but I think only from the point of view of the client ?
You: I just don't know Gareth, sorry. I haven't been working on that project. It may be that more viewer code is needed for caps to work.
Gareth Ellison: i aded the htpd while ago
Gareth Ellison: talk on IRC about that
otakup0pe Neumann: So what do they have you working on kelly ?
You: otakupope, I'm between projects and between dev studios. I am going to be spending most of today talking to dev studio leads to figure out where I want to work and what on.
otakup0pe Neumann: Cool.
otakup0pe Neumann: Kelly, the nomad coder.
You: It will either be Blacklight which is doing bug fixes and performance metrics, DaBoom which is doing conversion of services to webservices and more or icehouse which has been behind caps and more.
otakup0pe Neumann: I hacked up a libsl RESTbot
otakup0pe Neumann: Oh. I want to bug DaBoom people hehe.
You: Or there is a chance I will go rogue and pick up a couple projects that don't fit in those sutdios.
otakup0pe Neumann: so we can all enjoy the goodness that is AWS.
otakup0pe Neumann: Oh. You should add llSHA1 to LSL ;)
Drewan Keats: Kelly, are you in SF or remote?
You: I'm in Davis, CA
You: There is an office here, looking to hire more Devs.
Dzonatas Sol: I applied =)
You: :)
Gareth Ellison: how many iin the UK?
You: Uh .... I don't know about liaisons, but I think 2 devs are in UK
Summer Seale: there's one liason in the UK: Data Linden
Summer Seale: that i know of
You: Ah that is right, he is.
Summer Seale: ya he's a sweetie too.
otakup0pe Neumann: Who other than Babbage ?
Summer Seale: and gives out cute teddybears =)
Gareth Ellison: data is a UK guy?
Summer Seale: yup
Summer Seale: he's helped me on euro time before
Gareth Ellison: nice to have a treky linnden here
Summer Seale: hehe yup
Summer Seale: got a nice office too
Drewan Keats: What have you been working on most recently Kelly?
Summer Seale: woot rob linden is here too
You: The primary linux client dev is in UK
Summer Seale: hi Mr. Rob =)
otakup0pe Neumann: Oh hey Rob
Dzonatas Sol: heya Rob
otakup0pe Neumann: Fancy meeting you here !
Rob Linden waits for everyone to rez
Drewan Keats: Hey Rob.
You: Drewan I've been working on database load reduction - SQL query optimization, datapartitioning and building webservice data caches and similar
Summer Seale: you're kinda ruthed to me, Rob
You: Been doing that for the last 6 - 9 months.
Drewan Keats: Am I the only one not having lag here?
Gareth Ellison: i am siting!
You: maybe drewan
Summer Seale: so you're basically one of those in charge of speeding up the grid during these slowdowns with the huge weekend loads, Kelly? =)
Summer Seale: amazingly i don't have lag here yup
Summer Seale: but i never lag in linden office areas =)
otakup0pe Neumann: They are Class 8 Sims
You: Summer a large part of what has allowed us to go from 20k online at once at the begining of the year to almost 40k has been my work.
Gareth Ellison: i normaly do'nt lagg her
Summer Seale: ya i was here for the big christmas slowdown
Gareth Ellison: i'm laging so much my typing is n0bified
Summer Seale: i wasn't one of those who sent in angry emails tho. i know you guys are trying hard.
You: Right now .... our db does not appear to be causing our online at once cieling, and I'm not sure what is. Which is why blacklight is taking on improving our metrics as I mentioned earlier so it is easier to see what breaks when, where and why
otakup0pe Neumann: How many studios are there ?
Gareth Ellison: do you run loadt ests on smal test grids?
Drewan Keats: Are you capping onlines now? I was wondering why 40K hasn't been broken
Dzonatas Sol: It's going to be interesting when there is an official server farm outside the US =)
You: Icehouse, DaBoom, Blacklight, Ator and Enzo
You: No we are not intentionally capping onlines, but something seems to be drewan.
Summer Seale: oh is that why we have login troubles sometimes?
Summer Seale: i didn't know that
Gareth Ellison: what about load tests?
You: Yeah, basically.
Summer Seale: interesting
Summer Seale: not general knowlege =)
You: Gareth load tests of 40,000 online across almost 9k regions is a very, very difficult task.
Dzonatas Sol: Do you believe the limit may be more external?
otakup0pe Neumann: Kelly have you managed to pull it off ?
Gareth Ellison: simply spawnning logins from a sccript is imple
Gareth Ellison: lots of ways to test without full-blown clients
Gareth Ellison: if the login series at 40k login-only bots then it's the login server
Gareth Ellison: argh laag typos
You: It may be related to network issues which is a huge umbrella with many potential possibilities, or it could be databases in some way we aren't seeing or it could be something else!
otakup0pe Neumann: Maybe the hamsters are tired.
Summer Seale: lol that's what i was thinking the other night =)
Dzonatas Sol: Is it feasible for bots to connect to SL. Let's say many of them that would act like NPCs for WoW style gaming?
You: Login server is only used for logins, and there isn't any indication that the rate people are logging in is what is slowing us down. When we bring the grid back up logins are way higher than 'normal' - about 1k / minute.
Uildiar Kuhn gave you The Creation.
otakup0pe Neumann: yaye bots.
You: Dzonatas: libsl has bots, hacked clients could be made to be bots.
Summer Seale: btw..
Summer Seale: lol out of curiosity...
Summer Seale: why is there a casino ad across the water? isn't this like...linden village? =)
otakup0pe Neumann: They are not immune.
You: I don't know. I'm not commenting on any of the casino stuffs.
You: and linden village is in the middle of the real SL ;) so ... over there isn't linden village
Dzonatas Sol: So, I could host a tournament of bot killing? Or, would it be best that the limits of logins be solved first before we let the hord of bots connect?
otakup0pe Neumann: Kelly so is anyone using libsl internally at LL ?
otakup0pe Neumann: That you know of at least.
Summer Seale crashes
Summer Seale: missed your answer
You: otakupope I don't think we are beyond what QA does to verify reported exploits with it.
Summer Seale: ya why are there casino ads there i mean i thought linden land was kinda ad free and stuff?
Summer Seale: ao on
otakup0pe Neumann: hrm
You: Summer - that isn't linden land.
Summer Seale: oh i'm just curious like i'm not gonna be one to argue about casinos tho i personally don't like them. i just think it's kinda ironic.....
Summer Seale: oh okay
otakup0pe Neumann: How do you do load testing itnernally ?
otakup0pe Neumann: Preview pile-onst ?
otakup0pe Neumann: Lots of second life sessions running ?
otakup0pe Neumann: ons rather
Summer Seale: ao off
You: I don't think we currently do adequit load testing.
You: But I have also been distanced from QA lately so I'm not sure what they do.
You: Its 12:00! This was fun. I'll do it again next week. Maybe I can get the packet loss to stop here by then. (yeah, right)
Summer Seale: Kelly, i *have* noticed slower load times lately on textures. is that part of the reason you guys are moving that system to a new one? it takes longer to rez pictures etc
Summer Seale: lol ok
Drewan Keats: Thanks, Kelly.
Summer Seale: yes thank you =)
You: Summer Seale, yeah in part.
otakup0pe Neumann: kelly take it easy
Summer Seale: these sessions are always great and informative
Summer Seale: i always learn a lot
Summer Seale: ao on
Summer Seale: as i said, nice, cute, *and* a mac user =)
Summer Seale: how can you go wrong? =)
Summer Seale: haha =)
You: :D
Summer Seale: i'm on a PC right now but my main puter is a mac laptop <3
Dzonatas Sol: thanks for your time
Summer Seale: ya thank you i always love linden office sessions
You: I've got a Windows PC as my main dev box and a Macbook Pro for my laptop.
Rob Linden: I'll be having office hour at 3pm as well, if you'd all like to show up for that. see everyone later!
Summer Seale: ya i have a macbook pro 17" too
Summer Seale: ok thanks rob!
You: cya rob
You: I need to hunt down food, thanks all for showing up!
Summer Seale: hehe ya i'm in SF too
Dzonatas Sol: llPizza()
Summer Seale: near LL actually
Summer Seale: you guys have fun!