User:Andrew Linden/Office Hours/2009 08 25

Transcript

• [11:22] Andrew Linden: The answer he got back from his questions was mostly: yes it is a known problem, we have a solution planned, lots of people are involved
• [11:22] xstorm Radek: is it to do with a DNA timeout ???
• [11:22] Techwolf Lupindo: Do anyof the managers log into the world? They brag about it all the time. :-)
• [11:22] Andrew Linden: That is what I gathered from the thread for the most part (I only skimmed it)
• [11:22] BackHome Soon: https://blogs.secondlife.com/community/technology/blog/2009/06/15/improving-the-quality-of-group-chat
• [11:23] Arawn Spitteler understands the developers involved with Group Chat are intensively discussing the problem, over Group Chat
• [11:23] BackHome Soon: there was a "fix" rolled out -- I think in 1.25???, that only made it worse
• [11:23] Techwolf Lupindo: I hope one of those soluction is splitting groups into chat only and one into land funcitons.
• [11:23] Rex Cronon: low arawn
• [11:23] BackHome Soon: and had to be backed out quickly
• [11:23] Rex Cronon: lol*
• [11:23] Andrew Linden: Yeah, I think the solution is a real project -- it is going to take some time to scale that correctly. I dunno the timeline of the project.
• [11:24] lonetorus Habilis hopes we soon will see a media api irc client :)
• [11:24] xstorm Radek: it is more then one problem and whats making it seem bigger is how the problems work with each other to look like one big problem
• [11:24] Lestat Demain: why not just rip the code from msn it has an 'average; chat system
• [11:24] Techwolf Lupindo: The "fix" was flipping the Adult check on, that when it started all of a suden and they been scrambelling to fix it due to orders not to flip that switch back.
• [11:24] Uni Ninetails: occasionally i get this sense of the sheer weight of work ll devs have and it scared the rabbits outta me.
• [11:24] Arawn Spitteler: I'm surprised a VP inquiry would have been recent. I remember CG discussing Group Chat before he even studied office hours
• [11:25] BackHome Soon: a July 1st update from Sardonyx -- https://blogs.secondlife.com/community/technology/blog/2009/07/01/a-status-update-on-group-chat
• [11:25] Techwolf Lupindo reads
• [11:26] Simon Linden: yeah, it's been flaky for a while. There aren't going to be easy, small fixes ... the whole back-end needs an overhaul
• [11:26] Andrew Linden: looks like server 1.32 will be mostly internal overhaul -- very few features there if any.
• [11:26] BackHome Soon: YAY!
• [11:26] Andrew Linden: 1.32 is scheduled for November.
• [11:27] xstorm Radek: will there be a load test on the main grid in some sim ?
• [11:27] Andrew Linden: I think most of the overhaul in server-1.32 will be toward separating the SL viewer code from the SL server
• [11:27] Lodi McMillan excuses herself...RL conflicts..
• [11:28] s een subject to frequent freezes, today
• [11:28] Rex Cronon: tc
• [11:28] Andrew Linden: what kind of load test are you asking about xstorm?
• [11:29] xstorm Radek: find a sim with the larger number of support bug on it and get the bug testers there to try and crash it
• [11:29] Simon Linden: Did I miss an earlier mention of svc-22 ?
• [11:29] Vehicles crossing region borders aren't always treated as vehicles and can get incorrectly returned if the destination parcel is no-entry or parcel-full
• [11:29] xstorm Radek: keep a log of what happens
• [11:30] Techwolf Lupindo: I hit Crash Me a couple times.
• [11:30] Andrew Linden: no load tests for crashing simulators that I know of
• [11:30] office hours is half over
• [11:30] xstorm Radek: may be good to test later on so a record can be made ?
• [11:31] Andrew Linden: our simulator crash rate is lower than it has ever been. Instead of load testing for crashes we're using our "crash report" tool to try to focus on the most common crash modes
• [11:31] Andrew Linden: a load test for lag would be a different thing. I could see us attempting that...
• [11:31] Arawn Spitteler hadn't mentioned SVC-22, nor even SVC-93 yet: What'sthe dolar ratio, of Dev v QA Salary?
• [11:31] ROTATION and llSetRot incorrectly implemented for child prims
• [11:31] Lestat Demain: i woulda thought overloads was the most common mode
• [11:31] xstorm Radek: crash reports are not listing the sim stopping is it ?
• [11:31] Andrew Linden: if we had a team to work on the various known sources of lag
• [11:32] Andrew Linden: xstorm, our simulator crash report tool is working pretty well right now. yes, it lists the region name
• [11:32] lonetorus Habilis: often it starts out with residents having little clue on best buildig practices, bei it infinite non throttled for loops in lsl or very large textures
• [11:33] Lestat Demain: lag for client or sim cuz having a pingtime of 230ms + leads to problems u dont see
• [11:33] Simon Linden: The sim was fine, Rex
• [11:33] Arawn Spitteler: You were the only one, this time
• [11:33] Rex Cronon: weired
• [11:33] Arawn Spitteler: I've been crashing a lot myself, today
• [11:34] Uni Ninetails: actually funny you say that Emeralds been flakey today for some reason
• [11:34] xstorm Radek: corfeld has had so much lag in the last mounth that many times people can not move or even teleport
• [11:34] Arawn Spitteler: Viewerchanges don't seemto help, with today's lag
• [11:35] lonetorus Habilis: artoo, how many ppl can akeyo usually support, before its too laggy to walk around?
• [11:35] artoo Magneto: 15
• [11:35] xstorm Radek: and corfeld has just 5 people in the sim on the last bad lag
• [11:36] Uni Ninetails: ouch
• [11:36] lonetorus Habilis: and corfeld is also a full sim?
• [11:36] Arawn Spitteler: Is Cornfield on a NullSim?
• [11:36] Andrew Linden: corfeld currently has 14 people on it, FPS = 43
• [11:36] artoo Magneto: atm we have 4350 running scripts here.. i wonder whats average..
• [11:36] Lestat Demain: tried looking at av cost when that happens?
• [11:37] Uni Ninetails: club or a botnet on it? *ntes the cluster of dots
• [11:37] xstorm Radek: for corfeld 14 people is the max do to a club
• [11:37] xstorm Radek: thats not a standerd number of people
• [11:37] Andrew Linden: the lowest FPS I see for Corfiel's hourly backups is 31 FPS, but those are just hourly snapshots that last 1 minute
• [11:38] Andrew Linden: Oh I take that back. There is on 30FPS event
• [11:38] lonetorus Habilis: what about akeyo?
• [11:38] Arawn Spitteler: Hi, Xe, are you still getting unusual lags, today?
• [11:39] Xe Parx: terrible in 4 different sims
• [11:39] xstorm Radek: well if the FPS is that high whats stopping the people from moving at times ?
• [11:39] Xe Parx: avg of 7fps
• [11:39] Andrew Linden: Akeyo has 3 avatars, FPS = 43
• [11:39] lonetorus Habilis: yeah, it just came back online
• [11:39] lonetorus Habilis: after having been offline fr a hour or so
• [11:39] Uni Ninetails: woah akeyo (i assume thats the one that sells sneakers) looks busy sculpty wise
• [11:39] Andrew Linden: Akeyo has better FPS during the hourly backups... about 43 average it looks like
• [11:39] Xe Parx: changed all my parameters to the lowest possible values too
• [11:39] Xe Parx: 2 hours of this so far
• [11:40] lonetorus Habilis: the averages i measure myself, seem fine, around 44 etc, its the lag spikes that rind it all to a halt XD
• [11:40] Andrew Linden: well, the simulator running Akeyo looks happy, but the viewers visiting there might suffer
• [11:40] Ardy Lay: I have been sitting through periods of time when I seem to not be getting updates from the simulator I am in. This what you are discussing?
• [11:40] Xe Parx: right now i have 7.6 fps
• [11:40] Andrew Linden: Ardy Lay, is your viewer rendering new frames when that happens?
• [11:41] xstorm Radek: are you saying all the problems is client side ?
• [11:41] Andrew Linden: That is, can you move your avatar's head around and rotate the camera view?
• [11:41] Ardy Lay: Yes, animatins are running too.
• [11:41] Xe Parx: yes
• [11:41] Andrew Linden: Hrm... Ardy have you looked at the packet-loss meter when that happens?
• [11:41] Ardy Lay: I can turn but not change locations.
• [11:41] Xe Parx: i can
• [11:41] Arawn Spitteler guesses comm side
• [11:41] Lestat Demain: get a log of cordo70% of them are...set up in a ramdisk and watch nearly all the problems vanish
• [11:41] Andrew Linden: Ardy, it it happening in this region?
• [11:41] Techwolf Lupindo: I've been having problem of moving the past few days. I get stuck and can't move quiet often.
• [11:41] Ardy Lay: Packet loss meter on zero
• [11:42] xstorm Radek: when we freez in corfeld we can turn on the spot not move
• [11:42] Arawn Spitteler normally checks apperence, when he can't move
• [11:42] Andrew Linden: huh... xstorm that sounds like a server glitch alright
• [11:42] lonetorus Habilis: arawan, sounds like sim freezes on tp arrivals and departures
• [11:42] Ardy Lay: I am not trying to move so not noticing a problem at the moment.
• [11:42] lonetorus Habilis: err, xstorm even :)
• [11:43] xstorm Radek: and the map will not load and inventory will not load when it happens
• [11:43] Ardy Lay: I am setting up a network too so I can see if I have a connectivity issue.
• [11:43] Ardy Lay: network tool
• [11:43] Lestat Demain: that pause u mention is in idle time processing when downloading textures
• [11:43] Andrew Linden: hrm... yeah these symptoms sound like they might be caused by momentary blockage of the simulator
• [11:43] Andrew Linden: perhaps caused by teleport arrivals
• [11:43] Ardy Lay: Yes, arrivals do seem to be a cause.
• [11:44] Ardy Lay: They certainly are coincident with the pauses.
• [11:44] xstorm Radek: whish me to pull up consol debug next time and see what happens ?
• [11:44] Ardy Lay: Some last a long time.
• [11:44] Andrew Linden: Lestat, you're talking about the simulator stats? You see spikes in texture downloads?
• [11:44] Andrew Linden: Sure, I'd be interested in getting more clues as to where the lag spikes are that you're seeing
• [11:44] lonetorus Habilis: the length of the freeze of the sim, also chnages, so i suppose it depends on the numder of scripts needing to searilize, and the size of the textures the ariving person has on their avatar, that ahs to be pushed to all the other avatars in the region, the more avatars, the more bandwidth the sim neds to use, and the longer freeze?
• [11:44] Andrew Linden: However...
• [11:45] Lestat Demain: idle time goes up which drives up render time which slows down client response its sorta like a domino effect
• [11:45] Andrew Linden: I suspect the info is going to tell me they are in places that we already know are a problem.
• [11:45] Uni Ninetails: dunno i was the only one on a sim once, and another person tpd in and the sim froze for a full minute.
• [11:45] xstorm Radek: im starting to think im a sim bug magnet
• [11:46] lonetorus Habilis: the sim freezes is probably the single most irretating thing currently :)
• [11:46] Simon Linden: The freezes can also be affected by other regions running on the same server -- if two or three (or more) regions are trying to burn a lot of CPU, disk or bandwidth at the same time, they can all suffer
• [11:46] lonetorus Habilis: simon, indeed, when that happens, we restart the sim, to hopefully get started up oon a "better" server
• [11:47] Ardy Lay: You may get one others abandonned for the same reason.
• [11:47] Arawn Spitteler: My understanding is thatCPU Load will only effect that CPU, but that communications of that Server, can be competed for?
• [11:47] xstorm Radek: hold on so if a number of sim's are on the same server with sion chickens and teleport lag problems how many report have there been on that one sever ?
• [11:47] Simon Linden: That restart can also help clear things up sometimes ... one theory about the lag (particularly the mono script one) is that it gets worse over time
• [11:48] Ardy Lay: SOMETHING certainly gets worse over time.
• [11:48] Arawn Spitteler: Server Side Memory Leaks?
• [11:48] lonetorus Habilis: ardy, smelly feet?
• [11:48] Andrew Linden: Simon, I know you've looked into this problem and have thought about it. What do you think the next step would be if we had some spare devs to work on it (I know that we don't but purely for the exercise).
• [11:48] xstorm Radek: thinking so yes
• [11:48] Ardy Lay: I have gotten several mainland regions restarted recently.
• [11:49] Andrew Linden: Simon, do you think one tactic would be to just get HTTP texture out the door?
• [11:49] Simon Linden: Hmm, the TP arrival / region crossing glitches have some known bottlenecks ... one project would be to look at that code and speed it up or smooth it out
• [11:49] Uni Ninetails: dingdingding we have a possible winner
• [11:49] Andrew Linden: Or would it be to put a team under Babbage to get perftools actually working in 64bit?
• [11:50] BackHome Soon: wouldn't it be reasonable to first identify whether there's a memory leak at all? then you can look into finding WHAT'S actually leaking, then fix it
• [11:50] Ardy Lay: Sometimes it's just by getting stuck there and contacting support. They look and say "yes, that one needs restarted, can you move to another for a while?"
• [11:50] Simon Linden: The mono/memory/slow degradation is another problem that really needs someone's focused time ... at least a week to dig in and work on the memory tools, how to reproduce it, etc
• [11:50] Andrew Linden: BackHome, finding memory leaks can be tricky. We're looking into getting google-perftools working again.
• [11:51] Lestat Demain: eh u broke them?
• [11:51] BackHome Soon: oh, I agree, Andrew, but first do you even have a leak is my question
• [11:51] Andrew Linden: Unfortunately the simulator is so big that most memory leak detection tools won't work with it -- it runs too slow under them.
• [11:51] Techwolf Lupindo: How many mainland owers would like to be able to restart there mainland sim?
• [11:51] BackHome Soon: some quality time with top and ps and a little perl will tell you volumes
• [11:51] Simon Linden: There's no obvious memory leak ... i.e., we've seen the problem on regions that aren't bloated
• [11:52] Andrew Linden: Yes BackHome, simple easy to find memory leaks can be found with top
• [11:52] Andrew Linden: however some "memory leaks" are really memory fragmentation
• [11:53] BackHome Soon: right
• [11:53] xstorm Radek: if its bottlenecking what about moving the sim with heavy load on to sims with not much load at all ?
• [11:53] Andrew Linden: and if the memory leaks takes two weeks to become a problem... watching top is less useful
• [11:53] Techwolf Lupindo: How about putting out a call for sim owners to allow there sim to be place under the tools knowing it will suffer performance just to track down the problems?
• [11:53] BackHome Soon: and since in my RL, my company has moved away from C++ middle tier into J2EE, my whole outlook has changed...lol
• [11:53] Andrew Linden: we discovered that one of the problems with fixing the simulator crash rate is that long-term memory fragmentation is starting to bite harder
• [11:54] xstorm Radek: C++ had problems for some time now
• [11:54] Techwolf Lupindo: Firefox had that prolem.
• [11:54] Lestat Demain: dont have a defragger for memory?
• [11:54] Simon Linden: We have talked about better control and allocation of regions ... making sure popular, heavy loaded areas don't share the same server, but it soon gets into a complicated issue ... how do you rate 'good popular place' vs. 'bad design' ?
• [11:54] Rex Cronon: backhome. did thing get btter?
• [11:54] Andrew Linden: No Techwolf, I mean the simulator runs soooooo ssssslllloooowwww under the tools that it is completely unusable
• [11:55] Arawn Spitteler: Memory should havea slow- low priority defrag rolling all thetime
• [11:55] Techwolf Lupindo: Andrew, eek...
• [11:55] BackHome Soon: Rex, i do banking software -- heavy transactions, but nothing like SL volumes, but no, it ran faster under C++
• [11:55] Andrew Linden: the only promissing perf/memory tool is google perftools
• [11:55] office hours is almost over
• [11:55] Andrew Linden: however it currently does not work for us
• [11:56] Andrew Linden: the simulator runs on it fine, but the results that it is reporting are meaningless and clearly bogus
• [11:56] Techwolf Lupindo: After digging in the cmake code and discovering that the actuall mono engine is use for running mono scripts, I was like, "You got to be kidding?"
• [11:56] xstorm Radek: take it to a server side load graph ? that can move sim's with a heavy load to a server with less load to plus cut down on bottlenecking ?
• [11:56] Andrew Linden: when it suggests the process is taking up only 1MB when you know it is 200MB... something is wrong
• [11:56] Lestat Demain: hmmm thats amazing...all u programmers and no-ones written a memory monitor/defrag/cleaner?
• [11:56] xstorm Radek: *GIGGLES* :)~~~~
• [11:57] Andrew Linden: all u programmers...
• [11:57] BackHome Soon: hang on,Lestat, I'll write you a batch file
• [11:57] BackHome Soon: errr, shell script
• [11:57] BackHome Soon: lol
• [11:57] Andrew Linden: You make it sound like we have a surplus of dev resources
• [11:57] xstorm Radek: you make the program and tell us how great it works
• [11:57] Lestat Demain: well as far as i know 1 of the requirements to being a linden is to know something about programming/scripting/networking
• [11:58] Lestat Demain: or is that info outdated?
• [11:58] Arawn Spitteler: Not for all lidens
• [11:58] Simon Linden: yeah, I think there were some interview questions about some of those things...
• [11:58] BackHome Soon: LMAO Simon
• [11:58] Arawn Spitteler: WhiteBoard Interview?
• [11:58] xstorm Radek: there is a problem when it comes to tracking down memory leaks that way it put more load on a system and many times crash or never spots the problems
• [11:59] Techwolf Lupindo: Let prey that the server code isn't one huge Bash script. ;-)
• [11:59] lonetorus Habilis: dont bash bash
• [11:59] lonetorus Habilis: XD
• [11:59] Andrew Linden: xstorm, could you rephrase that question?
• [11:59] Simon Linden: The real problem with this bug is reproducing it ... that needs more effort. I made a short attempt generating zillions of mono scripts, but it ran fine
• [12:00] xstorm Radek: simple trying to find a memory leak in memory = loop
• [12:00] Ardy Lay: Hmm... the packet loss median bar is in the wrong place. It's too high in the floater.
• [12:00] lonetorus Habilis: simon i did that too, at crash me, and it went offline for 3 hours
• [12:00] Andrew Linden: bash is terrible for most programs longer than half a page long, and if you have to make lots of variables it gets quite messy
• [12:00] lonetorus Habilis: took about 20000 scripts in a box
• [12:00] Techwolf Lupindo: Put a bunch of sion chickens on there Simon.
• [12:00] lonetorus Habilis: mono
• [12:00] Thank you for coming to Linden office hours
• [12:00] Andrew Linden: I have a love+hate relationship with bash programming
• [12:01] Andrew Linden: I often start programming a tool in bash, only to ask my self "Why?!" later
• [12:01] BackHome Soon: it's handy to prototype something
• [12:01] Simon Linden: I'll do that if I can get off my current project and assigned to fix bugs. Until that happens, I'm not gonna have much time. We need more devs
• [12:01] Andrew Linden: Ack. I've got a meeting to attend now.
• [12:01] BackHome Soon: need a systems engineer? ;-)
• [12:01] Andrew Linden: thanks for coming everyone.
• [12:01] Uni Ninetails: Thanks Andrew, :) have fun
• [12:02] Lestat Demain: *waves and jumps up and down*
• [12:02] Simon Linden: Check the web site, we're hiring...
• [12:02] BackHome Soon: Thanks all -- enjoyed it and I'll be back
• [12:02] JayR Cela: thank you :_)
• [12:02] Rex Cronon: simon. thereare some interesting weapon huds. get a few people to try them while u watch for memory leaks:)
• [12:02] Simon Linden: Thanks all for coming, see you next time
• [12:02] xstorm Radek: to bad Sony , IBM and sun micro do not build LL a new type of server for free ;-)
• [12:02] Rex Cronon: tc
• [12:02] Ardy Lay: People, when looking at the site for job listings, find BOTH LISTS.
• [12:02] Morgaine Dinova: Bash is fine when it's used for its intended purpose, which is mainly launching apps and glueing together filter pipelines. If you use it for the wrong thing then it's poor, but that's not its fault, it's misuse.
• [12:02] lonetorus Habilis: simon, wnt that box os scripts that i used for crashing crashme?
• [12:02] Arawn Spitteler: Get a few lindens to tet teh eapon huds on the chickens of Crash Me?
• [12:02] JayR Cela: see ya later Rex / Morgaine / *waves* byee to everyone :_)
• [12:02] Simon Linden: Ionetorus - sure
• [12:02] Andrew Linden: Actually, we are indeed in a little hiring sprint now. Developers, QA engineers, and others.
• [12:03] Morgaine Dinova: See ya JayR
• [12:03] Rex Cronon: tc jayr
• [12:03] BackHome Soon: hmmmmmmmm
• [12:03] xstorm Radek: bye every one
• [12:03] BackHome Soon: BackHome Linden?????
• [12:03] Simon Linden: Bye
• [12:03] Lestat Demain: but wot if their overseas like au?
• [12:03] Rex Cronon: tc xstorm
• [12:03] xstorm Radek: thank you all for letting me drive you all crazy ;-)
• [12:03] Techwolf Lupindo: Andrew, where are the job posting for those? Everytime I check, its just for managers.
• [12:03] Morgaine Dinova: Ione: you'll get done for war crimes :-)
• [12:03] Yann Dufaux: sorry ㋡
• [12:04] Ardy Lay: Tech, I can help you find the other list.
• [12:04] Techwolf Lupindo: Ardy, thank, I can use the help to crack the code. :-)
• [12:04] Arawn Spitteler: IMs ar run through the local simulators?
• [12:05] Arawn Spitteler: Discussion in Advnced Scripters
• [12:05] xstorm Radek: :-)
• [12:05] Ardy Lay: First list, http://lindenlab.com/employment
• [12:05] Ardy Lay: Seconf list, http://lindenlab.hrmdirect.com/employment/openings.php?sort=da&
• [12:05] xstorm Radek: off to drive moles nuts
• [12:05] Uni Ninetails: sih back to planting trees
• [12:05] Ardy Lay: They are often listing different positions.
• [12:06] Uni Ninetails: Have a good one!
• [12:06] BackHome Soon: bye everyone
• [12:06] Rex Cronon: tc
• [12:06] lonetorus Habilis: yes, see ya all later
• [12:06] Ardy Lay: There, for all to see. Join the party, I wish I could.

Generated with SLog Wikifier