Simulator User Group/Transcripts/2014.01.21
| Prev 2014.01.14 | Next 2014.01.28 |
List of Speakers
| Adamburp Adamczyk | David Thurman | Duckie Dickins |
| Ender Hanner | Jenna Felton | Jonathan Yap |
| Kallista Destiny | Kelly Linden | Kennylex Luckless |
| Kitto Flora | Lexbot Sinister | Lucia Nightfire |
| Martinrj Fayray | Mona Eberhardt | Simon Linden |
| Steel Barrs | Tiberious Neruda | Yuzuru Jewell |
Transcript
[12:00] Simon Linden: Hello everyone
[12:00] Kennylex Luckless: Hello you.
[12:00] Simon Linden: I can get started with the server news ...
[12:01] Martinrj Fayray: hello all
[12:01] Simon Linden: Tomorrow the RC channels will all get updated with the same code, which has a crash fix in it
[12:02] Mona Eberhardt: Simon, I'd like to ask about preloading sounds. We have two functions, the deprecated llSoundPreload and the more recent llPreloadSound, which introduces a wait state. Recently, llSoundPreload was recommended again. However, we know it's deprecated. Are you guys planning to keep it as an option?
[12:02] Simon Linden: On other news, the profile feed snapshot upload is finally fixed
[12:02] Kennylex Luckless: Upload did fail for me this morning.
[12:02] Mona Eberhardt: Hi Kelly.
[12:03] Simon Linden: We rarely actually remove functions, Mona, unless there is something seriously wrong or damaging about it
[12:03] Mona Eberhardt: Thank you.
[12:03] Kitto Flora: llSoundPreload - never kenw of its existance. I use llPlaySOund at a very low level to force pre-laod
[12:03] Simon Linden: I haven't looked at those two but perhaps the recommendation to use llSoundPreload() was a mistake ... do you know where that was?
[12:04] Simon Linden: right -is that the correct name?
[12:05] Ender Hanner: Parrrdon me Simon....in rrregarrrds to yourrr Rrrc candidate crrrash fix...is this crrrash a full sim rrrestarrrt orrr just drrropping people without warrrning frrrom the sim but the sim still stays up?
[12:05] Lucia Nightfire: are you purring or is your "R" button screwy, heh
[12:05] Simon Linden: That crash is a full down-and-out crash, Ender
[12:05] Ender Hanner: Drrrat.
[12:05] Lucia Nightfire: lol
[12:05] Ender Hanner: Thank.
[12:05] Steel Barrs: He has a typing impediment.
[12:05] Ender Hanner: s
[12:05] Ender Hanner: Its my accent.
[12:05] Ender Hanner: >.>
[12:05] Lucia Nightfire: lol
[12:06] Tiberious Neruda: didn't know tigers purred
[12:07] Lucia Nightfire: all cats do afaik
[12:07] Ender Hanner: Werrrefolk arrre a pecularrr sorrrt.... >.>
[12:07] Ender Hanner: What is bad is me trrrying to talk without it.
[12:08] Ender Hanner: So much editing when I am not typing in SL to rrremove it.
[12:08] Ender Hanner: lol
[12:08] Steel Barrs: big cats generally can't purr. If they roar, they don't purr.
[12:08] Lexbot Sinister arrives with brainhurt from reading about taxes.
[12:08] Jonathan Yap: Activate autocorrect
[12:08] Ender Hanner: I have issues with autocorrrect on so many levels...
[12:08] Simon Linden: lol - this seems like a slow week for server questions or issues
[12:08] Tiberious Neruda: oh what fun...
[12:09] Ender Hanner: Well slight distrrraction sorrry.
[12:09] Ender Hanner: Back to my querrry line of thought.
[12:09] Lucia Nightfire: my questions typically would start with "When are we getting 'insert shiney name here'?"
[12:09] Tiberious Neruda: heheh
[12:09] Tiberious Neruda: yep.
[12:09] Lexbot Sinister: You keep not messing anything up, so whats can we say. Well, aside the tax fiasco. from Tommy Linden. If you know him, he really deserves a good push or cold coffee at least.
[12:09] Ender Hanner: We arrre experrrienceing in ourrr sim which has been rrrunning smoothly forrr yearrrs wherrre all ourrr visitorrrs arrre drrropped frrrom the serrrverrr without warrrning.
[12:10] Kitto Flora: Is that crash fix to do with physics and rezzeing big objects?
[12:10] Jonathan Yap: I can't log into jira--am getting this error: An error has occurred. Further information is available in the server log.
[12:10] Ender Hanner: You arrre able to immediately log back in and most of the time yourrr not wherrre you werrre and any rrrecent changes to the sim arrre rrremoved back to the last sim checkpoint.
[12:10] Simon Linden: No Kitto -- I can't go into the details on that crash because it's pretty easy to cause, if you know the sequence
[12:11] Tiberious Neruda: tax fiasco?
[12:11] Ender Hanner: I am hoping this fix you speak of will rrresolve ourrr issue howeverrr since in my experrrience a full serrrverrr rrreset doesnt allow you to immediately rrrelog back into the sim with no delay.
[12:12] Simon Linden: That's odd, Jonathan. Is anyone else having jira login issues? It seems fine from here
[12:12] Lexbot Sinister: W-8BEn and ITIN and scanning passports for a guy with a fake name. Linden Lab starts to look more and more like aphishing company
[12:12] Kitto Flora: Simon: Yes. Wengen continues to crash, almost regularly
[12:12] Adamburp Adamczyk: Kitto - I'll back you on that one
[12:12] Ender Hanner: Also the rrrestarrt norrrmall saves the sim as is...ourrr kicks out of the sim do not allow any saving we have had to post a notice stating that no no-copy objects arrre allowed to be rrrezzed because they arrre getting eaten by these drrrops.
[12:12] Adamburp Adamczyk: Wengen does seem to have a larger share of crashesd
[12:13] Kennylex Luckless: Snapshot did work: https://my.secondlife.com/kennylex.luckless/snapshots/52ded479f8a1ec303b000001
[12:13] Simon Linden: yeah, Wengen seems to crash 4-5 times a day, all in physics
[12:14] Steel Barrs: What RC is the sim on, Ender?
[12:14] Ender Hanner: We arrre experrriencing no detectable lag...the sim fps and physics rrremain at 45 fps consistently
[12:14] Martinrj Fayray: maybe try another browser or clear the browser cache jonathan, there was a jira software update yesterday
[12:14] Kennylex Luckless: Btw, is there any way to see what or who is causing time dilation in a SIM?
[12:15] Simon Linden: hmm, actually it really seems to hit a crash loop where it gets saved with griefer content
[12:15] Jonathan Yap: I was able to log in about 90 minutes ago
[12:15] Kennylex Luckless: Natoma has been under 25 % many times latest weeks.
[12:15] Martinrj Fayray: just tested
[12:16] Ender Hanner: He didnt give the name he just said the rrrolling rrrestarrrts would be on wednesdays now.
[12:16] Adamburp Adamczyk: wengen has a hub, adn a large rezz zone
[12:16] Ender Hanner: Not surrre if that will help?
[12:16] Simon Linden: The region/estate Debug tab has the "Top Scripts" and "Top Colliders" searches .. those are the best places to start
[12:16] Lucia Nightfire: unless you're in mainland, heh
[12:16] Ender Hanner: Island no neighborrrs.
[12:16] ER Isolation Hood whispers: Initialising Unit 's Communications Implant. Please wait...
[12:16] ER Isolation Hood whispers: Initialisation complete.
[12:17] Ender Hanner: we have been worrrking on an audit but we have plenty of scrrript time...it doesnt deviate much and this will happen if one perrrson is in sim orrr 20...
[12:17] Simon Linden: well - the times listed in the stats window (ctrl-shift-1) also give you a good idea where things are slow
[12:17] Lucia Nightfire: anyone with any object return level on any parcel in a region should be able to look at top scripts and colliders on mainland
[12:17] Ender Hanner: That is it...i have been rrrunning that....no detectable issues
[12:18] Ender Hanner: i have been rrreporrrting this issue forrr well overrr a week with updates everrry time the sim drrrops everrryone and rrreverrrts.
[12:18] Kennylex Luckless: I has use that but it hard to find user or object that may cause the TD-lag.
[12:18] Lucia Nightfire: the bomb has one running component that doesn't show on sim stats, fwiw, heh
[12:19] Ender Hanner: all we have been told is we shouldnt have anything overrr 3k meterrrs and that we should rrreduce scrrript and do and audit....the thing I dont underrrstand is we have been rrrunning smoothly forrr yearrrs.
[12:19] Ender Hanner: Both in ourrr old sim and this new one.
[12:19] Ender Hanner: the drrrops starrrted just beforrre Chrrristmas with no explination.
[12:20] Simon Linden: Sorry, I'm AFK a bit ... there's a minor crisis to look at
[12:20] Lucia Nightfire: I didn't hear any screaming yet
[12:21] Jonathan Yap: Kelly, are you looking at chat?
[12:21] ER Isolation Hood whispers: Initialising the Communications Implant. Please stand by...
[12:21] ER Isolation Hood whispers: Initialisation complete.
[12:22] Adamburp Adamczyk: baker hasn't switched you lot to decaff again has he?
[12:22] Kelly Linden: Kinda looking at chat Jonathan
[12:22] Duckie Dickins: maybe the NOC monkeys are slinging poo again
[12:23] Jonathan Yap: Kelly or Simon, could one of you give me an update if there has been any progress on the internal issue for bug-4340
[12:23] Jonathan Yap: https://jira.secondlife.com/browse/BUG-4340
[12:23] ER Isolation Hood whispers: Initialising the Communications Implant. Please stand by...
[12:23] ER Isolation Hood whispers: Initialisation complete.
[12:24] Kelly Linden: hrm, marked as a dupe of a feature request (VWR-11778)
[12:25] Kelly Linden: which has its own internal number ... but I see nothing new to report on it at this time Jonathan.
[12:25] Jonathan Yap: okay, thanks
[12:25] Jonathan Yap: I'll ask again when there is no crisis
[12:26] Kelly Linden: :)
[12:26] Simon Linden: I looked into that and I thought it wasn't quite as clean and simple for that field
[12:26] Jonathan Yap: Did you think someting was using it?
[12:28] Simon Linden: yes ... I'd have to double-check, however, but I recall looking through and finding it actually being used
[12:28] Jonathan Yap: ok, so much for my idea :)
[12:28] Simon Linden: Let me look closer at that message ... if we're lucky, we can append a block onto it
[12:29] Jonathan Yap wonders how my post-it notes Simon has tacked to his screen already
[12:29] ER Isolation Hood whispers: Initialising Communications Implant. Please stand by...
[12:29] ER Isolation Hood whispers: Ready.
[12:30] Simon Linden: oh right, this is in the binary bucket payload ... that was one reason it got complicated
[12:31] Mona Eberhardt: Kitto, I tried what you suggested and indeed force-preloading the sounds with llPlaySound at 0.0 volume works better; with llSoundPreload and llPreloadSound, I had the weird effect of the sounds being played multiple times without being triggered by the HUD.
[12:32] Jonathan Yap: I wish all the packets had a size field in them; then expansion would be pretty easy rather than impossible.
[12:32] Simon Linden: oh god yes ... the messaging system drives me nuts because it is so inflexible
[12:33] Jonathan Yap: Gone are the days when the grid would go offline and a mandatory viewer update would occur
[12:34] Kallista Destiny: Thank $DEITY$
[12:34] Kitto Flora: Mona - beware using volume of 0.0 - against the day when its decided that playing a sound at 0.0 is 'not worth doing' - use volume = 0.01, its still inaudible.
[12:34] Lucia Nightfire: yeah, those days forced peopel to go outside for an hour, heh
[12:34] Simon Linden: yes ... and it would be very, very painful to try and slide in a new messaging protocol while keeping the old one alive ... which is why we're still stuck with this one
[12:34] Lucia Nightfire: or a few
[12:34] Mona Eberhardt: OK, thanks.
[12:35] Kallista Destiny: an hour, 3 hours more like it.
[12:35] Ender Hanner: What do you feel is the best option forrr finding potential userrr content that could be causing a sim to crrrash frrrequently?
[12:35] Jonathan Yap: Do you allow anyone to rez ?
[12:36] ER Isolation Hood whispers: Initialising Communications Implant. Please stand by...
[12:36] ER Isolation Hood whispers: Ready.
[12:36] Ender Hanner: yes due to this being a combat sim...howeverrr even with the sim clean to only legal people and me being the only perrrson therrre it still happens.
[12:36] Duckie Dickins: downtime is the devil......People want uptime and they want it now, with no interruptions. Let your ISP drop your net access for an hour and you're ready to kill someone on the support line. :)
[12:37] Jonathan Yap: For testing purposes you could disable scripts from running
[12:38] Jonathan Yap: also keep as many people out as possible, in case someone is wearing something malicious
[12:38] Ender Hanner: As mentioned i have been alone with the sim clearrred of any unknown content and it has kicked me.
[12:38] Jonathan Yap: and then if there is no crash allow scripts in group to run
[12:39] Jonathan Yap: Maybe if there was no LL crisis going on right now Simon could look at the region's logs
[12:39] Ender Hanner: I can trrry the scrrript limting but will have to be carrreful since we arrre a mech combat sim and everrryone who plays must have theirrr worrrn scrrripts to worrrk orrr else it will kill ourrr ability to function.
[12:40] Jonathan Yap: Well, your ability to function is kind of awry right now
[12:40] Simon Linden: Ender- what is the region name?
[12:40] Ender Hanner: The Forge.
[12:40] Duckie Dickins: ahh yes I remember MCM now.. I visited there a few times.
[12:40] David Thurman: what the crisis is about?
[12:41] Duckie Dickins: Someone forgot to buy coffee beans for the coffee machine
[12:41] Jonathan Yap: The Forge has an interesting map tile
[12:41] Ender Hanner: We have been arrround since beforre 06 so having this happen just beforrre Chrrristmas has been rrreally frrrusterrrating. lol
[12:41] Duckie Dickins: after all...devs convert caffeine into code...and without caffeine...there is no code.
[12:41] Ender Hanner: Welcome to Space Jonathan. ?
[12:42] Lucia Nightfire: ground with space texture?
[12:42] Ender Hanner: ourrr main arrrea is a sim size jumpship....just doesnt happen to get caught by the photo.
[12:43] Ender Hanner: We have fourrr combat arrrenas as well but they arrre rrreally too high...lol
[12:44] Ender Hanner: By the way be carrreful what items you change the physics on....you can have a 256 prrrim object come in at a value of well overrr 5K..
[12:44] Steel Barrs: Due to historic reasons, the MCM system avoids SL ground. It could probably be made to work right, but there were issues in the beginning.
[12:44] Simon Linden: Hmmm ... you may be running into memory issues. I'd recommend re-starting the region when you can, but I see it's crashing regularly
[12:44] Ender Hanner: We have rrrestarrrted rrrepeatedly.
[12:45] Jonathan Yap: Simon, would there be some issue because it is now on a different simhost?
[12:45] Duckie Dickins: could any other regions using the same server be a problem?
[12:45] Simon Linden: No, it's been changing simhosts on almost every crash and restart
[12:45] Steel Barrs: we crash several times a day... are you suggesting we need to restart more often than that? I think we need to find what's eating up the memory if that's it and eliminate it.
[12:46] Ender Hanner: we just got moved to the RrrC code to trrry to fix the issue but we arrre still experrriencing the drrrops.
[12:46] Simon Linden: What does the traffic pattern look like? Are you busy all the time?
[12:46] Lucia Nightfire: is there any mesh penetrating the navmesh?
[12:46] Ender Hanner: I have drrropped when I am alone in sim and we have drrroped with 20
[12:47] Ender Hanner: seems to have no corrrilation...we have had no combat going on in any arrrena and some going on in severrral
[12:47] Simon Linden: Have you noticed the navmesh rebaking a lot?
[12:47] Steel Barrs: there's a lot of variability in traffic, but it's usually less than 10 people in the sim.
[12:47] Jonathan Yap: Can you disable the navmesh from rebaking by disabling pathfinding?
[12:47] Ender Hanner: Oddly enough afterrr ourrr issues starrrted we have been picking up trrraffic so this is making it morrre prrroblematic
[12:48] Ender Hanner: we have pathfinding disabled on ourrr sim.
[12:48] Simon Linden: I can see the crash rate has been rising over the last 2 weeks
[12:48] Steel Barrs: I believe we disabled pathfinding.
[12:48] Ender Hanner: We have not hearrrd the best things about leaving that enabled without a good rrreason.
[12:48] Ender Hanner: lol
[12:48] Simon Linden: yeah, turn it off if you're not using it
[12:49] Steel Barrs: we weren't using it, so it was best to turn it off until we need it.
[12:49] Ender Hanner: It has been disabled since it was an added option.
[12:49] Lexbot Sinister: oh, i got a question you might not be able to answer, but here goes : Can you check Im logs based on the location of where the avatar was during the conversation?
[12:50] Duckie Dickins: Would there be anything in the viewer log that might give a hint as to the cause of a crash..maybe some event triggered and seen by the viewer? There was a crash many years back due to broken objects in world that got corrupted due to a bad router at the data server and one no-mod item got altered from it and anytime I came within view of the item the viewer crashed and the sim restarted.
[12:50] Duckie Dickins: oops..data center
[12:51] Jenna Felton: is there a way to enforce an older server versions for a sim? or it is only possible on aditi?
[12:52] Simon Linden: We do that with the channels, Jenna ... a channel is assigned a version
[12:53] Jenna Felton: ok
[12:54] Jenna Felton: than you could try an older sim version on The Forest sim and see if the crash is stil there
[12:54] Jenna Felton: must be some version frol last year before xmass
[12:54] Jenna Felton: from*
[12:55] Ender Hanner: Simon would you be able to give us a morrre specific date wherrre you see the sim starrrted this crrrash patterrrn?
[12:55] Simon Linden: Lex - yes, we do have the position for chatting
[12:55] Jenna Felton: i meant "The Forge", o forest
[12:55] Ender Hanner: Also is therrre a way to cleanly see exactly how much memorrry ourrr sim is using to know if we arrre crrrossing that line?
[12:56] Ender Hanner: Also what is that line?
[12:56] Steel Barrs: Simon, would the logs indicate that the crashes are memory related, or is that just a guess?
[12:56] Simon Linden: I'm betting on memory right now
[12:56] Object: Hello, Avatar!
[12:56] Ender Hanner: That is the most conclusive and strrraight forrrwarrrd answerrr we have got in two weeks...lol
[12:57] Simon Linden: However, there really haven't been a lot of server changes in the last two months due to the pace of work and the holidays
[12:57] Lexbot Sinister: (i'm asking because there's a particular nasty Age Play club, that keeps moving around.. but essentially it would be good for LL to check every IM sent from that location, instead of trying to bait & AR by the regular resident)
[12:57] Simon Linden: Do you know if there is new content being used there? Has someone updated the combat system recently?
[12:57] Steel Barrs: I'm betting the only way to deal with memory issues is to reduce the amount of content?
[12:57] Lucia Nightfire: have you tried disabling physics to see if the crashes stop?
[12:58] Simon Linden: Well, the best way is for us to find a memory leak and fix it :)
[12:58] Ender Hanner: I do know....I am half of the ownerrrship of the company...we have not made any combat system updates rrrecently.
[12:58] Jenna Felton: actually there is a combination of server code and obkests on the sim :) probably there is a way to build up a part of rezed stuff and see if that has an effect on the problem
[12:58] Steel Barrs: there was a minor change with how the meter interacts with the teleport system, Ender.
[12:58] Jonathan Yap: One approach is to start disabling various features and see if the crashes stop, which doesn't really help too too much, but it wouldbe something
[12:59] Jonathan Yap: You could also cut your region into parcels and disable scripts from running on a per-parcel basis one at a time
[12:59] Ender Hanner: Well we would grrreatly apprrreciate any and all help we can get...thus farrr ourrr linden contact has basically said gut yourrr sim till it is stable and rrrebuild but has not given us any clean dirrrection of what the issue is exactly we arrre looking to prrrevent.
[12:59] Simon Linden: We don't keep much old crash data ... the database is pretty large as it is, and crashes older than a few weeks usually aren't that interesting since it needs to be fixed and tested against the latest version anyway. That said, it would be nice in a case like this
[13:00] Ender Hanner: Drrrat..good to know then.
[13:00] Duckie Dickins: I think that will end up messing with your traffic score, Jonathan. Not that traffic plays much of a role, although I suspect the web search uses it for relevance...
[13:00] Ender Hanner: so forrr finding ourrr memorrry usage?
[13:00] Jonathan Yap: Well, in this case with things not working, some kind of aggressive debugging has to be done
[13:00] David Thurman: how the other sims on the same server are doing?
[13:01] Simon Linden: I think we have a starting point to look at, anyway. The fact that it occurs when the region is empty or nearly empty shows it's probably something in-world ... and thus we may be able to reproduce it without needing a crowd
[13:01] Simon Linden: It keeps moving to other servers when it re-starts, so I don't think it's related to that
[13:02] Jonathan Yap: are the other simhosts all in the same datacenter?
[13:02] Ender Hanner: If the fix you spoke of doesnt effect the issue it should be no issue to experrrience it again the prrroblem is it is rrrandom.
[13:02] Steel Barrs: how do we know how much memory the region is using, and how much memory do they have available?
[13:02] Jonathan Yap: I remember a project a while ago to get neighbors into the same datacenter
[13:03] Ender Hanner: Because i basically live therrre as does alot of memberrrs we experrrience it often but just coming in you might not everrr catch a kick.
[13:03] Simon Linden: The hour is up and I need to get going ... thanks everyone for the good chat today
[13:03] Jonathan Yap: Thank you Simon & Kelly
[13:03] Ender Hanner: Thank you forrr the suggestion Jonathan.
[13:03] Ender Hanner nods
[13:03] Jenna Felton: i have to go too
[13:03] Steel Barrs: Thanks, Simon. TC
[13:03] Mona Eberhardt: Thanks for having us, Simon and Kelly. Take care!
[13:03] Kelly Linden: cheers
[13:03] Lexbot Sinister: Thanks for comming (unlike Nyx!!)
[13:03] Nal (nalates.urriah): Thx
[13:03] Steel Barrs: Thanks, Kelly.
[13:03] Simon Linden: Take care ... see you next time
[13:03] Jenna Felton: thank you lindes for the meeting and everyone have a great wek
[13:03] Ender Hanner: Thanks...now to find a way to see ourrr memorrry usage.
[13:03] Jenna Felton: week*
[13:04] Yuzuru Jewell: Thank you, Simon and Kelly.
| Prev 2014.01.14 | Next 2014.01.28 |