Simulator User Group/Transcripts/2012.11.27
| Prev 2012.11.20 | Next 2012.12.04 |
List of Speakers
| Andrew Linden | Baker Linden | Inara Pey |
| Jonathan Yap | Kallista Destiny | Lares Carter |
| MartinRJ Fayray | Motor Loon | Nalates Urriah |
| Qie Niangao | Rex Cronon | Simon Linden |
| Theresa Tennyson | Yuzuru Jewell |
Transcript
[12:00] Yuzuru Jewell: Hello, Nalates.
[12:00] Nalates Urriah: Hi
[12:00] Yuzuru Jewell: Hello, Inara.
[12:00] Motor Loon: Ey guys
[12:00] Nalates Urriah: Hey Yuz
[12:00] Meeter: Welcome to the Server User Group
[12:00] Yuzuru Jewell: Hello, Andrew.
[12:01] Andrew Linden: Hello everyone.
[12:01] Kallista Destiny: *waves*
[12:01] Inara Pey: Hi, andrew
[12:01] Inara Pey: Hi, Yuzuru, sorry, was afk
[12:01] Yuzuru Jewell: :D
[12:03] Andrew Linden: I was out of the office last week, but Simon gave me the transcripts and I just put them on the wiki a few minutes ago.
[12:03] Andrew Linden: Simon may show up later. I think he's AFK atm.
[12:03] Andrew Linden: I haven't been paying attention to the server update news this week
[12:04] Andrew Linden: but I think there was an update happening today.
[12:04] Kallista Destiny: Speak of the Devil
[12:04] Qie Niangao: Nothing scary in the forum thread: http://community.secondlife.com/t5/Second-Life-Server/Deploys-for-the-week-of-2012-11-26/td-p/1746697
[12:04] Inara Pey: Main channel? Appears to have been... it chased me across the grid
[12:04] Motor Loon: hehe
[12:04] Baker Linden: Hi everyone!
[12:05] Rex Cronon: hi everybody
[12:05] Inara Pey: Hi, Baker, Simon, all!
[12:05] Simon Linden: Hello everyone
[12:05] Yuzuru Jewell: Hello, Bake and Simon.
[12:05] Rex Cronon: hi simon, baker
[12:05] Yuzuru Jewell: Baker
[12:05] Rex Cronon: hi inara
[12:06] Andrew Linden: Simon, we're ready for any news about server updates.
[12:06] Simon Linden: ah, ok ...
[12:06] Rex Cronon: hi andrew
[12:06] Simon Linden: Today we rolled the version that was in the RC channels to the main grid
[12:06] Baker Linden: I don't have much news -- I'm still working on multithreading object rezzing, and I finally got things working -- now to see whether or not there's a benefit!
[12:06] MartinRJ Fayray: Hi
[12:07] Simon Linden: The RC channels will get updates tomorrow as well ... BlueSteel and LeTigre are getting a small maintenance fix
[12:07] Simon Linden: Magnum is getting that with a few additional changes that should help stability
[12:07] Motor Loon: Thats always good stuff
[12:08] Simon Linden: I think there are release notes ... 1 sec...
[12:08] Simon Linden: http://community.secondlife.com/t5/Second-Life-Server/Deploys-for-the-week-of-2012-11-26/td-p/1746697
[12:08] Simon Linden: I don't think there's much in there for new features
[12:09] Andrew Linden: The only news I've got is about the interestlist changes that we played around with during the Thursday beta grid meeting two weeks ago...
[12:09] Andrew Linden: There was a bug where some people's HUD attachments would show up on your screen, that one is fixed.
[12:09] Simon Linden: ah, there is a little bit - llGetObjectDetails() has some new options
[12:09] Rex Cronon: nice
[12:09] Simon Linden: They are OBJECT_PHYSICS, OBJECT_PHANTOM, OBJECT_TEMP_ON_REZ.
[12:10] Motor Loon: hehe yeah that was a wierd bug
[12:10] Simon Linden: That's it for release news
[12:11] Andrew Linden: Also, regarding the intrestlist stuff running on Ahern in the beta grid: the performance tests so far are looking as good or better than the old code, so this stuff may pass QA soon.
[12:11] Andrew Linden: That's all the news I've got.
[12:11] Andrew Linden: So I guess the table is open.
[12:11] Nalates Urriah: congrats
[12:12] Nalates Urriah: I saw Baker's Large Groups Edit has made it to the main viewer. Congrats Baker
[12:12] Theresa Tennyson: Did you find the problem with the interest list picking up people's HUD's?
[12:12] Baker Linden: Thanks Nal!
[12:12] Inara Pey: Yup, good news, Baker
[12:12] Andrew Linden: Yes Theresa, that HUD bug is fixed.
[12:12] Jonathan Yap: Is https://jira.secondlife.com/browse/SVC-7996 on anyone's to-do list?
[12:13] Simon Linden: yes, that's stuck on mine, Jonathan. I grabbed it so it wouldn't get lost
[12:13] Jonathan Yap: thanks Simon. I might, err, be bugging you every so often about it :)
[12:14] Simon Linden: It seems like a good idea but I haven't gotten around to it ... crashes have been demanding my attention
[12:14] Jonathan Yap: fixing crashes should come first
[12:15] Theresa Tennyson: DId you find out anything about the physics memory leak or the sudden loss of simulator frame rate?
[12:16] Motor Loon: ah thats tied to my question I guess:
[12:16] Motor Loon: "I wondered if anyone might be comments or ideas on a current server problem. One of our clients who rents a homestead from our estate, has had some pretty bad problems for the last 3 weeks or so. Basically the region seems to leak physics memory, starting out from a fresh restart with an allocation of around 50-60mb and slowly climbs to over 256mb. This ofcourse meaning that once it gets around around 230mb the RCCS kicks into emergency mode and prevents rezzing/attachments etc. on the region, forcing out resident to need to restart the sim every hour to have proper use of it. I understand a jira is already filed on it and techs are looking at it, but I don't know the jira no. and was hoping to a bit more info to give to my renter."
[12:16] MartinRJ Fayray: BUG-888 or BUG-772 are examples
[12:17] Andrew Linden: Theresa, you're asking about the memory and lag that was being looked at at last week's meeting?
[12:17] Motor Loon: (I do personally see anything on the sim that could be "problematic content", and I've heard anything from that it's a total mystery to that it might be releated to the havok 2012.1 release or even odd'er that it was somehow releated to particles (not sure why they would allocate physics memory though))
[12:17] Theresa Tennyson: Yes.
[12:17] Simon Linden: yeah, that's the problem we're looking at. We haven't been able to locate or reproduce the leak
[12:18] Motor Loon: The sim of mine where it happens constantly is: Rainbow Cove - if you wanna give it a check. Several tickets in on it aswell.
[12:19] MartinRJ Fayray: Motor you should file a Jira, Maestro is looking for examples of that bug
[12:19] Motor Loon: ok
[12:19] Motor Loon: But no ideas on what might be the cause?
[12:19] Simon Linden: Is there anything about activity there that is worth noting? More vehicles, combat bullets, or such than other regions?
[12:20] Andrew Linden: Dee Kass was reporting a problem with spiking memory in the "Jinxies" region two weeks ago, but moved to the Magnum channel and reported that things were looking good after that.
[12:20] Andrew Linden: Magnum was promoted today, so it may be that her problem was solved.
[12:20] Motor Loon: I havent seen anything. 3437 objects, stores/landscaping/trees/some moving animals but non-physical.
2244 active scripts, 338 active objects
No temp-on-rez objects
FPS great.
Physics time around 1.5ms
Scriptime around 3.8ms
Sparetime 16.2ms
Total time 22ms
Pathfinding 0.9ms
No objects under "top colliders"
[12:21] Motor Loon: Basically what I'd call a pretty normal homestead setup, mabye apart from a bit high primcount.
[12:21] Andrew Linden: Hrm... the region is on "Second Life Server" channel. I wonder if the promotion today to what was on Magnum fixed it.
[12:22] Simon Linden: How long does that usually take to bloat up ?
[12:22] Motor Loon: I'm pretty sure it had been rolled when I was there today, and monitored the memory allocation go from 60mb to 256+ while I was there for an hour
[12:22] Andrew Linden: Rainbow Cove is also on "Second Life Server" channel.
[12:22] Motor Loon: yes
[12:22] Andrew Linden: Ok, so you noticed the problem happen already today.
[12:22] Motor Loon: Rainbow Cove reaches 230mb around an hour after restart usually
[12:23] Motor Loon: yes, Ive seen it happen today
[12:23] Andrew Linden: Motor, and it happens with just one avatar in the region? It doesn't take a crowd to cause the problem?
[12:23] Motor Loon: Been going on for the last 3 weeks or so - resident claims not to have changed anything when it started, and sim had run just fine before then without these issue with the same content. Fits in with the release of Havok 2012.1 to mainchannel.
[12:23] Motor Loon: Yes, we was 2 avatars there today
[12:24] Andrew Linden: Ok, well a region that can reproduce it will help a lot.
[12:24] Motor Loon: It sure can
[12:24] Andrew Linden: Falcon may have some ideas on how to analyze the memory allocations, so we'll try to talk to him about it later today.
[12:25] Qie Niangao: FWIW, Rainbow Cove is at 181.7mb at the moment
[12:25] Motor Loon: Excellent, I'll just be glad if I can assume my client that its being worked on
[12:25] Motor Loon: assume=assure
[12:26] Theresa Tennyson: Just out of curiosity, do they have any dance poles?
[12:26] Motor Loon: you asking me Theresa?
[12:26] Theresa Tennyson: Yep.
[12:26] Qie Niangao: Theresa is looking for a hobby. :p
[12:26] Simon Linden: Something's definitely odd there ... I see the total memory size jumping up 50mb in 2 minutes, only one person there, while the object count stays the same
[12:26] Motor Loon: Thats not unlikely... he's using the sim for a gay-products store... there's also some adult club stuff there, Im sure there's "fun rooms" with poles or other animations
[12:27] Andrew Linden: Rainbow Cove has an uptime of about 1hr 30 minutes.
[12:27] Theresa Tennyson: I see extremely high memory use at Lolita Park, and I've heard of it on Zerango and Jinxies. Those are all strip clubs.
[12:27] Motor Loon: Yep simon... thats what I saw too... very big jumps in the allocation for no apparent reason, but combined over time a steady raise in the number
[12:28] Simon Linden: I'm going to go grab the logs and see if I can tell what happened in those 2 minutes ... at least that's a fairly small window of time
[12:29] Motor Loon: can it for some odd reason be related to it being a "adult" flagged sim?
[12:29] Andrew Linden: unlikely
[12:29] Qie Niangao: Meanwhile, completely off topic and low impact, but just in passing, the Map thing from a couple weeks ago still has some gaps. E.g., a strip visible on Oak Bluffs (or just open the Map in-world and find "Oak Bluffs"). There are others, but just a note that something is still slow/stuck.
[12:30] Meeter: Timecheck : User Group is half over
[12:30] Simon Linden: The maturity rating of a region is really only applied to access permissions and similar logic ... it shouldn't affect memory or features at all
[12:30] Motor Loon: k
[12:31] Andrew Linden: And "Sunny Point" the region that was not on the map can now be found there.
[12:31] Inara Pey: If they are all adult and with a similar theme, is it not worth looking at scripted objects they may have in common for a potential culprit?
[12:32] Andrew Linden: Yup, Oak Bluffs is not on the map.
[12:32] Theresa Tennyson: I'm wondering if it has something to do with dance poles - if it were sex beds the entire grid would have caught fire weeks ago.
[12:32] Jonathan Yap: Here we go again--I see Oak Bluffs on my old v1 viewer
[12:32] Motor Loon: does seem odd if its incidental that the affected sims all have to do with some sort of "adult" material.
[12:32] Jonathan Yap: It has a harbor
[12:33] Motor Loon: hehe Theresa
[12:33] Andrew Linden: and the the whole column is missing... there should be one region above and two below Oak Bluffs
[12:34] Andrew Linden: hrm... perhaps entire column is missing
[12:35] Qie Niangao: Most of the other continents look intact, but there's another column missing at Korzun
[12:35] Simon Linden: I dunno Theresa ... I think if it was dance poles we'd have melted down too. What do you think the pole:bed ratio is?
[12:36] Motor Loon: shouldnt really have anything to do with physics memory either
[12:38] Motor Loon: the things about it possibly being caused by particle emitters, is that just some nutty rumor or a theory at the lab?
[12:38] Qie Niangao: these "memory leak" sims (mostly? all?) have Pathfinding turned off, right? (which would rule out something funky with navmesh updating or anything like that)
[12:38] Simon Linden: Motor - is there a jira for Rainbow Cove?
[12:38] Andrew Linden: I don't see how particle emitters could affect the physics memory.
[12:39] Motor Loon: I havent filed a jira... we just did tickets on it
[12:39] Theresa Tennyson: What kinds of things use up physics memory? The baseline memory use seemed to get a lot higher after Pathfinding.
[12:39] Motor Loon: but
[12:39] Motor Loon: "Thank you for submitting a support request. We were made aware tonight by a previous resident who explained at length the issue regarding a memory leak on the Second Life server. It's very possible the latest SLS server deploy caused the issue we are now dealing with.A bug report is being made by the resident which will be escalated to our engineers for review. Vix Linden"
[12:39] Simon Linden: ok, thanks, I'll write up one for our internal tracking
[12:40] Motor Loon: ticket case 01558274 and 01550673
[12:41] Motor Loon: handled by Derrick and Vix
[12:43] Motor Loon steps on the cricket..
[12:43] Theresa Tennyson: Where are things at with threaded region crossing?
[12:43] Simon Linden: I'm doing paperwork, so if anyone has questions or topics, let's hear it...
[12:44] Simon Linden: Threaded region crossings are in QA now with one or two bugs that need to be fixed
[12:44] Simon Linden: ... basically the same as before Thanksgiving
[12:45] Theresa Tennyson: I went to the one region on Aditi that had it - I could cross in but not out.
[12:45] Lares Carter: Are the problems with Aditi being worked on?
[12:45] Simon Linden: The inventory problems on Aditi? yes
[12:45] Lares Carter: (failed logins, problems with agni inventory synch)
[12:46] Nalates Urriah: Lots of people are crying about ADITI login probs
[12:46] Andrew Linden: Problems with aditi logins today?
[12:46] Simon Linden: The database there gets overloaded a lot faster than we'd expect
[12:46] Lares Carter: I personally haven't had any, but I see a lot about it in the Second Life Beta group.
[12:46] Nalates Urriah: I haven't tried today.
[12:46] Rex Cronon: overloaded? what do u mean?
[12:47] Jonathan Yap: The disks fill up
[12:47] Simon Linden: right
[12:47] Rex Cronon: impossible
[12:47] Motor Loon: mabye try bigger disks?
[12:47] MartinRJ Fayray: What would we do without Motor :P
[12:47] Motor Loon: I know lol
[12:48] Simon Linden: There are good system people working on it ... more hardware is one possibility, but that takes more time and money. These aren't off-the-shelf disks from Fry's
[12:48] Motor Loon: Guess we'd just be sitting on some roof talking about things eh
[12:48] Rex Cronon: there is a way to eliminate overloading:)
[12:48] Simon Linden: I think there's a basic issue with bringing over too much data from the main grid ... we import a lot of accounts we really don't need to
[12:49] Nalates Urriah: If Fry's doesn't have it try eBay :)
[12:49] Simon Linden: but we don't want to go back to doing that manually
[12:49] Rex Cronon: make all accounts there temp. each time somebody logs in on beta an account that exists there but user not loggedin is removed:)
[12:50] Nalates Urriah: So... someone is going to have to develop a better import algorithm ?
[12:50] Andrew Linden: Ah I see the backlog in one of our internal IRC channels -- the beta inventory db servers are filling up.
[12:50] Simon Linden: Removing stuff is particularly painful ... and I mean really removing it, not just marking it deleted like a lot of DBs do intially
[12:51] Andrew Linden: Yeah, we've got lesser hardware serving the beta grid, which is why we don't just import all accounts
[12:51] Andrew Linden: you have to ask for your account to be added if it isn't already.
[12:51] Motor Loon: We just need a "come clean up your betagrid inventory" dat
[12:51] Motor Loon: day
[12:51] Qie Niangao: As Rex said, just wipe it weekly (or something), and always fail the first login attempt, then background a task to import just that user.
[12:51] Nalates Urriah: Who does one ask?
[12:52] Andrew Linden: I dunno Nalates, I think we used to route everyone to Oskar since he ran the beta user's group
[12:52] Rex Cronon: i think the lindens don't want to do that because all u made on beta will be deleted
[12:52] Motor Loon: How about running a script that purges all inventories on betagrid from avatars that hasnt access'd betagrid for 3 months or something
[12:52] Andrew Linden: and was often importing accounts to betea
[12:53] Qie Niangao: I gather that purging on a per-account basis is Hard.
[12:53] Motor Loon misses the little ninja pony... ;-(
[12:53] Andrew Linden: Right Motor, that script would have to be written, and the other for automatic import.
[12:53] Simon Linden: I know the performance of purging and actually reclaiming that space is really slow
[12:53] Andrew Linden: Right now it appears that we're writing some maintenance scripts for purging old beta accounts
[12:54] Motor Loon: should help atleast if not solve the problem fully
[12:54] Andrew Linden: once they are done it would probably be possible to connect those scripts to a cron job and some time threshold
[12:54] Rex Cronon: it would be nice to give a warning to somebody before u purge their account on beta...
[12:55] Nalates Urriah: I would guess the scripts used in the great DB cleanup of 2012 on the main grid could be used... to help out, yes, no?
[12:55] Motor Loon: if you havent been there for 3 months Rex... you're not gonna miss it
[12:55] Meeter: Timecheck : User Group is almost over
[12:55] Rex Cronon: well people need to be told that is what will happen
[12:56] Andrew Linden: Yeah, the fallout of any beta grid cleanup is not yet known. We may have an update by the Thursday Beta User Group meeting
[12:58] Nalates Urriah: That would be nice. We can publicize that the topic is up for discussion. See who shows up.
[12:59] Motor Loon: its probably mesh stuff thats taking up the extra inventory server space eh
[12:59] Simon Linden: I hope it can be done with older accounts that have never logged into Aditi
[13:00] Simon Linden: Being mesh or not doesn't really make a difference ... the inventory contains the info about all your folders and what assets are there, but not the asset data itself
[13:00] Meeter: Thank you for coming to the Server User Group
[13:00] Andrew Linden takes notes -- will try to learn more about the aditi problems by Thursday.
[13:00] Motor Loon: oh yeah thats right, asset seperate
[13:01] Motor Loon: Alrighty then, good meeting
[13:01] Nalates Urriah: So,,, there is only one asset db for AGNI and ADITI?
[13:01] Simon Linden: It's not really a database, Nal - it's just storage
[13:01] Qie Niangao: Thanks, Lindens... and congratulations on the interestlist progress, and the large group list deployment.
[13:01] Simon Linden: The inventory database contains the info about where those assets are
[13:02] Nalates Urriah: ok... but just one storage place then...
[13:02] Nalates Urriah: Thz Simon. Thank you lindens.
[13:02] Andrew Linden: The asset storage is just files on a big "hard drive" where "hard drive" may be a stack of servers and various backups.
[13:02] Inara Pey: Thanks, Andrew, Simon, Baker
[13:02] Lares Carter: Thanks for the meeting
[13:03] MartinRJ Fayray: goodbye everyone!
[13:03] Baker Linden: Have a great day everyone!
[13:03] Rex Cronon: tc everybody and have a nice day
[13:03] Rex Cronon: :)
[13:03] Inara Pey: And you!
[13:03] Simon Linden: Thanks everyone for coming today and the great conversation
| Prev 2012.11.20 | Next 2012.12.04 |