Open Source Meeting/2008-10-16

• [14:01] Carjay McGinnis: Hello invisible Rob
• [14:01] Rob Linden: howdy!
• [14:01] Khyota Wulluf: crap hope i can make it
• [14:01] Rob Linden: I'm a ninja!
• [14:01] Squirrel Wood: Mmm. Ninjas ^^
• [14:01] Carjay McGinnis: hehe
• [14:01] Khyota Wulluf: hi there swuirrel
• [14:01] Rob Linden: still invisible?
• [14:01] Khyota Wulluf: squirrel
• [14:01] Squirrel Wood: Hellos ^^
• [14:01] Carjay McGinnis: Helo Brad
• [14:02] Brad Linden: Hi
• [14:02] Khyota Wulluf: seen you a couple times already ^^
• [14:02] Carjay McGinnis: yes, for some strange reason both you and Brad
• [14:02] Carjay McGinnis: and q, hmm
• [14:02] Carjay McGinnis: this is strange
• [14:02] Rob Linden: Lindens, cloaks off! :-)
• [14:02] Squirrel Wood: I has an acorn
• [14:02] Carjay McGinnis: hehe
• [14:02] Q Linden: what's strainge?
• [14:02] Q Linden: besides my spelling, that is
• [14:02] Rob Linden: apparently, we're all invisible to Carjay
• [14:02] Carjay McGinnis: all Lindens invisible
• [14:02] Carjay McGinnis: once I zoom in it's ok
• [14:02] Q Linden: / we didn't tell him about the romulan takeover?
• [14:02] Rob Linden: I wonder if its an imposter issue then
• [14:02] Carjay McGinnis: something with LOD
• [14:03] Carjay McGinnis: hehe
• [14:03] Carjay McGinnis: imposters are off for me
• [14:03] Carjay McGinnis: that's the strange thing
• [14:03] Q Linden: well, you've uncovered us then.
• [14:03] Q Linden: we're all imposters
• [14:03] Q Linden: damn, did I use my outside voice?
• [14:03] Carjay McGinnis: hehe
• [14:03] Squirrel Wood: Holy Crip, He's a Crapple!
• [14:04] Rob Linden: well, we actually have an agenda and the right people to talk about it!
• [14:04] Carjay McGinnis: ello Soft
• [14:04] Soft Linden: The agenda is still last week's items.
• [14:04] Soft Linden: Ah, but Brad's here for the first!
• [14:04] Khyota Wulluf: i have a the feeling i really created a show last week :)
• [14:04] Brad Linden: hah
• [14:04] Carjay McGinnis: Hello Aimee
• [14:04] Rob Linden: http://wiki.secondlife.com/wiki/Open_Source_Meeting/Agenda
• [14:04] Aimee Trescothick: hey :)
• [14:05] Rob Linden: Brad is the "workingonit linden" in this case
• [14:05] Brad Linden: one of them at least
• [14:05] Brad Linden:  :)
• [14:05] Carjay McGinnis: ah, ok, unconvered!
• [14:05] Carjay McGinnis: oh, ok
• [14:06] Rob Linden: first agenda item: An update from WorkingOnIt Linden about [VWR-3943
• [14:06] Brad Linden: So, I'll get started...
• [14:06] Brad Linden: we've been looking at replacing smartheap with another heap library which'll help us debug these crashes better
• [14:07] Brad Linden: google's tcmalloc is the first one we're evaluating for that purpuse
• [14:07] Mm Alder: Have you made any progress on the bug itself?
• [14:08] Brad Linden: not exactly, the problem is it can be any of several bugs that show up similarly with the same error message
• [14:08] Mm Alder: What would those "several" be?
• [14:08] Brad Linden: that's why we're looking at tools like tcmalloc that will hopefully let us record more data about such crashes and get the information to fix them
• [14:09] Brad Linden: basically there are probably a bunch of places in the viewer where we're freeing memory twice or otherwise using memory after it's been freed
• [14:10] Brad Linden: and we need a systematic way to track each one down and make sure new ones don't get introduced
• [14:10] Brad Linden: there are tons of tools out there to help us do it
• [14:10] Saijanai Kuhn: hey all
• [14:10] Saijanai Kuhn: Haveyou tried the programming tools that Apple supplies?
• [14:10] Mm Alder: Most people don't have this problem. Do you see any similarities in the configurations of those experiencing the problem?
• [14:10] Brad Linden: I personally haven't but I think someone else has
• [14:11] Brad Linden: I doubt it has anything to do with their system configurations
• [14:11] Mm Alder: Why?
• [14:11] Saijanai Kuhn: they claim to do a lot of that stuff, and include recording of user interaction so you can run multiple tests with the same input
• [14:11] Brad Linden: it's most likely something caused by the places visited in world
• [14:12] Brad Linden: certain types of content may be more likely to trigger such errors
• [14:12] Carjay McGinnis: sounds like you don't see this error much during your tests?
• [14:12] Carjay McGinnis: I'm under Linux so never saw it but didn't experience it on Windows either so far
• [14:12] Brad Linden: we don't see it much in internal qa
• [14:12] Mm Alder: Do you see it at all?
• [14:13] Carjay McGinnis: hm, tough one then
• [14:13] Brad Linden: yeah
• [14:13] Mm Alder: Is it reproduceable?
• [14:13] Carjay McGinnis: I see random corruption but very very rarely
• [14:13] Brad Linden: no, not directly reproducable
• [14:14] Mm Alder: I think all of the reports on the JIRA say that it is very reproduceable for them.
• [14:14] Brad Linden: so the plan is to get a library like tcmalloc into the viewer to help record the heap state when crashes happen, and add extra info to these crash reports
• [14:14] Carjay McGinnis: will that slow down the viewer much? how is your experience with it?
• [14:14] Brad Linden: then we can start tracking down where the corruption happens
• [14:15] Brad Linden: when the extra logging is enabled from a debug setting, it slows things down to about 5fps
• [14:15] Brad Linden: from one of our tests
• [14:15] Carjay McGinnis: ok, so nothing for a release version
• [14:15] Rob Linden: Mm: is this a problem you see a lot?
• [14:15] Mm Alder: Have you been able to figure anything from the error reporting now?
• [14:16] Brad Linden: some of them
• [14:16] Mm Alder: Rob: I don't see it at all. I just think it needs to be fixed.
• [14:16] Brad Linden: but this is like whack-a-mole, you fix one heap corruption bug, and you have no idea how many are left hiding
• [14:17] Brad Linden: so it'll take time to get all of them fixed
• [14:17] Rob Linden: I'm assuming we don't have any sense of whether or not any of these problems are smartheap induced, do we?
• [14:17] Mm Alder: So why are some people so consistently unlucky about hitting the bugs?
• [14:17] Brad Linden: smartheap is likely not the cause, just a symptom
• [14:17] Q Linden: They're creatures of habit?
• [14:18] Rob Linden: Mm: it could be anything from content accessed, to hardware, to time of day
• [14:18] Brad Linden: they might be visiting places regularly in world that tickle the bug
• [14:18] Brad Linden: they might have an avatar that tickles the bug
• [14:18] Mm Alder: That would me it should be reproduceable for you as well.
• [14:18] Mm Alder: *mean*
• [14:18] Carjay McGinnis: hm, why?
• [14:19] Rob Linden: Mm: if we could isolate which of the thousand variables it is
• [14:19] Mm Alder: That's why I think it would be useful to look for commonalities in the configurations
• [14:20] Mm Alder: I think all of the reporters have high end systems
• [14:20] Carjay McGinnis: I think it's better to improve the tols
• [14:20] Carjay McGinnis: tools
• [14:20] Carjay McGinnis: get more information
• [14:20] Soft Linden: The high end system aspect is interesting.
• [14:20] Rob Linden: Mm: that's what getting tcmalloc wired up with the crash reporter buys us
• [14:21] Mm Alder: Rob, you can look now to see if there's anything there
• [14:21] Soft Linden: / This was why I had asked people check their texture memory setting. On my system, I was getting that error non stop until I reduced the detected vram to a sane amount.
• [14:21] Brad Linden: that's true
• [14:21] Carjay McGinnis: ah, I read that
• [14:21] Brad Linden: there are some recent graphics card drivers that have a known memory leak
• [14:21] Mm Alder: Can you contact residents directly to see if that fixes it for them?
• [14:21] Brad Linden: which triggers the smartheap error
• [14:22] Soft Linden: No affirmative responses on strange amounts detected though. If someone wants to help follow up with individual commenters it would be helpful.
• [14:22] Soft Linden: That doesn't have to be a Linden. The names are right in the JIRA comments.
• [14:22] Saijanai Kuhn: harpng back to the Apple INstruments tool for a sec. Seems to me it would be of value to use, especially if the memory leaks are consistent x-platform http://developer.apple.com/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/AboutTracing/chapter_2_section_4.html#//apple_ref/doc/uid/TP40004652-CH9-SW11
• [14:23] Rob Linden: Mm: getting tcmalloc wired in is going to be useful no matter what, not only in diagnosing the current batch of problems, but new issues that come up as well
• [14:23] Carjay McGinnis: indeed
• [14:23] Mm Alder: What is the outlook for getting tcmalloc in a RC?
• [14:23] Q Linden: It also, in my experience, helps with performance
• [14:23] Brad Linden: yeah, I'll definitely look into the apple tools again as well
• [14:23] Brad Linden: 1.23 is the tentative schedule
• [14:23] Q Linden: (my experience in previous lives, that is)
• [14:24] Brad Linden: so we'll be continuing to work on this with the tools we have during the 1.22 RC cycle
• [14:24] Rob Linden: Brad: is Smartheap going to be disabled in the 1.22 RC cycle?
• [14:25] Brad Linden: likely yes
• [14:25] Mm Alder: Will tcmalloc be able to find qraphics driver leaks?
• [14:25] Brad Linden: smartheap is incompatible with visual studio 2005 and later
• [14:25] Brad Linden: and from 1.22 on our standard builts will be made with 2005
• [14:25] Brad Linden: (unless stuff breaks and we have to revert)
• [14:25] Brad Linden:  ;)
• [14:26] Carjay McGinnis: hm, why did you wait so long to switch to VS2005?
• [14:26] Rob Linden: Havok
• [14:26] Mm Alder: What are the major items coming in 1.22?
• [14:26] Carjay McGinnis: thought that was no longer an issue?
• [14:26] Rob Linden: it's not :)
• [14:26] Brad Linden: we couldn't even start on the switch until havok was done
• [14:26] Carjay McGinnis: ah, ok, never change a running system
• [14:26] Carjay McGinnis: or if, then do it veeeeery slowly
• [14:27] Brad Linden: we just recently got it set up so our automated nightly build system could use both 2003 and 2005
• [14:27] Soft Linden: In a nutshell, no VS2005 libs for havok1, and we didn't want people using a different compiler for viewer and simulator development.
• [14:27] Carjay McGinnis: yes, reasonable
• [14:27] Carjay McGinnis: it's just that I thought havok4 was here for months
• [14:27] Rob Linden: hopefully I'm not jinxing things, but my understanding is that future Havok upgrades won't be nearly as painful as this one was
• [14:28] Brad Linden: smartheap was another reason we hadn't been switching
• [14:28] Soft Linden: Yes, but our release pipeline is pretty deep.
• [14:28] Mm Alder: What's in store for 1.22?
• [14:28] Brad Linden: we needed to do some testing to make sure that removing it didn't decrease the framerate
• [14:28] Carjay McGinnis: oh, ok, understood
• [14:28] Rob Linden: Mm: I think you can look at the latest release branch and get a pretty good idea
• [14:29] Rob Linden: doesn't remember any of the big ticket items
• [14:29] Brad Linden: 1.22 is primarily a maintenance release as far as I know
• [14:29] Brad Linden: lots of small bugfixes
• [14:29] Brad Linden: some render compatiblity stuff
• [14:29] Vent Sinatra: (ohoh :)
• [14:29] Mm Alder: We like bugfixes. :-)
• [14:29] Brad Linden: i think there was a batch of featurettes
• [14:30] Rob Linden: has 1.22 branched from release yet?
• [14:30] Brad Linden: I could be missing some things
• [14:30] Brad Linden: no, there are a couple of things still waiting to merge into release before it branches
• [14:30] Mm Alder: Brad, on a procedural note, why do Lindens rarely respond on the PJIRA?
• [14:30] Rob Linden: forgot that the public version is called "trunk" instead of "release"
• [14:31] Brad Linden: heh
• [14:31] Brad Linden: i can't speak for all lindens on that one Mm
• [14:31] Carjay McGinnis: oh, so Soft didn't pull off the internal rename yet *g*
• [14:31] Mm Alder: How about you on this particular issue? :-)
• [14:31] Soft Linden: No. Internally, trunk/ will come after another restructuring happening in the next quarter or so. Right now, it would create headaches.
• [14:31] Brad Linden: but especially on big issues like VWR-3943 where a lot of people are commenting it becomes very difficult to hold a dialog
• [14:33] Brad Linden: and pull out the useful information from the background
• [14:33] Mm Alder: A "we hear you" every now and then would make people happy.
• [14:33] Brad Linden: honestly
• [14:33] Brad Linden: my priority is getting the bug fixed
• [14:33] Q Linden: Carjay, it may be as simple as the fact that we have such a blizzard of internal communications to manage, adding additional channels is hard. The sustaining group is trying to make sure that we tickle issues as part of our triage process.
• [14:33] Mm Alder: And "we've narrowed it down to..." or "we think it's..." would give them something to work with.
• [14:33] Brad Linden: I'll leave keeping people happy to people who are better at it
• [14:34] Q Linden: We may not engage in chat but we're trying to keep the status up.
• [14:34] Vent Sinatra: assigning issues, or merging would already be of big help
• [14:34] Brad Linden: sorry if that came of snarky
• [14:34] Vent Sinatra: (I know, terrible task)
• [14:34] Rob Linden: Mm: I'd like to avoid going to meta on this sort of thing. what I've found....
• [14:34] Vent Sinatra: (if not impossible)
• [14:35] Rob Linden: ....is that when people in this meeting ask for status on a specific issue....
• [14:35] Rob Linden: ....we're able to get a pretty good answer
• [14:35] Rob Linden: Similar thing holds true for sldev@ (which has been kinda quiet lately)
• [14:35] Mm Alder: Well this one's been around since last December, and it sounds like you've made no real progress.
• [14:35] Rob Linden: the smartheap to tcmalloc issue is a great example of that
• [14:36] Rob Linden: Mm....what are you talking about
• [14:36] Rob Linden: please
• [14:36] Rob Linden: "no real progress"?
• [14:36] Mm Alder: The emphasis is on "sounds like"
• [14:37] Soft Linden: That issue is pretty much "SL runs out of memory sometimes for some people." Anything you can do to help narrow it down into something more actionable would be awesome.
• [14:37] Rob Linden: well, I guess my point is when someone (yourself) asked a pointed question about a specific issue, we got the dev who knows about it to come and talk to you all
• [14:37] Soft Linden: In this case, the right response to something that vague is to put tools in place.
• [14:37] Vent Sinatra: I would love to help.... i think many of us do, but how
• [14:38] Mm Alder: Like Vent said.
• [14:38] Rob Linden: Vent: bug fixes....start small, work your way up
• [14:38] Vent Sinatra: rob, actually i meant when encountering/reporting a particular bug/problem
• [14:38] Vent Sinatra: not 'helping' in general
• [14:38] Soft Linden: Vent/Mm, if you wanted to start pinging individual residents in that thread to collect more details, that would be a great start. Asking questions in the thread pretty much gets lost in all the replies.
• [14:39] Soft Linden: Making sure we have the same kind of information for each resident would be hugely helpful.
• [14:39] Vent Sinatra: mm, ok
• [14:39] Mm Alder: This is a bug fix, and I have tried helping, but if I have a question like "Is it unique to OpenGL 2.1.2 systems?" how do I get an answer
• [14:39] Rob Linden: one caveat on what Soft said: issues that don't have any activity for months is worth pinging in the issue itself
• [14:41] Rob Linden: Mm: on complicated issues like VWR-3943, one thing to do is perhaps start a wiki page or other resource that collects data
• [14:41] Mm Alder: But I can't get crash logs.
• [14:41] Rob Linden: ah, ok....crash logs are another story.
• [14:42] Carjay McGinnis: hm, well, i doubt a crash log would be helpful here
• [14:42] Rob Linden: I don't recall where we last left some of the issues of public accessibility
• [14:42] Rob Linden: the problem with crash logs is that they *might* contain personal info, so we can't just generally make them available
• [14:42] Mm Alder: Carjay, it would help determine if it depends on the configuration.
• [14:43] Carjay McGinnis: oh, ok, you mean the meta information, thought you were thinking about the stack trace, ok, true
• [14:43] Mm Alder: Rob, I understand. It means someone in Linden would have to look at them and work with the OS community.
• [14:44] Mm Alder: If no one is looking at them now, better logging won't help.
• [14:44] Soft Linden: So, we published the top crashers in the past. Release was going to take that over - I'm pretty sure that got dropped, but it's something we should bring back.
• [14:44] Rob Linden: Brad: do you know if an automated version of this is in the works? http://wiki.secondlife.com/wiki/Crash_Reports
• [14:45] Brad Linden: hmm
• [14:45] Brad Linden: I hadn't heard of anything in the works for that
• [14:45] Brad Linden: that would be cool
• [14:45] Rob Linden: ok....I think with all of the shuffling that has been going on, that may have slipped through the cracks
• [14:45] Brad Linden: goes to write a jira for myself to investigate that
• [14:46] Soft Linden: Beast published the last set. Asking for a one-time manual refresh while something like that's going on would be reasonable.
• [14:46] Rob Linden: indeed
• [14:46] Carjay McGinnis: I remember loking at it some months ago
• [14:46] Carjay McGinnis: looking
• [14:47] Mm Alder: But like Carjay said, the stack trace alone may not help much here.
• [14:47] Soft Linden: It may help though. With a leak, there's a good chance that something constantly allocating memory will tend to pop up in the call stack.
• [14:48] Rob Linden: Mm: if you're still on VWR-3943, that's something that we don't yet have good data for. that's what Brad and others is working on wiring up
• [14:48] Vent Sinatra: hmmm, the 'crashing' call might not be the leaker
• [14:48] Carjay McGinnis: hm, to be useful you also need the traces of all other threads
• [14:48] Vent Sinatra: but yeah, its a good start
• [14:49] Mm Alder: Rob, I meant configuration data, which you already get.
• [14:49] Rob Linden: Mm: we don't necessarily get that information correlated with a specific bug
• [14:50] Mm Alder: How do you mine the crash reports now?
• [14:50] Rob Linden: Mm: top crashers
• [14:50] Vent Sinatra: solved 90% of its own crashes with a new video card :)
• [14:51] Rob Linden: video card drivers constitute a large number of the problems, though we're certainly not off the hook
• [14:51] Mm Alder: But what do you look at for the top crashers? Just the stack traces?
• [14:51] Brad Linden: we look at the stack traces and the total logs
• [14:51] Rob Linden: defers to others who have done crash hunting
• [14:52] Brad Linden: you can see all teh info in SecondLifeCrashReport.log
• [14:52] Brad Linden: in %APPDATA%/SecondLife/logs
• [14:52] Mm Alder: I wondered if you had some program to digest the info.
• [14:52] Brad Linden: yeah we do
• [14:53] Mm Alder: Do you have a digest for this bug that you could share?
• [14:53] Brad Linden: I was just talking yesterday with someone about how to better share that tool with the community
• [14:54] Brad Linden: again, the problem is it can be any of several bugs that show up similarly with the same error message
• [14:54] Brad Linden: it's not one bug
• [14:54] Brad Linden: it's a meta-issue
• [14:55] Mm Alder: Does it get logged as a SmartHeap error?
• [14:55] Soft Linden: I think there may be something you're missing - SmartHeap is the general memory allocator here. It's not a "SmartHeap" error - it's an "out of memory" error.
• [14:55] Brad Linden: I believe we recently changed it
• [14:55] Brad Linden: now they'll show up with "bad allocation" at the end of the log
• [14:56] Soft Linden: Trying to treat "out of memory" as a single bug is like trying to treat "crashes" as a single bug.
• [14:56] Brad Linden: so yes, we are collecting them all in one place in the crash reporter
• [14:56] Squirrel Wood: do not forget the poor condition that the average users installations are in. outdated stuff, no updates, partial updates, keyloggers, trojans and other stuff cluttering the installation and making it pretty unstable, ...
• [14:56] Q Linden: ...and 3 competing anti-virus programs
• [14:56] Rob Linden: ....(claiming that the other 2 are viruses) :)
• [14:56] Squirrel Wood: one competing av program (dare I say Norton?) is more than enough ;)
• [14:57] Vent Sinatra: lol
• [14:57] Aimee Trescothick: lol, it's like a software drug overdose
• [14:57] Mm Alder: I guess I just get involved in mysteries. :-)
• [14:57] Mm Alder: I wan't to figure them out.
• [14:57] Brad Linden: heh, yeah this is a tricky one
• [14:57] Rob Linden: k.....wow, one agenda item nearly filled an hour, but I guess it wasn't really one item :)
• [14:57] Mm Alder: All the more reason. :-)
• [14:58] Soft Linden: Yus, and that's admirable. The more people dogging these issues, the better.
• [14:58] Q Linden: just like it really isn't one bug. :)
• [14:58] Mm Alder: I think I hear a dog not barking. :-)
• [14:58] Rob Linden: anything else we need to talk about quickly before we go?
• [14:58] Mm Alder: Thanks for coming and answering my questions, Brad.
• [14:59] Vent Sinatra: (i would love to test it in a. sim with MANY textures are no avatars, and the other we around). But yes, lets move :)
• [14:59] Brad Linden: no problem
• [14:59] Carjay McGinnis: oh, I almost read dodging there...
• [14:59] Brad Linden:  :)
• [14:59] Squirrel Wood: many textures => mall!
• [14:59] Squirrel Wood: like, textures'r'us
• [14:59] Vent Sinatra: mmmm good point quirrel
• [14:59] Carjay McGinnis: try bare rose but there are a lot of avatars around usualy
• [14:59] Vent Sinatra: i'll try that
• [14:59] Squirrel Wood: and bare rose, yes
• [14:59] Vent Sinatra: squirrel
• [15:00] Rob Linden: ok....well, thanks everyone for coming!
• [15:00] Squirrel Wood: Have a good weekend ^^
• [15:00] Vent Sinatra: is going to the mall !:)
• [15:00] Aimee Trescothick: thank you :)
• [15:00] Carjay McGinnis: thank you very much, Brad
• [15:00] Brad Linden:  :)
• [15:00] Carjay McGinnis: thanks Rob, Soft, Q
• [15:00] Vent Sinatra: thank you :)
• [15:00] Squirrel Wood: On to Benjamin ^^
• [15:00] Q Linden: see ya
• [15:00] Brad Linden: no problem
• [15:00] Mm Alder: Brad, stop by more often.
• [15:01] Khyota Wulluf: bye gys
• [15:01] Brad Linden: yeah, I've been meaning to for a while
• [15:01] Brad Linden: I should definitely come more often
• [15:01] Rob Linden: bye everyone!
• [15:01] Home: Going: to home
• [15:01] Carjay McGinnis: bye