Open Source Meeting/2009-01-29

Agenda

None posted

Transcript

[14:01] Q Linden: hey all
[14:01] Morgaine Dinova: Hiya Rob
[14:01] Atashi Toshihiko: Hello Rob
[14:01] Rob Linden: hi folks
[14:02] Adz Childs: :)
[14:03] Rob Linden: https://wiki.secondlife.com/wiki/Open_Source_Meeting/Agenda
[14:03] Rob Linden: (which doesn't appear to have anything on it)
[14:03] Rob Linden: skims the sldev@ traffic for items....
[14:04] Morgaine Dinova: I've barely looked at SLdev this year
[14:05] Rob Linden: discussion on the list has been about webmap api change, openal+mp3, some talk about llGetAgenLanguage, and build issues
[14:06] Rob Linden: pretty small group here today
[14:06] Morgaine Dinova: No Aimee, some catastrophe must have happenened ... tsunami? ;-)
[14:07] Thickbrick Sleaford: hello everyone
[14:07] Rob Linden: hi
[14:07] Morgaine Dinova: Hi Think
[14:07] Morgaine Dinova: Er, Tick
[14:07] Thickbrick Sleaford: heh
[14:07] Morgaine Dinova: Eeeek, my typing ...
[14:07] Morgaine Dinova: Hiya Gwen
[14:07] Q Linden: Hi Think, therefor Hi Aimee
[14:07] Gwen Hermit: hi all
[14:08] Rob Linden: so, anyone have anything in particular they want to bring up now?
[14:09] Rob Linden: alternatives: have Lindens give status updates on their stuff, and going through list of source issues on JIRA
[14:09] Soft Linden: yow, totally agenda-free
[14:09] Gwen Hermit: only random thing i've got is that the viewer seems to crash on linux if the kernel is using a 4K stack
[14:09] Gwen Hermit: and it takes the whole system down with it
[14:09] Rob Linden: what's the default?
[14:09] Gwen Hermit: might be just a weird thing with the nvidia drivers but other 3D apps work fine
[14:10] Rob Linden: I'm assuming setting the stack size is a custom kernel recompile
[14:10] Gwen Hermit: default is 8K i believe?
[14:10] Gwen Hermit: yeah
[14:10] Q Linden: "Don't do that."
[14:10] Gwen Hermit: like i said, fairly random issue
[14:10] Gwen Hermit: and only really occurs in weird scenarios
[14:10] Rob Linden: Q: that's exactly the reference I was going to make :)
[14:10] Gwen Hermit: though i do wonder what the viewer could be doing differently
[14:11] Rob Linden: "doctor doctor, it hurts when I do this"
[14:11] Q Linden: it's the recursive search through your email backlog
[14:11] Soft Linden: There's one thing we do differently than most any other real time app
[14:11] Soft Linden: It's the INSANE amount of texture transfer we do for a single frame.
[14:11] Soft Linden: Because we all love these zomg huge 1024x1024x32 textures, and lots of 'em.
[14:11] Gwen Hermit: heh
[14:12] Q Linden: I have a whole texture on this pixel right here.
[14:12] Gwen Hermit: i only ran into this bug when i was messing with ndiswrapper
[14:12] Soft Linden: That's like my render Moriarty. I always want to look back to that.
[14:12] Gwen Hermit: setup 4K stack to get ndiswrapper running
[14:12] Gwen Hermit: ndiswrapper didn't work, so i stopped using it but kept the custom kernel
[14:12] Soft Linden: Out of curiosity, has anyone here ever played with capping texture sizes to see if that helps on a low end system?
[14:12] Gwen Hermit: however, i did get crashes on SL
[14:13] Gwen Hermit: of the wonderful corrupt video output + broken record mplayer variety
[14:13] Soft Linden: If it turns out that limiting textures to 256 or 512x512 makes SL suddenly sing on an older Linux box or a netbook, you could suddenly be the biggest hero in SL.
[14:14] Morgaine Dinova: The lowest is 32x32, right?
[14:14] Thickbrick Sleaford: what about automatic atlasing? or would that be too expensive?
[14:14] Soft Linden: I don't know if 32x32 is the lowest. That's what sculpted prims use, but they might sample out of a 64x64.
[14:14] Soft Linden: There are 5 discard levels?
[14:15] Soft Linden: That would jibe with 32x32, if 1024x1024 is inclusive in that 5.
[14:15] Morgaine Dinova: Seem to recall Qarl saying there was some lower limit in jpeg2k, although I might have misheard.
[14:15] Rob Linden: there's probably utility in fixing the bugs triggered by having a small stack. I highly doubt we'd dedicate an engineer to doing it, but patches to fix problems there may still be considered useful (depending on the nature of the patch)
[14:16] Saijanai Kuhn: 64x64 for sculpties I think. Need that 33rd vertex
[14:16] Soft Linden: I thought they automatically wrapped around
[14:16] Saijanai Kuhn: though oblongs have their own minimum
[14:16] Zai Lynch: hm... i just tried to upload a texture with just 1 pixel. it gave me an error
[14:16] Morgaine Dinova: Hehe
[14:17] Saijanai Kuhn: probably a good thing...
[14:17] Thickbrick Sleaford: I'm pretty sure I've seen an 8x8 texture
[14:17] Gwen Hermit: rob - the problem with the small stack issue is it's such an odd setup
[14:17] Gwen Hermit: it'd be quite difficult to hunt down the cause
[14:19] Soft Linden: If you really wanted to track that down, I guess you could modify the kernel's exception handler to do kernel stack integrity checking on exit. Wouldn't be surprised if there were already a build option for it.
[14:19] Rob Linden: understood. like I said, not a pressing priority for us. I imagine Moore's law will make that sort of issue mostly moot
[14:19] Gwen Hermit: heh, for myself i've since updated my distro anyway
[14:19] Morgaine Dinova: Talking of which, how are we positioned for exploiting multiple cores in the future?
[14:20] Soft Linden: Or whoops - 2.6 doesn't use interrupts for kernel entry anymore
[14:20] Soft Linden: it's sysenter or whatever the funky Intel thing is.
[14:20] Soft Linden: Still - there's a standard handling point where you should be able to add or activate stack checking, and throw a signal back at the calling process to see what it was doing when it caused the overflow
[14:21] Soft Linden: stop signal, so you can attach gdb and poke at it
[14:21] Rob Linden: multicore - not sure. we're mainly just looking at making things generally more modular, which makes multicore optimizations at least more tenable
[14:21] Gwen Hermit: how do i attach gdb when the whole machine has completely locked up?
[14:21] Gwen Hermit: seriously, the whole thing completely locks up, no response across the network, nothing
[14:22] Morgaine Dinova: Bus lock? You mean the PCIe bus locking owing to some nastiness with the card?
[14:22] Morgaine Dinova: Ouch
[14:23] Gwen Hermit: shrugs
[14:23] Gwen Hermit: just corrupted video
[14:23] Gwen Hermit: and my mplayer instance in the background turns into a broken record
[14:23] Gwen Hermit: i get to hear trent reznor saying "a a a a a a"
[14:23] Rob Linden: GL problems tend to manifest themselves in drastic ways like that
[14:24] Q Linden: and that's different? ;)
[14:24] Zai Lynch: got to be the new album
[14:24] Morgaine Dinova: WOuld the viewer code even be relevant at that level? I'd have though the problem would have to be somewhere in nVidia/ATI territory, either in the driver or the libs.
[14:24] Gwen Hermit: it's odd because i haven't seen such a drastic crash on a linux box since the 90s, literally
[14:24] Gwen Hermit: morgaine - it appears to be, since other heavy 3D apps are fiine
[14:24] Gwen Hermit: *fine
[14:25] Rob Linden: Gwen: often we just tickle different parts of the video driver than other apps
[14:25] Rob Linden: ...so while the problem may only appear in Second Life, the root cause may still be in the driver
[14:25] Gwen Hermit: i know that VBOs tend to cause issues too
[14:25] Q Linden: could be network instead of video?
[14:25] Gwen Hermit: not full machine lockups
[14:26] Gwen Hermit: but still, causes that one process to crash, and sometimes takes down the X server
[14:26] Gwen Hermit: Q - what's your reasoning there?
[14:26] Rob Linden: network drivers are in kernel space too
[14:26] Q Linden: well, only that if you're blowing the stack, any system call could do it.
[14:27] Gwen Hermit: good point
[14:27] Rob Linden: if you're fooling around with ndiswrapper
[14:27] Soft Linden: Gwen - you'd make the stack larger so an overflow doesn't kill you, then you'd drop a value at the top of what would have been a 4k stack
[14:27] Gwen Hermit: rob - ndiswrapper wasn't loaded at this point
[14:27] Soft Linden: And then each time sysenter is used, you check if that value got overwritten on returning back to userland
[14:27] Rob Linden: ah, ok...well, so much for that part of the theory
[14:27] Gwen Hermit: ndiswrapper didn't support my card, so i stopped loading the ndiswrapper module but kept the custom kernel
[14:27] Soft Linden: You'd drop that marker check in /usr/src/linux/arch/i386/kernel/entry.S
[14:28] Soft Linden: at ENTRY(ret_from_sys_call)
[14:28] Gwen Hermit: i don't think i'm going to go to the fuss now that it works again having updated
[14:28] Gwen Hermit: if others have the same issue (which i'd think would be quite rare) then it might be worth doing something
[14:29] Soft Linden: Raise it on sldev if you see it? Certainly makes me curious!
[14:29] Gwen Hermit: i'll drop some details on slde
[14:29] Gwen Hermit: *sldev
[14:29] Morgaine Dinova: Rob, you mentioned the general trend towards making things more modular. How do we stand on use of threads in that area, well modularized or needs attention? (Not looked at that yet)
[14:29] Morgaine Dinova: Hiya CG
[14:29] Soft Linden: In the end though - wouldn't be surprised if it were in the bowels of nvidia's gl driver, and if they wouldn't work with ya on a nonstandard stack size.
[14:30] Gwen Hermit: yeah, it seems likely some feature used by SL which just happens to not be used by my other OpenGL apps
[14:30] Gwen Hermit: one more reason for open source drivers.....
[14:30] Morgaine Dinova: Aye
[14:30] Rob Linden: Morgaine: I'm not privy to a lot of the details at the level that you're probably interested in.
[14:31] Soft Linden: god, I wish there were -something- well-performing with open drivers. the x1300 is about as good as it gets :(
[14:31] Atashi Toshihiko: I get complete linux system lock-ups on my netbook, when using recent SL viewers - the only thing that works is ctrl-alt-delete which tells me that init is still running. But I can't even kill X windows. It's an Intel 945 chip in there though, not nvidia. SL 1.18 works, but nothing more recent.
[14:31] Soft Linden: er 3100. Or whatever the Intel chip is after the gma950 series
[14:31] Gwen Hermit: don't get me on the rantfest of manufacturers who think they need to keep their drivers secret....
[14:31] Morgaine Dinova: X3100
[14:31] Rob Linden: Morgaine: generally getting things into their own processes (like what SLVoice already is) is a direction that would definitely help out in that area
[14:32] Morgaine Dinova: Yes, definitely
[14:32] Q Linden: morgaine -- we're not yet focused on multithreading the viewer, but we ARE focused on refactoring improvements and beginning to bring modules under unit test
[14:32] Q Linden: Which will, as we do it, begin to enalbe the more interesting kinds of refactoring that would let us take advantage of multicore
[14:32] Q Linden: but we're just heading down that path right now
[14:33] Soft Linden: graphics is a -huge- patent minefield. Even the most obvious things a student would come up with are patented. I wonder sometimes if half the reason for keeping drivers closed is to avoid living in the courthouse.
[14:33] Morgaine Dinova: Q: super, scripted testing is an area I've got on my horizon too.
[14:33] Q Linden: as we get our frameworks in place, we'll be asking for help from the community
[14:33] Morgaine Dinova: The viewer is simply too complex for manual testing with any good degree of coverage.
[14:33] Q Linden: but first we have to get our feet under us
[14:33] Q Linden: agreed!
[14:34] Morgaine Dinova: Soft: ouch! You could be right :-(
[14:34] Gwen Hermit: crazy suggestion
[14:35] Gwen Hermit: probably based on the fact i've got a big thick erlang textbook sitting next to me
[14:35] Rob Linden: yeah, certainly, the point at which we have a good unit testing framework in place is a whole new vector for colaboration
[14:35] Gwen Hermit: refactor stuff so that every inworld object is some kind of lightweight process/actor
[14:35] Gwen Hermit: and then route network messages that move objects straight to the object in question
[14:36] Morgaine Dinova: Gwen: great, Erlang is an excellent way forward given current CPU directions
[14:36] Gwen Hermit: problem is that the viewer is C++, not erlang ;)
[14:36] Soft Linden: Gwen - it pretty much is like that on the sim side. There's an LLTask object that covers virtually everything.
[14:36] Rob Linden: Gwen, even just having obvious things like the inworld browser operating as a separate process is a big win
[14:36] Morgaine Dinova: And LL's interest in RabbitMQ ties into use of Erlang well.
[14:36] Soft Linden: Similar to the ViewerObject
[14:36] Q Linden: so you'll be disappointed when I release my secret Haskell-based viewer?
[14:37] Soft Linden: There's central dispatching though. Not routing to individual objects.
[14:37] Morgaine Dinova: Q: not at all :-)
[14:37] Gwen Hermit: come to think of it..... the ViewerObject class is quite close to an actor/process without persistent running code
[14:37] Gwen Hermit: you don't need to actually run anything persistently, just when a "message" comes in
[14:38] Soft Linden: Right. That's pretty common in games. A few methods that get called on all the objects round robin. May as well be a task scheduler as a loop.
[14:39] Soft Linden: Thing is - parallelizing that is only useful if it makes up a substantial share of the execution time
[14:39] Gwen Hermit: come to think of it, that's pretty much how any SL-compatible sim or other multiuser server app i've coded myself works
[14:39] Soft Linden: I think more time is spent in GL calls, etc which are tougher.
[14:39] Rob Linden: switching gears a little bit: how are the build instructions looking to everyone these days?
[14:39] Gwen Hermit: is it actually possible to do the GL calls in multiple threads?
[14:39] Q Linden: gaaaah please no
[14:39] Soft Linden: There's a switch you can flip on the mac that makes GL threaded, so long as you avoid a few dangerous combinations of activities. Do Linux and Windows have similar?
[14:40] Morgaine Dinova: Gwen: every non-trivial app evolves into becoming event-driven eventually ;-)
[14:40] Soft Linden: I don't know if there's a formal GL threading standard.
[14:40] Gwen Hermit: multithreaded OpenGL would seem an unnatural idea - the physical card is still a bottleneck
[14:40] Q Linden: And GL itself is a big honkin' state machine
[14:40] Gwen Hermit: bingo
[14:41] Gwen Hermit: but.... how much of the tricky stuff is actual OpenGL calls vs all the processing before sending off each frame?
[14:41] Soft Linden: Right. There's still a lot that can be done like locking/unlocking vertex buffers and textures though. The GPU might be blasting away with triangle lists and shaders while the app still wants to modify/create textures/lists for the next frame.
[14:41] Q Linden: there was a gfx card vendor that tried to do, um, "Tile-based" rendering? It was an attempt to do rendering independently for sections of the video card.
[14:41] Soft Linden: Yeah, that was PowerVR first, I think.
[14:42] Soft Linden: Apple just signed on as a licensor for that tech - speculation is that it's for the next generation iPhone, or some kind of game system
[14:42] Entering god: mode, level 200
[14:42] Gwen Hermit: heh, that method is used for commercial CGI films
[14:42] Leaving god: mode, level 200
[14:42] Gwen Hermit: pixar used it one title - unsure which one
[14:42] Morgaine Dinova: If the future brings us parallel GL, great, we'll use it. But really our goal is to factor our viewer internals to allow harness concurrency in the future, which is orthogonal to GL.
[14:43] Soft Linden: Sony actually played with depth-based tiling instead of screen position tiling. Nutty :)
[14:43] Rob Linden: Morgaine: what sort of viewer work are you doing?
[14:43] Morgaine Dinova: I'm on the Imprudence team, just started.
[14:43] Soft Linden: They had 16 rendering engines running in parallel that shared their portion of the frame buffer, and then a hardware combiner that pulled all the frames together as well as their Z buffers and mixed it all up.
[14:44] Soft Linden: Ran with 16 cards, each with two PS2 GPUs.
[14:44] Gwen Hermit: heh, sounds like an expensive bit of kit for the retail market....
[14:44] Thickbrick Sleaford: is that substantialy different from SLI?
[14:45] Soft Linden: I think SLI slices the screen horizontally, does top and bottom halves, doesn't it?
[14:45] Morgaine Dinova: Rob: my research background is in parallelism, so the questions stem mainly from that, rather than from Imprudence interest.
[14:45] Soft Linden: So it's like double or triple tiling. Big tiles.
[14:45] Q Linden: usually alternating lines
[14:45] Thickbrick Sleaford: used to do interleaved scan lines, in its previous incarnation, I think
[14:45] Q Linden: scan-line interleaved
[14:45] Soft Linden: Interesting
[14:46] Soft Linden: Must be pretty unfortunate when you want to source the previous frame as a texture, moved one pixel down. :)
[14:47] Atashi Toshihiko: I have another noob Opensource question - so I compile the viewer with debug options - how can I monitor or track what it's doing?
[14:47] Rob Linden: Atashi: which platform?
[14:47] Atashi Toshihiko: I would like to try and find out what happens when the Mac viewer hangs for a few seconds, and I'd like to have some kind of log or console that tells me second-by-second what is happening
[14:47] Gwen Hermit: Atashi - logs and gdb
[14:48] Atashi Toshihiko: Mac OS X 10.5.6
[14:48] Soft Linden: If you run it from a terminal, you'll also see the log information in real time. And debug mode does print a whole lot extra.
[14:49] Atashi Toshihiko: Ok - so if I launch SL from the terminal, the debug info scrolls on the console? Or I can redirect it to a log... that will help :)
[14:49] Aimee Trescothick: run it under xcode on the mac
[14:49] Gwen Hermit: ./secondlife &>secondlife.log
[14:49] Atashi Toshihiko: thank you :)
[14:49] Aimee Trescothick: and you can get the debug console in a window
[14:49] Gwen Hermit: in another terminal do tail -f secondlife.log
[14:50] Aimee Trescothick: Run menu, Console
[14:50] Gwen Hermit: actually, here's something
[14:50] Rob Linden: "less -F" is also handy....similar to tail -f, only with better scrollback
[14:50] Gwen Hermit: LL's sims log to syslog by default
[14:50] Gwen Hermit: as do the other server apps
[14:50] Morgaine Dinova: Add stderr to it, just in case .... viewer > log 2>&1
[14:50] Gwen Hermit: is there a fast way to do that with the client?
[14:50] Aimee Trescothick: but as you're on a mac, just use xcode, much easier :D
[14:51] Atashi Toshihiko: cool, thank you :)
[14:51] Soft Linden: A very fun/interesting way to learn more about the viewer is to find a bunch of the functions with process_ in the name and set breakpoints on those. Those are called as a result of messages from the simulator. Learning which ones are hit the most will put you well on the way to understanding the viewer.
[14:51] Morgaine Dinova: Soft: thankee!!!!
[14:51] Morgaine Dinova: Any more hints like that, much appreciated!
[14:51] Gwen Hermit: syslog for the viewer - simple way to do it?
[14:51] Soft Linden: If you spend too long looking at one when it's called, you'll need to restart the viewer as your connection will time out. But there are only about 10 of those that make up a good 99% of the traffic.
[14:52] Rob Linden: er...I take that back about less, it's "F" command after running less that gets you the tail -f behavior
[14:52] Gwen Hermit: don't have the precise reference to hand but somewhere in llapp::init() for the server code it sends to syslog
[14:52] Rob Linden: anyhow....other viewer development questions?
[14:52] Gwen Hermit: (thanks btw for giving us bits of server code in llcommon!)
[14:53] Rob Linden: Gwen, what's your development focus?
[14:53] Gwen Hermit: rob - most recently, not much, in general alternative servers and interop which goes beyond OGP
[14:53] Soft Linden: I wish we'd move more server code there. Things like llThrottle proved very useful on doing that.
[14:54] Soft Linden: For anyone looking for a good first project, actually - finding anything that can be done enough to be disruptive and trying to put an llThrottle on it is a nice one.
[14:54] Gwen Hermit: you all remember the controversial videos i made of myself TPing out of agni with my groups, avatar and inventory i presume?
[14:54] Gwen Hermit: anyway.... syslog
[14:55] Gwen Hermit: is there a quick way to toggle llapp to use syslog for the client?
[14:55] Gwen Hermit: you could easily start llapp in "server mode", but i'm guessing that alters other behaviours too
[14:57] Soft Linden: Not sure it would be very useful - still instructive though?
[14:58] Gwen Hermit: it's always good to use standardised logging ;)
[14:58] Gwen Hermit: even for desktop apps
[14:59] Rob Linden: how much more standard do you get than stdout? ;-)
[14:59] Rob Linden: well, we're just about out of time here
[14:59] Soft Linden: Ha. I guess our formatting could be much more standard, though. Positively schitzophrenic.
[15:00] Morgaine Dinova: Easy enough to pipe the viewer into logger ;-)))
[15:00] Soft Linden: I'd love to see what a logwatch specification would look like to cover all our output.
[15:00] Gwen Hermit: i like the formatting of the stdout log
[15:00] Thickbrick Sleaford: do you really want your syslog filled with stuff like "display_stats: FPS: 18.62"
[15:00] Thickbrick Sleaford: ?
[15:00] Gwen Hermit: thickbrick - stuff like that can be configured
[15:00] Gwen Hermit: i'm thinking of things like what sims i was connecting to could be useful to have in syslog
[15:00] Gwen Hermit: call me weird
[15:01] Soft Linden: The region IDs may be there, but probably not names. Yeah.
[15:01] Morgaine Dinova: Just pipe it into logger, using a local facility and a syslog config that dumps the output into a separate log, job done.
[15:01] Soft Linden: But ya - timing out - thanks for coming, all!
[15:02] Morgaine Dinova: Thanks all, take care :-)
[15:02] Aimee Trescothick: bye :)
[15:02] Atashi Toshihiko: Take care everyone :)
[15:02] Rob Linden: feel free to stay around an chat. thanks again for coming!
[15:02] Adz Childs: :)
[15:02] Rob Linden: see ya
[15:02] Morgaine Dinova: waves

Open Source Meeting/2009-01-29

Agenda

Transcript

Navigation menu

Search