Simulator User Group/Transcripts/2012.10.12
| Prev 2012.10.09 | Next 2012.10.16 |
List of Speakers
| Andrew Linden | Ardy Lay | Baker Linden |
| Barrington DeSantis | Inara Pey | Lares Carter |
| Motor Loon | Nalates Urriah | Qie Niangao |
| Rex Cronon | Siana Gearz | Simon Linden |
| TankMaster Finesmith | Theresa Tennyson | Toysoldier Thor |
Transcript
[16:01] Ardy Lay: It has changed directions
[16:01] TankMaster Finesmith: and there hs is
[16:01] Toysoldier Thor: can blame andrew for that wind
[16:01] Ardy Lay: Wind is calming now
[16:02] Toysoldier Thor: lol i see that
[16:02] Inara Pey: Eeps
[16:02] Ardy Lay: No going the other way. ;-)
[16:02] Barrington DeSantis: oops...sorry.
[16:02] Andrew Linden: What's that about the wind? Too frisky?
[16:02] Ardy Lay: -No +Now
[16:02] Toysoldier Thor: ardys stars on the head
[16:02] Toysoldier Thor: the wind was pretty strong here
[16:02] Ardy Lay: I'll let the sailors define what is "good wind".
[16:02] Simon Linden: Hmm, I wonder if I should get a wind sock
[16:02] Toysoldier Thor: lol no just stand ardy up here
[16:03] Andrew Linden: Yeah... I tried to mellow the wind out in the project I'm currently working on...
[16:03] Andrew Linden: I ended up with some instability. I have yet to fix it.
[16:03] Ardy Lay: Advanced - Render Metadata - Wind Vectors
[16:03] Toysoldier Thor: doesnt seem too bad
[16:03] Rex Cronon: hello everybody
[16:03] Ardy Lay: Well, this is supposed to float about like snowflakes.
[16:03] Toysoldier Thor: QIE !!! My fellow Sim Lag Troubleshooter !!!
[16:04] Qie Niangao smiles and waves
[16:04] Andrew Linden: Ok news...
[16:04] Toysoldier Thor: now yoru in trouble cause qie knows what hes talking about - unlike me
[16:04] Inara Pey waves to Qie
[16:05] Andrew Linden: I'm still working on interestlist stuff...
[16:05] Andrew Linden: I thought I'd have something working this week... it isn't quite working right.
[16:05] TankMaster Finesmith: interesting
[16:06] Andrew Linden: You can see it not working on Ahern on aditi
[16:06] TankMaster Finesmith: lol
[16:06] Rex Cronon: at firts i read as interesting too:)
[16:06] Andrew Linden: And if you see bad wind behavior in Ahern... well I know all about it.
[16:06] Rex Cronon: we go fly a kite?
[16:06] Theresa Tennyson: What's not working?
[16:06] TankMaster Finesmith: someone pass a lot of gass?
[16:06] Andrew Linden: Actually the interestlist is working for the most part
[16:07] Andrew Linden: and should be a little faster all around
[16:07] Theresa Tennyson: Did you break all my vehicles again?
[16:07] Rex Cronon: does the interest list work on objects attached to an avatar?
[16:07] Andrew Linden: but there are a few problems... maybe some things that should be in view don't show up...
[16:07] Andrew Linden: We're testing it right now.
[16:08] Andrew Linden: One major bug is that if you teleport in to a spot underground... your camera may be "broken" (cannot be moved or updated).
[16:08] Andrew Linden: So I've got to figure that one out soon.
[16:08] TankMaster Finesmith: oppsie
[16:08] Rex Cronon: if u tp above ground will it fix itself?
[16:08] Ardy Lay: I didn't know we could go underground.
[16:09] Andrew Linden: maybe Rex... you might have to TP out and then back again. Not sure.
[16:09] Andrew Linden: er... what I meant was... if you teleport to a place with high terrain...
[16:09] Andrew Linden: you will arrive underground first, and the simulator will figure that out and move you up to the surface
[16:09] Andrew Linden: and your camera will then be broken.
[16:10] Ardy Lay: Ah, thanks for explaining.
[16:10] Andrew Linden: Nevertheless, I'm confident that it will be faster... once I make it correct.
[16:10] Andrew Linden: There were two performance tests run on an earlier version...
[16:11] Andrew Linden: one test (mostly empty region with about 30 avatars running around) showed a slight decrease in performance... about 5% worse
[16:12] Andrew Linden: another test (crowd of avatars NOT looking at a pile of dynamic objects behind them) showed about 40% improvement (less time spent running the interestlist)
[16:12] Motor Loon: If you're fixing camera anyway, I'd sure like to see my flycam (using spacenavigator) not being messed up at each simcrossing
[16:12] Andrew Linden: so I went back to the code to try to fix the first test
[16:12] Andrew Linden: and I think I've got something that will be as good or better all around
[16:12] Andrew Linden: but we shall see.
[16:13] Rex Cronon: while u r at it, could it be possible to make so that the avatar can say what is the mas size of the interest list?
[16:13] Andrew Linden: Motor, in this case the "camera" I'm talking about is the server's notion of it... which is used to figure out what objects to send to the viewer or not
[16:13] Andrew Linden: the glitches in camera movement you're talking about are viewer bugs
[16:13] Motor Loon: ah
[16:13] Andrew Linden: the viewer tells the server where the camera is
[16:13] Rex Cronon: max size*
[16:14] Andrew Linden: and the server obeys... up to some distance
[16:14] Andrew Linden: The server will not accept camera positions that are too far from the avatar. It will clamp the distance.
[16:15] Andrew Linden: Not in the old (currently on agni) code... the viewer can specify an arbitararily far camera however...
[16:15] Ardy Lay: Did that flycam bug get introduced when the camera gimbal limits were changed?
[16:15] TankMaster Finesmith: so is that why distance objects ont render even if you cam to them from a few sims away?
[16:15] Andrew Linden: the viewer won't actually use that location... it will slave to the agent's position instead.
[16:15] Toysoldier Thor: it will? really??? I have had my cam position go realllll far from my avatar
[16:15] Andrew Linden: Yes TankMaster
[16:16] TankMaster Finesmith: ah ok, thx
[16:16] Toysoldier Thor: like 1500 meters up
[16:16] Andrew Linden: ToySoldier, the server will send updates for some objects that are very high (or low)
[16:16] Toysoldier Thor: ok that makes sense
[16:16] Andrew Linden: so the objects will be visible if you move your camera up there
[16:16] Toysoldier Thor: because some thing - a lot of things - are missing
[16:16] Andrew Linden: but if you move laterally then the object detail will degrade
[16:17] Andrew Linden: in the new code...
[16:17] Andrew Linden: the object detail will follow the camera (not the avatar)
[16:17] Motor Loon: makes much better sense
[16:17] Andrew Linden: as long as you stick within about 128 meters of the avatar
[16:17] Andrew Linden: beyond that the object detail will degrade
[16:18] Motor Loon: why only 128 m ?
[16:18] Toysoldier Thor: lol i remember a requested feature a long time ago that wanted the opposite - that cam views could not pass parcels (and even walls :)
[16:18] Toysoldier Thor: (for avatar privacy reasons)
[16:18] Motor Loon: I very often cam to the next sim to keep an eye on things
[16:18] Simon Linden: I can imagine lots of experience designs that would want that
[16:18] Motor Loon: 256m +
[16:18] Andrew Linden: Motor, that's the distance I'm using right now. Whatever it is... I'd like to keep it small enough so that your camera can't wander into regions that are more than one neighbor region away.
[16:19] Motor Loon: hm ok
[16:19] Motor Loon: well, aslong as you're open to change it down the road if it turns out to not be enough (hehe)
[16:19] Andrew Linden: Motor, you can set your camera draw distance very big and wander up to 128 meters away. You should be able to see more than one neighbor region away.
[16:19] Baker Linden: Hello everyone, sorry I'm late, I was cleaning up the kitchen. We had some cake for my birthday.
[16:19] Simon Linden: Even with that limit the new code should behave better, Motor. It will give you higher detail as you move the camera out far
[16:20] Andrew Linden: Happy Birthday Baker!
[16:20] Qie Niangao: Happy birthday, Baker
[16:20] Motor Loon: (very unfair to mention CAKE when I haven't had any!)
[16:20] Rex Cronon: the viewer can draw something like a ghost where user camera is:)
[16:20] Baker Linden: Sorry Motor
[16:20] TankMaster Finesmith: baker!!
[16:20] Toysoldier Thor: mmmm my chimichangas just arrived in front of my face as i sit here)
[16:20] Rex Cronon: hi baker
[16:20] Baker Linden: Here are some cakes in cup form
[16:20] Rex Cronon: happy birthday:)
[16:20] Lares Carter: Happy birthday Baker!
[16:20] Baker Linden: thank you
[16:21] Motor Loon: better than nothing I guess
[16:21] Baker Linden: lol
[16:21] Baker Linden: Once I saw a cupcake shaped cake made out of cupcakes.
[16:21] TankMaster Finesmith: lol
[16:21] Baker Linden: it was crazy meta
[16:22] Andrew Linden: Simon, what's the news on server updates?
[16:22] Baker Linden: anyway, my news is that I started working on figuring out a bug with teleports. I've been investigating it for a week and a half now, and I may have found a way to repro it consistently
[16:22] TankMaster Finesmith: just one bug? :P
[16:22] Baker Linden: I'm not sure if it's something I can fix, but I'll at least figure out why it's happening
[16:22] Baker Linden: well, a specific one
[16:22] Baker Linden: a timeout issue
[16:22] Simon Linden: Well, next week the code that's currently on BlueSteel and LeTigre will get deployed to the main grid. That happens Tuesday morning
[16:22] TankMaster Finesmith: ah
[16:23] Baker Linden: where attempting to teleport causes a timeout.
[16:23] Simon Linden: That has some database query optimizations for group data requests - not the stuff Baker was working on, but our worst query
[16:24] Simon Linden: On Wednesday there will be two more new sets of code in RC, but unfortunately (from my point of view) not the maintenance release with Baker's work
[16:24] Motor Loon: Oskars network tests on betagrid was most impressive (though I dunno if it will have a similar (or any) effect on maingrid)
[16:24] Simon Linden: The two updates shouldn't have visible changes.
[16:24] Motor Loon points at his new gold medal with huge pride in his eyes...
[16:24] Simon Linden: oh, and on Monday morning there may be some restarts to shuffle some regions to new hardware
[16:25] TankMaster Finesmith: woot, new hardware!
[16:25] Simon Linden: That's about it for releases
[16:25] Toysoldier Thor: simon...
[16:25] Motor Loon: new powercables for all servers woohoo...
[16:25] Toysoldier Thor: the new hardware - are you placing more VMs on a physical new box or there will be more horsepower available for the sims
[16:26] Simon Linden: yes and yes, I think
[16:26] TankMaster Finesmith: lol
[16:26] Simon Linden: There are more CPU cores, and the ones there are faster
[16:26] Ardy Lay: Why do people keep saying SL runs in VMs? DOes it?
[16:26] Simon Linden: No, it doesn't Ardy
[16:26] Andrew Linden: It doesn't run on VM
[16:26] Toysoldier Thor: hoe ever its termed
[16:26] Toysoldier Thor: instances?
[16:27] TankMaster Finesmith: some equate a region as a V
[16:27] TankMaster Finesmith: VM
[16:27] Toysoldier Thor: so are sim's application instances on one os instance?
[16:28] Andrew Linden: basically a debian box with some server processes
[16:28] Simon Linden: There's one OS running, and multiple processes
[16:28] Ardy Lay: All regions on a host are accessed via a single network address so I assume they are not in virtual machines.
[16:28] Toysoldier Thor: ook
[16:28] Toysoldier Thor: Qie - that is important to know
[16:28] Qie Niangao: yeah... for some reason, I always thought the sim process was wired to a core
[16:30] Toysoldier Thor: we were trying to develop theories why / how the borked lagged sim was showing major stress at the network transport layer
[16:30] Meeter: Timecheck : User Group is half over
[16:30] Toysoldier Thor: http://community.secondlife.com/t5/Second-Life-Server/Increase-in-Instant-SIM-LAG-amp-Crashes-During-Larger-Events/td-p/1683765/page/5
[16:31] Toysoldier Thor: did LL come up with any theories ? Bug-355
[16:32] Andrew Linden: There are too many unknowns for me to try to come up with a good theory on that one.
[16:32] Toysoldier Thor: another region owner contacted this week that friday nights they always encounter the sam type of huge lag - i instructed her as to what to capture and post it on the thread
[16:33] Andrew Linden: I'd want to know the date, the host, the region name at least.
[16:33] Qie Niangao: is there more information that could be collected to reduce those unknowns?
[16:33] Toysoldier Thor: we have been providing all that Andrew
[16:33] Toysoldier Thor: its in the thread
[16:33] Andrew Linden: Ah.
[16:34] Toysoldier Thor: Qie noticed the network process bogs down even at very low traffic loads - while a heavy loaded sim is running fine with no network stress
[16:34] Ardy Lay: When regions wet weird I fear to trust the statistics displayed by the viewer.
[16:35] Toysoldier Thor: well if someone could give us a better tool to capture sim stats - i am all game
[16:35] Ardy Lay: -wet +get
[16:36] Andrew Linden: So this sim lag is correlated with "large events". High avatar count, I take it.
[16:36] Theresa Tennyson: Sometimes yes, sometimes no.
[16:36] Ardy Lay: Oh, did we ever find out if residents with really bad network connections are causing issues when in areas with a lot of local chat?
[16:37] Toysoldier Thor: yes - all the details are in the thread - seems the large events trigger it then even when the avatar count goes down the sim seems to be half baked for a while and it seems its coming from the network not processing requests
[16:37] Rex Cronon: there are 3 things that affect lag: physics, scripts, and nr of avatars
[16:37] Simon Linden: oh, there are plenty of other things too
[16:38] Rex Cronon: ok. the main 3 things
[16:38] Motor Loon: ...and those 3 things are usually linked together too
[16:38] Toysoldier Thor: Pending Downloads
Bad: 531
Good: 1
Total Unacked Bytes:
Bad: 581.2 kb
Good: 3182.3
[16:38] Ardy Lay: Lag is caused by expectations of expediency.
[16:38] Andrew Linden: Well, when I get my interestlist changes ready for RC I'll follow up on reports of lag problems there.
[16:38] Toysoldier Thor: ok
[16:39] Toysoldier Thor: on those stats u see up there Andrew - the bad sim had 28 avatars and the good server had 77 avatars
[16:39] Theresa Tennyson: Will the interest list work also address "kill" notices that are lost, leaving ghost items?
[16:39] Andrew Linden: ok, so definitely not necessarily related to avatar count
[16:39] Toysoldier Thor: that is why Qie and i were puzzled
[16:39] Andrew Linden: Theresa, one of the things I want to do there is to send the KillObject messages more reliably.
[16:40] Andrew Linden: That won't be in the first iteration, but maybe in the second.
[16:40] Ardy Lay: Well, you do have multiple regions on a host. Could they be messing with each other?
[16:40] Toysoldier Thor: true
[16:40] Rex Cronon: u keep sending it until u receive a reply?
[16:40] Rex Cronon: i mean confirmation
[16:40] Andrew Linden: Also, I did a minor ajustment there... in the first iteration the server will be sending KillObject messages for ALL objects that are deleted, not just those that are in view, which will help, but only a little bit.
[16:41] Andrew Linden: Yes, Rex.
[16:41] Ardy Lay: An unpopulated garden region on a host with 6 mall regions and a live music event region goes "ugh".
[16:41] Qie Niangao: it will help quite a bit, actually.
[16:41] Motor Loon: Sounds like the may improve alot just that Andrew
[16:42] Motor Loon: All those airplanes I see hanging in the air when I cam around after a days work on a sim
[16:42] Theresa Tennyson: Mesh vehicles are also often not redrawn completely after sim crossings.
[16:42] Ardy Lay looks forward to seeing lots of green poofs.
[16:42] Qie Niangao: Ardy, who you callin' "green" ? :p
[16:42] Ardy Lay: Object delete updates are green ....
[16:43] Qie Niangao: (sorry. never mind. :o )
[16:43] Ardy Lay watches packets for a living now.
[16:43] Motor Loon: better than watching grass grow
[16:43] Motor Loon: somewhat
[16:43] TankMaster Finesmith: or paint dry
[16:44] Motor Loon: thats more like a weekend hobby aint it?
[16:44] TankMaster Finesmith: only if you paint
[16:44] Ardy Lay: I find lots of software bugs watching packets on networks.
[16:44] Toysoldier Thor: is there a scripted diag tool that is available to get a reliable accurate set of sim stats?
[16:45] Ardy Lay: I would expect scripts to be less reliable than the viewer statistics panel but I don't know.
[16:45] Rex Cronon: scripted dialog tool?
[16:46] Theresa Tennyson: That'd be getting into a Heisenberg place, wouldn't it?
[16:46] Siana Gearz: actually i like how paint dries. it has smell, it has those phases it goes through...
[16:46] Ardy Lay: I once reported a poorly performing region. I was informed that it had to be restarted because way too much logging was enabled. ;-)
[16:46] Toysoldier Thor: if the viewer stats could be inaccurate when a sim goes stale or bonkers
[16:47] Ardy Lay: Watching metal flake paint on a statically charged plastic surface can be amusing.
[16:47] Andrew Linden: What don't you trust about the simulator stats Toysoldier?
[16:47] Toysoldier Thor: ardy said we might not be able to trust the stats from a viewer
[16:48] Toysoldier Thor: when the sim is going bonkers
[16:48] Ardy Lay: I said I have my suspicions.
[16:48] Ardy Lay: They may be unfounded.
[16:48] Motor Loon: where do you think the viewers gets that data?
[16:48] Toysoldier Thor: so Andrew - can we trust the viewers sim stats?
[16:48] Ardy Lay: From the simulator that is going bonkers.
[16:49] Motor Loon: so getting it from the simulator via a script would be different how?
[16:49] Andrew Linden: Right. The viewer gets the stats from the simulator.
[16:49] Toysoldier Thor: ok so they can be trusted at all times - even when a sim might be going stale?
[16:49] Ardy Lay: Script handling changes whe a region is in duress.
[16:49] Ardy Lay: -whe +when
[16:49] Andrew Linden: Well, if the viewer has an actual connection to the simulator at all, then I would trust the stats coming from it.
[16:50] Toysoldier Thor: ok good to know
[16:50] Simon Linden: If the viewer stats are updating regularly, I think that should be pretty accurate data
[16:50] Toysoldier Thor: just wanted to confirm
[16:50] Ardy Lay: Yes
[16:50] Toysoldier Thor: good theory though ardy - i just wanted to make sure
[16:50] Qie Niangao: It's certainly possible that there's sometimes more network demand, events, etc., than the sim gets to record in stats (to viewer or scripts), but seeing what it gets around to reporting is still informative.
[16:50] Andrew Linden: The simulator stats are mostly timers and counters... even when the simulator is suffering from lag I'd expect the stats to be correct.
[16:51] Nalates Urriah: When the PFPS foes low the PING alwaus goes up. If you PING the server outside of the viewer, you get something completely different than what the viewer is reporting.
[16:51] Nalates Urriah: foes=goes
[16:51] Andrew Linden: PFPS = ?
[16:51] Nalates Urriah: Physics FPS
[16:52] Andrew Linden: Ah yes.
[16:52] Andrew Linden: What is happening there is that the incoming packets are processed in the main simulator loop
[16:52] Toysoldier Thor: that might be a different ping Nal... your ping from your pc is from your device across the internet to the server - the Sim ping might be a different source destination ping
[16:52] Motor Loon: Router ping
[16:52] Andrew Linden: so if the physics engine is causing the main simulator loop to run slow, then network packets pile up and wait
[16:52] Ardy Lay: "sim ping" in the viewer is kinda goofy. It includes viewer frame rendering time
[16:53] Andrew Linden: and their ping replies come back after a delay (after the physics frame is done)
[16:53] Nalates Urriah: I use the IP address and URL from Help->About... so I would think the URL would take the same path as the for PING and viewer...
[16:53] Andrew Linden: that is, the reponser to the ping messages is not running on a separate thread
[16:53] Ardy Lay: Same network path, different application path
[16:53] Andrew Linden: I think there is a similar dependency of ping on the *viewer* fps
[16:53] Nalates Urriah: OK that makes sense for why there is a difference.
[16:54] Ardy Lay: "sim ping" is not ICMP ECHO
[16:54] Andrew Linden: er... where responser should be *responder*
[16:54] Theresa Tennyson: So it's the time it takes to communicate with the server program itself rather than the machine its running on?
[16:54] Toysoldier Thor: interesting discussion
[16:55] Andrew Linden: right. I you ping the server from a terminal command line you'll get a hardware accelerated response
[16:55] Meeter: Timecheck : User Group is almost over
[16:55] Andrew Linden: which should be closer to the real network travel time between server and client
[16:56] Ardy Lay: Kinda like the difference between a line loop back and a payload loop back on a DDS circuit
[16:56] Toysoldier Thor: i hate to depart early..... waves to the uber grey matter crowd
[16:56] Toysoldier Thor: see u next week
[16:56] TankMaster Finesmith: take care toy
[16:56] Qie Niangao: have fun, Toy
[16:56] Nalates Urriah: bye Toy
[16:57] Motor Loon: oh dear... looks like Toy was responsable for 50% the lighting here
[16:57] TankMaster Finesmith: and rex the other 50%
[16:57] Motor Loon: hehe
[16:57] Motor Loon looks for his flashlight
[16:58] Theresa Tennyson: The sun's up, Motor...
[16:58] Ardy Lay: Huh? I see daylight
[16:58] Motor Loon: arh... Im just kidding guys
[16:58] Siana Gearz: Andrew, yes, ping is dependent on viewer framerate.
[16:59] Siana Gearz: because it's a processing roundtrip time of sorts and messages are only processed once a frame.
[17:00] Andrew Linden: yes, and that ping time is relevant to the UDP circuit code -- it's the effective ping it would want to use for delay interpolation (if it did any)
[17:00] Meeter: Thank you for coming to the Server User Group
[17:00] Ardy Lay: ... and something similar happens in the simulator
[17:00] Ardy Lay: The viewer has some moving object prediction in it, that is usually disabled.
[17:01] Ardy Lay looks.
[17:01] Rex Cronon: did i crash, or did the sim crash?
[17:01] TankMaster Finesmith: yes
[17:01] Andrew Linden: The sim didn't crash.
[17:01] Ardy Lay: Rex, just you, as usual.
[17:01] Rex Cronon: ok. i didn't crash. viewer lost connection
[17:02] TankMaster Finesmith: have a great weekend, simon and andrew
[17:02] Baker Linden: thanks for coming everyone
[17:02] TankMaster Finesmith: and baker
[17:02] Baker Linden: haha
[17:02] Simon Linden: Thanks for the discussion, everyone
[17:02] Ardy Lay: I suppose I could claim that we are all VERY fast at relogging and taking seats.
[17:02] TankMaster Finesmith: your to quiet over there :P
[17:02] Rex Cronon: tc everybody and have nice day:)
[17:03] Qie Niangao: Thanks Andrew, Simon, Baker... have fun, all.
[17:03] Baker Linden: I know, I've been waiting for my group stuff to get on RC so I can test it :)
[17:03] Andrew Linden: Cheers everyone.
[17:03] Baker Linden: then I can talk about stuff again
[17:03] TankMaster Finesmith: hehe
[17:03] TankMaster Finesmith: to bad it got clumped with bad code
[17:03] Baker Linden: yeah
[17:03] TankMaster Finesmith: and they can't unclump it
[17:03] Theresa Tennyson: Next week I hope to hit the roof skydiving - made it to the balcony a floor below today.
| Prev 2012.10.09 | Next 2012.10.16 |