Beta Server Office Hours/Minutes/2014-07-24

[14:59] I am not a furry: we'll be starting in a couple of minutes
[15:00] Chakat Northspring: Group Chat Chat  and Group Chat Test  as usual? ~
[15:01] I am not a furry: We should have one called Le bon chat
[15:01] Simon Linden: We're not going to do the group chat tests today, unfortunately
[15:01] I am not a furry: So, let's get going
[15:01] Simon Linden: I found a last-minute problem
[15:01] I am not a furry: http://wiki.secondlife.com/wiki/Server_Beta_User_Group
[15:01] Whirly Fizzle: OOh okay
[15:01] Caleb Linden: all ears
[15:02] iTalk: Pumpkin is my true love, my soul mate. I will forever cherish and adore her.
[15:02] iTalk: Pumpkin is my true love, my soul mate. I will forever cherish and adore her.
[15:02] Yuzuru: Hello, Caleb and Baker.
[15:02] Baker Linden: hello!
[15:02] I am not a furry: So, magnum went to Main Channel
[15:02] Rex Cronon: hello everybody
[15:02] I am not a furry: That's the SL Gaming prep work
[15:02] I am not a furry: everywhere on the grid now
[15:02] rcdsQueChatLog: State changed.{click me to get the url}
[15:02] I am not a furry: Seems pretty stable
[15:03] I am not a furry: The current maintance fix set went to all three RCs
[15:03] I am not a furry: That's the same set as was on BS/LT last week
[15:05] I am not a furry: In the Coming Soon department is more maintenance fixes
[15:05] I am not a furry: and, as you know, work continues on group chat
[15:05] I am not a furry: Simon, do you want to add anything here?
[15:06] I am not a furry: I'll take that as a no.  ;)
[15:06] Simon Linden: Sorry I have my head in the group chat code :P
[15:06] I am not a furry: ROFL
[15:06] Simon Linden: but no, I don't have anything to add
[15:07] I am not a furry: I like the recursion
[15:07] Rex Cronon: u must be looking at naked pointers:)
[15:07] I am not a furry: um, moving right along.... ;)
[15:07] Simon Linden: I can give a little update on what I'm working on with that if you want
[15:07] I am not a furry: go for it
[15:08] Simon Linden: Well, while the earlier update to group chat didn't give us any significant performance boost, we got a lot more information out of the servers
[15:08] Simon Linden: and what we found was a big part of the group chat system load is not the chat messages you care about
[15:08] Simon Linden: but the updates to who is in the session or not
[15:09] Oz Linden: note that those updates happen whether or not you're displaying who's in the session...
[15:09] Oz Linden: in every group you're in
[15:09] Simon Linden: you can actually see this in the viewer if you add a line of code to log something whenever an update comes in to tell you who's in the group chat ... you'll be surprised how many you get
[15:09] Rex Cronon: i guess having groups with over 1000 logged in members is not that easy to manange...
[15:10] Whirly Fizzle: Do you still get those updates for a group you closed group chat for?
[15:10] Simon Linden: Right Oz ... the load goes up as the group size goes up ... with a larger group, people are joining and leaving more often, and there are more people to update
[15:10] I am not a furry: yes
[15:10] I am not a furry: that messaging grows very big very fast
[15:10] Simon Linden: Whirly - no, if you close that group chat you should be leaving the session, and won't get messages or updates until you re-open it
[15:11] I am not a furry: for three people, 6 messages
[15:11] I am not a furry: for four people 12 messages
[15:11] I am not a furry: for five people, 20 messages
[15:11] I am not a furry: etc.
[15:12] Lucia Nightfire: #people * (#people - 1)
[15:12] I am not a furry: n(n-1)
[15:12] I am not a furry: yep
[15:12] I am not a furry: one message from each person to everyone else
[15:12] Simon Linden: I've been looking at adding better queueing to that, waiting a little bit and combining messages ... and I had something to test today, but I just discovered a glitch I need to re-write
[15:12] Rex Cronon: it might make things faster if the group chat server(s) didn't actively look for who is logged in or not
[15:12] Simon Linden: It doesn't do that, Rex
[15:13] Simon Linden: but it has a list of who's online and in the group chat at that moment .. it's adding and removing from that list, and the resulting updates, that are the problem
[15:13] Rex Cronon: so how come a group chat message pops up when i first logg in?
[15:13] I am not a furry: first login is different than checking if you are logged in
[15:14] Simon Linden: because as you log in, you automatically join all 42 of your groups chat sessions
[15:14] I am not a furry: later
[15:14] Rex Cronon: maybe we shouldn't automatically join?
[15:14] Simon Linden: and for all 42 groups - it sends a message to everyone in each one saying "Rex is here"!
[15:14] Whirly Fizzle: Heh, so upping the group limit really did make things worse
[15:14] Simon Linden: yes Whirly, it probably did
[15:14] I am not a furry: oh, yeah
[15:14] Lucia Nightfire: we need more groups, is what I get out of all this
[15:14] Lucia Nightfire: lol
[15:15] I am not a furry: /o\
[15:15] Kallista Arliss: more, smaller groups
[15:15] Oz Linden: and upping it again would make it even worse, so until this is dealt with, don't even ask... :-)
[15:15] Simon Linden: right - small groups work (aside from the rest of the server load) pretty well.   There isn't as much going on
[15:16] I am not a furry: also fewer updates: 10 groups of 10 send fewer updates than one group of 100
[15:16] Whirly Fizzle: Hmmm what about the chat lag in conference IM's? Thats often very bad. Same cause?
[15:16] I am not a furry: by an order of magnitude!
[15:17] Simon Linden: the ad-hoc chats, where you IM a bunch of your friends, are really the same thing as group chat.  It's all using the same code and servers
[15:17] Rex Cronon: so, how many dedicated group chat server we have?
[15:17] Rex Cronon: servers*
[15:18] Whirly Fizzle: Not enough? :D
[15:18] Lucia Nightfire: probably enough, but not for the infrastructure setup, heh
[15:18] Oz Linden: an exponential growth curve can't be dealt with by using linear resource growth, Rex; we've got to change the shape of the curve
[15:18] Simon Linden: adding more servers isn't an easy task, Rex, so that hasn't been considered.   We are doing a hardware change soon, however ... that might give us a little bit of help
[15:18] I am not a furry: Exactly, Oz
[15:19] Rex Cronon: those chat servers might distribute their load to the sim...
[15:20] Lucia Nightfire: people on substandard comps and hardware do enough of that as it is
[15:20] Whirly Fizzle: When a group chat server dies, Some (all?) group chat dies for groups with the same letter/number at the start of their key. So is there one chat server for 0, 1,2.... a, b, c etc? Or more then one?
[15:20] Lucia Nightfire: nto to mention bad connections, bad bandwidth settings
[15:20] Rex Cronon: its unlikely that a sim will hold 1000 group memebers:)
[15:21] Kallista Arliss: One thing I found , a few years ago, is that 1000 Mb interfaces can handle lots more packets/sec that 100 Mb this was discovered from a churning mail server.
[15:22] I am not a furry: I'm pretty sure the chat servers have better than 100Mb interaces.  ;)  i could be wrong, though, so don't quote that
[15:22] Simon Linden: I'd guess probably about 10x, Kallista :P
[15:22] I am not a furry: ha
[15:22] Kallista Arliss: more like 5-6 X
[15:22] Kallista Arliss: Overhead and all that
[15:22] Simon Linden: I dont' think we're hitting a network issue here ... it's that ugly exponential growth in updates
[15:23] Kallista Arliss: Still it does make a difference
[15:23] I am not a furry: me, too
[15:23] Simon Linden: it does, and the internal networks where our servers are hosted are pretty good.   We monitor traffic and they're able to take the usual load fine
[15:24] Whirly Fizzle: So if your group, even a small 5 person group, shares a chat server with a huge group that always has lots of chatters entering and leaving group chat, your small group is going to be affected by the updates from the busy group?
[15:24] Oz Linden: yes Whirly
[15:24] I am not a furry: But that's true of any shared enviornment
[15:24] Rex Cronon: its weird considering that the list of logged in users is not over 50k. things should be faster
[15:24] Simon Linden: yes Whirly ... it's sharing that same CPU
[15:25] Kallista Arliss: and the same network interface.
[15:25] And all I got was this lousy DN: wow, lots of lindens here
[15:25] I am not a furry: shh, Tank, you'll scare them off ;)
[15:26] I am not a furry: So, moving on from group chat
[15:26] And all I got was this lousy DN: if monty was here, it would be like a TPV meeting :P
[15:26] I am not a furry: heh
[15:26] I am not a furry: In the Coming Soon Department
[15:26] Rex Cronon: imo. i think that some large groups might be even wlling to pay to have a dedicated group chat server...
[15:26] And all I got was this lousy DN: up, scared one off! :D
[15:26] I am not a furry: heh
[15:27] And all I got was this lousy DN: oh maybe not
[15:27] I am not a furry: Anyway, we have another maintenance project in the pipeline
[15:27] I am not a furry: MOAR BUG FIXES!
[15:28] Whirly Fizzle: MOAR BUGS! \o/
[15:28] I am not a furry: lol, Whirly
[15:28] I am not a furry: And the other up and coming thing is trying to make texture downloads go faster
[15:28] Rex Cronon: more bugs comming?
[15:28] Oz Linden: hopefully shiny new ones
[15:28] And all I got was this lousy DN: lol
[15:28] I am not a furry: Which would make res time much quicker
[15:28] Kallista Arliss: shiny new bugs, whirly must be dancing in circles
[15:29] Rex Cronon: "[15:28] Rex Cronon: more bugs comming?
[15:28] Oz Linden: hopefully shiny new ones"
[15:29] Rex Cronon: lol
[15:29] I am not a furry: Shiny new bugs from the VW factory
[15:29] Whirly Fizzle: Is that sewrver end stuff for the HTTP pipeline viewer?
[15:30] Oz Linden: We're setting up some experiments with server side changes that will complement the pipeline viewer, but are not strictly speaking dependent on it
[15:31] Oz Linden: When we're confident that our test setup is ready, including how to measure the results, we'll invite you folks to help us test
[15:31] Whirly Fizzle: Cools
[15:32] I am not a furry: Thanks, Oz
[15:34] I am not a furry: And that's pretty much all I had to say
[15:34] And all I got was this lousy DN: couldnt hlp but notice othrs said most of it :P
[15:34] Oz Linden: sign of a good meeting moderator :-)
[15:34] And all I got was this lousy DN: :D
[15:35] Lucia Nightfire: anyone know why the lab is not responding to questions anymore on http://community.secondlife.com/t5/General-Discussion-Forum/Skill-Gaming-Policy-Thread/m-p/2771090#M187758
[15:35] Lucia Nightfire: they stopped on like page 28 or something, lol
[15:35] Oz Linden: Not our area, Lucia
[15:36] I am not a furry: I'm pretty sure /I/ would have stopped after a couple pages.  ;)
[15:36] Rex Cronon: u realize how long it takes to read 58 page of a forum post? no wonder ll got tired:)
[15:37] Lucia Nightfire: they were fielding questions though
[15:37] Oz Linden: I read 40 pages or so on one thread lately...
[15:37] Simon Linden: I'll pass that nudge on to the product people, Lucia
[15:37] Lucia Nightfire: thanks Simon
[15:37] And all I got was this lousy DN: got tiered repeating "asked and answered" every other post :P
[15:38] Rex Cronon: it might be better if there was live group chat to answer those questions...
[15:38] Kallista Arliss: Or minor variations on A&A
[15:38] Lucia Nightfire: they actually updated the skill gaming FAQ thanks to soem of those questions, but theres been quite a few more since they last responded
[15:38] Simon Linden: I'd rather they write it down and expand the FAQ, Rex
[15:38] Oz Linden: not if you're the one on the answering end, Rex
[15:39] Rex Cronon: ok. a faq section would be also useful
[15:39] Lucia Nightfire: yes, theres plenty of repeat questions too
[15:39] Rex Cronon: oz. people can be quite usefull in a group chat. there is at least one who has an answer:)
[15:39] Oz Linden: https://wiki.secondlife.com/wiki/Linden_Lab_Official:Second_Life_Skill_Gaming_FAQ
[15:40] Rex Cronon: for example: somebody asks a question and somebody else say: go look at faq for question nr. 23...
[15:41] I am not a furry: for some amusement on the whole, not read the comments and ask again, Rex, try googling "Rainbow-Cake Recipe Inspires Comment Apocalypse" ... it's a pretty funny read
[15:42] Oz Linden: time for me to go ... bye all
[15:42] Rex Cronon: tc oz
[15:42] I am not a furry: yep, time to wrap this
[15:42] Yuzuru: Thank you, OZ.
[15:42] Kallista Arliss: Thanks for coming OZ
[15:42] Rex Cronon: tc coyot
[15:42] Rex Cronon: tc all those leaving
[15:42] I am not a furry: u2 Rex
[15:43] Rex Cronon: i guess the group meeting ends early today?
[15:43] Lucia Nightfire: I want BBQ, heh
[15:44] I am not a furry: Rex: think of it as a fifteen minute gift from Linden Lab, free to do whatever you like with