User:Andrew Linden/Office Hours/2010 09 21

Transcript

[10:57]	Bronson Blackadder:	will be funny to see andrew and simon log in there right as the sim goes down
[10:57]	Qie Niangao:	hi Simon
[10:58]	Simon Linden:	hello
[10:58]	Gooden Uggla:	morning simon
[10:58]	Bronson Blackadder:	damn there goes that idea
[10:58]	Qie Niangao:	good choice of refuge. ;)
[10:58]	Ardy Lay:	Hello
[10:58]	Ardy Lay:	/me waits for a restart notice to show up here too.
[10:58]	Moon Metty:	hey Simon :)
[10:58]	Cummere Mayo:	so does this server deploy fix for real the account lockouts?
[10:58]	Simon Linden:	unfortuntely not yet
[10:59]	Cummere Mayo:	damn.
[10:59]	Simon Linden:	that should go into the "BlueSteel" channel on Thursday
[10:59]	Ardy Lay:	Ah, this region is not in BlueSteele? Okay
[10:59]	Simon Linden:	no, these two look like they were still 1.42
[10:59]	Cummere Mayo:	im getting tied of having my account locked out 10 minutes every time i crash
[10:59]	Ardy Lay:	My home is in a BlueSteel region
[11:00]	Sindra Aquila:	hi evertone
[11:00]	Ardy Lay:	Just takes me 1 minute to get back in after a crash.
[11:00]	Simon Linden:	there it goes
[11:01]	Ardy Lay:	Simon, your house evaporated. Does it do that often?
[11:01]	Cummere Mayo:	used to me. about a month ago it sarted lockign me out again. dunno why but it is..... irritating
[11:01]	Moon Metty:	hmmm, it didn't log me out
[11:01]	Simon Linden:	every crash and restart :)
[11:01]	Ardy Lay:	Hehe
[11:01]	Gooden Uggla:	the first 20 times you see that it's a little disturbing...
[11:01]	Bronson Blackadder:	ok I can confirm... the sim is gone
[11:01]	Second Life:	Second Life: Your region will restart in 5 minutes and 0 seconds for maintenance.
[11:01]	Gooden Uggla:	LOL
[11:01]	Ardy Lay:	Hehe
[11:01]	Sindra Aquila:	lol
[11:01]	Qie Niangao:	hehehe
[11:01]	Moon Metty:	ooooo!
[11:01]	Cummere Mayo:	hahahah
[11:01]	Bronson Blackadder:	ffs
[11:01]	Moon Metty:	exciting
[11:02]	Cummere Mayo:	lovely
[11:02]	Qie Niangao:	okay, *not* such a good choice of refuge
[11:02]	Ardy Lay:	Give the other 2 minutes to come back
[11:02]	Latif Khalifa:	/me jumps on the other side http://slurl.com/secondlife/Hanley/224/27/66
[11:02]	Ardy Lay:	It was fast when this happened to me at home.
[11:02]	Moon Metty:	it feels like a field trip :)
[11:02]	Cummere Mayo:	simon I blame you
[11:02]	Cummere Mayo:	/me giggles
[11:02]	Qie Niangao:	greetings Andrew... don't get too comfy
[11:02]	Sindra Aquila:	we need a campfire too
[11:02]	Andrew Linden:	Hi everyone.
[11:02]	Gooden Uggla:	we need a boat...
[11:02]	Cummere Mayo:	hi andrew 4 min till this one goes down too
[11:02]	Simon Linden:	Yeah, hopefully will be back soon. I have been seeing a lot of region crashes as they shutdown today ... there's some bug in there, but it doesn't seem to affect it much
[11:02]	Gooden Uggla:	:)
[11:02]	Sindra Aquila:	hello andrew
[11:02]	Andrew Linden:	So... we must be having a rolling update?
[11:03]	Gooden Uggla:	where to now?
[11:03]	Moon Metty:	hi Andrew :)
[11:03]	Gooden Uggla:	hiya andrew, yup
[11:03]	Cummere Mayo:	yeah but as simon said theres a hiccup to them. ^^
[11:03]	Simon Linden:	yes, the 1.42 bug fix (now 10.9.10.something) is getting the full grid now
[11:03]	Andrew Linden:	this spot will do fine
[11:03]	Second Life:	Second Life: Your region will restart in 3 minutes and 0 seconds for maintenance.
[11:03]	Gooden Uggla:	no, it's being rooled :)
[11:03]	Andrew Linden:	oh, you're right
[11:04]	Cummere Mayo:	famous last words andrew
[11:04]	Gooden Uggla:	where to?
[11:04]	Moon Metty:	lol
[11:04]	Moon Metty:	come on Denby! you can do it!
[11:04]	Cummere Mayo:	watch this region crash JUST as denby comes back
[11:05]	Andrew Linden:	I dunno. Someone here own a public region to which we can escape?
[11:05]	Jonathan Yap:	Torley has Here island
[11:05]	Gooden Uggla:	lol
[11:05]	Cummere Mayo:	um we could go to my linden home i guess
[11:05]	Second Life:	Second Life: Your region will restart in 1 minutes and 0 seconds for maintenance.
[11:05]	Simon Linden:	should be up really soon....
[11:06]	Gooden Uggla:	as they staqnd and get logged out
[11:06]	Second Life:	Second Life: Your region will restart in 0 minutes and 30 seconds for maintenance.
[11:06]	Andrew Linden:	Oh, I think denby is coming up
[11:06]	Second Life:	You just entered a region using a different server version, which may affect performance. [ http://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/10.9 View the release notes.]
[11:07]	Rex Cronon:	hello everybody
[11:07]	Simon Linden:	Hello
[11:07]	Sindra Aquila:	hi
[11:08]	Twisted Laws:	hello's
[11:08]	Rex Cronon:	did the sim just got on line?
[11:08]	Ardy Lay:	Rolling restart
[11:08]	Cummere Mayo:	im here lol
[11:08]	Simon Linden:	haha
[11:08]	Ardy Lay:	Second Life Server 10.9.10.210079
[11:08]	Andrew Linden:	/me is using viewer-1.23 today, I was testing the world map right before OH started
[11:08]	Moon Metty:	smooth
[11:08]	Liisa Runo:	something strange and horrible happened, server restarted, and my hippo server and update server got returned to my lost and found... they rely on the prim key to work, so now everything is ruined forever
[11:08]	Cummere Mayo:	haha i guess linden homes are on the same rollout pattern?
[11:09]	Ardy Lay:	I teleported from Longfellow to Denby just after the 30 second warning in Longfellow.
[11:09]	Moon Metty:	do you know the cause of that worldmap-lag?
[11:09]	Andrew Linden:	Now the viewer-1.23 UI is annoying. I can't remember where the buttions and options are.
[11:09]	Cummere Mayo:	oh no:( can they roll it back liisa?
[11:09]	Liisa Runo:	no idea..
[11:10]	Ardy Lay:	Andrew, yeah, placement changes are like that.
[11:10]	Gooden Uggla:	try to C&P a notice into 3 groups at once, 1.23 won't look too annoying anymore
[11:10]	Cummere Mayo:	why the heck is stuff getting returned by a sim restart? youre like the thrid person thats happened to that i know liisa
[11:10]	Andrew Linden:	hrm... I wonder why the hippo server was returned.
[11:11]	Gooden Uggla:	liisa ticket a rollback and explain on the ticket what the permanent loss is
[11:11]	Rex Cronon:	maybe your competition has someting to do with it;)
[11:11]	Andrew Linden:	Cummere, the third person in server-1.42 rollout?
[11:11]	Liisa Runo:	all the automatic update scripts in my products have the key of the update server hardcoded in to the script
[11:11]	Gooden Uggla:	give the specific time of return and anything else you can think of
[11:11]	Simon Linden:	hmm, I don't know why that stuff should be returned
[11:11]	Cummere Mayo:	third person today thats had stuff returned when the region restarted
[11:12]	Gooden Uggla:	sounds like a bug, huh?
[11:12]	Liisa Runo:	i must file a support ticket, if LL cant fix it, my business in SL is doomed
[11:12]	Rex Cronon:	liisa. didn't actually loose anything, besides have to rescript and re-reze things
[11:12]	Andrew Linden:	yeah. I've got this terrible feeling that maybe I'll have to look into that bug.
[11:12]	Rex Cronon:	having to*
[11:12]	Moon Metty:	could it be the group under which it was rezzed changed somehow?
[11:12]	Andrew Linden:	Not because I caused it, but it may be more important than whatever else I was working on.
[11:12]	Rex Cronon:	and that can really be a pain in the neck:(
[11:12]	Simon Linden:	There are very few changes between these versions too ... less than 10 bug fixes that I know of
[11:13]	Cummere Mayo:	simon can you rolll back liisa's region for her?
[11:13]	Arawn Spitteler:	/me just rezzed on Zindra Continent, because Longfello wasn't available
[11:13]	Liisa Runo:	no, i didnt loose anything, but all my customers have the key of my update script hardcoded, and since the server got returned, the key is now lost
[11:13]	Andrew Linden:	Liisa, is this a privately owned region?
[11:13]	Simon Linden:	No, it has to go via the usual support channels. They actually know how to do it right :)
[11:13]	Liisa Runo:	no
[11:13]	Andrew Linden:	Well, a rollback may just repeat the return -- if it is repeatable.
[11:13]	Gooden Uggla:	could have been introduced in 1.42 and sat there till this roll
[11:14]	Simon Linden:	Right
[11:14]	Liisa Runo:	anyway, my problems are not the topic today, ill file a ticket, lets get to the real topics
[11:14]	Moon Metty:	:)
[11:14]	Ardy Lay:	I suppose there is some reason the update server key has to be hardcoded.
[11:14]	Rex Cronon:	liisa i think the same thing will happen each time a new server version is uploaded:(
[11:14]	Andrew Linden:	Liisa, just in case I have to track it down... what were the details: object names and region name?
[11:14]	Gooden Uggla:	andrew, she has hundreds of customers that she'll lose contact with, that's poretty serious
[11:15]	Liisa Runo:	ill write a note to you Andrew
[11:15]	Gooden Uggla:	a rollback will give her time to contact them
[11:15]	Twisted Laws:	hope that isn't true, Rex. I've had an update server out for over 3 years
[11:15]	Andrew Linden:	Moon, you were wondering about the world map issue.
[11:16]	Moon Metty:	yes
[11:16]	Andrew Linden:	I did some of the digging on that problem, but was not involved in the fix.
[11:16]	Moon Metty:	i suspect the server is bombarded with requests
[11:16]	Andrew Linden:	Turns out it was an "asset problem"...
[11:16]	Moon Metty:	oh O_O
[11:16]	Cummere Mayo:	brb this water flicker bug is making me sick
[11:16]	Andrew Linden:	We cache some expensive database (db) queries as assets, that get updated periodically.
[11:16]	Rex Cronon:	IMO hardcoding keys is not a good idea
[11:17]	Andrew Linden:	The RegionName::MapTimeAssetID pairs were the product of such a cached asset.
[11:17]	Andrew Linden:	the asset was rather big... 30MB
[11:18]	Andrew Linden:	and our asset system is currently under an overhaul... we're changing how we cache the data, and putting independent caches in each colocation facility
[11:18]	Moon Metty:	yes
[11:18]	Andrew Linden:	the new caching system relies on "varnish" which is similar to "squid"
[11:19]	Andrew Linden:	and is an "HTTP accelerator" caching system, as I understand. I don't know much about it.
[11:19]	Andrew Linden:	However, as it turns out varnish does not "stream" cached assets, but will first make sure it knows the completed asset from the main server before it will forward it to the requester.
[11:19]	Cummere Mayo:	andrew could this overhaul also be causing the massive inventory losses residents have had lately?
[11:20]	Andrew Linden:	And the 30MB file was not coming down fast enough, so the requesters (some database servers on each simulator host) were timing out.
[11:20]	Moon Metty:	yes, hence the 60 second period
[11:20]	Andrew Linden:	So they fixed it by tweaking the cache duration for the relevant files, and maybe also some configs of other parts.
[11:21]	Andrew Linden:	I think they had a "quick temporary fix" but were talking about a longer-term fix
[11:21]	Moon Metty:	great, so the lag is already fixed?
[11:21]	Latif Khalifa:	why was it causing sim load?
[11:21]	Andrew Linden:	that would help ALL big assets, not just the cached map queries.
[11:21]	Andrew Linden:	Oh, the sim-load problem when opening the map very wide... it is a symptom of the old map system, inherent to viewer-1.23
[11:23]	Ardy Lay:	I saw some effect when opening and panning the map using V2.1.2 but nothing like V1.23.5 users wear causeing.
[11:23]	Latif Khalifa:	it didn't happen before
[11:23]	Andrew Linden:	Ardy, I tried to reproduce the lag you saw in viewer-2.1.2, but could not.
[11:24]	Andrew Linden:	Really Latif? The world-map lag in viewer-1.23 is new?
[11:24]	Ardy Lay:	Andrew, it's not horrible. Easy to see it on a quiet region/host.
[11:24]	Moon Metty:	the lag is new to me
[11:24]	Simon Linden:	Last week we had some people who had problems changing outfits ... they would appear naked or grey to others. That's been recently fixed
[11:24]	Andrew Linden:	Hrm... interesting.
[11:24]	Moon Metty:	(i wasn't aware of it before)
[11:24]	Cummere Mayo:	I see it too and never had it till recently
[11:25]	Rex Cronon:	what? viewers 1.23 cause server lag? really?
[11:25]	Andrew Linden:	/me worries.
[11:25]	Cummere Mayo:	and i aslo get it in the 2.12
[11:25]	Twisted Laws:	clients built off snowglobe 1.23 don't do it Rex
[11:25]	Cummere Mayo:	*2.1.2
[11:25]	Arawn Spitteler:	If a viewer can lag a server, I'd better not start hacking viewers.
[11:26]	Rex Cronon:	how can the viewer sombody is using cause server lag?
[11:26]	Gooden Uggla:	caches that don't cache hammer the asset system, this isn't new
[11:26]	Simon Linden:	The old map problem was when it zoomed way out
[11:26]	Latif Khalifa:	there is some bug in the server with map handling, this lagging is new
[11:26]	Rex Cronon:	r u guys sure that the moded viewers aren't the ones responsible?
[11:26]	Simon Linden:	It made way too many requests to find out where people were so it could draw green dots
[11:26]	Latif Khalifa:	worked fine for years doing exactly that
[11:27]	Moon Metty:	Rex, it's 1.23.5 that does it too
[11:27]	Cummere Mayo:	yes. im using the 2.1.2 dev build
[11:27]	Andrew Linden:	Ok thanks Latif. I'm making a note. I might have to look into that one too.
[11:27]	Qie Niangao:	I *thought* it was that, too, Simon, from the description of a 60-second cycle... but reports are that it lagged even with the green dots turned off.
[11:27]	Rex Cronon:	what kind of lag does 1.23.5 causes, and why?
[11:28]	Andrew Linden:	SVC-4575
[11:28]
[11:28]	NonSmokerGirl Tulip:	yay for the graphs. we need more graphs and charts!
[11:29]	Simon Linden:	hmm, yeah, I didn't dig into it myself so it may be different. I heard that being mentioned, however. There was a server-side fix a while back, and I don't think that went away, so this must be some variation or something completely different
[11:30]	Andrew Linden:	Ooops, no not that issue. I got the number wrong.
[11:30]	Ardy Lay:	I remember Vektor doing a test where he asked verybody to open that world map at the same time.
[11:30]	Moon Metty:	https://jira.secondlife.com/browse/SVC-6327
[11:30]	Ardy Lay:	It got ugly fast.
[11:30]	office hours	is half over
[11:30]	Andrew Linden:	Yes, thaks Moon. That is the one I was looking for.
[11:31]	Ardy Lay:	He told us one Linden in God mode could make it ugly.
[11:31]	Moon Metty:	i will open the world map for a sec
[11:31]	Rex Cronon:	if nothing else kills v1.23.5 this will:(
[11:32]	Andrew Linden:	yup, I saw that performance problem spkke Moon.
[11:32]	Latif Khalifa:	this is new bug in 1.42 and should be fixed, this was not an issue before
[11:32]	Cummere Mayo:	not as bad as it was, but im still getting some greying and some hesitation on lcient end. imm nto able to see if theres much differantce server end
[11:32]	Moon Metty:	the region has just been restarted, it doesn't look so bad now
[11:32]	Moon Metty:	i will try it again after the meeting :)
[11:33]	Andrew Linden:	Sigh... and I thought I was going to be able to start on region-crossings again.
[11:33]	Rex Cronon:	a paranoid person would think that it was done on purpose:)
[11:33]	Rex Cronon:	i am talking about 6327
[11:34]	Gooden Uggla:	and can't d/l SG anymore, yes
[11:35]	Andrew Linden:	Well. What little news I've got...
[11:35]	Cummere Mayo:	they finally take the sg download page ?
[11:35]	Cummere Mayo:	:(
[11:35]	Ardy Lay:	http://wiki.secondlife.com/wiki/Snowglobe
[11:35]	Andrew Linden:	I've finished fixing a rez-flood bug that could crash the simulator. It is in my andrew/maint-server repository that should go up for test (again, hopefully for the last time) tomorrow
[11:36]	Andrew Linden:	Also, a few minor fixes that will help maintain geometry-cache coherency
[11:36]	Andrew Linden:	That's all I got done since Friday.
[11:37]	NonSmokerGirl Tulip:	I ate some long grass.
[11:37]	Moon Metty:	:)
[11:37]	Andrew Linden:	BTW, Simon and I will be attending another meeting today that starts at 11:45.
[11:38]	Cummere Mayo:	andrew any wrod about the assets disappearing from database bugs?
[11:38]	Moon Metty:	did you get any information from the pile-up on aditi last thursday?
[11:39]	Andrew Linden:	Cummere, no word on that issue yet.
[11:39]	Simon Linden:	we did ... no miracle cures, but the memory usage is making more sense
[11:39]	Moon Metty:	aha
[11:39]	Cummere Mayo:	okay just asking
[11:39]	Cummere Mayo:	/me mourns her missing tiems
[11:39]	Ardy Lay:	I had a missing asset. It was missed in the asset store migration and ops recovered it for me.
[11:39]	Cummere Mayo:	*items
[11:40]	Jonathan Yap:	Cummere, can you get them on aditi?
[11:40]	Moon Metty:	those pile-ups often bring the region all the way down lol
[11:40]	Andrew Linden:	Cummere, remind me the names of the lost items? Do you have a jira open about it? or a notecard?
[11:40]	Andrew Linden:	The problem sounds similar to another bug I was supposed to do some legwork on, but the info in that bug was very old.
[11:41]	Cummere Mayo:	um there wee several jiras. not sure which one i commented on ><
[11:41]	Cummere Mayo:	it was a while ago
[11:41]	Gooden Uggla:	how's the progress on mono update coming?
[11:41]	Cummere Mayo:	but basically its jsut something is there one day, and the next its "unable to load xxx. object is missing form the database"
[11:42]	Andrew Linden:	Hrm... I wonder if the data is just too old. Cummere, do you recall approximately how long ago they were lost?
[11:42]	Andrew Linden:	And you still have the items in inventory Cummere?
[11:43]	Cummere Mayo:	i lose more every few days. and yes i still have some of them.
[11:43]	Cummere Mayo:	theres like 30 jira issues on it, all of them have been closed out to contact support
[11:43]	Cummere Mayo:	whcih is saying file a ticket and a jira
[11:43]	Cummere Mayo:	><
[11:43]	Cummere Mayo:	/me headdesks
[11:44]	Andrew Linden:	Well, send me an email, or notecard with some names. This relates to one bug I'm supposed to investigate, and I need some specific info to use in my digging.
[11:44]	Cummere Mayo:	/me nods
[11:44]	Cummere Mayo:	will do.
[11:45]	Andrew Linden:	Gooden, you asked about progress on Mono update. I think one of the other LL devs was able to make some progress there and is narrowing down on the bug -- it may actually be a Mono serialization bug.
[11:45]	Andrew Linden:	Ok. We've got to run
[11:45]	Moon Metty:	yup
[11:45]	Moon Metty:	thanks for the meeting :)
[11:45]	Simon Linden:	ah, right.
[11:45]	Andrew Linden:	see you all later. Thanks for coming.
[11:45]	Simon Linden:	Thanks everyone for coming
[11:45]	Qie Niangao:	Thanks Andrew and Simon
[11:45]	NonSmokerGirl Tulip:	thank you both!
[11:45]	Rex Cronon:	tc andrew
[11:45]	Simon Linden:	See you next time
[11:45]	Cummere Mayo:	tc and thanks!
[11:45]	Sindra Aquila:	thanks simon and andrew
[11:45]	Rex Cronon:	tc simon
[11:45]	Moon Metty:	good luck with your work!
[11:45]	Moon Metty:	see you soon
[11:46]	Sindra Aquila:	bye everybody
[11:46]	Gooden Uggla:	thanks, have a good one
[11:46]	Latif Khalifa:	take care guys, bye

Generated with SLog Wikifier