User:Babbage Linden/Office Hours/2009 11 18

Transcript of Babbage Linden's office hours:

[3:02]	Phantom Ninetails:	Good morning
[3:02]	Xugu Madison:	Gooood morning!
[3:02]	Cerdita Piek waves
[3:02]	Xugu Madison:	hey Babbage :)
[3:03]	Babbage Linden:	hi everyone
[3:03]	Cerdita Piek:	Hello Babbage
[3:03]	Babbage Linden:	thanks for coming
[3:03]	Phantom Ninetails:	Babbage arrived? Yay, hi Babbage
[3:03]	Nock Forager:	Hello Babbage.
[3:03]	Morgaine Dinova:	Hiya Babbage :-)
[3:03]	Babbage Linden:	let's give people a couple of minutes to turn up
[3:04]	Babbage Linden:	(as I'm amazingly not late for once)
[3:04]	Phantom Ninetails:	lol
[3:04]	Kaluura Boa:	hehehe
[3:04]	Nock Forager:	;-)
[3:05]	Morgaine Dinova:	Your late turnups are the stuff of legend, Babbage :-)
[3:05]	Morgaine Dinova:	Epics will be written about it
[3:05]	Phantom Ninetails:	lol
[3:05]	Babbage Linden:	hmm, maybe i should come back later...
[3:05]	Morgaine Dinova:	Hehe
[3:05]	Phantom Ninetails:	lol, yes
[3:05]	Xugu Madison:	Don't encourage him!
[3:06]	Morgaine Dinova:	Meanwhile .... crappy weather in UK
[3:06]	Phantom Ninetails:	（のヮの）
[3:06]	Xugu Madison:	Any specific topics we want to get covered today?
[3:06]	JB Hancroft:	I have one
[3:07]	Babbage Linden:	yes, let's talk about agenda
[3:07]	Babbage Linden:	i've been on holiday in cuba, so don't have a lot to report this week
[3:07]	Morgaine Dinova:	Seb probably wanted to talk about the document he sent you, but he doesn't seem to be online atm
[3:07]	Babbage Linden:	although in my absence, we deployed the tcmalloc simulator to aditit
[3:08]	Babbage Linden:	so i'd like to talk about testing that
[3:08]	Xugu Madison:	I'd love a general update on when we'll be seeing better script memory measurement tools, C# support, etc. - but this is all low priority if someone has actual new stuff to talk about :)
[3:08]	Babbage Linden:	and we've got Mono 2.7.x embedded in the simulator, which is great news
[3:08]	Xugu Madison:	2.7.x going rather faster then?
[3:09]	Babbage Linden:	2.7.x includes the CoreCLR security implementation
[3:09]	Babbage Linden:	which we're going to use for C#
[3:09]	Xugu Madison:	AWesome
[3:09]	Babbage Linden:	so, we're laying the foundations for getting to work on that
[3:09]	Xugu Madison:	Any news on releasing the C# API for us to stare blankly at ahead of time?
[3:10]	Babbage Linden:	once we have that sandbox in place we can also stop verifying assembly signatures, which will stop the sim freezes on mono script load too
[3:10]	artoo Magneto:	i was told the lag when people TP in and out comes from mono/lsl conversions or something... will that be improved ?
[3:10]	Babbage Linden:	artoo, I've been looking at that
[3:10]	Morgaine Dinova:	2.7? I seem to have missed some versions.
[3:10]	Babbage Linden:	and putting more instrumentation in to analyse it this morning
[3:10]	Phantom Ninetails:	Ooh, I hope that's true, Babbage. Finally getting somewhere on returning mono performance to what it was when we first got it sounds nice.
[3:11]	Babbage Linden:	there are 3 causes to the freezes phantom:
[3:11]	artoo Magneto:	yea would be great.. some busy places are pretty un-useable
[3:11]	Babbage Linden:	1) pathalogical behaviour in the scheduler, which we've fixed, but need to beta test
[3:12]	Babbage Linden:	2) signature verification on load, which we need to do to make sure assemblies were created by us
[3:12]	JB Hancroft:	any cache you might be doing a cache on the verification?
[3:12]	Babbage Linden:	3) script destruction, which i think is due to domain unload, but I'm looking in to that
[3:12]	Babbage Linden:	JB, yes, we cache scripts, so it only causes slow down on first load
[3:12]	Morgaine Dinova:	Babbage, although you've mentioned in the past that you wanted to send your implementation of fault recovery upstream, there is no prospect whatsoever about LL's multitasked VM from being opened, right?
[3:12]	Phantom Ninetails:	I see
[3:13]	Babbage Linden:	but if you get an av with 100+ assemblies TPing in to the region, you get a big pause
[3:13]	JB Hancroft:	I mean cache of the dig. signature verification, so it doesn't have to be re-done on every sim entry.
[3:13]	Babbage Linden:	morgaine, we submitted our signal chaining implementation to mono as it was generally useful to everyone and we didn't want to support it
[3:13]	Ann Otoole:	at metanomics the day the enterprise thing was announce the scripts per avatar average exceeded 64
[3:14]	Morgaine Dinova:	Babbage: very cool! Was Miguel receptive?
[3:14]	Babbage Linden:	sure, it's in Mono trunk now
[3:14]	Morgaine Dinova:	Woohoo!
[3:14]	Morgaine Dinova:	\o/
[3:14]	JB Hancroft:	What is signal chaining?
[3:14]	Babbage Linden:	the microthreading work is less generally useful, I've asked miguel if it would be a win for us to open us
[3:15]	Babbage Linden:	open it, rather
[3:15]	Babbage Linden:	he thinks few people would be interested in supporting it
[3:15]	Babbage Linden:	the current mono microthreading support is pretty niche
[3:15]	Babbage Linden:	so, for each mono extension, we need to weigh up the cost/benefit of opening it
[3:15]	Babbage Linden:	it made sense for signal chaining
[3:16]	Babbage Linden:	but probably not for microthreading
[3:16]	Morgaine Dinova:	Babbage: I agree that it wouldn't be *generally* useful to Mono community. However, it would be directly useful for Opensim, obviously. So the question really boils down to whether LL wants the SL scripting model to be disseminated, or wants thius particular capability to be an SL exclusive.
[3:16]	Babbage Linden:	JB, signal chaining is where you store the current signal handlers when you install new ones
[3:17]	Babbage Linden:	and then conditionally call the old handlers
[3:17]	JB Hancroft:	thanks... just reading it now at http://www.mono-project.com/Release_Notes_Mono_2.6#Signal_Chaining
[3:18]	Babbage Linden:	mono steals a lot of signal handlers when you embed it
[3:18]	Babbage Linden:	this extension makes it call the old ones when the exception occurs outside managed code
[3:18]	Babbage Linden:	allowing us to do emergency simstate saving
[3:19]	Babbage Linden:	morgaine, my conversations with miguel were part of that evaluation
[3:19]	Babbage Linden:	opensim is still welcome to implement microthreading to support sl compatibility, obviously
[3:20]	Babbage Linden:	we wanted to find out whether it made sense to open the implementation as well as the specfication
[3:22]	Morgaine Dinova:	Was there an outcome?
[3:22]	Babbage Linden:	JB (from a while ago) yes, we cache the results of the signature check, so you should only see the slow down on first load
[3:23]	Babbage Linden:	morgaine, no, but i think it's unlkely that we'll open the implementation if the only other project wanting to use it is opensim
[3:23]	Morgaine Dinova:	kk
[3:23]	Morgaine Dinova:	And thanks :-)
[3:23]	JB Hancroft:	First load into that sim, ok. When you TP to a new sim, it does it again?
[3:24]	Babbage Linden:	yes, the cache is not shared across sims
[3:24]	Babbage Linden:	so each time an assembly is seen by a new sim, a new verification occurs
[3:24]	JB Hancroft:	ok
[3:25]	Babbage Linden:	we talked about having sims trust each other a couple of weeks ago
[3:25]	JB Hancroft:	Do you have any stats on the average number of sim crossings/hour that an AV does?
[3:25]	Babbage Linden:	so if a sim is being given an assembly from another sim, it doesn't bother re-verifying
[3:26]	Babbage Linden:	you'd only get a freeze on initial save/rez then
[3:26]	JB Hancroft:	Oh, I thought from our chat that it wasn't the case...
[3:26]	Babbage Linden:	but, the better fix is to stop using it and use the mono sandbox instead
[3:26]	JB Hancroft:	I'll have to go back to the OH notes... hmmmm. thanks
[3:26]	Babbage Linden:	and to multithread rez/derez to get rid of freezes
[3:27]	Babbage Linden:	so, the other thing I wanted to talk about is testing the TCMalloc sim
[3:27]	JB Hancroft:	So the initial verification should be on login or wear, not on TP.
[3:27]	Morgaine Dinova:	Babbage: you'll need to put an "if (3rd_party_sim) {verify(assemly);}" in there in due course.
[3:28]	Babbage Linden:	morgaine, right
[3:28]	Babbage Linden:	and we'll also need an ABI specification for serialized scripts too
[3:28]	Morgaine Dinova nods
[3:28]	Babbage Linden:	(at which point we may as well be opening up our microthreading implementation...)
[3:29]	Morgaine Dinova:	Hahaha, good point
[3:29]	Babbage Linden:	so, anyway, tcmalloc
[3:29]	Babbage Linden:	let me find the regions on aditi
[3:30]	Babbage Linden:	ok, there are 4 regions called TCMallocTestN where N is 1-4
[3:31]	Babbage Linden:	if you could have a play on those and let me know how the performance compares to regions on agni I'd be very grateful
[3:31]	Morgaine Dinova winds Xugu up again, needs more power
[3:32]	Babbage Linden:	we're looking for region owners with laggy sims to compare their sims between the tcmalloc and normal 1.32 versions
[3:32]	Xugu Madison:	meep?
[3:32]	Babbage Linden:	once we find volunteers, we'll be converting the regions to run their regions on aditi
[3:32]	Babbage Linden:	but at the moment the regions are just empty
[3:33]	Babbage Linden:	and so good for testing scripts or other performance critical stuff
[3:33]	JB Hancroft:	what use cases will the changes in tcmalloc apply?
[3:33]	Babbage Linden:	like physics tests
[3:33]	Morgaine Dinova:	Very cool, Babbage. I've sent that info to Groupies.
[3:33]	Ann Otoole:	hmmm laggy in kali isle seems directly proportional to avatar count. so to simulate that we would need to have some parties on aditi and load up them wonderful resizers people should be deleting
[3:33]	JB Hancroft:	I do a lot of performance testing, you can load up my Homestead for testing
[3:33]	Babbage Linden:	thanks JB, I'll add you to the list of region owners we'll be contacting
[3:34]	JB Hancroft:	it's lightly load in terms of AVs, and I watch specific scripts I'm working on, very very carefully to try and tweak them.
[3:34]	JB Hancroft nods... glad to help
[3:34]	Babbage Linden:	in terms of what you should see, performance should be close to a normal 1.32 simulator
[3:34]	Ardy Lay:	My home, Lusk, is already on Aditi but the load there is the people, not the scenery.
[3:34]	Babbage Linden:	it may be slightly better as TCMalloc is a higher performance allocator
[3:34]	Ann Otoole:	anybody going to schedule a roaming party in aditi?
[3:35]	Babbage Linden:	but, the main reason we're moving to it is to allow memory profiling of production simulators
[3:35]	Morgaine Dinova nods
[3:35]	Babbage Linden:	so, when we notice a region bloating to 2GB+ we can see where the memory is going
[3:35]	Babbage Linden:	we need to be happy that performance is comparable on aditi
[3:36]	Babbage Linden:	if it is we can deploy the TCMalloc version across agni
[3:36]	Phantom Ninetails:	I wish we could get information about memory usage in the statistics floater
[3:36]	Babbage Linden:	and then start doing memory analysis
[3:36]	Morgaine Dinova:	Good idea Phantom
[3:36]	Ann Otoole:	i'd like to see memory use by at least attachment or hud
[3:36]	Babbage Linden:	Phantom, that may be possible once we have TCMalloc in
[3:36]	Phantom Ninetails:	Yay
[3:36]	Ann Otoole:	to identify scripts for permanent elimination from inventory
[3:37]	Babbage Linden:	i'd like to just see information about memory usage somewhere at the moment ;-)
[3:37]	Morgaine Dinova:	Hehe
[3:37]	Babbage Linden:	maybe we should plan to have our office hour on aditi next week?
[3:37]	Babbage Linden:	in TMallocTest1
[3:37]	Xugu Madison:	Definitely a good idea!
[3:37]	Morgaine Dinova:	Fine by me
[3:37]	Xugu Madison:	Just do put a sign out ahead of time to redirect people, right? :)
[3:37]	Babbage Linden:	(looks like it's TMallocTest, not TCMallocTest, typo...)
[3:38]	Morgaine Dinova:	kk
[3:38]	Babbage Linden:	yes, will do
[3:38]	Phantom Ninetails:	In the unlikely event I'm even up again at that time, I'd have no problems going there instead
[3:39]	Xugu Madison:	Babbage, could you perhaps do a US time office hour on Aditi - we can try getting in everyone who can't make 3am SLT, should help make it busy :)
[3:40]	Babbage Linden:	i might ask some of the other lindens to hold office hours on aditi ;-)
[3:40]	Babbage Linden:	office hours are exhausting even on UK time ;-)
[3:40]	Phantom Ninetails:	lol
[3:41]	Morgaine Dinova:	Really?
[3:41]	Morgaine Dinova:	We're not that hard :-)
[3:41]	Babbage Linden:	not this week :-D
[3:41]	Morgaine Dinova:	Hehe
[3:41]	Phantom Ninetails:	lol
[3:41]	Kaluura Boa:	It's running away from picks and forks which is exhausting...
[3:42]	Ardy Lay:	I would hate to have Blonden's crowd to deal with.
[3:42]	Babbage Linden:	so, I've also been thinking about the best way to do llSetLinkPrimitiveParams without delays
[3:42]	Morgaine Dinova:	Aye. Maybe people should be given a simple hex conversion test before being allowed in a tech OH ;-)
[3:42]	Babbage Linden:	to allow efficient scripts
[3:42]	Phantom Ninetails:	Ooh
[3:43]	Kaluura Boa:	Hooooo....
[3:43]	Babbage Linden:	as much as I dislike it I think just adding a new llSetLinkPrimitiveParamsNoDelay function might be the best way
[3:43]	Ann Otoole:	people that yell and demand figure out they are not going to be heard. works itself out over time
[3:43]	Kaluura Boa:	Plz, just a little shorter name..
[3:43]	Babbage Linden:	as it allows us to move to NoDelay versions of functions over time
[3:43]	Morgaine Dinova:	Nah, readable names are fine.
[3:44]	Phantom Ninetails:	Still don't know why it was decided a delay should be added to llDialog
[3:44]	Xugu Madison:	Please telle me library versioning is coming, though?
[3:44]	Babbage Linden:	in C# we can still make it less horrible by adding the different versions to different Classes/Namespaces
[3:44]	Kaluura Boa:	Long names will be fine when we'll have auto-compleion in editor
[3:44]	Morgaine Dinova nods
[3:44]	Babbage Linden:	Xugu, library versioning already exists for Mono scripts
[3:44]	Kaluura Boa:	auto-completion*
[3:44]	Xugu Madison:	Phantom, I believe because it grabs the viewer's attention and could be spam tool. I think things like dialogs should go into a per-script IM window and we have to go hunt them, but....
[3:45]	Babbage Linden:	you're running against v 1.0 now
[3:45]	Morgaine Dinova:	Kaluura: poor editor isn't Babbage's fault ^_^
[3:45]	Phantom Ninetails:	Xugu, it no longer grabs focus, and the delay has not stopped the spam scripts from crashing viewers
[3:45]	Babbage Linden:	but we can add a new version and still have old scripts link to the old onw
[3:45]	Xugu Madison nods to Phantom "Yeah, but once they put the delay in people started timing scripts around it :( "
[3:45]	Xugu Madison:	Babbage, great!
[3:46]	Babbage Linden:	so, we shouldn't have to add "Fixed" versions of methods any more
[3:46]	Phantom Ninetails:	I make scripts that make heavy use of dialogs, they by no means rely on this bad idea to function, they will only be better without the delay.
[3:47]	Babbage Linden:	although we'll have to decide how we let people opt in to new implementations when we version the libraries
[3:47]	Babbage Linden:	up until now we've been trying to avoid silently changing behaviour, even when it fixes things
[3:48]	Phantom Ninetails:	Yet a delay was added to lldialog.
[3:48]	Kaluura Boa:	I didn't notice it last time my client was crashed by a dialog spam...
[3:49]	Phantom Ninetails:	Yeah, it didn't stop the crashers
[3:49]	Phantom Ninetails:	It's only affecting legit scripts
[3:49]	Babbage Linden:	eventually I'd like to see us move all functions that should be trottled to use the http requestion throttle mechanism
[3:49]	Babbage Linden:	so, instead of using delays per script we have throttles per script resource pool
[3:49]	Phantom Ninetails:	That would be good.
[3:50]	Babbage Linden:	as delays just don't work when you can have any number of scripts, as you know
[3:50]	JB Hancroft:	Is there a writeup on how that works? I'd like to understand that better, please.
[3:50]	Babbage Linden:	there should be on the llHTTPRequest page
[3:50]	JB Hancroft:	ok
[3:51]	Babbage Linden:	at the moment they are throttled per object
[3:51]	Babbage Linden:	which still doesn't work
[3:51]	Babbage Linden:	they should be throttled per pool
[3:51]	Babbage Linden:	with the rate dependent on the pool size
[3:51]	Babbage Linden:	so, we can make simulators make a maximum of N requests/second
[3:52]	Babbage Linden:	and then divide N up between land and avatars as normal
[3:52]	Phantom Ninetails:	http://wiki.secondlife.com/wiki/LSL_http_server#Resource_Limitations
[3:52]	JB Hancroft:	I think there may be instances of people who want to higher-performance sims... is that being considered?
[3:52]	JB Hancroft:	thanks Phantom :)
[3:53]	Babbage Linden:	JB, yes
[3:53]	Babbage Linden:	there are a lot of discussions about different land skus
[3:53]	Phantom Ninetails:	Not sure if this is the limiting method he's referring to but it's one I like
[3:53]	Babbage Linden:	once we have script limits it will be relatively easy to make big memory skus for example
[3:54]	Babbage Linden:	phantom, yes it is
[3:54]	Phantom Ninetails:	Yay
[3:54]	Babbage Linden:	at the moment the http server limits only apply to URLs
[3:54]	JB Hancroft:	it would be nice to have a way to profile a sim, and see what the bottleneck is (I know, stating the obvious?)
[3:54]	Babbage Linden:	script limits will make the same system work for script memory
[3:54]	Ann Otoole:	so we would have to pay more than 300 bucks a month for a sim that works good? that'll go over well rofl
[3:54]	Babbage Linden:	and the throttling I'm talking about will make the same system work for throttling calls
[3:55]	JB Hancroft:	Ann, if I want to pay $500/month for a high-performance sim, there are some needs for that
[3:55]	Xugu Madison:	On the subject of sims, any news on class 6 sims/64 bit stuff?
[3:55]	Phantom Ninetails:	You could buy a whole "high performance" machine yourself for about that price, and run opensim
[3:56]	Babbage Linden:	we've been talking about 64 bit sims in relation to script limits
[3:56]	Ann Otoole:	thats great but 300 a month should get high performance since homesteads are supposed to be cut back versions yet they seem to fare better. but let's leave that debate for Jack's OH lmao
[3:56]	JB Hancroft:	Ann :)
[3:56]	Morgaine Dinova:	Pricing isn't Babbage's area. But as technology improves, service prices normally drop ... that doesn't seem to happen in SL. :-(
[3:57]	Phantom Ninetails:	Indeed
[3:57]	Babbage Linden:	when we switch to 64 bit, mono scripts can need nearly twice as much memory
[3:57]	Babbage Linden:	if you're using lots of lists for example
[3:57]	Babbage Linden:	which are implemented as ArrayLists of objects in .NET
[3:57]	JB Hancroft:	The HTTP limits are per-sim, and not prioritized to the owner in any way, so they are a potential denial-of-service target
[3:57]	Babbage Linden:	each pointer in the ArrayList gets twice as big
[3:57]	Babbage Linden:	(we saw that when we tried 64 bit sims before and had scripts failing)
[3:58]	JB Hancroft:	I can go into a sim, and soak up the resource... choking everything else that wants to get to HTTP?
[3:58]	Babbage Linden:	so, the plan is to pick numbers for script limits that will work for the current sims
[3:58]	Phantom Ninetails:	64 bit only has a rather small performance benefit anyway
[3:59]	Xugu Madison:	I thought the advantage to 64-bit was you could add a LOT more memory?
[3:59]	Babbage Linden:	then deploy 64 bit once the whole grid is running on hosts with more memory and double the script limits at the same time
[3:59]	Babbage Linden:	so, all scripts keep working
[3:59]	Phantom Ninetails:	Even with 32 bit you can have 64 GB of memory, using PAE
[3:59]	Babbage Linden:	all regions keep working
[3:59]	Babbage Linden:	and most scripts have more memory to play with
[4:00]	Babbage Linden:	(once we have C# and arrays of course, many of the reasons to use lists go away, allowing scripts to be even more efficient)
[4:00]	Morgaine Dinova:	Aye
[4:00]	Babbage Linden:	so, I'm glad we have a sensible 64 bit plan now
[4:00]	JB Hancroft:	A whole new set of performance profiling and utilization uses cases
[4:00]	Phantom Ninetails:	I would have thought by now we could get normal arrays in LSL..
[4:01]	Babbage Linden:	running 32 bit sims in an emulated environment on 64 bits hosts has been a pain for a while
[4:01]	Morgaine Dinova:	I never did understand why normal O(1) arrays wren't added to LSL. Don Linden implemented it some 3 years ago. Why didn't it get released?
[4:02]	Morgaine Dinova:	Even fixed size, non-growable would have been fine.
[4:02]	JB Hancroft:	Babbage - Do you use VM's at all, to host multiple sims? It seems like an opportunity to put hardware where and when it's needed.
[4:02]	Babbage Linden:	no idea Morgaine, I didn't know he had, I'll talk to him
[4:03]	Babbage Linden:	in general though, I'd rather get out of the language design game
[4:03]	Babbage Linden:	it's not our job
[4:03]	Morgaine Dinova:	Babbage: he reported it in the SL forums, and was just tidying up corners, he said (my words, but it's in the forums somewhere)
[4:03]	Babbage Linden:	as we've shown with LSL ;-)
[4:03]	Babbage Linden:	ok, I'll ask
[4:03]	Babbage Linden:	right
[4:03]	Babbage Linden:	times up
[4:03]	Babbage Linden:	I'm going to get back to digging in to Mono sim freezes
[4:03]	Ann Otoole:	have a good one
[4:03]	Babbage Linden:	next time: see you on aditi
[4:03]	Phantom Ninetails:	Thanks for the hour
[4:03]	JB Hancroft:	thanks, Babbage :)
[4:04]	Becky Pippen:	Thanks Babbage
[4:04]	JB Hancroft:	take care! :D
[4:04]	Cog: Hello, Avatar!
[4:04]	Morgaine Dinova:	Cool Babbage! Have fun there :-) And thanks :-)
[4:04]	Nock Forager:	Thanks babbage.
[4:04]	Imaze Rhiano:	bye babbage
[4:04]	Phantom Ninetails:	Fare well babbage, everyone
[4:04]	Imaze Rhiano:	anyone have chat log? I was fashionable late...
[4:05]	Nock Forager:	I'll put them up to the wiki in a hour or so.
[4:05]	Phantom Ninetails poof
[4:05]	Morgaine Dinova:	Thanks Nock :-)
[4:06]	Morgaine Dinova looks for Babbage's transcripts page
[4:07]	Cerdita Piek:	Thanks Babbage, take care :)
[4:07]	Imaze Rhiano:	TMalloc test oooo.....
[4:07]	Morgaine Dinova:	Ah, indexed on his own page, cool - https://wiki.secondlife.com/wiki/User:Babbage_Linden