User:Babbage Linden/Office Hours/2009 03 18

Transcript of Babbage Linden's office hours:

 Topics
 * MONO scheduler will change. If you using memory nealy 64K,
   you should check your script on Preview Grid. LINK
 * Script memory limit per percel, per avatar. LINK

[3:59]	Babbage Linden:	thanks for coming
[3:59]	Babbage Linden:	i moved my office hour to 3AM SLT last week
[3:59]	Roberto Salubrius:	geez it's 5am here but good to know
[3:59]	Babbage Linden:	and then managed to miss it this week due to UK daylight saving not happening yet
[4:00]	Babbage Linden:	it's 11AM in brighton where I'm based
[4:00]	Susan Whizenhunt laughs
[4:00]	Babbage Linden:	which is the best time for me as I don't have meetings with the US lindens until the afternooon
[4:01]	Nock Forager:	Timezone difference is one of big problem in SL :)
[4:01]	Susan Whizenhunt:	I've never met a Linden before, so I'm kind of in awe
[4:01]	Babbage Linden:	yes, this time is probably bad for most us residents
[4:01]	Babbage Linden:	but hopefully will be convinient for some people
[4:02]	Susan Whizenhunt:	Or it could be that I haven't had coffee yet
[4:02]	Babbage Linden:	so, normally i talk about what we've been up to regarding scripting
[4:02]	Babbage Linden:	and what our plans are
[4:02]	Roberto Salubrius:	good
[4:02]	Babbage Linden:	and then we talk about any scripting or technology issues people would like to discuss
[4:03]	Babbage Linden:	so, the biggest thing that happened this week is that the new mono scheduler past internal QA
[4:03]	Roberto Salubrius:	ohhhhhh
[4:03]	Roberto Salubrius:	care to share the ins and outs and implications
[4:03]	Roberto Salubrius:	or what can we expect
[4:04]	Babbage Linden:	the plan is that we will deploy that code to the beta grid after the 1.26 server release
[4:04]	Babbage Linden:	it's a set of bug fixes really
[4:04]	Roberto Salubrius:	now new limitations that we should be aware of ?
[4:04]	Babbage Linden:	previously it was possible to exploit the scheduler to get long time slices
[4:05]	Babbage Linden:	that should now be fixed
[4:05]	Babbage Linden:	also, it was possible to temporarily use more than 64K with mono scripts within a time slice
[4:06]	Roberto Salubrius:	mmm
[4:06]	Roberto Salubrius:	do you think that that might break content ?
[4:06]	Babbage Linden:	yes, it will, that's why we're running a beta
[4:07]	Babbage Linden:	so that residents can test their scripts
[4:07]	Babbage Linden:	scripts that are most often affected are those which use list operations like this
[4:07]	Babbage Linden:	l = llListSortList(l);
[4:08]	Babbage Linden:	during the evaluation of that expression there are 2 lists
[4:08]	Babbage Linden:	the copy passed to llListSortList
[4:08]	Babbage Linden:	and the current value of l
[4:08]	Babbage Linden:	after the expression there is only 1 list, l
[4:08]	Babbage Linden:	the copy passed to the function is freed
[4:08]	Babbage Linden:	previously you could get away with this
[4:09]	Babbage Linden:	but the new scheduler checks whenever a script might use too much memory
[4:09]	Babbage Linden:	the advantage of this is that scripts will no longer be affected by scheduler changes
[4:09]	Babbage Linden:	but when we make this change there will be some scripts that break
[4:09]	Babbage Linden:	so, anyone who pushes the limits of script memory in mono
[4:10]	Roberto Salubrius:	ListRandomize, ListFindinList ? those will be affected ?
[4:10]	Babbage Linden:	and uses long lists or strings as arguments to functions should be checked
[4:11]	Babbage Linden:	any scripts containing function calls which take strings or list parameters might be affected
[4:12]	Babbage Linden:	all scripts which run ok on the LSO VM should be fine
[4:12]	Babbage Linden:	but scripts that specifically take advantage of the extra memory mono provides might be affected
[4:12]	Babbage Linden:	it's unfortunate that we're having to make a change which might break content
[4:13]	Babbage Linden:	but hopefully this will be a one off
[4:13]	Roberto Salubrius:	mmm that worries me, not because I have done it "explicitly" but since I don't check ram as often it might hurt some content
[4:14]	Babbage Linden:	then i encourage you to check your scripts on the beta grid when we get the code there
[4:14]	Roberto Salubrius:	yea good to know I hope you make a big announcement
[4:14]	Babbage Linden:	we will make sure we communicate it widely closer to the time
[4:14]	Babbage Linden:	and we'll make sure we run the beta for a good while so people have time to check their code
[4:15]	Babbage Linden:	these changes won't make it on to the main grid until 1.27 server at the earliest
[4:15]	Roberto Salubrius:	will there be "brustable" moments
[4:15]	Babbage Linden:	?
[4:16]	Roberto Salubrius:	meaning that if a script goes over 64K
[4:16]	Roberto Salubrius:	for a small period of time
[4:16]	Roberto Salubrius:	it can burst to 65K for instance
[4:16]	Roberto Salubrius:	then go back to the original 64K ?
[4:17]	Babbage Linden:	memory can only be tested when the script is not running
[4:17]	Babbage Linden:	but what happens now is that the script will yield whenever it might have gone over 64K
[4:18]	Babbage Linden:	whereas before the script would yield at the end of it's time slice
[4:18]	Babbage Linden:	and if you had dropped back down below 64K by then, you got away with using more than 64K
[4:18]	Babbage Linden:	but that meant if anything changed to change when your script yielded
[4:18]	Babbage Linden:	you might stop being lucky and so run out of memory
[4:19]	Babbage Linden:	these changes decouple scheduling and memory accounting
[4:19]	Babbage Linden:	which means that if we need to change the scheduler in future
[4:19]	Babbage Linden:	we won't break scripts
[4:20]	Roberto Salubrius:	mmm well I guess we have to check then
[4:20]	Babbage Linden:	yes
[4:20]	Babbage Linden:	sorry for the extra hassle
[4:20]	Babbage Linden:	it's the right thing to do in the long run
[4:20]	Babbage Linden:	and it's best to do it now
[4:21]	Babbage Linden:	when only 10% of scripts are mono
[4:21]	Babbage Linden:	than when 90% of scripts are mono
[4:21]	Roberto Salubrius:	do you have a percentaje of the ram that people get to use
[4:21]	Roberto Salubrius:	I mean
[4:21]	Roberto Salubrius:	do you have a number and say ok some people go up to 65K
[4:21]	Roberto Salubrius:	or something like that ?
[4:22]	Roberto Salubrius:	I am going to the fact that is it possible to change the scheduler and while you are at it give us more ram
[4:22]	Babbage Linden:	yes, that's possible
[4:22]	Babbage Linden:	i'd like to see how many scripts are affected first
[4:23]	Roberto Salubrius:	that will allow a change for the scheduler, and well not break any content
[4:23]	Siann Beck:	So long as your script isn't using, say, 128k :)
[4:23]	Roberto Salubrius:	well it's good to know you are considering it
[4:24]	Babbage Linden:	there is a separate project we're working on which we limit script memory based on parcel ownership
[4:24]	Roberto Salubrius:	the problem is that we have no idea of how much ram a script might be using... or what is the real limit on the scheduler right now
[4:24]	Babbage Linden:	and give avatars and allowance for attachments
[4:24]	Babbage Linden:	which will work like prim limits
[4:25]	Roberto Salubrius:	so limit ram and or script time per avatar
[4:25]	Roberto Salubrius:	and per parcel ?
[4:25]	Babbage Linden:	yes
[4:26]	Babbage Linden:	when scripts are running in world, they will request memory from the parcel's script memory pool
[4:26]	Babbage Linden:	when scripts are attachments, they will request memory from the avatar's script pool
[4:26]	Roberto Salubrius:	mmmm.... I guess you guys are studing what an average avatar uses right now, add some and then limit up there.
[4:27]	Siann Beck:	Wouldn't you have a problem then of a script working in some places but not others? How do you communicate that effectively to non-techie customers?
[4:27]	Babbage Linden:	if the memory is not available, scripts won't rez
[4:27]	Babbage Linden:	the idea of having a constant size avatar pool is that if you can attach an object, it will work wherever you go
[4:28]	Babbage Linden:	with land pools, the process will work the same as with scripts
[4:28]	Babbage Linden:	instead of being told you don't have prims available, you will be told you don't have script memory available
[4:28]	Siann Beck:	Right, but if I'm selling a script or a scripted object, how do I tell prospective buyers, "This may or may not work on your parcel".
[4:28]	Nock Forager:	hmhm
[4:29]	Roberto Salubrius:	I really think that will make a lot of people angry
[4:29]	Babbage Linden:	you will need to sell it as a 6 prim, 16KB object
[4:29]	Babbage Linden:	people are widely aware of prim limits
[4:29]	Siann Beck:	Sure, but not memory limits.
[4:29]	Roberto Salubrius:	babbage so are we going to get the thing we have asked for a long time, to get to know the "script rendering cost" and now "memory cost" of each script ?
[4:30]	Babbage Linden:	"memory limits are like prim limits: you get them with your land"
[4:30]	Babbage Linden:	yes
[4:30]	Babbage Linden:	the first phase is to do the accounting, to work out how much memory scripts on avatars and parcels are using
[4:30]	Babbage Linden:	the next phase is to make that visible
[4:31]	Babbage Linden:	the final phase is to enforce it
[4:31]	Siann Beck:	And what happens when you implement it, and someone's existing content suddenly is over the memory limit?
[4:31]	Babbage Linden:	so, there will be a long period where you will be able to see how much memory is being used on your parcel
[4:31]	Siann Beck:	And they come screaming to the developer?
[4:31]	Roberto Salubrius:	Babbage I mean from your POV and me being a system adminstrator also and a RL programmer I understand, why this is being implemented
[4:32]	Roberto Salubrius:	or being looked at
[4:32]	Babbage Linden:	and if you're over you'll be able to reconfigure your land
[4:32]	Roberto Salubrius:	but from the other side of the fence, as a user, I see this as a possible form of mayhem
[4:32]	Babbage Linden:	the reason this is being worked on is that currently you can use as much memory as you like for scripts
[4:32]	Babbage Linden:	where regions have lots of scripts running, they cause simulators to swap
[4:33]	Babbage Linden:	which destroys the performance of all regions running on that simulator host
[4:33]	Babbage Linden:	in order to avoid lag we need to limit the size of the simulator processes
[4:33]	Roberto Salubrius:	but your virtualization system ... that already does not implement limits on the simulator usage ?
[4:34]	Babbage Linden:	it only enforces per script limits currently
[4:34]	Babbage Linden:	which is both annoying (because you can't make a big script)
[4:34]	Babbage Linden:	and useless (because you just add more scripts, memory usage is not limited)
[4:35]	Babbage Linden:	once we have parcel and avatar memory pools we can actually limit script memory usage
[4:35]	Babbage Linden:	(which stops the simulators swapping)
[4:35]	Babbage Linden:	and we can allow scripts to use more memory
[4:35]	Roberto Salubrius:	ok so the core of the issue was a problem implementing the virtualization when SL was convieved
[4:35]	Babbage Linden:	we could add library calls which request more memory for a script
[4:35]	Roberto Salubrius:	concieved
[4:36]	Babbage Linden:	yes, the LSO memory limit made it easy to implement scripts that can move around between simulators
[4:36]	Babbage Linden:	you just ship a 16K block to another machine
[4:36]	Babbage Linden:	with Mono there is no reason to limit a script to any amount
[4:36]	Babbage Linden:	big scripts will take longer to ship to another machine
[4:36]	Babbage Linden:	but some scripts should just be able to sit in a region using 2MB of memory if that makes sense
[4:37]	Babbage Linden:	but, you can't allow scripts to just keep requesting memory
[4:37]	Babbage Linden:	you need some kind of overall simulator limit
[4:37]	Siann Beck:	I'm going to have to run. See you all later!
[4:37]	Nock Forager:	bye Siann
[4:37]	Babbage Linden:	and the fairest way to do that is based on parcel ownership
[4:37]	Babbage Linden:	bye siann
[4:38]	Babbage Linden:	thanks for coming
[4:38]	Roberto Salubrius:	I know this is an open forum to talk about things, and this are just "ideas" right now, but I have to say, that well scripting on SL has never been easy, we get a lot of limitations, but now to also limit ram usage, like this, will make things a real nightmare for us coders.
[4:39]	Roberto Salubrius:	ok now I have a question, as we all know scripts grow
[4:39]	Roberto Salubrius:	or can grow
[4:39]	Roberto Salubrius:	for instance a simple visitor counter
[4:39]	Roberto Salubrius:	when you rez it it has a requirement of 2K of ram
[4:39]	Roberto Salubrius:	then it logs 100 visitors
[4:39]	Roberto Salubrius:	and it needs 4K of ram
[4:40]	Roberto Salubrius:	what is the scheme planned for issues like that one ?
[4:40]	Babbage Linden:	yes, so what you'd like to be able to do
[4:40]	Babbage Linden:	is rez the script with a default allocation of memory
[4:40]	Babbage Linden:	then have a library call that can get the current memory usage
[4:40]	Babbage Linden:	and another that can ask for more memory if required
[4:41]	Babbage Linden:	instead of running out of memory at 16KB or 64KB, it can grow as needed
[4:41]	Babbage Linden:	only running out of memory when it can no longer ask for more
[4:42]	Roberto Salubrius:	ok so we are gonna have to implement something like malloc, and free for our scripts
[4:42]	Babbage Linden:	up to you
[4:43]	Babbage Linden:	most people will want to build objects that have a fixed memory and prim requirement
[4:43]	Roberto Salubrius:	how will that work with previous content that have no idea of memory managment and renders it all to the server side ?
[4:43]	Babbage Linden:	"this object needs 6 prims and 64KB"
[4:43]	Roberto Salubrius:	renders memory usage to the server side I mean
[4:43]	Roberto Salubrius:	ugh memory managment
[4:43]	Babbage Linden:	but others will want to allow dynamically growing scripts
[4:43]	Babbage Linden:	depends what you're building
[4:44]	Babbage Linden:	previous content will just use fixed amounts
[4:44]	Babbage Linden:	LSL scripts will need 16KB
[4:44]	Babbage Linden:	Mono scripts will need 64KB
[4:44]	Babbage Linden:	but for new scripts you will have much more flexibility
[4:44]	Roberto Salubrius:	and that will be set per prim or per script ?
[4:45]	Babbage Linden:	and not need to chop your script in half and use link messages every time you need another 16KB
[4:45]	Babbage Linden:	scripts will have a memory requirement
[4:45]	Roberto Salubrius:	ok
[4:45]	Babbage Linden:	parcels and avatars will have a memory pool
[4:45]	Babbage Linden:	in order to rez a script you need to have enough memory available in the pool to fulfil the requirement
[4:46]	Babbage Linden:	just like a prim limit
[4:46]	Babbage Linden:	but separate
[4:46]	Babbage Linden:	you might have a 1 prim object that needs 1MB
[4:46]	Babbage Linden:	or a 100 prim object that needs 16KB
[4:46]	Babbage Linden:	depends what you're building
[4:46]	Roberto Salubrius:	no I understand the implementation, I am worried about the implications of this
[4:46]	Babbage Linden:	if you want to find out how it will work, go and play with HTTP In
[4:47]	Babbage Linden:	which uses this system to ration public URLs
[4:47]	Roberto Salubrius:	yes I worked with it already
[4:47]	Babbage Linden:	and was a good first resource to limit as it is a new resource
[4:47]	Babbage Linden:	so, you know how this is going to work already then
[4:47]	Roberto Salubrius:	I posted to the jira that how it was implemented I was able to render a sim with 0 http-in connections
[4:47]	Roberto Salubrius:	with one script
[4:48]	Roberto Salubrius:	i hogged all the connections
[4:48]	Roberto Salubrius:	for the sim ( when peri asked us to do the closed beta test )
[4:48]	Babbage Linden:	which is much better than being able to rez 100,000 scripts making the sim swap and completely unusable for all users
[4:48]	Babbage Linden:	;-)
[4:48]	Babbage Linden:	which happens currently
[4:49]	Babbage Linden:	not having limits doesn't work
[4:49]	Babbage Linden:	so we need to have limits and make them as workable as possible
[4:49]	Babbage Linden:	you could only hog all the connections as you were using a sandbox presumably
[4:49]	Roberto Salubrius:	ok I insist I do understand it all form a system admin POV, but as a user, I think it will be hard on people
[4:49]	Babbage Linden:	and also, you couldn't hog all the connections for all the avatars
[4:49]	Nock Forager:	Do you have in your mind the limitation numbers? KB/sqm or something.
[4:49]	Roberto Salubrius:	specially non techs
[4:50]	Babbage Linden:	we're currently analysing the script usage across the grid
[4:50]	Babbage Linden:	to come up with numbers that will break as little content as possible
[4:50]	Roberto Salubrius:	remember when http-in was first first implemented the first draft it had the connection limit based on the sim not on the parcel
[4:50]	Babbage Linden:	there will be edge cases were people are running 2000 scripts on a 512m parcel of land
[4:51]	Babbage Linden:	those will stop working
[4:51]	Roberto Salubrius:	ok
[4:51]	Babbage Linden:	but we should be able to make nearly all parcels work fine
[4:51]	Nock Forager:	2000 on 512sqm lol
[4:51]	Babbage Linden:	by picking the right numbers
[4:51]	Babbage Linden:	and actually, running tons of scripts on a tiny parcel is griefing all the other parcel owners anyway
[4:52]	Roberto Salubrius:	yes of course
[4:52]	Babbage Linden:	i agree that this will require lots of communication to get right
[4:52]	Babbage Linden:	but ultimately we need to do it
[4:52]	Roberto Salubrius:	and there are some attachments that will bring a sim down to it's knees like some multitools out there
[4:52]	Babbage Linden:	and second life will be a better place because of it
[4:52]	Babbage Linden:	exactly, those will stop rezzing
[4:53]	Babbage Linden:	but again, we will pick numbers for avatar pools that will mean that most people can continue to use the scripted attachments they currently use
[4:53]	Babbage Linden:	only extreme edge cases should be affected
[4:53]	Roberto Salubrius:	ok
[4:53]	Babbage Linden:	and if you set your land to not allow other people to rez objects
[4:53]	Roberto Salubrius:	well I guess we will have enough time on the beta grid to test it
[4:54]	Babbage Linden:	then you should be able to set up a parcel with a set of scripts
[4:54]	Babbage Linden:	and it will just keep working
[4:54]	Babbage Linden:	without swapping
[4:54]	Babbage Linden:	regardless of who turns up[
[4:54]	Babbage Linden:	yes, there will be lots of testing time
[4:54]	Susan Whizenhunt:	I'm not sure if this is off-topic, but is memory usage the main cause of lag? Or are there ways in which a simple script can be as taxing on a SIM?
[4:54]	Babbage Linden:	and lots of communications
[4:54]	Babbage Linden:	CPU time is already limited
[4:55]	Babbage Linden:	a sim will only run scripts for a few ms per frame
[4:55]	Babbage Linden:	but, when you have tons of scripts in memory
[4:55]	Babbage Linden:	and cause the simulator host to swap, everything lags
[4:55]	Babbage Linden:	for all regions running on that host
[4:55]	Babbage Linden:	making servers touch their hard drives is horrible for performance
[4:56]	Roberto Salubrius:	Susan you can basically crash a sim on 1 script
[4:56]	Babbage Linden:	so, if you have scripts that sit in tight loops doing complex calculations
[4:56]	Babbage Linden:	they will slow each other down
[4:56]	Babbage Linden:	but non-script performance should remain fine
[4:56]	Babbage Linden:	if you have lots of scripts filling memory
[4:57]	Babbage Linden:	then the performance of every aspect of every region on the host will be affected
[4:57]	Babbage Linden:	ok, we're just about out of time
[4:57]	Babbage Linden:	and i need to get back to work
[4:57]	Babbage Linden:	thanks for coming everyone
[4:57]	Roberto Salubrius:	one last question
[4:58]	Roberto Salubrius:	when will we see httpd in
[4:58]	Roberto Salubrius:	on the main grid
[4:58]	Babbage Linden:	1.27
[4:58]	Babbage Linden:	ok, i'm off
[4:58]	Babbage Linden:	hope to see you next week
[4:58]	Nock Forager:	THanks for the meeting Babbage.
[4:58]	Susan Whizenhunt:	Thanks
[4:58]	Roberto Salubrius:	thx
[4:58]	Roberto Salubrius:	have a nice day
[4:59]	Roberto Salubrius:	geez that only wories me
[4:59]	Roberto Salubrius:	or you guys as well ?
[5:00]	Nock Forager:	Need time to educate others...