User:Babbage Linden/Office Hours/2010 03 17

From Second Life Wiki
Jump to navigation Jump to search

Transcript of Babbage Linden's office hours:

[04:07] Latif Khalifa: hello Babbage :)

[04:07] Nock Forager: hello babbage

[04:07] Babbage Linden: hi everyone

[04:07] Imaze Rhiano: hi Babbie

[04:07] ANSI Soderstrom: hi all

[04:07] Babbage Linden: still very dark here when it's night...

[04:08] Babbage Linden: ok, what would everyone like to talk about today?

[04:08] Babbage Linden: i have a topic: script limits reserved memory mechanisms

[04:08] Babbage Linden: anything else/

[04:08] Babbage Linden: ?

[04:08] Ceawlin Steamweaver: I have questions regarding the fate of arbitrary limits imposted on http body sizes, if they will be removed when we get bigscripts. :3

[04:08] Ceawlin Steamweaver: Or even before. :3 Lol.

[04:09] Latif Khalifa: Babbage, I've seen your talk about c# api, and how you'd be publishing draft and soliciting feedback, perhaps you could tell us a bit about that process

[04:09] Babbage Linden: good one, any others?

[04:09] Babbage Linden: c# api process yep

[04:09] Babbage Linden: and fosdem talk

[04:10] Imaze Rhiano: any plans to get shared memory visible for memory usage window? or is that abandoned now?

[04:10] Babbage Linden: shared memory visibility

[04:10] Babbage Linden: ok

[04:10] Babbage Linden: that will probably keep us busy

[04:10] Babbage Linden: any more?

[04:11] Babbage Linden: ok, let's start

[04:11] Babbage Linden: so, we've been doing lots of thinking about memory reservation recently

[04:11] ScottJoins Alcove: Babbage Linden have you ever heared of Ratcloner101 ?

[04:11] Babbage Linden: we *really* want to be able to have maximum reserved memory sizes for scripts

[04:12] Babbage Linden: so that residents can set up scripted objects

[04:12] Babbage Linden: and know that they will continue to work indefinitely

[04:12] Ceawlin Steamweaver: Creator-set maximums or hard limits built in?

[04:12] Babbage Linden: so, that's the thing

[04:13] Babbage Linden: where creators set limits, it's fine

[04:13] Babbage Linden: while developing a scripted object you can test how much memory you need

[04:13] Latif Khalifa: sounds reasonable to me. guess the biggest trouble would be setting numbers to those limits and determening how much you can overcommit per sim not to end up with mostly unused memory

[04:13] Babbage Linden: and pick a limit that works

[04:13] Babbage Linden: the problem is for all the legacy scripted objects that don't currently have limits

[04:13] Babbage Linden: in those cases we have to be conservative

[04:13] Babbage Linden: setting 16KB for LSL scripts

[04:14] Babbage Linden: and 64KB for Mono scripts (which have been able to use 64KB up until now)

[04:14] Imaze Rhiano: is it going to be possible later allocate more/less memory? is it going to be function that, compiler parameter, pragma...?

[04:14] Babbage Linden: the effect of being conservative is that Mono scripts often look a lot worse to residents than LSL scripts

[04:14] Babbage Linden: even though they are generally better from our perspective

[04:15] Babbage Linden: as on average they are using 9KB of memory

[04:15] Babbage Linden: (although with ha 64KB maximum)

[04:15] Ceawlin Steamweaver: I think y'all should just depricate legacy LSL once the mono freeze bugs and such are worked out. ;P

[04:15] Ceawlin Steamweaver: /me ducks

[04:15] Babbage Linden: so, that is definitely an option we've been thinking about ceawllin

[04:15] Ceawlin Steamweaver: !!

[04:15] Babbage Linden: but have no plans to do it any time soon

[04:16] Latif Khalifa: there are gobs of content out there

[04:16] Babbage Linden: even if we do, we will be supporting running legacy LSL scripts

[04:16] Babbage Linden: the only change would be limiting compilation to LSL for new script

[04:16] Ann Otoole: so LL can force all scripts to recompile. whats the problem?

[04:16] Babbage Linden: ann, if a script recompiles, it resets

[04:17] Babbage Linden: as scripts keep data in memory, that's not good

[04:17] Babbage Linden: we would need to extract the state from the script and convert it to mono

[04:17] Babbage Linden: which is not trivial

[04:17] Ceawlin Steamweaver: That would suck, lol.

[04:17] Latif Khalifa: and bould for breakage in some unforseen edge case

[04:17] Ann Otoole: well there is stuff people bought that cannot be recompiled and the creators are not going to touch it so there needs to be a way to deal with that issue case

[04:17] Latif Khalifa: bound

[04:18] Babbage Linden: right, it's a hard one

[04:18] Babbage Linden: so, the current plan is to reserve 64KB for legacy (existing) mono scripts and 16KB for LSL scripts

[04:19] Latif Khalifa: well compiling new lso scripts could be disabled at some point. but i know people still compile lso on purpose, especially for scripted attachment as they perform better in region crossing

[04:19] Babbage Linden: is that going to be a problem? is it going to cause lots of people to convert back to LSL?

[04:19] Babbage Linden: if so, what should we do about it?

[04:19] Imaze Rhiano: well... you can't make an omelette without breaking eggs - and memory limits are going to break some content - that is pretty much purpose of it

[04:19] Ann Otoole: everyone has their own fav set of circumstances. i have a public photo garden with a lot of modeling pose balls and each has about 100 to 150 poses loaded into a list in memory so i need to know what the parcel limits are and how much damage i am looking at.

[04:20] Ceawlin Steamweaver: I think the best you can try to do is educate people that Mono is better, even though the numbers are larger, but if they are too hard-headed, that's their problem. >_>

[04:20] Babbage Linden: would showing the used memory along side the reserved memory help?

[04:20] Babbage Linden: or just be more confusing/

[04:20] Ardy Lay: Babbage, that's what a lot of people are asking for.

[04:20] Gypsy Paz: showing used memory would be awesome!!!

[04:20] Ceawlin Steamweaver: If you showed LSO/Mono cpu usage, they'd probably wise up. :P

[04:21] Imaze Rhiano: how about showing memory that shared between mono scripts? - and script time would be super too...

[04:21] Ann Otoole: i need to know what is used so i can see how much value i lose in the 300 bucks a month i have been payiong for sercvices that will now potentially be crippled with no reduction in fees

[04:21] Latif Khalifa: well except they are not charged against parcel memory limits for used memory... so it's a tricky thing

[04:21] Babbage Linden: ok, so that's a related issue

[04:22] Babbage Linden: if we were to surface shared bytecode memory how would that be displayed?

[04:22] Ceawlin Steamweaver: Even a shared memory guesstimate alongside of something somewhere would be helpful, even if it's not 100% accurate all the time...

[04:22] Babbage Linden: would you rez a script and have it shown as 64K reserved

[04:22] Ann Otoole: now if the pose script can be shared across a hundred units used by different people and it only counts once... but the thing is each ball loads a different set of animations to mem so how does that work?

[04:22] Babbage Linden: then rez another copy and have the reserved size go down to 60K

[04:23] Babbage Linden: then when you derez it have it go back up to 64K

[04:23] Ceawlin Steamweaver: 64k reserved, 2k used 1k shared (3k total) or something? The 3k total would remain the same, or something?

[04:23] Ann Otoole: i think it needs to show reality. what it really uses.

[04:23] Latif Khalifa: Ann, script memory usage counts both the compiled code, and the data codes uses. If you have exact same script and recompile they are going to share the code part

[04:23] Latif Khalifa: i mean don't recompile

[04:24] Imaze Rhiano: I don't have good answer to that Babbie... not sure is there any good way and same time keep it deterministic

[04:25] Ann Otoole: code is great. so the memory is wherte the limits are. so we need to know what memory is aggregated as used and know what the limits for the parcel are

[04:25] Babbage Linden: that's ok, we don't have all the answers yet, either, just wanted to let you know what we are thinking about

[04:25] Ann Otoole: then there is the issue of when an avatar sits on a pose ball that pose ball currently counts towards the avatar and against the parcel i bet

[04:25] Babbage Linden: i agree with showing reality

[04:25] Ceawlin Steamweaver: At the risk of making someone mad, I think it may be kinda hair-splitting anyway. There's like, bigger fish to fry wrt memory issues, in my opinion... I say get it working and then figure out how to expose the data. :P

[04:25] Babbage Linden: there may be a period where LSL looks more attractive

[04:26] Latif Khalifa: Ann, that was fixed in the last version of sim

[04:26] Babbage Linden: but in the long term I think people will use Mono

[04:26] Babbage Linden: another question is, what happens if Mono script memory usage increases

[04:26] Ceawlin Steamweaver: I dunno. Some people still swear that cars from the 1960s are better than new ones. <_< Lol.

[04:27] Babbage Linden: for example, if we move to 64 bit simulator code

[04:27] Babbage Linden: the pointer sizes in Mono scripts will increase

[04:27] Ceawlin Steamweaver: Is there any compelling reason to move to 64 bit sim code?

[04:27] Babbage Linden: increasing the mono memory usage

[04:28] Babbage Linden: ceawlin, we will eventually move to sim hosts with more memory

[04:28] ScottJoins Alcove: I been haveing problem getting viewer to start on windows 7 it saids the driver out dated?

[04:28] Babbage Linden: and more memory per process

[04:28] Babbage Linden: at that point we'll increase the memory available to scripts

[04:28] Babbage Linden: but their memory usage will increase too

[04:28] Ceawlin Steamweaver: Ahh, I guess someday a single process might exceed 2g huh? :<

[04:28] Latif Khalifa: 4gb

[04:29] Actingill Igaly: is it not worth waiting until that point to impose limits or is that too far away?

[04:29] Babbage Linden: do, we just rewrite the reserved memory numbers for Mono scripts?

[04:29] Ceawlin Steamweaver: Oh you can have 4g processes with a 32bit pae kernel? That is massive, lol.

[04:29] Babbage Linden: it's quite a long way away

[04:29] Babbage Linden: we want script limits before we upgrade to help with the swapping problems we have now

[04:29] Actingill Igaly: but would probably solve the problems people are going to get from script limits

[04:29] ScottJoins Alcove: I was woundering can i run 64 bit on 32 bit vaio ?

[04:29] Actingill Igaly: ah ok

[04:30] Latif Khalifa: well there are class 7 sims like this one, and they have more memory, but there is now 8 sims on it, not more room per sim

[04:30] Babbage Linden: but we also need to be able to make the script limits mechanism work in the face of scripts using more memory

[04:30] ScottJoins Alcove: CPU: Intel Core 2 Series Processor (2194 MHz)

  Memory: 2046 MB

  OS Version: Microsoft Windows Vista (Build 6000)

  Graphics Card Vendor: NVIDIA Corporation

  Graphics Card: GeForce 8600M GT/PCI/SSE2

  Windows Graphics Driver Version: 7.15.0011.5652

  OpenGL Version: 2.1.1

  libcurl Version: libcurl/7.18.1 OpenSSL/0.9.8j zlib/1.2.3

  J2C Decoder Version: Emerald Kakadu(v6.3.1)

  Audio Driver Version: FMOD version 3.750000

  LLMozLib Version: [LLMediaImplLLMozLib] - 2.01.31261 (Mozilla GRE version

  Packets Lost: 255/576596 (0.0%)

[04:30] Babbage Linden: (or less, although that is less of a problem)

[04:30] ScottJoins Alcove: ?

[04:31] Actingill Igaly: do we have any figures for the percentage of existing scripts that would fall foul of memory limits

[04:31] Babbage Linden: so, were doing a lot of thinking about that at the moment

[04:31] Babbage Linden: individual scripts won't fall foul of limits

[04:32] Babbage Linden: but some parcels or avatars will

[04:32] Babbage Linden: we're still working on the numbers

[04:32] Melfina Marshdevil: any idea what the initial numbers are?

[04:32] Actingill Igaly: but they are being compiled - thats a good thing :)

[04:32] Babbage Linden: yes, the numbers are being compiled

[04:32] Babbage Linden: we have some initial strawman numbers

[04:33] Actingill Igaly: care to share? :)

[04:33] Melfina Marshdevil: care to share them?

[04:33] Actingill Igaly: jinx

[04:33] Xugu Madison: Babbage, want to whisper them to me? :)

[04:33] Babbage Linden: but want to know how many people are going to be affected by them before we talk about them

[04:33] Babbage Linden: and I think jack is going to be communicating them when we're ready

[04:33] Ann Otoole: i'm curious about the relationship of memory to script time. Argent claimed on SLDEV list that the time is directly correlated to memory. and running out of spare time = page swapping. Is that true?

[04:33] Ann Otoole: being as how script time is what makes my sim lag

[04:34] Chaley May: what happens when an owner has scripts that use setscriptstate to activate another script but the script would go past the limit? would it allow for that script to activate and deactivate scripts of other people on the parcel?

[04:34] Latif Khalifa: you can use all of the script time and stil have no paging

[04:34] Babbage Linden: no, CPU and memory are different

[04:34] Babbage Linden: if you use lots of CPU time for scripts, only other scripts will slow down

[04:34] Ann Otoole: ok so these limits are not going to do squat for sim performance unless avatars are nerfed

[04:34] Babbage Linden: you might notice "lag" when interacting with scripted objects in that case

[04:35] Babbage Linden: as the script you're trying to interact with might be waiting for a time slice

[04:35] Xugu Madison: however memory usage can increase disk usage which can increase actual ("wall-clock") time taken

[04:35] Actingill Igaly: scripts that are deactivated still use memory so i assume deactivating them will have no effect on overal limits just cpu

[04:35] Ann Otoole: thats a lot of lost money. better start planning to change to a new account to avoid refund demands lol

[04:35] Babbage Linden: when scripts use too much memory and the simulator host starts swapping, everything slows down

[04:35] Babbage Linden: whole frames take longer

[04:35] Babbage Linden: and everything feels laggy, even when you're not interacting with scripts

[04:36] Babbage Linden: so, in that sense paging is worse

[04:36] Ann Otoole: well in my sim after watching carefully for over a year when a wad of fashionistas with cruddy scripts waddle in and spare time stays at zero then the rubber bannding begins along with pending downloads rising

[04:37] Latif Khalifa: script limits will help avoid swapping which is the biggest sim perfomance killer

[04:37] Babbage Linden: right, so it sounds like you have swapping there ann

[04:38] Babbage Linden: which script limits will help

[04:38] Ann Otoole: it is worth noting the frame time remains at 22.4 while this lag is present

[04:38] Ceawlin Steamweaver: Just think how much more efficient your priducts will be without all the junky slave scripts and being able to exceed 64k when you have to write something with a ton of code in it. It'll either equal out or improve, I bet. :P

[04:38] Ceawlin Steamweaver: *products, too

[04:38] Actingill Igaly: when scripts get deactivated, will there be a heirachy? I dont want someone with a huge overhead coming into my shop and stopping my demos from working. Will land owner scripts have priority?

[04:38] Babbage Linden: the goal is that scripts don't get deactivated

[04:39] Babbage Linden: parcel owners will get a pool of memory for scripts

[04:39] Ann Otoole: sadly most people wearing the really bad and most expensive shoes that cost 0.5msec a pair don't even use the damn scripts once they set their options.

[04:39] Babbage Linden: and if you can rez them they will keep running

[04:39] Actingill Igaly: so the server will prevent rezzing

[04:39] Babbage Linden: avatar attachments will use the avatar pool

[04:40] Ceawlin Steamweaver: Babbage, do you have any idea how much total ram will be given to scripts for the whole sim, even if just a guess?

[04:40] Babbage Linden: and our goal is to get the numbers to a point where even if the sim is full, with fully loaded avatars and fully used land it will still run without swapping

[04:40] Babbage Linden: yes, i do, but again, I don't want to talk numbers until we're ready

[04:40] Actingill Igaly: if thats the case then there probably isn't a need to worry about the script limits for the majority

[04:40] Ceawlin Steamweaver: Ok. :P

[04:41] Latif Khalifa: that goal would leave a lot of wasted memory in averge case

[04:41] Melfina Marshdevil: Ohhh what about customizing those limits. I heard object bonus effects the parcel memory pool.

[04:41] Latif Khalifa: since it's impossible to have every script in a sim use exactly the reserved memory

[04:41] Babbage Linden: latif, yes

[04:41] Melfina Marshdevil: also can you customize per avatar limits in the future? o.o

[04:42] Babbage Linden: but, lag is currently the biggest complaint, so that's what we're aiming to address

[04:42] Latif Khalifa: some overcommitment is neccessary imho

[04:42] Ardy Lay: The goal is not to achieve 100% utilisation, it's to prevent more than 100% demand causing paging.

[04:42] Babbage Linden: we are also looking at overcommitting

[04:42] Babbage Linden: so, we can make the best use of resources in most cases

[04:43] Babbage Linden: because not all systems in the simulator are limited

[04:43] Xugu Madison: Any hope for showing and taking into account bytecode sharing when calculating usage?

[04:43] Babbage Linden: even when we introduce script limits

[04:43] Babbage Linden: there will be some simulators that swap

[04:43] Ann Otoole: well my sim runs great until avatars exceed 10. i hope my sim doesnt get nerfed for no reason.

[04:43] Babbage Linden: so, our goal is to pick numbers that reduce the amount of swapping

[04:43] Babbage Linden: without inconviniencing many residents

[04:44] Babbage Linden: as those things trade off, we may end up with numbers that slightly over commit resources in some cases

[04:44] Babbage Linden: ann, it sounds like in your case that the fashionistas that visit will rationalise their bling a little

[04:45] Babbage Linden: and then your sim will run better when it's busy

[04:45] Ceawlin Steamweaver: Lol

[04:45] Imaze Rhiano: :P

[04:45] Babbage Linden: and there will be a new market in eco-bling for them ;-)

[04:45] Ann Otoole: ok babbage i don't like naming names but i will so you guys can go buy some stuff to test. Stiletto Moody shoes eat 0.500 msec a pair.

[04:46] Latif Khalifa: what is really important I think is to get the small script project out on the grid ASAP so content makers can adjust their product ahead of the time

[04:46] Ann Otoole: since LL heavily promotesd SM shoes you guys might want to help that company before they have a million refund demands

[04:46] Babbage Linden: latif, yes that is still the plan

[04:46] Actingill Igaly: Babbage - if i disable scripts for everyone except group, will people coming in still use memory to store the variables the script had on entry?

[04:46] Ceawlin Steamweaver: Large scripts too. :3 A lot of what I am working on these days wastes a ton of memory from having to split a script into 3 or 4. X_x

[04:47] Ann Otoole: and you can forget people buying a new pair of L$2000 shoes because you decided to install limits rofl. huge revolt.

[04:47] Latif Khalifa: Actingill, avatar memory is not counted against your parcel limit, so they will not affect it

[04:47] ANSI Soderstrom: sometimes i move over sims with "all scripts disabled" and cant walk faster as 1m per hour.... iam sure that he scripts are not the only "big" guiltily

[04:47] Actingill Igaly: ok sorry i missed the start of the convo :)

[04:47] Babbage Linden: ok, let's talk about the C# API

[04:48] Babbage Linden: the video that latif was talking about is here:

[04:48] Babbage Linden: it's my talk at FOSDEM earlier this year

[04:48] Babbage Linden: which talks a lot about second life, scripting and mono

[04:48] Babbage Linden: and our plans for the future

[04:49] Ann Otoole: C# will make me interested in scripting finally lol. till now it has been only as needed.

[04:49] Babbage Linden: including script limits and C#

[04:49] Ceawlin Steamweaver: Dang, I thought you were older. *sticks foot in mouth* :P

[04:49] Actingill Igaly: lol

[04:49] Babbage Linden: the slides are also available here:

[04:50] Babbage Linden: amongst other things it talks about designing the API that C# scripts will use

[04:50] Babbage Linden: and shows some C# code running on our current prototype

[04:51] Babbage Linden: we're working on getting a version 1 API defined in this sprint

[04:51] Babbage Linden: and enforced on the CoreCLR sandbox that we got working in SL last week

[04:51] Latif Khalifa: yey

[04:51] Babbage Linden: that consists of 2 parts

[04:52] Babbage Linden: defining an SL "profile" which is the subset of the .NET framework that is available

[04:52] Babbage Linden: and defining the SL specific APIs that scripts will use

[04:52] Babbage Linden: the profile is currently looking like it will initially be the EMCA-335 defined base class library

[04:53] Babbage Linden: minus the System.Net namespace which allows network access without throttling and duplicates a lot of the HTTP-In and HTTP-Out functionality

[04:53] Ceawlin Steamweaver: Awwwww, lol. XD

[04:53] Latif Khalifa: makes sense

[04:54] Melfina Marshdevil: no opening raw sockets? :P

[04:54] Latif Khalifa: haha

[04:54] Xugu Madison: /me nods "I saw some of this stuff on the Moonlight ML. Looks like you're making good progress!"

[04:54] Ceawlin Steamweaver: Man, if I could open a persistent TCP socket in LSL, I'd me in hawg heaven. :3

[04:54] Ceawlin Steamweaver: *be

[04:54] Ceawlin Steamweaver: *LSL/C#/w/e

[04:54] Babbage Linden: there are PDFs describing those libraries here:

[04:55] Latif Khalifa: I know there is a lot of people who breathe c# and are also heavy users of SL, so perhaps having some way of getting feedback on api proposa would be benefitial for all :)

[04:55] Melfina Marshdevil: yeah but just think what could happen if we could open tcp sockets to "anything" on LLNet

[04:55] Babbage Linden: we are planning to reuse the security attributes from Moonlight

[04:55] Babbage Linden: so, some of those APIs will not be fully available

[04:56] Babbage Linden: for example, you won't be able to use System.IO.File.Delete...

[04:56] Babbage Linden: so, to see which parts will be available, you can check the Silverlight documentation for those namespaces

[04:56] Babbage Linden: I'm just about to start a pass on the Linden Lab classes now

[04:57] Babbage Linden: most of the classes already exist and are being used by LSL scripts

[04:57] Babbage Linden: but will need their interfaces expanding to be generally useful for C# scripts

[04:57] Latif Khalifa: System.Collections will make us feel like we were released from a straight jacket of LSL, also byref symantic. Hope you don't forse linden api to be by val like LSL does

[04:57] Babbage Linden: for example Vectors and Quaternions need to be able to be parsed and formatted

[04:58] Babbage Linden: Latif, the ll* functions used by LSL and C# will be as they are currently

[04:58] Babbage Linden: but there will parts of the API designed to be used by C#

[04:59] Babbage Linden: one of the things we're considering is adding Idiomatic C# APIs over some of the complex ll* methods

[04:59] Ceawlin Steamweaver: <3

[04:59] Babbage Linden: ParticleSystemBuilders for example

[04:59] Babbage Linden: so, its early days yet

[04:59] Babbage Linden: but we hope to have an initial API defined by the end of the month

[05:00] Babbage Linden: and then Rand will be working on getting that on to a wiki where we can discuss the API in the open

[05:00] Babbage Linden: in the mean time watch the video for a sneak peak in to the current API

[05:00] Babbage Linden: and let me know if anything looks wrong and should be changed next weekk

[05:00] Ceawlin Steamweaver: Hey Babbage, sorry to interrupt, but can you give a yes or no on the "Lifting http body length limits when large scripts go in" question before you go? :3

[05:01] Babbage Linden: ceawlin, I think that makes a lot of sense

[05:01] Ceawlin Steamweaver: Ok, cool. Thankies. :P

[05:01] Babbage Linden: makes sense to allow XML and JSON representations to be accepted by HTTP-In at the same time too

[05:01] Babbage Linden: which is something I've wanted to do for ages

[05:01] Ceawlin Steamweaver: Yus.

[05:02] Babbage Linden: ok, I think we've got through everything

[05:02] Ceawlin Steamweaver: I have to break my stuff up into multiple chunks server-side to get it into LSL and then reassemble it, lol. X_x

[05:02] Actingill Igaly: me too

[05:02] Latif Khalifa: parsing it in LSL is pain though ;)

[05:02] Babbage Linden: and our times up

[05:02] Ceawlin Steamweaver: Thanks Babbage!

[05:02] Babbage Linden: latif, System.Xml is in the BCL :-D

[05:02] Actingill Igaly: cheers

[05:02] Latif Khalifa: \o/

[05:02] Babbage Linden: ok, see you all next time

[05:02] Babbage Linden: and thanks for coming

[05:02] Ann Otoole: thanks for clearing stuff up Babbage

[05:02] Melchizedek Blauvelt: Thanks for having us Babbage

[05:02] ANSI Soderstrom: ^^^

[05:02] Imaze Rhiano: thanks babbie

[05:02] Latif Khalifa: see you Babbage, take care

[05:03] Xugu Madison: Thanks Babbage!

[05:04] Ceawlin Steamweaver: That's just crazy. XD