User:Zero Linden/Office Hours/2007 Jul 26

Transcript of Zero Linden's office hours:

[7:36]	Zero Linden:	oh...
[7:36]	Zero Linden:	my eyes
[7:36]	Morgaine Dinova:	'Morning Zero
[7:36]	Rex Cronon:	hi zero
[7:36]	Zero Linden:	who is glowing?
[7:36]	JayR Cela:	morning Zero
[7:36]	Adam Xinpeng:	Heya Zero
[7:36]	Tree Kyomoon:	rex still needs to hatch
[7:36]	Zha Ewry:	Hmm. Glowing?
[7:36]	Morgaine Dinova:	Adam is glowing
[7:36]	Zha Ewry:	Hmm.
[7:37]	Wyn Galbraith:	People have to understand that SL is LL's business. Bottom line they have to comply with the gov.
[7:37]	Wyn Galbraith:	Rex has a shield on.
[7:37]	Rex Cronon:	oh, this is not an egg. is a f forcefield:)
[7:37]	Morgaine Dinova:	Or is the source of light invisible ... maybe the forcefield on Rex
[7:37]	Zha Ewry takes off her face light
[7:37]	Wyn Galbraith:	Should be a law against forcing fields.
[7:37]	Zero Linden:	well, it doesn't do much for people's complexions
[7:37]	Rex Cronon:	mine is a nice one:)
[7:38]	Wyn Galbraith almost got a mole for breakfest.
[7:38]	Tree Kyomoon:	unless we dont have a complexion
[7:38]	Rex Cronon:	if u don't bug it, it leaves u alone:)
[7:38]	Morgaine Dinova:	The only complexion it improves is Tree's skeleton's :-)
[7:38]	Zero Linden:	Tree - have you sprouted cillia, or have they always been there?
[7:38]	Wyn Galbraith wants to hang ornaments from Tree's ribs.
[7:38]	Tree Kyomoon:	its my super silicone hair do
[7:38]	Morgaine Dinova:	On a technical front .... how do we determine which object is a source of light, as in this case?
[7:39]	Tree Kyomoon:	withstands super high temperature without going all flat
[7:39]	Wyn Galbraith:	It's superglued to his skull.
[7:39]	Zero Linden:	huh - there is no beacon for that
[7:39]	Tree Kyomoon is pretty sure Adam is the GLowing culprit
[7:39]	Zero Linden:	quick - someone to the bat-source
[7:39]	Adam Xinpeng:	Nah
[7:39]	Adam Xinpeng:	I've got no attachment sequipped
[7:39]	Adam Xinpeng:	someone else
[7:39]	Zha Ewry:	Yeah, no beacon for a number of things
[7:40]	Rex Cronon:	there should be a beacon for noobs:)
[7:40]	otakup0pe Neumann:	i'm loaded to the gills with scripts.
[7:40]	Adam Xinpeng:	Heya P0pe
[7:40]	Morgaine Dinova:	Beacon for lights ... sounds like a job for the client hackers
[7:40]	Wyn Galbraith thought of wearing the templates to see where things landed.
[7:40]	Adam Xinpeng:	Hey Alpha
[7:40]	otakup0pe Neumann:	but i bet my 70+% packet loss isn't helping all these server much.
[7:40]	Adam Xinpeng:	Silly non-multithreaded sim. =p
[7:40]	Zha Ewry:	Looks to be a green source, toward Wyn and Adam...
[7:40]	Wright Juran:	:)
[7:40]	Wyn Galbraith has zero packet loss.
[7:41]	Wyn Galbraith:	Oh it's me!
[7:41]	Morgaine Dinova:	The sim is fine, time dilation 1.00
[7:41]	Wyn Galbraith:	My leg things have light enabled and it's green light.
[7:41]	Wyn Galbraith can't see it.
[7:41]	Rex Cronon:	the green light bug culprit has been found:)
[7:41]	otakup0pe Neumann:	That would explain the green light.
[7:41]	Zero Linden:	hey - that's a good topic - why we avoid threads at LL
[7:41]	otakup0pe Neumann:	:o
[7:41]	Zha Ewry nods
[7:42]	Morgaine Dinova:	Nice topic, go for it :-)
[7:42]	Adam Xinpeng:	Heh, we've gone the other direction with OpenSim - everything is threaded like crazy
[7:42]	Zha Ewry:	Because you hate spending weeks chasing obsucre low level locks?
[7:42]	Wyn Galbraith forgot, "Just made them last night. Was playing."
[7:42]	Rex Cronon:	because c++ threads have problems?
[7:42]	Zero Linden:	well then - let's get started - Welcome all to my office hours
[7:42]	Zero Linden:	As always, the transcript will be in the wiki - so speak freely, speak openly
[7:42]	Wyn Galbraith should put a on/of switch in them.
[7:42]	Morgaine Dinova:	I've just commissioned an 8-core server, bring those threads on :-)
[7:42]	Rex Cronon:	on click wyn
[7:43]	otakup0pe Neumann:	This does not give me very good mono-vibes.
[7:43]	Zha Ewry built a huge economics simulator based on c++ and threads and found it really, really hard to blend threads, c++ and sane locking
[7:43]	Adam Xinpeng:	Silly Mono users. =p
[7:43]	Zero Linden:	there - that should soften things a big
[7:43]	Zero Linden:	er, a bit
[7:43]	Saijanai Kuhn sotto voice: are we discussing monno?
[7:43]	Zha Ewry:	Threads I think
[7:44]	Zha Ewry:	some of the overall low level sim design choices
[7:44]	Zha Ewry:	And.. possibly, which would be interesting
[7:44]	Rex Cronon:	mono, the thing that is almost here but never makes it?
[7:44]	Morgaine Dinova:	Threading, hopefully. With special attention to the new wave of multi-core CPUs
[7:44]	otakup0pe Neumann:	Threads today. Zero you totally need to get babage in here with you some day !
[7:44]	Zha Ewry:	Threads is part of that theme..
[7:44]	Zero Linden:	no, we are not discussion mono - I'm still trying to get Babbage to come do that here - but he's busy opening the Brighton, U.K., office -
[7:44]	Zero Linden:	I think the big party was last night
[7:44]	Zero Linden:	or perhaps it is tonight - can't work out the time changes...
[7:44]	Tree Kyomoon:	nice!
[7:45]	Wyn Galbraith:	Don't see Pathfinder anymore since opening the Boston office. :(
[7:45]	Zha Ewry:	So.. No threads because?
[7:45]	Zero Linden:	So give him a week or three....
[7:45]	Morgaine Dinova:	Hi Khamon
[7:45]	Zero Linden:	BUT, I will report that there is some work now being done on mono, and as I said before, mono is being used as the driving use case for the second phase of het-grid
[7:45]	Khamon Fate:	hallo
[7:45]	Zero Linden:	BUT, back to threads
[7:46]	Zha Ewry:	Morning Khamon
[7:46]	otakup0pe Neumann:	heh. or three.
[7:46]	Zero Linden:	So, I don't think it is so much a language issue:
[7:46]	Rex Cronon:	hi
[7:46]	Khamon Fate:	Hi Zha
[7:46]	Zero Linden:	unless one is using an "exotic" language, say erlang, dealing with threads amounts to pretty much the same thing in any language
[7:46]	Khamon Fate:	Tree has nice threads
[7:46]	Zero Linden:	So, in my mind, here is the trade off with threads
[7:46]	Adam Xinpeng:	C# & Java do threads reasonably well (C# moreso, but Java can work)
[7:47]	Morgaine Dinova:	I vote for reimplementing everything in Erlang ... not entirely jokingly, hehe.
[7:47]	Rex Cronon:	java thread work quite well, but is not that simple even in java to handle threads
[7:47]	otakup0pe Neumann:	heh erlang.
[7:48]	Zero Linden:	Given X amount of work to be done it comes down to a trade off between the ease of writing that code, and the cost of managing it's execution
[7:48]	otakup0pe Neumann:	i know at least one linden is doing something with erlang.
[7:48]	Zero Linden:	if you have long computations, with little interaction, then threads are nice - there is little extra code to do locking, and you can write in a "straight-line" fashion
[7:49]	Zha Ewry nods
[7:49]	Zero Linden:	but if most of your work is tiny bursts of work, and there is a great deal of interaction between parts, then
[7:49]	Morgaine Dinova:	So what is LL's current position on making use of multiple cores? It's no longer viable to ven consider ignoring 1/2, 3/4, 7/8, 15/16 etc of your hardware capability.
[7:49]	Zero Linden:	the cost of sync.ing (locking) and task switching overwhelms the advantages
[7:49]	otakup0pe Neumann:	Morgaine I thought it did that ?
[7:49]	otakup0pe Neumann:	When I enable multiple threads CPU shoots past 100%
[7:49]	otakup0pe Neumann:	And my laptop because too hot to be a laptop.
[7:49]	Zero Linden:	so, if you can easily divide your work up into, say under 1ms. chunks, then cooperative switching between your own work queues is going to be much more efficient
[7:49]	Wright Juran:	on C++ related issue with threads, Intel released their threading building blocks under gpl the other day, for c++ programs it might be worth looking at
[7:50]	Zha Ewry assumes whe're mostly talking sim here
[7:50]	Zero Linden:	AND easier to code, because of no locking conerns
[7:50]	Saijanai Kuhn muses that SL seems to use a balanced load on his 2 CPUs on the mac, and that 2 clients double that use quite rationally
[7:50]	Wyn Galbraith has a dual core laptop too.
[7:50]	Adam Xinpeng:	Threads have the advantage of having cleaner code however.
[7:50]	Wyn Galbraith:	I thought the OS took care of delegating to the CPUs.
[7:50]	Adam Xinpeng:	plus, dual core machines, etc. The key is keeping the total count low and using a threadpool manager to avoid the big creation/handling penalties.
[7:51]	Zero Linden:	Adam, if and only if the cost of dividing up your work into chunks is larger than the cost of putting in the locks/sync-points
[7:51]	Zha Ewry:	Adam, that's only true.. if the coder doesn't have to keep managing a large number of object locks.
[7:51]	Zero Linden:	it is a trade off of program complexity, and I think it depends on the kind of code
[7:51]	Adam Xinpeng:	Zero: that's only if you rely on having locks & sync points.
[7:51]	Tree Kyomoon is learning a ton here...sweet
[7:51]	Wyn Galbraith:	That's why I come to these meetings Tree. :)
[7:51]	Zero Linden:	you must if you have a large data set that is touched by many different operations
[7:52]	Adam Xinpeng:	Wright (MW) here can probably explain a little more how we work on it with OpenSim, but we basically have a core "world" thread, and then per-user threads which handle network & packeting.
[7:52]	Zha Ewry:	If you don't... then you need to have some other mechanism to keep the data safe
[7:52]	Zero Linden:	the data model of sim seems to fall into thtat category for me
[7:52]	Adam Xinpeng:	The idea is that the "world" thread shuffles stuff into the user threads (no interaction outside of the main world thread), and creates a processing queue.
[7:52]	Zero Linden:	Ah- so let answer two stacked issues here: one cores, and the other networking
[7:53]	Zero Linden:	As for cores, in the viewer we have one very well partitioned computation: texture decoding - it falls into the longer computation, little data interaction category
[7:53]	Zero Linden:	so, THAT we put into a thread
[7:53]	Zero Linden:	and hence, get two cores in use
[7:53]	Zha Ewry likes to approach server core use by putting specific tasks in a thread (Which is in effect, what Adam is doing)
[7:54]	Zha Ewry:	Load up the cores with nice isolated tasks
[7:54]	Zero Linden:	in the future, threaded OpenGL drivers will get us another (Mac 10.5 will have 'em!)
[7:54]	Wright Juran:	the world thread which Adam talks about isn't actually a single thread, basically the way c# /.net handles the network packets is to use its thread pool, so a thread from that accepts the incoming packets and forwards them on to a clients thread that does the actual work
[7:54]	Morgaine Dinova:	Zero: as the number of cores rises (and it'll be in the thousands in just a few years), programmers are not going to be able to do any meaningful static allocation. Dynamic methods are going to be the only way forward ... much like the latest Intel programming system, Thread Building Blocks.
[7:54]	Wright Juran:	in some ways opensim is too multi threaded but I would rather that be the case than to be single threaded
[7:55]	Zero Linden:	Well, for networking, what we did, is build our own dispatched handler.....
[7:55]	otakup0pe Neumann:	wait are we talking client or server or either ?
[7:55]	Adam Xinpeng:	I think we are talking principle here.
[7:55]	otakup0pe Neumann:	i guess i should say 'viewer' and 'other'
[7:55]	Morgaine Dinova:	Both are relevant
[7:55]	Adam Xinpeng:	With examples.
[7:55]	otakup0pe Neumann:	k
[7:55]	Zero Linden:	in essence the same, only all within a single OS thread - we manage the packets distribute them to handlers, or "pumps-n-pipes" which manages
[7:55]	Zero Linden:	incrementing the work along as the data arrives
[7:55]	Adam Xinpeng:	The downside to that is, if you have one shoddy client, you have to waste time dealing with it.
[7:56]	Adam Xinpeng:	Under a multithreaded server - it's only that thread that slows down.
[7:56]	Zero Linden:	oN - at LL both sides use the same methodology
[7:57]	Morgaine Dinova:	Uh oh, "SL bans gambling" article just hit Slashdot ... expect the Slashdot effect.
[7:57]	Zero Linden:	AX - well, you mean on client with heavy network load?
[7:57]	Adam Xinpeng:	More along the lines of client on dialup.
[7:57]	Wright Juran:	one thing that we noticed about the linux client last week is it seems to be single threaded to such a extent that the whole program frezzes if the user even does a task such as goes to upload a texture and the file picker opens, the rest of the client stops, it won't respond to packets or anything
[7:57]	Adam Xinpeng:	Something where you have to waste time with retransmission, etc.
[7:57]	Zero Linden:	we can still partition how much time we spend handling messages from a given agent --
[7:57]	Zero Linden:	we don't need to slow the thread, we just throttle the processing of those messages
[7:57]	Adam Xinpeng:	You can, but that gets messier than just putting it in a thread.
[7:57]	Adam Xinpeng:	You end up doing the same sorts of things that the thread is doing anyway.
[7:58]	Zero Linden:	AX - I'm not sure really for us it is just list managment - but at this point we are disucssion programming style, not a performance issue I think
[7:58]	Zero Linden:	Now, in our python servers we do something else
[7:58]	Zero Linden:	and this is instructive, I think
[7:59]	Saijanai Kuhn:	One thing I have noticed about running 2 Mac clients in two different Mac accounts is that they both get progressively slower until both become unuseable.
[7:59]	Zha Ewry:	Well, that's an interesting trafde off zero. Either manage the locking, or the way you pass around units of owrk in the single thread
[7:59]	Morgaine Dinova:	Well most old programming styles can be kissed bye bye, now that we're in a multi-core world.
[7:59]	Zha Ewry:	Both are going to impose similar loads on the programmer
[7:59]	Zero Linden:	we use a combination of greenlet (simulate threads by copying around portions of the machine stack), and eventlet (cooperative co-routines)
[7:59]	Zero Linden:	(er, cooperative coroutines at network operations)
[8:00]	Zero Linden:	to make our server (called Backbone) be single threaded, but non blocking
[8:00]	Zero Linden:	in this way, you right straight line code, like a thread, but
[8:00]	Adam Xinpeng:	I'm assuming you must launch multiple instances per machine then to take advantage of multiple processors?
[8:00]	Morgaine Dinova:	Using standard libs for that greenlet/eventlet thing?
[8:00]	Zero Linden:	you know that there will be no switching untill you do a network or IO operation - hence you can deal with shared data fairly easily
[8:01]	Zero Linden:	AX - on our server side, indeed we do - on sim per core, one backbone per core
[8:01]	Morgaine Dinova:	AX: no, you can't run up multiple copies all operating on the same sim
[8:01]	Zero Linden:	actually, at present, we do one backbone per host - but we can run other stuff on those hosts as well --- we'll be doing multiple backbones per host (one per core) soon
[8:02]	Adam Xinpeng:	Morgaine: I'm talking processes.
[8:02]	Tree Kyomoon:	sheepishly raisess hand...wouldnt it make more sense to divide tasks to cores based on task rather than per sim?
[8:02]	Zha Ewry nods "Thast starts to come apart when you go from things like intel, with 2-16-64 cores, to things like Cell, or similar with orders of magnitudue more cores
[8:02]	Morgaine Dinova:	AX: indeed, but as Zero said, each of those sim processes has to operate on a different sim.
[8:02]	Adam Xinpeng:	Morgaine: Not nessecarily - under LL's simulator, sure.
[8:02]	Zero Linden:	Morgaine - I think both are available open source, though our internal eventlet is somewhat spiffed up - and Donovan Linden wants to make that available as soon as he gets the time to document the enhancemehts
[8:03]	Morgaine Dinova:	AX: ah, you're talking about OpenSim?
[8:03]	Morgaine Dinova:	Zero: kk
[8:03]	Adam Xinpeng:	Nah, this discussion is principle not example. Heh.
[8:03]	Adam Xinpeng:	But, what me/Zero were talking about was, since LL's software is all single threaded, they make up for it by launching multiple processors per machine
[8:03]	Adam Xinpeng:	Otherwise there'd be hardware going to waste.
[8:04]	Zero Linden:	Well, the issue is that the amount of processing a sim actually does is pretty low - even physics simulation and script execution are done in relatively small steps
[8:04]	Adam Xinpeng:	But even then - I imagine a process is heavier than threading
[8:04]	Zha Ewry:	Right, they load up the machines, by having bi g heavy processes to put on the cores
[8:04]	Zero Linden:	and, alas that work is not very contained with respect to the global state of the sim
[8:04]	Zero Linden:	a script tends to touch quite a lot of other state in the sim as it executes
[8:04]	Rex Cronon:	if physics processing is so low, why it is so low?
[8:04]	Zero Linden:	the physic sim. requires all object information
[8:04]	Tree Kyomoon:	--nice point rex
[8:04]	Rex Cronon:	i mean slow*
[8:04]	Morgaine Dinova:	It's not a matter of cores going to waste (they can always run other sims instead), but a matter of no more power being available per sim.
[8:05]	Zero Linden:	so - locking that state or sharing between tasks delegated to cores could be complex and cost a lot in locking
[8:05]	Zha Ewry ah, so phyiscs is a memory toucher/locker brute
[8:05]	Zha Ewry nods "that, alas, makes sense"
[8:05]	Zero Linden:	Actually, I think scripts would be the real killer
[8:05]	Adam Xinpeng:	Morgaine: I think we are talking about completly different things.
[8:05]	Zero Linden:	you can do SO much with a script - and so it really affects tons of state
[8:05]	Adam Xinpeng:	Heh, scripts dont have to be as bad as they are.
[8:05]	Morgaine Dinova:	Before, we could rely on speed improvement in CPUs to accelerate our apps. Now we can't anymore, because instead of greater speed we'll be getting more cores.
[8:06]	Morgaine Dinova:	So Moore's Law breaks down if your app can only use one core.
[8:06]	Jarod Godel:	I thought scripts had their own processes.
[8:06]	Zero Linden:	and let's not forget that, with languages like C++/Java/etc... where the programmer needs to be mindful of locking explicitly, it is easy for there to be difficult bugs
[8:06]	Adam Xinpeng:	Jarod: No, it's done in a single thread, each script is given a set number of operators to process every frame.
[8:06]	Morgaine Dinova:	Zero: that's a good reason for abandoning old languages. Programmers can adapt.
[8:06]	Adam Xinpeng:	LSL's actually horeendously inefficient at the low end.
[8:06]	Zero Linden:	Morgaine - for the short term future, we don't see us having systems with 64 cores.... but perhaps I'm not on the curve....
[8:07]	Tree Kyomoon would like to see LL become object oriented
[8:07]	Tree Kyomoon:	LSL *
[8:07]	Zero Linden:	AX - Oy Vey, do I know......
[8:07]	Adam Xinpeng:	Always the Niagara's, although their floating point performance sucks at the moment (although the v2.0 is supposed to fix tht)
[8:07]	Rex Cronon:	there is a big difference between java and c++. the java programs don't crash as hard and as often as one that were made using c++
[8:07]	Zero Linden is an old VM designer in prior jobs
[8:07]	Morgaine Dinova:	Zero: I've just bought myself an 8-core box for next to nothing. I expect to have 64 cores in under 3 years.
[8:08]	Adam Xinpeng:	Zero: Heh, we've spent the last two weeks looking at LSL and implementing it - we've gone the LSL->CIL route ourselves now, but I was writing an interpreter for a while and just looking at the bytecode design it's understandable why LSL is so slow. Heh.
[8:09]	Jarod Godel:	Adam, hunh. I could have sworn they each got their own virtual machine that ran and then talked to/update the sim. I'm aghast.
[8:09]	Tree Kyomoon:	try 1024 cores and 4 TB of RAM: http://money.cnn.com/news/newsfeeds/articles/prnewswire/AQM07923072007-1.htm
[8:09]	Zero Linden:	Morgaine - I'm not sure how we'll handle that - for us, threading doesn't look like a promising option.... we find little in the sim that could run as a paralell computation
[8:09]	Adam Xinpeng:	LL do have a virtual machine, but it's running in the same function as the rest of the simulator.
[8:09]	Adam Xinpeng:	Basically it's like:
[8:09]	Jarod Godel:	Zero, by VM, you mean Smalltalk, right?
[8:09]	Zero Linden:	though I suppose we could put some of our networking in another thread - somethign where the data flow is easily managed
[8:09]	Adam Xinpeng:	doPhysics(); doScripting(); doNetwork(); etc -- where doScripting() processes say 500 instructions then moves along.
[8:09]	Wyn Galbraith swoons over corepower.
[8:09]	Morgaine Dinova:	Zero: check out Intel's new Threading Building Blocks, which they've just open-sourced.
[8:10]	Zha Ewry thinks hard for a long minute
[8:10]	Morgaine Dinova:	Adam: what's the link for that work?
[8:10]	Jarod Godel:	Adam, oh. Then I really didn't understand. I thought scripts got forked off at a layer closer to the OS...separate processes that communicated asychronously.
[8:10]	Zero Linden:	Jarod - I've done a Smalltalk VM (w/Dan Ingalls), two Java VMs, and four custom langauge VMs
[8:10]	Jarod Godel:	Zero, whoa!
[8:10]	Jarod Godel:	cool
[8:11]	Zha Ewry:	Zero, how much of the sim work will get broken out when you go to the agent domain?
[8:11]	Adam Xinpeng:	Jarod: No - SL's sim runs in one giant single loop, not threaded at all from what I understand.
[8:11]	Jarod Godel:	Adam, well, yikes.
[8:11]	Zha Ewry:	As sims manage just physics, doesn't it get worse?
[8:11]	Adam Xinpeng:	Jarod: it's pretty standard for game servers - although they are moving towards threading these days.
[8:11]	Adam Xinpeng:	But, the reasons why Zero covered in the first half of this chat. Heh.
[8:11]	Zero Linden trying to respond to the firehose....
[8:12]	Morgaine Dinova:	I made a compiler and VM system for a 16-CPU multiprocessor for my research, way back. Lot of fun, especially when creating systems for profiling and observing timeslices of activity.
[8:12]	Jarod Godel:	Yeah, I'll read the log.
[8:12]	Zero Linden:	AX - yes, that is basically how the sim works - the problem is that few of those tasks break off easily into a separate thread due to date interactivitity (okay, I know, I'm being a broken record) but yes, some could be, like the networking
[8:13]	Tree Kyomoon thinks this conversation needs to be threaded :)
[8:13]	Morgaine Dinova:	You need VERY good tools to be able to understand what's going on in concurrent systems. The old vi/run-printf/vi cycle is useless.
[8:13]	Adam Xinpeng:	Heh
[8:13]	Saijanai Kuhn still wants fine grained text-coloring
[8:13]	Zero Linden:	When we go domains - then the running sims in the region domain are mostly doing three things a) physics , b) script execution, c) message management
[8:14]	Tree Kyomoon imagines live chat in message board format
[8:14]	Zero Linden:	which isn't much different then now, it's just that a huge list of agent mangement functions gets moved to the agent domain. But really those functions only add code complexity, not processing time
[8:14]	Jarod Godel:	Tree, it's called PM.
[8:14]	Zero Linden:	after all, how many of you are fiddiling with, say classifieds, at this moment
[8:15]	Jarod Godel:	You must have some kick ass compilers there at Linden Lab if code complexity doesn't increase processing time. :)
[8:15]	Rex Cronon:	so, it is wrong to chat and search for soemthing, while looking at the map? and doing a few IM?
[8:15]	Morgaine Dinova:	This subject is so big, maybe it needs splitting into server and client days.
[8:15]	Adam Xinpeng:	Jarod: Compilers are stunningly good these days.
[8:16]	Adam Xinpeng:	Except GCC.
[8:16]	Jarod Godel hugs his Perl interpreter.
[8:16]	Adam Xinpeng:	=p
[8:16]	Zero Linden:	Jarod - no same comilers you have - it's just that really, all those functions don't get called all that often
[8:16]	Zero Linden:	Now- there are other things we can do that just break off processing - and much easier than doing threads
[8:16]	Zero Linden:	A really good example is textures via HTTP
[8:16]	Zha Ewry:	Well, what I'm musing about, Zero, is whethere you end up having less to break off for cores in the sims when you do that.
[8:17]	Adam Xinpeng:	Zero: we are waiting with baited breath for that - UDP texture downloads are a mess!
[8:17]	Zero Linden:	So right now, the sim is doing a fair bit of work being a conduit of texture data
[8:17]	Morgaine Dinova:	Clients merely inject job requests into the server complex.
[8:17]	Adam Xinpeng:	Zero: we've been thinking of sending CAPS URLs for texture downloads to not be on the sim itself (ie have an 'asset cache proxy server' which is where the caps url for that goes)
[8:17]	Wright Juran:	yeah, any idea when texture downloads via http are going to be functional
[8:17]	Zero Linden:	your viewer wants texture X, the sim has to get it from the asset server (via a squid cache) and the feed it the viewer in UDP message chunks,
[8:17]	Zero Linden:	with all sorts of careful throttling code
[8:18]	Rex Cronon:	allow client to do that
[8:18]	Zero Linden:	AX - that code is already running here - exactly - caps for each texture - and the cap maps directly to the asset
[8:18]	Adam Xinpeng:	The client works & accepts it?
[8:18]	JayR Cela:	I thought the 1.8 release of the client was to be an end to UDP
[8:18]	otakup0pe Neumann:	here you bean internal grid ?
[8:19]	Adam Xinpeng:	We've given out the CAPS url's but they dont seem to get used.
[8:19]	Zero Linden:	hence, when the viewer gets the texture - it isn't even the sim that is involved, but Backbone and squid and apache
[8:19]	Morgaine Dinova:	Good decoupling
[8:19]	Zero Linden:	now - THOSE processes (at leasts squid and apache right now) are well sutied to threads and will soak up all those extra cores nicely
[8:19]	Zha Ewry:	/
[8:20]	Zha Ewry:	The 1.8 code just opens the door to migrate stuff off UDP, not makes it happen
[8:20]	Zero Linden:	backbone, when used in this caps/proxy mode too can be truly threaded (once it understands the reuqest) and so the proxy can be spun off into a thread
[8:20]	Adam Xinpeng:	Not quite.
[8:20]	Adam Xinpeng:	The 1.18 code just solidified the UDP template so that each packet is now numbered rather than indexed based on a crazy hashing scheme.
[8:20]	Adam Xinpeng:	The UDP templates are still there, and wont be going anywhere soon I dont think.
[8:20]	Jarod Godel:	backcone is python, iirc, so, yeah, it can be threaded...that's true.
[8:21]	Baba Yamamoto:	crap
[8:21]	Zero Linden:	AX - we have it running internally- I don't know Steve Linden's planned release of that functionality - but we all want it soon
[8:21]	Adam Xinpeng:	Heh cool - we will keep an eye out for it. :)
[8:21]	Baba Yamamoto:	am I late?
[8:21]	Zero Linden:	JayR - 1.18, message liberation, was the begining of migrating away from UDP - not an all out end of it
[8:21]	Rex Cronon:	only 9 minutes left
[8:21]	Adam Xinpeng:	UDP still makes sense for a lot of things.
[8:22]	JayR Cela:	ZERO thank you for clearing that up
[8:22]	Adam Xinpeng:	It's just the packets flagged "reliable" that it doesnt make sense for.
[8:22]	Baba Yamamoto:	you can't really drop udp all together, but forget trying to make it reliable
[8:22]	Zero Linden:	AX - 1.18 also introduced TCP/HTTP basesd ways of sending messages
[8:22]	Saijanai Kuhn:	just its existence reduced teh group IM bug by at least 50-90%
[8:22]	Adam Xinpeng:	Zero: CAPS was in before 1.18 though wasnt it?
[8:22]	Zha Ewry:	Even where UDP doesn't make sense, the 1.18 update only makes it possible to pull stuff up, doesn't move it.
[8:22]	otakup0pe Neumann:	CAPS has been in for ages ?
[8:22]	Adam Xinpeng:	Caps has been doing teleports since 1.16 I think
[8:22]	otakup0pe Neumann:	March or so ?
[8:23]	Baba Yamamoto:	yeah 1.16
[8:23]	Zero Linden:	of course, "reliable" packets make up the bulk of what is one the wire
[8:23]	otakup0pe Neumann:	heh.
[8:23]	Zero Linden:	yes, CAPS and messages over TCP/HTTP are different
[8:23]	Zha Ewry:	Those are all going over to TCP in time, zero?
[8:23]	Zero Linden:	let me explain
[8:23]	Zha Ewry:	The reliable ones?
[8:24]	Zero Linden:	CAPS lets you do HTTP/REST style semantics where the authorization to perform a function and the execution are separated
[8:24]	Zero Linden:	but for this discussion, the important point is that it is REST: you make a request and get a response
[8:24]	Zero Linden:	the message system, UDP or otherwise, has different semantics
[8:25]	Zero Linden:	you can only send a message - there is no reply at the message system level - only the application layer
[8:25]	Zero Linden:	though there are some options: reliablity
[8:25]	Zero Linden:	which only gives you a bit of info - it got there or no
[8:25]	Zero Linden:	so they are two different communications styles
[8:26]	Zero Linden:	The vast bulk of the code is written in the messaging style
[8:26]	Zha Ewry chacks her notes.. CPAS iscapabilitys?
[8:26]	Adam Xinpeng:	Hrrm, allowing myself to indulge in a question - why not just write a generic "message handler" which can read UDP or TCP packets via the same mechanism and just throw it down the different mechanisms depending on if reliable was set?
[8:26]	Zero Linden:	so we needed a way to get that stuff to have the resilliance that HTTP/XML style of communciations have
[8:26]	Morgaine Dinova:	Zero: until you redesign the single-sim handling to soak up new cores dynamically, you won't gain that scalability for events that we're after.
[8:26]	Zero Linden:	hence Msg. Lib.
[8:26]	otakup0pe Neumann:	Ok I actually have to run. Nice seeing everyone.
[8:27]	Zero Linden:	Gee, AX, that is exactly what Message liberation is
[8:27]	Adam Xinpeng:	Well, that' with the HTTP overhead.
[8:27]	Zero Linden:	CAPS - is capabilities, yes
[8:27]	Adam Xinpeng:	Unless your following msg_template under it?
[8:27]	Adam Xinpeng:	I'm more suggesting one msg_template, but it gets composed into a binary UDP or TCP packet based on the reliabilit/importance
[8:28]	Zero Linden:	AX - messages can be set to use TCP based transports, and then they are "freed" from the template - and are sent via LLSD based transports
[8:28]	Adam Xinpeng:	Obviously it's too late for any kind of suggestion - but I'm curious as to the use of HTTP there.
[8:28]	Jarod Godel:	I don't think anything is so unreliable any more that it can be handled by udp
[8:28]	Baba Yamamoto:	zero, how woud you design it if you were were starting everything today? Are you fixing past mistakes or designing for the future?
[8:28]	Jarod Godel:	not in second life anyway.
[8:28]	Adam Xinpeng:	Jarod: there's plenty.
[8:28]	Zha Ewry looks at Jarod
[8:28]	Jarod Godel:	aside from streaming media, name one.
[8:28]	Baba Yamamoto:	Will there be more big changes 2-3 years down the road that totally changes the protocol?
[8:28]	Adam Xinpeng:	Rapid object updates.
[8:28]	Zha Ewry:	Plenty of keystroke level updates which iif ou lose one.. you don't care
[8:28]	Jarod Godel:	inventory needs to be checked. chat needs to be checked.
[8:28]	Zero Linden:	There are three transports used for LLSD encoded messages: a) direct HTTP between sims (there is a URL, like /message that you POST to)
[8:28]	Adam Xinpeng:	IE, object is updating between X and Y - only the last one needs to be reliable.
[8:29]	Zha Ewry:	So, walk...
[8:29]	Zha Ewry:	Put avatar in position X.
[8:29]	Jarod Godel:	Updates need to be checked, or the world lags and breaks.
[8:29]	Zha Ewry:	Those, if we lose one... the next one will carry on the work
[8:29]	Adam Xinpeng:	Server go melty.
[8:29]	Adam Xinpeng:	plus then you have out of ordering issues
[8:29]	Zero Linden:	b) a capability that the viewer has for POSTing messages to the sim (winds up at /message/untrusted I think)
[8:29]	Jarod Godel:	no... rapid updates eed to be handled IN the client. :)
[8:29]	Adam Xinpeng:	What if it's going the other way?
[8:29]	Adam Xinpeng:	eg resizing a prim
[8:29]	Morgaine Dinova:	Jarod: wrong. You always have to design things to continue working under conditions of congestion. So UDP isn't an appropriate vehicle unless you add a reliability layer, or unless the application can survive with lost packets.
[8:30]	Adam Xinpeng:	Only the last position matters.
[8:30]	Zero Linden:	c) the Event Queue for sending messages from the sim to the viewer, where the viewer periodically fetches the pending events from the sim via a different capability
[8:30]	Adam Xinpeng:	Zero: speaking of event queue - who do we hit for that one?
[8:30]	Adam Xinpeng:	EventQueue strikes me as incredibly wasteful of sockets.
[8:30]	Jarod Godel:	Morgaine, UDP with a reliability layer is TCP
[8:30]	Zha Ewry:	There are a small number of places, where you just don't care if you drop a packet, That's when you can use UDP
[8:30]	Zha Ewry:	And.. If you do care, yes, you http, on top of tcp, and be done with it
[8:31]	Adam Xinpeng:	TCP is more than reliability.
[8:31]	Baba Yamamoto:	http is useful for some things but it's not exactly the most efficient way of sending a packet over tcp
[8:31]	Zero Linden cathcing up on back log
[8:31]	Zha Ewry:	In fact, if you don't care if you drop a packet, UDP self throttles more. (look at the transctips a few sessions back this came up, and Zero made some good
[8:31]	Zha Ewry:	points about it)
[8:31]	Khamon Fate:	thanks for hosting zero
[8:31]	Zero Linden:	So - I tend to agree with Baba - there is little left in SL that can be unrealible
[8:31]	Morgaine Dinova:	Jarod: not at all. TCP is not a layer above UDP. However, there are several systems that do implement a reliability layer on top of UDP, for example NFSv2.
[8:32]	Zero Linden:	even object updates are pain as you do have to deal with ordering (don't you all hate when the object you are editing snaps back...)
[8:32]	Morgaine Dinova:	Zero: yep
[8:32]	Zero Linden:	streaming media is one - but we just let other libs. and protocols handle that
[8:32]	Tree Kyomoon:	yes
[8:32]	Jarod Godel:	I can't think of any place (as I said, aside from streaming media) where UDP would trump TCP. That said, I can think of a few places where UDP could be eliminated to speed things up: rapid changes (my client can render walking and resizing after the fact just as well as during).
[8:33]	Zero Linden:	The event queue isn't that wasteful of sockets - or shouldn't be soon
[8:33]	Adam Xinpeng:	Well, it involves keeping a persistent listener out for each client
[8:33]	Adam Xinpeng:	If you only have 40 listeners in a sim that's fine
[8:34]	Adam Xinpeng:	but say you want 1000 users in a sim
[8:34]	Zero Linden:	basically, we do a trick called "the long poll" the viewr makes a HTTP POST and the sim holds the socket open until it has some events to send, or it all times out
[8:34]	Adam Xinpeng:	Then it adds up pretty quickly.
[8:34]	Zero Linden:	when events come, the viewer immediatly makes another -
[8:34]	Jarod Godel:	Morgain, I realize TCP and UDP are pretty much on the same level in the network stack, I was mostly making a facetious argument in favor of tcp.
[8:34]	Zero Linden:	if KEEP ALIVE is enabled, then this all happens over one socket
[8:34]	Zero Linden:	well, two things - 1000 sockets shouldn't be a problem on a modern OS at all
[8:34]	Morgaine Dinova:	Jarod: hehe. Well, they're both just tools, horses for courses :-)
[8:34]	Adam Xinpeng:	Heh true.
[8:35]	Zero Linden:	and 2) we are going to ahve to get MUCH better graphics cards before 1000 AVs can be in a sim!
[8:35]	Jarod Godel:	Giddyap NetBios... AWAY!
[8:35]	Adam Xinpeng:	It just seems rather inefficient to just opening a TCP session and using normal two way comms.
[8:35]	Adam Xinpeng:	Zero: Not if they dont see each other.
[8:35]	Morgaine Dinova:	Zero: not better graphics cards, better rendering systems. A better algorithm trumpt h/w power every time.
[8:35]	Zero Linden:	Adam - if they don't see eachother - than there is no need for them to be in the same sim, eh?
[8:36]	Adam Xinpeng:	Sure there is, say watching a common event.
[8:36]	Adam Xinpeng:	Yes you could mirror the sims - but why waste the hardware if you can avoid it?
[8:36]	Wright Juran:	what about for big events, is there a need to see every other avatar?
[8:36]	Tree Kyomoon still thinks there should be "audience mode" for avatars
[8:36]	Jarod Godel:	Adam, you can watch the same event a sim away.
[8:36]	Zero Linden:	Adam, not clear that SL is the right vechilce for non-interactive viewing
[8:36]	Adam Xinpeng:	Heh, we will have to get Zero playing with OpenSim. heh.
[8:36]	Jarod Godel:	Tree, someone was using upstream.tv for something very similar to that.
[8:36]	Morgaine Dinova:	Zero: machinima
[8:37]	Zero Linden:	If you want 1000 people to view your broadcast, but not be able to see eachother... there is QuickTime streaming..... :-)
[8:37]	Jarod Godel:	Log an avatar in, stream the screen view, and let people watch the video.
[8:37]	Adam Xinpeng:	Yeah but quicktime streaming doesnt have alt zoom
[8:37]	Wright Juran:	or chat
[8:37]	Tree Kyomoon:	or ability to tip the performer by cliking their jar
[8:37]	Adam Xinpeng:	heh
[8:37]	Zha Ewry:	Well, if you want to have more ave in the same sim, you can also contrsin them, so the rendring cost goes way down
[8:38]	Tree Kyomoon:	exactly zha
[8:38]	Zero Linden wonders if he'd like his office hours with 1000 avatars, rendered each a nothing but a name badge, all chatting....
[8:38]	Zha Ewry:	Make them sit, and not change outfits, so you can bake a lot of that and simply leave it static
[8:38]	Morgaine Dinova:	Adam: I'll give OpenSim a try as soon as I put the 8-core box online.
[8:38]	Wright Juran:	we would have had to play find the glowing source
[8:38]	Tree Kyomoon:	well, we wont get sting performing in SL with 40 people in the audience unless the tickets cost 40,000 dollars
[8:38]	Rex Cronon:	everybody can have a blob of a different color:)
[8:38]	Wright Juran:	wouldn't
[8:38]	Zero Linden:	we can all be lifeless plywood cubes...... (no offence, Cube Linden, if you are reading this....)
[8:38]	Adam Xinpeng:	Morgraine: still alpha, but you can do some neat stuff with it if your a coder.
[8:39]	Zha Ewry grins
[8:39]	Adam Xinpeng:	Lots of stuff got broken in the 0.2->0.3 transition so that's being patched up at the moment.
[8:39]	Morgaine Dinova:	Adam: yep, developer here. SVN?
[8:39]	Zero Linden:	well friends, my hour is up and I've got things like coding and protocols to do
[8:39]	Adam Xinpeng:	Grab the info off the developers page @ opensimulator.org
[8:39]	Zero Linden:	thank you all for coming
[8:39]	Wyn Galbraith:	There's worst things in life then being a plywood cube.
[8:39]	Wyn Galbraith:	Thanks for the meeting Zero.
[8:39]	Zha Ewry:	Well, Zero, I think at about 10 serious tech people asking you questions you begin to drown.
[8:39]	Morgaine Dinova:	Thanks Zero.
[8:39]	JayR Cela:	thanks Zero
[8:39]	Zha Ewry:	1000 woud be specaturular
[8:39]	Tree Kyomoon:	thanks zero, you handled it well
[8:39]	Zha Ewry:	And, Thanks Zero
[8:39]	Saijanai Kuhn:	un unemployed plywood cube...
[8:39]	Adam Xinpeng:	Yep, thanks again. :)
[8:39]	Zha Ewry:	Great topic
[8:39]	Tree Kyomoon:	you must have a multithreaded brain
[8:39]	Morgaine Dinova:	Zero prolly needs an hour of coffee after these sessions.
[8:39]	Zero Linden:	Till next week