User:Zero Linden/Office Hours/2007 Aug 16

Transcript of Zero Linden's office hours:

[7:34]	Johanna Hyacinth:	Hello, Scope
[7:34]	Johanna Hyacinth:	Good morning, Zero :)
[7:34]	Rex Cronon:	hi scope
[7:34]	Zero Linden:	morning all
[7:34]	Wyn Galbraith:	Good morning Zero.
[7:34]	Rex Cronon:	hi zero
[7:34]	Panacea Pangaea:	Hi :-)
[7:35]	Saijanai Kuhn:	morning teacher
[7:36]	Johanna Hyacinth:	So, how does this work - do we just all ply you with questions? :)
[7:36]	Zero Linden:	oh, no
[7:36]	Zero Linden:	I'll wait a few more minutes
[7:36]	Zero Linden:	but we generally have a meandering discussion on the technology that runs Sl
[7:36]	Zero Linden:	A word to new folks
[7:36]	Johanna Hyacinth:	Aha, I see. :)
[7:36]	Rex Cronon:	i still haven't been able to read the transcripts for last tuesday(about mono)
[7:36]	Wyn Galbraith:	Meandering... LOL
[7:37]	Zero Linden:	the transcripts are posted to the public wiki -- speak freely, speak openly!
[7:37]	Zero Linden:	Sorry Rex....
[7:38]	Panacea Pangaea:	I'm probably in a group way over my head ...
[7:39]	Panacea Pangaea:	I just had a couple of specific questions about Silo :-\
[7:39]	Saijanai Kuhn:	little man
[7:39]	Saijanai Kuhn:	dancing ascii
[7:39]	Johanna Hyacinth:	While we're waiting for you to begin, who would I talk to about how SLVoice.exe and SLVoiceAgent.exe work? There's an issue in Jira about voice on Linux that mentions running the voice component on another machine, but I'll be darned if I can figure out how to make that work.
[7:39]	Zero Linden:	hmmm..... small turnout today
[7:39]	Saijanai Kuhn:	dancing kirby
[7:39]	Wyn Galbraith remembers ascii :)
[7:39]	Zero Linden:	well....
[7:39]	Saijanai Kuhn wants to bring ascii grpahics back to SL.
[7:39]	Zero Linden:	I was going to talk a bit about "High Availability"
[7:40]	Saijanai Kuhn:	all it needs is a fixed width font
[7:40]	Saijanai Kuhn:	in llSetText
[7:40]	Zero Linden:	or rather, our quest to figure out what to do about it
[7:40]	Zero Linden:	Alas, I can't speak much to the voice client side implementation
[7:40]	Zero Linden:	other than knowing it is in a separate process......
[7:40]	Johanna Hyacinth:	OK
[7:41]	Wyn Galbraith:	High availability of what, Zero?
[7:41]	Zero Linden:	Well, anything, really
[7:41]	Saijanai Kuhn:	people have shown me evidence that when the CPU activity of the voice client spikes, it impacts the client to the point of freezing
[7:41]	Wyn Galbraith:	That can be a huge subject.
[7:41]	Panacea Pangaea:	Is that the Sun thingy?
[7:41]	Zero Linden:	Here's a more contained statement of the problem
[7:41]	Rex Cronon:	u can use sculpt letters saijanai:)
[7:41]	Zero Linden:	No, it is a general system concept
[7:42]	Zero Linden:	though they often tout their ability to do it
[7:42]	Saijanai Kuhn:	little man
[7:42]	Zero Linden:	for us, the problem is really simply this
[7:42]	Zero Linden:	we have several 1000 client processes on several 1000 machines each trying to talk to some common service
[7:43]	Zero Linden:	for example, every simulator needs to be able to ask about the status of any arbitrary avatar
[7:44]	Zero Linden:	while we can put caches in front of that central service, not all problems work that way - sometimes the data is too dynamic for that to help.
[7:44]	Zero Linden:	another example would be, "which machine is running region Foo"?
[7:44]	Zero Linden:	that is a rather important data set, that needs to be be both highly available, and is very dynamic
[7:44]	Zero Linden:	now it is easy for us to write a nice little python based service, and set this up on a machine
[7:44]	Rex Cronon:	we talking about voice here?
[7:45]	Zero Linden:	no, just any service that the grid needs to run, where that service is essentially central in nature (if not implementation)
[7:45]	Zero Linden:	like "where is this region running?" or "is avatar X on line?" or "who are the members of this group?"
[7:46]	Zero Linden:	so we have, say 12k clients (the regions), hitting some central server with HTTP queries, and we ahve this central server providing answers and managing some dynamic data
[7:46]	Zero Linden:	and....
[7:47]	Zero Linden:	poof.... the central server dies, or looses network connectivity, or even worse, just slows to a crawl
[7:47]	Zero Linden:	We need a way for some other central machine to take over that function... because without that function, the grid dies
[7:48]	Zero Linden:	So, there are two halves to this problem
[7:48]	Zero Linden:	a) we need a way to ensure that the dataset gets moved to the new machine, and the new machine to take over, without corrupting the dataset
[7:48]	Zero Linden:	and
[7:49]	Zero Linden:	b) we need a way to make sure that the 12k simulators now all start asking their queires of the new server, and skip the old
[7:49]	Zero Linden:	Now, it turns out that a) is the less difficult part - we can serialize the data, replay journal logs, or use a redudant database cluster, etc....
[7:49]	Zero Linden:	the harder part is getting 12k clients to all of a sudden know to talk to a different machine
[7:50]	Zero Linden:	Following me?
[7:50]	Panacea Pangaea:	I think so
[7:50]	Rex Cronon:	think static variables:)
[7:51]	Zero Linden:	So, Look at this and think.... well, we could sovle this by, for each service, figuring out some way for the client to work around dead servers and find new ones....
[7:51]	Zero Linden:	but since were likely to have dozens of such things, solving each time seems awkward
[7:52]	Zero Linden:	Righ now we solve it layer below that: Each client, rather than making the HTTP call directly, calls a service layer and says "I want to make a HTTP call to the X service"
[7:52]	Zero Linden:	the service layer picks up a configuration file from the machine that says, for each service, which machine and URL to use
[7:53]	Zero Linden:	So, if a central server dies, we can update the configuration file to point to a new server, and all the clients just change
[7:53]	Scope Cleaver:	They're IP's?
[7:53]	Zero Linden:	we were even clever enough to make the service layer check that file live, and notice when it gets updated
[7:53]	Zero Linden:	Scope - usually DNS names
[7:54]	Scope Cleaver:	Ah okay, in that case why do you fear they might not contact the new machine?
[7:54]	Zero Linden:	BUT - turns out getting a .xml file updated on 4k machines isn't always reliable..... and slower than you'd think!
[7:54]	Zero Linden:	That's why!
[7:54]	Zero Linden:	And once you've got two or more co-location facilities (as we do), then getting it to all the machines in a timely and reliable manner becomes even more difficult
[7:55]	Zero Linden:	So, we could attack these problems, and figure out a way to get the configuration info distributed faster and more reliably
[7:55]	Zero Linden:	OR
[7:55]	Zero Linden:	we could attack the high-availability issue at a layer below it
[7:55]	Zero Linden:	and build clusters of machines for each service
[7:56]	Zero Linden:	and have the machines use techniques like "IP-Takeover" and "Fencing" to cause one machine to take over for another in a failure without any client knowing about it
[7:56]	Panacea Pangaea:	That sounds like a better idea
[7:57]	Zero Linden:	This stuff seems more reliable - and certainly faster, BUT - it involves a much more compilcated operations set up.....
[7:57]	Zero Linden:	right now we basically just roll racks of machines, configured into small subnets, into the co-lo and plug 'em in
[7:58]	Zero Linden:	this would require machines with multiple ethernet interfaces and networks, and controllable power supplies
[7:58]	Panacea Pangaea:	Not sure I understand the last part ... power supplies ...
[7:59]	Panacea Pangaea:	How does that become an issue?
[7:59]	Scope Cleaver:	controlable PSU's
[8:00]	Zero Linden:	well- here is what things like "Red Hat Cluster Software" do: If the controlling server notices that machine A isn't responding.... it assigns machine B to take over the services that A was running.
[8:00]	Zero Linden:	Then machine B gets all the data ready, and before it starts, it TURNS OFF THE POWER on machine A!
[8:01]	Zero Linden:	This is to ensure that the same service isn't running on two machines.... like you wouldn't want two different machines with their own idea of your L$ ballance, say
[8:01]	Panacea Pangaea:	Indeed
[8:01]	Zero Linden:	Then, machine B reconfigures it's second ethernet port to have the same IP address as machine A *had*
[8:01]	Zero Linden:	and off it goes
[8:02]	Zero Linden:	all the clients have no idea that machine B has now become the service
[8:02]	Panacea Pangaea:	And the problem with implementing that is ..?
[8:02]	Rex Cronon:	why would u need to turn power off? can't u just reset? and a takes over only until b resets. or if b can't reset?
[8:02]	Zero Linden:	so this is nice, but requires, as you see, two ethernet nets (one for control, one for the service) and the ability for machines to power eachother
[8:02]	Zero Linden:	Rex - well, yes, you can reset, and presumably when machine A comes back it asks the controller what it should be doing
[8:02]	Scope Cleaver:	How robust is it?
[8:03]	Zero Linden:	I think the reason they power off is that it is possible for machines to wedge in ways that mean they don't reset.... since
[8:03]	Zero Linden:	since machine A is going off-line for a while, it is safest to just power it off, then on, and start from a cold, fresh state
[8:03]	Scope Cleaver:	And if there is a general power failure how do you trace back which machine was live?
[8:03]	Panacea Pangaea:	The important thing is that it is fast, reliable, and seamless
[8:04]	Zero Linden:	Panacea, the problem with doing this is that it requires careful configuration of clusters of machines into groups
[8:04]	Rex Cronon:	turning power on/off repeatedly places stress on electronic components. therefore reducing their life
[8:04]	Zero Linden:	Operations people in general, and ours specifically, prefer that there is one and only one configuration of machine. Or at most a very small number
[8:05]	Scope Cleaver:	You wouldnt want to end up with two worlds would you? lol
[8:05]	Zero Linden:	that way the hardware is in general interchangeable, and the configuration can be done mechanically, in bulk, and before we know the particualr use a machine will be used for
[8:05]	Zero Linden:	remember that we are putting something like 150 to 250 regions on-line a week...
[8:06]	Zero Linden:	that is 30 to 80 machines a week being installed
[8:06]	Zero Linden:	so, we buy whole racks of configured machines (41 servers to a rack) and just roll 'em in and turn 'em on
[8:07]	Wyn Galbraith:	Do they ghost the setup to make it easier to bring systems back onine?
[8:07]	Zero Linden:	if we decided we needed five machines in the next week to implement some service like this, then someone has to go down to the co-lo,
[8:07]	Panacea Pangaea:	You said earlier that caching wasn't dynamic enough - is there any way to create a more dynamic cache?
[8:07]	Zero Linden:	pick five machines, add ethernet interfaces, and power control, and re-wire 'em, then configure them to be differen than the others
[8:08]	Kooky Jetaime quietly mumbles to himself..how can I have "gesture missing from database" when I'm not loading any gestures......
[8:08]	Scope Cleaver:	Which of the options scales best?
[8:08]	Wyn Galbraith would love to see the control room.
[8:08]	Zero Linden:	so far, we haven't found a caching solution for highly dynamic data on our scale that works. memcached is sort of what you want, but it wasn't designed to work over 5k machines
[8:08]	Rex Cronon:	there is a bug on jira about it kookie
[8:08]	Rex Cronon:	kooky*
[8:09]	Zero Linden:	and, caching still doesn't solve what to do when the central service itself dies
[8:09]	Zero Linden:	Wyn - there are two, a co-lo in SF and a co-lo in Dallas ---
[8:09]	Zero Linden:	I'm not sure they are that interesting... I've not been to either, but I've spent plenty of time in the co-lo facility that houses my personal server
[8:10]	Panacea Pangaea:	Another centyral server would still have to take over
[8:10]	Wyn Galbraith:	I bet it's a very busy place. So what are you planning to do, Zero.
[8:10]	Zero Linden:	there is about a 5min. wow factor, looking at a very neat layout of several hundred racks, and many thousands of machines, and more bandwidth than you could shake a stick at
[8:10]	Zero Linden:	but it quickly becomes just a noisy, cramped place to try to get some work done
[8:11]	Kooky Jetaime:	But even more than you could shake a stick at, still isn't enough is it :)
[8:11]	Wyn Galbraith:	It sounds like you need a traffic controller. I hope they keep it neat.
[8:12]	Zero Linden:	No - but when you play around with a single machine in such places it is cool.... Like asking your linux server to update it's packages, and it says there are 150 packages to download and update... and it does the download in under a minute
[8:12]	Zero Linden:	really, accessing the internet at 100BaseT is awesome!
[8:13]	Zero Linden:	Wyn- I don't know what we are planning on doing - that is one of my major tasks this month -- to figure it out
[8:13]	Kooky Jetaime:	Just don't plan a nice month long vacation
[8:14]	Panacea Pangaea:	What I was imagining as you described the problem was countless little balls being kept airborn by the server - the data requested ...
[8:14]	Zero Linden:	right now we do a combination of application layer specific solutions (when you try to TP to a region, and the region goes down as you are doing it, the sim rertires and does rediscovery to find where the region came up )
[8:14]	Panacea Pangaea:	Like the server is juggling the balls ...
[8:14]	Zero Linden:	and the service layer solution (the service.xml configuration file)
[8:14]	Rex Cronon:	more like eggs
[8:15]	Panacea Pangaea:	And what you want is for another server to be able to take over the juggling without dropping any
[8:15]	Zero Linden:	No, my vacation was earlier this year - our family added a baby - I took my time then!
[8:15]	Wyn Galbraith:	Is there no where else in the industry that has a similar situation? You need some way to keep track of who's doing what.
[8:15]	Zero Linden:	Panacea - yes, preferably without the server noticing at all
[8:15]	Kooky Jetaime:	Congrats
[8:16]	Panacea Pangaea:	So the balls are really the interface
[8:16]	Wyn Galbraith:	Almost as if you need a whole other set of systems that do just traffic control.
[8:16]	Panacea Pangaea:	like a dynamic cache
[8:16]	Wyn Galbraith:	Each layer added though adds an element of time and error.
[8:16]	Saijanai Kuhn wonders how Wow and the like handle it...
[8:17]	Wyn Galbraith was thinking the same thing Saijanai, "Not that they'd tell."
[8:17]	Zero Linden:	Well, if there were a manager server... what would happen if THAT went down ... it needs to be backed up too..... and then who manages THAT cluster...
[8:17]	Zero Linden:	well, rather than having "turtles all the way down"
[8:17]	Panacea Pangaea:	:-\
[8:17]	Turtle: What do you want
[8:17]	Turtle: Seriously, what do you want?
[8:17]	Zero Linden:	usually you have all the cluster machines use a form of voting and majority rule to manage themselves
[8:18]	Rex Cronon:	ai?
[8:18]	Zero Linden:	hmmm.... do we want to trust 'em? give 'em that much power and they might turn....
[8:18]	Wyn Galbraith:	Have you ever considered using a Cray?
[8:19]	Zero Linden:	When I was at Apple, they bought a Cray... i have the dubious distinction of being the guy who named it
[8:19]	Zero Linden:	"TMA-1"
[8:19]	Saijanai Kuhn waits for it...
[8:19]	Wyn Galbraith:	Very cool, I started my career with Control Data.
[8:19]	Saijanai Kuhn:	what does TMA stand for
[8:20]	Kooky Jetaime:	I seem to know this name..but can't place it.
[8:20]	Wyn Galbraith:	Spent many hours behind the mainframe in the wire racks. To Much Action?
[8:20]	Zero Linden:	'cmon on... no one? Hint: It is a movie reference
[8:20]	Kooky Jetaime:	Oh
[8:20]	Kooky Jetaime:	oh
[8:20]	Kooky Jetaime:	I got it
[8:20]	Wyn Galbraith:	Time Machine?
[8:20]	Kooky Jetaime:	Tycho Magnetic Anomoly 1
[8:20]	Kooky Jetaime:	The Moon Monolith
[8:20]	Zero Linden:	Bingo! Kooky wins a Linden Bear
[8:21]	Saijanai Kuhn:	ah...
[8:21]	Wyn Galbraith:	I don't remember that movie.
[8:21]	Kooky Jetaime:	Its 2001
[8:21]	Saijanai Kuhn:	2001
[8:21]	Wyn Galbraith facepalms.
[8:21]	Zero Linden:	remember the shape of a Cray-1? The circle of slabs with the bench seat around it?
[8:21]	Wyn Galbraith:	I know that movie. The Cray does lend itself to that.
[8:21]	Wyn Galbraith:	Yes.
[8:21]	Rex Cronon:	the moon monolith:)
[8:21]	Kooky Jetaime:	Not that most people remember they called it tma-1.. I sure didn't..... I was reading about it a few months ago..
[8:22]	Wyn Galbraith needs coffee, "Haven't had any yet."
[8:22]	Zero Linden:	http://en.wikipedia.org/wiki/Cray_X-MP
[8:22]	Zero Linden:	that's the machine, only Apple's was purple
[8:22]	Wyn Galbraith:	Anyway, would a super dupper mainfame help?
[8:23]	Zero Linden:	wow - this fact is mentioned at the bottom of this page: http://en.wikipedia.org/wiki/The_Monolith
[8:24]	Wyn Galbraith:	Should add the facts to that Zero.
[8:24]	Kooky Jetaime:	Zero- maybe you should add the citation and take the credit :)
[8:24]	Zero Linden:	no, really, a larger computer doesn't help --
[8:24]	Wyn Galbraith:	LOL, owe me a coke, Kooky!
[8:24]	Zero Linden:	these problems aren't really ones about speed, or memory size - they are about failure - and failure happens in large comptuers too
[8:25]	Wyn Galbraith:	It does, but if you use one super computer... oh, but then if that went down, you'd be stuck.
[8:25]	Kooky Jetaime:	Well, if there is anything that SL needs raw processing power for, pick up a couple of Tesla's (nVidia) they are basically 2 TFLOP desktop servers they've designed utilizing their new GPUs
[8:25]	Wyn Galbraith:	It's an interesting problem, Zero.
[8:25]	Zero Linden:	The only think in our system right now that needs major processing is....
[8:25]	Zero Linden:	...anyone wanna guess?....
[8:25]	Kooky Jetaime:	Physics?
[8:25]	Zero Linden:	...ignoring the viewers....
[8:25]	Kooky Jetaime:	Assets?
[8:25]	Wyn Galbraith:	My inventory?
[8:26]	Panacea Pangaea:	LOL
[8:26]	Kooky Jetaime:	The Database is my guess
[8:26]	Zero Linden:	nope - assets don't need compute power, just bandwidth
[8:26]	Saijanai Kuhn:	tech support?
[8:26]	Zero Linden:	nope - database generally runs out of IO bandwidth to the disks
[8:26]	Rex Cronon:	i know. sculpties:)
[8:26]	Wyn Galbraith:	LOL Sai.
[8:26]	Kooky Jetaime:	ignoring the viewers, whats left? Bear Design?
[8:26]	Zero Linden:	pyhsics does, or can take CPU power, but usually when it does it is "borked" and it would just use all it could get and still fail!
[8:27]	Saijanai Kuhn:	ascii text graphics
[8:27]	Zero Linden:	...wait for it...
[8:27]	Zero Linden:	The Map!
[8:27]	Saijanai Kuhn blinks slowly
[8:27]	Kooky Jetaime raises an eyebrow
[8:27]	Zero Linden:	That is because it is the only rendering task we do on our side!
[8:27]	Zero Linden:	and we have to render EVERYTHING
[8:27]	Wyn Galbraith:	OH! Let's get rid of the map ;)
[8:27]	Rex Cronon:	i think qarl said that nobody knows how it works
[8:27]	Zero Linden:	it is big!
[8:27]	Zero Linden:	and it needs to compute average colors for every texture.....
[8:28]	Zero Linden:	it is a large, unwieldly process, even divided into chunks
[8:28]	Wyn Galbraith:	It's huge! I was looking at it the other day, that's how I explore, and I thought, I'd hate to keep track of all this stuff.
[8:28]	Panacea Pangaea:	Seems to me that unless the server can let everyone know with its dying breath to use another server, you will either need a front end of some sort or run round updating everyones config :-\
[8:28]	Kooky Jetaime:	i know it seemed like we went a month or two between map updates
[8:28]	Saijanai Kuhn:	there's where that new nvidia super computer comes in...
[8:28]	JetZep Zabelin:	WHy does the map have to refresh so often when it seems snapshots arent taken very often
[8:28]	Zero Linden:	perhaps
[8:28]	Rex Cronon:	why don't u render only the terain, and let each viewer render the objects?
[8:28]	Zero Linden:	one problem with the map is taht our machines have no 3d graphics cards in 'em
[8:28]	Zero Linden:	really!
[8:28]	Wyn Galbraith:	How does something like Google maps keep track of all that stuff?
[8:29]	Zero Linden:	and the software only implementations of OpenGL are, well, slow
[8:29]	Zero Linden:	since there isn't much call for 'em, there is little need to optimize 'em
[8:29]	Zero Linden:	and really, 3d hw really really does do this stuf better!
[8:29]	Zero Linden:	so the maps are rendering entirely in software... which makes it SLOW
[8:30]	Kooky Jetaime:	Zero, even if it is done slowly, couldn't mapping be relegated to the sims as an "idle time" process? When its vacant/has extra frame time start working on a map... once it completes, upload the graphic to the asset server and then just pull from there?
[8:30]	Zero Linden:	so - yes, if operations could stomach another "special purpose configuration" -- or even "special purpose machine", then fancy SGI boxes could help
[8:31]	Saijanai Kuhn:	or just a few hundred medium-end game machines
[8:31]	Kooky Jetaime:	its not like the image needs to be updated every day.... once a week I think is the goal, there seems like there should be enough time per week to crank out 1 image
[8:31]	Zero Linden:	Kooky-well, perhaps, but we are learly of putting a process that compute intensive on a machines that we want to be responsive
[8:31]	Zero Linden:	even at lower priority
[8:31]	Rex Cronon:	so how often is right now the map being updated?
[8:32]	Zero Linden:	I don't know
[8:32]	Wyn Galbraith:	So if the map problem was handled better, what would that mean?
[8:32]	Saijanai Kuhn:	is easy to test. Build something visible from space and see how long it takes for it to show up
[8:32]	Wyn Galbraith just checked the map for items she's moved form one sim to another.
[8:33]	Rex Cronon:	it must be interesting when it has to handle megaprim sculpties:)
[8:33]	Kooky Jetaime:	Zero- how about a change flag on each Sim...... until a sim has a change in excess of a certain percentage, until that amount is reached, just use a previous rendering
[8:33]	Panacea Pangaea:	It can take several days sometimes
[8:33]	Zero Linden:	ooooo Skype is down... suppose we aren't the only ones with operations troubles....
[8:33]	Kooky Jetaime:	might save a few cycles
[8:34]	Zero Linden:	I noticed that Facebook disabled logins for a few hours yesterday
[8:34]	Wyn Galbraith:	Tweaksru.com was down for days.
[8:34]	Zero Linden:	Kooky - you'd be shocked how much changes! SL is very dynamic over much of it!
[8:34]	Zero Linden:	well all... it is time for me to go
[8:34]	Zero Linden:	thanks for coming.....
[8:35]	Zero Linden:	'till next week
[8:35]	Johanna Hyacinth:	Thanks for the chat!
[8:35]	Rex Cronon:	bye zero
[8:35]	Kooky Jetaime:	See ya Tuesday
[8:35]	Saijanai Kuhn:	later