User:Zero Linden/Office Hours/2008 May 15

• [8:35] Zero Linden: Gooooood morning all
• [8:35] JayR Cela: hi there zero :_)
• [8:35] Anya Heberle: Good Morning sleepy head
• [8:35] Arawn Spitteler: wonders what would happen, if he set integer size_y to 0: If I do that, I get more pixels on the basic texture. Hi, Zero.
• [8:35] Rex Cronon: hi zero
• [8:35] Marc Montague: Good morning to you Zero
• [8:36] Zero Linden: I'm almost ready here
• [8:36] joao Mastroianni: hello^^
• [8:36] Zero Linden: there - coffee
• [8:36] JayR Cela: Arawn / might work
• [8:36] Rex Cronon: hi
• [8:36] Zero Linden: much better
• [8:36] Zero Linden: d'oh - just spilled it all behind my head
• [8:36] Zero Linden: dang animation interactions!
• [8:36] joao Mastroianni: hello zero its a pleasure^^
• [8:36] Wyn Galbraith: Coffee is so important. *sips*
• [8:36] Anya Heberle: lol
• [8:36] Zha Ewry: Coffee management issues
• [8:36] JayR Cela: Hi there Zha :_)
• [8:36] Black Kitten: Coffee : Hot strong coffee!!
• [8:36] Zero Linden: I suppose the only real way to get that sort of thing right is to have a semantic tag on each motion in an animation and then use some
• [8:37] Zero Linden: AI system to figure out what take priority over what....
• [8:37] Zero Linden: yes - holding a cup takes priority over putting your hands behind your head...
• [8:37] Zero Linden: but beating down the attacking zombie takes precedence over spilling your coffee
• [8:37] dragonflymystic Freschi: yeah ... havent had my iced tea
• [8:37] Tao Takashi: Hi there
• [8:37] JayR Cela: lol :_)
• [8:38] Zha Ewry: sets AWG tag
• [8:38] dragonflymystic Freschi: dont drink coffee anymore
• [8:38] Rex Cronon: hi tao
• [8:38] JayR Cela: hi tao :_)
• [8:38] joao Mastroianni: can i ask a question zero?
• [8:38] couch sit: 1 RED.: BigMike Bukowski, say '/1 Hide' to hide me, or '/1 Show' to make me show. Or just right-click and sit on me to use me.
• [8:39] Dolf Rhino: Hi
• [8:39] Zero Linden: Agena items, anyone?
• [8:39] Zero Linden: Joao - have an item?
• [8:39] joao Mastroianni:  ??
• [8:39] Tao Takashi: I wonder if there is a plan to register some domain for ogp to make the URL shorter :)
• [8:39] joao Mastroianni: item?
• [8:39] Zero Linden: is your question an item for the agenda?
• [8:39] JayR Cela: things seem to be running pretty smothly lately
• [8:39] Rex Cronon: subjects of discussion for todays office hours
• [8:39] joao Mastroianni: mmm
• [8:40] joao Mastroianni: well dont know just a doubt
• [8:40] joao Mastroianni: about the servers
• [8:40] Anya Heberle: are allLindensCertifyable
• [8:40] Anya Heberle: and if so
• [8:40] Tao Takashi: we can make it an agenda item if there's not a quick answer
• [8:40] Anya Heberle: .. Can I be one
• [8:40] Arawn Spitteler: was thinking, this morning, that some way is needed, to keep German IPAddresses from seeing anything a Nazi would consider Naughty.
• [8:40] Zero Linden: what is it, joao?
• [8:41] joao Mastroianni: well now ive been receiving lots of warnings that im entering in a diferent simulator
• [8:41] joao Mastroianni: in Linden village
• [8:41] Zero Linden: Arawn - without human decision in that loop, it is a big AI research project
• [8:41] joao Mastroianni: but from 1 day to another they change
• [8:41] JayR Cela: thats normal joao
• [8:41] joao Mastroianni: so wich server is
• [8:41] joao Mastroianni: class 4 or 5
• [8:41] Zero Linden: Yes - that is because which regions are running a "test" simulator change from time to time
• [8:41] joao Mastroianni: or is there a new kind?
• [8:41] JayR Cela: as LL updates the server code in sections
• [8:42] Tao Takashi: JayR: did you happen to find that link again? :)
• [8:42] joao Mastroianni: ahh
• [8:42] Zero Linden: it's not the class that is different, it is the version of the simulator software
• [8:42] joao Mastroianni: ahh
• [8:42] JayR Cela: Tao @ what link was that ?
• [8:42] joao Mastroianni: ok and how we know wich ias the faster?
• [8:42] joao Mastroianni: if we can know?
• [8:42] Saijanai Kuhn: only one kind of code per compjuter though, as I understand it, so you should see 4 simulataors with thesame code grouped together
• [8:42] Tao Takashi: about IBM's VW interop announcement or whatever it was. maybe Zha knows more? :)
• [8:42] Arawn Spitteler: The warning flags don't tell us anything, but that we can ask for the same lack of information
• [8:43] Bell Boyd: frequent minor chnges in Havek4?
• [8:43] Bell Boyd: Havok
• [8:43] Tao Takashi: the release notes are simply missing in that popup
• [8:43] JayR Cela: @Tao / oh yeah / darnit I cant remember now
• [8:43] Zero Linden: Saijani - that is correct, a version of the software loads onto a single machine, which simulates mutliple regions
• [8:43] joao Mastroianni: anyway i must congratulate you guys this is faster^^
• [8:43] Zero Linden: (one per CPU, or in some cases (voids) 4 per CPU)
• [8:43] Saijanai Kuhn: are they contiguous regions, Zero?
• [8:44] Zero Linden: since all the regions on a machine share some server processes (dataserver, backbone, squid, apache),
• [8:44] Free Radar: HUD v1.1 by Crystal Gadgets
• [8:44] Zero Linden: they must all be at the same rev
• [8:44] Conover's Flight-Helper: 6.3.3 (WEAR ME!): Flight-helper is ready and operational.
• [8:44] Free Radar: HUD v1.1 by Crystal Gadgets
• [8:44] Saijanai Kuhn: topolgogically continguous?
• [8:44] Saijanai Kuhn: share thesame sim border?
• [8:44] Zero Linden: The coordination of which regions can run where is a very very complex task
• [8:44] Tao Takashi: just be on the same machine
• [8:44] Arawn Spitteler: understands they don't have to be on the same grid.
• [8:45] Tao Takashi: they might be anywhere as I understand it
• [8:45] Zero Linden: we've been doing it manually for months (Go Release Team, Go!)
• [8:45] Zero Linden: But very very soon (already) we will be using "The Conductor" that will coordinate all of this automatically
• [8:45] Tao Takashi: hopefully ;-)
• [8:45] Zero Linden: including finding spare machines, figuring out what SW load is needed for waiting regions, loading it, and then starting and running them
• [8:45] JayR Cela: whats that Zero ?
• [8:45] Zha Ewry: tao? I'm not quite sure what you're asking
• [8:45] Saijanai Kuhn: periapse gave a talkon this at the Mono office hours. Put a link on the AW Grouipes page
• [8:46] joao Mastroianni: zero the servers when we change there is fawls cause the data is not yet in the next server right?
• [8:46] Zero Linden: No, to answer the question I think was asked above, no, the regions on a single host are not, in general, topologically adjacent
• [8:46] Saijanai Kuhn: https://wiki.secondlife.com/wiki/Mono/2008-04-04
• [8:47] Tao Takashi: Zha: JayR mentioned some announcement from IBM at the last dataportability meetup but couldn't find the link anymore.. so I don't know myself what it might have been ;-) Was there any announcement by IBM regarding VW interop recently?
• [8:47] joao Mastroianni: ok but we use small servers,isnt possible
• [8:47] Dr Scofield: is that deliberate or just coincidence?
• [8:47] Dr Scofield: and, hi :-)
• [8:47] joao Mastroianni: to use a big server to many regions and so no fawls
• [8:47] JayR Cela: well I think region crossings are getting a bit better lately
• [8:47] Rex Cronon: hi
• [8:47] joao Mastroianni: sorry if my question is ignorant-...
• [8:47] Saijanai Kuhn: 15:35ish in the transcripts, Periapse talks about what Conductor will do
• [8:48] Zero Linden: Dr. S. - it is just coincidence - the rigidity of tying regions to particular machines which would be needed to keep adjacent ergions on a host isn't worth the loss of flexibility
• [8:48] Anya Heberle: yes in thelastyear there islittle difference in the lag of simswith adjacents sims
• [8:48] Anya Heberle: I second that
• [8:48] Anya Heberle: its great
• [8:48] Anya Heberle: SLB4 was poor slow because ofadjacentssims
• [8:48] Zero Linden: especially when you realize that regions are STILL goign to have to talk to neighbors
• [8:48] Dr Scofield: yep, true, didn't think that far
• [8:49] Arawn Spitteler: Fawls?
• [8:49] Zero Linden: and it also means that dense areas will have a higher likely hood of having all four regions hogging the same host
• [8:49] Zero Linden: The conductor, once it is deployed, and works will finally be the first time that we can have an automated
• [8:49] JayR Cela: ahh / simm corners comes to mind
• [8:49] Zero Linden: intellegence placing regions - it could be extended easily to look at historical load for a region, and the load on
• [8:50] Zero Linden: existing machines with an available CPU, and place it accordingly
• [8:50] Zero Linden: That kind of load balancing wasn't possible with the old system
• [8:51] Dr Scofield: do you have a mechanism that would "split" a region on to two servers when the load gets too high? (like OpenSim's region splitting? though i haven;t tried that myself)
• [8:52] Arawn Spitteler: doesn't think that will happen, until regions are ready to run at reduced size.
• [8:52] Zero Linden: No, we don't, for several reasons
• [8:53] Zero Linden: As a C++ application, migrating live data structures involves coding - and a fair bit of it
• [8:53] Dr Scofield: ok. good point
• [8:53] Zero Linden: Live migration of connected avatars could be difficult
• [8:53] Zero Linden: (as the viewer connects dirrectly to a socket on the sim host... how do you migrate that live to another host? - we'd need protocol in the viewer to support it
• [8:54] JayR Cela: i recently to a visit to OpenLife / and used the RealExtend client
• [8:54] JayR Cela: so so results
• [8:56] Zero Linden: I'd like to turn our attention to RHTTP
• [8:56] JayR Cela: k :_)
• [8:56] Zero Linden: https://wiki.secondlife.com/wiki/Reverse_HTTP
• [8:56] Tao Takashi: Raw HTTP ;-)
• [8:57] Zero Linden: If you haven't read it, and are interested, please do
• [8:57] Arawn Spitteler: Retro-Hyper
• [8:57] Zero Linden: I believe that on Tuesday, Icehouse folks are comign to the AWGroupies meeting to talk, yes?
• [8:57] Zha Ewry: yes yes
• [8:57] JayR Cela: IceHouse ?
• [8:57] Zha Ewry: Donovan is on tap for that
• [8:58] Zero Linden: The cool thing is - it works - and in python it is under 100 lines of code
• [8:58] Zero Linden: though, frankly it would need to be a bit longer "for real"
• [8:58] JayR Cela: what is it ?
• [8:58] Tao Takashi: did he put that in eventlet?
• [8:58] Zero Linden: It is a"reverse HTTP" -
• [8:58] Zero Linden: Tao - no it works with the standard python HTTP client/server libs!
• [8:58] Tao Takashi: cool
• [8:59] Zero Linden: RHTTP is the following simply idea
• [8:59] Dyne Talamasca: neat
• [8:59] joao Mastroianni: the server becomes client and the client server,basicly is that?
• [8:59] Zero Linden: establish a standard run-of-the-mill HTTP connection from viewer to simulator (for example) intending to do a polled request for events
• [8:59] Zero Linden: BUT
• [8:59] Zero Linden: use the HTTP/1.1 facilities of protocol upgrade (The Upgrade: header and the 101 status code)
• [9:00] Zero Linden: to agree to switch to "PTTH" (reverse HTTP)
• [9:00] Tao Takashi: you should promote this wildly, make browsers and servers implement it and everybody will be happy! :-)
• [9:00] Zero Linden: at whcih point the protocol is *exactly* HTTP from the simulator (as client) to the viewer (as server)
• [9:00] Saijanai Kuhn: Thiswill make it really easy to create remote control test harnesses
• [9:01] Tao Takashi: of course IE would make it with their own spin on it ;-)
• [9:01] Zero Linden: With teh right support in existing HTTP client and server libraries, this can be very trivial to implement ---
• [9:01] Zero Linden: basically a library has to allow you to tell it to be a HTTP client or server with a given, already connected, socket
• [9:01] Zero Linden: many do, as they separate out the "connect" phase from the rest of the protocol
• [9:01] Zero Linden: but some do not....
• [9:02] Zero Linden: that wouldd be a barrier to entry
• [9:02] Tao Takashi: are you planning to release the code?
• [9:02] Zero Linden: I'm pretty sure that Donovan will be releaseing the Python code
• [9:02] Zero Linden: we haven't done this in C++ yet
• [9:02] Zero Linden: Alas, it is probably NOT doable from JavaScript
• [9:02] Tao Takashi: well, I am more interested in Python anyway ;-)
• [9:02] Zero Linden: as the interface to behing a HTTP client is very highlevel, and there is no library for being an HTTP server
• [9:02] Tao Takashi: but JS would have been nice of course
• [9:03] Rex Cronon: does this RHTTP work when the user is behind a firewall?
• [9:03] Zha Ewry: javascript, It hink, would have to be done in the JS container
• [9:03] Rex Cronon: or if user is using a proxy?
• [9:03] Zha Ewry: JS inherently doesn't have the library
• [9:04] Zha Ewry: The container, does, for all sorts of good reasons
• [9:04] JayR Cela: I think is avaiable with latest verson of NetBeans
• [9:04] Zero Linden: Rex - yes it does
• [9:04] Zero Linden: the very reason for it's existance
• [9:04] Zero Linden: However, there is a hitch
• [9:04] Zero Linden: (as there always are)
• [9:05] Zero Linden: The HTTP/1.1 upgrade mechanism is connection level, not end-to-end transport level
• [9:05] Zero Linden: so you are only upgrading the first link
• [9:06] Zero Linden: If you are contacting a proxy explicitly to do your HTTP out, then this Upgrade:; is a request to the proxy, not the far end
• [9:06] Zero Linden: The only other thing to use the Upgrade mechanism is
• [9:07] Zero Linden: Upgradeing to TLS within HTTP: RFC 2817
• [9:07] Zero Linden: [1]
• [9:07] Zero Linden: (Note to be confused with HTTP over TLS, RFC 2818)
• [9:07] Dr Scofield: hmmm....so any proxied connection is hosed?
• [9:08] Zero Linden: (*not*)
• [9:08] Zero Linden: Well, any explicitly proxied connection needs to be treated special
• [9:08] Zero Linden: you need to ask the proxy to perform a CONNECTION to the far end
• [9:08] Zero Linden: *then* proceed with the Upgrade:
• [9:08] Zha Ewry: Do we have anythoughts on how to test this, in the small, before depending on it?
• [9:08] Zero Linden: this is defined in the proxy spec, but it isn't clear to us if it is actually implemented, and if so, is it commonly administratively allowed
• [9:09] Zero Linden: Zha - well, the first step was getting an implementation to work with - and that is what Donovan did this week
• [9:09] Zha Ewry: nods
• [9:09] Zero Linden: Another hanging question is will this work through
• [9:09] Zero Linden: transparent proxies?
• [9:09] Zha Ewry: I'm thinking of a way to get the pair of endpoints in a lot of places and test it
• [9:10] Zero Linden: if you connect from A to B, but somwhere in the nextwork, a proxy inserts itself, so the flow is really A to P to B
• [9:10] Rex Cronon: u can add it as a module to the rcviewer
• [9:10] Rex Cronon: and those interested can test it
• [9:10] Zero Linden: One can't expect A to know that it's connection level aspects (there are several in HTTP) apply to P, P must act transparently
• [9:10] Zero Linden: BUT,
• [9:10] Zero Linden: who knows if exsiting transparent Proxies were designed and tested to handle Upgrade
• [9:11] Saijanai Kuhn: the code is already linked to from the reveres_http page on the wiki
• [9:11] Zero Linden: from the total lack of RFC 2817 implementation
• [9:11] Zero Linden: we can't infer wide spread support
• [9:11] Zero Linden: for Upgrade:
• [9:12] Saijanai Kuhn: The micro_POST-server command line mechaism I came up with for the login test harness would an ideal way to test this
• [9:13] Zero Linden: a brief google search implies that no browsers support it, thought it has been contemplated for Firefox as of 2006
• [9:13] Zero Linden: Apache 2.1 supports it (well, the support is claimed for 2.2, but the docs for 2.1 have the directive to support it)
• [9:14] Zero Linden: Of course, if we say this is just optional, that you can fall back on the event queue -- then we may be good no matter what the support for now
• [9:14] Zero Linden: Ah, so the code is!
• [9:14] Saijanai Kuhn: login script contacts the controller script server, which reverses itself to become a client POSTING GUI commands back to the test-harness
• [9:15] Saijanai Kuhn: tada! instant distributed testing over the internet
• [9:17] Tree Kyomoon: other than testing, what could you practically do with reverse http?
• [9:17] Tao Takashi: that looks like eventlet though ;-)
• [9:18] Saijanai Kuhn: gets rid of tunnelingissue for direct http hookup
• [9:18] Dr Scofield: no more event queue
• [9:18] Dr Scofield: ideially
• [9:18] Dr Scofield: ideally
• [9:18] Zero Linden: Tao - I think he's just using eventlet there to run two processes (the server and the client) from within the same code
• [9:18] Tao Takashi: trying right now to understand it ;-)
• [9:18] Zero Linden: The RHTTP stuff uses wsgi, which is generic
• [9:18] Tao Takashi: Linden Lab seems not to be a fan of docstring ;-)
• [9:19] Zero Linden: Of course, really, if you think about it
• [9:19] Zero Linden: having the rhttp client end (the simulator in our example), hold this socket open, with keep alives, so that it can send requests to the rhttp server (the viewer)
• [9:19] Zero Linden: is really exactly the same network state
• [9:20] Zero Linden: as the eventqueue long poll, where the viewer has made a request, and the simulator holds onto that socket, not replying, until it has something to send
• [9:20] Zero Linden: From a resource perspective, it is *EXACTLY* the same
• [9:20] Zero Linden: however
• [9:21] Zero Linden: some HTTP server libraries might balk at the long held open socket, whereas http client libs that support keep-alive don't at all
• [9:22] Zero Linden: though - to be honest, for any http server libraies that would appropriate to implement either of these, this isn't a problem
• [9:22] Zero Linden: I see the biggest advantage of this over the event queue
• [9:22] Zero Linden: is that there is no re-encapsulation of the requests
• [9:22] Zero Linden: the encapsualtion *IS* http, just as it would be in the other direction
• [9:22] Rex Cronon: holding so many open ports server side doesn't waste a lot of resources?
• [9:23] Saijanai Kuhn: would it make sense to ever do a double switch? Post to one, switch, post back, switch, etc?
• [9:23] Zero Linden: Where is this 'so many' idea coming from?
• [9:23] Zero Linden: A connected server port costs no more and no less than a client port
• [9:23] Zero Linden: If the simulator is going to send stuff to the viewer it needs a socket
• [9:23] Zero Linden: it makes no difference if it creates a client port and connects and holds that
• [9:23] Dr Scofield: grins
• [9:24] Dr Scofield: channelling
• [9:24] Zero Linden: or if it accepts a server connection (binds) and holds that
• [9:24] Dr Scofield: via the sub-ether
• [9:24] Zero Linden: same cost, same resources
• [9:24] Zero Linden: right now each simulator holds a UDP connection open to each agent
• [9:24] Zero Linden: and accepts at most one incoming TCP connection for each event queue
• [9:25] Zero Linden: so - two per agent.....
• [9:25] Zero Linden: this doesn't seem like a large number to me.
• [9:25] Tao Takashi: except when we scale the region to hold a million avatars :)
• [9:25] Dr Scofield: with 40 avs max per island, that's just 80
• [9:25] Tree Kyomoon: zero is this a TCP send/receive/send kind of thing, or you keep an open connection continuously ?
• [9:25] Rex Cronon: i was thinking that u want like 1000 avatars per sim
• [9:26] Dr Scofield: so that would be just 2000
• [9:26] Rex Cronon: there*
• [9:26] Rex Cronon: thre are 4 sims per server
• [9:26] Dr Scofield: 8000 then
• [9:27] Dr Scofield: ports is not the issue, network buffer might
• [9:27] Rex Cronon: wouldn't having that many open connections create lag?
• [9:27] Zero Linden: Thats just 128M of buffer memory
• [9:27] Saijanai Kuhn: if you can scale to that many, you could probably start worrying about auxiliary servers to handle the extra load
• [9:27] Zero Linden: So - UDP connections must be held open for the duration
• [9:28] Zero Linden: TCP connections can be held open, or closed and reopened as needed
• [9:28] Zero Linden: having the port open, other than the memory footprint, doesn't cost much
• [9:28] Zero Linden: for UDP and TCP w/o keep-alive( TCP KA) there is *NO* network cost at all
• [9:28] Zero Linden: and no processing cost (as long as epoll or some such newer API than select is used)
• [9:29] Rex Cronon: if things are as u say, than there might be no problem
• [9:29] Zero Linden: If you have TCP keep-alive enabled, it is just one packet, send/received by the kernel every so often
• [9:29] Zero Linden: looks up the interval
• [9:29] Saijanai Kuhn: we can test it easily enough. Use Dovana's script as its own test harness to spawn a zillion processes and see what happens
• [9:30] Tiffany Sicling: will that work through NAT and firewalls without port forwarding ?
• [9:30] Tree Kyomoon: so PTTH does both TCP and UDP...interesting
• [9:30] Dr Scofield: should, since you are using an existing connection
• [9:30] Dr Scofield: which the client opened
• [9:30] Saijanai Kuhn: that's the beauty. the SERVER POSTS to the CLIENT first time through, so there's no need for port forwarding
• [9:31] Zero Linden: heh - the default TCP keep-alive interval is....
• [9:31] Zero Linden: ...wait for it.....
• [9:31] Zero Linden: ....2 hours!
• [9:31] Zero Linden: so, let's not worry 'bout that
• [9:31] Zero Linden: Tree - NOOOO PTTH only does HTTP
• [9:31] Tree Kyomoon: unless a romulan quantum singularity drive happens to be near by of course
• [9:32] Tedd Maa: cat /proc/sys/net/ipv4/tcp_keepalive_* -- but different on Windows
• [9:32] Arawn Spitteler: worries a bout the machine, which countes each of 7,200,000 milliseconds
• [9:32] Zero Linden: Ya know, we've had our TCP streams affected by Romulan Quantum Singularity Drive before... and it's not pretty
• [9:32] Saijanai Kuhn: its just a flag in the python library
• [9:32] Saijanai Kuhn: But Zero, can you be sure it was THAT, and not some OTHER Quantum Sigularity Drive?
• [9:33] Zero Linden: We had an Ensign from Engineering analyize the warp signature.... there was no doubt
• [9:33] Saijanai Kuhn: I'm convinced
• [9:34] Rex Cronon: i think that Quantum Sigularity Drive should be taken into consideration only after u have at least one in your possesion:)
• [9:34] Tree Kyomoon: (they speed up /slow down and reverse time...making the packets update time potentially disasterous)
• [9:34] Zero Linden: (Alas, Ensign Expendable died collecting that signature.... but such is the cost of Science!)
• [9:34] Zha Ewry: Hard to do QA otherwise
• [9:34] Saijanai Kuhn: tried for one of those positoins, bet multi-universal meta-tensors was beyond him
• [9:34] Tiffany Sicling: I think the improbability drive is better and is more suited
• [9:34] Saijanai Kuhn: just as well, it sounds like
• [9:34] Tree Kyomoon: 42
• [9:35] Zero Linden: Tiffany- indeed, and gets your packets delivered faster, and with extra icing!
• [9:35] Zero Linden: okay all
• [9:35] Zero Linden: thank you for coming
• [9:35] Arawn Spitteler: didn't like the Red Shirt, they threatened to make us wear.
• [9:35] Dr Scofield: thx for your time
• [9:35] Zero Linden: I'll see you all on Tuesday