User:Which Linden/Office Hours/2007 Dec 13

Transcript of Which Linden's office hours:

[11:00]	Saijanai Kuhn:	Speaking of stray pieces of bamboo...
[11:00]	WarKirby Magojiro:	hi there which
[11:00]	WarKirby Magojiro:	heh
[11:00]	Which Linden:	morning, all
[11:00]	WarKirby Magojiro:	very unusual look indeed
[11:00]	Saijanai Kuhn:	a friend wants to know why you don't have christmas tree lights, Which
[11:01]	Which Linden:	oh! I should do that
[11:01]	Saijanai Kuhn:	a little lone ornament ala the Peanuts tree
[11:01]	Which Linden:	Hah, that would be awesome
[11:02]	Which Linden:	sometime, when I have the time
[11:02]	WarKirby Magojiro:	these charts look quite relevant to a problem I've been having with customers
[11:02]	Which Linden:	What problem is that?
[11:03]	WarKirby Magojiro:	3 of them in 2 weeks, claimed they paid my vendors and recieved nothing
[11:03]	Groove Market:	They are all certified? lol...
[11:03]	WarKirby Magojiro:	I recieve no L$ from them, and nothing shows up in records
[11:03]	WarKirby Magojiro:	they could just be trying to scam me, though
[11:03]	Which Linden:	Oh, yeah
[11:04]	Which Linden:	One thing I'd like to see in the future is not only guaranteed transactions from vendors, but also a record of what transactions have been attempted or are in-flight
[11:04]	WarKirby Magojiro nods
[11:04]	Which Linden:	Like a little hud that shows a list of transactions that are in-progress
[11:04]	Tao Takashi:	Hi
[11:04]	WarKirby Magojiro:	Also, one thing that's definitely an issue
[11:04]	WarKirby Magojiro:	is the money event failing to trigger
[11:04]	Which Linden:	Hey, Tao, looks like you make it back again!
[11:05]	WarKirby Magojiro:	I recieve money, but the vendors do nothing
[11:05]	Tao Takashi:	yes, this time no login failure before getting here  ;-)
[11:06]	Which Linden:	Yeah, we really shouldn't be relying on lsl for transactions
[11:06]	WarKirby Magojiro:	hmm
[11:06]	WarKirby Magojiro:	standard transactions are very limited, though
[11:06]	Which Linden:	However, there's kind of a turtles-all-the-way down problem in that, if you're using lsl to initiate a transaction, that initiation can fail
[11:07]	Which Linden:	Hell, even the viewer can drop a "buy" packet on the floor
[11:07]	WarKirby Magojiro:	can't send records to external databases, for example
[11:07]	WarKirby Magojiro:	I see
[11:07]	WarKirby Magojiro:	hmm
[11:07]	Which Linden:	that's why I think the "in progress" thing is so important -- it can tell you what the server thinks you did
[11:07]	WarKirby Magojiro nods
[11:08]	Which Linden:	That's for the glorious future, where everything works and is beautiful
[11:08]	Which Linden:	:-)
[11:08]	Which Linden:	So, topics for today
[11:08]	WarKirby Magojiro:	sounds wonderful
[11:09]	Which Linden:	We decided that we're going to go with "shuffle" for the escrow for now, and we'll handle the group payout problem in some other wya
[11:10]	Tao Takashi:	just looking at the code
[11:10]	Which Linden:	I haven't yet put up last week's transcript, since my main computer is in the shot
[11:10]	Which Linden:	shop
[11:10]	Tao Takashi:	ah, you should buy it then finally  ;-)
[11:10]	Which Linden:	But it was a pretty productive discussion about the problem, and ways to cram it into shuffle.
[11:10]	Which Linden:	Hah
[11:11]	Which Linden:	So, um, I feel like we should talk about external apis
[11:11]	Tao Takashi:	I need one for uploading images!  ;-)
[11:11]	Which Linden:	I think I've nailed down the escrow initiation api, but that will probably be internal-only
[11:11]	Which Linden:	...for a while at least
[11:12]	Which Linden:	Probably the first escrow-related api will be to gather and distribute objects as part of a transaction....
[11:13]	Tao Takashi:	so from looking at the test code of the escrow I have no idea what you are doing there  ;-)
[11:13]	Which Linden:	ha ha ha, yeah, we're gonna do a code cleanup pass on that today
[11:13]	Which Linden:	It's a mess
[11:13]	Tao Takashi:	it might help to write doc strings as soon as you write down the function  ;-)
[11:13]	Lom Hax:	What do I need to do to get a look at the source?
[11:13]	Tao Takashi:	http://svn.secondlife.com/trac/certified_http/browser/trunk/certified_http
[11:14]	Tao Takashi:	this URL
[11:14]	Lom Hax loads
[11:14]	Tao Takashi:	and then escrow_test.py
[11:14]	Tao Takashi:	which is what I am looking at
[11:14]	Which Linden:	The agentdb tests are completely messed up because they have three layers:
[11:14]	Which Linden:	the local-only tests, which make sure the agentdb code does the right thing
[11:15]	Which Linden:	the arms-length tests which just call the local functions via chttp
[11:15]	Which Linden:	and the two-arms-length tests which call an escrow-like thing that does an additional layer of indirection
[11:16]	Which Linden:	and there's versions of all these things that are a) sequential, b) sequential with faults and c) concurrent with faults
[11:16]	Which Linden:	It's actually amazing that the code is so small
[11:16]	Tao Takashi:	maybe 3 test classes or so would make sense then
[11:17]	Tao Takashi:	with one test setup superclass or so
[11:17]	Tao Takashi:	(again a reason why I like to put these things into a separate tests directory)
[11:18]	Which Linden:	We do already have multiple classes in agentdb_test.py, but yeah, it is just OneLongFile
[11:18]	Which Linden:	Wanna hear about the bug we found in saranwrap?
[11:18]	Tao Takashi:	what again was saranwrap?  :)
[11:19]	Tao Takashi:	was it that database stuff?
[11:19]	Which Linden:	Saranwrap is more general, but yeah, we're basically using it for database stuff
[11:19]	Which Linden:	It throws an entire object graph into a separate process.
[11:19]	Saijanai Dagger:	I'm a wanna be bot don't mind me
[11:19]	Which Linden:	Whoa, is this clones day, Sai?
[11:19]	Which Linden:	Hah
[11:20]	Saijanai Kuhn:	OK, now that IS a bot...
[11:20]	Saijanai Kuhn:	evoked via python
[11:20]	Which Linden:	Oh, neat. What's it do?
[11:21]	Saijanai Kuhn:	sstands there loking stupid
[11:21]	Saijanai Kuhn:	just got it to work
[11:21]	Saijanai Kuhn:	right now
[11:21]	Tao Takashi:	define objects, Python objects?
[11:21]	Saijanai Kuhn:	one more packet beyond the script I have up on the wiki
[11:22]	Saijanai Kuhn:	since I"m not responding, the server wil log me out soon
[11:22]	Which Linden:	There it goes
[11:22]	Saijanai Kuhn:	er sooner
[11:22]	Which Linden:	So, saranwrap can either wrap a module or a particular object
[11:22]	Which Linden:	And basically every object in the "client" process gets a proxy object on the "parent" side
[11:23]	Tao Takashi gives up on the chttp code and waits for the docstrings to come along so he at least know what those classes are supposed to do  ;-)
[11:23]	Which Linden:	The oplog code is quite well-commented if you're looking for a place to start
[11:24]	Which Linden:	Each operation on a saranwrap proxy object passes messages over the pipe to the client process
[11:24]	Which Linden:	Telling it what to do, in essence
[11:24]	Tao Takashi:	ah ok, thanks
[11:24]	Which Linden:	Perhaps we should have called it "puppeteer" instead
[11:25]	Which Linden:	The whole point is to get eventlet to cooperatively yield the current coroutine when you've told the remote object to do something, so you can do work while the remote object blocks
[11:25]	Which Linden:	It could also be used to distribute computationally intensive tasks around. But you're probably better off using pyro for stuff like that
[11:26]	Which Linden:	Ah, OI Public, the most wretched hive of scum and villainy in Second Life
[11:26]	Which Linden:	:-)
[11:26]	WarKirby Magojiro:	>.> ?
[11:27]	Tao Takashi:	I think I am not deep enough into that coroutines/eventlet stuff
[11:27]	Which Linden:	You don't have to be -- think of them as threads that cannot be preempte
[11:27]	Which Linden:	preempted
[11:28]	Saijanai Kuhn:	separate process but living in the same adress space?
[11:28]	Which Linden:	yeah, basically
[11:28]	Which Linden:	well, that's what threads are
[11:29]	Saijanai Kuhn:	heavier-weight threads?
[11:29]	Which Linden:	coroutines are special-er (and lighter-weight) since they can make the assumption that they will not be interrupted
[11:29]	Saijanai Kuhn:	ah, ok.
[11:29]	Saijanai Kuhn:	so no blocking/nonblocking code...
[11:30]	Saijanai Kuhn:	would they be for event loops and the like?
[11:30]	Tao Takashi:	ok
[11:30]	Which Linden:	Yeah, like httpd.server has an accept loop\
[11:30]	Which Linden:	That then farms out connections to a pool of coroutines
[11:31]	aliceinwire Bleac:	hello
[11:31]	Which Linden:	good day!
[11:31]	WarKirby Magojiro waves
[11:31]	Lom Hax nods
[11:31]	Which Linden:	Here's the event loop in httpd: https://svn.secondlife.com/trac/eventlet/browser/trunk/eventlet/httpd.py#L507
[11:32]	aliceinwire Bleac:	hello warkirby  :)
[11:32]	Tao Takashi:	http://en.wikipedia.org/wiki/Coroutine
[11:32]	Which Linden:	that pool.execute_async call in there transfers control over to another coroutine
[11:33]	Which Linden:	What eventlet does to make coroutine programming easier is that it has a main, scheduler coroutine, and whenever you do I/O, it switches to that main coroutine
[11:34]	Which Linden:	So, for example, if you're doing httpc.get(url), as soon as you have to start waiting for the remote host, eventlet switches away from the coroutine that called get() and into another coroutine, to allow it to do some work
[11:35]	Which Linden:	Actually it probably switches a few times, since I think you can block when writing output as well
[11:35]	Which Linden:	Anyhow, MySQLdb is blocking, and it's in C code so we can't get around it easily
[11:36]	Which Linden:	So that's why we stick it in a separate process via saranwrap, and we can use eventlet to cooperatively yield when we're telling the client process what to do
[11:36]	Which Linden:	But, we had code that looked like this: _write_request(); _read_response()
[11:36]	Which Linden:	and these two had to be called in sequence!
[11:37]	Tillie Ariantho:	oop
[11:37]	Which Linden:	But, as you will note, the *entire point* of eventlet is that it goes out and does some other work while I/O is happening
[11:37]	Which Linden:	so, uh, the bug was that you'd call write_request, and eventlet would activate some other coroutine that would also call write_request
[11:38]	Which Linden:	and then the read_responses would be out-of-order
[11:38]	Which Linden:	D'oh
[11:38]	Tao Takashi:	nice  ;)
[11:38]	Which Linden:	took us a day of hair-pulling to figure that one out
[11:39]	Tillie Ariantho:	:-P
[11:39]	Which Linden:	The crappy quick solution is to put a lock around those two lines, so that no other coroutine can call write_request while we're waiting for a response
[11:39]	Which Linden:	But locks are the domain of the threading world, it kinda sucks that we're using them here
[11:40]	Affluent Destiny:	hi guys
[11:40]	Which Linden:	top o the afternoon!
[11:41]	Which Linden:	So, that was the saranwrap bug, I hope I didn't bore you
[11:41]	Which Linden:	Or leave out critical details
[11:41]	Tillie Ariantho:	No, always interesting.  :)
[11:42]	Which Linden:	So, um, what else should we talk about? Or I lecture on?
[11:43]	Lom Hax:	I'm just getting my feet wet, so I'll listen to most anything.
[11:43]	Tao Takashi:	still studying source code  ;-)
[11:43]	Tillie Ariantho:	hum...
[11:44]	Tao Takashi:	like starting with the webcrawler and seeing what's happening
[11:44]	Which Linden:	Yeah, that's pretty nice
[11:45]	Which Linden:	Incidentally, I added another section to the Mulib/Examples page illustrating how to use it as a cgi
[11:45]	Which Linden:	But the wsgi interface is probably more interesting, does anyone have any experience with that?
[11:45]	Tillie Ariantho:	What's wsgi? ,)
[11:45]	Tao Takashi:	only regarding with Pylons
[11:45]	Tao Takashi:	Webserver Standard Gateway Interface, a Python standard
[11:46]	Tao Takashi:	http://www.wsgi.org/wsgi
[11:46]	Which Linden:	I get the impression that it's like FastCGI but Python-specific
[11:46]	Tillie Ariantho:	ah okay.
[11:46]	Tao Takashi:	Ian is quite involved in that
[11:46]	Which Linden:	Ian who?
[11:46]	Tao Takashi:	Bicking
[11:46]	Which Linden:	Mm.
[11:47]	Tao Takashi:	but I found a problem with it when trying to do POST
[11:47]	Tao Takashi:	well, with paste
[11:47]	Tao Takashi:	which builds on that
[11:47]	Tao Takashi:	as apparently you can only use form-encoded posts with it, not e.g. application/xml
[11:47]	Tao Takashi:	which I guess should be possible
[11:47]	Tao Takashi:	(but haven't looked at the standard)
[11:47]	Tao Takashi:	which is my problem right now in the login server code
[11:48]	Which Linden:	oh, hmm... yeah, wsgi I though provided a callback to retrieve the post body full stop
[11:49]	Tao Takashi:	well, paste's request is so clever to feed socket_fp directly into FieldStorage() from the cgi module
[11:49]	Tao Takashi:	which eats everything and as it does not find and form does not return anything
[11:49]	Tao Takashi:	and socket_fp is empty after that
[11:49]	Tillie Ariantho:	uh oh. .)
[11:49]	Tao Takashi:	so it assumes form-encoded stuff
[11:50]	Tao Takashi:	there maybe is the question anyway what should be done with that data. It's maybe not always a good idea to store it in RAM
[11:50]	Which Linden:	too clever by half, that sounds like it would probably deviate from the spec
[11:50]	Tao Takashi:	need to read the spec again  ;-)
[11:50]	Tao Takashi:	but I doubt that it says that POST data always is multipart
[11:52]	Which Linden:	It'd be cool if someone were to try out eventlet.wsgi and report back on how well it works
[11:52]	Tao Takashi:	the good thing abour WSGI is I think that you can plug certain components together
[11:52]	Which Linden:	Looking at it, I'm not even sure how you do it
[11:52]	Tao Takashi:	Pylons is completely based on that
[11:53]	Which Linden:	It actually looks like eventlet.wsgi is a wsgi server
[11:54]	Which Linden:	I thought it would be a client
[11:54]	Which Linden:	Bah
[11:54]	Tao Takashi:	honestly I never looked that deeply into WSGI
[11:54]	Tao Takashi:	I think a web browser is a client  ;-)
[11:54]	Tao Takashi:	regarding WSGI there are mostly apps and some server which uses these apps I think
[11:55]	Which Linden:	Yeah, and the server could be, like apache, or nginx
[11:55]	Tao Takashi:	yes, there is mod_wsgi for apache
[11:55]	Which Linden:	Anyhow, once someone finds that they want to use it, let me or Donovan know
[11:56]	Tao Takashi:	I think I still need to fully grok WSGI
[11:56]	Which Linden:	Another thing I'd love for people to try out is getting db_pool to work with other databases besides mysql
[11:56]	Tao Takashi:	but I hope to sit together in January with a WSGI expert at the Snow Sprint where I also wanted to talk him into doing some AWG work  :-)
[11:56]	Tao Takashi:	LL should send somebody to that sprint  ;-)
[11:56]	Lom Hax:	LL may be at the point where oracle makes sense
[11:57]	Tillie Ariantho makes a note about that wsgi stuff.  :)
[11:57]	Which Linden:	What's the snow sprint?
[11:58]	Tao Takashi:	an annual Plone/Python/Zope sprint in the alps
[11:58]	Which Linden:	Ah, neat
[11:58]	Which Linden searches for a joke about the alps
[11:58]	Tao Takashi:	indeed  :)
[11:58]	Which Linden can't find any
[11:59]	Tao Takashi:	http://www.openplans.org/projects/snow-sprint-2008/project-home
[11:59]	Tao Takashi:	openplans also uses WSGI btw  ;-)
[11:59]	Which Linden:	Ok, I'm getting pulled away here, I gotta sit down and do this code cleanup on certified http.
[11:59]	Tao Takashi:	and Ian Bicking is heavily involved
[12:00]	Which Linden:	I'll give it a look, thanks for the pointer, Tao!
[12:00]	Tao Takashi:	do that  :-) I need to get a coffee  :-)
[12:00]	aliceinwire Bleac:	:)
[12:00]	Which Linden:	Thank you all for coming today!
[12:00]	aliceinwire Bleac:	i gice it a look too
[12:00]	Tao Takashi:	and it would be great to have more LL people at Python conferences, esp. in Germany  :-)
[12:00]	Tao Takashi:	good way to get more people interested in participating
[12:00]	Tillie Ariantho:	Tao: .P
[12:00]	Which Linden:	The next two weeks' office hours are canceled
[12:00]	Tillie Ariantho:	Any around frankfurt? .)
[12:01]	Tillie Ariantho:	Which: Why that! haha. .P
[12:01]	Which Linden:	I'm gonna be drinkin' egg nog, you see
[12:01]	Which Linden:	Requires all my concentration
[12:01]	Tao Takashi:	well, there is at least the python user group cologne. not sure if Frankfurt has one
[12:01]	Which Linden:	:-)
[12:01]	Tillie Ariantho:	Anything special this month... .P
[12:01]	Which Linden:	Catch you all in the new year! Have good holidays!
[12:01]	Tao Takashi:	no idea, it's def. not my birthday