User:Which Linden/Office Hours/2008 Jan 31

From Second Life Wiki
Jump to: navigation, search

Transcript of Which Linden's office hours:

[11:04] Stormbringer Blackflag: hey
[11:04] Which Linden: Hey stormbringer
[11:04] Stormbringer Blackflag: am i late, early or jsut confused?
[11:04] Which Linden: Early, I guess
[11:05] Which Linden: People generally are reminded to come to my office hours
[11:05] Which Linden: Perhaps I should have not started the practice of announcing them, because now people probably rely on the announcement rather than memory
[11:05] Stormbringer Blackflag: heh
[11:06] Stormbringer Blackflag: i have them in my iCal :P
[11:06] Stormbringer Blackflag: so what's on for today?
[11:07] Which Linden: I'd like to continue the discussion from last time
[11:07] Which Linden: About how to do light, or untrusted chttp
[11:07] Which Linden: Hey Zha
[11:08] Zha Ewry: Hey all. Trippled book, but listening
[11:08] Stormbringer Blackflag: hey
[11:09] Which Linden: Ok, so last time we discussed the option of shorter-duration, ah, "leases"
[11:09] Which Linden: WindowS?
[11:09] Zha Ewry: Leases
[11:09] Zha Ewry: This is "how long the server has to keep the tranaaction held?"
[11:09] Which Linden: Yes
[11:09] Zha Ewry nods
[11:10] Zha Ewry: That lowers the cost of being a c-http server?
[11:10] Which Linden: So in effect the client is leasing storage
[11:10] Which Linden: Right now chttp assumes that the client has certain properties
[11:10] Which Linden: Which e.g. the viewer cannot live up to
[11:10] Marty McGettigan: hello
[11:11] Which Linden: Hey Marty
[11:11] Which Linden: For example, it requires that the client send the same request, even after a reboot
[11:12] Which Linden: And also that the client retry for up to 15 days
[11:12] Which Linden: :-)
[11:12] Zha Ewry: Ouch :-)
[11:12] Which Linden: Work for our current use cases, but obviously of limited wider appeal
[11:13] Which Linden: So, last week we discussed a way of negotiating that duration
[11:14] Zha Ewry nods
[11:14] Which Linden: I.e. the server would declare a range of durations that it would support, the client would specify a duration in its request, and if there was a mismatch, the server would reject with a 403 or 406
[11:14] Which Linden: Not so different from content-type negotiation
[11:15] Which Linden: One thing that was left unresolved was the question of discovery
[11:15] Saijanai Kuhn: Have you guys discussed caps bundling oris this another issue?
[11:15] Which Linden: I.e. how does the client figure out a priori what the server's range is?
[11:15] Which Linden: Sai: different issue
[11:15] Zha Ewry nods
[11:15] Saijanai Kuhn: KK sorry to be late
[11:15] Zha Ewry: That needs to be done
[11:16] Which Linden: Sorry I didn't see you arrive
[11:16] Zha Ewry: Is it a get?
[11:16] Zha Ewry: Just get an attribute?
[11:16] Saijanai Kuhn: dropoped in from on high a few secs ago
[11:16] Zha Ewry: or.. an acive negotiation?
[11:16] Zha Ewry: RESTfully, put, and see if the other end returns what you put?
[11:17] Which Linden: I was thinking, what if the server returns its range in a header when it 403s
[11:17] Zha Ewry: Hmm
[11:17] Which Linden: A nd then if you want to check the range you can just send a request with a patently invalid duration?
[11:17] Zha Ewry is leary of side effects
[11:17] Zha Ewry: This isn't quite one
[11:18] Zha Ewry: But. it is awfully close
[11:18] Which Linden: IT seems cheesy, I agree
[11:18] Which Linden: I guess the question is, how often do you really care?
[11:18] Marty McGettigan: i have to leave again. i will have a closer look at chttp from the web first. always eager to learn. see you later.
[11:18] Which Linden: Ok, thanks for stopping by Marty
[11:18] Zha Ewry: The alternative, tho
[11:19] Zha Ewry: Is what? Put an "atribute" resource out that you can get/set?
[11:19] Which Linden: Sai, btw, we're talking about shorter-duration chttp so that e.g. the viewer could speak it without undue server load
[11:19] Which Linden: Zha: Yeah, or use OPTIONS
[11:19] Zha Ewry: REST gets a little wonky. (And, yes, I know, the SOAPy solution is to put i on th port)
[11:20] Zha Ewry: Can you lay out the options approach. (not one I've looked at to be honest)
[11:20] Which Linden: Basically, you make a request to the target url with the http method OPTIONS
[11:20] Zha Ewry nods
[11:20] Which Linden: and it would return some metadata about the chttp node there, including the range
[11:21] Which Linden: I'm leery of overloading a whole method like that though
[11:21] Zha Ewry: Thats a standard 1.1 method, but
[11:21] Zha Ewry: Not for this use
[11:21] Which Linden: OK, yeah, I actually am not that familiar with it
[11:22] Which Linden: "The OPTIONS method represents a request for information about the communication options available on the request/response chain identified by the Request-URI. "
[11:22] Which Linden: Dunno, seems like it would cover this case, but then again it's painfully vague
[11:23] Zha Ewry: right. One also wonders how much support there is for it in the standard packages
[11:23] Zha Ewry: I mean one can always stuff a request onto an HTTP port
[11:23] Zha Ewry: but.. It's relevent
[11:23] Zha Ewry: (The mroe we make people hand code stuff to use it, the less popular, this gets)
[11:23] Which Linden: Mmmm.... good call
[11:23] Saijanai Kuhn: not sure I recall seeing it in the standard python module...
[11:24] Which Linden: I think most http modules I've seen lately accept any arbitrary http method
[11:24] Zha Ewry nods
[11:24] Zha Ewry: True
[11:24] Saijanai Kuhn loves grabbing single lines from other modlues and adding stuff to the basehttp class
[11:24] Zha Ewry: The web 2.0 world pushes us that way
[11:25] Which Linden: Which is ironic in some ways since the whole point of REST is that you're limited to 6
[11:25] Zha Ewry: Soo. Lets assume we deinfe, a way of saying "from 1 to 2,000 minutes"
[11:25] Zha Ewry: Does that happen "per client" or "per service endpoint?"
[11:26] Which Linden: The service endpoint has a range like that
[11:26] Which Linden: (per url, I'd say)
[11:26] Which Linden: And the client sends a header x-cttp-duration: 200 minutes
[11:26] Which Linden: or something like that
[11:26] Which Linden: and then the idea is that the client retries for 200 minutes and the server has to guarantee idempotency for that duration
[11:27] Zha Ewry nods
[11:27] Which Linden: So, if you were talking to clients that you knew were unreliable you would set your server to have a range of 0-1 minute
[11:28] Which Linden: I guess the question is, if the duration is so short, does anything even need to per persisted to disk?
[11:29] Which Linden: I'd say, no
[11:29] Which Linden: But in the general case how can you decide?
[11:29] Which Linden: I.e. if the server has a range of 0-30 minutes, clearly it must persist the 30-minute ones
[11:29] Which Linden: But if it gets a request that's 5 minutes.... is that short enough that it doesn't have to persist?
[11:30] Zha Ewry: How much book keeping are we adding to the server to handle a larger range?
[11:30] Which Linden: It's not a huge amount
[11:30] Zha Ewry: And.. even five minutes, if you're assuming its your source of a log, or lock
[11:30] Zha Ewry: You need to persist it, because, you may be locking a big transaction
[11:30] Which Linden: Rigt
[11:31] Which Linden: But if you know that your crash/restore time exceeds the duration, yuo don't have to persist
[11:32] Which Linden: I guess what I'm getting at is that it
[11:32] Which Linden: it is far better to not have to persist at all
[11:32] Which Linden: persisting for a minute doesn't reduce your load very much below persisting for 30 days
[11:33] Which Linden: in fact it might increase your load cause you're deleting things from disk so frequently, whereas with a 30-day window you can clean hourse during a downtime
[11:33] Which Linden: I guess there's nothing preventing the 1-minute server from cleaning house as infrequently as it wants
[11:33] Which Linden: the duration is just a lower bound
[11:34] Which Linden: So, what other ways can the client behave badly?
[11:35] Which Linden: One thing is that the client could reuse the same message-id for various distinct requests
[11:35] Saijanai Kuhn: in the context of this discussion? or in general...
[11:35] Which Linden: A chttp client, Sai, so I guess in the context of this discussion
[11:35] Saijanai Kuhn: sorry having constant crashes on aditi right now
[11:35] Zha Ewry: Whoops. Zero Calling.
[11:35] Zha Ewry: Yea
[11:35] Zha Ewry: I htink, the question is whether you can trust that window, or not
[11:36] Zha Ewry: If you're asking this to be "reliabel" that's sort of the base
[11:36] Zha Ewry: and. Yeah, the 1 minute vs 30 is interesting
[11:36] Zha Ewry: Tho
[11:36] Zha Ewry: You can talways hold longer
[11:36] Zha Ewry: No penalty for that, is there?
[11:36] Which Linden: Yeah, seems legit
[11:37] Zha Ewry: Ok. Interesting.
[11:37] Zha Ewry: Thinka botu that,f or a oment.
[11:37] Zha Ewry: I need to hop here
[11:37] Which Linden: Cool, thanks for coming!
[11:37] Zha Ewry: Pleasure
[11:37] Zha Ewry: Ths is important stuff
[11:37] Which Linden: I appreciate your input
[11:37] Saijanai Kuhn: later Zha
[11:38] Which Linden: I think we're covered in the case of the client making multiple distinct requests with the same message-id, cause we currently hash the request and reject if the hash doesn't match
[11:39] Which Linden: (which implies that hashing the body is a legit message-id-generation strategy, actually)
[11:39] Saijanai Kuhn: interesting so all proper possible messages are covered? IOW, you only have a very lmited number that you have to worry about?
[11:41] Which Linden: When I say "covered" I mean "protected to some degree from abuse"
[11:41] Which Linden: But actually now that I think about it, just checking the hash is not a completely trivial operation since it has to look it up on disk
[11:41] Saijanai Kuhn: not sure what you mean...
[11:42] Saijanai Kuhn: the message comes in in xml-LLSD format right?
[11:42] Which Linden: Lemme back up
[11:42] Which Linden: So the client sets a message id on each request
[11:42] Which Linden: And the client is required to retry with exactly the same request
[11:43] Saijanai Kuhn: ah ok
[11:43] Which Linden: So, if we were relying on that behavior the client could cause problems by keeping the message id constant andsending a buncha different bodies/headers
[11:43] Which Linden: But we decided to double-check the client in that case, so when a message comes in we store the message id, and also the hash
[11:44] Which Linden: (the hash is computed over all the headers and the body)
[11:44] Which Linden: So if the client tries this little trick the server is all "nuh-uh!"
[11:44] Saijanai Kuhn: ok, so the hash is a temporary thing computed on-the-fly for agiven client's mess[s]
[11:44] Saijanai Kuhn: message[s]
[11:44] Stormbringer Blackflag: you could also cache the hashes so you don't haev to go back to the disk each time a duplicate request comes in
[11:45] Saijanai Kuhn: no some giant dictionary for all possible propermesages
[11:45] Which Linden: It's computed for each message received, but the hash of the first message is stored for as long as the rest of the data pertaining to that message is stored
[11:45] Which Linden: Storm: true
[11:45] Saijanai Kuhn: KK its asneeded
[11:45] Which Linden: Oh, yeah, so after 30 days or 30 minutes or whatever, the hash is discarded
[11:46] Which Linden: along with the rest of the info about the request
[11:46] Which Linden: Hey Taja
[11:46] Taja Beatty: Hello Which....hope you do not mind me dropping in
[11:46] Which Linden: No problem! The morethe merrier!
[11:46] Saijanai Kuhn: Which is explaining things to ME, I'm sure he wont mind someone else asking questions. (ie a little more knowledgeable)
[11:47] Taja Beatty: think though I will be quiet and listen and not interrupt until I understand what the chat is about ha ha
[11:47] Taja Beatty: me more knowledgable?
[11:47] Taja Beatty: doubt that
[11:48] Taja Beatty: but I am always willing to learn
[11:48] Which Linden: I'm just looking over the chttp spec's Requirements section, to see if the client has any onerous requirements
[11:48] Which Linden: Oh, yeah, the clck skew thing
[11:48] Which Linden: clock skew
[11:49] Which Linden: We're assuming that the client and server don't have too much skew
[11:49] Which Linden: But, I can't recall why that's so
[11:49] Which Linden: Ah, ah, I remember
[11:50] Which Linden: We're assuming that the client doesn't change its clock so much that it retries expired messages
[11:51] Which Linden: I.e. it says to itself, "I'll retry this message until 12:15" then it gets hit by a DeLorean and it's 10 am as far as it's concerned
[11:51] Stormbringer Blackflag: well, once a message is expored, the server doesn't have to guarantee idempotency any more, so it can just reject it
[11:51] Which Linden: But then it has to remember that it saw that message before
[11:51] Stormbringer Blackflag: oh, right. hmm.
[11:51] Which Linden: I.e., I've been assuming that after the time expires the server forgets that it ever heard about the message
[11:52] Which Linden: We could require that the server not forget
[11:52] Which Linden: Specifically to deal with this problem
[11:52] Which Linden: Forever
[11:53] Which Linden: We do state in the spec that it's not unreasonable to store small data like this
[11:53] Which Linden: Hah, out-of-order chat.
[11:53] Stormbringer Blackflag: or it could just immediately reject clock-skewed messages, but that might acuse other problems that i haven't thought of yet.
[11:53] Which Linden: We actually do that already
[11:53] Stormbringer Blackflag: oh
[11:53] Which Linden: Well, wait, no
[11:53] Which Linden: Two different skews
[11:54] Which Linden: We currently reject messages wherein the client's idea of the current date differs too much from the server's idea of the current date
[11:54] Which Linden: I don't think there's a good way of detecting when the client has "gone back in time" so to speak
[11:54] Saijanai Kuhn: unlessyou timetsamp every message
[11:54] Which Linden: I guess you could do that
[11:55] Which Linden: And say, "the client thought it was 2pm last time, now it thinks it's 1am"
[11:55] Stormbringer Blackflag: i thnik if you keep taht rejection policy, it solves the problem
[11:56] Stormbringer Blackflag: because if the client sends a request when the clock is off, it gets rejhected. the nif the clock is fixed and they resend it, it gets accepted
[11:56] Which Linden: Yeah, I guess so, if we keep the client-server skew below the duration of the message
[11:56] Stormbringer Blackflag: yeah
[11:56] Saijanai Kuhn: of course a simple increasing ID number would do the same
[11:56] Which Linden: That is a good point
[11:56] Which Linden: Sai: ?
[11:56] Saijanai Kuhn: yo?
[11:56] Saijanai Kuhn: hmm?
[11:57] Saijanai Kuhn: whats up?
[11:58] Which Linden: Why's the increasing id number the same?
[11:58] Which Linden: (sorr,y got distracted irl)
[11:59] Saijanai Kuhn: if you don't use a higher ID number, then you've lost track of where you are in the process
[11:59] Which Linden: Oh I see
[11:59] Saijanai Kuhn: if you DO, it seems one could assume that the client is ware that time has passed...
[12:01] Which Linden: Hmm... but one could envision the client storing the increasing id number somewhere on disk, rebooting with a bad bios battery (and thus getting its clock reset to 1970) and therefore retrying with correcty-increasing ids for 38 years. :-)
[12:01] Saijanai Kuhn: true
[12:01] Which Linden: Ah, shit, the client doesn't even send down its current date
[12:02] Which Linden: It only ever sends down the inception date
[12:02] Which Linden: So we can't even store the client's last date
[12:03] Which Linden: OK, well, this is something that we can probably solve given enough brainpower :-)
[12:03] Which Linden: I have to go right now though
[12:03] Which Linden: So, thanks so much for coming, Sai and Storm
[12:04] Which Linden: This was a fruitful discussion