Difference between revisions of "Certified HTTP API"

From Second Life Wiki
Jump to: navigation, search
(Client: More refactoring)
 
(11 intermediate revisions by one other user not shown)
Line 1: Line 1:
= Certified HTTP API =
+
<< Back to [[Certified HTTP Project]]
  
 
This is the ever-evolving api for a Python implementation of [[Chttp | Certified HTTP]].
 
This is the ever-evolving api for a Python implementation of [[Chttp | Certified HTTP]].
Line 5: Line 5:
 
== Client ==
 
== Client ==
  
Th major problem we face on the client is resumption.  If you send a message, then you crash, what makes you not decide to send the same message with a new message-id?  If you planned on sending a message, then taking some actions based on the response, how can you ensure that said actions get carried out on resumption?  In the general sense, this implies that a certified http client must be able to capture and store a [http://en.wikipedia.org/wiki/Continuation continuation].
+
The are two types of messages you'll want to send from the client:
 +
* Fire-and-forget: you don't want to do anything with the response of the message, you just want to make sure the message gets to its destination.
 +
* Resumable: When the response comes, there is some computation that needs to be performed on it, or the message is one in a series of certified messages.
  
There may be a way to capture a continuation automatically, but it seems unlikely that we'll be able to solve that problem in general, much more likely that the developer writing the chttp client will have to write continuation-friendly code.  We need to have an API that allows you to conveniently store and restore the enough application state to serve as a continuation, and to unit test that you actually got all the state you needed!
+
=== Fire and Forget ===
 +
 
 +
This API should be straightforward:
 +
 
 +
certified_http.put(url, body)
 +
 
 +
If the client crashes and resumes, it should not reenter that code path at all.  If it does, then we have no way to detect the duplication.  The certified_http system should retry the message if necessary (discarding the body of the response).
 +
 
 +
=== Resumable ===
 +
 
 +
If you planned on sending a message, then taking some actions based on the response, how can you ensure that said actions get carried out on resumption?  In the general sense, this implies that a certified http client must be able to capture and store a [http://en.wikipedia.org/wiki/Continuation continuation]. 
 +
 
 +
There may be a way to capture a continuation automatically, but it seems unlikely that we'll be able to solve that problem in general, much more likely that the developer writing the chttp client will have to write continuation-friendly code.  We need to have an API that allows you to conveniently store and restore the enough application state to serve as a continuation, and to unit test that you actually captured all the state you needed! What we've chosen to do is called "deterministic replay".
 +
 
 +
The basic idea here is that the resumable section of code is wrapped up in a function.  The function should be written with as few external dependencies as possible, i.e. "pure".  The oplog argument allows you to store away data whose creation is not atomic with the execution of the method.
  
def start_transferring_foo(req)
 
    # might not be necessary if we can inspect up the stack.
 
    with_chttp(this_other_function, req)
 
 
   
 
   
  def this_other_function(req, chttp_context):
+
  def start_transferring_foo(oplog, *args):
    body = req.readbody()
+
   
+
 
     # this context method saves away the non-deterministic thing
 
     # this context method saves away the non-deterministic thing
 
     # during the initial execution, and spits out the saved value on subsequent runs
 
     # during the initial execution, and spits out the saved value on subsequent runs
     local = chttp_context.persist(some_non_deterministic_thing)
+
     local = oplog.persist(some_non_deterministic_thing)
 
      
 
      
     response = chttp_context.post(escrow_url, "start transaction")
+
     response = certified_http.post(oplog, escrow_url, "start transaction")
   
+
    # see if we can make this post method capture a retroactive continuation?
+
    # or at least remove the need for an interstitial function definition
+
 
      
 
      
 
     coupon_code = local + response['random number']
 
     coupon_code = local + response['random number']
 +
 +
# this wraps the worker function with another whose signature is *args,
 +
# and constructs an oplog upon invocation
 +
 +
start_transferring_foo = certified_http.resumable(start_transferring_foo)
 +
 +
# in python 2.5 you would precede start_transferring_foo with @certified_http.resumable
  
 
== Server ==
 
== Server ==
  
The paradigm on the server is actually a little easier to conceptualize, since we already have a handle method on the server that we expect to be called asynchronously.  We still have to be careful about relying on module-level objects and other external contexts.
+
The server is all about resumability.  The [[mulib]] style of programming is to dispatch to a single method that handles the entire request, so it's very natural to layer resumability on top.  We still have to be careful about relying on module-level objects and other external contexts.
  
  class Resource(mu.Resource)
+
  class Endpoint(certified_http.Resource)
 
   
 
   
def handle_foo(self, req):
+
  def handle_foo(self, oplog, req):
    # we're already in a chttp context
+
      # do stuff here!
 +
 
 +
== Persistence ==
 +
 
 +
Here's a little diagram of the oplog and its relationship to code.
 +
 
 +
[[Image:oplog.png]]
 +
 
 +
The oplog is the core concept, but the coordinator for all the oplogs is the Persistence object, of which there will be as many subclasses as we have underlying mechanisms.
 +
 
 +
<pre>
 +
class Persistence(object):
 +
  def __init__(*args):
 +
    # do implementation-specific initialization here
 +
 
 +
  def message(id):
 +
    # retrieves or creates a message with the specified id
 +
 
 +
  def oplogs():
 +
    # returns a list of open oplogs -- i.e. oplogs that haven't completely run through
 +
</pre>
 +
 
 +
 
 +
<pre>
 +
class Oplog(object):
 +
  def do(method, *args, **kwargs):
 +
    # if not resuming, calls the method and stores away the return value
 +
    # if resuming, simply returns the stored return value
 +
 
 +
  def message(*args):
 +
    # special version of do that stores the arguments of the message *before* executing the function
 +
 
 +
</pre>
 +
 
 +
 
 +
<pre>
 +
class Message(object):
 +
  def oplog():
 +
    # returns the associated oplog
 +
 
 +
  # member variable request is the request part of the HTTP message
 +
  request
 +
 
 +
  # response part of the HTTP message
 +
  response
 +
</pre>

Latest revision as of 16:26, 5 December 2007

<< Back to Certified HTTP Project

This is the ever-evolving api for a Python implementation of Certified HTTP.

Client

The are two types of messages you'll want to send from the client:

  • Fire-and-forget: you don't want to do anything with the response of the message, you just want to make sure the message gets to its destination.
  • Resumable: When the response comes, there is some computation that needs to be performed on it, or the message is one in a series of certified messages.

Fire and Forget

This API should be straightforward:

certified_http.put(url, body)

If the client crashes and resumes, it should not reenter that code path at all. If it does, then we have no way to detect the duplication. The certified_http system should retry the message if necessary (discarding the body of the response).

Resumable

If you planned on sending a message, then taking some actions based on the response, how can you ensure that said actions get carried out on resumption? In the general sense, this implies that a certified http client must be able to capture and store a continuation.

There may be a way to capture a continuation automatically, but it seems unlikely that we'll be able to solve that problem in general, much more likely that the developer writing the chttp client will have to write continuation-friendly code. We need to have an API that allows you to conveniently store and restore the enough application state to serve as a continuation, and to unit test that you actually captured all the state you needed! What we've chosen to do is called "deterministic replay".

The basic idea here is that the resumable section of code is wrapped up in a function. The function should be written with as few external dependencies as possible, i.e. "pure". The oplog argument allows you to store away data whose creation is not atomic with the execution of the method.


def start_transferring_foo(oplog, *args):

   # this context method saves away the non-deterministic thing
   # during the initial execution, and spits out the saved value on subsequent runs
   local = oplog.persist(some_non_deterministic_thing)
   
   response = certified_http.post(oplog, escrow_url, "start transaction")
   
   coupon_code = local + response['random number']

# this wraps the worker function with another whose signature is *args,
# and constructs an oplog upon invocation

start_transferring_foo = certified_http.resumable(start_transferring_foo)

# in python 2.5 you would precede start_transferring_foo with @certified_http.resumable

Server

The server is all about resumability. The mulib style of programming is to dispatch to a single method that handles the entire request, so it's very natural to layer resumability on top. We still have to be careful about relying on module-level objects and other external contexts.

class Endpoint(certified_http.Resource)

  def handle_foo(self, oplog, req):
      # do stuff here!

Persistence

Here's a little diagram of the oplog and its relationship to code.

Oplog.png

The oplog is the core concept, but the coordinator for all the oplogs is the Persistence object, of which there will be as many subclasses as we have underlying mechanisms.

class Persistence(object):
  def __init__(*args):
    # do implementation-specific initialization here

  def message(id):
    # retrieves or creates a message with the specified id
  
  def oplogs():
    # returns a list of open oplogs -- i.e. oplogs that haven't completely run through


class Oplog(object):
  def do(method, *args, **kwargs):
    # if not resuming, calls the method and stores away the return value
    # if resuming, simply returns the stored return value

  def message(*args):
    # special version of do that stores the arguments of the message *before* executing the function


class Message(object):
  def oplog():
    # returns the associated oplog

  # member variable request is the request part of the HTTP message
  request

  # response part of the HTTP message
  response