PyOGP Client Library

PyOGP

• Roadmap
• PyOGP Client Library:
  • Coding Guidelines
  • Package Development
  • Client Development Sandbox
  • Package Unittests

Intro

Pyogp is an young open source python client library akin to libopenmetaverse (nee libsl.) Hosted on svn.secondlife.com, it does require a contributor's agreement for commit access, and currently has a few contributors from the Second Life community.

We'll likely be moving to hg relatively soon...

Conceived as a mechanism for testing OGP grid changes, it carries on with a charter of enabling automated testing of Second Life grids, enabling exploration into the client/server relationship of Second Life, and enabling prototyping of various client applications.

• mailing list: pyogp@lists.secondlife.com
• irc: irc://irc.freenode.com/#pyogp
• wiki: http://wiki.secondlife.com/wiki/Pyogp
• licensing: Apache 2.0 license, copyright Linden Lab

Goals

In the very near future, we can have tests available to be run as soon as a deploy is completed that exercise a simulator/grid in the same way we do a smoke test. We will use these as automated tests run at build time, post deploy validation, and regression testing of simulators and backend systems.

This provides early feedback on code quality. QA is then able to dive deeper in testing the changes specific to a branch.

Having this library available also allows us to test potential changes before we have finalized design and are ready to submit to QA. Not sure how something will play out? Try it, and test it with PyOGP....

Structure

PyOGP is comprised of two python packages.

pyogp.lib.base - consists of basic networking, messaging systems (UDP and event queue) and capabilities, custom datatypes, and a low level message related event system

pyogp.lib.client - consists of 'convenience' classes mapping methods and handlers to specific messages (e.g. Agent().say(msg) tells the client to send the ChatFromViewer message to the host region). Raises application oriented events based on processing of messages (these are currently sparsely implemented).

Dependencies

The packages that make up PyOGP have some dependencies on python modules not included in a standard install, or sometimes not available on an older Python distribution.

PyOGP is known to work well in Python 2.4.x and 2.5.x. Known issues are preventing successful use on Python 2.6.

pyogp.lib.base dependencies:

 from setup.py
 
    install_requires=[
        'setuptools',
        # -*- Extra requirements: -*-
        'uuid',
        'elementtree',
        'indra.base',
        'WebOb',
        'wsgiref',
        'eventlet==0.8.14'
        'eventlet==0.8.14'

pyogp.lib.client dependencies:

    install_requires=[
        'setuptools',
        # -*- Extra requirements: -*-
        'pyogp.lib.base'

pyogp.apps dependencies:

    install_requires=[
        'setuptools',
        'pyogp.lib.client'
        ],

PyOGP aims to be compatible across platforms, though there are known problems with various environments. PyOGP runs well on Windows XP with Python 2.5.x, on Mac with Python 2.5.x, and on Linden hosts running Python 2.4.4. Problems have been observed on Windows Vista and 7 (due to Python 2.6 and or pyopenssl related problems). We'll be focusing on ensuring better compatibility soon.

Current functional coverage

Anything not listed as covered is probably not yet covered.

pyogp.lib.base:

• base udp messaging system (message.*)
  • UDP serialization/deserialization
  • message_template.msg parsing
• base event queue messaging system (event_queue.EventQueueClient())
• capabilities and their methods. Seed capabilities are a special case. (caps.Capability())
• Message-based events (message.message_handler)

pyogp.lib.client:

• agents (agent.Agent())
  • L$ balance request, friending, and walk/fly/sit/stand actions...
• OGP agentdomain (agentdomain.AgentDomain())
• application level events (event_system.AppEventsHandler())
• some object handling (objects.*)
  • edit name, description, next-owner permissions and more
  • object creation is possible
• some inventory handling (inventory.*)
  • login inv skeletons
  • fetching inventory, including AIS (caps based Agent Inventory Services))
  • some creating of new inventory items (LSL scripts, notecards)
• regions (region.Region())
  • host and neighboring regions are handled slightly differently
  • udp and event queue connections are optionally enabled for each case
  • currently only pulling caps available to the agent via the seed cap (plus using the inventory related caps in the AIS context)
• some appearance handling (appearance.AppearanceManager),
• parcels
• chat
• some ImprovedInstantMessage handling
  • raises events on received instant messages
  • can deal with inventory offers/accepts/declines
  • other cases in this message are currently only logged
• groups
• group chat
• LSL script uploading

How to install and use

Standalone dev environment using buildout

On a desktop, one can checkout a dev environment for working with pyogp. Currently, buildout is used to configure the environment. One may optionally use a virtualenv python env.

Dependencies: buildout takes care of everything, grabbing needed modules etc.

Wiki instructions: https://wiki.secondlife.com/wiki/Pyogp/Client_Lib/The_Development_Sandbox

Using only pyogp.lib.base (aka the Linden context)

Updated info available via Enus as needed, will be updating docs asap.

Sample Scripts

There are a variety of scripted examples that have been used to exercise functionality as it is added to the library. These persist as coded documentation.

Note:

Sample scripts are brought into the buildout as an external package, pyogp.apps, and are executable in buildout's bin/ directory. The source code is available in https://svn.secondlife.com/svn/linden/projects/2008/pyogp/pyogp.apps/trunk/pyogp/apps/examples/ .

"python {script} -h" will display {meager} usage for each script.

bin/AIS_inventory_handling

bin/agent_login

bin/agent_manager

bin/appearance_management

bin/chat_and_instant_messaging

bin/group_chat

bin/group_creation

bin/inventory_handling

bin/inventory_transfer

bin/inventory_transfer_specify_agent

bin/login

bin/multi_region_connect

bin/object_create_edit

bin/object_create_permissions

bin/example/object_creation

bin/example/object_properties

bin/example/object_tracking

bin/example/region_connect

etc

Agent Login

Single Agent Login

non-eventlet context: many test cases may be written this way. the main script process is blocking and will terminate the client when completed.

<python>from pyogp.lib.client.agent import Agent

client = Agent()

client.login(options.loginuri, args[0], args[1], password, start_location = options.region</python>

Eventlet context: spawn a client in a co-routine, allowing persistent presence until forcefully terminated.

<python>from eventlet import api

from pyogp.lib.client.agent import Agent from pyogp.lib.client.settings import Settings

settings = Settings()

settings.ENABLE_INVENTORY_MANAGEMENT = True settings.MULTIPLE_SIM_CONNECTIONS = False

client = Agent(settings = settings)

api.spawn(client.login, options.loginuri, 'first', 'last', 'password', start_location = options.region)

1. wait for the agent to connect to it's region

while client.connected == False:

   api.sleep(0)

while client.region.connected == False:

   api.sleep(0)

1. once connected, live until someone kills me

while client.running:

   api.sleep(0)</python>

Multiple Agent Login

Eventlet and non-eventlet spawning methods apply to the agent manager in the same way as they apply to single agent usage. Each agent instance in logged in in a separate coroutine.

<python>from pyogp.lib.client.agent import Agent from pyogp.lib.client.agentmanager import AgentManager from pyogp.lib.client.settings import Settings

settings = Settings()

params = [['agent1', 'lastname', 'password'], ['agent2', 'lastname', 'password']] agents = []

1. prime the Agent instances

for params in clients:

   agents.append(Agent(settings, params[0], params[1], params[2]))

agentmanager = AgentManager() agentmanager.initialize(agents)

1. log them in

for key in agentmanager.agents:

   agentmanager.login(key, options.loginuri, options.region)

1. while they are connected, stay alive

while agentmanager.has_agents_running():

   api.sleep(0)</python>

Events & Callbacks

The event implementation in pyogp follows the observer pattern, where observers subscribe to and are notified when an event occurs. Data is passed throughout the client instance via events.

There are 2 types of internal event systems:

• MessageManager - is an attribute of a Region and every packet received/sent is filtered through here. subscriptions are by message name
  • MessageHandler - is an attribute of MessageManager, and every categorized message received from the event queue or udp dispatcher is filtered through here. (message as defined in message_template.msg, or one of ['ChatterBoxInvitation', 'ChatterBoxSessionEventReply', 'ChatterBoxSessionAgentListUpdates', 'ChatterBoxSessionStartReply', 'EstablishAgentCommunication']. There may be unhandled messages, I just haven't seen em yet :))
• EventsHandler - an attribute of an Agent, also able to be passed in, that is intended as the primary interface of a pyogp application into the internal state and data events within the lib.

Message Events

In the most fundamental implementation of event usage, all packets are passed through a MessageManager() instance for evaluation. Observers may register to receive udp packets serialized into the form of Message() instances. The MessageHandler() is a consolidation point for subscribing to messages keyed by message name, and created on demand via subscription.

See pyogp.lib.base.message.message_handler.MessageHandler() for more details.

The pyogp agent's Region() instances each monitor their stream of packets (e.g. the host region: agent.region.message_manager.message_handler). (Perhaps this should be changed to a generalized Network() class where all packets (coupled to their originating regions) are evaluated.

Event firing passes data on to a callback handler defined in the subscription, in the form of (handler, *args, **kwargs).

The Agent class monitors the ImprovedInstantMessage packet: <python>... onImprovedInstantMessage_received = self.region.message_handler.register('ImprovedInstantMessage') onImprovedInstantMessage_received.subscribe(self.onImprovedInstantMessage) ...

def onImprovedInstantMessage(self, packet):

   """ handles the many cases of data being passed in this message """

   {code} # parse and handle the data...

</python>

The messaging system then fires the event when an ImprovedInstantMessage message is received which calls onImprovedInstantMessage method above to handle the message

Unsubscribing from an event: <python> onImprovedInstantMessage_received.unsubscribe(self.onImprovedInstantMessage) </python>

Event System Events

New! And not widely used yet...

The application level event system is a separate implementation in pyogp.lib.base.event_system.EventsHandler(). This allows for a timeout to be specified for the subscription to a particular event.

The api for subscribing to these events is similar to the MessageHandler() examples above, with an additional timeout parameter passed in the _register() method. When the specified timeout expires, the subscription returns None and expires the subscription.

The first use of this can be seen with the InstantMessageReceived() implementation as handled in Agent().onImprovedInstantMessage(), and usage of such in the sample script in the dev buildout bin/chat_and_instant_messaging.

Logging

Uses python's standard logging module (http://docs.python.org/library/logging.html). The library defines logging events throughout, it is up to the application/script to determine the output.

Hooking logging into a new module:

<python>from logging import getLogger

1. initialize logging

logger = getLogger('pyogp.lib.base.agent')

class Agent(object):

   """ our agent class """

   def __init__(self, params):

       self.params = params

       logger.debug("Initializing agent with params: %s" % (params))</python>

An application can then set up the logging output as follows (or any other way it pleases):

<python>console = logging.StreamHandler() formatter = logging.Formatter('%(asctime)-30s%(name)-30s: %(levelname)-8s %(message)s') console.setFormatter(formatter) logging.getLogger().addHandler(console) logging.getLogger().setLevel(logging.DEBUG)</python>

The output to console is then:

<bash>2009-04-21 22:08:58,681 pyogp.lib.base.agent  : DEBUG agent with params: params</bash>

Pyogp Unit Tests

See PyOGP_Package_Unittests.

Writing Test Cases

Tests can be written using standard unittest. The tests in pyogp.interop cover some ogp and a couple of legacy cases, these need to be updated to work.

Testing only the call to login.cgi is unique, we don't need to spawn the client in a coroutine, nor do we need to keep the client alive, we just need to post to the login endpoint and evaluate the response.

pyogp.interop.tests.test_legacy_login

<python>import unittest, doctest import ConfigParser from pkg_resources import resource_stream import time import uuid import pprint

from pyogp.lib.base.agent import Agent from pyogp.lib.base.datatypes import UUID from pyogp.lib.base.exc import LoginError from pyogp.lib.base.settings import Settings

import helpers

class AuthLegacyLoginTest(unittest.TestCase):

   def setUp(self):
       
       # initialize the config
       self.config = ConfigParser.ConfigParser()
       self.config.readfp(resource_stream(__name__, 'testconfig.cfg'))
               
       self.test_setup_config_name = 'test_interop_account'
       
       self.firstname = self.config.get(self.test_setup_config_name, 'firstname')
       self.lastname = self.config.get(self.test_setup_config_name, 'lastname')
       self.password = self.config.get(self.test_setup_config_name, 'password')
       self.agent_id = self.config.get(self.test_setup_config_name, 'agent_id')
       self.login_uri = self.config.get(self.test_setup_config_name, 'login_uri')
       self.region = self.config.get('test_interop_regions', 'start_region_uri') 

       self.successful_login_reponse_params = ['last_name', 'sim_ip', 'inventory-lib-root',
         'start_location', 'inventory-lib-owner', 'udp_blacklist', 'home', 'message', 
         'agent_access_max', 'first_name', 'agent_region_access', 'circuit_code', 'sim_port',
         'seconds_since_epoch', 'secure_session_id', 'look_at',  'ao_transition', 'agent_id',
         'inventory_host', 'region_y', 'region_x', 'seed_capability', 'agent_access', 'session_id',
         'login']

       self.settings = Settings()
       self.settings.MULTIPLE_SIM_CONNECTIONS = False

       self.client = Agent(self.settings, self.firstname, self.lastname, self.password)

   def tearDown(self):
       
       if self.client.connected:
           self.client.logout()
       
   def test_base_login(self):
       """ login with an account which should just work """

       self.client.settings.ENABLE_INVENTORY_MANAGEMENT = False

       self.client.login(loginuri = self.login_uri, start_location = self.region, connect_region = False)

       # Make sure that the login response attributes propagate properly, 
       # Make sure the login against a grid has worked
       self.assertEquals(self.client.grid_type, 
                         'Legacy', 
                         'Storing the wrong grid type based on a \'legacy\' login request')
       self.assertEquals(self.client.firstname, self.firstname)
       self.assertEquals(self.client.lastname, self.lastname)
       self.assertEquals(self.client.lastname, self.lastname)
       self.assertEquals(self.client.name, self.firstname + ' ' + self.lastname)
       self.assertEquals(self.client.connected, True)
       self.assertNotEquals(self.client.agent_id, None)
       self.assertEquals(str(self.client.agent_id), self.agent_id)
       self.assertNotEquals(self.client.session_id, None)
       self.assertNotEquals(self.client.secure_session_id, None)

       self.assertEquals(self.client.login_response['last_name'], self.lastname)
       self.assertEquals(self.client.login_response['first_name'], '"' + self.firstname + '"')
       self.assertEquals(self.client.login_response['login'], 'true')
       self.assertEquals(self.client.login_response['secure_session_id'], str(self.client.secure_session_id))
       self.assertEquals(self.client.login_response['session_id'], str(self.client.session_id))
       self.assertEquals(self.client.login_response['agent_id'], str(self.client.agent_id))
       self.assertNotEquals(self.client.login_response['seed_capability'], )

       fail = 0
       fail_extra = 0
       fail_missing = 0
       extra_keys = 
       missing_keys = 
       
       for key in self.client.login_response:
           try:
               self.successful_login_reponse_params.index(key) # if the key is in our valid list, sweet
           except:
               fail_extra = 1
               extra_keys = extra_keys + ' ' + key

       for key in self.client.login_response:
           try:
               self.successful_login_reponse_params.index(key) # if the key is in our valid list, sweet
           except:
               fail_missing = 1
               missing_keys = missing_keys + ' ' + key
      
       self.assertEquals(fail_extra, 0, 'login response has additional keys: ' + extra_keys)
       self.assertEquals(fail_missing, 0, 'login response is missing keys: ' + missing_keys)

   def test_login_with_bad_password(self):

       self.client.settings.ENABLE_INVENTORY_MANAGEMENT = False

       self.assertRaises(LoginError, 
                         self.client.login, 
                         loginuri = self.login_uri, 
                         password = 'BadPassword', 
                         start_location = self.region, 
                         connect_region = False)

def test_suite():

   from unittest import TestSuite, makeSuite
   suite = TestSuite()
   suite.addTest(makeSuite(AuthLegacyLoginTest))
   return suite</python>

Adding Functionality

Wrapping Packets

PyOGP, like the Viewer, communicates with the Second Life simulator by sending messages over UDP.

In order to extend PyOGP, you'll enable a new UDP message.

Example: Sending an IM

To send an IM to the simulator, send an ImprovedInstantMessage packet. The base class for message packets is defined in base/message/message.py

Packets are assembled using a Message() instance which has the message name and Block() instances passed in through its constructor. Similarly, Blocks are assembled by passing in the Block name and the value name and values for each of the Block values. Note: It is important that the message name, block name, and value names and types should match what is specified in the message template.

Example: <python> def send_ImprovedInstantMessage(self, AgentID = None, SessionID = None,

                               FromGroup = None, ToAgentID = None, 
                               ParentEstateID = None, RegionID = None, 
                               Position = None, Offline = None, 
                               Dialog = None, _ID = None, Timestamp = None, 
                               FromAgentName = None, _Message = None, 
                               BinaryBucket = None):
       """ 
       sends an instant message packet to ToAgentID. this is a 
       multi-purpose message for inventory offer handling, im, group chat, 
       and more 
       """</python>

Assemble the message: <python>

       packet = Message('ImprovedInstantMessage', 
                        Block('AgentData', 
                              AgentID = AgentID, 
                              SessionID = SessionID), 
                        Block('MessageBlock', 
                              FromGroup = FromGroup, 
                              ToAgentID = ToAgentID, 
                              ParentEstateID = ParentEstateID, 
                              RegionID = RegionID, 
                              Position = Position, 
                              Offline = Offline, 
                              Dialog = Dialog, 
                              ID = UUID(str(_ID)), 
                              Timestamp = Timestamp, 
                              FromAgentName = FromAgentName, 
                              Message = _Message, 
                              BinaryBucket = BinaryBucket))</python>

Send the message: <python>

       self.region.enqueue_message(packet, True)</python>

Sphinx (api docs)

Api documentation is now available for pyogp packages (well, not for apps yet, but someday....)!

In pyogp/docs in each of the pyogp.lib.base and pyogp.lib.client packages, one will find source, last revision, and build files for sphinx based documents.

Sample output is available at {package root}/docs/html/index.html.

README.txt contains build instructions, as follows:

This checkout contains the most recently complied version of the documentation in docs/html/.

To rebuild the sphinx doc set:

Get sphinx!!!

Either use your virtualenv, or your native python install and run: 
    easy_install -U Sphinx

Then, from the docs dir:

1. python refresh.py

refresh.py stages the sphinx .rst files, and then runs 'sphinx-build -a -c source/configure/' source/ html/

The docs/html/ directory will contain the fully compiled documentation set.
Please check in updated docs if you add functionality.

Roadmap

There is so much yet to implement that it is frightening. Here's what's up in the near term for pyogp:

• enabling parcel testing
• teleport (in OGP and in the 'legacy' context)
  • teleport works in OGP, but is not encapsulated in a method yet
• permissions system testingto save QA from 3-5 day regression passes on perms
• appearance - this will require enabling upload and download, plus baking. Anyone have some spare time? :)