PyOGP Client Library

PyOGP

• Roadmap
• PyOGP Client Library:
  • Coding Guidelines
  • Package Development
  • Client Development Sandbox
  • Package Unittests

Intro

Pyogp is an young open source python client library akin to libopenmetaverse (nee libsl.) Hosted on svn.secondlife.com, it does require a contributor's agreement for commit access, and currently has a few contributors from the Second Life community.

Conceived as a mechanism for testing OGP grid changes, it carries on with a charter of enabling testing of Second Life grids.

• mailing list: pyogp@lists.secondlife.com
• irc: irc://irc.freenode.com/#pyogp
• wiki: http://wiki.secondlife.com/wiki/Pyogp
• licensing: Apache 2.0 license, copyright Linden Lab

Goals

Why?

Because.

In the very near future, we can have tests available to be run as soon as a deploy is completed that exercise a simulator/grid in the same way we do a smoke test. Perhaps we run these tests as a post deploy step.

This provides early feedback on code quality. QA is then able to dive deeper in testing the changes specific to a branch. Perhaps pyogp has been revved (in a side branch?) and knows about the changes the linden branch was made, and can test those directly with new test suites as well. The possibilities are endless.

Having this library available also allows us to test potential changes before we have finalized design and are ready to submit to QA. Not sure how something will play out? Try it, and test it with pyogp...

Current functional coverage

Anything not listed as covered is probably not yet covered.

Pyogp knows about (modules in parenthesis are prefaced with pyogp.lib.base. in practice):

• agents (agent.Agent())
• OGP agentdomain (agentdomain.AgentDomain())
• base udp messaging system (message.*)
• base event queue messaging system (event_queue.EventQueueClient())
• capabilities and their methods. Seed capabilities are a special case. (caps.Capability())
  • currently only pulling caps available to the agent via the seed cap (plus using the inventory related caps in the AIS context)
• internal event systems
  • packets (message.packethandler.PacketHandler())
  • event queue (event_queue.EventQueueHandler()
  • application level - new! (event_system.EventsHandler())
• some object handling (objects.*)
  • name, description, next-owner permissions and more
• some inventory handling (inventory.*)
  • login inv skeletons
  • fetching inventory, including AIS (caps based Agent Inventory Services))
• regions (region.Region())
  • host and neighboring regions are handled slightly differently)
  • udp and eq connections are optionally enabled for each case

Pyogp can also do a little bit of:

• chat
• some ImprovedInstantMessage handling
  • raises events on received instant messages
  • can deal with inventory offers/accepts/declines
  • other cases in this message are currently only logged
• groups
• group chat

How to install and use

Standalone dev environment

On a desktop, one can checkout a dev environment for working with pyogp. Currently, buildout is used to configure the environment. One may optionally use a virtualenv python env.

svn co https://svn.secondlife.com/svn/linden/projects/2008/pyogp/buildouts/libdev/trunk

Dependencies: buildout takes care of everything, grabbing needed modules etc.

Wiki instructions: https://wiki.secondlife.com/wiki/Pyogp/Client_Lib/The_Development_Sandbox

Sample Scripts

There are a variety of scripted examples that have been used to exercise functionality as it is added to the library. These persist as coded documentation.

{script} -h will display {meager} usage for each script.

pyogp/lib/base/examples/sample_AIS_inventory_handling.py

pyogp/lib/base/examples/sample_agent_login.py

pyogp/lib/base/examples/sample_agent_manager.py

pyogp/lib/base/examples/sample_appearance_management.py

pyogp/lib/base/examples/sample_chat_and_instant_messaging.py

pyogp/lib/base/examples/sample_group_chat.py

pyogp/lib/base/examples/sample_group_creation.py

pyogp/lib/base/examples/sample_inventory_handling.py

pyogp/lib/base/examples/sample_inventory_transfer.py

pyogp/lib/base/examples/sample_inventory_transfer_specify_agent.py

pyogp/lib/base/examples/sample_login.py

pyogp/lib/base/examples/sample_multi_region_connect.py

pyogp/lib/base/examples/sample_object_create_edit.py

pyogp/lib/base/examples/sample_object_create_permissions.py

pyogp/lib/base/examples/sample_object_creation.py

pyogp/lib/base/examples/sample_object_properties.py

pyogp/lib/base/examples/sample_object_tracking.py

pyogp/lib/base/examples/sample_region_connect.py

Agent Login

Single Agent Login

non-eventlet context: many test cases may be written this way. the main script process is blocking and will terminate the client when completed.

    from pyogp.lib.base.agent import Agent

    client = Agent()

    client.login(options.loginuri, args[0], args[1], password, start_location = options.region)

eventlet context: spawn a client in a coroutine, allowing persistent presence until forcefully terminated.

    from eventlet import api

    from pyogp.lib.base.agent import Agent
    from pyogp.lib.base.settings import Settings

    settings = Settings()

    settings.ENABLE_INVENTORY_MANAGEMENT = True
    settings.MULTIPLE_SIM_CONNECTIONS = False

    client = Agent(settings = settings)

    api.spawn(client.login, options.loginuri, 'first', 'last', 'password', start_location = options.region)

    # wait for the agent to connect to it's region
    while client.connected == False:
        api.sleep(0)

    while client.region.connected == False:
        api.sleep(0)

    # once connected, live until someone kills me
    while client.running:
        api.sleep(0)

Multiple Agent Login

Eventlet and non-eventlet spawning methods apply to the agent manager in the same way as they apply to single agent usage. Each agent instance in logged in in a separate coroutine.

    from pyogp.lib.base.agent import Agent
    from pyogp.lib.base.agentmanager import AgentManager
    from pyogp.lib.base.settings import Settings

    settings = Settings()

    params = [['agent1', 'lastname', 'password'], ['agent2', 'lastname', 'password']]
    agents = []

    # prime the Agent instances
    for params in clients:

        agents.append(Agent(settings, params[0], params[1], params[2]))

    agentmanager = AgentManager()
    agentmanager.initialize(agents)

    # log them in
    for key in agentmanager.agents:
        agentmanager.login(key, options.loginuri, options.region)

    # while they are connected, stay alive
    while agentmanager.has_agents_running():
        api.sleep(0)

Events & Callbacks

The event implementation in pyogp follows the observer pattern, where observers subscribe to and are notified when an event occurs. Data is passed throughout the client instance via events.

There are 3 types of internal event systems (overkill? perhaps. at least the PacketHandler and EventQueueHandler can be coalesced into a single class at some point in the future).

• PacketHandler - is an attribute of a Region and every packet received/sent is filtered through here. subscriptions are by packet (message) name
• EventQueueHandler - is an attribute of the Agent, and every categorized message received from the event queue is filtered through here. (message as defined in message_template.msg, or one of ['ChatterBoxInvitation', 'ChatterBoxSessionEventReply', 'ChatterBoxSessionAgentListUpdates', 'ChatterBoxSessionStartReply', 'EstablishAgentCommunication']. there may be unhandled messages, I just haven't seen em yet :))
• EventsHandler - an attribute of an Agent, also able to be passed in, that is intended as the primary interface of a pyogp application into the internal state and data events within the lib.

Message Events

In the most fundamental implementation of event usage, all packets are passed through a packet handler for evaluation. Observers may register to receive udp packets serialized into the form of UDPPacket() instances. The PacketHandler() is a consolidation point for subscribing to PacketReceivedNotifier() instances keyed by message message name, and created on demand via subscription.

See pyogp.lib.base.message.packethandler.PacketHandler() for more details. Event queue messages are treated similarly, through a separate consolidation point (EventQueueHandler()).

The pyogp agent's Region() instances each monitor their stream of packets (e.g. the host region: agent.region.packet_handler). (Perhaps this should be changed to a generalized Network() class where all packets (coupled to their originating regions) are evaluated.

Event firing passes data on to a callback handler defined in the subscription, in the form of (handler, *args, **kwargs).

The Agent class monitors the ImprovedInstantMessage packet:

    ...

    onImprovedInstantMessage_received = self.region.packet_handler._register('ImprovedInstantMessage')
    onImprovedInstantMessage_received.subscribe(self.onImprovedInstantMessage)

    ...

    def onImprovedInstantMessage(self, packet):
        """ handles the many cases of data being passed in this message """

        {code} # parse and handle the data...

The messaging system then fires the event when an ImprovedInstantMessage packet is received:

    ... packet = UDPPacket(context & name = 'ImprovedInstantMessage')

    self.packet_handler.handle(packet)

The onImprovedInstantMessage method above then does it's thing with the data received.

Unsubscribing from an event:

    onImprovedInstantMessage_received.unsubscribe(self.onImprovedInstantMessage)

Event System Events

New! And not widely used yet...

The application level event system is a separate implementation in pyogp.lib.base.event_system.EventsHandler(). This allows for a timeout to be specified for the subscription to a particular event.

The api for subscribing to these events is similar to the PacketHandler() examples above, with an additional timeout parameter passed in the _register() method. When the specified timeout expires, the subscription returns None and expires the subscription.

The first use of this can be seen with the InstantMessageReceived() implementation as handled in Agent().onImprovedInstantMessage(), and usage of such in the sample script in the dev buildout bin/chat_and_instant_messaging.

Logging

Uses python's standard logging module (http://docs.python.org/library/logging.html). The library defines logging events throughout, it is up to the application/script to determine the output.

Hooking logging into a new module:

from logging import getLogger, CRITICAL, ERROR, WARNING, INFO, DEBUG

# initialize logging
logger = getLogger('pyogp.lib.base.agent')
log = logger.log

class Agent(object):
    """ our agent class """

    def __init__(self, params):

        self.params = params

        log(DEBUG, "Initializing agent with params: %s" % (params))

An application can then set up the logging output as follows (or any other way it pleases):

        console = logging.StreamHandler()
        formatter = logging.Formatter('%(asctime)-30s%(name)-30s: %(levelname)-8s %(message)s')
        console.setFormatter(formatter)
        logging.getLogger('').addHandler(console)
        logging.getLogger('').setLevel(logging.DEBUG)

The output to console is then:

2009-04-21 22:08:58,681       pyogp.lib.base.agent          : DEBUG    agent with params: params

Pyogp Unit Tests

See unittest.html in the embedded Pyogp/Client_Lib#Sphinx_.28api_docs.29.

or

Run 'bin/client_unittest'

We need more coverage here!

Writing Test Cases

Tests can be written using standard unittest. The tests in pyogp.interop cover some ogp and a couple of legacy cases, these need to be updated to work.

Testing only the call to login.cgi is unique, we don't need to spawn the client in a coroutine, nor do we need to keep the client alive, we just need to post to the login endpoint and evaluate the response.

pyogp.interop.tests.test_legacy_login

import unittest, doctest
import ConfigParser
from pkg_resources import resource_stream
import time
import uuid
import pprint

from pyogp.lib.base.agent import Agent
from pyogp.lib.base.datatypes import UUID
from pyogp.lib.base.exc import LoginError
from pyogp.lib.base.settings import Settings

import helpers

class AuthLegacyLoginTest(unittest.TestCase):
   
    def setUp(self):
        
        # initialize the config
        self.config = ConfigParser.ConfigParser()
        self.config.readfp(resource_stream(__name__, 'testconfig.cfg'))
                
        self.test_setup_config_name = 'test_interop_account'
        
        self.firstname = self.config.get(self.test_setup_config_name, 'firstname')
        self.lastname = self.config.get(self.test_setup_config_name, 'lastname')
        self.password = self.config.get(self.test_setup_config_name, 'password')
        self.agent_id = self.config.get(self.test_setup_config_name, 'agent_id')
        self.login_uri = self.config.get(self.test_setup_config_name, 'login_uri')
        self.region = self.config.get('test_interop_regions', 'start_region_uri') 

        self.successful_login_reponse_params = ['last_name', 'sim_ip', 'inventory-lib-root', 'start_location', 'inventory-lib-owner', 'udp_blacklist', 'home', 'message', 'agent_access_max', 'first_name', 'agent_region_access', 'circuit_code', 'sim_port', 'seconds_since_epoch', 'secure_session_id', 'look_at',  'ao_transition', 'agent_id', 'inventory_host', 'region_y', 'region_x', 'seed_capability', 'agent_access', 'session_id', 'login']

        self.settings = Settings()
        self.settings.MULTIPLE_SIM_CONNECTIONS = False

        self.client = Agent(self.settings, self.firstname, self.lastname, self.password)

    def tearDown(self):
        
        if self.client.connected:
            self.client.logout()
        
    def test_base_login(self):
        """ login with an account which should just work """

        self.client.settings.ENABLE_INVENTORY_MANAGEMENT = False

        self.client.login(loginuri = self.login_uri, start_location = self.region, connect_region = False)

        # make sure that the login response attributes propagate properly, and, make sure the login against a grid has worked
        self.assertEquals(self.client.grid_type, 'Legacy', 'Storing the wrong grid type based on a \'legacy\' login request')
        self.assertEquals(self.client.firstname, self.firstname)
        self.assertEquals(self.client.lastname, self.lastname)
        self.assertEquals(self.client.lastname, self.lastname)
        self.assertEquals(self.client.name, self.firstname + ' ' + self.lastname)
        self.assertEquals(self.client.connected, True)
        self.assertNotEquals(self.client.agent_id, None)
        self.assertEquals(str(self.client.agent_id), self.agent_id)
        self.assertNotEquals(self.client.session_id, None)
        self.assertNotEquals(self.client.secure_session_id, None)

        self.assertEquals(self.client.login_response['last_name'], self.lastname)
        self.assertEquals(self.client.login_response['first_name'], '"' + self.firstname + '"')
        self.assertEquals(self.client.login_response['login'], 'true')
        self.assertEquals(self.client.login_response['secure_session_id'], str(self.client.secure_session_id))
        self.assertEquals(self.client.login_response['session_id'], str(self.client.session_id))
        self.assertEquals(self.client.login_response['agent_id'], str(self.client.agent_id))
        self.assertNotEquals(self.client.login_response['seed_capability'], '')

        fail = 0
        fail_extra = 0
        fail_missing = 0
        extra_keys = ''
        missing_keys = ''
        
        for key in self.client.login_response:
            try:
                self.successful_login_reponse_params.index(key) # if the key is in our valid list, sweet
            except:
                fail_extra = 1
                extra_keys = extra_keys + ' ' + key

        for key in self.client.login_response:
            try:
                self.successful_login_reponse_params.index(key) # if the key is in our valid list, sweet
            except:
                fail_missing = 1
                missing_keys = missing_keys + ' ' + key
       
        self.assertEquals(fail_extra, 0, 'login response has additional keys: ' + extra_keys)
        self.assertEquals(fail_missing, 0, 'login response is missing keys: ' + missing_keys)

    def test_login_with_bad_password(self):

        self.client.settings.ENABLE_INVENTORY_MANAGEMENT = False

        self.assertRaises(LoginError, self.client.login, loginuri = self.login_uri, password = 'BadPassword', start_location = self.region, connect_region = False)

def test_suite():
    from unittest import TestSuite, makeSuite
    suite = TestSuite()
    suite.addTest(makeSuite(AuthLegacyLoginTest))
    return suite

Adding Functionality

Wrapping Packets

PyOGP, like the Viewer, communicates with the Second Life simulator by sending messages over UDP.

In order to extend PyOGP, you'll enable a new UDP message.

Example: Renaming an Object

To rename an object on the simulator, send an ObjectName message packet.

The templates for message packets are defined as classes in base/message/packets.py:

class ObjectNamePacket(object):
    ''' a template for a ObjectName packet '''

    def __init__(self, AgentDataBlock = {}, ObjectDataBlocks = []):
        """ allow passing in lists or dictionaries of block data """
        self.name = 'ObjectName'

Packets usually contain an AgentData block. They may also contain other blocks.

        if ObjectDataBlocks == []:
            # initialize an empty list for blocks that may occur > 1 time in the packet
            self.ObjectDataBlocks = []    # list to store multiple and variable block types

            # a sample block instance that may be appended to the list
            self.ObjectData = {}
            self.ObjectData['LocalID'] = None    # MVT_U32
            self.ObjectData['Name'] = None    # MVT_VARIABLE
        else:
            self.ObjectDataBlocks = ObjectDataBlocks

In this case, an ObjectName block can operate on a single or multiple objects.

Let's add the object name functionality to the Object class in base/objects.py.

    def set_object_name(self, agent, Name):
        """ update the name of an object."""

        packet = ObjectNamePacket()

First, get a new ObjectNamePacket object.

        # build the AgentData block
        packet.AgentData['AgentID'] = uuid.UUID(str(agent.agent_id))
        packet.AgentData['SessionID'] = uuid.UUID(str(agent.session_id))

Then set the packet's AgentData block:

        ObjectData = {}
        ObjectData['LocalID'] = self.LocalID
        ObjectData['Name'] = Name

        packet.ObjectDataBlocks.append(ObjectData)

And the updated name of the object.

        agent.region.enqueue_message(packet())

Then send the packet.

Where Does LocalID come from?

So how do you get an object's local ID?

Patience. It comes in the CompressedObjectUpdate packet (see Event Callbacks.)

Use the my_objects(), find_objects_by_name(), or find_objects_within_radius() methods of Objects (accessed through client.region.objects, see Agent Login.)

These methods return lists of objects on the simulator.

You'll need to use the Wait() utility method to pause your client to wait for the CompressedObjectUpdate packets with the detailed information on the scene to appear.

Then you can iterate over the resulting list of objects and call the update methods described above.

    # let's see what's nearby
    objects_nearby = client.region.objects.find_objects_within_radius(20)
 
    for item in objects_nearby:
        item.select(client)
 
    waiter = Wait(15)
 
    for item in objects_nearby:
        item.deselect(client)
 
    my_objects = client.region.objects.my_objects()

Sphinx (api docs)

Api documentation is now available for pyogp!

The docs directory in pyogp.lib.base contains source, last revision, and build files for sphinx based documents.

Sample output is available at {libdev root}/docs/html/index.html.

README.txt contains build instructions, as follows:

This checkout contains the most recently complied version of the documentation in docs/html/.

To rebuild the sphinx doc set:

Get sphinx!!!

Either use your virtualenv, or your native python install and run: 
    easy_install -U Sphinx

Then, from the docs dir:

1. python source/build.py
2. sphinx-build -a -c source/configure/ source/ html/

The docs/html/ directory will contain the fully compiled documentation set.
Please check in updated docs if you add functionality.

Roadmap

There is so much yet to implement that it is frightening. Here's what's up in the near term for pyogp:

• enabling parcel testing
• teleport (in OGP and in the 'legacy' context)
  • teleport works in OGP, but is not encapsulated in a method yet
• permissions system testingto save QA from 3-5 day regression passes on perms
• appearance - this will require enabling upload and download, plus baking. Anyone have some spare time? :)