Difference between revisions of "Pyogp/Documentation/Specification/pyogp.lib.base"

From Second Life Wiki
Jump to navigation Jump to search
Line 24: Line 24:
<b>send_retry(host, message_buf=None)</b> - sends a message using the RETRY packet flag, but delegates sending to send_messsage.<br>
<b>send_retry(host, message_buf=None)</b> - sends a message using the RETRY packet flag, but delegates sending to send_messsage.<br>
<b>send_reliable(host, retries, message_buf=None)</b> - sends a message using the RELIABLE flag, as well as sets the packet's retry count, but delegates sending to send_messsage.<br>
<b>send_reliable(host, retries, message_buf=None)</b> - sends a message using the RELIABLE flag, as well as sets the packet's retry count, but delegates sending to send_messsage.<br>


===Receiving a message===
===Receiving a message===
Line 36: Line 34:


<b>process_acks()</b> - this function resends all packets that we have sent out that haven't been acked in time, as well as sending out all the acks of the messages we have received from the server that expect to be acked.
<b>process_acks()</b> - this function resends all packets that we have sent out that haven't been acked in time, as well as sending out all the acks of the messages we have received from the server that expect to be acked.
===Design Decision===
You'll notice that, in most cases, the user never has direct access to a created or receiving message. The user can, of course, get the message directly by going through the Message System's builder and reader (messagesystem.builder.current_msg), the design is not meant to be used in such a way. The user is not meant to manipulate a message directly but to use builders and readers for such a thing. Even then, the user shouldn't even be using builders and readers but going through the message system, which uses them.
*Advantages:
#Separation of concerns. This means the MsgData class represents a message, the builder builds the message, and the reader reads a message. Each has its own particular function and only that function. There is no object that has more than one piece of functionality.
#Ease of use - the MessageSystem provides the high-level functionality to bring all the pieces together to allow the user to not need to build any message by hand, do any serialization or connection


==UDP Template Messaging==
==UDP Template Messaging==

Revision as of 10:57, 11 August 2008

ToDo: fill out this page Enus Linden 10:10, 24 July 2008 (PDT)

see Pyogp/Client_Lib

Message System/API

The way that Pyogp currently does messaging is through the Message System. The Message System provides the API through which you create a connection (or connections), build a send and receive messages. The Message System encapsulates all the other functionality that is needed do start working with messages. That is, it handles all parsing of the message template (message_template.msg), the message list (message.xml), creates dictionaries out of them, creates a UDP socket to send and receives messages from, sets up HTTP connections, and has builders and readers necessary to build and read the template formatted messages and the llsd formatted messages. It also handles all the maintaining of packets that need to be acked, keeping track of which we (being the client) need to ack and which we want acked by the server, as well as sending the acks or resending unacked packets.

The Message System object should really be what any user of Pyogp should need to deal with either sending or receiving messages. The user shouldn't need to use a reader or a builder directly. It also shouldn't have to establish any connections or do any direct sending or receiving of messages over a socket or connection. The user should always go through the Message System. The Message System is meant to be driven by some outside source, either a client, a test, or just another application. It has none of its own loops or threads.

Initializing the system

MessageSystem(port) - the port of which the Message System will receive messages. This is currently not used and may even be removed later.

Building a message

There are a few methods in the Message System that are available as high-level api calls. When a user wants to create a message, he or she does not have to know which type of message the server it expects it to be formatted as. For this reason, the Message System determines which format to use and delegates the creation of building the message to the corresponding builder.

new_message(message_name) - begins creating a new message. The Message System determines which type of message is being built (either the template flavor or the llsd flavor). This determines which builder should be used. The creation of the new message is then delegated to the corresponding builder (transparent to the user).
next_block(block_name) - this tells the builder that we want to begin building the named block of the message. This is just delegated to a builder (see below for the details of how it works).
add_data(var_name, data, data_type) - adds data to the variable in the current block with the var_name name, delegated to the builder.

Sending a message

Once a message has been built, the user can then send the message to a given host (with a host being a combination of the ip address and port).

send_message(host, message_buf=None) - this sends the message that we most recently built using the new_message, next_block, add_data functions. It sends the message to the host specified. The message_buf parameter allows for the user to pass in a message to send that wasn't built using the Message System (such as those created by directly using the builder of choice). This function also makes sure the created message is serialized, adds on any packet flags, adds the sequence number for the packet, adds the packet identification, and finally the payload. It also adds any acks on to the end and compresses the message using a zero-coding.
send_retry(host, message_buf=None) - sends a message using the RETRY packet flag, but delegates sending to send_messsage.
send_reliable(host, retries, message_buf=None) - sends a message using the RELIABLE flag, as well as sets the packet's retry count, but delegates sending to send_messsage.

Receiving a message

receive_check() - determines if there is a message waiting on the socket. Does a single pass to get a single message. If there is a message waiting, it processes the message, reading its binary form into a deserialized form, and determining its flags (that is, does it need to be acked, does it have acks attached, it is zero-coded, etc). The read message can then be accessed through the methods:
get_received_message() - this returns the whole message, in the form of a MsgData object
get_data(block_name, var_name, data_type, block_number=0) - this gets data from a particular block. The block_number is used when the message has multiple or variable blocks.

Maintenance

The Message System also has some other features that allow users an easy to way make sure all maintenance is handled properly. Maintenance includes acking packets and removing stale packets (ones that cannot be resent anymore and haven't been acked in time).

process_acks() - this function resends all packets that we have sent out that haven't been acked in time, as well as sending out all the acks of the messages we have received from the server that expect to be acked.

Design Decision

You'll notice that, in most cases, the user never has direct access to a created or receiving message. The user can, of course, get the message directly by going through the Message System's builder and reader (messagesystem.builder.current_msg), the design is not meant to be used in such a way. The user is not meant to manipulate a message directly but to use builders and readers for such a thing. Even then, the user shouldn't even be using builders and readers but going through the message system, which uses them.

  • Advantages:
  1. Separation of concerns. This means the MsgData class represents a message, the builder builds the message, and the reader reads a message. Each has its own particular function and only that function. There is no object that has more than one piece of functionality.
  2. Ease of use - the MessageSystem provides the high-level functionality to bring all the pieces together to allow the user to not need to build any message by hand, do any serialization or connection

UDP Template Messaging

There are a few main components to sending a UDP message, the message template, the parser, the builder, and the reader.

Message Template

Can be found message_template.msg
The message template is the file that outlines all the different types of UDP messages that can be sent. It breaks each message down by the message header information, its blocks, and the block data.
In order to communicate between the client and sim (or anything else) we need to be able to parse the message template, determine which messages can be sent, build messages based on the format they are specified to be in, and then read incoming messages.

Message Template Parser

In order to build or read (and therefore send or receive) any UDP messages, we have to first parse the message template. The parser parses the message template, searching for each message listed in it, and then iterates through the message's blocks and the block data.
The parser goes through the message template and constructs a data object of type MessageTemplate, MessageTemplateBlock, and MessageTemplateVariable. These types are used to store general information about the message that is read from the message template. In other words, these objects created by the parser hold no incoming or outgoing data. They are simply used as templates to build data out of. They allow us to know the header information for the message, the blocks it is supposed to have, and the data that the blocks are supposed to have.
The parser reads things such as the message frequency, the message number (which is a unique value that is matched with frequency), its trust, encoding, and deprecation.

  • The template also stores something called the hex num, which is the combination of the frequency and the message number stored as a hex number. It is stored here because the hex value never changes for the templates, and rather than building the hex value every time a message is going to be sent, it is just stored in the template for quicker access.

It also reads the block information such as its type (one of single, multiple, or variable)

  • If the block is of type multiple, then something called the block number is also stored. This number represents how many of the given block MUST be written for a message.
  • If the block is of type variable, then any number of the given block can be written for a message.

Finally, it reads the block data for things such as the data type (one of many types) and its size.

  • The string type read from the template is converted to a class variable (like an enum)
  • The size of the variable is gotten from our sizeof function in message_types.py
    • Although, if the variable is of type variable or fixed, the maximum number of bytes that the variable can be is stored (it is also listed in the message template as a third parameter, such as { Data Variable 2 }, which says the variable called "Data" is of type "Variable" where the variable can store up to 2 bytes worth of data.

The output of the parser is a list of message template objects, where each object has its list of blocks, and where each block has its list of variables.

Message Template Dict

The parser outputs a list of message templates. In order to make accessing this list easier and more efficient, a dict has been created. This takes the list and makes dicts out of it, one that maps the template name to the template, and one that maps the frequency/num combination to the template. This way, we can get any template by its name or frequency/num combination.

Message Template Builder

The builder is used to create messages that can be sent through UDP. It is used to make sure that the messages being built are in accordance with the message template and have all the necessary blocks and data that go along with the message. The builder creates message objects using the MsgData, MsgBlockData, and MsgVariableData classes. These objects differ from the template versions in that they hold the actual data. They don't hold general information but only exactly what will be sent through UDP. However, they hold the data in object form, and therefore are not serialized. The builder also serializes the message once it is finished being built.
Messages are built in a sequence of steps:

  1. new_message(message_name) - this method sets up a new message to begin being built. The message is filled in with all the block stubs that it needs to have.
  2. next_block(block_name) - sets the block that we are building to be the block with the given block name. It also fills in the stubbed block with all the variables that the block needs to have.
    • Note that if we are trying to set the block to one that doesn't exist, or that has already been created, we will get errors. However, if the block is of type multiple or variable, then we can create more than one block with the same name. A multiple type block means that there is a fixed number of blocks that the message must have, so we can add that number of blocks to the message (but no more than that number). A variable type block means that there can be any number of blocks, so we can add any number of this block to the message (meaning, we can call next_block() with the same block_name any number of times).
  3. add_data(var_name, data, data_type) - this adds the data to the block. There are a couple checks to make sure that the data you pass it matches the data it is expecting (from the template).
    • When we add data we store the size of the data. Now, normally we know the size directly from the data type (where both type and size are stored in the template). But when the data is of type variable then the template doesn't store the size of the data (because it can't, the data can be any size). However, the template stores how many bytes the size can be. So, for type variable, we have to determine the size of the actual data being written and store that instead.
  4. build_message() - this goes through the message, each of its blocks and data, and serializes the data into a string that can be sent over UDP. Before it does so, it makes sure the data added to the message is correct, that it has all the blocks and variables it is supposed to have, and in the right format. This returns the message and size that has been serialized.
    • To figure out the format of the message and what the serializes is doing, check out the Pyogp/Client_Lib/Notes page.
    • This uses the DataPacker to pack the data correctly. The DataPacker takes the data and the data_type and determines how to serialize the data given the type.

Message Template Reader

The reader attempts to read a message that has been received through UDP. The reader can only process one message at a time. There are a few steps to processing and using the read data:

  1. validate_message(buffer,size) - this attempts to decode the message and figure out what type of message it is. It also checks to make sure the message is valid (in the message template), and keeps track of the template if it is.
  2. read_message(buffer) - goes through the message, skipping over the header and pre-header information (that gets added on by some other process) and processes the blocks and the data. This also makes sure that the message was validated first. Simply iterates over the buffer, reading block information and data information and constructs a MsgData object out of it. This deserializes the data back into binary form.
    • Note that if the block is multiple or variable, it repeatedly reads the block data until it has processed all of the message data.
    • The DataUnpacker is used to deserialize the data. It is given the data and type.
  3. get_data(block_name, var_name, data_type, block_number=0) - this returns the data that was deserialized for the message. We give it the block name of which block to find the variable, the data type to make sure that the user is aware of the type (error checks) and as an optional argument, the block_number (which is only used when the message has many blocks of the same time, aka, the block type is multiple or variable).
  4. {optional}clear_message() - this gets the reader ready to read a new message. The reader won't crash without doing this, but warnings will be issued to make sure this is the desired behavior.