Pyogp/Documentation/Specification/pyogp.lib.base

UDP Messaging

API

The UDP messaging system is broken up into a few parts. They are: UDP Dispatcher, Circuit, Message (with Block), Packet, UDPSerializer, UDPDeserialzer, and UDPNetClient. I will discuss each of these to explain how each component should be used.

Message Template

Can be found message_template.msg
Before I can discuss any of the design components, I should first explain the message template. The message template is a file that outlines all the different UDP messages that can be sent over a circuit. It breaks each message down by the message header information, its blocks, and the block data.
In order to communicate between the client and sim (or anything else) we need to be able to parse the message template, determine which messages can be sent, build messages based on the format they are specified to be in, and then read incoming messages. This is what the Message Template Parser is for.

Message Template Parser

In order to build or read (and therefore send or receive) any UDP messages, we have to first parse the message template. The parser parses the message template, searching for each message listed in it, and then iterates through the message's blocks and the block data.
The parser goes through the message template and constructs a data object of type MessageTemplate, MessageTemplateBlock, and MessageTemplateVariable. These types are used to store general information about the message that is read from the message template. In other words, these objects created by the parser hold no incoming or outgoing data. They are simply used as templates to build data out of. They allow us to know the header information for the message, the blocks it is supposed to have, and the data that the blocks are supposed to have.
The parser reads things such as the message frequency, the message number (which is a unique value that is matched with frequency), its trust, encoding, and deprecation.

• The template also stores something called the hex num, which is the combination of the frequency and the message number stored as a hex number. It is stored here because the hex value never changes for the templates, and rather than building the hex value every time a message is going to be sent, it is just stored in the template for quicker access.

It also reads the block information such as its type (one of single, multiple, or variable)

• If the block is of type multiple, then something called the block number is also stored. This number represents how many of the given block MUST be written for a message.
• If the block is of type variable, then any number of the given block can be written for a message.

Finally, it reads the block data for things such as the data type (one of many types) and its size.

• The string type read from the template is converted to a class variable (like an enum)
• The size of the variable is gotten from our sizeof function in message_types.py
  • Although, if the variable is of type variable or fixed, the maximum number of bytes that the variable can be is stored (it is also listed in the message template as a third parameter, such as { Data Variable 2 }, which says the variable called "Data" is of type "Variable" where the variable can store up to 2 bytes worth of data.

The output of the parser is a list of message template objects, where each object has its list of blocks, and where each block has its list of variables.

Message Template Dict

The parser outputs a list of message templates. In order to make accessing this list easier and more efficient, a dict has been created. This takes the list and makes dicts out of it, one that maps the template name to the template, and one that maps the frequency/num combination to the template. This way, we can get any template by its name or frequency/num combination. This class is stored as a utility (like a Singleton that can be gotten with ZCA's getUtility() function).

Messages and Packets

MsgData
In order to create a message that can be sent over a socket, we have at the very lowest level the classes MsgData, MsgBlockData, and MsgVariableData. These classes represent the components that are used when building a packet's message data. A message is made up of blocks, and each of the blocks is made up of some variables. These classes are similar to the MessageTemplate, MessageTemplateBlock, and MessageTemplateVariable classes, except these classes are designed to hold actual data that can be serialized and sent over a network.
To create a message, one could create a new MsgData object, and create the corresponding blocks and variables that make up the message, and add it to the message. This will create an object that knows about its blocks, and where the blocks know about their variables. One thing to know is that the MsgData object has a dictionary of blocks, mapped by name (the variable called "blocks" in MsgData). These blocks are actually block lists. The reason the blocks are actually lists is because a block can be of the type MULTIPLE or VARIABLE, and so any given message can have the same block repeated (essentially meaning the MsgData's blocks dictionary would map the different blocks to the same element in the dictionary, an overwrite). So, if you do something like message_data.blocks, just remember that blocks is a list.
Packets
Now, the MsgData object is just the message data for something to be sent through udp. It is the payload for the packet. There are some other things we need in order to send a message. For this, we have a Packet. Packets hold information such as the flags that the packet will be sent with, the id of the packet (id being the sequence number, or the order that the packet was sent over a udp connection), its allowed number of retries, and the time it will expire. Packets also keep track of the acks that have been attached to the packet, and of course, it has the message data or payload for the Packet. Again, just like the MsgData, the Packet object is a high-level object that still needs to be serialized.

UDPSerializer and UDPDeserializer
With a Packet object in hand, one can then use the UDPSerializer to serialize that packet into a series of bytes that can finally be sent over a network. The UDPSerializer takes a Packet, determines which type of message it is by looking at the message template and reading information from the message data in the Packet, and attempts to serialize the Packet based on how the message template says it should look. If there is any discrepancy in what data the Packet has and what data the message template says it should have, an error will be thrown and the serialization will fail. Note that the UDPSerializer also packs things such as the flags onto the front, the message header information, the payload, and even adds the acks onto the end of the packet.

The way Pyogp's serialization works is that (just like everything else) we have an interface defined for ISerialization and IDeserialization, then we implement that interface with our serializers and deserializers (in this case, UDPSerializer and UDPDeserialzer), and tell zca the type of object we are attempt to (de)serialize. So, for instance, what we would do to serializer a Packet is:
serializer = ISerializer(myPacket) data = serializer.serialize()
This tells ZCA to look for the serializer that adapts our Packet object (in other words, knows how to change our Packet into a string of bytes). Then we just serialize it and get our byte payload. This serialized data has everything we need to send it over the network, including all the flags, the message template information, headers, payload, and any acks attached onto the end. +++

The way that Pyogp currently does udp messaging is through what is called the UDPDispatcher. The UDPDispatcher provides the API through which you create a connection (or connections), build a send and receive messages.

That is, it handles all parsing of the message template (message_template.msg), the message list (message.xml), creates dictionaries out of them, creates a UDP socket to send and receives messages from, sets up HTTP connections, and has builders and readers necessary to build and read the template formatted messages and the llsd formatted messages. It also handles all the maintaining of packets that need to be acked, keeping track of which we (being the client) need to ack and which we want acked by the server, as well as sending the acks or resending unacked packets.

The Message System object should really be what any user of Pyogp should need to deal with either sending or receiving messages. The user shouldn't need to use a reader or a builder directly. It also shouldn't have to establish any connections or do any direct sending or receiving of messages over a socket or connection. The user should always go through the Message System. The Message System is meant to be driven by some outside source, either a client, a test, or just another application. It has none of its own loops or threads.

Initializing the system

MessageSystem(port) - the port of which the Message System will receive messages. This is currently not used and may even be removed later.

Building a message

There are a few methods in the Message System that are available as high-level api calls. When a user wants to create a message, he or she does not have to know which type of message the server it expects it to be formatted as. For this reason, the Message System determines which format to use and delegates the creation of building the message to the corresponding builder.

new_message(message_name) - begins creating a new message. The Message System determines which type of message is being built (either the template flavor or the llsd flavor). This determines which builder should be used. The creation of the new message is then delegated to the corresponding builder (transparent to the user).
next_block(block_name) - this tells the builder that we want to begin building the named block of the message. This is just delegated to a builder (see below for the details of how it works).
add_data(var_name, data, data_type) - adds data to the variable in the current block with the var_name name, delegated to the builder.

Sending a message

Once a message has been built, the user can then send the message to a given host (with a host being a combination of the ip address and port).

send_message(host, message_buf=None) - this sends the message that we most recently built using the new_message, next_block, add_data functions. It sends the message to the host specified. The message_buf parameter allows for the user to pass in a message to send that wasn't built using the Message System (such as those created by directly using the builder of choice). This function also makes sure the created message is serialized, adds on any packet flags, adds the sequence number for the packet, adds the packet identification, and finally the payload. It also adds any acks on to the end and compresses the message using a zero-coding.
send_retry(host, message_buf=None) - sends a message using the RETRY packet flag, but delegates sending to send_messsage.
send_reliable(host, retries, message_buf=None) - sends a message using the RELIABLE flag, as well as sets the packet's retry count, but delegates sending to send_messsage.

Receiving a message

receive_check() - determines if there is a message waiting on the socket. Does a single pass to get a single message. If there is a message waiting, it processes the message, reading its binary form into a deserialized form, and determining its flags (that is, does it need to be acked, does it have acks attached, it is zero-coded, etc). The read message can then be accessed through the methods:
get_received_message() - this returns the whole message, in the form of a MsgData object
get_data(block_name, var_name, data_type, block_number=0) - this gets data from a particular block. The block_number is used when the message has multiple or variable blocks.

Maintenance

The Message System also has some other features that allow users an easy to way make sure all maintenance is handled properly. Maintenance includes acking packets and removing stale packets (ones that cannot be resent anymore and haven't been acked in time).

process_acks() - this function resends all packets that we have sent out that haven't been acked in time, as well as sending out all the acks of the messages we have received from the server that expect to be acked.

Design Decision

You'll notice that, in most cases, the user never has direct access to a created or receiving message. The user can, of course, get the message directly by going through the Message System's builder and reader (messagesystem.builder.current_msg), the design is not meant to be used in such a way. The user is not meant to manipulate a message directly but to use builders and readers for such a thing. Even then, the user shouldn't even be using builders and readers but going through the message system, which uses them.

• Advantages:

1. Separation of concerns: this means the MsgData class represents a message, the builder builds the message, and the reader reads a message. Each has its own particular function and only that function. There is no object that has more than one piece of functionality.
2. Ease of use: the MessageSystem provides the high-level functionality to bring all the pieces together to allow the user to not need to build any message by hand, do any serialization or connections, keep track of circuit information (which sequence # is next, packet acking, etc), build or read packet header information (flags). Also allows the user to not need to know what formats the server expects to receive messages in. The MessageSystem handles this distinction based on message type.
3. Generic messages: any message can be built, read, sent, and received through this manner. All messages are read from the message template and so they can all be built with this generic representation of messages, builders, and readers.

• Disadvantages:

1. No message object: the messages being built and sent are stored within the builders, readers, and MessageSystem, and so it may not be what users expect. For instance, users may be used to using a message OBJECT directly and passing a given message to something else to send it (rather than just calling send_message()). On the same line, people may be used to having explicit connections that they send/receive on. E.g. conn1.send(message1) rather than messageSystem.send_message() which determines which connection to send the message to.
2. Related to number 1, because we have no message object, there is no way to save and reuse objects. We have to reconstruct the message every time we wish to send the same message. This isn't necessarily true because we ARE currently saving packets that have been sent (so that we can resend them if they don't get acked in time). They DO exist, the design just keeps the user from ever needing to have direct access. This may be one case that the Message System could give the user a direct message if he or she so asks for it. This is currently possible; it's just not explicitly in the API.
3. Sequential building of message: the messages are built by doing a series of new_message, next_block, add_data calls. Some may want to build a message by having an object (see note above) that they can do direct calls to. e.g. message.name = "Locklainn" message.agent_id = uuid.UUID('blahblah'). The problem here lies in the MULTIPLE and VARIABLE type blocks. For instance, a single message can have the same variable multiple times in the message. So you can't simply do message.name="NAME" because the variable might exist in multiple blocks, and so which block does that variable belong to? Another way one might do it is by having something such as message.name_1= message.name_2=, but this doesn't seem to gain us anything in terms of ease of use in building a message. A given message can have any number of these blocks, so to handle the same thing one would have to use lists or some other container structure. One might have to do the following: message.blocks = [['name':"Locklainn", 'agent_id':UUID("blahblah")]['session_id':123531, 'circuit_code':123656]].

When sending a message, the user specifies the host that will receive the message.

• Advantages:

1. The same messaging system can be used to send to multiple hosts without any reconfiguration. The other method would be to couple the messaging system with its targeted receiver so that when sending a message it will always go to that receiver. This binds the messaging system to communicating with only a single host, unless there is added functionality to allow a list of target hosts that one can send to, which effectively brings us back to allowing the message system to send to any host (the original design).

see Pyogp/Client_Lib

Old UDP Messaging

Message System/API

The way that Pyogp currently does messaging is through the Message System. The Message System provides the API through which you create a connection (or connections), build a send and receive messages. The Message System encapsulates all the other functionality that is needed do start working with messages. That is, it handles all parsing of the message template (message_template.msg), the message list (message.xml), creates dictionaries out of them, creates a UDP socket to send and receives messages from, sets up HTTP connections, and has builders and readers necessary to build and read the template formatted messages and the llsd formatted messages. It also handles all the maintaining of packets that need to be acked, keeping track of which we (being the client) need to ack and which we want acked by the server, as well as sending the acks or resending unacked packets.

The Message System object should really be what any user of Pyogp should need to deal with either sending or receiving messages. The user shouldn't need to use a reader or a builder directly. It also shouldn't have to establish any connections or do any direct sending or receiving of messages over a socket or connection. The user should always go through the Message System. The Message System is meant to be driven by some outside source, either a client, a test, or just another application. It has none of its own loops or threads.

Initializing the system

MessageSystem(port) - the port of which the Message System will receive messages. This is currently not used and may even be removed later.

Building a message

There are a few methods in the Message System that are available as high-level api calls. When a user wants to create a message, he or she does not have to know which type of message the server it expects it to be formatted as. For this reason, the Message System determines which format to use and delegates the creation of building the message to the corresponding builder.

new_message(message_name) - begins creating a new message. The Message System determines which type of message is being built (either the template flavor or the llsd flavor). This determines which builder should be used. The creation of the new message is then delegated to the corresponding builder (transparent to the user).
next_block(block_name) - this tells the builder that we want to begin building the named block of the message. This is just delegated to a builder (see below for the details of how it works).
add_data(var_name, data, data_type) - adds data to the variable in the current block with the var_name name, delegated to the builder.

Sending a message

Once a message has been built, the user can then send the message to a given host (with a host being a combination of the ip address and port).

send_message(host, message_buf=None) - this sends the message that we most recently built using the new_message, next_block, add_data functions. It sends the message to the host specified. The message_buf parameter allows for the user to pass in a message to send that wasn't built using the Message System (such as those created by directly using the builder of choice). This function also makes sure the created message is serialized, adds on any packet flags, adds the sequence number for the packet, adds the packet identification, and finally the payload. It also adds any acks on to the end and compresses the message using a zero-coding.
send_retry(host, message_buf=None) - sends a message using the RETRY packet flag, but delegates sending to send_messsage.
send_reliable(host, retries, message_buf=None) - sends a message using the RELIABLE flag, as well as sets the packet's retry count, but delegates sending to send_messsage.

Receiving a message

receive_check() - determines if there is a message waiting on the socket. Does a single pass to get a single message. If there is a message waiting, it processes the message, reading its binary form into a deserialized form, and determining its flags (that is, does it need to be acked, does it have acks attached, it is zero-coded, etc). The read message can then be accessed through the methods:
get_received_message() - this returns the whole message, in the form of a MsgData object
get_data(block_name, var_name, data_type, block_number=0) - this gets data from a particular block. The block_number is used when the message has multiple or variable blocks.

Maintenance

The Message System also has some other features that allow users an easy to way make sure all maintenance is handled properly. Maintenance includes acking packets and removing stale packets (ones that cannot be resent anymore and haven't been acked in time).

process_acks() - this function resends all packets that we have sent out that haven't been acked in time, as well as sending out all the acks of the messages we have received from the server that expect to be acked.

Design Decision

You'll notice that, in most cases, the user never has direct access to a created or receiving message. The user can, of course, get the message directly by going through the Message System's builder and reader (messagesystem.builder.current_msg), the design is not meant to be used in such a way. The user is not meant to manipulate a message directly but to use builders and readers for such a thing. Even then, the user shouldn't even be using builders and readers but going through the message system, which uses them.

• Advantages:

1. Separation of concerns: this means the MsgData class represents a message, the builder builds the message, and the reader reads a message. Each has its own particular function and only that function. There is no object that has more than one piece of functionality.
2. Ease of use: the MessageSystem provides the high-level functionality to bring all the pieces together to allow the user to not need to build any message by hand, do any serialization or connections, keep track of circuit information (which sequence # is next, packet acking, etc), build or read packet header information (flags). Also allows the user to not need to know what formats the server expects to receive messages in. The MessageSystem handles this distinction based on message type.
3. Generic messages: any message can be built, read, sent, and received through this manner. All messages are read from the message template and so they can all be built with this generic representation of messages, builders, and readers.

• Disadvantages:

1. No message object: the messages being built and sent are stored within the builders, readers, and MessageSystem, and so it may not be what users expect. For instance, users may be used to using a message OBJECT directly and passing a given message to something else to send it (rather than just calling send_message()). On the same line, people may be used to having explicit connections that they send/receive on. E.g. conn1.send(message1) rather than messageSystem.send_message() which determines which connection to send the message to.
2. Related to number 1, because we have no message object, there is no way to save and reuse objects. We have to reconstruct the message every time we wish to send the same message. This isn't necessarily true because we ARE currently saving packets that have been sent (so that we can resend them if they don't get acked in time). They DO exist, the design just keeps the user from ever needing to have direct access. This may be one case that the Message System could give the user a direct message if he or she so asks for it. This is currently possible; it's just not explicitly in the API.
3. Sequential building of message: the messages are built by doing a series of new_message, next_block, add_data calls. Some may want to build a message by having an object (see note above) that they can do direct calls to. e.g. message.name = "Locklainn" message.agent_id = uuid.UUID('blahblah'). The problem here lies in the MULTIPLE and VARIABLE type blocks. For instance, a single message can have the same variable multiple times in the message. So you can't simply do message.name="NAME" because the variable might exist in multiple blocks, and so which block does that variable belong to? Another way one might do it is by having something such as message.name_1= message.name_2=, but this doesn't seem to gain us anything in terms of ease of use in building a message. A given message can have any number of these blocks, so to handle the same thing one would have to use lists or some other container structure. One might have to do the following: message.blocks = [['name':"Locklainn", 'agent_id':UUID("blahblah")]['session_id':123531, 'circuit_code':123656]].

When sending a message, the user specifies the host that will receive the message.

• Advantages:

1. The same messaging system can be used to send to multiple hosts without any reconfiguration. The other method would be to couple the messaging system with its targeted receiver so that when sending a message it will always go to that receiver. This binds the messaging system to communicating with only a single host, unless there is added functionality to allow a list of target hosts that one can send to, which effectively brings us back to allowing the message system to send to any host (the original design).

UDP Template Messaging

There are a few main components to sending a UDP message, the message template, the parser, the builder, and the reader.

Message Template Builder

The builder is used to create messages that can be sent through UDP. It is used to make sure that the messages being built are in accordance with the message template and have all the necessary blocks and data that go along with the message. The builder creates message objects using the MsgData, MsgBlockData, and MsgVariableData classes. These objects differ from the template versions in that they hold the actual data. They don't hold general information but only exactly what will be sent through UDP. However, they hold the data in object form, and therefore are not serialized. The builder also serializes the message once it is finished being built.
Messages are built in a sequence of steps:

1. new_message(message_name) - this method sets up a new message to begin being built. The message is filled in with all the block stubs that it needs to have.
2. next_block(block_name) - sets the block that we are building to be the block with the given block name. It also fills in the stubbed block with all the variables that the block needs to have.
   • Note that if we are trying to set the block to one that doesn't exist, or that has already been created, we will get errors. However, if the block is of type multiple or variable, then we can create more than one block with the same name. A multiple type block means that there is a fixed number of blocks that the message must have, so we can add that number of blocks to the message (but no more than that number). A variable type block means that there can be any number of blocks, so we can add any number of this block to the message (meaning, we can call next_block() with the same block_name any number of times).
3. add_data(var_name, data, data_type) - this adds the data to the block. There are a couple checks to make sure that the data you pass it matches the data it is expecting (from the template).
   • When we add data we store the size of the data. Now, normally we know the size directly from the data type (where both type and size are stored in the template). But when the data is of type variable then the template doesn't store the size of the data (because it can't, the data can be any size). However, the template stores how many bytes the size can be. So, for type variable, we have to determine the size of the actual data being written and store that instead.
4. build_message() - this goes through the message, each of its blocks and data, and serializes the data into a string that can be sent over UDP. Before it does so, it makes sure the data added to the message is correct, that it has all the blocks and variables it is supposed to have, and in the right format. This returns the message and size that has been serialized.
   • To figure out the format of the message and what the serializes is doing, check out the Pyogp/Client_Lib/Notes page.
   • This uses the DataPacker to pack the data correctly. The DataPacker takes the data and the data_type and determines how to serialize the data given the type.

Message Template Reader

The reader attempts to read a message that has been received through UDP. The reader can only process one message at a time. There are a few steps to processing and using the read data:

1. validate_message(buffer,size) - this attempts to decode the message and figure out what type of message it is. It also checks to make sure the message is valid (in the message template), and keeps track of the template if it is.
2. read_message(buffer) - goes through the message, skipping over the header and pre-header information (that gets added on by some other process) and processes the blocks and the data. This also makes sure that the message was validated first. Simply iterates over the buffer, reading block information and data information and constructs a MsgData object out of it. This deserializes the data back into binary form.
   • Note that if the block is multiple or variable, it repeatedly reads the block data until it has processed all of the message data.
   • The DataUnpacker is used to deserialize the data. It is given the data and type.
3. get_data(block_name, var_name, data_type, block_number=0) - this returns the data that was deserialized for the message. We give it the block name of which block to find the variable, the data type to make sure that the user is aware of the type (error checks) and as an optional argument, the block_number (which is only used when the message has many blocks of the same time, aka, the block type is multiple or variable).
4. {optional}clear_message() - this gets the reader ready to read a new message. The reader won't crash without doing this, but warnings will be issued to make sure this is the desired behavior.