Linden Lab Official:Media Rendering Plugin Operation and Data Flow

Second Life APIs

• Registration API

• Map API

• Inventory API
• Live Data Feeds
• Media Plugin API
  • Frequently Asked Questions
  • Technical Overview
  • Using the Plugin Test App
  • Getting Started
    • Hello World Plugin Code
  • Plugin Operation and Data Flow
  • Messages
    • Internal Messages
• Marketplace API for Viewers

Unofficial APIs

• Search API
• Snapshot API
• World API
• Picture Service

High level view

From a high level view, when the Viewer wants to render a media URI:

1. The Viewer examines the media URI. If the URI scheme is http: or https:, it retrieves the MIME type of the resource with an asynchronous HTTP HEAD request. It then looks up the URI scheme and/or MIME type in the mime_types.xml file in the viewer install to determine which plugin handles that media type.
2. The Viewer begins listening on a local TCP socket and launches the PLS, passing it the socket's port number
3. The PLS connects to the viewer's TCP socket and sends a 'hello' message
4. The viewer sends the PLS a "load_plugin" message with the filename of the plugin to load
5. The PLS loads the plugin and sends it an "init" message
6. The plugin sends "init_response" and "texture_params" messages in response to the "init" message

At this point, the plugin is initialized and ready to start rendering.

• Most of the messages can be received in any order, so the plugin should be prepared to handle any message at any time.
  • notably, the viewer may not send a size_change message before it sends a load_uri, so the plugin needs to be prepared to begin loading a media URI even if it doesn't have a pixel buffer yet
• The viewer will send a load_uri message when the plugin should load a new media URI.
  • note that the plugin may receive multiple load_uri messages, and should navigate or switch to the new media URI each time it receives one
• The PLS will send "idle" messages to the plugin periodically, which the plugin should use to update the media
• When the plugin updates the pixel buffer, it must send an "updated" message with top/left/bottom/right parameters so that the host knows the buffer has changed

This continues until the viewer doesn't need the plugin anymore.

1. When the viewer is ready for a plugin to exit, it closes the local TCP socket to that plugin's PLS
   • if the viewer crashes, the operating system will close the socket as part of its process cleanup, so the PLS will also catch this case
2. When the PLS notices that the socket has been closed, it sends a final "cleanup" message to the plugin
3. Once the plugin returns from the message handler, the PLS exits immediately

Detailed view

The following sections give a detailed view of what happens during each stage of plugin operation.

Initialization

During initialization, a plugin tells the PLS the set of messages it supports, including versioning information to maintain backwards compatibility.

Shared memory size negotiation

Once the plugin is running, the plugin and the Viewer negotiate the size and format of the shared memory pixel buffer into which the plugin draws with the texture_params, size_change_request, and size_change messages. The dimensions of the pixel buffer may change multiple times over the life of the plugin, so this process may happen repeatedly.

The plugin specifies the buffer's pixel size and format by sending a texture_params message. Indexed pixel modes and pixels which use less than eight bits per channel are not currently supported. Recommended modes are 24 bit RGB or 32 bit ARGB, or variants thereof (BGRA, etc).

Size negotiation is a bit more complicated:

• Viewer initially sets up the dimensions of the pixel buffer.
  • If the plugin doesn't have any special sizing requirements, it can just draw to the buffer and never send a size_change_request message.
• If the plugin sends a size_change_request message, the Viewer takes that as the "native size" of the media.
  • Media is allowed to change sizes multiple times during playback, such as with streaming QuickTime movies.
• If the media is playing as parcel media and the "auto scale" option is set, the draw dimensions may be increased to the next power of two.
• Viewer sends the requested drawing dimensions, the actual size of the buffer, and the name of the shared memory segment to the plugin in a size_change_response message.
  • The plugin must draw within the specified dimensions, and should NOT send a size_change_request in response to the size_change_response

When changing buffer sizes, the new size may require a larger or smaller memory buffer than the old size. In this case, the shared memory segment name received with the size_change_response may be different than the one the plugin was previously drawing to. If the plugin receives a size_change_response with a shared memory segment name it hasn't seen yet, it MUST not draw to the old shared memory segment using the new parameters, or it may overrun the end of the buffer and crash.

In this case, the plugin should expect to receive a shm_added message for the new shared memory segment either before or after the size_change_response, and must wait until it has received both before drawing to the new buffer.

Likewise, when a plugin receives a shm_remove message and it's drawing to the shared memory buffer with that name, it needs to stop doing so, since the memory buffer will be deallocated once the message handler returns.

When the plugin is running

TODO: what specific messages are sent and received by the plugin? what are the mandatory messages? list other possible messages?

Time-based media rendering plugins

TODO

Browser-like media rendering plugins

TODO

Error handling

TODO: how should the plugin handle errors? how does the PLS handle plugin errors?

Exit and cleanup

TODO: what does the plugin need to do when it exits? what does the PLS need to do?