Linden Lab Official:Media Rendering Plugin System Technical Overview

Overview

The Second Life media plugin system enables integration of external media types into the Second Life Viewer, enabling the Viewer to render media both inworld and in the embedded web browser via media rendering plugins. Future releases of the Viewer will incorporate the media plugin system to provide built-in support for a variety media types and provide an open API for third-party developers to create plugins for additional media types.

Using the media plugin system, developers will be able to create Viewer plugins to render a variety of media types. This open API will enable interesting applications and use cases, for example: shared desktops, native document display, third-party application integration, and so on.

Additionally, the media plugin system is the first use of a general Viewer plugin architecture that in the future may be used for other types of plugins, for example to support additional input devices, image decoding (for example, JPEG2000), and voice services.

This design does not address a system for distributing plugins automatically, for example in the way that Firefox add-ons work. Clearly, something similar will be required eventually but is beyond the scope of this document.

Advantages

In this system, each plug-in runs as a separate process, which:

Improves Viewer stability: if the plugin crashes, the Viewer does not, though it may have invalid (or generic) media data. Currently, a small but significant portion of Viewer crashes occur when a media component crashes.
Eliminates build-time dependencies, making it easier to author plugins.
Eases plugin development:
- Plugin developers do not need to rebuild the entire Viewer, a complex and time-consuming task.
- Plugin developers can use different compiler versions, perhaps even different languages.

Additionally, the architecture:

Takes advantage of multi-core systems.
Facilitates support for multiple media sources per parcel or per prim.

Plugin system architecture

A plugin is a platform-native dynamic library (.DLL on Windows, .dylib on Mac OS, and .so on Linux).

Although the architecture provides for plugins to load directly into the Viewer process (for example for certain plugins developed by Linden Lab), externally-developed plugins generally run as a separate process to enhance Viewer stability.

NOTE: You write to the same API whether the plugin runs as a separate process or not; so this factor does not affect how you write the plugin. This document focuses on plugins that run as separate processes, since in general that is the case for third-party plugins.

The figure below illustrates the media plugin architecture for a single plugin. In general, there may be multiple plugins running at one time.

Plugin loader shell

The plugin loader shell (PLS) loads and hosts plugins and provides for communication with the Viewer process.

The PLS is a separate process (for example, slplugin.exe on Windows), which makes it is easier to navigate through operating system firewalls (and other programs that limit outgoing network connections), since the end-user needs to unblock it only once.

The Viewer instructs the PLS to load a given plugin via internal messages once it establishes the control channel. The PLS manages relative priorities and CPU usage of plugins by setting the operating system priority of the process and throttling messages to the plugin.

The Viewer communicates with the PLS via internal messages. The PLS handles certain internal messages (some shared memory setup and process priority control) instead of passing them through to the plugin.

Messages

All communication in the system is via messages. Messages consist of a message class and name (both human-readable) and a collection of data represented by LLSD, so values may be rich data types such as arrays or other containers. The plugin and the PLS serialize and send messages back and forth using the control channel, a local TCP socket.

Messages are self-contained and cannot pass pointers, since in general they are serialized and passed across process boundaries.

Individual messages are unidirectional; messages are sent:

From the PLS to the plugin.
From the plugin to the PLS.

Additionally, The Viewer and the PLS exchange internal messages.

Sending a message to a plugin may cause it to send a message back to the PLS as a response, depending on the contents of the message. Plugins may also send unsolicited messages to the plugin loader shell (i.e. messages that aren't responses to a particular message the plugin received).

Messages are completely asynchronous. They may be queued for an arbitrary amount of time before they're delivered, so it's not possible to directly query a plugin and get an answer back. The command sets for specific plugin types need to take this into account: instead of having commands to query for the state of a plugin, the plugin should send unsolicited messages to the host when its state changes in meaningful ways, so that the PLS can maintain a local shadow of that state from which it will answer queries. The proxy object handles this logic, as well as message encoding and decoding.

Messages are guaranteed to be delivered in the same order they were sent. If a plugin crashes, the PLS notifies any clients with an interface to that plugin.

Sending a message to a plugin may cause it to send a response. Whether it does depends on the semantics of the particular message. Plugins can also send unsolicited messages (messages that are not a direct response to another message). Such messages are used to keep the plugin's proxy object's cache of the plugin's state up to date, among other things.

Shared Memory

The Viewer sets up memory segments shared with plugin processes. Shared memory segments are identified by a name unique within each instance of the plugin. See Project Notes for more information.

Plugins can negotiate (via specific messages) for the host to set up segments of shared memory between the plugin and the viewer. When plugins are hosted by the plugin loader shell, these will be interprocess shared memory segments. (When directly loaded by the viewer, they will be simple pointers, but they will still negotiate setup in the same way, and the pointer will be owned by the Viewer.) This is especially useful for media plugins, since they need to share large amounts of data with the viewer (specifically, pixel data containing rendered media).

Media plugins

A media plugin instance is dedicated to a particular media stream. A shared memory buffer is allocated to the stream, and the stream renders into the buffer continuously.

The internet media type (aka MIME type) defines the plugin can render a given media type. An XML file in the Viewer installation specifies which internet media type maps to which media plugin.

However, internet media type is not the only factor that determines which plugin renders a given media type. While a plugin specifies the types of media it can render, the Viewer determines which plugin to use based on other factors (beyond the scope of this document), such as user preferences, presence of other plugins, and so on. Thus, while you as a plugin developer indicate what media your plugin is capable of rendering, you cannot guarantee that a given media type will be rendered by the plugin.

Data flow

Typical data flow when the Viewer wants to render a media URI:

The Viewer retrieves the MIME type of the resource with an asynchronous HTTP HEAD request. If there is no MIME type associated with this URI (a VNC session for example) then it uses type or subtype in the "x-" or "vnd-" range.
Later, when the MIME type is known, the Viewer uses type and subtype fields to determine which media plugin to use via a lookup in an XML file.
The Viewer creates an "instance" of the relevant plugin.
When the plugin is fully created and initialized, it informs the Viewer that it is ready.
Viewer asks the plugin to start rendering data.
Plugin receives the message and starts to render the media.
When the media data being rendered changes, the plugin tells the Viewer that something changed and the Viewer updates itself accordingly.

LSL access

The system will also provide access via Linden Scripting Language (LSL) to media data on inworld objects (prims). Media data includes things like media's default URL, current URL, height and width, auto-play setting, permissions, and so on.

The basic LSL operations provided are:

Set media params on a prim face, initializing it if not there.
Get media params on a prim face.
Clear media from a prim face.

Code

Latest branch: https://svn.secondlife.com/svn/linden/branches/2009/plugin-api/

After checking out the branch, run develop.py normally, then build/run the "media_plugin_test" target in the SecondLife project.