Linden Lab Official:Media Rendering Plugin System Technical Overview
System overview
The Second Life media rendering plugin system enables the Second Life Viewer to render rich media both inworld via parcel media and in the embedded web browser via media rendering plugins. A media rendering plugin is a modular software library created to display a specific media type. The Second Life Viewer incorporates the media rendering plugin system to provide built-in support for a variety media types as well as an open API for developers to create plugins for additional media types. The media rendering plugins provided with the current Viewer include QuickTime and HTML rendering using WebKit, and GStreamer.
Using the media rendering plugin system, developers can create Viewer plugins to render a variety of media types. This open API enables interesting applications and use cases, for example: shared desktops, native document display, third-party application integration, etc.
System architecture
Each plugin runs as a separate process, so that if the plugin crashes, the Viewer does not.
System diagram
This diagram illustrates the media rendering plugin system architecture, with multiple plugins running.
System components
This section describes the role and characteristics of each system component in the system diagram. For more information on how the components work together during different stages of plugin operation, see Media Rendering Plugin Operation and Data Flow.
Media rendering plugin
A plugin is a platform-native dynamic library (.DLL on Windows, .dylib on Mac OS, and .so on Linux). It can be built independently of the Viewer.
A media rendering plugin is dedicated to rendering a particular media stream. A shared memory buffer is allocated to the media stream, and the stream writes into the buffer continuously.
Each media rendering plugin can render a specific internet media type (MIME type). An XML file in the Viewer installation specifies the mappings between internet media types and specific media rendering plugins. A URI scheme (for example, RTSP or HTTP) can also specify which plugin to use.
Although a plugin specifies the types of media it can render, the Viewer determines which plugin to use in any given instance based on other factors, such as user preferences, presence of other plugins, and others beyond the scope of this document. Thus, while you as a plugin developer indicate what media your plugin is capable of rendering, you cannot guarantee that a given media type will be rendered by the plugin.
Plugin loader shell: SLPlugin
A plugin loader shell (PLS) loads and hosts each plugin, and provides for communication with the Viewer process.
The PLS is a separate executable (SLPlugin.exe on Windows, SLPlugin on Mac and Linux). One PLS process instance runs for each plugin instance, and one plugin instance runs for each instance of media.
The Viewer instructs a PLS to load a given plugin via internal messages once it establishes the control channel. The PLS helps manage relative priorities and CPU usage by setting the operating system priority of the process and throttling messages to the plugin.
The Viewer communicates with each PLS via internal messages. The PLS independently handles internal system messages to and from the Viewer (some shared memory setup and process priority control) without intervention from the plugin.
Viewer
The Viewer is responsible for rendering all inworld elements to the display. The Viewer renders media rendering plugin content by reading and processing data the plugin writes to its shared memory segment.
Proxy object
A proxy object is an object in the Viewer which has detailed knowledge of the behaviors and messages of a particular plugin instance. Each plugin instance has a corresponding proxy object in the viewer.
Any part of the Viewer that needs to use the services of a plugin does so by interacting with the plugin's proxy object. The proxy object provides any necessary interfaces for interacting with the plugin, and mirrors any plugin state information that is useful to clients of that plugin.
The proxy object may persist for some time after the plugin instance goes away, so that all clients of the plugin can be cleanly notified. If a plugin crashes and is relaunched, the proxy object doesn't need to be destroyed, although any clients using the proxy object will be notified that the plugin's state has been reset.
Each proxy object can cause messages to be sent to the plugin for which it is a proxy. Likewise, each proxy object has a mechanism (a listener-type interface) that allows other code to sign up to receive relevant messages from the plugin.
The plugin enables the proxy object to maintain accurate state information by sending messages to the proxy object whenever its state changes in ways meaningful to its clients. State queries can then be answered directly by the proxy object without having to wait for a round-trip query via messages.
The proxy object builds up messages (for outgoing messages) and decodes messages (for incoming messages) internally, so that the details of message formats are hidden from code that doesn't need to know about them. If the semantics of messages need to change over time, the proxy object is where compatibility with older plugins/message formats will be maintained.
Messages
All communication in the system is done via messages.
Messages consist of a message class, name, and a collection of data represented by LLSD, so values can be rich data types such as arrays or other containers. The plugin and the plugin loader shell send messages back and forth using the control channel, a local TCP socket. The Viewer and the PLS also exchange internal messages.
Messages are asynchronous. They may be queued for an arbitrary amount of time before they're delivered, so it's not possible to directly query a plugin and get an answer back immediately. Instead of the Viewer or PLS querying for the state of a plugin, the plugin sends status update messages to the PLS when its state changes, so that the PLS can notify the proxy object in the Viewer. The proxy object maintains a local shadow of the plugin state for the Viewer to query.
Some types of messages sent from the PLS to the plugin require the plugin to send a response to the PLS. Plugins can also send unsolicited messages to the PLS (i.e. messages that aren't responses to a particular message the plugin received).
Messages are guaranteed to be delivered in the same order they were sent. If a plugin crashes, the PLS notifies any client with an interface to that plugin.
Plugins should not try to pass pointers via messages, since plugins run as a separate process, and a pointer in one process is meaningless in another. The exception is shared memory messages, which convey memory pointers between a PLS and plugin.
Control channel
The Viewer and plugin loader shell use the control channel to pass messages back and forth.
To establish the control channel, the Viewer listens on a local TCP socket, and passes the socket's port to the PLS. The PLS connects back to the indicated local TCP port.
To keep latency low, the system does not use the control channel for bulk data transfer. If large amounts of data need to be passed between the plugin host and a plugin, that data should not be embedded in messages sent across the control channel. Instead use either shared memory or separate streams negotiated via messages.
The Viewer sets up memory segments that are shared with plugin processes. Shared memory segments can be created, resized, and removed.
A plugin instance sends requests to its plugin loader shell to set up an interprocess shared memory segment between itself and the Viewer. The memory segment is used to share large amounts of data, such as rendered pixel data or audio data.
A plugin can have more than one shared memory segment, each of which is identified by name. The names are implementation-defined and different on each platorm, but will always be unique across all plugin instances running on a machine.
Shared memory setup follows the same rules as any message exchange: when a plugin sends a request to create or modify a segment, it must receive a confirmation message before proceeding as if the change has been made.
A plugin exchanges messages with its PLS to set up and remove shared memory segments. The PLS modifies and forwards the shared memory messages to and from the Viewer -- messages do not go directly from the plugin to the Viewer.
There are native implementations for the shared memory segment with a lightweight platform abstraction on Windows, Mac and Linux. These use the CreateFileMapping API on Windows and either mmap() or shm_open() on Mac and Linux. The details of this implementation are hidden from the plugins, since they reside in the Viewer and plugin loader shell.
Possibilities for future development
The media rendering plugin system is the first use of a general Viewer plugin architecture that in the future may be used for other types of plugins, for example to support additional input devices, image decoding (for example, JPEG2000), and voice services.
This document does not address a system for distributing plugins automatically, for example in the way that Firefox add-ons work.