Difference between revisions of "Linden Lab Official:Media Rendering Plugin System Technical Overview"

From Second Life Wiki
Jump to navigation Jump to search
 
(6 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Media Plugin Nav}}
{{:API Portal/navigation|media}}
__TOC__
__TOC__
== Overview ==


The Second Life media plugin system enables the Second Life Viewer to render rich media both inworld (on prims) and in the embedded web browser via ''media rendering plugins''.  A media rendering plugin is a modular software library created to display a specific media type.  The Second Life Viewer will incorporate the media plugin system to provide built-in support for a variety media types as well as an open API for developers to create plugins for additional media types.  The initial built-in media plugins include QuickTime and HTML rendering using WebKit.
== System overview ==


Using the media plugin system, developers will be able to create Viewer plugins to render a variety of media types.  This open API will enable interesting applications and use cases, for example: shared desktops, native document display, third-party application integration, and so on.
The Second Life media rendering plugin system enables the Second Life Viewer to render rich media both inworld via [[Parcel media|parcel media]] and in the embedded web browser via ''media rendering plugins''. A media rendering plugin is a modular software library created to display a specific media type. The Second Life Viewer incorporates the media rendering plugin system to provide built-in support for a variety media types as well as an open API for developers to create plugins for additional media types. The media rendering plugins provided with the current Viewer include QuickTime and HTML rendering using WebKit, and GStreamer.


Additionally, the media plugin system is the first use of a general Viewer plugin architecture that in the future may be used for other types of plugins, for example to support additional input devices, image decoding (for example, JPEG2000), and voice services.
Using the media rendering plugin system, developers can create Viewer plugins to render a variety of media types. This open API enables interesting applications and use cases, for example: shared desktops, native document display, third-party application integration, etc.


This design does ''not'' address a system for distributing plugins automatically, for example in the way that Firefox add-ons work. Clearly, something similar will be required eventually but is beyond the scope of this document.
== System architecture ==


=== Advantages ===
Each plugin runs as a separate process, so that if the plugin crashes, the Viewer does not.


In this system, each plug-in runs as a separate process, which has the following advantages:
=== System diagram ===
* '''Improves Viewer stability''': if the plugin crashes, the Viewer does not, though it may have invalid (or generic) media data.  Currently, a small but significant portion of Viewer crashes occur when a media component crashes.
This diagram illustrates the media rendering plugin system architecture, with multiple plugins running.
* '''Eliminates build-time dependencies''', making it easier to author plugins.
* '''Eases plugin development''':
** Plugin developers do not need to rebuild the entire Viewer, a complex and time-consuming task.
**  Plugin developers can use different compiler versions, perhaps even different languages.


Additionally, the architecture:
[[Image:Media rendering plugin system.png|Media rendering plugin system diagram]]
* '''Takes advantage of multi-core systems'''.
* '''Facilitates support for multiple media sources''' per parcel or per prim.


== Plugin system architecture ==
=== System components ===
This section describes the role and characteristics of each system component in the system diagram. For more information on how the components work together during different stages of plugin operation, see [[Media Rendering Plugin Operation and Data Flow]].


A ''plugin'' is a platform-native dynamic library (.DLL on Windows, .dylib on Mac OS, and .so on Linux). 
==== Media rendering plugin ====


Although the architecture provides for plugins to load directly into the Viewer process (for example for certain plugins developed by Linden Lab), externally-developed plugins generally run as a separate process to enhance Viewer stability.
A plugin is a platform-native dynamic library (.DLL on Windows, .dylib on Mac OS, and .so on Linux). It can be built independently of the Viewer.


NOTE: You write to the same API whether the plugin runs as a separate process or not; so this factor does not affect how you write the plugin.
A ''media rendering plugin'' is dedicated to rendering a particular media stream. A shared memory buffer is allocated to the media stream, and the stream writes into the buffer continuously.  
This document focuses on plugins that run as separate processes, since in general that is the case for third-party plugins.


The figure below illustrates the media plugin architecture for a single plugin. In general, there may be multiple plugins running at one time.
Each media rendering plugin can render a specific [http://en.wikipedia.org/wiki/Internet_media_type internet media type] (MIME type). An XML file in the Viewer installation specifies the mappings between internet media types and specific media rendering plugins. A URI scheme (for example, RTSP or HTTP) can also specify which plugin to use.


[[Image:Media API.png|right]]
Although a plugin specifies the types of media it can render, the Viewer determines which plugin to use in any given instance based on other factors, such as user preferences, presence of other plugins, and others beyond the scope of this document. Thus, while you as a plugin developer indicate what media your plugin is capable of rendering, you cannot guarantee that a given media type will be rendered by the plugin.


=== Plugin loader shell ===
==== Plugin loader shell: SLPlugin ====


The plugin loader shell (PLS) loads and hosts plugins and provides for communication with the Viewer process.
A plugin loader shell (PLS) loads and hosts each plugin, and provides for communication with the Viewer process.  


The PLS is a separate process (for example, slplugin.exe on Windows), which makes it is easier to navigate through operating system firewalls (and other programs that limit outgoing network connections), since the end-user needs to unblock it only once. One PLS instance runs for each plugin running.
The PLS is a separate executable (SLPlugin.exe on Windows, SLPlugin on Mac and Linux). One PLS process instance runs for each plugin instance, and one plugin instance runs for each instance of media.  


The Viewer instructs the PLS to load a given plugin via internal messages once it establishes the control channel. The PLS manages relative priorities and CPU usage of plugins by setting the operating system priority of the process and throttling messages to the plugin.
The Viewer instructs a PLS to load a given plugin via internal messages once it establishes the control channel. The PLS helps manage relative priorities and CPU usage by setting the operating system priority of the process and throttling messages to the plugin.


The Viewer communicates with the PLS via [[Media Plugin System Internal Messages|internal messages]]. The PLS handles certain internal messages (some shared memory setup and process priority control) instead of passing them through to the plugin.
The Viewer communicates with each PLS via [[Media Plugin System Internal_Messages|internal messages]]. The PLS independently handles internal system messages to and from the Viewer (some shared memory setup and process priority control) without intervention from the plugin.


=== Messages ===
==== Viewer ====


All communication in the system is via ''messages''. Messages consist of a message class and name (both human-readable) and a collection of data represented by [[LLSD]], so values may be rich data types such as arrays or other containers. 
The Viewer is responsible for rendering all inworld elements to the display. The Viewer renders media rendering plugin content by reading and processing data the plugin writes to its [[#Shared memory | shared memory segment]].
The plugin and the PLS serialize and send messages back and forth using the [[Media Plugin System Project Notes#Control_channel|control channel]], a local TCP socket.  


Messages are self-contained and cannot pass pointers, since in general they are serialized and passed across process boundaries. 
==== Proxy object ====


Individual messages are unidirectional; messages are sent:
A proxy object is an object in the Viewer which has detailed knowledge of the behaviors and messages of a particular plugin instance. Each plugin instance has a corresponding proxy object in the viewer.
* From the PLS to the plugin.
* From the plugin to the PLS.


Additionally, The Viewer and the PLS exchange [[Media Plugin System Internal Messages|internal messages]].
Any part of the Viewer that needs to use the services of a plugin does so by interacting with the plugin's proxy object. The proxy object provides any necessary interfaces for interacting with the plugin, and mirrors any plugin state information that is useful to clients of that plugin.


Sending a message to a plugin ''may'' cause it to send a message back to the PLS as a response, depending on the contents of the message. Plugins may also send unsolicited messages to the plugin loader shell (i.e. messages that aren't responses to a particular message the plugin received).
The proxy object may persist for some time after the plugin instance goes away, so that all clients of the plugin can be cleanly notified. If a plugin crashes and is relaunched, the proxy object doesn't need to be destroyed, although any clients using the proxy object will be notified that the plugin's state has been reset.


Messages are completely asynchronous.  They may be queued for an arbitrary amount of time before they're delivered, so it's not possible to directly query a plugin and get an answer back. The command sets for specific plugin types need to take this into account: instead of having commands to query for the state of a plugin, the plugin should send unsolicited messages to the host when its state changes in meaningful ways, so that the PLS can maintain a local shadow of that state from which it will answer queries.  The [[Media Plugin System Project Notes#Proxy_object|proxy object]] handles this logic, as well as message encoding and decoding.
Each proxy object can cause messages to be sent to the plugin for which it is a proxy. Likewise, each proxy object has a mechanism (a listener-type interface) that allows other code to sign up to receive relevant messages from the plugin.


Messages are guaranteed to be delivered in the same order they were sent. If a plugin crashes, the PLS notifies any clients with an interface to that plugin.
The plugin enables the proxy object to maintain accurate state information by sending messages to the proxy object whenever its state changes in ways meaningful to its clients. State queries can then be answered directly by the proxy object without having to wait for a round-trip query via messages.


Sending a message to a plugin may cause it to send a response.  Whether it does depends on the semantics of the particular message.  Plugins can also send unsolicited messages (messages that are not a direct response to another message). Such messages are used to keep the plugin's proxy object's cache of the plugin's state up to date, among other things.
The proxy object builds up messages (for outgoing messages) and decodes messages (for incoming messages) internally, so that the details of message formats are hidden from code that doesn't need to know about them. If the semantics of messages need to change over time, the proxy object is where compatibility with older plugins/message formats will be maintained.


=== Shared Memory ===
==== Messages ====
The Viewer sets up memory segments shared with plugin processes.  Shared memory segments are identified by a name unique within each instance of the plugin.  See [[Media_Plugin_System_Project_Notes#Shared_memory|Project Notes]] for more information.


Plugins can negotiate (via specific messages) for the host to set up segments of [[Media Plugin System Project Notes#Shared_Memory | shared memory]] between the plugin and the viewer.  When plugins are hosted by the plugin loader shell, these will be interprocess shared memory segments.  (When directly loaded by the viewer, they will be simple pointers, but they will still negotiate setup in the same way, and the pointer will be owned by the Viewer.)  This is especially useful for media plugins, since they need to share large amounts of data with the viewer (specifically, pixel data containing rendered media).
All communication in the system is done via [[Media Plugin System Messages|messages]].  


== Media plugins ==
Messages consist of a message class, name, and a collection of data represented by [[LLSD]], so values can be rich data types such as arrays or other containers. The plugin and the [[#Plugin loader shell : Slplugin | plugin loader shell]] send messages back and forth using the [[Media Plugin System Project Notes#Control_channel|control channel]], a local TCP socket. The Viewer and the PLS also exchange [[Media Plugin System Internal Messages|internal messages]].


A media plugin instance is dedicated to a particular media stream. A shared memory buffer is allocated to the stream, and the stream renders into the buffer continuously.
Messages are asynchronous. They may be queued for an arbitrary amount of time before they're delivered, so it's not possible to directly query a plugin and get an answer back immediately. Instead of the Viewer or PLS querying for the state of a plugin, the plugin sends status update messages to the PLS when its state changes, so that the PLS can notify the proxy object in the Viewer. The proxy object maintains a local shadow of the plugin state for the Viewer to query.


The [http://en.wikipedia.org/wiki/Internet_media_type internet media type] (aka MIME type) specifies the plugin that can render a given media type.  An XML file in the Viewer installation specifies which internet media type maps to which media plugin.  
Some types of messages sent from the PLS to the plugin require the plugin to send a response to the PLS. Plugins can also send unsolicited messages to the PLS (i.e. messages that aren't responses to a particular message the plugin received).
You can also use URI scheme (for example, RTSP or HTTP) to determine which plugin to use,.


However, while a plugin specifies the types of media it can render, the Viewer determines which plugin to use in any given instance based on other factors (beyond the scope of this document), such as user preferences, presence of other plugins, and so on. Thus, while you as a plugin developer indicate what media your plugin is capable of rendering, you cannot guarantee that a given media type will be rendered by the plugin.
Messages are guaranteed to be delivered in the same order they were sent. If a plugin crashes, the PLS notifies any client with an interface to that plugin.


=== Data flow ===
Plugins should not try to pass pointers via messages, since plugins run as a separate process, and a pointer in one process is meaningless in another. The exception is shared memory messages, which convey memory pointers between a PLS and plugin.


Typical data flow when the Viewer wants to render a media URI:
==== Control channel ====
 
The Viewer and [[#Plugin loader shell: SLPlugin|plugin loader shell]] use the control channel to pass [[Media Plugin System Messages|messages]] back and forth.
 
To establish the control channel, the Viewer listens on a local TCP socket, and passes the socket's port to the PLS. The PLS connects back to the indicated local TCP port.
 
To keep latency low, the system does not use the control channel for bulk data transfer. If large amounts of data need to be passed between the plugin host and a plugin, that data should ''not'' be embedded in messages sent across the control channel. Instead use either [[#Shared memory | shared memory]] or separate streams negotiated via messages.
 
==== Shared memory ====
 
The Viewer sets up memory segments that are shared with plugin processes. Shared memory segments can be created, resized, and removed.
 
A plugin instance sends requests to its [[#Plugin loader shell : SLPlugin | plugin loader shell]] to set up an interprocess shared memory segment between itself and the Viewer. The memory segment is used to share large amounts of data, such as rendered pixel data or audio data.
 
A plugin can have more than one shared memory segment, each of which is identified by name.  The names are implementation-defined and different on each platorm, but will always be unique across all plugin instances running on a machine.
 
Shared memory setup follows the same rules as any message exchange: when a plugin sends a request to create or modify a segment, it must receive a confirmation message before proceeding as if the change has been made.
 
A plugin exchanges messages with its PLS to set up and remove shared memory segments. The PLS modifies and forwards the shared memory messages to and from the Viewer -- messages do not go directly from the plugin to the Viewer.
 
===== Shared memory implementation =====
There are native implementations for the shared memory segment with a lightweight platform abstraction on Windows, Mac and Linux. These use the CreateFileMapping API on Windows and either mmap() or shm_open() on Mac and Linux. The details of this implementation are hidden from the plugins, since they reside in the Viewer and plugin loader shell.
 
== Possibilities for future development ==
The media rendering plugin system is the first use of a general Viewer plugin architecture that in the future may be used for other types of plugins, for example to support additional input devices, image decoding (for example, JPEG2000), and voice services.
 
This document does ''not'' address a system for distributing plugins automatically, for example in the way that Firefox add-ons work.


# Viewer retrieves the MIME type of the resource with an asynchronous HTTP [http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.4 HEAD] request. If there is no MIME type associated with this URI (a VNC session for example) then it uses type or subtype in the "x-" or "vnd-" range.  When the MIME type is known, the Viewer uses type and subtype fields to determine which media plugin to use via a lookup in an XML file.
# Viewer creates an "instance" of the relevant plugin.
# When the plugin is fully created and initialized, it informs the Viewer that it is ready.
# Viewer asks the plugin to start rendering data.
# Plugin receives the message and starts to render the media.
# When the media data being rendered changes, the plugin tells the Viewer that something changed and the Viewer updates itself accordingly.
[[Category:Media]]
[[Category:Media]]
[[Category:Design Documents]]

Latest revision as of 14:31, 4 May 2011

System overview

The Second Life media rendering plugin system enables the Second Life Viewer to render rich media both inworld via parcel media and in the embedded web browser via media rendering plugins. A media rendering plugin is a modular software library created to display a specific media type. The Second Life Viewer incorporates the media rendering plugin system to provide built-in support for a variety media types as well as an open API for developers to create plugins for additional media types. The media rendering plugins provided with the current Viewer include QuickTime and HTML rendering using WebKit, and GStreamer.

Using the media rendering plugin system, developers can create Viewer plugins to render a variety of media types. This open API enables interesting applications and use cases, for example: shared desktops, native document display, third-party application integration, etc.

System architecture

Each plugin runs as a separate process, so that if the plugin crashes, the Viewer does not.

System diagram

This diagram illustrates the media rendering plugin system architecture, with multiple plugins running.

Media rendering plugin system diagram

System components

This section describes the role and characteristics of each system component in the system diagram. For more information on how the components work together during different stages of plugin operation, see Media Rendering Plugin Operation and Data Flow.

Media rendering plugin

A plugin is a platform-native dynamic library (.DLL on Windows, .dylib on Mac OS, and .so on Linux). It can be built independently of the Viewer.

A media rendering plugin is dedicated to rendering a particular media stream. A shared memory buffer is allocated to the media stream, and the stream writes into the buffer continuously.

Each media rendering plugin can render a specific internet media type (MIME type). An XML file in the Viewer installation specifies the mappings between internet media types and specific media rendering plugins. A URI scheme (for example, RTSP or HTTP) can also specify which plugin to use.

Although a plugin specifies the types of media it can render, the Viewer determines which plugin to use in any given instance based on other factors, such as user preferences, presence of other plugins, and others beyond the scope of this document. Thus, while you as a plugin developer indicate what media your plugin is capable of rendering, you cannot guarantee that a given media type will be rendered by the plugin.

Plugin loader shell: SLPlugin

A plugin loader shell (PLS) loads and hosts each plugin, and provides for communication with the Viewer process.

The PLS is a separate executable (SLPlugin.exe on Windows, SLPlugin on Mac and Linux). One PLS process instance runs for each plugin instance, and one plugin instance runs for each instance of media.

The Viewer instructs a PLS to load a given plugin via internal messages once it establishes the control channel. The PLS helps manage relative priorities and CPU usage by setting the operating system priority of the process and throttling messages to the plugin.

The Viewer communicates with each PLS via internal messages. The PLS independently handles internal system messages to and from the Viewer (some shared memory setup and process priority control) without intervention from the plugin.

Viewer

The Viewer is responsible for rendering all inworld elements to the display. The Viewer renders media rendering plugin content by reading and processing data the plugin writes to its shared memory segment.

Proxy object

A proxy object is an object in the Viewer which has detailed knowledge of the behaviors and messages of a particular plugin instance. Each plugin instance has a corresponding proxy object in the viewer.

Any part of the Viewer that needs to use the services of a plugin does so by interacting with the plugin's proxy object. The proxy object provides any necessary interfaces for interacting with the plugin, and mirrors any plugin state information that is useful to clients of that plugin.

The proxy object may persist for some time after the plugin instance goes away, so that all clients of the plugin can be cleanly notified. If a plugin crashes and is relaunched, the proxy object doesn't need to be destroyed, although any clients using the proxy object will be notified that the plugin's state has been reset.

Each proxy object can cause messages to be sent to the plugin for which it is a proxy. Likewise, each proxy object has a mechanism (a listener-type interface) that allows other code to sign up to receive relevant messages from the plugin.

The plugin enables the proxy object to maintain accurate state information by sending messages to the proxy object whenever its state changes in ways meaningful to its clients. State queries can then be answered directly by the proxy object without having to wait for a round-trip query via messages.

The proxy object builds up messages (for outgoing messages) and decodes messages (for incoming messages) internally, so that the details of message formats are hidden from code that doesn't need to know about them. If the semantics of messages need to change over time, the proxy object is where compatibility with older plugins/message formats will be maintained.

Messages

All communication in the system is done via messages.

Messages consist of a message class, name, and a collection of data represented by LLSD, so values can be rich data types such as arrays or other containers. The plugin and the plugin loader shell send messages back and forth using the control channel, a local TCP socket. The Viewer and the PLS also exchange internal messages.

Messages are asynchronous. They may be queued for an arbitrary amount of time before they're delivered, so it's not possible to directly query a plugin and get an answer back immediately. Instead of the Viewer or PLS querying for the state of a plugin, the plugin sends status update messages to the PLS when its state changes, so that the PLS can notify the proxy object in the Viewer. The proxy object maintains a local shadow of the plugin state for the Viewer to query.

Some types of messages sent from the PLS to the plugin require the plugin to send a response to the PLS. Plugins can also send unsolicited messages to the PLS (i.e. messages that aren't responses to a particular message the plugin received).

Messages are guaranteed to be delivered in the same order they were sent. If a plugin crashes, the PLS notifies any client with an interface to that plugin.

Plugins should not try to pass pointers via messages, since plugins run as a separate process, and a pointer in one process is meaningless in another. The exception is shared memory messages, which convey memory pointers between a PLS and plugin.

Control channel

The Viewer and plugin loader shell use the control channel to pass messages back and forth.

To establish the control channel, the Viewer listens on a local TCP socket, and passes the socket's port to the PLS. The PLS connects back to the indicated local TCP port.

To keep latency low, the system does not use the control channel for bulk data transfer. If large amounts of data need to be passed between the plugin host and a plugin, that data should not be embedded in messages sent across the control channel. Instead use either shared memory or separate streams negotiated via messages.

Shared memory

The Viewer sets up memory segments that are shared with plugin processes. Shared memory segments can be created, resized, and removed.

A plugin instance sends requests to its plugin loader shell to set up an interprocess shared memory segment between itself and the Viewer. The memory segment is used to share large amounts of data, such as rendered pixel data or audio data.

A plugin can have more than one shared memory segment, each of which is identified by name. The names are implementation-defined and different on each platorm, but will always be unique across all plugin instances running on a machine.

Shared memory setup follows the same rules as any message exchange: when a plugin sends a request to create or modify a segment, it must receive a confirmation message before proceeding as if the change has been made.

A plugin exchanges messages with its PLS to set up and remove shared memory segments. The PLS modifies and forwards the shared memory messages to and from the Viewer -- messages do not go directly from the plugin to the Viewer.

Shared memory implementation

There are native implementations for the shared memory segment with a lightweight platform abstraction on Windows, Mac and Linux. These use the CreateFileMapping API on Windows and either mmap() or shm_open() on Mac and Linux. The details of this implementation are hidden from the plugins, since they reside in the Viewer and plugin loader shell.

Possibilities for future development

The media rendering plugin system is the first use of a general Viewer plugin architecture that in the future may be used for other types of plugins, for example to support additional input devices, image decoding (for example, JPEG2000), and voice services.

This document does not address a system for distributing plugins automatically, for example in the way that Firefox add-ons work.