Difference between revisions of "Linden Lab Official:Media Rendering Plugin System Technical Overview"

From Second Life Wiki
Jump to navigation Jump to search
 
(35 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Media Plugin Nav}}
{{:API Portal/navigation|media}}
__TOC__
__TOC__
== Overview ==


'''Goal''': To facilitate easier integration of external media into the Second Life Viewer.
== System overview ==


This document presents two separate (although related) designs:
The Second Life media rendering plugin system enables the Second Life Viewer to render rich media both inworld via [[Parcel media|parcel media]] and in the embedded web browser via ''media rendering plugins''. A media rendering plugin is a modular software library created to display a specific media type. The Second Life Viewer incorporates the media rendering plugin system to provide built-in support for a variety media types as well as an open API for developers to create plugins for additional media types. The media rendering plugins provided with the current Viewer include QuickTime and HTML rendering using WebKit, and GStreamer.
* A general-purpose [[#Plugin_Architecture|plugin architecture]] for the Second Life Viewer.
* Specific use of the plugin architecture for [[#Media_Plugins|media rendering plugins]] within the Second Life Viewer.


Plugins of different types can use the plugin architecture.  Although the initial implementation is for media plugins, potentially it could also be used for:
Using the media rendering plugin system, developers can create Viewer plugins to render a variety of media types. This open API enables interesting applications and use cases, for example: shared desktops, native document display, third-party application integration, etc.
* Input devices
* Image decoding (JPEG2000)
* Voice services


In the future, developers may wish to load plugins directly into the Viewer's address space.  This design supports that mode of operation, but the initial implementation loads all plugins as external processes.
== System architecture ==


Loading media rendering plug-ins ras separate processeses:
Each plugin runs as a separate process, so that if the plugin crashes, the Viewer does not.
* '''Improves stability''' in Second Life client: if the plugin crashes, the Viewer is able to continue, albeit with invalid (or generic) media data.  Currently, a small but significant portion of Viewer crashes occur when a media component crashes.
* '''Eliminates build-time dependencies''', making it easier to author plugins.
* '''Eases plugin development''':
** Plugin developers will not need to rebuild the entire Viewer, a complex and time-consuming task.
**  Plugin developers will be able to use different compiler versions, perhaps even different languages.


The asynchronous nature of the architecture:
=== System diagram ===
* '''Takes advantage of multi-core systems''' (generally speaking).
This diagram illustrates the media rendering plugin system architecture, with multiple plugins running.
* '''Facilitates support for multiple media sources''' per parcel or per prim.


=== Why? ===
[[Image:Media rendering plugin system.png|Media rendering plugin system diagram]]


Implementing rendering for media types in separate plugins makes each easier to create, modify and maintain.
=== System components ===
This section describes the role and characteristics of each system component in the system diagram. For more information on how the components work together during different stages of plugin operation, see [[Media Rendering Plugin Operation and Data Flow]].
Given the creativity and ingenuity of Residents, an increased selection of robust media implementations in Second Life will enable more interesting applications and use cases, for example, shared desktops, native document display, third-party application integration, and so on.


This redesign also:
==== Media rendering plugin ====


* Facilitates moving towards a more fine-grained architecture to ease maintenance and testing of the Second Life Viewer.
A plugin is a platform-native dynamic library (.DLL on Windows, .dylib on Mac OS, and .so on Linux). It can be built independently of the Viewer.
* Enables similar changes to filter through to other parts of the Second Life code base; for example, image support (J2K image modules) and user input devices (for example Space Navigator and VR Goggles).
* Allows for multiple implementations per media type and/or per platform (Mozilla/IE/WebKit for the embedded browser for example).


This design does ''not'' address a system for distributing plugins automatically, for example in the way that Firefox add-ons work. Clearly, something similar will be required eventually but is beyond the scope of this document.
A ''media rendering plugin'' is dedicated to rendering a particular media stream. A shared memory buffer is allocated to the media stream, and the stream writes into the buffer continuously.  


=== Terminology ===
Each media rendering plugin can render a specific [http://en.wikipedia.org/wiki/Internet_media_type internet media type] (MIME type). An XML file in the Viewer installation specifies the mappings between internet media types and specific media rendering plugins. A URI scheme (for example, RTSP or HTTP) can also specify which plugin to use.
In this document "media" refers to QuickTime media and the in-Viewer web browser (currently implemented by LLMedia and its subclasses), as well as other media types which may be added in the future.


The term "client" refers to code in the Viewer that uses the services of a plugin.
Although a plugin specifies the types of media it can render, the Viewer determines which plugin to use in any given instance based on other factors, such as user preferences, presence of other plugins, and others beyond the scope of this document. Thus, while you as a plugin developer indicate what media your plugin is capable of rendering, you cannot guarantee that a given media type will be rendered by the plugin.


=== Code ===
==== Plugin loader shell: SLPlugin ====


Latest branch: '''https://svn.secondlife.com/svn/linden/branches/2009/plugin-api/'''
A plugin loader shell (PLS) loads and hosts each plugin, and provides for communication with the Viewer process.  


After checking out the branch, run develop.py normally, then build/run the "media_plugin_test" target in the SecondLife project.
The PLS is a separate executable (SLPlugin.exe on Windows, SLPlugin on Mac and Linux). One PLS process instance runs for each plugin instance, and one plugin instance runs for each instance of media.  


== Technical overview ==
The Viewer instructs a PLS to load a given plugin via internal messages once it establishes the control channel. The PLS helps manage relative priorities and CPU usage by setting the operating system priority of the process and throttling messages to the plugin.


Plugin implementations run as a separate process. A plugin and the Viewer exchange pixel data using shared memory.
The Viewer communicates with each PLS via [[Media Plugin System Internal_Messages|internal messages]]. The PLS independently handles internal system messages to and from the Viewer (some shared memory setup and process priority control) without intervention from the plugin.


Communication between a plugin and the Viewer is:
==== Viewer ====
* Bi-directional via a local TCP channel.
* Message-based and completely asynchronous.


Typical data flow when the Viewer wants to render a media URI:
The Viewer is responsible for rendering all inworld elements to the display. The Viewer renders media rendering plugin content by reading and processing data the plugin writes to its [[#Shared memory | shared memory segment]].


* The Viewer retrieves the MIME type of the resource with an asynchronous HTTP [http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.4 HEAD] request. If there is no MIME type associated with this URI (a VNC session for example) then it uses type or subtype in the "x-" or "vnd-" range.
==== Proxy object ====
* Later, when the MIME type is known, the Viewer uses type and subtype fields to determine which media plugin to use via a lookup in an XML file.
* The Viewer creates an "instance" of the relevant plugin.
* When the plugin is fully created and initialized, it informs the Viewer that it is ready.
* Viewer asks the plugin to start rendering data.
* Plugin receives the message and starts to render the media.
* When the media data being rendered changes, the plugin tells the Viewer that something changed and the Viewer updates itself accordingly.


A proxy object is an object in the Viewer which has detailed knowledge of the behaviors and messages of a particular plugin instance. Each plugin instance has a corresponding proxy object in the viewer.


[[Image:LLPlugin_Diagram.png]]
Any part of the Viewer that needs to use the services of a plugin does so by interacting with the plugin's proxy object. The proxy object provides any necessary interfaces for interacting with the plugin, and mirrors any plugin state information that is useful to clients of that plugin.


== Plugin architecture ==
The proxy object may persist for some time after the plugin instance goes away, so that all clients of the plugin can be cleanly notified. If a plugin crashes and is relaunched, the proxy object doesn't need to be destroyed, although any clients using the proxy object will be notified that the plugin's state has been reset.


=== Plugin Architecture Overview ===
Each proxy object can cause messages to be sent to the plugin for which it is a proxy. Likewise, each proxy object has a mechanism (a listener-type interface) that allows other code to sign up to receive relevant messages from the plugin.


At the lowest level, a "plugin" is be a loadable piece of object code in platform-native format (.DLL, .so, .dylib).
The plugin enables the proxy object to maintain accurate state information by sending messages to the proxy object whenever its state changes in ways meaningful to its clients. State queries can then be answered directly by the proxy object without having to wait for a round-trip query via messages.


Although the architecture provides for plugins to load directly into the Viewer process (for example for certain plugins developed by Linden Lab), externally-deveoloped plugsin generally run as a separate process to enhance stability. NOTE: You write to the same API whether the plugin runs as a separeate process or not; so this factor does not affect how you write the plugin.
The proxy object builds up messages (for outgoing messages) and decodes messages (for incoming messages) internally, so that the details of message formats are hidden from code that doesn't need to know about them. If the semantics of messages need to change over time, the proxy object is where compatibility with older plugins/message formats will be maintained.
This document focuses on plugins that run as separate processes, since in general that is the case for third-party plugins.


The [[#Plugin_Loader_Shell|plugin loader shell]] loads plugins, provides for communication with the Viewer process, and acts as a host for the plugin.
==== Messages ====


Plugins communicate with the Viewer through [[#Messages | messages]]. Messages consist of a message class and name (both human-readable) and a collection of key-value pairs (values with human-readable names).  Messages are represented with LLSD, so values may be rich data types such as arrays or other containers. 
All communication in the system is done via [[Media Plugin System Messages|messages]].  


Individual messages are unidirectional, though messages are sent:
Messages consist of a message class, name, and a collection of data represented by [[LLSD]], so values can be rich data types such as arrays or other containers. The plugin and the [[#Plugin loader shell : Slplugin | plugin loader shell]] send messages back and forth using the [[Media Plugin System Project Notes#Control_channel|control channel]], a local TCP socket. The Viewer and the PLS also exchange [[Media Plugin System Internal Messages|internal messages]].
* From the plugin loader shell to the plugin.
* From the plugin to the plugin loader shell.


Sending a message to a plugin ''may'' cause it to send a message back to the PLS as a response, depending on the contents of the message.  Plugins may also send unsolicited messages to the plugin loader shell (i.e. messages that aren't responses to a particular message the plugin received).
Messages are asynchronous. They may be queued for an arbitrary amount of time before they're delivered, so it's not possible to directly query a plugin and get an answer back immediately. Instead of the Viewer or PLS querying for the state of a plugin, the plugin sends status update messages to the PLS when its state changes, so that the PLS can notify the proxy object in the Viewer. The proxy object maintains a local shadow of the plugin state for the Viewer to query.


The plugin and the PLS send messages back and forth using the [[#Control_Channel|control channel]], a local TCP socket. Since messages are passed over the control channel, they may need to be serialized.
Some types of messages sent from the PLS to the plugin require the plugin to send a response to the PLS. Plugins can also send unsolicited messages to the PLS (i.e. messages that aren't responses to a particular message the plugin received).


Generally the PLS and the plugin run in a different process space, so you cannot pass pointers in a message.
Messages are guaranteed to be delivered in the same order they were sent. If a plugin crashes, the PLS notifies any client with an interface to that plugin.


It's important to note that messages are completely asynchronous.  They may be queued for an arbitrary amount of time before they're delivered, so it's not possible to do a direct query of a plugin and get an answer back. The command sets for specific plugin types need to take this into account -- instead of having commands to query for the state of a plugin, the plugin should send unsolicited messages to the host when its state changes in meaningful ways, so that the PLS can maintain a local shadow of that state from which it will answer queries.  This logic, as well as the logic for message encoding/decoding, is handled by a [[#Proxy_Object|proxy object]] -- a class in the Viewer tailored for a particular type of plugin.
Plugins should not try to pass pointers via messages, since plugins run as a separate process, and a pointer in one process is meaningless in another. The exception is shared memory messages, which convey memory pointers between a PLS and plugin.


Plugins can negotiate (via specific messages) for the host to set up segments of [[#Shared_Memory | shared memory]] between the plugin and the viewer.  When plugins are hosted by the plugin loader shell, these will be interprocess shared memory segments.  (When directly loaded by the viewer, they will be simple pointers, but they will still negotiate setup in the same way, and the pointer will be owned by the Viewer.)  This is especially useful for media plugins, since they need to share large amounts of data with the viewer (specifically, pixel data containing rendered media).
==== Control channel ====


=== Plugin loader shell ===
The Viewer and [[#Plugin loader shell: SLPlugin|plugin loader shell]] use the control channel to pass [[Media Plugin System Messages|messages]] back and forth.


The plugin loader shell (PLS) is a separate process (slplugin.exe) that hosts all plugins and executes the code for a plugin as an external process.
To establish the control channel, the Viewer listens on a local TCP socket, and passes the socket's port to the PLS. The PLS connects back to the indicated local TCP port.


Since the PLS is a single executable, makes it easier for the end-user to navigate through the Windows firewall (and other programs which limit outgoing network connections), since the user only need to be unblock it one time.
To keep latency low, the system does not use the control channel for bulk data transfer. If large amounts of data need to be passed between the plugin host and a plugin, that data should ''not'' be embedded in messages sent across the control channel. Instead use either [[#Shared memory | shared memory]] or separate streams negotiated via messages.


Code in the PLS handles certain messages (such as certain parts of shared memory setup and process priority control) instead of passing them through to the plugin.
==== Shared memory ====


When the PLS is initially launched, it won't know which plugin it is expected to load. The Viewer supplies this information via internal messages once it establishes the control channel.
The Viewer sets up memory segments that are shared with plugin processes. Shared memory segments can be created, resized, and removed.


The PLS manages relative priorities and CPU usage of plugins by setting the operating system priority of the process and throttling messages to the plugin.
A plugin instance sends requests to its [[#Plugin loader shell : SLPlugin | plugin loader shell]] to set up an interprocess shared memory segment between itself and the Viewer. The memory segment is used to share large amounts of data, such as rendered pixel data or audio data.


A plugin can have more than one shared memory segment, each of which is identified by name.  The names are implementation-defined and different on each platorm, but will always be unique across all plugin instances running on a machine.


Shared memory setup follows the same rules as any message exchange: when a plugin sends a request to create or modify a segment, it must receive a confirmation message before proceeding as if the change has been made.


=== Messages ===
A plugin exchanges messages with its PLS to set up and remove shared memory segments. The PLS modifies and forwards the shared memory messages to and from the Viewer -- messages do not go directly from the plugin to the Viewer.
All communication with the plugin will be via ''messages''. At the binary/function call level, the only thing plugins can do is send and receive messages.  This simplifies the binary interface to plugins, and eliminates reasons to change the interface.


A message consists of a message class and message name (human-readable strings) and zero or more key-value pairs.  A key-value pair is a name (also human-readable) and some data. Messages are represented using [[LLSD]], and are serialized/deserialized using the existing mechanism for LLSD serialization.
===== Shared memory implementation =====
There are native implementations for the shared memory segment with a lightweight platform abstraction on Windows, Mac and Linux. These use the CreateFileMapping API on Windows and either mmap() or shm_open() on Mac and Linux. The details of this implementation are hidden from the plugins, since they reside in the Viewer and plugin loader shell.


Messages are self-contained and cannot pass pointers, since in general they are serialized and passed across process boundaries.
== Possibilities for future development ==
The media rendering plugin system is the first use of a general Viewer plugin architecture that in the future may be used for other types of plugins, for example to support additional input devices, image decoding (for example, JPEG2000), and voice services.


Individual messages are unidirectional, asynchronous, and may be queued for an unspecified amount of time before they're delivered.  Messages ARE guaranteed to be delivered in the same order they were sent. If a plugin crashes, any clients with an interface to that plugin will be notified, and must assume that any messages sent before the notification may have been lost
This document does ''not'' address a system for distributing plugins automatically, for example in the way that Firefox add-ons work.


Sending a message to a plugin may cause it to send a response.  Whether it does depends on the semantics of the particular message.  Plugins can also send unsolicited messages (messages that are not a direct response to another message).  Such messages are used to keep the plugin's proxy object's cache of the plugin's state up to date, among other things.
[[Category:Media]]
 
[[Category:Design Documents]]
=== Shared Memory ===
The Viewer sets up shared memory segments shared with  plugin processes.  Shared memory segments are identified by a name  unique within each instance of the plugin.  Memory segments can be created, resized (reallocated), and destroyed.
 
== Media Plugins ==
 
A media plugins instance is dedicated to a particular media stream.  A shared memory buffer is allocated to the stream, and it renders into it continuously.
 
* The [http://en.wikipedia.org/wiki/Internet_media_type internet media type] (aka MIME type) defines the media implementation used to render a given type.
 
* Internet media type is not the only factor that determines which plugin renders a given media type.  While a plugin specifies the types of media it can render, the Viewer determines which plugin to use based on other factors (beyond the scope of this document), such as user preferences, the presence of other plugins, and so on.  Thus, while you as a plugin developer indicate what media your plugin is capable of rendering, you cannot guarantee that a given media type will be rendered by the plugin.
 
* An XML file in the Viewer installation specifies which internet media type maps to which media plugin.
 
=== Plugin implementation ===
* A shared library that contains the code to render media
 
* One shared library per media implementation.
 
* If a plugin encounters invalid media data or network failures it can simply exit, since this will not affect the Viewer process.

Latest revision as of 14:31, 4 May 2011

System overview

The Second Life media rendering plugin system enables the Second Life Viewer to render rich media both inworld via parcel media and in the embedded web browser via media rendering plugins. A media rendering plugin is a modular software library created to display a specific media type. The Second Life Viewer incorporates the media rendering plugin system to provide built-in support for a variety media types as well as an open API for developers to create plugins for additional media types. The media rendering plugins provided with the current Viewer include QuickTime and HTML rendering using WebKit, and GStreamer.

Using the media rendering plugin system, developers can create Viewer plugins to render a variety of media types. This open API enables interesting applications and use cases, for example: shared desktops, native document display, third-party application integration, etc.

System architecture

Each plugin runs as a separate process, so that if the plugin crashes, the Viewer does not.

System diagram

This diagram illustrates the media rendering plugin system architecture, with multiple plugins running.

Media rendering plugin system diagram

System components

This section describes the role and characteristics of each system component in the system diagram. For more information on how the components work together during different stages of plugin operation, see Media Rendering Plugin Operation and Data Flow.

Media rendering plugin

A plugin is a platform-native dynamic library (.DLL on Windows, .dylib on Mac OS, and .so on Linux). It can be built independently of the Viewer.

A media rendering plugin is dedicated to rendering a particular media stream. A shared memory buffer is allocated to the media stream, and the stream writes into the buffer continuously.

Each media rendering plugin can render a specific internet media type (MIME type). An XML file in the Viewer installation specifies the mappings between internet media types and specific media rendering plugins. A URI scheme (for example, RTSP or HTTP) can also specify which plugin to use.

Although a plugin specifies the types of media it can render, the Viewer determines which plugin to use in any given instance based on other factors, such as user preferences, presence of other plugins, and others beyond the scope of this document. Thus, while you as a plugin developer indicate what media your plugin is capable of rendering, you cannot guarantee that a given media type will be rendered by the plugin.

Plugin loader shell: SLPlugin

A plugin loader shell (PLS) loads and hosts each plugin, and provides for communication with the Viewer process.

The PLS is a separate executable (SLPlugin.exe on Windows, SLPlugin on Mac and Linux). One PLS process instance runs for each plugin instance, and one plugin instance runs for each instance of media.

The Viewer instructs a PLS to load a given plugin via internal messages once it establishes the control channel. The PLS helps manage relative priorities and CPU usage by setting the operating system priority of the process and throttling messages to the plugin.

The Viewer communicates with each PLS via internal messages. The PLS independently handles internal system messages to and from the Viewer (some shared memory setup and process priority control) without intervention from the plugin.

Viewer

The Viewer is responsible for rendering all inworld elements to the display. The Viewer renders media rendering plugin content by reading and processing data the plugin writes to its shared memory segment.

Proxy object

A proxy object is an object in the Viewer which has detailed knowledge of the behaviors and messages of a particular plugin instance. Each plugin instance has a corresponding proxy object in the viewer.

Any part of the Viewer that needs to use the services of a plugin does so by interacting with the plugin's proxy object. The proxy object provides any necessary interfaces for interacting with the plugin, and mirrors any plugin state information that is useful to clients of that plugin.

The proxy object may persist for some time after the plugin instance goes away, so that all clients of the plugin can be cleanly notified. If a plugin crashes and is relaunched, the proxy object doesn't need to be destroyed, although any clients using the proxy object will be notified that the plugin's state has been reset.

Each proxy object can cause messages to be sent to the plugin for which it is a proxy. Likewise, each proxy object has a mechanism (a listener-type interface) that allows other code to sign up to receive relevant messages from the plugin.

The plugin enables the proxy object to maintain accurate state information by sending messages to the proxy object whenever its state changes in ways meaningful to its clients. State queries can then be answered directly by the proxy object without having to wait for a round-trip query via messages.

The proxy object builds up messages (for outgoing messages) and decodes messages (for incoming messages) internally, so that the details of message formats are hidden from code that doesn't need to know about them. If the semantics of messages need to change over time, the proxy object is where compatibility with older plugins/message formats will be maintained.

Messages

All communication in the system is done via messages.

Messages consist of a message class, name, and a collection of data represented by LLSD, so values can be rich data types such as arrays or other containers. The plugin and the plugin loader shell send messages back and forth using the control channel, a local TCP socket. The Viewer and the PLS also exchange internal messages.

Messages are asynchronous. They may be queued for an arbitrary amount of time before they're delivered, so it's not possible to directly query a plugin and get an answer back immediately. Instead of the Viewer or PLS querying for the state of a plugin, the plugin sends status update messages to the PLS when its state changes, so that the PLS can notify the proxy object in the Viewer. The proxy object maintains a local shadow of the plugin state for the Viewer to query.

Some types of messages sent from the PLS to the plugin require the plugin to send a response to the PLS. Plugins can also send unsolicited messages to the PLS (i.e. messages that aren't responses to a particular message the plugin received).

Messages are guaranteed to be delivered in the same order they were sent. If a plugin crashes, the PLS notifies any client with an interface to that plugin.

Plugins should not try to pass pointers via messages, since plugins run as a separate process, and a pointer in one process is meaningless in another. The exception is shared memory messages, which convey memory pointers between a PLS and plugin.

Control channel

The Viewer and plugin loader shell use the control channel to pass messages back and forth.

To establish the control channel, the Viewer listens on a local TCP socket, and passes the socket's port to the PLS. The PLS connects back to the indicated local TCP port.

To keep latency low, the system does not use the control channel for bulk data transfer. If large amounts of data need to be passed between the plugin host and a plugin, that data should not be embedded in messages sent across the control channel. Instead use either shared memory or separate streams negotiated via messages.

Shared memory

The Viewer sets up memory segments that are shared with plugin processes. Shared memory segments can be created, resized, and removed.

A plugin instance sends requests to its plugin loader shell to set up an interprocess shared memory segment between itself and the Viewer. The memory segment is used to share large amounts of data, such as rendered pixel data or audio data.

A plugin can have more than one shared memory segment, each of which is identified by name. The names are implementation-defined and different on each platorm, but will always be unique across all plugin instances running on a machine.

Shared memory setup follows the same rules as any message exchange: when a plugin sends a request to create or modify a segment, it must receive a confirmation message before proceeding as if the change has been made.

A plugin exchanges messages with its PLS to set up and remove shared memory segments. The PLS modifies and forwards the shared memory messages to and from the Viewer -- messages do not go directly from the plugin to the Viewer.

Shared memory implementation

There are native implementations for the shared memory segment with a lightweight platform abstraction on Windows, Mac and Linux. These use the CreateFileMapping API on Windows and either mmap() or shm_open() on Mac and Linux. The details of this implementation are hidden from the plugins, since they reside in the Viewer and plugin loader shell.

Possibilities for future development

The media rendering plugin system is the first use of a general Viewer plugin architecture that in the future may be used for other types of plugins, for example to support additional input devices, image decoding (for example, JPEG2000), and voice services.

This document does not address a system for distributing plugins automatically, for example in the way that Firefox add-ons work.

Retrieved from "https://wiki.secondlife.com/w/index.php?title=Linden_Lab_Official:Media_Rendering_Plugin_System_Technical_Overview&oldid=1142372"