Linden Lab Official:Media Rendering Plugin System Technical Overview

From Second Life Wiki
Jump to navigation Jump to search

Overview

Goal: To facilitate easier integration of external media into the Second Life Viewer.

This document presents two separate (although related) designs:

Plugins of different types can use the plugin architecture. Although the initial implementation is for media plugins, potentially it could also be used for:

  • Input devices
  • Image decoding (JPEG2000)
  • Voice services

In the future, developers may wish to load plugins directly into the Viewer's address space. This design supports that mode of operation, but the initial implementation loads all plugins as external processes.

Loading media rendering plug-ins ras separate processeses:

  • Improves stability in Second Life client: if the plugin crashes, the Viewer is able to continue, albeit with invalid (or generic) media data. Currently, a small but significant portion of Viewer crashes occur when a media component crashes.
  • Eliminates build-time dependencies, making it easier to author plugins.
  • Eases plugin development:
    • Plugin developers will not need to rebuild the entire Viewer, a complex and time-consuming task.
    • Plugin developers will be able to use different compiler versions, perhaps even different languages.

The asynchronous nature of the architecture:

  • Takes advantage of multi-core systems (generally speaking).
  • Facilitates support for multiple media sources per parcel or per prim.

Why?

Implementing rendering for media types in separate plugins makes each easier to create, modify and maintain.

Given the creativity and ingenuity of Residents, an increased selection of robust media implementations in Second Life will enable more interesting applications and use cases, for example, shared desktops, native document display, third-party application integration, and so on.

This redesign also:

  • Facilitates moving towards a more fine-grained architecture to ease maintenance and testing of the Second Life Viewer.
  • Enables similar changes to filter through to other parts of the Second Life code base; for example, image support (J2K image modules) and user input devices (for example Space Navigator and VR Goggles).
  • Allows for multiple implementations per media type and/or per platform (Mozilla/IE/WebKit for the embedded browser for example).

This design does not address a system for distributing plugins automatically, for example in the way that Firefox add-ons work. Clearly, something similar will be required eventually but is beyond the scope of this document.

Terminology

In this document "media" refers to QuickTime media and the in-Viewer web browser (currently implemented by LLMedia and its subclasses), as well as other media types which may be added in the future.

The term "client" refers to code in the Viewer that uses the services of a plugin.

Code

Latest branch: https://svn.secondlife.com/svn/linden/branches/2009/plugin-api/

After checking out the branch, run develop.py normally, then build/run the "media_plugin_test" target in the SecondLife project.

Technical overview

Plugin implementations run as a separate process. A plugin and the Viewer exchange pixel data using shared memory.

Communication between a plugin and the Viewer is:

  • Bi-directional via a local TCP channel.
  • Message-based and completely asynchronous.

Typical data flow when the Viewer wants to render a media URI:

  • The Viewer retrieves the MIME type of the resource with an asynchronous HTTP HEAD request. If there is no MIME type associated with this URI (a VNC session for example) then it uses type or subtype in the "x-" or "vnd-" range.
  • Later, when the MIME type is known, the Viewer uses type and subtype fields to determine which media plugin to use via a lookup in an XML file.
  • The Viewer creates an "instance" of the relevant plugin.
  • When the plugin is fully created and initialized, it informs the Viewer that it is ready.
  • Viewer asks the plugin to start rendering data.
  • Plugin receives the message and starts to render the media.
  • When the media data being rendered changes, the plugin tells the Viewer that something changed and the Viewer updates itself accordingly.


LLPlugin Diagram.png

Plugin architecture

At the lowest level, a "plugin" is a platform-native dynamic library (.DLL on Windows, .dylib on Mac OS, and .so on Linux.

Although the architecture provides for plugins to load directly into the Viewer process (for example for certain plugins developed by Linden Lab), externally-deveoloped plugsin generally run as a separate process to enhance stability. NOTE: You write to the same API whether the plugin runs as a separate process or not; so this factor does not affect how you write the plugin. This document focuses on plugins that run as separate processes, since in general that is the case for third-party plugins.

The plugin loader shell (PLS) loads and hosts plugins and provides for communication with the Viewer process.

Plugins communicate with the PLS (and thus the Viewer) through messages. Messages consist of a message class and name (both human-readable) and a collection of key-value pairs (values with human-readable names). Messages are represented with LLSD, so values may be rich data types such as arrays or other containers.

Individual messages are unidirectional; messages are sent:

  • From the PLS to the plugin.
  • From the plugin to the PLS.

Sending a message to a plugin may cause it to send a message back to the PLS as a response, depending on the contents of the message. Plugins may also send unsolicited messages to the plugin loader shell (i.e. messages that aren't responses to a particular message the plugin received).

The plugin and the PLS send messages back and forth using the control channel, a local TCP socket. Since messages are passed over the control channel, they may need to be serialized.

Generally the PLS and the plugin run in a different process space, so you cannot pass pointers in a message.

Messages are completely asynchronous. They may be queued for an arbitrary amount of time before they're delivered, so it's not possible to do a direct query of a plugin and get an answer back. The command sets for specific plugin types need to take this into account: instead of having commands to query for the state of a plugin, the plugin should send unsolicited messages to the host when its state changes in meaningful ways, so that the PLS can maintain a local shadow of that state from which it will answer queries. The proxy object handles this logic, as well as message encoding and decoding,

Plugins can negotiate (via specific messages) for the host to set up segments of shared memory between the plugin and the viewer. When plugins are hosted by the plugin loader shell, these will be interprocess shared memory segments. (When directly loaded by the viewer, they will be simple pointers, but they will still negotiate setup in the same way, and the pointer will be owned by the Viewer.) This is especially useful for media plugins, since they need to share large amounts of data with the viewer (specifically, pixel data containing rendered media).

Plugin loader shell

The plugin loader shell (PLS) is a separate process (slplugin.exe) that hosts all plugins and executes the code for a plugin as an external process.

Since the PLS is a single executable, it is easier for the end-user to navigate through Windows firewall (and other programs which limit outgoing network connections), since the user only need to be unblock it one time.

The PLS handles certain messages (some shared memory setup and process priority control) instead of passing them through to the plugin.

When the PLS is initially launched, it won't know which plugin it is expected to load. The Viewer supplies this information via internal messages once it establishes the control channel. The PLS manages relative priorities and CPU usage of plugins by setting the operating system priority of the process and throttling messages to the plugin.

Messages

All communication with the plugin will be via messages. At the binary/function call level, the only thing plugins can do is send and receive messages. This simplifies the binary interface to plugins, and eliminates reasons to change the interface.

A message consists of a message class and message name (human-readable strings) and zero or more key-value pairs. A key-value pair is a name (also human-readable) and some data. Messages are represented using LLSD, and are serialized/deserialized using the existing mechanism for LLSD serialization.

Messages are self-contained and cannot pass pointers, since in general they are serialized and passed across process boundaries.

Individual messages are unidirectional, asynchronous, and may be queued for an unspecified amount of time before they're delivered. Messages ARE guaranteed to be delivered in the same order they were sent. If a plugin crashes, any clients with an interface to that plugin will be notified, and must assume that any messages sent before the notification may have been lost

Sending a message to a plugin may cause it to send a response. Whether it does depends on the semantics of the particular message. Plugins can also send unsolicited messages (messages that are not a direct response to another message). Such messages are used to keep the plugin's proxy object's cache of the plugin's state up to date, among other things.

Shared Memory

The Viewer sets up memory segments shared with plugin processes. Shared memory segments are identified by a name unique within each instance of the plugin. See Project Notes for more information.

Media Plugins

A media plugins instance is dedicated to a particular media stream. A shared memory buffer is allocated to the stream, and it renders into it continuously.

  • The internet media type (aka MIME type) defines the media implementation used to render a given type.
  • Internet media type is not the only factor that determines which plugin renders a given media type. While a plugin specifies the types of media it can render, the Viewer determines which plugin to use based on other factors (beyond the scope of this document), such as user preferences, the presence of other plugins, and so on. Thus, while you as a plugin developer indicate what media your plugin is capable of rendering, you cannot guarantee that a given media type will be rendered by the plugin.
  • An XML file in the Viewer installation specifies which internet media type maps to which media plugin.

Plugin implementation

  • A shared library that contains the code to render media
  • One shared library per media implementation.
  • If a plugin encounters invalid media data or network failures it can simply exit, since this will not affect the Viewer process.
Retrieved from "https://wiki.secondlife.com/w/index.php?title=Linden_Lab_Official:Media_Rendering_Plugin_System_Technical_Overview&oldid=456583"