Linden Lab Official:Media Rendering Plugin System Technical Overview

From Second Life Wiki
Revision as of 10:30, 24 June 2009 by Samuel Linden (talk | contribs) (Created page with '= A design for a plugin based media library for the Second Life client = == Links == * For overview timeline, see '''Media Plugin Timeline''' * For latest branch: '''svn+ssh...')
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

A design for a plugin based media library for the Second Life client

Links

Goal

  • To facilitate the easier integration of new and improved media into the Second Life client.

Why do we need a new system

  • The library that provides media support for the Second Life client supports the addition of new media implementations. However, in order to add or modify one you have to rebuild the entire client since it's implemented entirely in a static library. This can be a complex and time consuming task. If the implementation for each media type was a separate component, accessible via a narrow, well defined API/interface, it would be substantially easier to create, modify and maintain them. Clearly, given the creativity and ingenuity of our residents, an increased selection of robust media implementations in Second Life will enable more interesting applications and use cases - shared desktops, native document display, 3rd party application integration etc.
  • A small but significant portion of the client crashes happen when a media component crashes deep inside its own code - currently this means Mozilla/Gecko, QuickTime or GStreamer. If this happens there is very little we can do to fix it. If the media code crashes while it is running in a separate process then the user experience may be less than ideal but the client will not crash.
  • This redesign is part of an engineering effort to move towards a more finely grained architecture that will make it easier to maintain and test the Second Life client.
  • A robust implementation of a generic plugin architecture will allow similar changes to filter through to other parts of the Second Life code base - for example, image support (J2K image modules) and user input devices (Space Navigator, VR Goggles etc.).
  • The asynchronous nature of the proposed design means that developing versions in the future that support multiple sources per parcel/prim will be substantially easier to implement. In addition, there will be free (potential) performance gains as systems move towards a multicore architecture.
  • The new design will allow for multiple implementations per media type and/or per platform (Mozilla/IE/WebKit for the embedded browser for example).
  • Part of the deliverables for this project is a well defined API including documentation and sample code that illustrates how to write your own plugin.
  • This design does *not* address a system for distributing plugins automatically in the same way that Firefox add-ons work for example. Clearly, something similar will be required eventually but is beyond the scope of this document.

30 second overview

  • plugin implementations run as a separate process.
  • Pixel data is exchanged between the plugin and client using shared memory.
  • Bi-directional communication between plugin and client is via a local TCP channel.
  • Communication between plugin and client is message based and completely asynchronous.
  • Typical data flow when the client wants to render a media URI
    • The client asks the media manager to create a media implementation that can render media at the given URI.
    • The plugin manager retrieves the MIME type of the resource at this URI type via an asynchronous HTTP HEAD request. If there is no MIME type associated with this URI (a VNC session for example) then one is assigned to it by specifying a type or subtype in the "x-" or "vnd-" range.
    • Later, when the MIME type is known, the type and subtype fields are used to determine which media implementation will be used to render it via a lookup in an XML file.
    • The plugin manager creates an "instance" of the relevant plugin.
    • When the plugin is fully created and initialized, it informs the plugin manager that it is ready.
    • The plugin manager tells the client that the media can now be rendered.
    • Sometime later, the client asks the plugin manager to tell the plugin to start rendering data.
    • Sometime later, the plugin receives the message and starts to render the media.
    • When the media data being rendered changes, the plugin tells the media manager which in turn tells client that something changed and the client updates itself accordingly.


LLPlugin Diagram.png

Detailed description

General comments

This document really encompasses two separate (although related) designs:

In this document, we use the word "media" to refer to QuickTime media and the in-viewer web browser (which are currently implemented by LLMedia and its subclasses), as well as other media types which may be added in the future.

We use the word "client" to mean any code in the viewer which uses the services of a plugin.

The intention is that the plugin architecture will be usable for plugins of different types. Although the initial implementation will probably only be used for media plugins, we're also thinking about how it could be used for:

  • input devices
  • image decoding (JPEG2000)
  • voice services

This design proposes that the media implementations are separate processes. The reasons for this are:

  • Improved stability in Second Life client - if the plugin crashes, the viewer is able to continue, albeit with invalid (or generic) media data.
  • It will be significantly easier to author plugins if there are no build-time dependencies.
  • Plugins can be written in different versions of the compiler - perhaps even different languages altogether.
  • The inherently asynchronous nature of a multi-process design will (generally speaking) work more effectively on multi-core systems.

There may be future uses of the plugin architecture where it makes more sense to have plugins load directly into the viewer's address space. This design is intended to support that mode of operation as well, but the initial implementation will load all plugins in an external process.

Plugin Architecture

Plugin Architecture Overview

At the lowest level, a "plugin" will be a loadable piece of object code in platform-native format (a .DLL, .so, etc.). Since these types of interfaces are inherently fragile, we will keep this part of the interface as simple as possible, so that it won't have to change over time.

Plugins may be loaded in one of two ways: directly into the viewer process, or in a separate process. The same plugin should be able to be loaded either way, and the code inside the plugin shouldn't have to do anything differently.

Plugins which are loaded into their own processes will be loaded by a plugin loader shell which implements interprocess communication with the viewer process and acts as a host for the plugin.

All communication between the viewer and plugins will be through messages. Messages consist of a message class and name (both human-readable) and a collection of key-value pairs (values with human-readable names). Messages are represented with LLSD, so values may be rich data types such as arrays or other containers.

Individual messages are unidirectional, and messages will be sent in both directions (from the host to the plugin, and from the plugin to the host). Sending a message to a plugin _may_ (depending on the contents of the message) cause it to send a message back to the host as a response. Plugins may also send unsolicited messages to the host (i.e. messages that aren't responses to a particular message the plugin received).

Messages will be passed to and from the plugin across a control channel. The control channel will be a stream -- the current plan is to use a TCP socket, but a unix domain socket or pipe would also work on platforms that support them. Since messages will be passed over the control channel, they may need to be serialized. They may also be received in a different process space, so passing pointers in a message is not generally useful.

It's important to note that messages are completely asynchronous. They may be queued for an arbitrary amount of time before they're delivered, so it's not possible to do a direct query of a plugin and get an answer back. The command sets for specific plugin types need to take this into account -- instead of having commands to query for the state of a plugin, the plugin should send unsolicited messages to the host when its state changes in meaningful ways, so that the host can maintain a local shadow of that state from which it will answer queries. This logic, as well as the logic for message encoding/decoding, will be handled by a proxy object -- an instance of a C++ class in the viewer that is tailored for a particular type of plugin.

Plugins can negotiate (via specific messages) for the host to set up segments of shared memory between the plugin and the viewer. When plugins are hosted by the plugin loader shell, these will be interprocess shared memory segments. (When directly loaded by the viewer, they will be simple pointers, but they will still negotiate setup in the same way, and the pointer will be owned by the plugin manager.) This is especially useful for media plugins, since they need to share large amounts of data with the viewer (specifically, pixel data containing rendered media).

Plugin Manager

The plugin manager code in the viewer will be responsible for launching the plugin loader shell which hosts each plugin.

The plugin manager code will monitor the plugin process and notice if it crashes. Some form of heartbeat from the plugin may additionally be implemented to detect if the process hangs. If the plugin dies, or hangs and needs to be killed, the client will be alerted

  • The Media plugin proxies may use this information to display a generic "media is not available" message in place of the rendered media.

The plugin manager will create and own all proxy object instances. Clients that need to have a plugin instantiated will make those requests through the plugin manager, and proxy objects will use the plugin manager internally to communicate with their plugins.

Plugin Loader Shell

The plugin loader shell is the executable that is responsible for loading and executing the code for a plugin as an external process.

Making this a single executable (instead of having a separate executable for each plugin) will make navigating through the Windows firewall (and other programs which ask the user about processes that make outgoing network connections) less painful - it will only need to be unblocked once.

Code in the plugin loader shell will handle certain messages (such as certain parts of shared memory setup and process priority control) instead of passing them through to the plugin.

When the plugin loader shell is initially launched, it won't know which plugin it is expected to load. This information will be supplied via internal messages once it establishes the control channel.

The plugin loader will manage relative priorities and CPU usage of plugins by setting the operating system priority of the process and throttling messages to the plugin.

In the future, the plugin loader may be responsible for querying a central authority for a certificate of authenticity for a plugin it has been asked to load. Ultimately, we will have different levels of signing so users can differentiate between plugins from Linden Lab, plugins from partner companies and 3rd party providers (Granitehead Chislehome and his penis plugins). Users will be able to set a security "zone" so that no plugin outside of their zone is ever loaded. The default setting is probably obvious. None of this will be implemented in this phase of the project and is only mentioned here in case it's appropriate to lay some groundwork earlier on.

Control Channel

The control channel will be used bidirectionally to pass messages back and forth to the plugin.

To establish the control channel, the plugin manager will listen on a local TCP socket, and pass that socket's port to the plugin loader shell as a command-line argument on launch. The plugin loader shell will connect back to the indicated TCP port on localhost.

To keep latency low, the control channel will not be used for bulk data transfer. If large amounts of data need to be passed between the plugin host and a plugin, that data should _not_ be embedded in messages sent across the control channel. One of these methods should be used instead:

  • shared memory segments
  • separate streams negotiated via messages

Messages

All communication with the plugin will be via messages. At the binary/function call level, the only thing plugins can do is send and receive messages. This keeps the binary interface to plugins very simple, and means that it should not have to change over time.

A message consists of a message class and message name (which will be human-readable strings) and 0 or more key-value pairs. A key-value pair is a name (also human-readable) and some data. Messages are represented using LLSD, and are serialized/deserialized using the existing mechanism for LLSD serialization.

Messages must be self contained and cannot pass pointers, since they may need to be serialized, and may be passed across process boundaries. There may be a couple of exceptions to this rule, for messages that are sent from the plugin loader shell to the plugin itself when setting up shared memory regions.

Individual messages are unidirectional, asynchronous, and may be queued for an unspecified amount of time before they're delivered. Messages ARE guaranteed to be delivered in the same order they were sent. If a plugin crashes, any clients with an interface to that plugin will be notified, and must assume that any messages sent before the notification may have been lost

Sending a message to a plugin may or may not induce it to send back a response. Whether it does or not will be dependent on the semantics of that particular message. Plugins can also send unsolicited messages (i.e. messages that are not a direct response to another message) to the host -- this will be used for keeping the plugin's proxy object's cache of the plugin's state up to date, among other things.

We will define sets of messages which map to a logical hierarchy of plugin types. I expect the initial hierarchy to look something like this:

  • Internal messages -- for communication between the viewer and the plugin loader shell. These should never be delivered to a plugin.
  • Base messages -- all plugins must implement a set of messages used for initialization, connection management, and shared memory setup
    • Media messages -- media plugins will implement this set, which will cover negotiating a pixel buffer and handling updates of subsections of it, and specifying a source URL for media, and passing mouse and keyboard events
      • Time-based media messages -- play, pause, stop, seek, etc.
      • Browser-like media messages -- forward, back, reload, etc

We'll need to think about how to categorize these. For example, it may make more sense to break out "interactive media" messages (mouse and keyboard interaction) into a separate class.

Part of a plugin's initialization will include it telling the plugin host which sets of messages it supports. This will include versioning information, so that if we have to change the semantics of messages over time, backwards compatibility can be maintained.

Incoming messages will be queued by each end, and delivered to the plugin and code in the viewer serially. There will be a simple mechanism for both the plugin and the plugin client to put their input queues "on hold", to avoid potential concurrency issues. When the input queue is on hold, no new messages will be delivered to that end's message handler. By default, the queue will be on hold for the duration of the call to the message handler.

Shared Memory

The plugin manager will contain code which can set up shared memory segments that are shared between the viewer and plugin processes. Shared memory segments will be identified by a name (which should be unique within each instance of the plugin). Memory segments can be created, resized (a.k.a reallocated), and destroyed.

Shared memory segment setup uses message semantics -- when you make a request to the plugin manager to modify a segment, nothing actually happens until you receive a message telling you it completed.

The plugin will interact with the plugin host through messages to set up and tear down shared memory segements, but some of those messages may be handled (or partially handled and changed) by the plugin host shell -- they will not go directly over the control channel.

We will create native implementations for the shared memory segment with a lightweight platform abstraction. This would use the CreateFileMapping API on Windows and either mmap() or shm_open() on Mac and Linux. The details of this implementation will be hidden from plugin authors, since it will reside in the viewer and the plugin loader shell.

Proxy Object

A proxy object is an instance of a C++ class in the viewer which has detailed knowledge of the behaviors and messages of a particular type of plugin. Each instance of a plugin being managed by the plugin manager will have a corresponding instance of a proxy object in the viewer.

Parts of the viewer code that need to use the services of a plugin will do so by interacting with the plugin's proxy object. The proxy object will provide any necessary interfaces for interacting with the plugin, and will mirror any necessary state information about the plugin that may need to be queried by clients of that plugin.

The proxy object instance may persist for some time after the plugin instance goes away, if necessary (i.e. so that all clients of the plugin can be cleanly notified). If a plugin crashes and is relaunched, the proxy object need not be destroyed, although any clients using the proxy will be notified that the plugin's state has been reset.

The proxy object may mirror some of the state of the plugin. This can be maintained by the plugin sending unsolicited state messages when its state changes in ways meaningful to its clients. State queries can then be answered directly by the proxy object without having to do a round-trip query with messages.

The hierarchy of proxy object classes may mirror the hierarchy of message types where it makes sense to do so. For example, there will probably be different proxy object classes for browser-like and time-based media.

Each proxy object will have member functions that can be called which will cause messages to be sent to the plugin for which it is a proxy. Likewise, each proxy object will have a mechanism (probably a listener-type interface) that will allow other code to sign up to receive relevant messages from the plugin.

The functions in the proxy object that clients use will have a conventional function-argument signature. The message will be built up (for outgoing messages) or decoded (for incoming messages) inside the proxy object's implementation, so that the details of message formats aren't exposed to code that doesn't need to know them. If the semantics of messages need to change over time, this is the level where compatibility with older plugins/message formats will be maintained.

Media Plugins

Media plugins can render in one of two modes: continuous and one-shot.

In continuous mode, the plugin instance is dedicated to a particular media stream. A shared memory buffer is allocated to the stream, and it renders into it continuously.

In one-shot mode, a message is sent to the plugin requesting that it render a particular media URL into a shared memory buffer. It renders the media once, then sends a message indicating the render has completed. This may be used when rendering large numbers of media streams at lower priorities (such as when there are a large number of streams in the user's view, but most of them are far away).

  • The media implementation used to render a given type of media is defined both by the type and subtype fields of its MIME type (more correctly called the Internet Media Type nowadays).
  • QUESTION: Should MIME type (more correctly called Internet Media Type now) be the only parameter used to determine which media implementation to use or should other parts of the URI such as scheme or extension also play a part too?
    • ANSWER: I think the decision of which plugin to use for a particular piece of media is outside the scope of the API design. A plugin can specify which types of media it's capable of rendering, but the policy of which plugin to use may depend on other factors, such as user preferences, the presence of other plugins, and the "kill switch" for turning off plugins in which exploits are found. All we should say to the plugin writer is that they will only be asked to render media of types they specify.
  • A local XML file specifies which MIME type or scheme maps to which media implementation.
  • QUESTION: Should this XML mapping file be stored online at a Linden Lab domain and cached? That would make changing the mapping easy which would be useful for switching off plugins if exploits are found. In the absence of a plugin distribution system for this version, we could implement a layered system (like XUI files) where a local version of the file is overlaid at runtime if it exists.
    • ANSWER: I think this also falls outside the scope of the API design -- we don't need to specify the policy completely at this point, just say "there will be a mechanism for determining which plugin renders which media".
  • This component will contain a handler that observes the control channel and catches messages from plugins. (This is subject to a design decision about how messages are routed - via the plugin manager or directly from plugin to client). It is possible that this handler should be a standalone component that is shared with the plugin code since they both have to observe the control channel and act on messages.


Plugin implementation

  • A shared library that contains the code to render media
  • One shared library per media implementation.
  • One invocation of the plugin per media source (per URI).
    • However, consideration should be given to a system that allows multiple media sources (URIs) to share a plugin invocation. For example, for rendering one-shot Web pages, it might be perfectly acceptable for a part of the client to wait a short time before displaying a Web page whilst a different page is rendered.
    • In addition, it may be optimal to cache one or more plugin invocations - for example, a significant amount of content in the Second Life client uses the embedded Web Browser so when the login page is closed and the user logs in, there is probably no point in destroying the embedded Web browser media implementation.
  • Ideally, there should be a way to disable plugins remotely. If as discussed elsewhere, the mapping from MIME type to implementation is stored in a remote file, this will be straightforward to implement. If not, some other "kill switch" mechanism should be considered.
  • Each plugin should support a default "Loading" 'image' that is displayed during init / load process. There should be a static one built into the plugin itself and perhaps the ability to specific one via local/remote URI.
  • Plugins should be able to deal with media sizes changes over the life of the stream. For example, this happens frequently in a QuickTime RTSP stream or when a Web based UI is re sizeable.
  • Needless to say, plugins must be able to deal with invalid media data or network failures and not crash. At the very least they should revert to their "Loading..." screen and ideally report back on what happened.
  • Each plugin will contain a handler that observes the control channel and catches messages from the plugin manager. (This is subject to a design decision about how messages are routed - via the plugin manager or directly from plugin to client). It is possible that this handler should be a standalone component that is shared with the plugin manager code since they both have to observe the control channel and act on messages.
  • An addition to the design that allows the plugin implementation shared libraries to be loaded directly by the plugin manager into the same process space is being considered. There are some circumstances when this might be preferred and whilst this functionality may not be needed immediately, it will be useful in the future. For example, in a world where many simultaneous media sources are required, having a large number of separate processes may consume a lot of system resource. If a plugin has been shown to be reliable, loading in into the same process may be reasonable. Certainly from the plugin author's point of view (and ideally for most of the system), the interface should be the same for both use cases.

Message definitions

Arrows indicate which direction each message is sent:

> for host-to-plugin (or host-to-launcher shell for internal messages)

< for plugin-to-host.

Keys for each command are set off as bullet items, like this:

  • foo: a sample argument

internal messages

Sent between the viewer and the launcher shell ONLY. Plugins will not need to handle these messages.

< hello
first command sent from the launcher shell to the viewer when it comes up
> load_plugin
tells the loader to load a plugin and send the base init message
  • file (string): the full path to the plugin DLL/dylib/so file.
< load_plugin_response
plugin loaded and initialized
  • versions (map): LLSD key/value map, where each key is a message class and its value is a version string. All versions start at "1.0".
  • plugin_version (string): Freeform version string for the plugin itself. This is expected to contain the versions of libraries the plugin uses (i.e. quicktime, webkit, etc.) and may end up in the viewer's about box.
< heartbeat
plugin launcher shell sends these once per second to show that the plugin isn't locked up
> shutdown
request an orderly shutdown
< shutdown_response
shutdown has completed, about to exit
> shm_add
shared memory segment has been set up in the parent
  • name (string)
  • size (S32)
< shm_add_response
shared memory has been mapped into the child
  • name (string)
> shm_remove
shared memory segment will be removed once you respond
  • name (string)
< shm_remove_response
plugin is done with the shared memory segment, safe to delete
  • name (string)
> sleep_time
set the interval between idle messages to the plugin
  • time (float): time in seconds (default is 1/100)

base messages

All plugins need to handle these messages

> init
< init_response
  • versions (map): LLSD key/value map, where each key is a message class and its value is a version string.
    • All versions start at "1.0".
    • the host will need to check these against what it understands and send the proper versions of each message set to the plugin
  • plugin_version (string): Freeform version string for the plugin itself. This is expected to contain the versions of libraries the plugin uses (i.e. quicktime, webkit, etc.) and may end up in the viewer's about box.
> idle
call-in for plugin to do processing, no response
  • time (float): interval (in seconds) at which idle messages are being sent
> shutdown
request an orderly shutdown
< shutdown_response
completed cleanup, ready to be killed
> shm_added
shared memory segment has been set up
  • name (string)
  • size (S32)
  • address (string)
    • Since this is going from the loader shell to the plugin (i.e. not crossing a process boundary), the pointer will actually be valid
    • LLSD doesn't have an encoding for a pointer (or even an unsigned 32 bit value), so this will be the pointer's value as a hexadecimal string
> shm_remove
shared memory segment will be removed once you respond
  • name (string)
< shm_remove_response
plugin is done with the shared memory segment, safe to delete
  • name (string)

media messages

< texture_params
  • default_width (S32) OPTIONAL -- default width for the texture
  • default_height (S32) OPTIONAL -- default height for the texture
  • depth (S32) -- pixel size in bytes
  • internalformat (U32) -- texture internalformat for glTexImage2D
  • format (U32) -- texture format for glTexImage2D
  • type (U32) -- texture type for glTexImage2D
  • swap_bytes(bool) OPTIONAL -- true if the host should use glPixelStorei(GL_UNPACK_SWAP_BYTES, 1) when loading the data
    • if not specified, defaults to false.
  • coords_opengl(bool) -- true if the plugin wants to work in OpenGL coordinates (where (0,0) is the lower left corner of the texture), false to work in a coordinate system where (0,0) is the upper left
    • Note that the hexadecimal GLenum values for these are fixed by the OpenGL ARB, so passing them across process boundaries should be safe
    • Although the pixel size in bytes is completely determined by the format and type, deriving it from only those values is complex. To simplify things, the plugin must provide the pixel size to the host.
  • allow_downsample(bool) OPTIONAL -- If true, the host may negotiate a smaller texture when the plugin's priority is reduced.
    • The plugin should not set this if downscaling the media is more expensive than drawing it at native size
  • padding(S32) OPTIONAL -- Specifies how the rows of the texture should be padded. Useful values:
    • 0 (the default) -- no padding is added
    • -1 -- pad width to the next power of two
    • 16 -- start each row on a 16 byte boundary -- NOTE: may not work correctly with odd-sized pixels like GL_RGB
    • 32 -- start each row on a 32 byte boundary -- NOTE: may not work correctly with odd-sized pixels like GL_RGB
> load_uri
  • uri (string)
> key_event
  • event (string) one of:
    • down
    • repeat
    • up
  • key(S32) -- platform-specific keycode
  • modifiers -- string containing zero or more of the following, concatenated together with '|' characters in between. There may be a spurious trailing '|'.
    • control
    • alt -- means the 'option' key on the mac
    • shift
    • meta -- means the 'command' key on the mac
> text_event -- text input event not associated with a single keystroke
  • text(string) -- encoded in utf8
> mouse_event
  • event (string) : one of:
    • down
    • move
    • up
    • double_click
  • x -- x coordinate in texture space
  • y -- y coordinate in texture space (taking into account whether vertical flip was requested or not)
  • modifiers -- string containing zero or more of the following, concatenated together with '|' characters in between. There may be a spurious trailing '|'.
    • control
    • alt -- means the 'option' key on the mac
    • shift
    • meta -- means the 'command' key on the mac
> scroll_event
scrollwheel
  • x (integer) : horizontal scroll amount (positive or negative)
  • y (integer) : vertical scroll amount (positive or negative)
  • modifiers -- string containing zero or more of the following, concatenated together with '|' characters in between. There may be a spurious trailing '|'.
    • control
    • alt -- means the 'option' key on the mac
    • shift
    • meta -- means the 'command' key on the mac
< updated
  • left(S32)
  • top(S32)
  • right(S32)
  • bottom(S32)
  • current_time(float) OPTIONAL -- the current playhead time in seconds
  • duration(float) OPTIONAL -- the total duration of the media in seconds
  • current_rate(float) OPTIONAL -- the rate at which the media is currently playing
< media_status
  • status (string), one of:
    • <empty string> -- not initialized or no media loaded
    • loading -- loading or navigating
    • loaded -- preroll/navigation complete
    • error -- navigation/preroll failed
    • playing -- playing (only for time-based media)
    • paused -- paused (only for time-based media)
< size_change_request
when the stream wants to change its native size
  • width(S32)
  • height(S32)
  • name(string) -- the name of the shared memory segment the texture will draw into
    • this may or may not induce the host to send a size_change message. The plugin may stop drawing to the texture until an appropriate size_change message has been received.
> size_change
either host-initiated or after a request from the plugin
  • width(S32)
  • height(S32)
  • texture_width(S32)
  • texture_height(S32)
  • name(string) -- the name of the shared memory segment the texture will draw into
< size_change_response
sent by the plugin when the size change is complete
  • width(S32)
  • height(S32)
  • texture_width(S32)
  • texture_height(S32)
  • name(string) -- the name of the shared memory segment the texture will draw into
< cursor_changed
  • name (string), one of:
    • <empty string>
    • arrow
    • ibeam
    • splith
    • splitv
    • hand
> set_priority
tells the plugin its relative priority
  • priority(string): one of:
    • stopped: media is not playing, shouldn't need to update at all
    • hidden: media is not being displayed or is out of view, don't need to do graphic updates, but may still update audio, playhead, etc.
    • low: media is in the far distance, may be rendered at reduced size
    • normal: default priority
    • high: media has user focus and/or is taking up most of the screen
> edit_cut
do an edit-menu "cut" operation to the system clipboard
> edit_copy
do an edit-menu "copy" operation to the system clipboard
> edit_paste
do an edit-menu "paste" operation from the system clipboard
< edit_state
update the state of the edit menu items in the host
  • cut(bool) OPTIONAL
  • copy(bool) OPTIONAL
  • paste(bool) OPTIONAL
    • items that are not present in the update message should have their state unchanged.
    • All states start out false in the host.
    • False means the operation is not possible and the menu should be disabled.

media_browser messages

> focus
  • focused (bool): true if the browser instance has focus
> clear_cache
> clear_cookies
> enable_cookies
  • enable (bool)
> proxy_setup
  • enable (bool)
  • host (string)
  • port (S32)
> browse_stop
> browse_reload
  • ignore_cache(bool): if true, bypass cache for this reload
> browse_forward
> browse_back
> set_status_redirect (we use a 404 redirect in Web based search)
  • code(S32): the code to redirect on (should only be 404 for now)
  • url(string): the URL to redirect to
< navigate_begin
  • uri (string): uri the browser is navigating to
< navigate_complete
  • uri (string): uri the browser was navigating to
  • result_code (S32): 200, 404, etc.
  • result_string (string): "OK", "Not Found", etc.
  • history_back_available (bool): true if the "back" button should be enabled
  • history_forward_available (bool): true if the "forward" button should be enabled
< progress
  • percent (S32): percent of page load that has completed
< status_text
  • status (string): new browser status string
< location_changed
  • uri (string): new browser uri for the navigation bar
< click_href
  • uri (string): uri of the link
  • target (string): target property of the link
< click_nofollow
  • uri (string): uri of the link
> init_history
  • history (LLSD, array of strings): array of string URIs to add to the browser history

media_time messages

> stop
> start
  • rate (float) OPTIONAL: rate to play at -- 0.0 is the movie's default rate, 1.0 is normal forward (on most movies), -1.0 is reverse, greater than 1.0 is fast foward, etc.
> pause
> seek
  • time (float)
> set_loop
  • loop (boolean): true to make the media loop, false to make it play once
> set_volume
  • volume (float)

Changes to existing system

Client side code

  • A large section of the client code that deals with moving bytes from memory into a texture managed by the client that OpenGL understands will have to be rewritten. There are several reasons for this - firstly, that code has grown "organically" and needs a significant clean-up. Secondly, the process by which dynamic textures are created and managed hasn't changed since QuickTime was added to the client 4 years ago. Lastly and most importantly, the programming paradigm changes completely with this new design from a synchronous model where you call functions to a message based model. When you send a message to a plugin, consider it as being shot into the dark. You may (or may not) get a response back at some future time. Whether a particular message will cause a response to be sent will be defined by the particular type of plugin.
  • plugin will specify which transport controls should be used via a fragment of XML.
  • The number and type of transport controls can vary over time and should support an enabled and disabled state.
  • QUESTION: How will we do this? Built in set of icons that XML file references plus ability to specify your own or something more straightforward?

Existing media implementations

  • New versions of the embedded Web browser (using LLMozLib) and movie support (QuickTime and GStreamer) will be written using the new API.
  • Ideally, we will include at least one additional Linden Lab written plugin and one more 3rd party plugin.
  • Some simple examples will be included in the API documentation.

Testing philosophy

  • Lots of QA and user testing via a 'First Look' release and blog post encouraging users to test the more esoteric existing media set ups.
  • Script to grab all media URIs from the grid, add them to a playlist that is used as input to a test harness that tries to open and render each one for a few seconds. Clearly this won't test the visuals but may find potential crashers. A lot of this exists already. It's takes a long time - several days to grab URIs and a day to run through them.
  • We should think about unit tests for each media implementation. A script can run through the directory and load each one, initialize it and render a known media URL at a given "timecode". The output can be captured and written to a file. We can either visually examine the file or look for an all black frame (for example) and pass/fail automatically. Not perfect but it would catch big failures.

Public API

  • Developers who want to write plugins will do so via a public API
  • The API describes the interface between the plugin and the plugin manager. (This is subject to a design decision about how messages are routed - via the plugin manager or directly from plugin to client)
  • API documentation will consist of the header file itself and example code that illustrates how to write a plugin from scratch. If the header file is sufficiently well commented, other written documentation will not be necessary.
Retrieved from "https://wiki.secondlife.com/w/index.php?title=Linden_Lab_Official:Media_Rendering_Plugin_System_Technical_Overview&oldid=404923"