Linden Lab Official:Media Plugin System Messages

Media Rendering Plugin System

• Frequently Asked Questions
• Technical Overview
• Using the Plugin Test App
• Getting Started
  • Hello World Plugin Code
• Plugin Operation and Data Flow
• Messages
  • Internal Messages

NOTE: This document reflects ongoing internal work on the Media Plugin System. The details may not match the current publicly-available code; however the overall approach and high-level features reflect the project direction.

Overview

This article specifies the messages used by the Second Life Viewer media plugin system.

Arrows indicate which direction each message is sent:

Messages the plugin receives: > for messages the plugin loader shell sends to the plugin process. For internal messages, these are from Viewer to plugin loader shell.

Messages the plugin sends: < for messages the plugin sends to the plugin loader shell. For internal messages, these are from plugin loader shell to Viewer.

Keys for each command are set off as bullet items, like this:

• foo: a sample argument

Buffer size negotiation

The size and format of the pixel buffer which media plugins draw into is negotiated between the plugin and the host via the texture_params, size_change_request, and size_change messages.

The pixel size and format used for the buffer are determined entirely by the plugin, according to what it sends in the texture_params message. Indexed pixel modes and pixels which use less than 8 bits per channel are not currently supported. Recommended modes are 24 bit RGB or 32 bit ARGB, or variants thereof (BGRA, etc).

Size negotiation is a bit more complicated:

• the host initially sets up the dimensions of the pixel buffer
  • if the plugin doesn't have any special sizing requirements, it can just draw to the buffer and never send a size_change_request message
• if the plugin sends a size_change_request message, the host takes that as the "native size" of the media
  • media is allowed to change sizes multiple times during playback, such as with streaming QuickTime movies
• If the media is playing as parcel media and the "auto scale" option is set, the draw dimensions may be increased to the next power-of-two
• if the plugin sets the allow_downsample field in the texture_params message and the media's priority has been lowered, the host may reduce the requested draw dimensions
  • the plugin should only set allow_downsample if it can draw the media at different scales without changing the overall look of the media, such as scaling down a video stream
  • media which looks different when drawn at different pixel scales (such as a web browser where font sizes depend on pixel scale) should NOT set allow_downsample
  • if it's more expensive to draw media downsampled than at native resolution, the plugin should NOT set allow_downsample -- it's supposed to be a way to reduce load on the system, and if it doesn't accomplish that it shouldn't be used
• if the plugin sets the 'padding' field in the texture_params message, the buffer's byte width will be padded to the requested alignment
  • note that it's possible to request pixel size/alignment combinations that don't come out even when using 3-byte RGB pixels. If the host detects this, it will disable padding.
• the host sends the requested drawing dimenions, the actual size of the buffer, and the name of the shared memory segment to the plugin in a size_change_response message
  • the plugin must draw within the specified dimensions, and should NOT send a size_change_request in response to the size_change_response

When changing buffer sizes, the new size may require a larger or smaller memory buffer than the old size. In this case, the shared memory segment name received with the size_change_response may be different than the one the plugin was previously drawing to. If the plugin receives a size_change_response with a shared memory segment name it hasn't seen yet, it MUST not draw to the old shared memory segment using the new parameters, or it may overrun the end of the buffer and crash.

In this case, the plugin should expect to receive a shm_added message for the new shared memory segment either before or after the size_change_response, and must wait until it has received both before drawing to the new buffer.

Likewise, when a plugin receives a shm_remove message and it's drawing to the shared memory buffer with that name, it needs to stop doing so, since the memory buffer will be deallocated once the message handler returns.

Internal messages

The Viewer and plugin loader shell exchange internal messages. Plugins do not need to handle these messages.

Messages plugin loader shell sends

hello

This is the first message the plugin loader shell sends to the Viewer when it initializes.

load_plugin_response

This message indicates the plugin is loaded and initialized.

Parameters

Key	Type	Description
versions	Map	Lists version corresponding to each plugin class. Key: message class name Value: version string. All versions start at "1.0".
plugin_version	String	Free-form version string for the plugin. This is expected to contain the versions of libraries the plugin uses (for example, quicktime, webkit, and so on) and may end up in the Viewer's about box.

heartbeat

Plugin loader shell sends these once per second to indicate that the plugin isn't locked up.

shm_add_response

This message indicates shared memory has been mapped into the child.

Parameters

Key	Type	Description
name	String	?

shm_remove_response

Plugin is done with the shared memory segment, safe to delete.

Parameters

Key	Type	Description
name	String	?

Messages plugin loader shell receives

load_plugin

Tells the loader to load a plugin and send the base init message.

Parameters

Key	Type	Description
file	String	Full path to the plugin DLL/dylib/so file.

shm_add

shared memory segment has been set up in the parent

Parameters

Key	Type	Description
name	string	?
size	S32	?

shm_remove

Shared memory segment will be removed once you respond

Parameters

Key	Type	Description
name	String	Name of shared memory segment to remove.

sleep_time

Set the interval between idle messages to the plugin.

Parameters

Key	Type	Description
time	Float	Time in seconds (default is 1/100)

Base messages

All plugins need to handle these messages.

Messages the plugin receives

init

Initialization message.

idle

Call-in for plugin to do processing, no response.

Parameters

Key	Type	Description
time	float	interval (in seconds) at which idle messages are being sent.

cleanup

the plugin is about to be unloaded, and should perform any cleanup it needs to do beforehand

shm_added

Shared memory segment has been set up.

Parameters

Key	Type	Description
name	string
size	S32	?
address	string	Since this is going from the loader shell to the plugin (i.e. not crossing a process boundary), the pointer will actually be valid. LLSD doesn't have an encoding for a pointer (or even an unsigned 32 bit value), so this will be the pointer's value as a hexadecimal string.

shm_remove

Shared memory segment will be removed once you respond.

Parameters

Key	Type	Description
name	String	Name of memory segment to remove.

Messages the plugin sends

init_response

Parameters

Key	Type	Description
versions	Map	Lists version corresponding to each plugin class. Key: message class name Value: version string. All versions start at "1.0".
plugin_version	String	Free-form version string for the plugin. This is expected to contain the versions of libraries the plugin uses (for example, quicktime, webkit, and so on) and may end up in the Viewer's about box.

shm_remove_response

Pugin is done with the shared memory segment, safe to delete.

Parameters

Key	Type	Description
name	String	Name of shared memory segment to delete.

Media messages

Messages the plugin receives

load_uri

• uri (string)

key_event

• event (string) one of:
  • down
  • repeat
  • up
• key(S32) -- platform-specific keycode
• modifiers -- string containing zero or more of the following, concatenated together with '|' characters in between. There may be a spurious trailing '|'.
  • control
  • alt -- means the 'option' key on the mac
  • shift
  • meta -- means the 'command' key on the mac

text_event

Text input event not associated with a single keystroke

• text(string) -- encoded in utf8

mouse_event

• event (string) : one of:
  • down
  • move
  • up
  • double_click
• x -- x coordinate in texture space
• y -- y coordinate in texture space (taking into account whether vertical flip was requested or not)
• modifiers -- string containing zero or more of the following, concatenated together with '|' characters in between. There may be a spurious trailing '|'.
  • control
  • alt -- means the 'option' key on the mac
  • shift
  • meta -- means the 'command' key on the mac

scroll_event

Scrollwheel

• x (integer) : horizontal scroll amount (positive or negative)
• y (integer) : vertical scroll amount (positive or negative)
• modifiers -- string containing zero or more of the following, concatenated together with '|' characters in between. There may be a spurious trailing '|'.
  • control
  • alt -- means the 'option' key on the mac
  • shift
  • meta -- means the 'command' key on the mac

size_change

Either host-initiated or after a request from the plugin

• width(S32)
• height(S32)
• texture_width(S32)
• texture_height(S32)
• name(string) -- the name of the shared memory segment the texture will draw into

set_priority

Tells the plugin its relative priority

• priority(string): one of:
  • stopped: media is not playing, shouldn't need to update at all
  • hidden: media is not being displayed or is out of view, don't need to do graphic updates, but may still update audio, playhead, etc.
  • low: media is in the far distance, may be rendered at reduced size
  • normal: default priority
  • high: media has user focus and/or is taking up most of the screen

edit_cut

Do an edit-menu "cut" operation to the system clipboard

edit_copy

Do an edit-menu "copy" operation to the system clipboard

edit_paste

Do an edit-menu "paste" operation from the system clipboard

Messages the plugin sends

texture_params

• default_width (S32) OPTIONAL -- default width for the texture
• default_height (S32) OPTIONAL -- default height for the texture
• depth (S32) -- pixel size in bytes
• internalformat (U32) -- texture internalformat for glTexImage2D
• format (U32) -- texture format for glTexImage2D
• type (U32) -- texture type for glTexImage2D
• swap_bytes(bool) OPTIONAL -- true if the host should use glPixelStorei(GL_UNPACK_SWAP_BYTES, 1) when loading the data
  • if not specified, defaults to false.
• coords_opengl(bool) -- true if the plugin wants to work in OpenGL coordinates (where (0,0) is the lower left corner of the texture), false to work in a coordinate system where (0,0) is the upper left
  • Note that the hexadecimal GLenum values for these are fixed by the OpenGL ARB, so passing them across process boundaries should be safe
  • Although the pixel size in bytes is completely determined by the format and type, deriving it from only those values is complex. To simplify things, the plugin must provide the pixel size to the host.
• allow_downsample(bool) OPTIONAL -- If true, the host may negotiate a smaller texture when the plugin's priority is reduced.
  • The plugin should not set this if downscaling the media is more expensive than drawing it at native size
• padding(S32) OPTIONAL -- Specifies how the rows of the texture should be padded. Useful values:
  • 0 (the default) -- no padding is added
  • -1 -- pad width to the next power of two
  • 16 -- start each row on a 16 byte boundary -- NOTE: may not work correctly with odd-sized pixels like GL_RGB
  • 32 -- start each row on a 32 byte boundary -- NOTE: may not work correctly with odd-sized pixels like GL_RGB

updated

• left(S32)
• top(S32)
• right(S32)
• bottom(S32)
• current_time(float) OPTIONAL -- the current playhead time in seconds
• duration(float) OPTIONAL -- the total duration of the media in seconds
• current_rate(float) OPTIONAL -- the rate at which the media is currently playing

media_status

• status (string), one of:
  • <empty string> -- not initialized or no media loaded
  • loading -- loading or navigating
  • loaded -- preroll/navigation complete
  • error -- navigation/preroll failed
  • playing -- playing (only for time-based media)
  • paused -- paused (only for time-based media)

size_change_request

When the stream wants to change its native size

• width(S32)
• height(S32)
• name(string) -- the name of the shared memory segment the texture will draw into
  • this may or may not induce the host to send a size_change message. The plugin may stop drawing to the texture until an appropriate size_change message has been received.

size_change_response

Sent by the plugin when the size change is complete

• width(S32)
• height(S32)
• texture_width(S32)
• texture_height(S32)
• name(string) -- the name of the shared memory segment the texture will draw into

cursor_changed

• name (string), one of:
  • <empty string>
  • arrow
  • ibeam
  • splith
  • splitv
  • hand

edit_state

Update the state of the edit menu items in the host

• cut(bool) OPTIONAL
• copy(bool) OPTIONAL
• paste(bool) OPTIONAL
  • items that are not present in the update message should have their state unchanged.
  • All states start out false in the host.
  • False means the operation is not possible and the menu should be disabled.

Media_browser messages

> focus

• focused (bool): true if the browser instance has focus

> clear_cache

> clear_cookies

> enable_cookies

• enable (bool)

> proxy_setup

• enable (bool)
• host (string)
• port (S32)

> browse_stop

> browse_reload

• ignore_cache(bool): if true, bypass cache for this reload

> browse_forward

> browse_back

> set_status_redirect (we use a 404 redirect in Web based search)

• code(S32): the code to redirect on (should only be 404 for now)
• url(string): the URL to redirect to

< navigate_begin

• uri (string): uri the browser is navigating to

< navigate_complete

• uri (string): uri the browser was navigating to
• result_code (S32): 200, 404, etc.
• result_string (string): "OK", "Not Found", etc.
• history_back_available (bool): true if the "back" button should be enabled
• history_forward_available (bool): true if the "forward" button should be enabled

< progress

• percent (S32): percent of page load that has completed

< status_text

• status (string): new browser status string

< location_changed

• uri (string): new browser uri for the navigation bar

< click_href

• uri (string): uri of the link
• target (string): target property of the link

< click_nofollow

• uri (string): uri of the link

> init_history

• history (LLSD, array of strings): array of string URIs to add to the browser history

Media_time messages

> stop

> start

• rate (float) OPTIONAL: rate to play at -- 0.0 is the movie's default rate, 1.0 is normal forward (on most movies), -1.0 is reverse, greater than 1.0 is fast foward, etc.

> pause

> seek

• time (float)

> set_loop

• loop (boolean): true to make the media loop, false to make it play once

> set_volume

• volume (float)