Difference between revisions of "Media Plugin System Project Notes"

From Second Life Wiki
Jump to navigation Jump to search
 
(17 intermediate revisions by 3 users not shown)
Line 3: Line 3:
== Changes to existing system ==
== Changes to existing system ==


=== Client side code ===
A large section of the client code that deals with moving bytes from memory into a texture managed by the client that OpenGL understands has been rewritten. There are several reasons for this:
* A large section of the client code that deals with moving bytes from memory into a texture managed by the client that OpenGL understands will have to be rewritten. There are several reasons for this - firstly, that code has grown "organically" and needs a significant clean-up. Secondly, the process by which dynamic textures are created and managed hasn't changed since QuickTime was added to the client 4 years ago. Lastly and most importantly, the programming paradigm changes completely with this new design from a synchronous model where you call functions to a message based model.  When you send a message to a plugin, consider it as being shot into the dark. You may (or may not) get a response back at some future time. Whether a particular message will cause a response to be sent will be defined by the particular type of plugin.


* plugin will specify which transport controls should be used via a fragment of XML.
* This code has grown "organically" and needs a significant clean-up.
* The process by which dynamic textures are created and managed hasn't changed since QuickTime was added to the Viewer four years ago.
* Most importantly, the programming paradigm changes completely with this new design from a synchronous model where you call functions to a message-based model.


* The number and type of transport controls can vary over time and should support an enabled and disabled state.
A plugin will specify which transport controls should be used via a fragment of XML.


* <font color="red">QUESTION: How will we do this? Built in set of icons that XML file references plus ability to specify your own or something more straightforward?</font>
The number and type of transport controls can vary over time and should support an enabled and disabled state.


== Other notes ==


=== One-shote rendering ===


In one-shot mode, a message is sent to the plugin requesting that it render a particular media URL into a shared memory buffer.  It renders the media once, then sends a message indicating the render has completed.  This may be used when rendering large numbers of media streams at lower priorities (such as when there are a large number of streams in the user's view, but most of them are far away).
== Technical notes ==


=== Plugins in the Viewer process ===
=== Shared memory ===
The Viewer sets up shared memory segments shared with  plugin processes.  Shared memory segments are identified by a name  unique within each instance of the plugin.  Memory segments can be created, resized (reallocated), and destroyed.


An addition to the design that allows the plugin implementation shared libraries to be loaded directly by the plugin manager into the same process space is being considered. There are some circumstances when this might be preferred and whilst this functionality may not be needed immediately, it will be useful in the future. For example, in a world where many simultaneous media sources are required, having a large number of separate processes may consume a lot of system resource. If a plugin has been shown to be reliable, loading in into the same process may be reasonable. Certainly from the plugin author's point of view (and ideally for most of the system), the interface should be the same for both use cases. {{Comment|Reword, but want devs to think about possibility of loading into Viewer process.}}
Shared memory segment setup uses message semantics: when you make a request to the Viewer to modify a segment, nothing actually happens until you receive a message telling you it completed.  


The plugin interacts with the Plugin Loader Shell (PLS) through messages to set up and tear down shared memory segments, but some of those messages may be handled (or partially handled and changed) by the PLS -- they do not go directly over the control channel.


=== Existing media implementations ===
We will create native implementations for the shared memory segment with a lightweight platform abstraction. This would use the CreateFileMapping API on Windows and either mmap() or shm_open() on Mac and Linux.  The details of this implementation will be hidden from plugin authors, since it will reside in the viewer and the plugin loader shell.
* New versions of the embedded Web browser (using LLMozLib) and movie support (QuickTime and GStreamer) will be written using the new API.


* Ideally, we will include at least one additional Linden Lab written plugin and one more 3rd party plugin.
=== Proxy object ===


* Some simple examples will be included in the API documentation.
A proxy object is an instance of a C++ class in the Viewer which has detailed knowledge of the behaviors and messages of a particular type of plugin.  Each instance of a plugin being managed by the plugin manager will have a corresponding instance of a proxy object in the viewer.
 
Parts of the viewer code that need to use the services of a plugin will do so by interacting with the plugin's proxy object.  The proxy object will provide any necessary interfaces for interacting with the plugin, and will mirror any necessary state information about the plugin that may need to be queried by clients of that plugin.
 
The proxy object instance may persist for some time after the plugin instance goes away, if necessary (i.e. so that all clients of the plugin can be cleanly notified).  If a plugin crashes and is relaunched, the proxy object need not be destroyed, although any clients using the proxy will be notified that the plugin's state has been reset.
 
The proxy object may mirror some of the state of the plugin.  This can be maintained by the plugin sending unsolicited state messages when its state changes in ways meaningful to its clients.  State queries can then be answered directly by the proxy object without having to do a round-trip query with messages.
 
Each proxy object will have member functions that can be called which will cause messages to be sent to the plugin for which it is a proxy.  Likewise, each proxy object will have a mechanism (probably a listener-type interface) that will allow other code to sign up to receive relevant messages from the plugin.
 
The functions in the proxy object that clients use will have a conventional function-argument signature.  The message will be built up (for outgoing messages) or decoded (for incoming messages) inside the proxy object's implementation, so that the details of message formats aren't exposed to code that doesn't need to know them.  If the semantics of messages need to change over time, this is the level where compatibility with older plugins/message formats will be maintained.
 
=== Control channel ===
 
The Viewer and plugin will use the control channel to pass [[#Messages|messages]] back and forth.
 
To establish the control channel, the Viewer listens on a local TCP socket, and passes the socket's port to the PLS.  The plugin loader shell connects back to the indicated local TCP port.
 
To keep latency low, the system does not use the control channel for bulk data transfer.  If large amounts of data need to be passed between the plugin host and a plugin, that data should ''not'' be embedded in messages sent across the control channel.  Instead use either:
* Shared memory segments.
* Separate streams negotiated via messages.


== Testing philosophy ==
== Testing philosophy ==
* Lots of QA and user testing via a 'First Look' release and blog post encouraging users to test the more esoteric existing media set ups.
* Script to grab all media URIs from the grid, add them to a playlist that is used as input to a test harness that tries to open and render each one for a few seconds. Clearly this won't test the visuals but may find potential crashers. A lot of this exists already. It's takes a long time - several days to grab URIs and a day to run through them.


* We should think about unit tests for each media implementation. A script can run through the directory and load each one, initialize it and render a known media URL at a given "timecode". The output can be captured and written to a file. We can either visually examine the file or look for an all black frame (for example) and pass/fail automatically. Not perfect but it would catch big failures.
Think about unit tests for each media implementation. A script can run through the directory and load each one, initialize it and render a known media URL at a given "timecode". The output can be captured and written to a file. We can either visually examine the file or look for an all black frame (for example) and pass/fail automatically. Not perfect but it would catch big failures.
[[Category:Media]]
[[Category:Design Documents]]

Latest revision as of 12:35, 19 November 2009

Changes to existing system

A large section of the client code that deals with moving bytes from memory into a texture managed by the client that OpenGL understands has been rewritten. There are several reasons for this:

  • This code has grown "organically" and needs a significant clean-up.
  • The process by which dynamic textures are created and managed hasn't changed since QuickTime was added to the Viewer four years ago.
  • Most importantly, the programming paradigm changes completely with this new design from a synchronous model where you call functions to a message-based model.

A plugin will specify which transport controls should be used via a fragment of XML.

The number and type of transport controls can vary over time and should support an enabled and disabled state.


Technical notes

Shared memory

The Viewer sets up shared memory segments shared with plugin processes. Shared memory segments are identified by a name unique within each instance of the plugin. Memory segments can be created, resized (reallocated), and destroyed.

Shared memory segment setup uses message semantics: when you make a request to the Viewer to modify a segment, nothing actually happens until you receive a message telling you it completed.

The plugin interacts with the Plugin Loader Shell (PLS) through messages to set up and tear down shared memory segments, but some of those messages may be handled (or partially handled and changed) by the PLS -- they do not go directly over the control channel.

We will create native implementations for the shared memory segment with a lightweight platform abstraction. This would use the CreateFileMapping API on Windows and either mmap() or shm_open() on Mac and Linux. The details of this implementation will be hidden from plugin authors, since it will reside in the viewer and the plugin loader shell.

Proxy object

A proxy object is an instance of a C++ class in the Viewer which has detailed knowledge of the behaviors and messages of a particular type of plugin. Each instance of a plugin being managed by the plugin manager will have a corresponding instance of a proxy object in the viewer.

Parts of the viewer code that need to use the services of a plugin will do so by interacting with the plugin's proxy object. The proxy object will provide any necessary interfaces for interacting with the plugin, and will mirror any necessary state information about the plugin that may need to be queried by clients of that plugin.

The proxy object instance may persist for some time after the plugin instance goes away, if necessary (i.e. so that all clients of the plugin can be cleanly notified). If a plugin crashes and is relaunched, the proxy object need not be destroyed, although any clients using the proxy will be notified that the plugin's state has been reset.

The proxy object may mirror some of the state of the plugin. This can be maintained by the plugin sending unsolicited state messages when its state changes in ways meaningful to its clients. State queries can then be answered directly by the proxy object without having to do a round-trip query with messages.

Each proxy object will have member functions that can be called which will cause messages to be sent to the plugin for which it is a proxy. Likewise, each proxy object will have a mechanism (probably a listener-type interface) that will allow other code to sign up to receive relevant messages from the plugin.

The functions in the proxy object that clients use will have a conventional function-argument signature. The message will be built up (for outgoing messages) or decoded (for incoming messages) inside the proxy object's implementation, so that the details of message formats aren't exposed to code that doesn't need to know them. If the semantics of messages need to change over time, this is the level where compatibility with older plugins/message formats will be maintained.

Control channel

The Viewer and plugin will use the control channel to pass messages back and forth.

To establish the control channel, the Viewer listens on a local TCP socket, and passes the socket's port to the PLS. The plugin loader shell connects back to the indicated local TCP port.

To keep latency low, the system does not use the control channel for bulk data transfer. If large amounts of data need to be passed between the plugin host and a plugin, that data should not be embedded in messages sent across the control channel. Instead use either:

  • Shared memory segments.
  • Separate streams negotiated via messages.

Testing philosophy

Think about unit tests for each media implementation. A script can run through the directory and load each one, initialize it and render a known media URL at a given "timecode". The output can be captured and written to a file. We can either visually examine the file or look for an all black frame (for example) and pass/fail automatically. Not perfect but it would catch big failures.