Media Plugin System Project Notes
Changes to existing system
Client side code
- A large section of the client code that deals with moving bytes from memory into a texture managed by the client that OpenGL understands will have to be rewritten. There are several reasons for this - firstly, that code has grown "organically" and needs a significant clean-up. Secondly, the process by which dynamic textures are created and managed hasn't changed since QuickTime was added to the client 4 years ago. Lastly and most importantly, the programming paradigm changes completely with this new design from a synchronous model where you call functions to a message based model. When you send a message to a plugin, consider it as being shot into the dark. You may (or may not) get a response back at some future time. Whether a particular message will cause a response to be sent will be defined by the particular type of plugin.
- plugin will specify which transport controls should be used via a fragment of XML.
- The number and type of transport controls can vary over time and should support an enabled and disabled state.
- QUESTION: How will we do this? Built in set of icons that XML file references plus ability to specify your own or something more straightforward?
Other notes
In the future, the plugin loader may be responsible for querying a central authority for a certificate of authenticity for a plugin it has been asked to load. Ultimately, we will have different levels of signing so users can differentiate between plugins from Linden Lab, plugins from partner companies and 3rd party providers (Granitehead Chislehome and his penis plugins). Users will be able to set a security "zone" so that no plugin outside of their zone is ever loaded. The default setting is probably obvious. None of this will be implemented in this phase of the project and is only mentioned here in case it's appropriate to lay some groundwork earlier on.
- One invocation of the plugin per media source (per URI).
- However, consideration should be given to a system that allows multiple media sources (URIs) to share a plugin invocation. For example, for rendering one-shot Web pages, it might be perfectly acceptable for a part of the client to wait a short time before displaying a Web page whilst a different page is rendered.
- In addition, it may be optimal to cache one or more plugin invocations - for example, a significant amount of content in the Second Life client uses the embedded Web Browser so when the login page is closed and the user logs in, there is probably no point in destroying the embedded Web browser media implementation.
One-shot rendering
In one-shot mode, a message is sent to the plugin requesting that it render a particular media URL into a shared memory buffer. It renders the media once, then sends a message indicating the render has completed. This may be used when rendering large numbers of media streams at lower priorities (such as when there are a large number of streams in the user's view, but most of them are far away).
Plugins in the Viewer process
An addition to the design that allows the plugin implementation shared libraries to be loaded directly by the plugin manager into the same process space is being considered. There are some circumstances when this might be preferred and whilst this functionality may not be needed immediately, it will be useful in the future. For example, in a world where many simultaneous media sources are required, having a large number of separate processes may consume a lot of system resource. If a plugin has been shown to be reliable, loading in into the same process may be reasonable. Certainly from the plugin author's point of view (and ideally for most of the system), the interface should be the same for both use cases. Reword, but want devs to think about possibility of loading into Viewer process.
The Viewer sets up shared memory segments shared with plugin processes. Shared memory segments are identified by a name unique within each instance of the plugin. Memory segments can be created, resized (reallocated), and destroyed.
Shared memory segment setup uses message semantics: when you make a request to the Viewer to modify a segment, nothing actually happens until you receive a message telling you it completed.
The plugin interacts with the PLS through messages to set up and tear down shared memory segements, but some of those messages may be handled (or partially handled and changed) by the PLS -- they do not go directly over the control channel.
We will create native implementations for the shared memory segment with a lightweight platform abstraction. This would use the CreateFileMapping API on Windows and either mmap() or shm_open() on Mac and Linux. The details of this implementation will be hidden from plugin authors, since it will reside in the viewer and the plugin loader shell.
Existing media implementations
- New versions of the embedded Web browser (using LLMozLib) and movie support (QuickTime and GStreamer) will be written using the new API.
- Ideally, we will include at least one additional Linden Lab written plugin and one more 3rd party plugin.
- Some simple examples will be included in the API documentation.
Proxy object
A proxy object is an instance of a C++ class in the viewer which has detailed knowledge of the behaviors and messages of a particular type of plugin. Each instance of a plugin being managed by the plugin manager will have a corresponding instance of a proxy object in the viewer.
Parts of the viewer code that need to use the services of a plugin will do so by interacting with the plugin's proxy object. The proxy object will provide any necessary interfaces for interacting with the plugin, and will mirror any necessary state information about the plugin that may need to be queried by clients of that plugin.
The proxy object instance may persist for some time after the plugin instance goes away, if necessary (i.e. so that all clients of the plugin can be cleanly notified). If a plugin crashes and is relaunched, the proxy object need not be destroyed, although any clients using the proxy will be notified that the plugin's state has been reset.
The proxy object may mirror some of the state of the plugin. This can be maintained by the plugin sending unsolicited state messages when its state changes in ways meaningful to its clients. State queries can then be answered directly by the proxy object without having to do a round-trip query with messages.
The hierarchy of proxy object classes may mirror the hierarchy of message types where it makes sense to do so. For example, there will probably be different proxy object classes for browser-like and time-based media. We started out this way, but ended up merging the browser-like and time-based proxy objects into a single class, LLPluginClassMedia.
Each proxy object will have member functions that can be called which will cause messages to be sent to the plugin for which it is a proxy. Likewise, each proxy object will have a mechanism (probably a listener-type interface) that will allow other code to sign up to receive relevant messages from the plugin.
The functions in the proxy object that clients use will have a conventional function-argument signature. The message will be built up (for outgoing messages) or decoded (for incoming messages) inside the proxy object's implementation, so that the details of message formats aren't exposed to code that doesn't need to know them. If the semantics of messages need to change over time, this is the level where compatibility with older plugins/message formats will be maintained.
Control channel
The Viewer and plugin will use the control channel to pass messages back and forth.
To establish the control channel, the Viewer listens on a local TCP socket, and passes the socket's port to the PLS. The plugin loader shell connects back to the indicated local TCP port.
To keep latency low, the system does not use the control channel for bulk data transfer. If large amounts of data need to be passed between the plugin host and a plugin, that data should not be embedded in messages sent across the control channel. Instead use either:
- Shared memory segments.
- Separate streams negotiated via messages.
Testing philosophy
- Lots of QA and user testing via a 'First Look' release and blog post encouraging users to test the more esoteric existing media set ups.
- Script to grab all media URIs from the grid, add them to a playlist that is used as input to a test harness that tries to open and render each one for a few seconds. Clearly this won't test the visuals but may find potential crashers. A lot of this exists already. It's takes a long time - several days to grab URIs and a day to run through them.
- We should think about unit tests for each media implementation. A script can run through the directory and load each one, initialize it and render a known media URL at a given "timecode". The output can be captured and written to a file. We can either visually examine the file or look for an all black frame (for example) and pass/fail automatically. Not perfect but it would catch big failures.