Movie System GStreamer

NOTE: This system is deprecated. Use the new Media Rendering Plugin Framework instead.

INTRODUCTION

QuickTime API exists only for MSWindows and Macs
We need to support Linux clients and other platforms
Would prefer an Open-Source video playback solution to the proprietary QuickTime API

Lots of existing in-world media stream URLs assuming a QuickTime media engine
Thus a solution needs to make it possible to play the majority of these legacy streams
We don't want to get into the business of bundling codecs, for size and licensing reasons
We don't want *hard* runtime dependencies on uncommon or hard-to-bundle libraries

Format-agnostic, streaming-agnostic, codec-agnostic runtime plugin system makes specific format support (i.e. playback of QT MOVs) someone else's problem.
Designed to be embedded in applications
Potentially supports streaming of existing QT content and a lot more.
De facto standard on reasonably modern Linux distributions; probably already installed (GStreamer 0.10.x).
Open-Source core and utility classes, with provision for a spectrum of licenses for specific plugins
Can be extended by the embedding application without modifying GStreamer itself
Can handle a lot of the output format conversion quite transparently
Very few cross-platform alternatives with comparable functionality!

Not very API or ABI stable
C-oriented API with its own bizarro object system
Spotty documentation
Mac and Windows support fairly young
No guarantee that a user has the right plugins for a particularly useful level of movie compatibility as far as Second Life content is concerned

Implement a new subclass of LLMediaImpl, wire it up to the SL app
Use GStreamer's playbin plugin to automatically set up the majority of the decode pipeline based on a provided media source URI
Implement custom audio and video sinks (as 'static', app-specific plugins) and plug those into the decode pipeline for this media source
We use APR to dlopen() the gstreamer libs and extract its symbols dynamically instead of mandating GStreamer 0.10 as a hard runtime dependancy, improving our runtime compatibility.

We need to be able to put decoded movie frames into OpenGL textures for display inside our world. These textures are subject to particular format and size constraints.

Letting GStreamer open the audio device directly is simply not very reliable on Linux (and possibly other unixoids); the SL application will already be using the audio device, and multiplexing (simultaneous app audio) support on Linux has proven to be very spotty. Thus the idea is to consume GStreamer's audio data ourself and feed it into FMOD(/whatever) for mixing into the existing SL audio.
- Note: This aspect has been punted to the future while we figure out whether this is significant issue in 'real' usage.
  - I can't speak for other distributions but in Fedora we fixed this a year ago by using dmix by default in FC5. We fixed it even more with pulseaudio in F8. Seg Baphomet 11:43, 9 December 2007 (PST)

GStreamer runs plugins in their own threads, so we need to protect access to data shared with main app. Decode is quite independent of the SL application, so the frame rate of the decoded video may be greater or lower than the frame date at which SL is consuming or displaying at.
- Implement slvideo plugin to consume decoded video frames
- slvideo's capability-negotiation with the GStreamer core ensures that we only receive video frames in RGBA or BGRA format, ready-scaled to power-of-two frame dimensions as required by OpenGL.
- When GStreamer pushes a new video frame to us, we lock the slvideo instance object and copy that frame (its pixel data, dimensions and format) into fields within it, atomically as far as other lock-observing threads are concerned. We also raise a flag as an advisory to a consuming thread (the application) that a new frame was copied into here since the flag was last lowered.

The SL application polls the video system once per SL frame by the nature of LLMediaImpl. This is the point at which it expects its own view of the video data to be updated if necessary, and be told whether there is work for it to be done this frame so it can resize/update its own private GL structures for the new frame.
- We lock the slvideo object for this stream and examine its retained frame.
- If that retained frame is not flagged as being new, we unlock the object and tell SL that there's nothing to be done, otherwise we lower the flag and continue:
- If the retained frame is a different size to the previous then we unlock the slvideo object and tell SL that we've resized.
- We copy the frame data over to SL's internal buffer, unlock the slvideo object and tell SL that an update to the GL texture is needed.