Parcel media

From Second Life Wiki
Revision as of 14:08, 28 September 2007 by Samuel Linden (talk | contribs)
Jump to navigation Jump to search

Media On A Parcel

This project comprises tasks related to supporting various different types of media in Second Life. Currently we support a single quicktime compliant URL per parcel, which is associated with a texture ID and thus displayed on any prim using that texture. We would like to expand this to include multiple URLs per parcel, and abstract the underlying media layer to allow for the easy addition of new media support. Eventually we will also support a single URL per material component, and thereby allow users to display media on any face of a prim.


Goals

The goal of this project is to expand the current functionality for playing movies on a parcel to include a variety of media types. The media system itself will be rewritten to be more scalable and allow for user created media handling and flexible URL types. It will also be extricated from the viewer and exist as a standalone application with a minimum of external dependencies. This will allow for external testing of the component.

Design

User interaction model

Creating a media URL

Land Parcel Media Options.PNG
  • Parcel owner right clicks on land and selects "About Land" and chooses Media tab
  • Enters media details:
    • media URL
    • media Type (dropdown menu from LLMediaRegistry file)
    • texture to replace
    • auto scaling checkbox
      • media size (if required, otherwise grey'ed out)
  • Hits (new) 'Apply' button
  • Client uses HTTP 'GetHeaders' to retrieve the MIME type of the media URL
  • 'Pleasing' client UI shows progress bar whilst HTTP request happens
  • If this fails (or it's not an HTTP URL) then some more primitive heuristics are used (string matching etc.) and a MIME type generated
  • Media details (including MIME type) are sent to the simulator and stored

URLSetFlow.jpg

Consuming a media URL

  • Resident enters a parcel which has a media URL defined
  • Simulator sends media information as a fragment of LLSD.
  • Transport control appears based on the media type (since the simulator knows the type now via MIME)
  • User selects transport control that 'starts' media (E.G. play the movie for QuickTime)
  • Media type (via MIME) used to select the corresponding media impl
    • Note: consider re-getting MIME type via HTTP headers in case the contents of the source URL changed since it was added
  • Media impl created and local init function called (typically lightweight init that wasn't done at startup)
  • Media Manager calls method in the impl via well defined abstract interface - e.g. 'start()'
  • Media impl does the right thing

URLDecodingFlow.jpg


Changes to UI

  • Dialog to set media URL now needs to support:
    • Selecting which media URL/texture you are changing
    • Allow name and description fields for media (to help with selecting which one you want to see later)
    • Allow for setting media type from a dropdown populated by LLMediaRegistry
    • Media width/height (if required, e.g. for Web browser)
    • Button to get MIME type and "GETTING..." progress bar
  • Playback UI now needs to support
    • More than one media source per parcel (selection via name perhaps)
    • Different transport controls based on media type (e.g. Play/pause for QuickTime, back,forward,refresh for Web)
      • May punt on this for now since Richard is making large changes in that area too - will talk to him about it.


Top level class hierarchy

File:Llmedia class structure.png
  • Singleton: LLMediaManager. All access to media is via this class.
    • Note: name change from LLMediaEngine was done on purpose to strongly differentiate it from current version
  • Singleton: LLMediaRegistry. determines, stores and updates media registry XML file
  • Singleton: LLMediaFactory. Creates instances of LLMediaImpls based on the URL MIME type.
  • Pure virtual base class: LLMediaBase.h:
    • provides well defined interface through which media impls are accessed
  • LLMediaImplCommon.cpp/.h:
    • Support for common functionality between impls - E.G. setting flags, allocating memory etc.
    • Each media impl derives from this class
  • Media impls: LLMediaImpl*.cpp/.h:
    • Impls for each media type
    • May be more than one per media type - e.g. Web may have LLMediaImplLLMozLib and LLMediaImplLLOpera
  • Inheritance chain:
    • LLmediaBase is an abstract class
    • LLMediaImplCommon derrives from LLmediaBase
    • LLMediaImplCommon provides default (stubbed) impls for LLmediaBase interface
    • LLMediaImpl* derrives from LLMediaImplCommon and overrides the methods it needs to change
  • Observer/emitter implementation: LLMediaObserver.h and LLMediaEmitter.h

Media Registration

  • Media types are associated with Impls through the Media Registry XML file
  • Each entry in the file contains the following:
    • Unique Name - e.g. Quicktime
    • MIME-type reference - e.g. video/quicktime
    • Impl reference - e.g. QuicktimeImpl

The Name field need not be unique, however there is no current use-case that would support duplicate names.


Implementation

System startup

  • LLMediaRegistry is loaded and
  • At startup, global init function (static) called for each media impl type. (breaks abstraction but any initialization that takes a long time is done here)


Creating a media URL

  • Client side:
    • Retrieve media URL
    • Gets HTTP headers using CURL and retrieves MIME type from Content-type: field
      • If no HTTP headers (or not HTTP URL) use heuristics to find media type.
      • Default to Web content
    • Retrieve texture ID, name, description, width, height and send to simulator as a block of LLSD using new message system.
  • Server side:
    • Simulator stores media information


Playing back a media URL

  • Client side:
    • Resident enters a parcel
    • Asks simulator for media details
    • Media details arrive via new message system
    • Display UI (transport controls) based on media type
    • When user presses 'start':
    • Look up media type and create appropriate impl
    • Client sets itself up to observe Media Manager
    • Store impl and return media ID
    • Client uses this media ID to reference this media source
    • Load media using this ID (E.G. LLMediaManager::getInstance()->play( mMediaId ); )
    • Start playing back media
    • When new frame of data ready, media manager emits event that client sees
    • Client decides whether to display it or not.
    • Client sees other events emitted from Media Manager (for example, media size changes, media stopped, media started)
  • Server side:
    • Client asks for media details
    • Send media details to client using new message system


Message Details

The simulator changes for this project are reasonably small. The current message will be deprecated in favor of an LLSD style TCP message that will allow for a more robust media format and multiple URL's per packet. The structure will look like:

Media LLSD Message
Value Type Comment
URL String Link to the Media
MediaType String MIME Type
Flags U32 Flags associated with the media
Name String A user specified name for the media
Description String A user specified description for the media


Media Types

  • Implemented in this project
    • QuickTime (refactor existing one and make sure we're using the latest version of the QuickTime SDK)
    • LLMozLib (Will use existing version that's in the client - updating that is another project)
  • Implemented shortly afterwards:
    • LLOperaLib (Alternative impl for handling Web based content - based on Opera browser 9.5)


Media Registry File

  • The Media Registry file provides data-driven association between media types and decoders (Impls)
  • The Type Name can be used for type-specific UI
  • Type Name is specified in About Land UI
Media Registry File
Status MIME Type Impl Type Type
enabled image/jpeg LLQUICKTIME image
enabled image/bmp LLQUICKTIME image
enabled image/png LLQUICKTIME image
enabled stream/mov LLQUICKTIME movie
enabled stream/mpeg LLQUICKTIME movie
enabled text/html LLOPERA web
disabled text/html LLMOZLIB web
XUI Combobox definition
User Friendly Name Type UI controls visible
Image image none
Movie movie Source image size, auto-scale
Web Page web Source page size
Alternate Approach
UI String MIME Type Enabled Impl Type UI Controls Visible
Image (JPEG) image/jpeg enabled LLQUICKTIME image
Image (BMP) image/bmp enabled LLQUICKTIME image
Image (PNG) image/png enabled LLQUICKTIME image
Movie (QuickTime) stream/mov enabled LLQUICKTIME movie
Movie (MPEG) stream/mpeg enabled LLQUICKTIME movie
Web Page text/html enabled LLOPERA web
Web Page text/html disabled LLMOZLIB web
XUI Combobox definition
Type UI controls visible
image none
movie Source image size, auto-scale
web Source page size
  • media_registry.xml lives in the /app_settings directory
  • XUI for pop-up contains user-friend name -> type mapping

Changes to LSL

  • All LSL media commands now needs to specify some kind of media index since we can have more than one per parcel
  • Extra commands needed to:
    • specify MIME type directly (maybe)
    • set MIME type based on HTTP headers (remember this process may take some time)
    • specify media width/height


Milestone targets

Milestone 1

  • Tentative Alpha Date: October 31st
  • Tentative Release Date: December 1st

Features

  • Allow parcel owner to specify Web based media URL that gets rendered in the same way as QuickTime does currently
  • No interactivity
  • Dynamic web content is supported
  • Includes foundation for later work - for example:
    • flexible media type discovery via MIME types
    • multiple URLs per parcel
    • unification of Web based media and LLWebBrowser control widget

Tasks

  • Media Manager -- 10 days
    • LLMediaManager (callum)
    • LLMediaRegistry (callum)
      • Make and test version of LLMediaRegistry xml (callum)
      • Create and test CURL getHeader functionality (callum)
    • LLMediaFactory (callum)
    • LLMediaBase (callum)
    • LLMediaImplCommon (callum)
  • Media Impls -- 15 days
    • LLMediaImplQuicktime (callum)
    • LLMediaImplMozilla (callum)
  • Build unit test app (callum) -- 5 days
  • Viewer Changes -- 7 days
    • Alter About Land UI xui and .cpp file (james)
    • Tie in to new transport controls UI (sam)
    • Alter media rendering code to use data written to LLTextureRaw (callum/steve)
  • Simulator/Database -- 10 days
    • Deprecate UDP msg and create TCP msg from About Land panel to Sim using capability (sam)
    • Alter Simulator code to use new msg (sam)
    • Create code to send URL to viewer via secure TCP (sam)
    • Add database support for new parcel media fields (sam)
      • Generate upgrade scripts (sam)
  • Write Milestone 1 test plan (sam) -- 1 day

Milestone 2

Features

  • Adds multiple URLs per parcel

Tasks

  • Alter About Land UI to support multiple URLs
  • Alter transport controls to support multiple URLs
  • Alter msg to support multiple URLs
  • Write simulator code to sanity check URL limit
  • Add database handling for multiple URLs
    • Generate upgrade script
  • Write Milestone 2 test plan


Milestone 3a

  • Adds interactivity
    • Individual interactivity (all interaction is local)


Milestone 3b

  • Poor mans shared interactivity via clicks or URLS (mouse clicks or URLs passed around parcel)
  • Will need a permissions system so parcel owner can specify who is able to interact with media and how.


Milestone 3c

    • True shared interactivity via browser running on server (sim)


Milestone 4

  • Media per prim.
  • Almost certainly a completely separate project but listed here for completeness.


Limitations

  • One URL per parcel initially supported.
  • Limited support of media types in early releases
  • No Interaction initially
  • Limited media size?


Performance

  • Open questions remain about performance of the HTTP::get from the simulator. This might need to be done asynchronously, or on the simulator to prevent hitches
  • The Media Manager will cache URLs to prevent re-getting same data again
  • Might have to limit number of URLs once multi-url support is added


Security concerns

  • Don't autoplay media content in case it p0wns the parcel visitor
  • How does the new secondlife:// play ball with media (I.E. don't open a classified floater when I walk onto your parcel)
  • We don't handle media dialogs for passwords etc. so any site that uses .htaccess to protect its contents or one using SSL won't work. Should we do the work to support this (HARD)?


Use cases

Milestone 1

  • Parcel owner wants to display some static signage on his parcel
    • This can be done but the visitor will have to hit 'start' before they see it.
  • Parcel owner wants to display dynamic content on his parcel - e.g. Twitter stream
    • (once visitor presses 'start', this will work as expected
  • Parcel owner puts a Web page containing a slide show that has transport controls built into the Web page
    • This works but each parcel visitor will see the slide show independently. If one user presses 'NEXT SLIDE' control, no one else will see it.
  • Parcel owner puts a Web page containing a slide show that has transport controls via a scripted object on the parcel
    • Each parcel visitor will see the same URL
  • Parcel owner puts a Web page containing a slide show that has transport controls via a scripted object on the parcel and wants to disallow visitors changing the URL
    • [My lack of knowledge of LSL here means I can't answer this but I think it should be possible to only allow the transport control object (and therefore the URL) to be used by the parcel owner]
  • Parcel owner puts a Web page containing a Flash game on his parcel
    • The current version of LLMozLib doesn't support Flash interaction so this won't work.
  • Parcel owner puts a Web page containing a Javascript game on his parcel
    • This milestone doesn't support interaction with parcel media so this won't work
  • Parcel owner wants to hide content behind a username/password set via .htaccess
    • Won't work since we don't support username/password dialog - will have to implement using Web content


Milestone 2

  • Parcel owner wants a movie screen and a Web page with movie details on the same parcel.
    • Works - raises some interesting questions about transport controls that appear at the bottom of the screen


Milestone 3a

  • Parcel owner wants to leave a Web browser on their parcel so that visitors can browser
    • Works as expected


Milestone 3b

  • Parcel owner wants to create a controlled shared experience - viewing a sequence of images with next/previous links in the page itself
    • Works as expected
  • Parcel owner wants to join his fellow 'parcelees' and browse the Internet together
    • Will work depending on the content but any page that is generated dynamically based on time, cookies etc. will most likely not work as expected.


Milestone 3c

  • Parcel owner wants to join his fellow 'parcelees' and browse the Internet together
    • Works as expected

Also: some ideas here: http://wiki.secondlife.com/wiki/User:Zero_Linden/Office_Hours/HTML_on_a_Prim_Use_Cases

Test Plan

To Be Determined


Miscellaneous unanswered questions

  • media engine should emit event when media is ready - client observes this events and acts on it when it comes in
  • limit maximum media size (2k x2k)?
  • timed updated per media impl ? N per second
  • what do we need to consider to make this work later on a HUD
  • Should we be able to render HTML content from notecards
  • Should we be able to render HTML content from LSL

Sample media URLs

```QuickTime``` Movies non-streaming (.mov)

```QuickTime``` Movies streaming (.mov)

MPEG4 (.mp4)

Flash 5 (.swf)

  • only Flash5 or earlier will play - movies created with latest version or FlashMX probably won't work
  • although we have no interactivity with flash movies, if there is a single "click to play" at the start, pressing play multiple times will often start it playing
  • http://www.campchaos.com/content/232_napsterbad_56k.swf

Animated GIF (.gif)

```QuickTime``` VR panoramas (.mov)

```QuickTime``` VR objects (.mov)

Real Time Streaming (Video Camera)

  • rtsp://216.23.173.121/mystream.sdp (live stream from hosting company we're using for GDC party)
  • rtsp://aragorn.lindenlab.com/mystream.sdp (from callum's imac - not always on and only available internally)

Streaming via HTTP

```QuickTime``` Text files

URL that causes a crash with QT6 installed http://www.americafree.tv/unicast_mov/H264test.mov

Retrieved from "https://wiki.secondlife.com/w/index.php?title=Parcel_media&oldid=33555"