Difference between revisions of "Parcel media"

From Second Life Wiki
Jump to navigation Jump to search
(New page: {{TOCright}} ='''Media On A Parcel'''= This project comprises tasks related to supporting various different types of media in Second Life. Currently we support a single quicktime complian...)
 
m (1 revision: Redirecting KB articles to current versions on the Lithium platform)
 
(49 intermediate revisions by 18 users not shown)
Line 1: Line 1:
{{TOCright}}
{{#widget:Redirect|url=/t5/English-Knowledge-Base/Land-parcel media/ta-p/700097}}
='''Media On A Parcel'''=
This project comprises tasks related to supporting various different types of media in Second Life.  Currently we support a single quicktime compliant URL per parcel, which is associated with a texture ID and thus displayed on any prim using that texture.  We would like to expand this to include multiple URLs per parcel, and abstract the underlying media layer to allow for the easy addition of new media support.  Eventually we will also support a single URL per material component, and thereby allow users to display media on any face of a prim.
 
 
== Jira Tasks ==
Main task: https://jira.lindenlab.com/browse/SL-54603
 
== Goals ==
The goal of this project is to expand the current functionality for playing movies on a parcel to include a variety of media types.  The media system itself will be rewritten to be more scalable and allow for user created media handling and flexible URL types.  It will also be extricated from the viewer and exist as a standalone application with a minimum of external dependencies.  This will allow for external testing of the component.
 
== Why Do This ==
===Problems We Are Solving===
* The current system will scale poorly given our long-term goal of a media-rich environment.
* The media type to media decoder associations are hard coded.
* The system is dependent on the Viewer, which contributes to viewer complexity and hampers viewer abstraction initiatives.
* It's currently only possible to have one media URL per parcel even if your system can support more.
* Current location of embedded browser (hosted in client UI code via LLWebBrowserCtrl) cannot be used as a media source. A better approach is to move the implementation to the media library and replace the "Web Browser Control" widget with a more generalized "Media Widget".
* Current implementation is strongly coupled to parcel concept. Long term goals require more flexible support so that media can be used anywhere in client - for example, the HUD.
 
===Pros===
* Rewriting and documenting the Media Library API will allow for easier integration of new media types, which hopefully will involve our open source community.
* Extricating the media library from the Viewer as much as possible will decrease Viewer complexity and hopefully contribute to greater Viewer stability long term.  Debugging the Media Library will be easier as well. 
* Adding support for new media types, such as HTML, is an integral part of our long-term growth strategy and tracks closely with a couple of our Global Initiatives.  Specifically '''[https://jira.lindenlab.com/browse/SL-53206 Establish new revenue streams]''' and '''[https://jira.lindenlab.com/browse/SL-52680 Make SL's architecture the standard for 3D worlds]''' have dependencies on this project.  For example user hosted AdSense is not possible without HTML support.
* Creating a data-driven media association path will remove the compile requirements for changing media type to impl associations.
 
===Cons===
* Any change to this code has the potential to introduce bugs and break functionality. 
* This project does conceive of new features, and therefore is not purely a refactoring of existing code. 
* While media support on a parcel does address long-term goals, it is less relevant to near-term stability and new-user experience initiatives.
 
== Design ==
 
===User interaction model===
====Creating a media URL====
[[Image:Land_Parcel_Media_Options.PNG|thumb]]
[[Image:mediaui2.jpg|thumb]]
* Parcel owner right clicks on land and selects "About Land" and chooses Media tab
 
* Enters media details:
** media URL
** media Type (dropdown menu from LLMediaRegistry file)
** texture to replace
** auto scaling checkbox
*** media size (if required, otherwise grey'ed out)
 
* Hits (new) 'Apply' button
 
* Client uses HTTP 'GetHeaders' to retrieve the MIME type of the media URL
 
* 'Pleasing' client UI shows progress bar whilst HTTP request happens
 
* If this fails (or it's not an HTTP URL) then some more primitive heuristics are used (string matching etc.) and a MIME type generated
 
* Media details (including MIME type) are sent to the simulator and stored
[[Image:URLSetFlow.jpg|420px]]
====Consuming a media URL====
 
* Resident enters a parcel which has a media URL defined
 
* Simulator sends media information as a fragment of LLSD.
 
* Transport control appears based on the media type (since the simulator knows the type now via MIME)
 
* User selects transport control that 'starts' media (E.G. play the movie for QuickTime)
 
* Media type (via MIME) used to select the corresponding media impl
** Note: consider re-getting MIME type via HTTP headers in case the contents of the source URL changed since it was added
 
* Media impl created and local init function called (typically lightweight init that wasn't done at startup)
 
* Media Manager calls method in the impl via well defined abstract interface - e.g. 'start()'
 
* Media impl does the right thing
[[Image:URLDecodingFlow.jpg|420px]]
 
 
===Changes to UI===
* Dialog to set media URL now needs to support:
** Selecting which media URL/texture you are changing
** Allow name and description fields for media (to help with selecting which one you want to see later)
** Allow for setting media type from a dropdown populated by LLMediaRegistry
** Media width/height (if required, e.g. for Web browser)
** Button to get MIME type and "GETTING..." progress bar
 
* Playback UI now needs to support
** More than one media source per parcel (selection via name perhaps)
** Different transport controls based on media type (e.g. Play/pause for QuickTime, back,forward,refresh for Web)
*** May punt on this for now since Richard is making large changes in that area too - will talk to him about it.
 
 
 
===Top level class hierarchy===
[[Image:llmedia_class_structure.png|thumb]]
* Singleton: LLMediaManager. All access to media is via this class.
** Note: name change from LLMediaEngine was done on purpose to strongly differentiate it from current version
 
* Singleton: LLMediaRegistry. determines, stores and updates media registry XML file
 
* Singleton: LLMediaFactory.  Creates instances of LLMediaImpls based on the URL MIME type.
* Pure virtual base class: LLMediaBase.h:
** provides well defined interface through which media impls are accessed
 
* LLMediaImplCommon.cpp/.h:
** Support for common functionality between impls - E.G. setting flags, allocating memory etc.
** Each media impl derives from this class
 
* Media impls: LLMediaImpl*.cpp/.h:
** Impls for each media type
** May be more than one per media type - e.g. Web may have LLMediaImplLLMozLib and LLMediaImplLLOpera
 
* Inheritance chain:
** LLmediaBase is an abstract class
** LLMediaImplCommon derrives from LLmediaBase
** LLMediaImplCommon provides default (stubbed) impls for LLmediaBase interface
** LLMediaImpl* derrives from LLMediaImplCommon and overrides the methods it needs to change
 
* Observer/emitter implementation: LLMediaObserver.h and LLMediaEmitter.h
 
=== Media Registration ===
* Media types are associated with Impls through the Media Registry XML file
* Each entry in the file contains the following:
** Unique Name - e.g. Quicktime
** MIME-type reference - e.g. video/quicktime
** Impl reference - e.g. QuicktimeImpl
 
The Name field need not be unique, however there is no current use-case that would support duplicate names.
 
 
 
==Implementation==
 
 
===System startup===
* LLMediaRegistry is loaded and
* At startup, global init function (static) called for each media impl type. (breaks abstraction but any initialization that takes a long time is done here)
 
 
===Creating a media URL===
* Client side:
** Retrieve media URL
** Gets HTTP headers using CURL and retrieves MIME type from Content-type: field
*** If no HTTP headers (or not HTTP URL) use heuristics to find media type.
*** Default to Web content
** Retrieve texture ID, name, description, width, height and send to simulator as a block of LLSD using new message system.
 
* Server side:
** Simulator stores media information
 
 
===Playing back a media URL===
* Client side:
** Resident enters a parcel
** Asks simulator for media details
** Media details arrive via new message system
** Display UI (transport controls) based on media type
** When user presses 'start':
** Look up media type and create appropriate impl
** Client sets itself up to observe Media Manager
** Store impl and return media ID
** Client uses this media ID to reference this media source
** Load media using this ID  (E.G. LLMediaManager::getInstance()->play( mMediaId ); )
** Start playing back media
** When new frame of data ready, media manager emits event that client sees
** Client decides whether to display it or not.
** Client sees other events emitted from Media Manager (for example, media size changes, media stopped, media started)
 
* Server side:
** Client asks for media details
** Send media details to client using new message system
 
 
 
=== Message Details ===
The simulator changes for this project are reasonably small.  The current message will be deprecated in favor of an LLSD style TCP message that will allow for a more robust media format and multiple URL's per packet.  The structure will look like:
 
{| border="1"
|+ Media LLSD Message
! Value !! Type !! Comment
|-
| URL ||String || Link to the Media
|-
|-
| MediaType||String || MIME Type
|-
|-
| Flags||U32|| Flags associated with the media
|-
|-
|Name||String || A user specified name for the media
|-
|-
|Description||String || A user specified description for the media
|-
|}
 
 
 
===Media Types===
* Implemented in this project
** QuickTime (refactor existing one and make sure we're using the latest version of the QuickTime SDK)
** LLMozLib (Will use existing version that's in the client - updating that is another project)
 
* Implemented shortly afterwards:
** LLOperaLib (Alternative impl for handling Web based content - based on Opera browser 9.5)
 
* Implemented sometime in the future by us or ideally, by residents:
** PDF (perhaps via this cross platform SDK http://www.foxitsoftware.com/pdf/sdk/dll/)
** Ultra lightweight text renderer (perhaps via AntiGrain http://www.antigrain.com/demo/index.html)
** Ultra lightweight SVG renderer (perhaps via AntiGrain http://www.antigrain.com/svg/index.html)
** Flickr -e.g. flickr://keyword/dog/rand  or flickr://interestiness/date/20070410/rand
** VNC
 
 
 
===Media Registry File===
* The Media Registry file provides data-driven association between media types and decoders (Impls)
* The Type Name can be used for type-specific UI
* Type Name is specified in About Land UI
 
{| border="1"
|+ Media Registry File
! Status!! MIME Type!! Impl Type !! Type
|-
|enabled||image/jpeg||LLQUICKTIME||image
|-
|enabled||image/bmp||LLQUICKTIME||image
|-
|enabled||image/png||LLQUICKTIME||image
|-
|enabled||stream/mov||LLQUICKTIME||movie
|-
|enabled||stream/mpeg||LLQUICKTIME||movie
|-
|enabled||text/html||LLOPERA||web
|-
|disabled||text/html||LLMOZLIB||web
|-
|}
 
{| border="1"
|+ XUI Combobox definition
! User Friendly Name !! Type !! UI controls visible
|-
|Image||image||none
|-
|Movie||movie||Source image size, auto-scale
|-
|Web Page||web||Source page size
|-
|}
 
----
 
{| border="1"
|+ Alternate Approach
! UI String !! MIME Type !! Enabled !! Impl Type !! UI Controls Visible
|-
|Image (JPEG) ||image/jpeg||enabled||LLQUICKTIME||image
|-
|Image (BMP) ||image/bmp||enabled||LLQUICKTIME||image
|-
|Image (PNG) ||image/png||enabled||LLQUICKTIME||image
|-
|Movie (QuickTime) ||stream/mov||enabled||LLQUICKTIME||movie
|-
|Movie (MPEG) ||stream/mpeg||enabled||LLQUICKTIME||movie
|-
|Web Page||text/html||enabled||LLOPERA||web
|-
|Web Page||text/html||disabled||LLMOZLIB||web
|-
|}
 
{| border="1"
|+ XUI Combobox definition
! Type !! UI controls visible
|-
|image||none
|-
|movie||Source image size, auto-scale
|-
|web||Source page size
|-
|}
 
* media_registry.xml lives in the /app_settings directory
* XUI for pop-up contains user-friend name -> type mapping
 
===Changes to LSL===
* All LSL media commands now needs to specify some kind of media index since we can have more than one per parcel
* Extra commands needed to:
** specify MIME type directly (maybe)
** set MIME type based on HTTP headers (remember this process may take some time)
** specify media width/height
 
 
 
==Milestone targets==
===Milestone 1===
* Tentative Alpha Date: October 31st
* Tentative Release Date: December 1st
====Features====
* Allow parcel owner to specify Web based media URL that gets rendered in the same way as QuickTime does currently
* No interactivity
* Dynamic web content is supported
* Includes foundation for later work - for example:
** flexible media type discovery via MIME types
** multiple URLs per parcel
** unification of Web based media and LLWebBrowser control widget
 
====Tasks====
* Media Manager -- 10 days
** LLMediaManager ('''callum''')
** LLMediaRegistry ('''callum''')
*** Make and test version of LLMediaRegistry xml ('''callum''')
*** Create and test CURL getHeader functionality ('''callum''')
** LLMediaFactory ('''callum''')
** LLMediaBase ('''callum''')
** LLMediaImplCommon ('''callum''')
* Media Impls -- 15 days
** LLMediaImplQuicktime ('''callum''')
** LLMediaImplMozilla ('''callum''')
* Build unit test app ('''callum''') -- 5 days
* Viewer Changes -- 7 days
** Alter About Land UI xui and .cpp file ('''james''')
** Tie in to new transport controls UI ('''sam''')
** Alter media rendering code to use data written to LLTextureRaw ('''callum/steve''')
* Simulator/Database -- 10 days
** Deprecate UDP msg and create TCP msg from About Land panel to Sim using capability ('''sam''')
** Alter Simulator code to use new msg ('''sam''')
** Create code to send URL to viewer via secure TCP ('''sam''')
** Add database support for new parcel media fields ('''sam''')
*** Generate upgrade scripts ('''sam''')
* Write Milestone 1 test plan ('''sam''') -- 1 day
 
===Milestone 2===
====Features====
* Adds multiple URLs per parcel
 
====Tasks====
* Alter About Land UI to support multiple URLs
* Alter transport controls to support multiple URLs
* Alter msg to support multiple URLs
* Write simulator code to sanity check URL limit
* Add database handling for multiple URLs
** Generate upgrade script
* Write Milestone 2 test plan
 
 
===Milestone 3a===
* Adds interactivity
** Individual interactivity  (all interaction is local)
 
 
===Milestone 3b===
* Poor mans shared interactivity via clicks or URLS (mouse clicks or URLs passed around parcel)
* Will need a permissions system so parcel owner can specify who is able to interact with media and how.
 
 
===Milestone 3c===
** True shared interactivity via browser running on server (sim)
 
 
===Milestone 4===
* Media per prim.
* Almost certainly a completely separate project but listed here for completeness.
 
 
 
== Limitations ==
* One URL per parcel initially supported.
* Limited support of media types in early releases
* No Interaction initially
* Limited media size?
 
 
 
== Performance ==
* Open questions remain about performance of the HTTP::get from the simulator.  This might need to be done asynchronously, or on the simulator to prevent hitches
* The Media Manager will cache URLs to prevent re-getting same data again
* Might have to limit number of URLs once multi-url support is added
 
 
 
==Security concerns==
* Don't autoplay media content in case it p0wns the parcel visitor
* How does the new secondlife:// play ball with media (I.E. don't open a classified floater when I walk onto your parcel)
* We don't handle media dialogs for passwords etc. so any site that uses .htaccess to protect its contents or one using SSL won't work. Should we do the work to support this (HARD)?
 
 
 
==Use cases==
===Milestone 1===
 
* Parcel owner wants to display some static signage on his parcel
** This can be done but the visitor will have to hit 'start' before they see it.
 
* Parcel owner wants to display dynamic content on his parcel - e.g. Twitter stream
** (once visitor presses 'start', this will work as expected
 
* Parcel owner puts a Web page containing a slide show that has transport controls built into the Web page
** This works but each parcel visitor will see the slide show independently. If one user presses 'NEXT SLIDE' control, no one else will see it.
 
* Parcel owner puts a Web page containing a slide show that has transport controls via a scripted object on the parcel
** Each parcel visitor will see the same URL
 
* Parcel owner puts a Web page containing a slide show that has transport controls via a scripted object on the parcel and wants to disallow visitors changing the URL
** [My lack of knowledge of LSL here means I can't answer this but I think it should be possible to only allow the transport control object (and therefore the URL) to be used by the parcel owner]
 
* Parcel owner puts a Web page containing a Flash game on his parcel
** The current version of LLMozLib doesn't support Flash interaction so this won't work.
 
* Parcel owner puts a Web page containing a Javascript game on his parcel
** This milestone doesn't support interaction with parcel media so this won't work
 
* Parcel owner wants to hide content behind a username/password set via .htaccess
** Won't work since we don't support username/password dialog - will have to implement using Web content
 
 
===Milestone 2===
* Parcel owner wants a movie screen and a Web page with movie details on the same parcel.
** Works - raises some interesting questions about transport controls that appear at the bottom of the screen
 
 
===Milestone 3a===
* Parcel owner wants to leave a Web browser on their parcel so that visitors can browser
** Works as expected
 
 
===Milestone 3b===
* Parcel owner wants to create a controlled shared experience - viewing a sequence of images with next/previous links in the page itself
** Works as expected
 
* Parcel owner wants to join his fellow 'parcelees' and browse the Internet together
** Will work depending on the content but any page that is generated dynamically based on time, cookies etc. will most likely not work as expected.
 
 
===Milestone 3c===
* Parcel owner wants to join his fellow 'parcelees' and browse the Internet together
** Works as expected
 
Also: some ideas here: http://wiki.secondlife.com/wiki/User:Zero_Linden/Office_Hours/HTML_on_a_Prim_Use_Cases
 
== Test Plan==
To Be Determined
 
 
 
==Miscellaneous unanswered questions==
* media engine should emit event when media is ready - client observes this events and acts on it when it comes in
* limit maximum media size (2k x2k)?
* timed updated per media impl ? N per second
* what do we need to consider to make this work later on a HUD
* Should we be able to render HTML content from notecards
* Should we be able to render HTML content from LSL
 
== Sample media URLs ==
'''```QuickTime``` Movies non-streaming (.mov)'''
*http://movies.apple.com/movies/miramax/sin_city/sin_city_480.mov
*http://movies.apple.com/movies/disney/the_incredibles/the_incredibles-tlr_a480.mov
*http://movies.apple.com/movies/disney/cars/cars-teaser_320.mov
*http://www2.bc.edu/~lewbel/shortsha.mov
*http://movies.apple.com/movies/fox/fantastic_four/fantastic_four_480.mov
*http://www.madoverlord.com/Movies/TSN/I-TornadoMer-Megabyte.mov
*http://webdev.lindenlab.com/media/guinesssurfer.mov (amazing ad a friend worked on)
*http://webdev.lindenlab.com/media/johnniewalkerfish.mov (more of the same)
 
'''```QuickTime``` Movies streaming (.mov)'''
*http://stream.qtv.apple.com/events/jan/macworld/2005/macworld_hi_300_100_56_ref.mov
 
'''MPEG4 (.mp4)'''
*http://www.archive.org/download/doa_1949/doa_1949_256kb.mp4
 
'''Flash 5 (.swf)'''
*only Flash5 or earlier will play - movies created with latest version or FlashMX probably won't work
*although we have no interactivity with flash movies, if there is a single "click to play" at the start, pressing play multiple times will often start it playing
*http://www.campchaos.com/content/232_napsterbad_56k.swf
 
'''Animated GIF (.gif)'''
*```http://www.geocities.com/callumprentice/lindenlab/weather.gif```
 
'''```QuickTime``` VR panoramas (.mov)'''
*http://www.panoramas.dk/movies3/ts2004_3600r.mov
 
'''```QuickTime``` VR objects (.mov)'''
*http://images.apple.com/hardware/gallery/media/pmac_g5_jun2004_480.mov
 
'''Real Time Streaming (Video Camera)'''
*rtsp://216.23.173.121/mystream.sdp (live stream from hosting company we're using for GDC party)
*rtsp://aragorn.lindenlab.com/mystream.sdp  (from callum's imac - not always on and only available internally)
 
'''Streaming via HTTP'''
* http://stream.qtv.apple.com/events/jan/macworld/2005/macworld_hi_300_100_56_ref.mov
 
'''```QuickTime``` Text files'''
*rudimentary markup language - supports fonts, colours, size, position etc.
*syntax here: http://www.apple.com/quicktime/tools_tips/tutorials/textdescriptors.html
*http://www.geocities.com/callumprentice/lindenlab/example.txt  (temporary location)
 
'''URL that causes a crash with QT6 installed'''
http://www.americafree.tv/unicast_mov/H264test.mov
 
[[Category:Design Documents]]
[[Category:System Architecture]]

Latest revision as of 10:25, 2 March 2011

Redirecting to http://community.secondlife.com/t5/English-Knowledge-Base/Land-parcel media/ta-p/700097

Retrieved from "https://wiki.secondlife.com/w/index.php?title=Parcel_media&oldid=1136257"