HTTP Texture - Revision history

Gwyneth Llewelyn: /* Caching */ Replace external link with an internal one; sprinkle a bit of WikiText around

2024-02-19T19:25:13Z

Caching: Replace external link with an internal one; sprinkle a bit of WikiText around

Strife Onizuka at 22:20, 26 September 2013

2013-09-26T22:20:31Z

← Older revision		Revision as of 15:20, 26 September 2013
Line 363:		Line 363:
	Pre-existing Jira Issues		Pre-existing Jira Issues
	{\|		{\|
	{{Issue\|VWR-8473\|type=fs\|Add a menu choice to force re-download of ruthed avs, missing textures or shapes}}		{{Issue/V1\|VWR-8473\|type=fs\|Add a menu choice to force re-download of ruthed avs, missing textures or shapes}}
	{{Issue\|SVC-3446\|type=fs\|Ability to load textures on prim faces via http/https, as well as, for sculpt maps}}		{{Issue/V1\|SVC-3446\|type=fs\|Ability to load textures on prim faces via http/https, as well as, for sculpt maps}}
	{{Issue\|VWR-217\|type=fs\|New Feature -> LSL -> Dynamic Web Textures}}		{{Issue/V1\|VWR-217\|type=fs\|New Feature -> LSL -> Dynamic Web Textures}}
	\|}		\|}

Rand Linden at 18:35, 5 May 2011

2011-05-05T18:35:37Z

← Older revision		Revision as of 11:35, 5 May 2011
Line 1:		Line 1:
			{{Archive}}
	== Deployment ==		== Deployment ==
	HTTP textures have been deployed to Second Life in 2010, August.		HTTP textures have been deployed to Second Life in 2010, August.
Line 372:		Line 373:

	[[Category:Open Source Portal]]		[[Category:Open Source Portal]]
	~~[[Category:Linden Lab Projects]]~~

Opensource Obscure: /* Deployment */ update to deployment status

2010-08-04T20:17:35Z

Deployment: update to deployment status

← Older revision		Revision as of 13:17, 4 August 2010
Line 2:		Line 2:
	HTTP textures have been deployed to Second Life in 2010, August.		HTTP textures have been deployed to Second Life in 2010, August.
	* August, 3: HTTP textures enabled on 30% of Second Life regions. [https://lists.secondlife.com/pipermail/server-beta/2010-August/000083.html Read Oskar Linden's message] to the [https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta server-beta mailing list].		* August, 3: HTTP textures enabled on 30% of Second Life regions. [https://lists.secondlife.com/pipermail/server-beta/2010-August/000083.html Read Oskar Linden's message] to the [https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta server-beta mailing list].
			* August, 4: HTTP textures enabled on 80% of Second Life regions.

	== Overview ==		== Overview ==

Yoz Linden: /* HTTP API */ Added HTTP docs

2010-08-03T23:52:52Z

HTTP API: Added HTTP docs

← Older revision		Revision as of 16:52, 3 August 2010
Line 27:		Line 27:
	== HTTP API ==		== HTTP API ==

	This is the method used to retrieve textures or texture-fragments from the grid. At time of writing, requests are made		This is the method used to retrieve textures or texture-fragments from the grid. At time of writing, requests are made directly to the sim host; this may change in the future.

	===Texture retrieval===		===Texture retrieval===
Line 43:		Line 43:
	===Partial Range===		===Partial Range===

	To retrieve a ~~portion~~ of the texture, specify a <tt>Range</tt> header as described in [http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html RFC-2616]		To retrieve a fragment of the texture, specify a <tt>Range</tt> header as described in [http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html RFC-2616]

	Example:		Example:

Yoz Linden at 23:50, 3 August 2010

2010-08-03T23:50:39Z

@@ Line 24: / Line 24: @@
 * Except for jpegs specified with an "http://" URLs, all other images are still using the old fashion protocol
 * There are still quite a bit of crashes here and there... Don't use such a build for your regular SL activities (search for bugs logged against ''http-texture branch'' in PJIRA for more info)
 == Files Involved ==

Opensource Obscure: ops

2010-08-03T20:58:14Z

ops

← Older revision		Revision as of 13:58, 3 August 2010
Line 1:		Line 1:
	== Deployment ==		== Deployment ==
	HTTP textures ~~to Second Life~~ have been deployed to Second Life in 2010, August.		HTTP textures have been deployed to Second Life in 2010, August.
	* August, 3: HTTP textures enabled on 30% of Second Life regions. [https://lists.secondlife.com/pipermail/server-beta/2010-August/000083.html Read Oskar Linden's message] to the [https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta server-beta mailing list].		* August, 3: HTTP textures enabled on 30% of Second Life regions. [https://lists.secondlife.com/pipermail/server-beta/2010-August/000083.html Read Oskar Linden's message] to the [https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta server-beta mailing list].

Opensource Obscure: added section about HTTP textures currently being deployed to Second Life

2010-08-03T20:51:58Z

added section about HTTP textures currently being deployed to Second Life

← Older revision		Revision as of 13:51, 3 August 2010
Line 1:		Line 1:
	== ~~Introduction~~ ==		== Deployment ==
			HTTP textures to Second Life have been deployed to Second Life in 2010, August.
			* August, 3: HTTP textures enabled on 30% of Second Life regions. [https://lists.secondlife.com/pipermail/server-beta/2010-August/000083.html Read Oskar Linden's message] to the [https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta server-beta mailing list].

			== Overview ==
	Questions this document will answer:		Questions this document will answer:
	* What is the "http-texture" project?		* What is the "http-texture" project?

Sheet Spotter: Added simplified class diagrams for "Texture Threads" and "Texture Workers"

2009-05-21T02:17:38Z

Added simplified class diagrams for "Texture Threads" and "Texture Workers"

← Older revision		Revision as of 19:17, 20 May 2009
Line 182:		Line 182:

	=== Threads ===		=== Threads ===
			[[Image:TextureThreads20090520.png\|200px\|thumb\|right\|Texture Threads]]
			[[Image:TextureWorkers20090520.png\|200px\|thumb\|right\|Texture Workers]]
			The relationship between the threads and workers described in this section is shown in the "Texture Threads" and "Texture Workers" simplified class diagrams (found on the right hand side). You can view larger versions of the diagrams by clicking on them, or you can open the larger versions in another browser window. (Some of the variable names from the source code have been renamed in the diagrams to simplify the explanation.)

	All threads in the viewer derives from LLThread which is a wrapper around the APR thread mechanism. This package implements threads (LLThread), mutexes (LLMutex) and mutex conditions (LLCondition).		All threads in the viewer derives from LLThread which is a wrapper around the APR thread mechanism. This package implements threads (LLThread), mutexes (LLMutex) and mutex conditions (LLCondition).

Merov Linden: LLImageDecodeThread

2009-05-12T00:24:10Z

LLImageDecodeThread

← Older revision		Revision as of 12:25, 19 February 2024
Line 171:		Line 171:
	=== Caching ===		=== Caching ===

	See also the public wiki [~~http://wiki.secondlife.com/wiki/Texture_Cache~~ Texture Cache] documentation for a general overview. The writing here under covers things that are specific to the http-texture branch and questions recently asked on the sldev mailing list.		See also the public wiki [[Texture Cache]] documentation for a general overview. The writing here under covers things that are specific to the http-texture branch and questions recently asked on the sldev mailing list.

	The texture caching system implemented in lltexturecache.cpp is rather simple in its concept although it's not a simple store of raw files. The main elements of this system are:		The texture caching system implemented in lltexturecache.cpp is rather simple in its concept although it's not a simple store of raw files. The main elements of this system are:
Line 186:		Line 186:
	===== cache/texture.cache =====		===== cache/texture.cache =====

	This file simply stores the first 600 byte chunk of raw data from the original stream into a single stream of texture "headers" (note that this is a conventional name and might store more than just metadata). This system is effective in SL because the viewer retrieves textures from the network in small packets of raw data and is able to decode a partially downloaded texture. So there's no penalty in storing the first retrieved packet in a special unique "header" file when they are first streamed down. The rest of the texture file is stored as a separate "body" file.		This file simply stores the first 600-byte chunk of raw data from the original stream into a single stream of texture "headers" (note that this is a conventional name and might store more than just metadata). This system is effective in SL because the viewer retrieves textures from the network in small packets of raw data and is able to decode a partially downloaded texture. So there's no penalty in storing the first retrieved packet in a special unique "header" file when they are first streamed down. The rest of the texture file is stored as a separate "body" file.

	Each record in that file is exactly TEXTURE_CACHE_ENTRY_SIZE (600 bytes) in size per texture.		Each record in that file is exactly TEXTURE_CACHE_ENTRY_SIZE (600 bytes) in size per texture.
Line 195:		Line 195:

	In addition to the UUID (mID), each entry also contains:		In addition to the UUID (mID), each entry also contains:
	* mImageSize : the original raw file size		* <code>mImageSize</code>: the original raw file size
	* mBodySize : the size of the body file only (i.e. mBodySize = mImageSize - TEXTURE_CACHE_ENTRY_SIZE in the current case where the offset is null or zero if the file is smaller than TEXTURE_CACHE_ENTRY_SIZE)		* <code>mBodySize</code>: the size of the body file only (i.e. <code>mBodySize = mImageSize - TEXTURE_CACHE_ENTRY_SIZE</code> in the current case where the offset is null or zero if the file is smaller than TEXTURE_CACHE_ENTRY_SIZE)
	* mTime : a time stamp stating when last the texture was used.		* <code>mTime</code>: a time stamp stating when last the texture was used.

	To optimize the position of frequently used textures and purge old ones, the cache is organized using an LRU (Least Recently Used) algorithm.		To optimize the position of frequently used textures and purge old ones, the cache is organized using an LRU (Least Recently Used) algorithm.
Line 214:		Line 214:
	==== Reading and Writing the Cache ====		==== Reading and Writing the Cache ====

	Because of this (somewhat) arbitrary split between header cache and body cache, there is a little bit of copy bits acrobatic to do to recreate a seamless stream when reading the cache back. This code is implemented in LLTextureCacheRemoteWorker and can be traced between the (mState == HEADER) and (mState == BODY) sections in LLTextureCacheRemoteWorker::doRead().		Because of this (somewhat) arbitrary split between header cache and body cache, there is a little bit of copy bits acrobatic to do to recreate a seamless stream when reading the cache back. This code is implemented in LLTextureCacheRemoteWorker and can be traced between the <code>(mState == HEADER)</code> and <code>(mState == BODY)</code> sections in LLTextureCacheRemoteWorker::doRead().

	One puzzling data member of LLTextureCacheRemoteWorker to consider is mOffset. This covers data that are reserved for extra information in the formatted image buffer at creation before the readFromCache() is invoked (see LLTextureFetchWorker::doWork() in lltexturefetch.cpp). This quantity is fixed, file format specific and never changed. Those data are also not part of the raw stream of image data and should be taken out of the stream when reading the cache back.		One puzzling data member of LLTextureCacheRemoteWorker to consider is mOffset. This covers data that are reserved for extra information in the formatted image buffer at creation ''before'' the readFromCache() is invoked (see LLTextureFetchWorker::doWork() in lltexturefetch.cpp). This quantity is fixed, file format specific and never changed. Those data are also ''not'' part of the raw stream of image data and should be taken out of the stream when reading the cache back.

	This is the idea behind the various offset and skipping of data made between the (mState == HEADER) and (mState == BODY) sections in doRead().		This is the idea behind the various offset and skipping of data made between the <code>(mState == HEADER)</code> and <code>(mState == BODY)</code> sections in doRead().

	In truth, we haven't found any instance of that code being exercised with anything else but (mOffset == 0). So it's possible that this code is somewhat buggy (there's no unit test for it).		In truth, we haven't found any instance of that code being exercised with anything else but <code>(mOffset == 0)</code>. So it's possible that this code is somewhat buggy (there's no unit test for it).

	In the http-texture branch, the cache now supports jpeg cached files in addition to j2c files. However, it does not support offset writing so, again, there's no chance for the code to be ever exercised with anything else but (mOffset == 0).		In the http-texture branch, the cache now supports jpeg cached files in addition to j2c files. However, it does not support offset writing so, again, there's no chance for the code to be ever exercised with anything else but <code>(mOffset == 0)</code>.

	For this reason, we believe we should consider suppressing support for this mOffset reading/writing as it makes the code more complex and is likely to crash anyway since writing cache in http-texture does not support it.		For this reason, we believe we should consider suppressing support for this <code>mOffset</code> reading/writing as it makes the code more complex and is likely to crash anyway since writing cache in http-texture does not support it.

	=== Threads ===		=== Threads ===

← Older revision		Revision as of 17:24, 11 May 2009
Line 274:		Line 274:
	There's only one mutex (mCreationMutex) and one list (mCreationList) in the class. All access (read or write) to the mCreationList are guarded by mutex locking/unlocking. The list is a temporary buffer of data used to instantiate requests on the queued thread. Requests are then processed in order and the responder called when done.		There's only one mutex (mCreationMutex) and one list (mCreationList) in the class. All access (read or write) to the mCreationList are guarded by mutex locking/unlocking. The list is a temporary buffer of data used to instantiate requests on the queued thread. Requests are then processed in order and the responder called when done.

	Computations performed on the requests themselves are not guarded by mutexes as no one access the request data but the decoding thread. Once a work order is done, its result is made accessible to the calling thread by calling ~~the~~ responder.		Computations performed on the requests themselves are not guarded by mutexes as no one access the request data but the decoding thread. Once a work order is done, its result is made accessible to the calling thread by calling a responder.

	~~Looking into details, here~~'s ~~again~~ a narrative of how things work:		Here's a detailed narrative of how things work:
	* LLTextureFetchWorker::doWork(): when the worker reaches the (mState == DECODE_IMAGE) state, a call to decodeImage() on the LLImageDecodeThread is made.		* LLTextureFetchWorker::doWork(): when the worker reaches the (mState == DECODE_IMAGE) state, a call to decodeImage() on the LLImageDecodeThread is made.
	* As part of the call, a pointer to a newly created DecodeResponder object is passed. This responder will be called once the decoding is complete.		* As part of the call, a pointer to a newly created DecodeResponder object is passed. This responder will be called once the decoding is complete.
Line 284:		Line 284:
	* For all subsequent LLTextureFetchWorker::doWork() calls, the worker will check the mDecoded flag which will be switched to true only when the responder's completed() method will be called by the LLImageDecodeThread thread.		* For all subsequent LLTextureFetchWorker::doWork() calls, the worker will check the mDecoded flag which will be switched to true only when the responder's completed() method will be called by the LLImageDecodeThread thread.
	* During that time, the LLImageDecodeThread continues processing:		* During that time, the LLImageDecodeThread continues processing:
	** The way LLQueuedThread are implemented, they run an infinite loop and ~~call~~ update() over and over (unless the thread is paused or idle but, let's not make things too complicated in this story, let's just assume the thread is running).		** The way LLQueuedThread are implemented, they run an infinite loop and wait on condition. The main thread calls update() over and over updating the condition and making sure the processing thread gets some action (unless the thread is paused or idle but, let's not make things too complicated in this story, let's just assume the thread is running).
	** LLImageDecodeThread::update() is called, locks the list again and iterates through it.		** LLImageDecodeThread::update() is called, locks the list again and iterates through it.
	** For each record in that list, a new ImageRequest object is instantiated and is added to the queue using addRequest().		** For each record in that list, a new ImageRequest object is instantiated and is added to the queue using addRequest().
Line 290:		Line 290:
	** LLImageDecodeThread::processRequest() performs the actually decompression and decoding of the image data.		** LLImageDecodeThread::processRequest() performs the actually decompression and decoding of the image data.
	** LLImageDecodeThread::finishRequest() does some basic clean up and calls the responder's completed() method that will flip the mDecoded flag in the fetch worker.		** LLImageDecodeThread::finishRequest() does some basic clean up and calls the responder's completed() method that will flip the mDecoded flag in the fetch worker.

			For another view of that thread, you can take a look at its unit test in indra/llimage/tests/llimageworker_test.cpp. This unit test exercises the thread class in both a threaded and non threaded way.

	==== LLLFSThread ====		==== LLLFSThread ====