HTTP Texture

Introduction

Questions this document will answer:

• What is the "http-texture" project?
• Where are the files involved with this project? (modified compared to trunk)
• How is this supposed to work?
• What about all those threads and their use?
• What about CURL use?

Objective of the http-texture Project

The initial idea of the project is to allow the viewer to point to any image file of any format anywhere on the internet, pull it using the http protocol and use it as a regular texture anywhere textures are used.

This involves a bunch of moving (and new) parts:

• Using any file format as textures (instead of just j2c)
• Pulling image files using HTTP (instead of UDP)
• Getting resources from any host (instead of just the LL asset server)

This is a rather ambitious and wide ranging change. As of today (March 24th, 2009), the main plumbing has been implemented with the following caveats:

• Only jpeg images are supported as another file format
• Such images must be loaded at once (no progressive rendering yet)
• Except for jpegs specified with an "http://" URLs, all other images are still using the old fashion protocol
• There are still quite a bit of crashes here and there... Don't use such a build for your regular SL activities (search for bugs logged against http-texture-10 in JIRA for more info)

Files Involved

Here are the files involved in the implementation of this feature:

/indra/llcommon/llqueuedthread.cpp        // Modifications to the virtual method LLQueuedThread::threadedUpdate()
/indra/llcommon/llqueuedthread.h          // (does nothing in the root class, see derived class)

/indra/llimage/llimage.cpp                // Initializes the image worker thread LLImageWorker::initImageWorker()
/indra/llimage/llimagedxt.cpp
/indra/llimage/llimageworker.cpp          // Main implementation of the threading for llimage objects (completely redone)
/indra/llimage/llimageworker.h

/indra/llmessage/llcurl.cpp               // Threaded CUrl implementation
/indra/llmessage/llcurl.h                 // (all of this has already been merged in trunk)
/indra/llmessage/llhttpclient.cpp         //
/indra/llmessage/llhttpclient.h           //
/indra/llmessage/llhttpassetstorage.cpp   //
/indra/llmessage/llurlrequest.cpp         //
/indra/llmessage/llurlrequest.h           //

/indra/newview/llappviewer.cpp            // Calls the init of the image decoding threads
/indra/newview/llconsole.cpp              // Changed to use mutexes and different line coloring
/indra/newview/llconsole.h                // 
/indra/newview/lltexturecache.cpp         // Changes with locked mutex, cache but without offset writing and other limitations
/indra/newview/lltexturecache.h           // Adds open file book keeping mOpenFiles
/indra/newview/lltexturefetch.cpp         // Adds most of the changes related to http texture fetching and threading
/indra/newview/lltexturefetch.h           //
/indra/newview/lltextureview.cpp          // Lots of fonts and text changes. Similar to llconsole in scope.
/indra/newview/llviewerimage.cpp          // Creates the texture request from the passed URL
/indra/newview/llviewerimage.h            // Holds the URL for the texture in mURL
/indra/newview/llviewerimagelist.cpp      // Manages the list of textures
/indra/newview/llviewerimagelist.h        // Defines getImageFromUrl()

Implementation

Fetching

Common Mechanism

The fetching mechanism uses threads to get data while the main rendering loop goes on. All threads used here inherit from LLThread so a good understanding of this class (and of LLMutex, defined in the same llthread file) is important to understand the texture fetching mechanism.

4 threads are used when requesting a tile/image/texture:

• the LLTextureCache thread: used to retrieve all textures from the local cache. This one is actually not used for http fetched textures.
• the LLTextureFetch thread: used to request textures (all types: j2c, jpg, etc...) from servers, support a variety of protocols (UDP, HTTP). That's the "front" thread of sort, the one that we hit first when posting a texture fetch request and that handles the delegation of tasks to other threads.
• the LLImageDecodeThread thread: used to convert the raw data loaded from the image file (or gathered in chunks by the network thread) into pixel data. In http texture, this is used at the end to decode the whole jpeg image in one single chunk of work.
• curl thread

Synchronization between threads is done using LLMutex, a wrap around Apache mutex to lock/unlock critical sections.

Threaded code is really hairy to track with work orders being created, checked, yielding, running, aborted, ending but not quite done, etc... It's hard to find ones way through the maze. So it's a good idea to keep in mind a little narrative, a story of what is supposed to happen. It's also a good handle to compare the code with the intent of the designer and, hopefully, fix problems.

So, how is an http texture supposed to be fetched and find its way in the viewer's data structure?

The general idea of texture fetching is the following:

• code asks for an image using LLViewerImageList::getImageFromUrl(): ideally, this is all the high level code needs to know. Textures retrieved through http are simply passed to the LLViewerImageList which handles all the fetch, load, etc... of the texture and returns an LLViewerImage pointer to the caller. The whole thing is supposed to be handled completely transparently underneath.
• this getImageFromUrl() call instantiates an LLViewerImage creation, passing the url to the constructor and adds it to the list of managed images
• the list updateImagesFetchTextures() is regularly called on the main thread
• this causes updateFetch() to be called on each image on the list
• the updateFetch() eventually calls createRequest() on the LLTextureFetch thread using the image url
• the LLTextureFetch thread adds a worker to its list to handle the request
• depending on the url and image id, the LLTextureFetchWorker calls the LLTextureCached thread or not. Note that, as of today:
  • all files starting with "file://" are treated as local files and use the local cache worker to load. This is the case of all textures used in the UI and skins.
  • all textures known by their UUID use the remote cache worker which tests if there is a local cached texture (stored from a previous session) and, if not, makes a network request. This is the code path used for all j2c scene textures.
  • all other textures are "http://" jpeg images and will trigger an http request (see below)
• if the cache has the image, it loads it and passes the resulting raw and decoded image to the fetch worker
• if not, a network request is started, if the http flag is on, this will be an HTTP request. Note that, as of today:
  • all j2c textures are using the old UDP request scheme
  • only jpeg images are using the HTTP request scheme. This code path is exercised only in the map panel which fetches its image data from the Amazon S3 repository.
• the image file is fetched
• when done, the raw image is decoded using the LLImageDecodeThread thread
• when decoded, the request is deleted and the LLViewerImage updated with the decoded image

HTTP Fetch

For the HTTP fetch mechanism, here's a zoomed in trace of what happens starting with the texture fetch request creation:

• LLTextureFetch::createRequest() is called when an http url is detected
  • LLTextureFetchWorker::LLTextureFetchWorker(): a worker is added to the queue
  • LLTextureFetchWorker::startWork() starts by initializing a bunch of things on the worker
    • LLTextureFetchWorker::setDesiredDiscard(): the code currently requests the entire image to be loaded by the worker before going back to the main thread
  • LLTextureFetchWorker::doWork() is called on the thread and allocate some time to each workers in sequence. The state of the work is updated depending of what's achieved:
    • mState = INIT
    • mState = LOAD_FROM_TEXTURE_CACHE : which actually does nothing currently for http fetched textures (no cached)
    • mState = SEND_HTTP_REQ
      • LLTextureFetch::addToHTTPQueue() and HTTPGetResponder::HTTPGetResponder() : creates a request on the http thread
    • mState = WAIT_HTTP_REQ: waits for the http thread to come back
    • during that time, the other threads might request the status of the worker using LLTextureFetch::getRequestFinished and LLTextureFetch::getFetchState
      • HTTPGetResponder::completedRaw(): eventually, the http thread completes
      • LLTextureFetch::removeFromHTTPQueue(): the fetch order is taken off the http queue
      • LLTextureFetchWorker::callbackHttpGet(): the callback is executed and the http worker deleted
    • mState = DECODE_IMAGE: is the new state of the fetch worker
      • It instantiates a DecodeResponder() on the LLImageDecodeThread and the LLTextureFetchWorker will check for it to complete
    • When completed, the file is written to cache (not implemented yet for http textures!)
    • LLTextureFetchWorker::finishWork and LLTextureFetchWorker::endWork are called
    • The request is then deleted, first it's removed from the queue, then the worker is deleted:
      • LLTextureFetch::deleteRequest()
      • LLTextureFetch::removeRequest()
      • LLTextureFetch::removeFromNetworkQueue()
      • LLTextureFetchWorker::deleteOK
      • LLTextureFetchWorker::clearPackets is eventually called

Notes

• For the moment, the file (jpg) is downloaded at once (no partial download implemented yet)
• http downloaded files are not cached yet. In most cases though, it's not very relevant since the LLViewerImageList maintains live instances of all loaded images so the cache would be useful only between sessions
• The viewer uses only non nested mutex (APR_THREAD_MUTEX_UNNESTED).
• Threads manage work units as lists (maps actually) of LLWorkerClass objects.

Caching

See also the pubic wiki Texture Cache documentation for a general overview. The writing here under covers things that are specific to the http-texture branch and questions recently asked on the sldev mailing list.

The texture caching system implemented in lltexturecache.cpp is rather simple in its concept although it's not a simple store of raw files. The main elements of this system are:

• the header cache
• the "body" files
• the LRU and time stamping mechanism

LLTextureCache operations are implemented in a worker thread (inheriting from LLWorkerThread) so all the operations on the cache system are done by worker objects (inheriting from LLWorkerClass) in parallel to the rest of the viewer's operations.

Header Cache

The idea of the Header Cache is to allow fast retrieval of basic texture information (file size, image size, color model, other metadata...) without having to open and close each cached file. Instead, a single cache file containing all of these information for all cached textures is used.

cache/texture.cache

This file simply stores the first 600 byte chunk of raw data from the original stream into a single stream of texture "headers" (note that this is a conventional name and might store more than just metadata). This system is effective in SL because the viewer retrieves textures from the network in small packets of raw data and is able to decode a partially downloaded texture. So there's no penalty in storing the first retrieved packet in a special unique "header" file when they are first streamed down. The rest of the texture file is stored as a separate "body" file.

Each record in that file is exactly TEXTURE_CACHE_ENTRY_SIZE (600 bytes) in size per texture.

cache/texture.entries

To retrieve a texture header knowing its UUID, the viewer uses a map giving for each UUID an index in the texture.cache file. This (UUID,index) table is saved in the cache/texture.entries file, alongside cache/texture.cache.

In addition to the UUID (mID), each entry also contains:

• mImageSize : the original raw file size
• mBodySize : the size of the body file only (i.e. mBodySize = mImageSize - TEXTURE_CACHE_ENTRY_SIZE in the current case where the offset is null or zero if the file is smaller than TEXTURE_CACHE_ENTRY_SIZE)
• mTime : a time stamp stating when last the texture was used.

To optimize the position of frequently used textures and purge old ones, the cache is organized using an LRU (Least Recently Used) algorithm.

Body Files

Once the first packets are received and stored in the header cache file, the rest of the file (nicknamed "BODY") is stored in a texture specific file. Note that, because of this design:

• the header part is not being duplicated in the body file so, in effect, those cached texture files are not usable as is
• if the total file size is smaller than the size of a header cache entry (TEXTURE_CACHE_ENTRY_SIZE), the whole file is actually in the header cache and no body texture cached file is created

The body file is stored in the cache/textures folder hierarchy using a name built from the UUID of the texture asset as: /[0-F]/UUID.texture

[0-F] being the first digit (in hexadecimal) of the UUID. This split between folders is to avoid running into file count limits in a folder on some platforms.

Reading and Writing the Cache

Because of this (somewhat) arbitrary split between header cache and body cache, there is a little bit of copy bits acrobatic to do to recreate a seamless stream when reading the cache back. This code is implemented in LLTextureCacheRemoteWorker and can be traced between the (mState == HEADER) and (mState == BODY) sections in LLTextureCacheRemoteWorker::doRead().

One puzzling data member of LLTextureCacheRemoteWorker to consider is mOffset. This covers data that are reserved for extra information in the formatted image buffer at creation *before* the readFromCache() is invoked (see LLTextureFetchWorker::doWork() in lltexturefetch.cpp). This quantity is fixed, file format specific and never changed. Those data are also *not* part of the raw stream of image data and should be taken out of the stream when reading the cache back.

This is the idea behind the various offset and skipping of data made between the (mState == HEADER) and (mState == BODY) sections in doRead().

In truth, we haven't found any instance of that code being exercised with anything else but (mOffset == 0). So it's possible that this code is somewhat buggy (there's no unit test for it).

In the http-texture branch, the cache now supports jpeg cached files in addition to j2c files. However, it does not support offset writing so, again, there's no chance for the code to be ever exercised with anything else but (mOffset == 0).

For this reason, we believe we should consider suppressing support for this mOffset reading/writing as it makes the code more complex and is likely to crash anyway since writing cache in http-texture does not support it.

Threads

All threads in the viewer derives from LLThread which is a wrapper around the APR thread mechanism. This package implements threads (LLThread), mutexes (LLMutex) and mutex conditions (LLCondition).

All threads involved with texture handling are of the LLQueuedThread class (derived from LLThread), meaning that they essentially run all the time and handle work bundle.

Most of those (actually all of them except LLImageDecodeThread) are "worker threads" of the LLWorkerThread class (derived from LLQueuedThread) and handle specialized work orders (derived from LLWorkerClass) in a worker queue. This allows lengthy operations (like fetching textures...) to be delegated to the thread and, asynchronously, checked for completion while the application continues to render.

The list of "workers" are organized as maps, usually indexed by images UUIDs.

While debugging or tracing threads, a good advice is to "follow the mutex". You need to identify which section of memory each mutex or condition is protecting and make sure lock/unlock are done at the right moment when writing or reading those sections of memory.

LLTextureFetch

This is the most complete thread of the list. The complexity comes from the fact that the work order on that thread do instantiate other work orders on other threads to get the work done. An additional complexity is that there are several queues and, therefore, several mutexes handled on that thread making the deciphering of the state of the work orders on the thread a tad difficult.

Basically though, the thread follows for each work order the narrative described above. Here's a complete trace of how the LLTextureFetch thread handles an http texture retrieval from the request creation to the image decoding passed to the image list:

LLTextureFetch::createRequest
LLTextureFetchWorker::LLTextureFetchWorker
LLTextureFetchWorker::calcWorkPriority, priority = 136633646
LLTextureFetchWorker::startWork
LLTextureFetchWorker::setDesiredDiscard, discard = 5, size = 33554432
LLTextureFetch::getFetchState: Fetch state = 1
LLTextureFetchWorker::doWork: Handle mState INIT
LLTextureFetchWorker::clearPackets
LLTextureFetchWorker::doWork: Handle mState LOAD_FROM_TEXTURE_CACHE
LLTextureFetchWorker::doWork: Handle mState SEND_HTTP_REQ
LLTextureFetch::getHTTPQueueSize
LLTextureFetch::getTextureBandwidth
LLTextureFetch::removeFromNetworkQueue
LLTextureFetch::addToHTTPQueue
HTTPGetResponder::HTTPGetResponder
LLTextureFetchWorker::doWork: Handle mState WAIT_HTTP_REQ
LLTextureFetch::getFetchState: Fetch state = 7
LLTextureFetchWorker::setImagePriority, priority = 1.018e+006
LLTextureFetch::getRequestFinished: Request finished = 0
HTTPGetResponder::completedRaw
LLTextureFetch::removeFromHTTPQueue
LLTextureFetchWorker::callbackHttpGet
HTTPGetResponder::~HTTPGetResponder
LLTextureFetch::getFetchState: Fetch state = 8
LLTextureFetchWorker::doWork: Handle mState DECODE_IMAGE
Push a DecodeResponder() on the LLImageDecodeThread
LLTextureFetchWorker::doWork: Handle mState DECODE_IMAGE_UPDATE
LLTextureFetchWorker::doWork: Handle mState DONE
LLTextureFetchWorker::finishWork, param = 0, completed = 1
LLTextureFetchWorker::endWork, param = 0, aborted = 0
LLTextureFetch::deleteRequest()
LLTextureFetch::removeRequest()
LLTextureFetch::removeFromNetworkQueue()
LLTextureFetchWorker::deleteOK
LLTextureFetchWorker::~LLTextureFetchWorker
LLTextureFetchWorker::clearPackets

LLTextureCache

The LLTextureCache thread takes care of saving and loading image files on the local file system, aka "cached" files as those files are supposed to be downloaded from SL servers or other web resources (e.g. Amazon S3). Note that in fact, some of those "cached" files are downloaded at install time with the application bundle so they are not truly "cached" but, heck... This thread allows the rest of the viewer application to request image files using file names, URLs or UUIDs transparently and rely on the LLTextureCache thread to load the cached version if it has already been downloaded.

Note: in the current implementation, images that are fetched using an HTTP URL are never cached on the local system. For this to work, we would need a convention to convert URLs to local file name (using a parsing convention) and there's no such system in place yet. Besides, the system would have to handle obsolete files, modified files, etc... There is however a system to convert UUIDs into local file name through LLTextureCache::getLocalFileName(). For the moment though, we do not create a UUID for each HTTP fetched texture so this caching system is not usable in that case.

The LLTextureCache thread is a worker thread (like almost all other threads in the viewer) meaning that it is started at launch and keeps running for the whole life time of the application. It handles "work orders" through a pooled list of objects types LLTextureCacheWorkers. There are actually 2 flavors of this worker:

• LLTextureCacheRemoteWorker
• LLTextureCacheLocalFileWorker: used to load local files (which file names start in "file://"), all UI textures are loaded using that worker.

One or the other flavor is created when the cache is queried through a readFromCache() call on the LLTextureCache thread. This call is done by an LLTextureFetchWorker object in the LLTextureFetch thread. For the moment, such calls are done only for local files (file name starting with "file://") and for files known only by their UUID. Files known by an "http://" URL are not hitting that call at all.

To load a local file (UI texture for instance), the sequence of calls is as follow:

LLTextureFetchWorker::CacheReadResponder::CacheReadResponder:()
LLTextureCache::readFromCache()
LLTextureCacheLocalFileWorker::LLTextureCacheLocalFileWorker()
LLTextureCacheWorker::read()
LLTextureCacheWorker::startWork()
LLTextureCacheLocalFileWorker::doRead()     // loads the entire file using LLAPRFile::readEx()
LLTextureCacheWorker::finishWork()
LLTextureCache::ReadResponder::setData: setData: imagesize = 1068, imagelocal = 1
LLTextureCache::addCompleted()
LLTextureFetchWorker::CacheReadResponder::completed: CacheReadResponder::completed() // will lock the fetch queue and transfer the image
   LLTextureFetch::lockQueue()
   LLTextureFetch::unlockQueue()
LTextureCacheWorker::endWork()
LLTextureCacheWorker::~LLTextureCacheWorker()
LLTextureFetchWorker::DecodeResponder::DecodeResponder:()
LLTextureFetchWorker::DecodeResponder::completed:()
   LLTextureFetch::lockQueue()
   LLTextureFetch::unlockQueue()
LLTextureCache::readComplete()

LLImageDecodeThread

This thread is used in http texture to decompress the image texture once the raw image has been downloaded. If you're trying to get familiar with threads in the viewer, it's interesting to read that code in detail as it is short and gives a good overview of how the various objects involved in thread signaling work in the viewer. You'll see thread, requests, mutex and responders in action in 150 or so lines of code.

There's only one mutex (mCreationMutex) and one list (mCreationList) in the class. All access (read or write) to the mCreationList are guarded by mutex locking/unlocking. The list is a temporary buffer of data used to instantiate requests on the queued thread. Requests are then processed in order and the responder called when done.

Computations performed on the requests themselves are not guarded by mutexes as no one access the request data but the decoding thread. Once a work order is done, its result is made accessible to the calling thread by calling the responder.

Looking into details, here's again a narrative of how things work:

• LLTextureFetchWorker::doWork(): when the worker reaches the (mState == DECODE_IMAGE) state, a call to decodeImage() on the LLImageDecodeThread is made.
• As part of the call, a pointer to a newly created DecodeResponder object is passed. This responder will be called once the decoding is complete.
• That call to decodeImage() pushes a new record on the mCreationList in the LLImageDecodeThread. The mCreationMutex is locked/unlocked for this call.
• At that point, no decoding code proper is really done or even started, there's just a record for a decoding request on the list. The request itself is not created either.
• The decodeImage() call returns to the LLTextureFetch thread and the state of the worker is changed to (mState == DECODE_IMAGE_UPDATE)
• For all subsequent LLTextureFetchWorker::doWork() calls, the worker will check the mDecoded flag which will be switched to true only when the responder's completed() method will be called by the LLImageDecodeThread thread.
• During that time, the LLImageDecodeThread continues processing:
  • The way LLQueuedThread are implemented, they run an infinite loop and call update() over and over (unless the thread is paused or idle but, let's not make things too complicated in this story, let's just assume the thread is running).
  • LLImageDecodeThread::update() is called, locks the list again and iterates through it.
  • For each record in that list, a new ImageRequest object is instantiated and is added to the queue using addRequest().
  • The generic LLQueuedThread code then calls LLImageDecodeThread::processRequest() and LLImageDecodeThread::finishRequest().
  • LLImageDecodeThread::processRequest() performs the actually decompression and decoding of the image data.
  • LLImageDecodeThread::finishRequest() does some basic clean up and calls the responder's completed() method that will flip the mDecoded flag in the fetch worker.

LLLFSThread

The LLLFSThread (LFS stands for "Local File System") is used to queue file access on the client machine. It can be used to both load and save data on the local file system.

In the case of texture loading, it's used to read images from the cache if the USE_LFS_READ compilation option is set (which is not set by default).

By default (USE_LFS_READ not defined), the image files are loaded using LLAPRFile which is a wrapper class around the Apache file access facility.

Links

Intro to threads and mutexes:

• Threads
• Mutex
• Condition Variables

Jira Issues

New Jira Issues

Pre-existing Jira Issues

		VWR-8473			Add a menu choice to force re-download of ruthed avs, missing textures or shapes
		SVC-3446			Ability to load textures on prim faces via http/https, as well as, for sculpt maps

Development of this feature

As of this writing (2009-03-30), development on this branch is being done in the http-texture branch on svn.secondlife.com. See HTTP Texture Development for more information about contributing