Local data storage methods used in the viewer
This page is a work in progress and should not be considered complete at this time.
This page is intended to have two different purposes.
- Document where the VFS is used in the viewer source
- Provide a direction forward for removing/replacing the VFS
VFS Initialization, Running, and Shutdown
These are all carried out by the main viewer file newview/llappviewer.cpp
Initialization
<php>268 // VFS globals - see llappviewer.h 269 LLVFS* gStaticVFS = NULL;</php>
<php>302 // File scope definitons 303 const char *VFS_DATA_FILE_BASE = "data.db2.x."; 304 const char *VFS_INDEX_FILE_BASE = "index.db2.x.";</php>
<php>765 // 766 // Initialize the VFS, and gracefully handle initialization errors 767 // 768 769 if (!initCache()) 770 { 771 std::ostringstream msg; 772 msg << 773 gSecondLife << " is unable to access a file that it needs.\n" 774 "\n" 775 "This can be because you somehow have multiple copies running, " 776 "or your system incorrectly thinks a file is open. " 777 "If this message persists, restart your computer and try again. " 778 "If it continues to persist, you may need to completely uninstall " << 779 gSecondLife << " and reinstall it."; 780 OSMessageBox( 781 msg.str().c_str(), 782 NULL, 783 OSMB_OK); 784 return 1; 785 }</php>
Main Loop
<php>889 //------------------------------------------- 890 // Run main loop until time to quit 891 //------------------------------------------- ( . . . ) 1052 while(1) 1053 { 1054 S32 work_pending = 0; 1055 S32 io_pending = 0; 1056 work_pending += LLAppViewer::getTextureCache()->update(1); // unpauses the texture cache thread 1057 work_pending += LLAppViewer::getImageDecodeThread()->update(1); // unpauses the image thread 1058 work_pending += LLAppViewer::getTextureFetch()->update(1); // unpauses the texture fetch thread 1059 io_pending += LLVFSThread::updateClass(1); 1060 io_pending += LLLFSThread::updateClass(1); 1061 if (io_pending > 1000) 1062 { 1063 ms_sleep(llmin(io_pending/100,100)); // give the vfs some time to catch up 1064 } ( . . . ) 1085 //LLVFSThread::sLocal->pause(); // Prevent the VFS thread from running while rendering. 1086 //LLLFSThread::sLocal->pause(); // Prevent the LFS thread from running while rendering.</php>
Shutdown
<php>1148 // End TransferManager before deleting systems it depends on (Audio, VFS, AssetStorage) 1149 #if 0 // this seems to get us stuck in an infinite loop... 1150 gTransferManager.cleanup(); 1151 #endif</php>
<php>1212 // Wait for any pending VFS IO 1213 while (1) 1214 { 1215 S32 pending = LLVFSThread::updateClass(0); 1216 pending += LLLFSThread::updateClass(0); 1217 if (!pending) 1218 { 1219 break; 1220 } 1221 llinfos << "Waiting for pending IO to finish: " << pending << llendflush; 1222 ms_sleep(100); 1223 }</php>
<php>1281 // 1282 // Shut down the VFS's AFTER the decode manager cleans up (since it cleans up vfiles). 1283 // Also after viewerwindow is deleted, since it may have image pointers (which have vfiles) 1284 // Also after shutting down the messaging system since it has VFS dependencies 1285 // 1286 LLVFile::cleanupClass(); 1287 llinfos << "VFS cleaned up" << llendflush;</php>
<php>1327 // Let threads finish 1328 LLTimer idleTimer; 1329 idleTimer.reset(); 1330 const F64 max_idle_time = 5.f; // 5 seconds 1331 while(1) 1332 { 1333 S32 pending = 0; 1334 pending += LLAppViewer::getTextureCache()->update(1); // unpauses the worker thread 1335 pending += LLAppViewer::getImageDecodeThread()->update(1); // unpauses the image thread 1336 pending += LLAppViewer::getTextureFetch()->update(1); // unpauses the texture fetch thread 1337 pending += LLVFSThread::updateClass(0); 1338 pending += LLLFSThread::updateClass(0); 1339 F64 idle_time = idleTimer.getElapsedTimeF64(); 1340 if (!pending || idle_time >= max_idle_time) 1341 { 1342 llwarns << "Quitting with pending background tasks." << llendl; 1343 break; 1344 } 1345 }</php>
<php>1361 // This should eventually be done in LLAppViewer 1362 LLImageJ2C::closeDSO(); 1363 LLImageFormatted::cleanupClass(); 1364 LLVFSThread::cleanupClass(); 1365 LLLFSThread::cleanupClass(); 1366 1367 llinfos << "VFS Thread finished" << llendflush; 1368 1369 #ifndef LL_RELEASE_FOR_DOWNLOAD 1370 llinfos << "Auditing VFS" << llendl; 1371 gVFS->audit(); 1372 #endif 1373 1374 // For safety, the LLVFS has to be deleted *after* LLVFSThread. This should be cleaned up. 1375 // (LLVFS doesn't know about LLVFSThread so can't kill pending requests) -Steve 1376 delete gStaticVFS; 1377 gStaticVFS = NULL; 1378 delete gVFS; 1379 gVFS = NULL;</php>
VFS Usage While Client is Running
(work in progress)
Reasons for VFS removal/replacement
As has been discussed in the sl-dev mailing list and in the JIRA, the VFS does not scale up very well at all since it functions as a private RAM-disk for non-texture assets downloaded by the viewer. The on-disk VFS files are used only for storage when the viewer is not running. At startup, the files are parsed for errors and loaded into memory, and continue to occupy memory until the viewer exits and the contents are written back to the disk files.
As a RAM disk, increasing the cache size to larger than the available free memory causes the computer to run out of physical RAM and require the use of virtual memory and disk swapping. This is counterproductive when the intent of increasing the VFS size is to cache more of the frequently-used data to speed up the client and reduce network traffic.
In order to permit the cache of assets currently stored in the VFS to grow to gigabytes of storage without requiring gigabytes of physical memory to store it, the existing VFS will have to be removed from the viewer and replaced with a traditional file/directory-based disk cache.
In the interests of copyright protection, a directory-based non-texture asset cache should include some form of low-impact data obfuscation, to make it slightly difficult for the copyrighted works of other SL users held in the cache to be casually copied and examined.