Local data storage methods used in the viewer

This page is a work in progress and should not be considered complete at this time.

This page is intended to have two different purposes.

• Document where the VFS is used in the viewer source
• Provide a direction forward for removing/replacing the VFS

The following list of source references is based on viewer source released 1.19.1.4 in the SVN repository, and will need future updating as the viewer source continues to evolve:

• Link to 1.19.1.4 in the SVN repository

VFS Initialization, Running, and Shutdown

In Windows, these actions are carried out by the main viewer file newview/llappviewer.cpp

Initialization

<php>268 // VFS globals - see llappviewer.h 269 LLVFS* gStaticVFS = NULL; ( . . . ) 302 // File scope definitons 303 const char *VFS_DATA_FILE_BASE = "data.db2.x."; 304 const char *VFS_INDEX_FILE_BASE = "index.db2.x."; ( . . . ) 394 " -nothread run vfs in single thread\n" </php>

<php>548 int parse_args(int argc, char **argv) ( . . . ) 824 else if (!strcmp(argv[j], "-nothread")) 825 { 826 LLVFile::ALLOW_ASYNC = FALSE; 827 llinfos << "Running VFS in nothread mode" << llendl; 828 } </php>

<php>982 bool LLAppViewer::init() ( . . . ) 1208 // 1209 // Initialize the VFS, and gracefully handle initialization errors 1210 // 1211 1212 if (!initCache()) 1213 { 1214 std::ostringstream msg; 1215 msg << 1216 gSecondLife << " is unable to access a file that it needs.\n" 1217 "\n" 1218 "This can be because you somehow have multiple copies running, " 1219 "or your system incorrectly thinks a file is open. " 1220 "If this message persists, restart your computer and try again. " 1221 "If it continues to persist, you may need to completely uninstall " << 1222 gSecondLife << " and reinstall it."; 1223 OSMessageBox( 1224 msg.str().c_str(), 1225 NULL, 1226 OSMB_OK); 1227 return 1; 1228 } </php>

<php>1965 bool LLAppViewer::initThreads() ( . . . ) 1972 LLVFSThread::initClass(enable_threads && true); 1973 LLLFSThread::initClass(enable_threads && true); ( . . . ) 2914 LLSplashScreen::update("Initializing Texture Cache..."); 2915 2916 // Init the texture cache 2917 // Allocate 80% of the cache size for textures 2918 BOOL read_only = mSecondInstance ? true : false; 2919 const S32 MB = 1024*1024; 2920 S64 cache_size = (S64)(gSavedSettings.getU32("CacheSize")) * MB; 2921 const S64 MAX_CACHE_SIZE = 1024*MB; 2922 cache_size = llmin(cache_size, MAX_CACHE_SIZE); 2923 S64 texture_cache_size = ((cache_size * 8)/10); 2924 S64 extra = LLAppViewer::getTextureCache()->initCache(LL_PATH_CACHE, texture_cache_size, read_only); 2925 texture_cache_size -= extra; 2926 2927 LLSplashScreen::update("Initializing VFS..."); 2928 2929 // Init the VFS 2930 S64 vfs_size = cache_size - texture_cache_size; 2931 const S64 MAX_VFS_SIZE = 1024 * MB; // 1 GB 2932 vfs_size = llmin(vfs_size, MAX_VFS_SIZE); 2933 vfs_size = (vfs_size / MB) * MB; // make sure it is MB aligned 2934 U32 vfs_size_u32 = (U32)vfs_size; 2935 U32 old_vfs_size = gSavedSettings.getU32("VFSOldSize") * MB; 2936 bool resize_vfs = (vfs_size_u32 != old_vfs_size); 2937 if (resize_vfs) 2938 { 2939 gSavedSettings.setU32("VFSOldSize", vfs_size_u32/MB); 2940 } 2941 llinfos << "VFS CACHE SIZE: " << vfs_size/(1024*1024) << " MB" << llendl; 2942 2943 // This has to happen BEFORE starting the vfs 2944 //time_t ltime; 2945 srand(time(NULL)); // Flawfinder: ignore 2946 U32 old_salt = gSavedSettings.getU32("VFSSalt"); 2947 U32 new_salt; 2948 char old_vfs_data_file[LL_MAX_PATH]; // Flawfinder: ignore 2949 char old_vfs_index_file[LL_MAX_PATH]; // Flawfinder: ignore 2950 char new_vfs_data_file[LL_MAX_PATH]; // Flawfinder: ignore 2951 char new_vfs_index_file[LL_MAX_PATH]; // Flawfinder: ignore 2952 char static_vfs_index_file[LL_MAX_PATH]; // Flawfinder: ignore 2953 char static_vfs_data_file[LL_MAX_PATH]; // Flawfinder: ignore 2954 2955 if (gMultipleViewersOK) 2956 { 2957 // don't mess with renaming the VFS in this case 2958 new_salt = old_salt; 2959 } 2960 else 2961 { 2962 do 2963 { 2964 new_salt = rand(); 2965 } while( new_salt == old_salt ); 2966 } 2967 2968 snprintf(old_vfs_data_file, LL_MAX_PATH, "%s%u", // Flawfinder: ignore 2969 gDirUtilp->getExpandedFilename(LL_PATH_CACHE,VFS_DATA_FILE_BASE).c_str(), 2970 old_salt); 2971 2972 // make sure this file exists 2973 llstat s; 2974 S32 stat_result = LLFile::stat(old_vfs_data_file, &s); 2975 if (stat_result) 2976 { 2977 // doesn't exist, look for a data file 2978 std::string mask; 2979 mask = gDirUtilp->getDirDelimiter(); 2980 mask += VFS_DATA_FILE_BASE; 2981 mask += "*"; 2982 2983 std::string dir; 2984 dir = gDirUtilp->getExpandedFilename(LL_PATH_CACHE,""); 2985 2986 std::string found_file; 2987 if (gDirUtilp->getNextFileInDir(dir, mask, found_file, false)) 2988 { 2989 snprintf(old_vfs_data_file, LL_MAX_PATH, "%s%s%s", dir.c_str(),

                                  gDirUtilp->getDirDelimiter().c_str(), found_file.c_str());
                                  // Flawfinder: ignore 

2990 2991 S32 start_pos; 2992 S32 length = strlen(found_file.c_str());

                                  /* Flawfinder: ignore*/ 

2993 for (start_pos = length - 1; start_pos >= 0; start_pos--) 2994 { 2995 if (found_file[start_pos] == '.') 2996 { 2997 start_pos++; 2998 break; 2999 } 3000 } 3001 if (start_pos > 0) 3002 { 3003 sscanf(found_file.c_str() + start_pos, "%d", &old_salt); 3004 } 3005 llinfos << "Default vfs data file not present, found " << old_vfs_data_file << llendl; 3006 llinfos << "Old salt: " << old_salt << llendl; 3007 } 3008 } 3009 3010 snprintf(old_vfs_index_file, LL_MAX_PATH, "%s%u", // Flawfinder: ignore 3011 gDirUtilp->getExpandedFilename(LL_PATH_CACHE,VFS_INDEX_FILE_BASE).c_str(), 3012 old_salt); 3013 3014 stat_result = LLFile::stat(old_vfs_index_file, &s); 3015 if (stat_result) 3016 { 3017 // We've got a bad/missing index file, nukem! 3018 llwarns << "Bad or missing vfx index file " << old_vfs_index_file << llendl; 3019 llwarns << "Removing old vfs data file " << old_vfs_data_file << llendl; 3020 LLFile::remove(old_vfs_data_file); 3021 LLFile::remove(old_vfs_index_file); 3022 3023 // Just in case, nuke any other old cache files in the directory. 3024 std::string dir; 3025 dir = gDirUtilp->getExpandedFilename(LL_PATH_CACHE,""); 3026 3027 std::string mask; 3028 mask = gDirUtilp->getDirDelimiter(); 3029 mask += VFS_DATA_FILE_BASE; 3030 mask += "*"; 3031 3032 gDirUtilp->deleteFilesInDir(dir, mask); 3033 3034 mask = gDirUtilp->getDirDelimiter(); 3035 mask += VFS_INDEX_FILE_BASE; 3036 mask += "*"; 3037 3038 gDirUtilp->deleteFilesInDir(dir, mask); 3039 } 3040 3041 snprintf(new_vfs_data_file, LL_MAX_PATH, "%s%u", // Flawfinder: ignore 3042 gDirUtilp->getExpandedFilename(LL_PATH_CACHE,VFS_DATA_FILE_BASE).c_str(), 3043 new_salt); 3044 3045 snprintf(new_vfs_index_file, LL_MAX_PATH, "%s%u",

                gDirUtilp->getExpandedFilename(LL_PATH_CACHE, VFS_INDEX_FILE_BASE).c_str(),
                // Flawfinder: ignore 

3046 new_salt); 3047 3048 3049 strncpy(static_vfs_data_file, gDirUtilp->getExpandedFilename(LL_PATH_APP_SETTINGS,

                "static_data.db2").c_str(), LL_MAX_PATH -1);          // Flawfinder: ignore 

3050 static_vfs_data_file[LL_MAX_PATH -1] = '\0'; 3051 strncpy(static_vfs_index_file, gDirUtilp->getExpandedFilename(LL_PATH_APP_SETTINGS,

                "static_index.db2").c_str(), LL_MAX_PATH -1);                // Flawfinder: ignore 

3052 static_vfs_index_file[LL_MAX_PATH -1] = '\0'; 3053 3054 if (resize_vfs) 3055 { 3056 llinfos << "Removing old vfs and re-sizing" << llendl; 3057 3058 LLFile::remove(old_vfs_data_file); 3059 LLFile::remove(old_vfs_index_file); 3060 } 3061 else if (old_salt != new_salt) 3062 { 3063 // move the vfs files to a new name before opening 3064 llinfos << "Renaming " << old_vfs_data_file << " to " << new_vfs_data_file << llendl; 3065 llinfos << "Renaming " << old_vfs_index_file << " to " << new_vfs_index_file << llendl; 3066 LLFile::rename(old_vfs_data_file, new_vfs_data_file); 3067 LLFile::rename(old_vfs_index_file, new_vfs_index_file); 3068 } 3069 3070 // Startup the VFS... 3071 gSavedSettings.setU32("VFSSalt", new_salt); 3072 3073 // Don't remove VFS after viewer crashes. If user has corrupt data, they can reinstall. JC 3074 gVFS = new LLVFS(new_vfs_index_file, new_vfs_data_file, false, vfs_size_u32, false); 3075 if( VFSVALID_BAD_CORRUPT == gVFS->getValidState() ) 3076 { 3077 // Try again with fresh files 3078 // (The constructor deletes corrupt files when it finds them.) 3079 llwarns << "VFS corrupt, deleted. Making new VFS." << llendl; 3080 delete gVFS; 3081 gVFS = new LLVFS(new_vfs_index_file, new_vfs_data_file, false, vfs_size_u32, false); 3082 } 3083 3084 gStaticVFS = new LLVFS(static_vfs_index_file, static_vfs_data_file, true, 0, false); 3085 3086 BOOL success = gVFS->isValid() && gStaticVFS->isValid(); 3087 if( !success ) 3088 { 3089 return false; 3090 } 3091 else 3092 { 3093 LLVFile::initClass(); 3094 return true; 3095 } 3096 }</php>

Main Loop

<php>1344 bool LLAppViewer::mainLoop() 1345 { 1346 //------------------------------------------- 1347 // Run main loop until time to quit 1348 //------------------------------------------- ( . . . ) 1497 while(1) 1498 { 1499 S32 work_pending = 0; 1500 S32 io_pending = 0; 1501 work_pending += LLAppViewer::getTextureCache()->update(1);

                                          // unpauses the texture cache thread 

1502 work_pending += LLAppViewer::getImageDecodeThread()->update(1);

                                          // unpauses the image thread 

1503 work_pending += LLAppViewer::getTextureFetch()->update(1);

                                          // unpauses the texture fetch thread 

1504 io_pending += LLVFSThread::updateClass(1); 1505 io_pending += LLLFSThread::updateClass(1); 1506 if (io_pending > 1000) 1507 { 1508 ms_sleep(llmin(io_pending/100,100));

                                                 // give the vfs some time to catch up 

1509 } 1510 1511 F64 frame_time = frameTimer.getElapsedTimeF64(); 1512 F64 idle_time = idleTimer.getElapsedTimeF64(); 1513 if (frame_time >= min_frame_time && 1514 idle_time >= min_idle_time && 1515 (!work_pending || idle_time >= max_idle_time)) 1516 { 1517 break; 1518 } 1519 } 1520 frameTimer.reset(); 1521 1522 // Prevent the worker threads from running while rendering. 1523 // if (LLThread::processorCount()==1)

                                    //pause() should only be required when on a single processor client... 

1524 if (run_multiple_threads == FALSE) 1525 { 1526 LLAppViewer::getTextureCache()->pause(); 1527 LLAppViewer::getImageDecodeThread()->pause(); 1528 // LLAppViewer::getTextureFetch()->pause();

                                          // Don't pause the fetch (IO) thread 

1529 } 1530 //LLVFSThread::sLocal->pause();

                                 // Prevent the VFS thread from running while rendering. 

1531 //LLLFSThread::sLocal->pause();

                                 // Prevent the LFS thread from running while rendering.</php>

Shutdown

<php>1561 bool LLAppViewer::cleanup() ( . . . ) 1587 // End TransferManager before deleting systems it depends on (Audio, VFS, AssetStorage) 1588 #if 0 // this seems to get us stuck in an infinite loop... 1589 gTransferManager.cleanup(); 1590 #endif ( . . . ) 1662 // Wait for any pending VFS IO 1663 while (1) 1664 { 1665 S32 pending = LLVFSThread::updateClass(0); 1666 pending += LLLFSThread::updateClass(0); 1667 if (!pending) 1668 { 1669 break; 1670 } 1671 llinfos << "Waiting for pending IO to finish: " << pending << llendflush; 1672 ms_sleep(100); 1673 } ( . . . ) 1729 // 1730 // Shut down the VFS's AFTER the decode manager cleans up (since it cleans up vfiles). 1731 // Also after viewerwindow is deleted, since it may have image pointers (which have vfiles) 1732 // Also after shutting down the messaging system since it has VFS dependencies 1733 // 1734 LLVFile::cleanupClass(); 1735 llinfos << "VFS cleaned up" << llendflush; ( . . . ) 1789 while(1) 1790 { 1791 S32 pending = 0; 1792 pending += LLAppViewer::getTextureCache()->update(1); // unpauses the worker thread 1793 pending += LLAppViewer::getImageDecodeThread()->update(1); // unpauses the image thread 1794 pending += LLAppViewer::getTextureFetch()->update(1); // unpauses the texture fetch thread 1795 pending += LLVFSThread::updateClass(0); 1796 pending += LLLFSThread::updateClass(0); ( . . . ) 1819 // This should eventually be done in LLAppViewer 1820 LLImageJ2C::closeDSO(); 1821 LLImageFormatted::cleanupClass(); 1822 LLVFSThread::cleanupClass(); 1823 LLLFSThread::cleanupClass(); 1824 1825 llinfos << "VFS Thread finished" << llendflush; 1826 1827 #ifndef LL_RELEASE_FOR_DOWNLOAD 1828 llinfos << "Auditing VFS" << llendl; 1829 gVFS->audit(); 1830 #endif 1831 1832 // For safety, the LLVFS has to be deleted *after* LLVFSThread. This should be cleaned up. 1833 // (LLVFS doesn't know about LLVFSThread so can't kill pending requests) -Steve 1834 delete gStaticVFS; 1835 gStaticVFS = NULL; 1836 delete gVFS; 1837 gVFS = NULL; </php>

VFS Usage While Client is Running

(work in progress)

Reasons for VFS removal/replacement

As has been discussed in the sl-dev mailing list and in the JIRA, the VFS does not scale up very well at all since it functions as a private RAM-disk for non-texture assets downloaded by the viewer. The on-disk VFS files are used only for storage when the viewer is not running. At startup, the files are parsed for errors and loaded into memory, and continue to occupy memory until the viewer exits and the contents are written back to the disk files.

As a RAM disk, increasing the cache size to larger than the available free memory causes the computer to run out of physical RAM and require the use of virtual memory and disk swapping. This is counterproductive when the intent of increasing the VFS size is to cache more of the frequently-used data to speed up the client and reduce network traffic.

In order to permit the cache of assets currently stored in the VFS to grow to gigabytes of storage without requiring gigabytes of physical memory to store it, the existing VFS will have to be removed from the viewer and replaced with a traditional file/directory-based disk cache.

In the interests of copyright protection, a directory-based non-texture asset cache should include some form of low-impact data obfuscation, to make it slightly difficult for the copyrighted works of other SL users held in the cache to be casually copied and examined.