Difference between revisions of "VFS: understanding the Index file layout"

From Second Life Wiki
Jump to navigation Jump to search
 
(clarify, VFS still in use, but just not for textures)
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This is an attempt at dissecting the data arrangement within a new Index file.
On this page you can find a short description of the index file layout including some basic nice to knows. Up until client verion 1.13.2 the VFS was used to store all asset types, but textures have been split off as a direct file store in the cache directory. The VFS continues to be in use, however, and is used to store many non-texture assets.<BR>
 
<BR>
This is the complete raw bytecode of a new Index, created by:
The index-file contains <b>34-byte long records</b> which contain the following fields (listed in order):<BR>
* setting the cache size to 50 megabytes
<b>Location (4 Bytes):</b> The location within the cache file counted in bytes from start of file.<BR>
* setting the bandwidth to 50 kilobits
<b>Length (4 Bytes):</b> The reserved amount of space in the cache file for this item, counted in bytes (multiples of 1024 as 1024 is the block size)<BR>
* exiting the client
<b>Accesstime (4 Bytes):</b> Timestamp of last access, is updated when file is read from cache and is set on intial addition in the cache. Counted in seconds since 1/1/1970 (commonly known as unix time or POSIX time)<BR>
* deleting any existing cache folder
<b>UUID (16 Bytes):</b> The unique ID of the asset<BR>
* starting the client
<b>Filetype (2 Bytes):</b> The asset type, mostly 0 (textures)<BR>
* letting the world begin to load for about 30 seconds
<b>Size (4 Bytes):</b> The size, counted in bytes, of the data currently stored for this asset. This can be significantly lower than the Length<BR>
* quit the client
<BR>
* dump the Index with a hex editor
Loading the world with a blank cache and subsequent analysis of the index file shows that the caching system is very inefficient. Due to its implementation files can not be stored fragmented hence the expected required amount of space is immediately reserved in the cache. Yet most files are not download in their entirety. Just a few minutes after login a clean cache will have over 50MB of cache reserved which means small caches are totally pointless as it'll start removing items shortly after you start the client.<BR>
 
<BR>
Here are the raw data bytes with an offset header at the beginning of each 16-byte line. This is from the file <b>C:\Documents and Settings\admin\Application Data\SecondLife\cache\index.db2.x.8714</b>
<FONT COLOR=RED>NOTE:</FONT>As said above, as of 1.13.3 the system changes radically for textures and the newly implemented texture cache is a lot more efficient.
 
0000: 00 00 00 00 00 04 00 00 0e 22 b1 45 04 60 8b 19
0010: 16 68 39 9c 57 90 b7 df 8d 9d c9 cf 15 00 70 00
0020: 00 00 00 04 00 00 00 04 00 00 0e 22 b1 45 03 dc
0030: 9f f9 46 43 55 d9 84 3b 51 45 42 dc a5 48 15 00
0040: 3e 00 00 00 00 08 00 00 00 04 00 00 0e 22 b1 45
0050: 4b 70 87 16 cf 36 8b 73 83 ae 3c 10 f4 02 1b f4
0060: 15 00 4e 00 00 00 00 0c 00 00 00 04 00 00 0e 22
0070: b1 45 0a 01 0d 21 3f 86 00 2b c8 b4 3f 13 1a da
0080: a3 b6 15 00 28 01 00 00 00 10 00 00 00 48 00 00
0090: 11 22 b1 45 fc 4b 9f 0b d0 08 45 c6 96 a4 01 dd
00a0: 94 7a c6 21 00 00 27 12 00 00 00 58 00 00 00 84
00b0: 01 00 0f 22 b1 45 f2 50 fd e6 91 04 7c 78 a4 82
00c0: 53 74 39 cb 6d 21 00 00 63 06 00 00 00 dc 01 00
00d0: 00 04 00 00 15 22 b1 45 d4 e9 38 f2 68 ed 87 62
00e0: 53 85 45 c8 12 dd 08 bc 0d 00 06 02 00 00 00 e0
00f0: 01 00 00 04 00 00 15 22 b1 45 7a 2a 9c ac 9f 4c
0100: f1 f4 0d 83 c6 a2 7a 72 2a 7f 0d 00 12 03 00 00
0110: 00 e4 01 00 00 88 01 00 1a 22 b1 45 7d bb 2c cc
0120: 22 85 91 31 34 8d db 95 97 39 26 73 00 00 d7 21
0130: 00 00 00 6c 03 00 00 28 00 00 15 22 b1 45 16 3c
0140: ae 3c ba e6 c9 2f 44 27 20 8c bb 2f 76 d2 00 00
0150: 77 02 00 00 00 94 03 00 00 1c 00 00 15 22 b1 45
0160: 55 ee 69 93 b1 df 2d 68 7a 8b 9a 04 6f 4f 7d 1b
0170: 00 00 77 02 00 00 00 b0 03 00 00 24 00 00 15 22
0180: b1 45 8a 6b 75 08 02 85 07 d1 12 4e 98 c0 b9 f0
0190: 10 1c 00 00 77 02 00 00 00 d4 03 00 00 04 00 00
01a0: 15 22 b1 45 a9 d1 0d d1 f2 d2 4f fd 7a e3 3d 65
01b0: 41 7c 77 82 0d 00 f4 02 00 00 00 d8 03 00 00 08
01c0: 00 00 16 22 b1 45 a8 29 d0 33 f9 ee 87 7b 25 84
01d0: 7c 0b d2 77 38 9d 0d 00 27 04 00 00 00 e0 03 00
01e0: 00 04 00 00 17 22 b1 45 3f 6e a2 f6 58 0e 8d 56
01f0: f0 a1 00 48 4e 34 03 a0 00 00 7a 01 00 00 00 e4
0200: 03 00 00 24 00 00 17 22 b1 45 31 5d e2 8a 1d 21
0210: 8d 55 63 38 4e 3e 0c 2f e7 ca 00 00 94 23 00 00
0220: 00 08 04 00 00 14 00 00 21 22 b1 45 4c 8c 3c 77
0230: de 8d bd e2 b9 b8 32 63 5e 0f d4 a6 01 00 d0 11
0240: 00 00 00

Latest revision as of 10:44, 7 June 2008

On this page you can find a short description of the index file layout including some basic nice to knows. Up until client verion 1.13.2 the VFS was used to store all asset types, but textures have been split off as a direct file store in the cache directory. The VFS continues to be in use, however, and is used to store many non-texture assets.

The index-file contains 34-byte long records which contain the following fields (listed in order):
Location (4 Bytes): The location within the cache file counted in bytes from start of file.
Length (4 Bytes): The reserved amount of space in the cache file for this item, counted in bytes (multiples of 1024 as 1024 is the block size)
Accesstime (4 Bytes): Timestamp of last access, is updated when file is read from cache and is set on intial addition in the cache. Counted in seconds since 1/1/1970 (commonly known as unix time or POSIX time)
UUID (16 Bytes): The unique ID of the asset
Filetype (2 Bytes): The asset type, mostly 0 (textures)
Size (4 Bytes): The size, counted in bytes, of the data currently stored for this asset. This can be significantly lower than the Length

Loading the world with a blank cache and subsequent analysis of the index file shows that the caching system is very inefficient. Due to its implementation files can not be stored fragmented hence the expected required amount of space is immediately reserved in the cache. Yet most files are not download in their entirety. Just a few minutes after login a clean cache will have over 50MB of cache reserved which means small caches are totally pointless as it'll start removing items shortly after you start the client.

NOTE:As said above, as of 1.13.3 the system changes radically for textures and the newly implemented texture cache is a lot more efficient.

Retrieved from "https://wiki.secondlife.com/w/index.php?title=VFS:_understanding_the_Index_file_layout&oldid=70480"