SLSquid Proxy
The main scope of this proxy is to not burden the Second Life simulation servers with the network traffic that can be made external. For example, we want to increase the quality of the images being sent to the viewers without an increased quantity of network traffic on the simulation servers. Over the course of many ideas, talks, benchmarks, and other errata, we want enable lossless images being sent to the viewer by the use of a SLSquid proxy mesh-network.
A Vision
Kamilion Schnook wrote an excellent vision about SLSquid from the discussion on SLDev:
Okay, here's another idea -- Squid currently supports Store Digests; Which is a bitmap that lossilly represents the contents of an entire squid cache, biased to hits. It also supports Client Streams; which allow other applications to access the squid cache.
It also runs natively on windows built with mingw32.
Technically, it shouldn't be too hard to build a control program that pulls the store digest from squid, and integrates a torrent tracker, torrent client, and some form of SuperTracker client that keeps track of the various "SLSquid" servers.
This solves most of the problem with 'revealed' IP addresses.
We would end up with a meshed network of SLSquids that could transfer textures between each other in the background.
At that point, the SuperTracker is managing a round robin DNS containing all of the known SLSquids, which the SL client could be pointed at.
This same sort of DNS management is commonly used in some larger IRC networks, where you connect to irc.network.com and you're redirected to a server appropriate for your geographical region.
This also keeps a central point of authority which would be useful to block abusive users and/or servers.
- From here, we have a meshed network of interconnected aggregate caches that serve content between each other on demand with the slower torrent protocol, while serving content to SL clients via fast HTTP from the cache. It would also allow later addition of the SL clients directly downloading full 'packs' of data from the caches, containing an entire object's worth of data: Say you have a linked object with 163 prims, each prim displaying 5 independent textures, all of them completely different. That's 815 textures of various sizes; not including sculpt textures. Add those in to get say, an even 900 textures. Plus this particular object has 20 sounds, and 80 animations, for an even 1000 assets. Let's say it's a dancefloor.
This entire objectmass can be placed into a single .torrent, with all 1000 of those assets referenced. SLSquids or SLClients could pull down an entire object relatively quickly.
One of the slowest parts of the torrent protocol is the fact that is has to discover peers; which we're 'solving' by making the torrent tracker a peer also, which has the benefit of quickly connecting to other peers that are also connecting to the torrent tracker's client. In this way, DHT networks can be formed quickly, and blocks can begin transferring immediately. Also, since bittorrent supports variable block sizes, faster block completion can be achived by using small blocks, since we rely on the fact that we're not transferring massive amounts of data in each load -- even a object with many assets would likely be under 10MB, and BT excels at packs of small files. Object-level torrents would be a huge advantage over single asset methods, mainly due to the fact that I havn't really seen the viewer load a partial object; only octree a full object's visible prims away.
Torrent clients also have quite good bandwidth limiting, and a forced limit in the SL client could easily be chosen by default, such as 3KB/sec upload. Since the SLSquids take care of 90% of the block transfers, client to client transfers are pretty much just icing on the cake at that point, and don't really matter either way.
This is a perfect little 3rd party community project that requires minimal serverside changes, uses mostly unmodified existing
applications (Squid, ctorrent/libtorrent/torrentflux, bttrack.py/bnbteasytracker/phpbttrkplus/bytemonsoon), and is mostly
glue code sticking the ragtag little group together; in fact, I wouldn't be surprised it if could be done easily in a dynamic language
like perl, python, php, or ruby.
I'm not sure if we've gone over to asset transfer over HTTP yet, but I do know it's in the roadmap already, and the sooner we get a handle on a decent way to pull this off, the more we can gear right into that roadmap.
It also decreases what I would assume to be a fairly massive bandwidth and database load on the LL grid which translates to improved quality of experiance for all; allowing their transactions to deal with assets less and other systems like search, presence, messaging and scripts to be more reliable and concurrent, hopefully allowing far more people to be on the grid at once.
It allows one more thing that could be quite useful; the ability to use SL texture assets outside of SL by requesting them from a cache. This would be handy for sites like SLExchange to display images of products directly from SL, using the images most item creators already use for vendors and primbox storefronts.
Once again, this is a third party program that *supplements* SL -- it doesn't need to be part of the client. Everything necessary is already win32 compatible so we could provide a simple installer package for windows for "private home network" caches, and linux/bsd/solaris packages for larger (public?) servers.
Basic How-To
This section covers the basic how to drop-in and go with the most default squid cache but configured for Second Life on your system.
Please refer to the main download page for squid for the right binary.
Windows:
Comments
This also tends to fix most of the Web Textures problems, as SLSquid could easily retrieve, compress, cache, serve, and share grid-external assets, while keeping user IPs private and mitigate load on external webservers. Technically, it would also be able to serve dynamic content as well with Squid's powerful redirection API. -- Kamilion Schnook 16:40, 08 July 2007 (PDT)
Forums for discussing and developing SLSquid are now located here: SLLabs SLSquid Forums
Other services should be available soon for this project, including a SVN repository.
Any interesting data will also be added to this wiki. -- Kamilion Schnook 16:50, 08 July 2007 (PDT)