Amazon S3 Support: Our Approach to External Storage Services

In case you haven't heard about it, let me announce it properly. JomSocial 1.6 will have a powerful, seamless support for Amazon S3 storage. For now, the support includes both videos and photos, where you can serve the files directly from Amazon's servers, bypassing your servers, thus reducing site load and bandwidth consumption.

When we started thinking about support for remote storage, there are a couple of ideas that we considered. For the technically inclined, providing a virtualized file system, where local file and remote file looks exactly the same, seems like a good idea. However, to ignore latency issues would be a huge mistake. Access to local files are FAST and accessing remote file is orders or magnitude slower. That is a fact.

So, we decided that we needed to provide the external storage support that is easy to use, work seamlessly, while maintaining fast operation for all file-system related operation. Our design also uses pluggable storage system, meaning that, it is rather easy to add new storage type (your own remote server, ftp server support are planned!)

How does it work?

Once you set photos to use a different storage type, cron process will work in the background to transfer all your photos to the other storage. This means that, when you actually change the storage type at the backend, there is no need to transfer the whole content (which could be in tens of GB for large site).

Cron processes will automatically do this transfer for you, in an orderly, gradual fashion. This allow your site to run smoothly while data are being transferred to the remote storage. Unused files in your local server will then be deleted automatically.

This also work flawlessly in reverse. Say you have tons of data in remote storage and now wanted to move it all to local server, or any other storage type, you'll only need to flip a switch in the site backend and transfer will start in the background.

Can you simplify this further please...

Really, what it all means is that our remote storage support works fast, seamless and headache free! Oh yes, any 3rd party plugin will also be able to use this services!