Amazon S3 Support: Our Approach to External Storage Services

Posted by: azrul on Jan 12, 2010  

Tagged in: storage , development

In case you haven't heard about it, let me announce it properly. JomSocial 1.6 will have a powerful, seamless support for Amazon S3 storage. For now, the support includes both videos and photos, where you can serve the files directly from Amazon's servers, bypassing your servers, thus reducing site load and bandwidth consumption.

When we started thinking about support for remote storage, there are a couple of ideas that we considered. For the technically inclined, providing a virtualized file system, where local file and remote file looks exactly the same, seems like a good idea. However, to ignore latency issues would be a huge mistake. Access to local files are FAST and accessing remote file is orders or magnitude slower. That is a fact.

So, we decided that we needed to provide the external storage support that is easy to use, work seamlessly, while maintaining fast operation for all file-system related operation. Our design also uses pluggable storage system, meaning that, it is rather easy to add new storage type (your own remote server, ftp server support are planned!)

How does it work?

Once you set photos to use a different storage type, cron process will work in the background to transfer all your photos to the other storage. This means that, when you actually change the storage type at the backend, there is no need to transfer the whole content (which could be in tens of GB for large site).

Cron processes will automatically do this transfer for you, in an orderly, gradual fashion. This allow your site to run smoothly while data are being transferred to the remote storage. Unused files in your local server will then be deleted automatically.

This also work flawlessly in reverse. Say you have tons of data in remote storage and now wanted to move it all to local server, or any other storage type, you'll only need to flip a switch in the site backend and transfer will start in the background.

Can you simplify this further please...

Really, what it all means is that our remote storage support works fast, seamless and headache free! Oh yes, any 3rd party plugin will also be able to use this services!

16 Response(s)

You must be logged in to post a comment. Please register if you do not have an account yet.
yoankatz
yoankatz
May 30, 2011
Report Reply
http://www.bloomdigital.co.il
vote down 0 vote up
thecooper
thecooper
June 02, 2010
Report Reply
I try to use Amazon S3 with JomSocial.
If I switch to the remote Amazon S3--- does this mean, that all my pictures and videos will be deleted from my local host?

It would be great to read an better explanation. Thanks for all your good ideas!!
vote down 2 vote up
jvfrance10
May 12, 2010
Report Reply
I presently store my web site videos on Amazon S3 but serve the videos to each site user through RTMP Amazon Cloudfront. The videos that arrive to the user are presented faster than my local web site and the Cloudfront can handle unlimited number of users.
I have presented a request to Hardwood Design - HWDMEDIASHARE to provide software that integrates with Jomsocial, uploads and converts videos and photos using an Amazon EC2 instance and stores the files on Amazon S3 and serves the videos and photos to the user through Amazon Cloudfront. I know Jomsocial and hwdmediashare teams are working together, why not collaborate on this ?
Amazon has a powerful solution and is inexpensive.
vote down 0 vote up
Alessandro_Roma
March 16, 2010
Report Reply
Hello... what about to create an extention that permit us to put our media files to a personal(other) server, not just amazon s3 server....
vote down 2 vote up
tamber
January 22, 2010
Report Reply
What about us that have thousands of media files on our sides already? Will this affect the existing media? How can we easily export our existing media to S3 storage?
vote down 1 vote up
cloudberryman
January 21, 2010
Report Reply
I always enjoy learning what other people think about Amazon Web Services and how they use them. Check out my very own tool CloudBerry Explorer that helps to
manage S3 on Windows . It is a freeware. http://s3.cloudberrylab.com/
vote down -3 vote up
Ric Raftis
Ric Raftis
January 20, 2010
Report Reply
Are there more detailed instructions on setting up this facility? User guide is light on. For example, do you have to set up a bucket or just name it in the backend with your security keys? Are adjustments required to the cron job?
vote down -1 vote up
Wilco Wietsma
January 20, 2010
Report Reply
That's fucking awesome! Can't wait!
vote down -2 vote up
maxx
January 15, 2010
Report Reply
that's cool
vote down 0 vote up
Enrique Vargas
Enrique Vargas
January 14, 2010
Report Reply
Great news!
vote down 3 vote up
Matamari
January 12, 2010
Report Reply
All you need in a neat and tidy package :-)
vote down 4 vote up
Tiago Neves
Tiago Neves
January 12, 2010
Report Reply
This is a great performance boost option. Congratulations on this development.
You mention "it is rather easy to add new storage type" - is there any tips on how to accomplish other storage integration? I would like to develop one for Rackspace Cloud Files also.
Thanks.
vote down 1 vote up
fuqaha
fuqaha
January 12, 2010
Report Reply
If you are experienced enough, this should help (JS 1.6.x): componentscom_communitylibrariesstorage.
All the best. ;)
vote down 0 vote up
Tiago Neves
Tiago Neves
January 12, 2010
Report Reply
Thanks. I'll take a look into it in coming days to see what I can come up with...
vote down 1 vote up
jomlover
January 12, 2010
Report Reply
Wow... I'm impressed. I love the automatic gradual transfer. That's the spirit of cloud system. Set and stop worrying.
vote down 3 vote up
busy

[ Back ]