JomSocial Servers Down without Dyno

Early in the morning yesterday (Malaysian time),  Syam(our server admin) was awaken by a downtime notification on JomSocial servers. He tried to connect and reboot the server manually but there was no response from the server, which is a rare case. We contacted RimuHosting and they informed us that they were having a power disruption on their servers and were working on it.

We received several complaints from our users especially from those who use version older than 2.2. In the older version of JomSocial, the update checker is set to check available update on JomSocial.com server. Since the server is down, it could not connect to our servers resulting to the slowdown. This issue is fixed in later versions. Besides, there was supposed to be a DNS fail-over that would redirect all requests to a backup server. Unfortunately, it didn't happen. 

It turns out, the power disruption was not only affecting RimuHosting servers, but also affected the whole Colo4 datacenter. Colo4 is used by many online service providers, including Pingability.com which we used to monitor servers uptime and to provide fail-over in case of a failure. That explains why our trusted 500 Dyno couldn't make it to the scene during the downtime.

The access to our server was restored around 6PM(CST). Everything is back online. For those who affected by this downtime, we extend our deepest apology for the  inconvenience caused.

From this incident, we've prepared a couple of improvements to your open source community software and downtime recovery plan. Our DNS fail-over is now hosted in different geographical locations to make sure our 500 Dyno will take over the scene when everything else is down.