Update on Rojo outage

We experienced database corruption as a result of the catastrophic power failure at our datacenter.  The affected tables are being repaired, but due to the massive amount of content that Rojo indexes, it is a protracted process that may take a couple more days.

This datacenter situation was a perfect storm of failures.  We do keep both master and slave database servers to guard against reasonably foreseeable events, but in this case both our master and replicated databases were corrupted.  Unfortunately, we do not have a usable third backup of the particular table that was affected.  This is a situation that "should" never happen on many levels, sparked not only by a major power outage but the subsequent spontaneous failure of multiple backup power generators at our collocation facility -- despite, we are assured, regular testing.

Rojo will indeed be restored.  We are investigating some ways we might be able to make some functions available earlier, but full functionality will have to wait until the repair is complete.

We apologize for the service interruption.  We will be back online as soon as we possibly can, and appreciate your patience in the mean time.

Comments

[Forex Trading]

When your site will be online? How much are the damages? Big or not so big?

[Paul Johnson]

It has been three days now and no updates what is going on? In situations like these communication is key and silence is deadly.

[MAt]

I think its time to go to other place now. This site has become unreliable.

[Noah Tutak]

I agree with the comments above - every day that goes by without an update I move more feeds to my other readers (google and bloglines).

[el camino]

good bye ROJO!
www.abriendo.wordpress.com

[ツ NiceFace]

terrible, justo que era la primera vez que lo iba a utilizar se cae el sistema, que tristeza :(

[Stephen R]

This is amazing. Please let all of us know what database and platform you were running on. I've never seen a database melt down like this. Most databases can survive a sudden shutdown, like power failure. To me, this sounds like there was a real lack of disaster recovery planning, as well as a good amount of human error after the fact.

[Eric]

I'd flipped to Google Reader about a month ago when Rojo was acting up and not updating one of my feeds; I'd flip back to Rojo to get the archive of things I'd saved previously. Sooooo glad I made my migration a month ago rather than waiting!

[micheal kramar]

And I now pronounce Rojo dead...

I liked Rojo, but it never gained wider popularity (one has to analyze the underlying dynamics carefully to understand why), and now this has happened... The odds are highly unlikely that it will survive.

I just hope that I can get my list of feeds some how... I was subcribed to tens of feeds, of which I can never remember them all... Perhaps this is a good opportunity for me to get rid of the ones that are not memorable / attention worthy...

Anyways, RIP rojo. I feel sorry for you guys who work there, and the decision makers @ six apart who acquired it :(

[Mat]

I was hoping that they were recovered by now. Just hoping I can use this service again, because its the best rss reader interface out there (although I modified it a little with my own css), I like it even more than the software ones. If the can't keep it alive I would like they sold as a software so I can install it on my server for my personal use.
Its a real shame that a blackout can destroy a web service, there are disaster recovery planning and backups that are the ABC of any company that relies o computer for they're business.
Im still hoping they put this service online again so I can give them just one more chance to trust them, but a week offline its a dead business for me.
Please, rojo, give us some feedback, its really bad PR as it is for you to not talking to your customers.

[Miroslav]

Could you please let us know when will be ROJO back again ?

[Benedict Herold]

Has rojo converted itself from RSS reader to link building blog? I'm not happy with PR either.!

[Peter]

rojo is dead. unbelievable that the site is still not up again and you don't even keep your users informed at all. byebye rojo

[bob pasin]

ummm... I can understand that rojo is dead.

But is also six apart dead?

This is a PR catastrophe for six apart, as much it is for rojo...

I mean COME ON! If you don't have anything to say, just say... "we don't know what to say, but we're doing our best". Even that would be better than saying nothing. PLEASE, DO / SAY SOMETHING!

I'm calling VC's who funded six apart to duty. How can you be so irresponsible with your investment?

[Chris]

Bob, and all,
Our apologies for not updating sooner. We haven't updated because unfortunately there hasn't been a lot to update on at this point that's different from our post last week. We are undergoing a massive repair effort which exceeded our initial expectations for how long it would take. So yes, we are indeed working on it and are confident that the service will come back online.
Chris