Oct 23 04:15 All Services Restored
Oct 23 04:15
Upon Amazon restoring our EBS we found that one of the drives was corrupt. Angelo restored the drive from backup and we where finally able to bring the remaining services back online. Our backup management plan proved effective. Over the next 24 hours we will review all of our server configurations and put in place new infrastructure to reduce the risk of having a pro-longed outage in the future.
Please note as indicated below Our Media Hosting and our Stats Service never went offline. Our fail over plan worked as expected and all media and data served/collected during the primary provider outage.
Oct 23: 00:44am EST
The Amazon Service outage affecting many sites across the country is effecting Blubrry.com and our Publishing platform. We are awaiting Amazon to allow us access to the two remaining drives that have not came online yet. Our entire suite of services is backed up daily so all data is safe regardless
Our Media Hosting and our Stats Service has not been offline, all media continues to be delivered and logs captured for statistics . What is currently affected is our publishing interface and of course the main website. We expect resolution of this in shortly and apologize for the outage. If your show publishing schedule has been affected by this outage be sure to submit a support ticket once we come back online.
Our media statistics service was designed to stay online with multiple data center failures and today that design proved itself. “No data has been lost” and all media continues to be served.
We need to improve the rest of our operations resilience to these types of outages. I will be working with the team as soon as we recover from the Amazon outage and talk about fail over contingencies for the rest of our services. Down time in any instance is not acceptable and we will put in provisions to reduce the chance of further outages.