Incident report for January 7th 2017

Magda Neagu - Tuesday, January 10, 2017


Hi everyone,

We have recently discovered a fault with one of the machines on our Sydney Datacenter. The tasks on this particular machine have been blocked since January 7th, leading to a queue of unsent notifications, tasks and errors with API integrations. We are currently in the process of resolving these issues and will reinstate all items by later today. This will lead to all queued tasks being sent, with a delay of up to 72 hours from the initial time they should have been processed. A short summary of the impact is available below:

  • Affected sites: a percentage of the sites hosted on our Sydney datacenter
  • Affected tasks (highest impact. Smaller impact and security related tasks not listed): API integrations, Customer Support Ticketing, Recurrent Order Invoicing, Comment Spam Processing
  • Impact of the incident: delay of up to 72h in sending of several tasks and notifications
  • Loss of data: none. All items will be restored, with the maximum delay mentioned.

This incident has not impacted in any way sites hosted in the European or United States Datacenters.

We apologize for the issues encountered. As an additional measure to prevent this from happening in the future, our operations team is introducing additional monitoring on all machines to raise awareness faster when such incidents occur. Should you experience any further issues please do not hesitate to reach out to us.


Update: as of 12:00 PM CET today, January 11th, we can confirm that all back items have been processed without any loss of data. This includes recurring orders as well as all API notifications that were stuck in the queue. Once again, please accept our apologies for the problems caused by this glitch. 

The Business Catalyst team.