Cristinel Anastasoaie

Incident report - recent downtime for AU data center

As you are probably aware, sites on all data centers have experienced some downtime these two weeks. First, we apologize for any inconvenience this might have caused you, and offer you a detailed explanation on what happened and what measures we're taking to prevent this in the future.

Starting on September 26th, sites on all data centers have begun experiencing intermittent downtimes. Sites on our Asia Pacific data center have experienced longer and more frequent downtime sessions than what we have announced in the AWS maintenance blog post.

The downtime has been caused by three distinct events and was amplified by timing:

  • Amazon AWS infrastructure upgrade - this operation implied many server restarts and failing over from one Amazon availability zone onto another and then back (basically, we had to execute a scheduled disaster recovery procedure). Our team has worked 24/7 to make this major AWS-wide infrastructure upgrade as smooth as possible to all our customers. During these procedures, the sites on the data center under maintenance became totally unavailable while sites on the other two data centers kept their front-ends running but had most of the back-end services disabled because we needed to stop the data replication between data centers. While Amazon has performed the restarts outside business hours for each region, the restarts of NA and Europe data centers fell during AU business hours and thus had some impact on all sites by preventing customers to access some of the back-end services. We are looking into implementing some architectural changes that will limit the impact of such operations from one data center to the other.
  • Load balancer crash - this week we have encountered a load balancer crash. We have worked with the vendor to identify the root cause and we decided to upgrade the system’s firmware; this procedure is almost completed now and we are closely monitoring the load balancer for any unforeseen issues that might arise.
  • A network connectivity issue between Amazon datacenters triggered an automatic fail over of the database servers to the backup servers. This type of operation usually generates a downtime of up to several minutes. We are currently trying to identify a potential network architecture change that could help mitigate this type of occurrence.

Once again, our apologies for any inconvenience this incident might have caused. Both our team and Amazon are fully committed to provide the upmost level of security and reliability to all our customers and we continuously dedicate efforts to improve on these fronts.

Sincerely,

The Adobe Business Catalyst Team

View Comments

To ensure the highest levels of performance and reliability, we've scheduled a database server upgrade on our Australia and North America data centers. To minimize the customer impact, the upgrade is scheduled at the most convenient hours for the regions and will take up to 6 hours to complete. During the maintenance procedure, creating and updating content, Partner registration, trial site creation, publish from Muse, sFTP, APIs and some site admin sections will not be available for 4 hours on ALL data centers. Additionally, all sites on the AU and NA data center will experience up to 9 minutes downtime sometimes during the maintenance window. Except for the scheduled 9 minutes downtime, the website front-ends will not be impacted by the maintenance.

Maintenance schedule:

  • Start date and time: Sunday, September 14th, 12:00 PM UTC (check data center times)
  • Duration: We are targeting a 6 hours maintenance window

Customer impact:

  • Partner registration, Trial site creation Muse Publish, APIs, FTP and some admin section will not be available for 4 hours on all data centers
  • All websites and services on AU and NA data centers will experience up to 9 minutes downtime sometimes within the maintenance window
  • Creating or updating content on sites located on AU and NA data center will be unavailable during the maintenance procedure

For up to date information about system status, check the Business Catalyst System Status page. We apologize for any inconvenience caused by these service interruptions. Please make sure that your customers and team members are made aware of these important updates.

Thank you for your understanding and support,

The Adobe Business Catalyst Team

View Comments
Cristinel Anastasoaie

Scheduled system maintenance on AU datacenter - August 3rd 2014

To ensure the highest levels of performance we will be updating file servers located in our Australia data center. To minimize the customer impact, the update is scheduled at the most convenient hours for the region and will take up to one hour to complete. During the maintenance procedure, publish from Muse, sFTP,  and File Uploads will experience a 30 minutes downtime. Additionally, the site indexing services will not run for approximately 10 hours starting 8 hours before the update window. The website front-ends will not be impacted by the maintenance.

Maintenance schedule:

  • Start date and time: Sunday, August 3rd, 00:00 AM AEST (check datacenter times)
  • Duration: We are targeting a 30 minutes maintenance window

Customer impact:

  • Muse Publish, sFTP, File Uploads, Site replication and some admin section will experience service interruptions
  • Updated site indexes will be available at 2:30 AM AEST

For up to date information about system status, check the Business Catalyst System Status page. We apologize for any inconvenience caused by these service interruptions. Please make sure that your customers and team members are made aware of these important updates.

Thank you for your understanding and support,

The Adobe Business Catalyst Team

View Comments
Cristinel Anastasoaie

Scheduled system maintenance on EU datacenter - June 16th 2014

To ensure the highest levels of performance and reliability, we've scheduled a database server upgrade on our EU AWS data center. To minimize the customer impact, the upgrade is scheduled at the most convenient hours for the region and will take up to 4 hours to complete. During the maintenance procedure, creating and updating content, Partner registration, trial site creation, publish from Muse, sFTP, APIs and some site admin sections will not be available. Additionally, all sites on the EU data center will experience a 10 minutes downtime sometimes during the maintenance window. Except for the scheduled 10 minutes downtime, the website front-ends will not be impacted by the maintenance.

Maintenance schedule:

  • Start date and time: Monday, June 16th, 3:00 AM UTC (check data center times)
  • Duration: We are targeting a 4 hours maintenance window

Customer impact:

  • Partner registration, Trial site creation Muse Publish, APIs, FTP and some admin section will not be available through the entire maintenance window
  • All websites and services on EU data center will experience a 10 minutes downtime sometimes within the maintenance window
  • Creating or updating content on the impacted sites will be unavailable during the maintenance procedure

For up to date information about system status, check the Business Catalyst System Status page. We apologize for any inconvenience caused by these service interruptions. Please make sure that your customers and team members are made aware of these important updates.

Thank you for your understanding and support,

The Adobe Business Catalyst Team

View Comments
Cristinel Anastasoaie

Business Catalyst Service Maintenance, June 2012 - Updated

Last update - June 27, 01:30 PDT - To ensure the highest security and performance levels for our services, we're applying a software update on all our database servers. To minimize the customer impact, the updates are scheduled at the most convenient hours for each of the data centers and will take up to eight hours to complete.

During every of the three maintenance procedures, the services requiring users to login will experience 2 downtime windows of up to 30 minutes each (first one at the beginning and the second at the end), impacting sites on ALL datacenters

Additionally, throughout the maintenance procedure, website front-ends for sites hosted on the datacenter under maintenance will experience intermittent service interruptions due to database failover procedures. Please find below the maintenance schedule and the list of affected services:

Start of maintenance Duration Datacenter Customer impact Systems affected
Thursday, June 14th, 2012, 1:00 AM PDT (check local time) 7 hours North America
  • 2 x 25 minutes downtime for all services requiring users to login; this will impact sites on all datacenters
  • several intermittent website front-end interruptions for sites on North America datacenter
  • Admin Console, FTP, APIs, Partner Portal
  • Websites front-end
Tuesday, June 19th, 2012, 11:00 AM PDT (check local time) 4 hours Europe
  • 2 x 25 minutes downtime for all services requiring users to loginthis will impact sites on all datacenters
  • several intermittent website front-end interruptions for sites on Europe datacenter
  • Admin Console, FTP, APIs, Partner Portal
  • Websites front-end
Saturday, June 30th, 2012, 3:30 AM PDT (check local time) 8 hours Asia Pacific
  • 2 x 30 minutes downtime for all services requiring users to loginthis will impact sites on all datacenters
  • several intermittent website front-end interruptions for sites on Asia Pacific datacenter
  • Admin Console, FTP, APIs, Partner Portal
  • Websites front-end

For up to date information about system status, check the Business Catalyst System Status page.

We apologize for any inconveniences generated by these service interruptions. Please make sure that your customers and team members are aware of these important updates.

Thank you for your understanding and support,

The Adobe Business Catalyst Team

View Comments
Cristinel Anastasoaie

Business Catalyst Service Maintenance - November 12

To ensure the highest reliability and performance levels for our services, we've scheduled a database server upgrade on our Asia Pacific datacenter. The upgrade is scheduled for Saturday, November 12 at 1:00 AM AEDT time (check local time) and will take one hour to complete.

During the upgrade, customers of our Asia Pacific datacenter (including the Business Catalyst website) will experience 3 windows of 5 minutes each of service interruption. 

Please find below the maintenance schedule and the list of affected services:

  • Start of maintenance: Saturday, November 12, 1:00 AM AEDT time (check local time)
  • End of maintenance: Saturday, November 12, 2:00 AM AEDT time (check local time)
  • Duration: 2 hours
  • Systems affected: Site front-ends, Admin console, Partner Portal, FTP services, API services
  • Customer impact: 3 windows of 5 minutes each of service interruptions

We sincerely apologize for any inconveniences generated by these service interruptions.

The Business Catalyst Team

View Comments
Cristinel Anastasoaie

Scheduled System Update - October 31st, November 1st and 2nd

We are planning to update our database and server infrastructure between 31 October and 2nd of November. For each datacenter, the update will take up to 6 hours and will cause two downtime sessions of up to 15 minutes each, one at the start of the update and another one at the end. During the downtime, the following Business Catalyst services will be unavailable:

  • Admin Console Access
  • Partner Portal
  • FTP
  • Dreamweaver extension
  • Muse
  • Business Catalyst APIs
  • Partner registration
  • Trial site creation

Additionally, the during each of the planned downtimes, the Business Catalyst front-end service will experience up to 1 minute of service interruption that will display a "Site under maintenance" page for site visitors.

Please find below the schedule and expected downtime hours for each of the data centers.

Monday, October 31st, Asia Pacific datacenter update:

  • Duration: 6 hours and 15 minutes
  • Start time: Mon, 21 Oct, 21:00 Sydney time (check local time)
  • Downtime (affecting all sites): up to 15 min, starting 21:00 and ending 21:15 (check local time);
  • End of maintenance: Tue, 1 Nov, 3:15 AM (check local time)
  • Downtime (affecting all sites): up to 15 min starting 3:00 AM and ending 3:15 AM (check local time)

Tuesday, November 1st, North America datacenter update

  • Duration: 6 hours and 15 minutes
  • Start time: 1:00 AM PDT (check local time)
  • Downtime (affecting all sites): up to 15 min, starting 1:00 AM and ending 10:15 AM (check local time)
  • End of maintenance: 7:15 AM PDT (check local time)
  • Downtime (affecting all sites): up to 15 min starting with 7:00 AM and ending 7:15 AM (check local time)

Wednesday, November 2nd, Europe datacenter update

If you have any questions, please contact Business Catalyst support team.

View Comments
Cristinel Anastasoaie

Ottawa data center migration - Live status

Last update: 06:20 PM EDT 29 May

Migration from Legacy Ottawa Datacenter to New Jersey data center is COMPLETED.

Pre-deployment

Task Status
Set-up high speed direct datalink between Legacy Ottawa DC and new New Jersey DC
Done
Set-up reverse proxy server  Done
Execute migration dry-run Successful
Send first (out of 3) warning email communication to Customers Done
Enable Information Site at http://status-ottawa.businesscatalyst.com
Done
Send second (out of 3) warning email communication to Customers Done
Send third and final email communication to Customers Done
Disable system alerts in Ottawa Datacenter
Done

Deployment

Task Status
Start maintenance Done
Stop email campaigns, workflow notifications on Ottawa Done
Enable maintenance page for sites on Ottawa Done
Shut down FTP & web servers on Ottawa Done
Shut down FTP, disable and queue email campaigns and workflow notifications in New Jersey  Done
Replicate Ottawa content (database & assets) in New Jersey Done
Start final content synchronization (databases and assets) between Ottawa and New Jersey Done
Reconfigure migrated sites in New Jersey Done
Restart web servers on New Jersey Done
Validate data on New Jersey Done
Reconfigure DNS settings Done
Start-up reverse proxy server on New Jersey Done
Shut down maintenance pages Done
Enable system alerts on New Jersey Done
Close maintenance with final validation checks
Done


Thank you for your patience and cooperation,
Cristinel Anastasoaie
Adobe Business Catalyst Product Manager

View Comments

The final details for our North America data center (located in Ottawa) have been set. The move has been scheduled to occur on Sunday, 29 May 2011, starting with 12:00 AM EDT (check local times here).

During the migration, we will be moving all Business Catalyst sites and application infrastructure hosted on our legacy North America datacenter (located in Ottawa) to our new United States datacenter (located in New Jersey). The migration will take approximately 8 hours; during this time, any sites that are hosted on Ottawa datacenter will not be accessible and instead be replaced by a system maintenance message.

Sites hosted on other datacenters including Sydney, Dublin, and the new US datacenter will be unaffected. Partner Portal access and site creation will also be unaffected.

  • What's Happening?: We are migrating all sites and Business Catalyst application infrastructure from Ottawa to New Jersey in one bulk-move
  • Start Time: 12:00AM EDT Sunday 29 May 2011  (check local times here)
  • End Time: 8:00AM EDT Sunday 29 May 2011 (check local times here)
  • How Long Will It Take? We will have a scheduled maintenance window of 8 hours, during which all sites hosted on Ottawa will be unavailable (both frontend and Admin console access). Once the maintenance is complete all sites will automatically be up again with both frontend and Admin Console access.
  • What are we doing? We are going to replicate all databases between Ottawa and New Jersey. We will also setup a high-speed direct datalink between the 2 locations, to ensure databases are kept in sync prior and during the migration. At the scheduled time of the migration we will reconfigure DNS settings and make other related Business Catalyst architectural changes to point to the New Jersey datacenter. We will also need to restart all web servers.
  • Customer Action Required - sites hosted on legacy Ottawa datacenter with redelegated DNS: After the migration is complete you will need to update your DNS nameservers from ns[01,02,03].businesscatalyst.com to ns[1,2,3].worldsecuresystems.com.
  • Customer Action Required - sites hosted on legacy Ottawa datacenter with 3rd party hosted DNS: After the migration is complete you will need to change your DNS settings with your DNS host i.e GoDaddy etc, to point to the IP address of the New Jersey data center (192.150.2.140). This will re-enable FTP access to your site.
  • Customer Action Required - sites hosted on legacy Ottawa datacenter need to update payment gateway settings: Following the migration, customers using Optimal Payments, EBS, SagePay or Internet Secure payment gateways will need to update the Business Catalyst IP address in their payment gateway account. For each site affected, you will need to login to the payment gateway provider's Administration Console and update the provided IP address from 69.20.239.58 to 192.150.2.4

Communication Plan for Affected Sites

Like for the previous migrations, we've set-up a dedicated site to host information about migration scehdule and required actions: http://status-ottawa.businesscatalyst.com/

In addition to that, we're also going to send a series of email notifications to affected Partners and affected site owners who are billed directly by Business Catalyst. White-labeled Partners will need to notify their own customers about the scheduled maintenance. We have prepared sample emails which you can download here. Here is a list of emails you should expect until the migration:

  1. Monday 9 May - 1st Notification is sent to affected Partners and Site Owners with details of the scheduled migration and action required afterwards
  2. Thursday 12 May - Email containing a list of affected sites with their domains and whether the domain is redelegated to BC or hosted with a 3rd party registrar is sent to affected Partners
  3. Tuesday 17 May - Email containing a list of websites for which partners or customers need to update Business Catalyst IP in their payment gateway account; email will be sent to affected partners
  4. Moday 23 May - 2nd Notification is sent to affected Partners and site owners with details of the scheduled migration and action required afterwards
  5. Friday 27 May - 3rd and Final notification is sent to affected Partners and site owners.
  6. Monday 30 May - Partners and site owners will receive a notification when the maintenance is completed

Feel free to ask any questions through the comments thread below, and we'll answer as soon as possible.

Thank you,
Cristinel Anastasoaie
Adobe Business Catalyst Product Manager

View Comments
Cristinel Anastasoaie

Update: January 2011 Stability and Email Issues

Earlier this year we experienced a series of incidents that caused significant problems for a large number of customers. We had issues with our hosted email services followed by stability problems on our legacy Sydney data center.

In light of this, we are issuing a full month credit for all paid sites that were hosted on the legacy Sydney data center, and those that were using internally hosted email services.

To be more specific, the following sites will receive a full month credit:

  • All paid sites hosted on the AU data center that have been upgraded on or before January 31st
  • All paid sites using the internal Business Catalyst hosted mail service across all data centers, which have been upgraded on or before January 31st 2011

All customers that were affected by these incidents will receive an email announcing the full month of credit.

In addition, we have made some updates to the Partner Portal and site Admin Console, to help partners identify sites that have received the credit and also to help customers understand the period for which the credit has been applied. Thus, the site list in Partner Portal has been updated with a new “Credit” column:

The Partner Portal > Clients > Site details user interface has also been updated to include a notice about the credit:

We’ve updated the Site admin user interface to highlight the period for which the credit has been received:

To issue the full month of credit, we will skip a month of invoicing for websites with monthly billing, and simply postpone the invoice for one month for sites on annual billing. If your site is billed monthly, no invoice will be generated between 15 Apr and 15 May 2011. If your site is billed yearly, the next due invoice will be postponed by one month.

The updates are going to be rolled out in our next week release.

Since the series of incidents, we have managed to solve all email issues by shifting to an externally hosted email provider and migrating all sites from the legacy Sydney data center to a new location.

To prevent similar incidents on our other two legacy datacenters, we have accelerated the migration schedule for both legacy Europe (London) and North America (Ottawa) data centers and we plan to complete this in the first half of 2011.

Once again, we apologize for the inconvenience caused by these outages and thank you for your support. If you have any other questions, please submit a Support Request via your Partner Portal or reach us on Live Chat.

The Business Catalyst Team

View Comments