[Updated - May 14th 11:30 PST]: We've just sent a follow-up email to explain the incident. Since we've used the BC Email Marketing engine, the broadcast has been limited to BC partners and business owners that had the email address registered with BC and opt-in to receive emails.

As you are probably aware, customers have received a warning email with the following subject: "Warning: You Are Close To Reaching Your Email Newsletter Limit". The email was sent from a Business Catalyst email address or the partner email address (for white-labeled Partner Portals). This email has been accidentally sent and does not reflect the actual email credits a site has.

First, we apologize for any inconvenience this might have caused you, and offer you a detailed explanation on what happened and what measures we're taking to prevent this in the future.

The accidental emails were caused by an incorrect parameter update following a minor system update on Monday, May 11th:

  • Due to an incorrect parameter update and use, the internal engines checking the email quota on Business Catalyst paid sites (Partner sites are also considered paid sites) have considered all sites reached their email quota and automatically triggered the email credit limit warning email
  • The error impacted a large number of Business Catalyst paid and partner sites
  • The email has been sent to Business Catalyst partners and to all admin users with Administer System privileges
  • The email has been sent from a Business Catalyst email address or from the Partner email address (if the site was under a white-labeled Partner Portal)
  • The email has been also sent to Creative Cloud members which have published a site on Business Catalyst

To contain the problem and to prevent such incidents in the future we have:

  • Temporarily disabled the automated warning email engine
  • Investigate the code updates to find the root cause of the incident
  • Fixed the incorrect parameter issue
  • Updated the warning email engine to add an additional check at this level

A service update will be done shortly and restart the automated warning email engine.

Once again, our apologies for any inconvenience this incident might have caused. Our team is fully committed to provide the utmost level of support to all our customers and we continuously dedicate efforts to improve our services.

Sincerely,

The Adobe Business Catalyst Team

View Comments

To ensure the highest levels of performance and reliability, we've scheduled a database server upgrade on our Australia and North America data centers. To minimize the customer impact, the upgrade is scheduled at the most convenient hours for the regions and will take up to 6 hours to complete. During the maintenance procedure, creating and updating content, Partner registration, trial site creation, publish from Muse, sFTP, APIs and some site admin sections will not be available for 4 hours on ALL data centers. Additionally, all sites on the AU and NA data center will experience up to 9 minutes downtime sometimes during the maintenance window. Except for the scheduled 9 minutes downtime, the website front-ends will not be impacted by the maintenance.

Maintenance schedule:

  • Start date and time: Sunday, September 14th, 12:00 PM UTC (check data center times)
  • Duration: We are targeting a 6 hours maintenance window

Customer impact:

  • Partner registration, Trial site creation Muse Publish, APIs, FTP and some admin section will not be available for 4 hours on all data centers
  • All websites and services on AU and NA data centers will experience up to 9 minutes downtime sometimes within the maintenance window
  • Creating or updating content on sites located on AU and NA data center will be unavailable during the maintenance procedure

For up to date information about system status, check the Business Catalyst System Status page. We apologize for any inconvenience caused by these service interruptions. Please make sure that your customers and team members are made aware of these important updates.

Thank you for your understanding and support,

The Adobe Business Catalyst Team

View Comments
Cristinel Anastasoaie

Scheduled system maintenance on AU datacenter - August 3rd 2014

To ensure the highest levels of performance we will be updating file servers located in our Australia data center. To minimize the customer impact, the update is scheduled at the most convenient hours for the region and will take up to one hour to complete. During the maintenance procedure, publish from Muse, sFTP,  and File Uploads will experience a 30 minutes downtime. Additionally, the site indexing services will not run for approximately 10 hours starting 8 hours before the update window. The website front-ends will not be impacted by the maintenance.

Maintenance schedule:

  • Start date and time: Sunday, August 3rd, 00:00 AM AEST (check datacenter times)
  • Duration: We are targeting a 30 minutes maintenance window

Customer impact:

  • Muse Publish, sFTP, File Uploads, Site replication and some admin section will experience service interruptions
  • Updated site indexes will be available at 2:30 AM AEST

For up to date information about system status, check the Business Catalyst System Status page. We apologize for any inconvenience caused by these service interruptions. Please make sure that your customers and team members are made aware of these important updates.

Thank you for your understanding and support,

The Adobe Business Catalyst Team

View Comments
Cristinel Anastasoaie

Business Catalyst Service Maintenance, June 2012 - Updated

Last update - June 27, 01:30 PDT - To ensure the highest security and performance levels for our services, we're applying a software update on all our database servers. To minimize the customer impact, the updates are scheduled at the most convenient hours for each of the data centers and will take up to eight hours to complete.

During every of the three maintenance procedures, the services requiring users to login will experience 2 downtime windows of up to 30 minutes each (first one at the beginning and the second at the end), impacting sites on ALL datacenters

Additionally, throughout the maintenance procedure, website front-ends for sites hosted on the datacenter under maintenance will experience intermittent service interruptions due to database failover procedures. Please find below the maintenance schedule and the list of affected services:

Start of maintenance Duration Datacenter Customer impact Systems affected
Thursday, June 14th, 2012, 1:00 AM PDT (check local time) 7 hours North America
  • 2 x 25 minutes downtime for all services requiring users to login; this will impact sites on all datacenters
  • several intermittent website front-end interruptions for sites on North America datacenter
  • Admin Console, FTP, APIs, Partner Portal
  • Websites front-end
Tuesday, June 19th, 2012, 11:00 AM PDT (check local time) 4 hours Europe
  • 2 x 25 minutes downtime for all services requiring users to loginthis will impact sites on all datacenters
  • several intermittent website front-end interruptions for sites on Europe datacenter
  • Admin Console, FTP, APIs, Partner Portal
  • Websites front-end
Saturday, June 30th, 2012, 3:30 AM PDT (check local time) 8 hours Asia Pacific
  • 2 x 30 minutes downtime for all services requiring users to loginthis will impact sites on all datacenters
  • several intermittent website front-end interruptions for sites on Asia Pacific datacenter
  • Admin Console, FTP, APIs, Partner Portal
  • Websites front-end

For up to date information about system status, check the Business Catalyst System Status page.

We apologize for any inconveniences generated by these service interruptions. Please make sure that your customers and team members are aware of these important updates.

Thank you for your understanding and support,

The Adobe Business Catalyst Team

View Comments
Cristinel Anastasoaie

FedEx Scheduled Maintenance - May 26 and June 9

Please be advised that the FedEx production environments will undergo a planned system update on May 26 and on June 9, which will result into up to 6 hours of downtime for each day. During the downtime, the FedEx Rating, Shipping, Tracking and Pickup  services will be unable to process transactions for all customers.

Start Date and Time End Date and Time Duration Functionality Affected
Saturday, 
May 26, 2012
9 p.m. CT
Sunday,
May 27, 2012
1 a.m. CT
4 hours Rating
Shipping
Tracking
Pickup
Saturday, 
June 9, 2012
9 p.m. CT
Saturday, 
June 9, 2012
11 p.m. CT
2 hours Rating
Shipping
Saturday, 
June 9, 2012
9 p.m. CT
Sunday, 
June 10, 2012
3 a.m. CT
6 hours Tracking
Pickup

Please notify your customers about the scheduled maintenance.


View Comments
Dragos Manescu

Major DNS Upgrade and Service Maintenance on February 8

The Business Catalyst  team has just finished a major upgrade of the DNS architecture that will increase the performance and the availability of the system.
This is an infrastructure upgrade that will improve both system scalability and resilience, while keeping the existing DNS management user experience unchanged.

The upgrade is scheduled for Wednesday, February the 8th,  from 05:00 to 11:00 AM EST (check local time) and during this time frame functionalities related to domain management,  website creation and new partner registration  will be suspended.

Please find below a schedule of the maintenance:
  • Time frame for this operation: 05:00 to 11:00 AM EST (check local time); the duration will be of about 6 hours.
  • Systems affected: 
    • DNS management for existing websites
    • Set-up of internal or external MX records or addition of new email addresses
    • Site Cancelation from the Partner Portal
    • Partner Activation:  if a customer has registered as a partner prior to the above time frame but hasn’t visited the Partner Portal to activate her account, she will have to wait until 11:00 EST (check local time) for this activation
    • New website creation
    • New partner registration
We sincerely apologize for any inconveniences generated by these service interruptions.
 
The Business Catalyst Team
View Comments
Cristinel Anastasoaie

Business Catalyst Service Maintenance - November 12

To ensure the highest reliability and performance levels for our services, we've scheduled a database server upgrade on our Asia Pacific datacenter. The upgrade is scheduled for Saturday, November 12 at 1:00 AM AEDT time (check local time) and will take one hour to complete.

During the upgrade, customers of our Asia Pacific datacenter (including the Business Catalyst website) will experience 3 windows of 5 minutes each of service interruption. 

Please find below the maintenance schedule and the list of affected services:

  • Start of maintenance: Saturday, November 12, 1:00 AM AEDT time (check local time)
  • End of maintenance: Saturday, November 12, 2:00 AM AEDT time (check local time)
  • Duration: 2 hours
  • Systems affected: Site front-ends, Admin console, Partner Portal, FTP services, API services
  • Customer impact: 3 windows of 5 minutes each of service interruptions

We sincerely apologize for any inconveniences generated by these service interruptions.

The Business Catalyst Team

View Comments
Cristinel Anastasoaie

Scheduled System Update - October 31st, November 1st and 2nd

We are planning to update our database and server infrastructure between 31 October and 2nd of November. For each datacenter, the update will take up to 6 hours and will cause two downtime sessions of up to 15 minutes each, one at the start of the update and another one at the end. During the downtime, the following Business Catalyst services will be unavailable:

  • Admin Console Access
  • Partner Portal
  • FTP
  • Dreamweaver extension
  • Muse
  • Business Catalyst APIs
  • Partner registration
  • Trial site creation

Additionally, the during each of the planned downtimes, the Business Catalyst front-end service will experience up to 1 minute of service interruption that will display a "Site under maintenance" page for site visitors.

Please find below the schedule and expected downtime hours for each of the data centers.

Monday, October 31st, Asia Pacific datacenter update:

  • Duration: 6 hours and 15 minutes
  • Start time: Mon, 21 Oct, 21:00 Sydney time (check local time)
  • Downtime (affecting all sites): up to 15 min, starting 21:00 and ending 21:15 (check local time);
  • End of maintenance: Tue, 1 Nov, 3:15 AM (check local time)
  • Downtime (affecting all sites): up to 15 min starting 3:00 AM and ending 3:15 AM (check local time)

Tuesday, November 1st, North America datacenter update

  • Duration: 6 hours and 15 minutes
  • Start time: 1:00 AM PDT (check local time)
  • Downtime (affecting all sites): up to 15 min, starting 1:00 AM and ending 10:15 AM (check local time)
  • End of maintenance: 7:15 AM PDT (check local time)
  • Downtime (affecting all sites): up to 15 min starting with 7:00 AM and ending 7:15 AM (check local time)

Wednesday, November 2nd, Europe datacenter update

If you have any questions, please contact Business Catalyst support team.

View Comments
Cristinel Anastasoaie

PayPal Scheduled Maintenance - September 24, 11:30 PM PDT

In order to improve its hardware infrastructure, PayPal is migrating its services to a new data center facility. To complete the move, the PayPal production environments will have a planned maintenance window of 2 hours scheduled to start on September 24 at 11:30 PM PDT (check local time here) and ending at 4:00 AM PDT.

During the downtime, the PayPal services will be unavailable for all customers for intervals between 30 minutes and 4 hours.

These dates and times are subject to change. Please monitor the following blog for the most up-to-date communication:

https://www.paypaltech.com/payflow

Please notify your customers about the scheduled maintenance.

View Comments
Cristinel Anastasoaie

Ottawa data center migration - Live status

Last update: 06:20 PM EDT 29 May

Migration from Legacy Ottawa Datacenter to New Jersey data center is COMPLETED.

Pre-deployment

Task Status
Set-up high speed direct datalink between Legacy Ottawa DC and new New Jersey DC
Done
Set-up reverse proxy server  Done
Execute migration dry-run Successful
Send first (out of 3) warning email communication to Customers Done
Enable Information Site at http://status-ottawa.businesscatalyst.com
Done
Send second (out of 3) warning email communication to Customers Done
Send third and final email communication to Customers Done
Disable system alerts in Ottawa Datacenter
Done

Deployment

Task Status
Start maintenance Done
Stop email campaigns, workflow notifications on Ottawa Done
Enable maintenance page for sites on Ottawa Done
Shut down FTP & web servers on Ottawa Done
Shut down FTP, disable and queue email campaigns and workflow notifications in New Jersey  Done
Replicate Ottawa content (database & assets) in New Jersey Done
Start final content synchronization (databases and assets) between Ottawa and New Jersey Done
Reconfigure migrated sites in New Jersey Done
Restart web servers on New Jersey Done
Validate data on New Jersey Done
Reconfigure DNS settings Done
Start-up reverse proxy server on New Jersey Done
Shut down maintenance pages Done
Enable system alerts on New Jersey Done
Close maintenance with final validation checks
Done


Thank you for your patience and cooperation,
Cristinel Anastasoaie
Adobe Business Catalyst Product Manager

View Comments