News
Current Reflexion Email Delays - Mail service disruption
Posted by Andrew Serody on 30 July 2019 03:02 PM

Update:

8/1/19 - 12:00pm EDT

All Reflexion services continue to process email with no delivery queue delays. This has been the case since services had been fully restored approximately 14 hours ago. The infrastructure and engineering teams continue to monitor the service 24x7 to ensure this remains the case. 

We have identified the source of the incident which caused the excessive IO that led to this issue and we are in the process of drafting a thorough root cause analysis (RCA) that will include the details of what happened, how it happened, and steps we are taking (and have already taken) to ensure that this type of incident does not occur in the future.  We will make the RCA available once it is completed.  

To reiterate, all Reflexion services are fully operational and delivering mail in real-time. The Reflexion system did not lose any email during this incident and eventually processed and delivered all inbound and outbound messages received by the service. Also, at no time did this incident affect Sophos Central or related services as the Reflexion system is completely independent of Sophos Central-managed products and services.

We will provide an update when there is something additional to inform on.

Sophos Reflexion Support





Update:

8/1/19 - 9:30am EDT

All Reflexion services continue to deliver inbound and outbound mail flow in real time.  The infrastructure and engineering teams continue to diligently monitor the service. 

While we are still finalizing the root cause of the problem, we have determined that during the initial incident the IO limits of our storage infrastructure were exceeded. 

We sincerely apologize for the inconvenience this has caused and we will provide a thorough root cause analysis (RCA) once it’s completed.

We will continue to provide updates as there is additional information around this incident to report.

Sophos Reflexion Support





Update 

7/31/19 - 9:00 pm EDT

All Reflexion services have been completely restored, and all queued emails have been delivered.  The infrastructure and engineering teams continue to diligently monitor the service. 

While we are still finalizing the root cause of the problem, we have determined that during the initial incident the IO limits of our storage infrastructure were exceeded. 

We sincerely apologize for the inconvenience this has caused and are continuing the investigation. We will provide a thorough root cause analysis (RCA) once it’s complete. 

Sophos Reflexion Support








Update:

7/31/19 - 8:15pm EDT

The Reflexion infrastructure and engineering teams continue to monitor the delay in email processing and working to ensure mail flow returns to normal as quickly as possible. 

Sophos Reflexion Support



Update:

7/31/19 - 2:30 pm EDT

Issues with Reflexion Email Security Service
 

All inbound and outbound mail delivery through Reflexion is returning to normal, although some end users may periodically notice delays due to the high volume of emails we are currently processing. We expect that any delays will continue to improve as the queues are processed. We will continue to provide updates as the operation progresses back to a fully healthy state with no delays.  

Please know that all email that was queued in the last 24 hours has been safely processed and will be sent and received as the flow resumes - no emails were lost by the Reflexion operation as a result of this issue. 

We expect improvements throughout the day. The operations and development teams are diligently monitoring the service to ensure all Reflexion customer email is fully resumed. Note: This issue affected customers of the Reflexion service and did not impact any of our services in Sophos Central.  

While we are still finalizing the root cause of the problem, we have determined that during the initial incident the IO limits of our storage infrastructure was exceeded, due to a routine maintenance process that was executed incorrectly. We are continuing the investigation and will provide a thorough root cause analysis (RCA) once complete. 

We sincerely apologize for the inconvenience this has caused and will continue to provide updates during the day. 


Sophos Reflexion Support




Update:

7/31/19 7:15 AM EDT

As of 6:30 AM EDT, Reflexion services continue to process the backlog of email that was delayed as a result the outage yesterday.  No email was lost, and we expect all queued email to be delivered over the next few hours.  Our infrastructure engineering teams continue to monitor and are actively working to resolve the issue and restore regular mail flow as quickly as possible.  

Reflexion Outbound mail is delivering without issue. 

Again, we sincerely apologize for the inconvenience this has caused and will provide an update once the queues return to normal or there is additional information to share. 

Sophos Reflexion Support






Update:

7/30/19 – 11:00PM EDT

All Reflexion services have been restored and all messages queued in Reflexion have been processed.  The inbound and outbound emails that have been queued are now being received, processed, and delivered as quickly as possible.  No email was lost, and all queued email should be delivered over the next few hours. 

Again, we sincerely apologize for the inconvenience this has caused and will provide a final update once the queues return to normal. 

Sophos Reflexion Support




Update: 

7/30/19 – 9:00PM EDT

Reflexion is still experiencing a delay in the processing of inbound and outbound email. 

Delivery of outbound email flow is processing but at a slower rate as the delivery queues are catching up and working to return to normal levels.

We are still experiencing a delay in delivery of inbound email, and our infrastructure engineering team is actively working to resolve the issue and restore mail flow as quickly as possible.

 

We are working as fast as possible to resolve the issue. We sincerely apologize for the inconvenience this has caused and will provide an update as soon as we have additional information.

 

Sophos Reflexion Support




Update:


7/30/19 - 5:50PM EDT

Reflexion is currently experiencing the following:

We are currently experiencing a delay in the processing of inbound and outbound email.

Delivery of outbound email flow is processing but at a slower rate as the delivery queues are catching up and working to return to normal levels.

We are still experiencing a delay in delivery of inbound email and outbound email.

Engineering is actively working to resolve mail flow service as quickly as possible. We will provide an update as soon as we have additional information.
Reflexion Support




Update:

7/30/19 - 4:15PM EDT

Reflexion is currently experiencing the following:

We are currently experiencing a delay in the processing of inbound and outbound email.

Delivery of outbound email flow is processing but at a slower rate as the delivery queues are catching up and working to return to normal levels.

We are still experiencing a delay in delivery of inbound email and outbound email.

Engineering is actively working to resolve mail flow service as quickly as possible. We will provide an update as soon as we have additional information or at 5:30 pm EDT.

Reflexion Support


Update:

7/30/19 - 3:30PM EDT

Reflexion is currently experiencing the following:

We are currently experiencing a delay in the processing of inbound and outbound email.

Engineering is actively working to resolve mail flow service as quickly as possible.  We will provide an update as soon as we have additional information or at 4:00 pm EDT.
 
Reflexion Support





7/30/19 - 3:00PM EDT

Reflexion is currently experiencing the following:

We are currently deferring inbound email for RTC  and delivery of  outbound email is currently delayed but will be delivered.

Engineering is actively working to resolve mail flow service as quickly as possible.  We will provide an update as soon as we have additional information or at 4:00 pm EDT.
 
Reflexion Support