Exoprise continuously monitors the operations of its service — using CloudReady sensors from all over the world in addition to other services. If there are any interruptions in service, planned or unplanned, a note will be posted here.
|Core Servers||Public Sites||Alarms||Dependant Services||Service Comms API|
Wednesday September 15th, 2021 7:15 AM
- Some machines in the PUB-TOKYO region experienced outages during a Windows update and took longer to recover than expected. We are investigating the issue to determine the root cause and to prevent this in the future.
Thursday August 19th, 2021 12:00 PM
- It looks like Microsoft has finally (after 24 hours) recognized the error of its ways and corrected an account prompt AFTER signing in. The changes appear to be laboriously rolling out across their sign in infrastructure. We’ve made no changes to our sensors.
Thursday August 19th, 2021 9:50 AM
- We have identified a core issue that is causing the sensors to periodically fail to sign in. It looks like Microsoft has recently changed the logon procedure to include an extra prompt to select an account when its not necessary.
We actually don’t know why they are prompting for account selection. It seems like an inadvertent change and partial deploy on Microsoft’s part where they didn’t tell anyone about their new sign-on flow. Additionally, its appears to only be partially deployed which is causing further problems. Finally, it only appears to be affecting Teams making it further seem like a bug.
We continue to investigate workarounds but they have to be executed carefully because Microsoft will likely roll it back once they realize their foolish mistake.
Thursday August 19th, 2021 8:30 AM
- We are investigating a Teams sign-in interaction that is causing increased sensor errors. We expect to have updates later this afternoon and will post more information here as it is discovered.
Saturday June 12th, 2021 1:00 PM
- We’ve identified a fix for the small number of customers that are affected by this condition with OWA sensors. We are continuing to test in development and stage and plan to deploy the fix at some point during the next two business days.
Friday June 11th, 2021 5:00 PM
- There are limited reports of OWA sensors periodically failing for a limited set of tenants in different regions. We are investigating the issue and attempting to pinpoint differences. We expect to have more to report on Monday, June 14th.
Thursday April 22st, 2021 2:44 PM
- Virtual machines in the Ohio region have been rebooted and are operational. We apologize for the inconvenience caused by this outage and will investigate quicker failover detection and remediation.
Thursday April 22st, 2021 2:00 PM
- We are investigating an outage for some of the machines in the Ohio region. We hope to have an update within the next 45 minutes.
Thursday April 1st, 2021 6:20 PM
- Azure DNS is recovering and so are the core Exoprise services. Exoprise recently aligned itself with our customers by moving pieces of the Exoprise infrastructure to Azure. This inadvertently introduced new dependencies on Azure services which we are examining for improvement. More information about the outage
Thursday April 1st, 2021 5:40 PM
- Core services that are dependent on Azure are experiencing DNS issues. We will update this page by 6:00 PM at the latest.
Wednesday March 31st, 2021 11:43 AM
- We rolled back DNS maintenance for our core services and recovery should be complete.
Wednesday March 31st, 2021 11:23 AM
- We are performing some emergency DNS maintenance for our core services. We expect recovery to be complete within the next 10-15 minutes. We will update this page within that time period.
Monday March 8th, 2021 1:00 AM
- Maintenance to the PUB-VIRGINIA has completed. We apologize for any inconvenience this may have caused.
Monday March 8th, 2021 9:00 AM
- Public hosting sites in the Virginia (US-EAST) region are undergoing maintenance and load balancing. We hope to have the procedures finished by noon today. We will update this page as soon as it is finished. In the meantime, some performance oriented alarms from sensors running in this region may be elevated.
Friday March 5th, 2021 1:26 PM
- A fix for the new Teams meeting experience has been deployed and is being rolled out across the Teams AV sensors. We continue to investigate various tenant channels for Teams.
Friday March 5th, 2021 8:29 AM
- The Teams AV sensors are failing for certain tenants as an updated Teams meeting experience is deployed across servers and tenants. We are investigating the issue and hope to have a resolution by the end of the day.
Thursday January 28th, 2021 3:00 PM
- We updated our Teams AV sensor to support newer Microsoft Teams interfaces that are being deployed. A very small subset of tenants and users were affected by the updated. The Teams AV sensor should update itself within the next 2 hours.
Thursday January 28th, 2021 5:30 AM
- We are investigating interface and API upgrades to Microsoft Teams that may be causing sensor errors across different tenants. We hope to have updates available by the end of the day.
Monday December 7th, 2020 7:45 PM
- Microsoft Teams Graph API and Bot infrastructure has recovered and begun to return to normal operation. We continue to monitor the infrastructure and the Exoprise Teams AV Bot. Additionally, the team is awaiting results of submitted requests for additional information from Microsoft.
Monday December 7th, 2020 6:00 PM
- We continue to investigate new rate throttling or implementation bugs from the Microsoft Graph API for joining Teams calls.
Monday December 7th, 2020 2:20 PM
- Microsoft Teams is experiencing errors and poor integration with the Exoprise Teams AV Bot. We are investigating the issue and will update later today by 6pm.
Thursday August 20, 2020 1:45 AM
- Microsoft Teams appears to have recovered at this time and the sensors were able to create Audio Video conferences again. This issue was caused by a prolonged Microsoft Teams outage and manifested through disconnected AV conference sessions.
Wednesday August 19, 2020 5:00 PM
- Fixes were deployed for the OneDrive sensor to account for newly released and updated OneDrive pages that contained errors on the page and with Microsoft deployment. The sensors will update themselves over the next few hours.
Wednesday August 19, 2020 12:00 PM
- Microsoft Teams is having outage issues for the past few hours and this is being reflected via the Teams AV sensors. We are continuing to observe the outage details and our sensor executions.
Tuesday August 18, 2020 6:00 PM
- The Exoprise Teams AV Bot was upgraded to the latest Microsoft Communications SDK codebase. Previously the Teams AV Bot was on a Beta SDK and the Bot was updated to the version 1 GA. We are continuing to deploy the updates to the Bot and investigate any new scaling requirements that may be required from the new SDK. There may be intermittent errors from interaction with the Teams AV Bot.
Tuesday August 19, 2020 3:00 PM
- Fixes were deployed for the SharePoint sensor to account for newly released and updated SharePoint pages that contained errors on upload pages. OneDrive is still being affected.
Friday July 31st, 2020 12:00 PM
- Microsoft is actively changing SharePoint and OneDrive modern page and those changes are being deployed across subsets of tenants. This is affecting SharePoint upload pages and OneDrive pages which are broken. The issue has been reported to Microsoft and we are awaiting fixes.
Wednesday July 15th, 2020 1:00 PM
- A fix was deployed for Okta SSO integration and monitoring.
Tuesday July 14th, 2020 4:00 PM
- Okta sign in forms have been updated and okta-based Single Sign-on may no longer be working for browser tests. We are investigating the issue. The current suggested workaround is to switch to “managed” (non-Federated) accounts for tenants that are leveraging Okta SSO.
Thursday June 11th, 2020 6:05 PM
- Servers in the PUB-CANADA region completed their updates and recovered. We will investigate better planning and quicker updates in the future.
Thursday June 11th, 2020 5:10 PM
- Servers in the PUB-CANADA region are being upgraded and taking longer than expected. We will update this page when the upgrades are complete.
Thursday June 11th, 2020 10:00 AM
- The Azure US-EAST region VoIP sensors have recovered. We will update this page with the cause when we receive one from Microsoft.
Thursday June 11th, 2020 8:00 AM
- Microsoft is reporting Azure issues in the US-EAST region that is affect the Azure East VoIP sensors. We will post an update when we get one.
Tuesday June 2nd, 2020 7:30 PM
- We deployed a fix for the Teams AV sensor and it should be rolling out to any of the sensors over the next few hours. Microsoft changed the internal call initiation interface.
Tuesday June 2nd, 2020 6:30 PM
- Call creation to the Exoprise AV Bot is failing and we continue to investigate. We hope to have an update by tomorrow morning.
Tuesday June 2nd, 2020 4:30 PM
- We are investigating issues with our Exoprise AV Bot for the Teams AV Sensor. We hope to have and update or fix within the next two hours.
Thursday May 28th, 2020 4:00 PM
- Email sensors and autodiscover testing for Office 365 have not reported any failures within the past 24 hours. There may still be cached DNS entries out there for poorly configured DNS servers or providers.
Thursday May 28th, 2020 8:30 AM
- DNS entries for bad autodiscover servers appear to be subsiding as caches are updated depending on your provider. We will continue to monitor the situation.
Wednesday May 27th, 2020 8:00 AM
- We are investigating issues with Exchange Online and Exchange Web Services. There are potential issues with Autodiscover and their round-robin DNS. Autodiscover servers are redirecting to broken autodiscover services. These are Microsoft issues and the offending autodiscover servers are not responding and causing timeouts.
- A potential workaround is to avoid autodiscover and utilize OAuth or hardcode the Exchange Web Services URL: https://outlook.office365.com/EWS/Exchange.asmx
We will update this page when we understand more or if the autodiscover servers, DNS entries are fixed.
Tuesday May 26th, 2020 8:00 AM
- We are investigating issues with SharePoint sensors and accelerating a possible upgrade. Microsoft’s modern SharePoint interfaces have been updated and are causing incompatibilities with our sensors. Investigations are on-going and some customers may have received upgrades already.
- A possible workaround is to disable the modern SharePoint interface for the sites that are being monitored.