Rackspace hosted Exchange suffered a devastating failure beginning December 2, 2022 and is still ongoing as of 12:37 AM December fourth. Initially referred to as connectivity and login concerns, the guidance was ultimately updated to reveal that they were dealing with a security event.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the early morning hours of December 2, 2022. At first there was no word from Rackspace about what the issue was, much less an ETA of when it would be solved.
Clients on Buy Twitter Verified reported that Rackspace was not responding to support e-mails.
This has been rather the day with #Rackspace. Every hosted exchange client has been down for 14 hours or two. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they fell victim to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace client independently messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the past 16 hours.
Not sure how many companies that is, but it’s considerable.
They’re serving a 554 long delay bounce so people emailing in aren’t familiar with the bounce for numerous hours.”
The main Rackspace status page used a running update of the failure but the preliminary posts had no info other than there was a failure and it was being investigated.
The first official update was on December second at 2:49 AM:
“We are examining an issue that is impacting our Hosted Exchange environments. More details will be published as they appear.”
Thirteen minutes later on Rackspace began calling it a “connection concern.”
“We are examining reports of connectivity concerns to our Exchange environments.
Users might experience an error upon accessing the Outlook Web App (Webmail) and syncing their e-mail customer(s).”
By 6:36 AM the Rackspace updates explained the ongoing problem as “connection and login issues” then later on that afternoon at 1:54 PM Rackspace announced they were still in the “investigation stage” of the failure, still trying to determine what failed.
And they were still calling it “connectivity and login problems” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
Four hours later Rackspace described the scenario as a “substantial failure”and started offering their customers free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround till they understood the problem and might bring the system back online.
The main guidance specified:
“We experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any further issues while we continue work to bring back service. As we continue to resolve the origin of the problem, we have an alternate solution that will re-activate your ability to send out and get emails.
At no cost to you, we will be supplying you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 up until more notification.”
Rackspace Hosted Exchange Security Occurrence
It was not up until almost 24 hours later at 1:57 AM on December 3rd that Rackspace formally revealed that their hosted Exchange service was struggling with a security event.
The statement further exposed that the Rackspace professionals had actually powered down and detached the Exchange environment.
“After further analysis, we have identified that this is a security event.
The recognized impact is separated to a part of our Hosted Exchange platform. We are taking required actions to evaluate and secure our environments.”
Twelve hours later that afternoon they upgraded the status page with more info that their security team and outdoors experts were still dealing with solving the blackout.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has actually not launched details of the security occasion.
A security event normally involves a vulnerability and there are 2 severe vulnerabilities presently in the wile that were covered in November 2022.
These are the two most existing vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Demand Forgery (SSRF) attack permits a hacker to check out and change information on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an aggressor is able to run malicious code on a server.
An advisory released in October 2022 described the effect of the vulnerabilities:
“A confirmed remote aggressor can carry out SSRF attacks to escalate privileges and execute arbtirary PowerShell code on vulnerable Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mailbox server, the enemy can potentially access to other resources via lateral movement into Exchange and Active Directory environments.”
The Rackspace interruption updates have actually not suggested what the specific issue was, only that it was a security event.
The most existing status update since December fourth specified that the service is still down and consumers are motivated to migrate to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in addressing the occurrence. The schedule of your service and security of your information is of high significance.
We have committed extensive internal resources and engaged world-class external know-how in our efforts to decrease negative effects to consumers.”
It’s possible that the above noted vulnerabilities belong to the security occurrence affecting the Rackspace Hosted Exchange service.
There has been no announcement of whether consumer info has actually been compromised. This event is still continuous.
Featured image by Best SMM Panel/Orn Rin