Status pages
At approximately 8:19PM EDT, StatusCast’s engineers were alerted that some status page and admin applications were inaccessible. The team identified that its hosting partner, Microsoft, was experiencing some issues in its US East region related to app services and SQL databases connections. As of 9:03PM EDT services have been restored and StatusCast’s team is currently working with Microsoft to fully investigate the incident. Once the team has completed it’s investigation we will follow up with an RCA.
At this time StatusCast should be operating fully as expected, if you continue to have any further issues please contact us at support@statuscast.com
As of 9:03PM EDT services have been restored and StatusCast’s team is currently working with Microsoft to fully investigate the incident. Once the team has completed it’s investigation we will follow up with an RCA.
In working with Microsoft, StatusCast’s team confirmed that the disruption was due to an outage with SQL Databases located in Azure’s US East region which is where StatusCast is primarily hosted:
StatusCast itself was impacted by this outage from approximately 8:19 PM EDT and had fully recovered by 9:03 PM EDT. StatusCast’s team will continue to work closely with Microsoft to further optimize its offering to help ensure that impact of service provider outages is as minimal as possible.
We are very excited to announce that StatusCast has been acquired by 4Me! Since 2013 we have been working hard to close the gap between service outages and those who are impacted, and this acquisition is one large step further in our journey of providing critical information to those who need it most.
The inclusion of StatusCast's features will aid 4Me in it's mission to modernize service management for organizations. Click here to read more!
We want to keep you fully informed about recent security enhancements on our platform, especially if you are utilizing third-party agents, bots, uptime monitors, or accessing our services through APIs with nonstandard user agents.
Recent Security Enhancements
As part of our commitment to maintaining robust digital security, we have increased the surface security of our firewall and denial of service protection services. This proactive measure is crucial to counter the significant rise in distributed denial of service (DDoS) attacks and unauthorized penetration attempts.
If you're using third-party agents, bots, uptime monitors, or APIs with nonstandard user agents, you may experience inconsistencies in accessing our services. We understand the importance of uninterrupted access to our services for your operations. If you encounter any access difficulties, please contact us immediately. Our team is ready to work closely with you to ensure any problems are swiftly resolved.
We appreciate your understanding and cooperation as we enhance our security measures. Thank you for your continued trust in our services. Together, we ensure a secure and efficient digital experience.
The StatusCast team will be performing a maintenance January 7, 8:00am EST, the estimated duration is 1h. During this period StatusCast's technical team will be required to make a DNS update to *.status.page traffic which may cause brief interruptions in connecting to StatusCast. CNAMEs will continue to work as expected during this time. While we understand the importance of uninterrupted service, these updates are crucial to ensuring a secure and efficient environment for all our users.
We appreciate your understanding and thank you for your continued trust in our services. If you have any urgent concerns or questions related to this maintenance, please feel free to reach out to our support team at support@statuscast.com.
This maintenance has been completed.
At approximately 9:38AM EST StatusCast engineers detected malicious activity targeting our services. The attack, aimed at overwhelming our service and causing disruptions, was neutralized, ensuring minimal impact on our operations by 9:49AM EST. We will continue to monitor the platform's health and will perform an in-depth investigation to the malicious activity targeted against StatusCast.
Services continue to operate as expected. StatusCast's team will provide additional information around this event once our investigation has been fully completed.
The maintenance that was scheduled for today, December 31st at 8:00AM EST has been rescheduled for January 7th at 8:00AM EST. We apologize for any inconvenience, if you have any questions please send them to support@statuscast.com.
Earlier today StatusCast's support team received reports of users sporadically getting a 403 "Unauthorized" error when authenticating to the status page and admin portal. Engineers investigated the reports and confirmed that one server in rotation was at fault and have performed an update to resolve the issue. If you continue to receive 403 errors when authenticating please reach out to us at support@statuscast.com
The StatusCast team will be performing a maintenance on December 17, 8:00am EST, the estimated duration is 2h. We do not expect any impact to your service but in some cases there may be a brief interruption.
This maintenance has been completed.
Starting August 30th, 2023 for Public Status Pages that allow SMS subscriptions StatusCast will now require that a valid email address be confirmed before a person can fully establish a new SMS subscription.
This change in subscription workflow is to help prevent malicious parties from attempting to commit SMS fraud which has become a growing concern for many SaaS companies dealing with mass notifications. We here at StatusCast have witnessed this trend, in the past 6 months the quantity of malicious traffic attempting to commit SMS fraud has increased drastically. While we have continued to implement industry best practices to safeguard against this sort of activity, ultimately real user confirmation is the most effective way to prevent such unwanted attention.
At this time services have been restored and should be operating as normal. If you continue to have any issues please contact support@statuscast.com to open a ticket. We will follow to this event with an RCA detailing what occurred and how we will handle this moving forward.
Describe the full incident details below:
On July 21st, 2023 at approximately 9:40 EDT StatusCast’s engineers received alerts that the application was displaying a HTTP Error 500.30 error when attempting to access any *.status.page status page or admin portal. During this period any notifications in progress or from schedule maintenance would have continued to work as expected. Additionally, during this period anyone using StatusCast’s legacy(*.statuscast.com) version of the application was not impacted.
Describe action taken by StatusCast to mitigate issue:
Engineers immediately began to investigate the cause of the problem. StatusCast’s service provider, Azure, indicated that it was undergoing maintenance in the region that StatusCast’s is primarily hosted on(US East). Engineers got in contact with Microsoft to confirm and to get additional insight as the issue itself was impacting the failover region(US West). During this process StatusCast deployed an additional instance to another Azure region which experienced the same errors as both East and West.
The root cause of the problem ultimately was related to Azure’s maintenance and the availability of one of StatusCast’s databases used for managing connections to the application. Leading up to the outage StatusCast’s operations team was preparing for its monthly penetration test which regularly involves a fresh test database for a reserved test application. The updated connection was not properly propagated to all of StatusCast’s application servers and traffic manager which unfortunately caused the subsequent errors.
Once the issue had been identified StatusCast’s engineers were quickly able to restore service. StatusCast development team will be performing an emergency patch today(July 21st, 2023) to ensure that an issue like this can be caught without the application going unavailable.