System Status, Increased Capacity and improved Failure Handling
Posted on Jul 02, 2015
This past spring, we made some major overhauls to our infrastructure to improve the security and integrity of our systems; in layman’s terms, we made it harder for our servers to be knocked off line and improved our ability to serve live stats to our viewers. This summer, we’ve continued to build on that, as we continue to strive for our goal of 99.99% uptime.
StatBroadcast’s System Status Panel
We have now implemented our System Status panel which can be accessed from http://systemstatus.statbroadcast.com; this site is hosted completely independently from all other tiers of our infrastructure, so even in the event of a system outage, this site will also be accessible. This site will not only report the status of all of the different tiers of our system, but will also list any current known issues within the system. The result: if you encounter a problem using our service, this site will help you determine if it is an issue on your end or on our end, and in the event of an outage, will give us an outlet for communicating vital information to all of our university partners.
Since March 2015, we have made improvements to increase our overall capacity. In addition to making our stats service more efficient, enabling us to serve more users at the same time, we have also doubled the amount of server resources and bandwidth available to us at any given time. Additionally, as traffic ramps up and we add more resources, those resources are now added at double their previous rate. These three changes we believe will enable us to handle even the busiest days of the collegiate sports schedule without any hiccups or downtime.
In addition, we have also added more failure checks and handling throughout every aspect of our service. In cases where our servers are approaching capacity, new users will be queued until additional server capacity has been added to handle them. This will eliminate the past issues of a new flood of users knocking the service offline and affecting those already on the system viewing stats; additionally, instead of receiving a never-ending browser loading screen or wondering what is wrong, this system also allows us to more quickly trap errors, give better feedback to our users about what’s going on and immediately report issues to our support team for quick resolution.
We’re confident and hopeful that all of these system improvements will aid in our ability to provide you with not only the best stats service, but uninterrupted service, for the upcoming 2015-16 academic year and thank you for your continued support. If you have any questions about the system improvements we have made or about the outages that occurred during Spring 2015, please don’t hesitate to reach out to our team.