SoniXCast Whitelabel partners, producers and customers,
Yesterday, August 7th, 2017 at around 19:00 hours we were noticing higher than usual packet loss on the network hardware servicing CABHS32, CABHS33, CABHS36 and CABHS37 clusters. At or around 03:00 this morning (August 8th, 2017) the aforementioned hardware failed catastrophically and an emergency replacement was performed. Unfortunately that also required restarting the cluster which meant services were offline for a few minutes and those services where monitoring was disabled were not restarted automatically.
In the coming days we will replace the current network hardware with something much more robust that can handle the influx of listeners from sonixfm.com and the global public relay network. In the meantime, we have temporarily disabled search queries for whitelabel partners pending the hardware upgrade. The global public relay was not affected and remains as before available.
The cause for the outage was determined to be a material defect in the networking hardware and although the replacement hardware is expected to function correctly, we predict much higher loads as RDNS propagates and more and more listeners are routed through our network. We are therefore erring on the side of caution and will upgrade the entire SoniXCast AnyCastIP network with better quality hardware.
Whitelabel partners, producers and customers need not be concerned about the hardware upgrade as the systems are clustered and “hot-pluggable” meaning no further disruption of service is expected.
Buffering issues within the global public network should now have been eliminated, however as we replace networking hardware, we expect any buffering issues that may remain to be eradicated.
BoomBox users are encouraged to login to the control panel and check that their services are running correctly. Please, do not hesitate to contact support with any issues.
Your SoniXCast Network Team