As previously reported, The US Justice Department recently indicted 13 conspirators of which one Sergey Pavlovich Polozov was personally responsible for hacking into SoniXCast systems in 2014. We now know why don’t we?
Just in case you missed it
The indicted Mr. Polozov was tasked with putting together systems in North America to be used to mount the so called “Information War” against the United States 2016 elections. In the runup, in June 2014 he along with 2 other co-conspirators broke into and took control of SoniXCast Virtual Private Servers in Toronto, Montreal and New York City. Since we couldn’t wrest control back from the hackers, we had to destroy the VPS’s and rebuild customer accounts and services from backups of our customer database. That was an “all hands on deck” effort which took about a week until everything was back to normal.
However, the hackers didn’t cover their tracks very well and we were able to obtain their personal information which we passed onto the Department of Homeland Security. As far as we know, Canada does not have an active counter espionage unit. To date: no Federal Agency has ever contacted us for further information on the subject, so we exacted some self help against the hackers that was sure to have caused some inconvenience.
One of the co-conspirators was the German national Udo Poschen who also Trolled for the German Performance Rights Organization GEMA. In a separate lawsuit, we were able attain Mr. Poschens bank records that ultimately showed that he had received payments both from the Russians and GEMA which was also reported to the authorities. Since then, Mr. Poschen has been pretty much inactive on the Internet.
But that is only a drop in the bucket
On an average day, our network team responds to over 300 network attacks. Most are very amatuer attempts to brute force into our systems or some really laughable ransom attempts, but at least once a week we receive a serious threat. An overwhelming amount (over 80%) come from US-based systems including the most recent attack that took out the router (according to OVH) in our Montreal Data Center. Over 60% originate from Russia. The remaining 40% originate from Europe and China.
So, What’re ya doing to mitigate the effects?
Immediately after the 2014 instance we moved sensitive data into a private network inaccessible from the Internet and implemented aggressive backup plans to ensure we could quickly rebuild our systems in the event of catastrophe. We also developed a sort of “Hot Swap” concept that would allow us to quickly bring a mirror of each server online in the event of attack.
The biggest crux in our efforts has been the provider. We have to host in Canada because of the broadcast license and despite popular belief, the Canadians just aren’t as technically savvy as US providers are. Further, it takes an act of congress to get exasperated OVH technician to do anything. The folks at iWeb are much more responsive (and friendly), but still technically really, really weak.
Edge firewalling is not even in the average Canadian providers vocabulary and the concept of intrusion detection systems might as well be aliens from outer space. Leaving providers like us to resort to self help in securing our systems against hackers and the occasional overzealous technician (they cause more downtime than you think).
Recently we came across a US provider who built their own cutting edge data center in Canada and other worldwide locations. After testing their technology for months, we feel confident that we’ve found a data center provider that can meet our security and networking needs and have begun moving our infrastructure over to them.
Moving is a complex multi-pronged effort that will take months to complete fully. We’ve already moved most of our web infrastructure over (which is why the websites are faster now). Next will be the cluster and relay networks, commercial partners, then finally retail customer services. We expect some obvious short (seconds maybe minutes) downtimes with the final phase, but we’ll give customers plenty of time to plan for it.
More importantly we are moving from a network that is ripe for the picking by hackers and where monkeys hammer on the hardware in order to keep it running to a shiny new government grade adaptive network where the features are not just marketing hype and SECURITY is written large.
For example; one of the things we tested was the server “Hot Swap” technique we pioneered, but never got to work efficiently because of the limitations of the provider. In a recent test with over 50 server services running, we were able to swap a server within a single ping (10ms) and not one listener dropped. Cool Huh?