For all the time that I have been managing veterinary hospitals I have also had oversight responsibility for the computers and technological systems (I.T.) employed at those hospitals. Coming from a technical background in the entertainment lighting world, this just seemed to be a natural extension of my existing skill sets – the things that make me good (hopefully) at what I do.
One of the things that I have always been passionate about is data security. I’ve written articles and had them published on the subject. I made sure my hospitals had proper backup procedures, good anti-virus protection, updates ran regularly, and great firewalls. I’d always felt that there is always an element of carelessness, or lack of understanding of the risks, when hearing about those that have fallen foul of hardware failures or ransomware. Nothing in my years of experience did anything to dissuade me from this impression.
Until one of my hospitals was hacked.
The story starts first thing on a Sunday morning when I get a call from the office manager on site saying that they are unable to get their practice management software up and running. I try logging in remotely and get nowhere so I make the five-minute drive to take a look. What meets me on the screen of the server is a message that runs my blood cold and leads to feeling of despair sinking into my stomach.
“Your System has been hacked. All of your data has been encrypted. To release your, data payment must be made via Bitcoin….”
We have backups, we have a replication server, we’ll be fine.
We were not.
The ransomware attack had been possible due to one the connections that we used to allow doctors to write up their medical records from home. In addition, there had been an old user with administrator rights that had somehow been overlooked and led to the hacker being able to access the server. The height of irony was that we were in the process of moving over to a more secure system to allow remote access when this attack took place. If we had been a couple of weeks further along this attack would not have been possible as it unfolded.
Our server was encrypted, our replication server was encrypted, our daily incremental backup drive was encrypted, our weekly full backup drive was encrypted, and several workstations were also encrypted. We had no internet, no practice management software, and so no access to medical records, schedule, email, or files.
Paying was not an option on general principles.
Our last hope was offsite cloud backup.
This backup had been fully protected and within 24 hours we were able to have remote access to this so we could access schedule and records.
We tried for three days to download the massive database onto a drive to allow us to restore the server. After three days of failure, in part cause by file size, an inopportune Windows automatic update, and network / computer stability issues, our cloud backup vendor arranged for a physical drive to be sent to us. Once the drive arrived (at 8PM at night several days later) the instructions were unclear as to how to access what had been sent to us and the cloud backup provider did not have anyone on staff late at night who knew how the drive had been prepared. We finally restored functionality to most of the hospital on the seventh day after the attack.
Lessons learned
Our I.T. vendor had, for the most part, been great. They understood the position we were in and I, in turn, protected them from the owners and staff who were rightly upset and frustrated. I had several moments of frustration myself , particularly when it came to getting a physical drive from the cloud backup vendor which turned into a comedy of errors. But both sides were able to work on the problem and maintain a professional atmosphere. We’ve had a long relationship with our I.T. support vendor and they have been very good to us in turn. They understood our need to go with other suppliers for things such as phone systems and servers but were still being prepared to help support those items and the overall health of our networks. Without that long term relationship, and atmosphere of mutual trust, things could have been very difficult indeed.
We used our barely functioning network to try and download a huge amount of data. We should have done this offsite, at one of our other locations. We should have also immediately requested a physical drive to be sent to us. I offered multiple times to get on plane and courier the drive personally, however, this was turned down but did add to the pressure on the cloud backup company to get their act together.
Try to be calm. After all was said the done the total loss of business for the week that we were unable to either take care of that week or squeeze into the following week was estimated at 4% – well within the normal variation from week to week. Not even close to the amount to bother our insurance company with. Clients will understand. Deal with what you can, improvise, and communicate as much as possible with everyone.
The major lesson that I learned, however, was one of humility. Anything can he hacked. All it takes is time and a willingness to spend that time. There was, in the heat of the moment, a number of times when the blame game reared its head. I made the decision to not allow that from anyone, feeling that if there was any blame it was the wrong time to even talk about it. What I ultimately realized that what is important was not in preventing a hack, but our resilience if the face of that attack. It is not a matter of if, but when. We lost no data – I consider that a great victory. We lost little to no business – I also consider that a victory. We also came out the other side of the ransomware attack with a much stronger awareness and agreement on the importance of cyber security.
Humility is not an excuse, or a reason to not try everything possible to prevent issues. But it helps with the realization that all systems are vulnerable. That the very things that make I.T. systems so great and useful, are also the things that can lead to vulnerabilities.
Humility is directly related to resilience. When bad things happen what is important is that we can recover from them as quickly and easily as possible- not to pretend that there are no bad things or that we are immune to them.
Mike Falconer 