Archives for category: I.T.

I have a pitch for the reissue or follow up to “The Revenge of Analog” for Mr. Sax. He should call it “The Revenge of the Revenge of Analog.” If two years of a global pandemic have taught us nothing us, and I believe it has taught us a great deal, it is that the primary thesis of Mr. Sax’s excellent 2016 book is even more right than I think even he believed possible.

Real things matter.

Digital has transformed our world and for most people this is a good thing. Digital makes life easier and more productive. It allows easy access to information like never before and it allows for an ease of communication that is straight out of science fiction. The author’s point is not that digital is necessarily a bad thing, but that to live in a solely digital only world is a cold and sterile existence that can be lacking in creativity and positive unintended, consequences.

Using examples such as books, vinyl records, music production, movies, education, paper notebooks, the design process, and games, “The Revenge of Analog” makes the case that with digital it is all to easy to fit ideas to fit processes and so by extension limit those ideas.  There is also a drive, often by those who are older and want to be seen as innovative and “with it,” to focus on the technology and then try to apply it to problems rather than start with the problem and see what solutions might work- technological or otherwise. Indeed, one of the more intriguing facts in the book is that it is often those who have grown up with digital that are the ones that see the most value in analog records, books, and notebooks for example. There is value in inconveniences if the experience is more authentic.

 The wider point is that technology and digital media are just tools. Badly implemented tools, or tools that are adopted without first understanding the problem, are destined to fail. However, what is less well understood is that when tools are easy to use and do solve multiple issues, they can also reduce the value of an experience in the mind of the participant. To embrace analog items in our digital world is not a repudiation of that world – it is an acknowledgment of its shortcomings and a possible solution to them. Digital processes in the creative world can lead to homogeneity – there is nothing more creatively open than a blank piece of paper.

One of the realizations from the pandemic that almost everyone can agree on, was that meetings over zoom, for example, are not a good replacement for meetings in person. That while some people liked working from home, others found it isolating and lacking in comradery. The pandemic almost universally proved that remote education is fraught with difficulties for both students and teachers. A class being together with a teacher has value that far exceeds the delivery of knowledge.

If there was any doubt after reading Mr. Sax’s excellent book, the pandemic removed it all.

So Mr. Sax, The Revenge of the Revenge of Analog?

Just after finishing the first draft of this review, I saw that David Sax has a new book coming out – “The Future is Analog.” So much for “The Revenge of the Revenge of Analog.”

For all the time that I have been managing veterinary hospitals I have also had oversight responsibility for the computers and technological systems (I.T.) employed at those hospitals. Coming from a technical background in the entertainment lighting world, this just seemed to be a natural extension of my existing skill sets – the things that make me good (hopefully) at what I do.

One of the things that I have always been passionate about is data security. I’ve written articles and had them published on the subject. I made sure my hospitals had proper backup procedures, good anti-virus protection, updates ran regularly, and great firewalls.  I’d always felt that there is always an element of carelessness, or lack of understanding of the risks, when hearing about those that have fallen foul of hardware failures or ransomware. Nothing in my years of experience did anything to dissuade me from this impression.

Until one of my hospitals was hacked.

The story starts first thing on a Sunday morning when I get a call from the office manager on site saying that they are unable to get their practice management software up and running. I try logging in remotely and get nowhere so I make the five-minute drive to take a look. What meets me on the screen of the server is a message that runs my blood cold and leads to feeling of despair sinking into my stomach.

“Your System has been hacked. All of your data has been encrypted. To release your, data payment must be made via Bitcoin….”

We have backups, we have a replication server, we’ll be fine.

We were not.

The ransomware attack had been possible due to one the connections that we used to allow doctors to write up their medical records from home. In addition, there had been an old user with administrator rights that had somehow been overlooked and led to the hacker being able to access the server. The height of irony was that we were in the process of moving over to a more secure system to allow remote access when this attack took place. If we had been a couple of weeks further along this attack would not have been possible as it unfolded.

Our server was encrypted, our replication server was encrypted, our daily incremental backup drive was encrypted, our weekly full backup drive was encrypted, and several workstations were also encrypted. We had no internet, no practice management software, and so no access to medical records, schedule, email, or files.

Paying was not an option on general principles.

Our last hope was offsite cloud backup.

This backup had been fully protected and within 24 hours we were able to have remote access to this so we could access schedule and records.

We tried for three days to download the massive database onto a drive to allow us to restore the server. After three days of failure, in part cause by file size, an inopportune Windows automatic update, and network / computer stability issues, our cloud backup vendor arranged for a physical drive to be sent to us. Once the drive arrived (at 8PM at night several days later) the instructions were unclear as to how to access what had been sent to us and the cloud backup provider did not have anyone on staff late at night who knew how the drive had been prepared. We finally restored functionality to most of the hospital on the seventh day after the attack.

Lessons learned

Our I.T. vendor had, for the most part, been great. They understood the position we were in and I, in turn, protected them from the owners and staff who were rightly upset and frustrated. I had several moments of frustration myself , particularly when it came to getting a physical drive from the cloud backup vendor which turned into a comedy of errors. But both sides were able to work on the problem and maintain a professional atmosphere. We’ve had a long relationship with our I.T. support vendor and they have been very good to us in turn. They understood our need to go with other suppliers for things such as phone systems and servers but were still being prepared to help support those items and the overall health of our networks. Without that long term relationship, and atmosphere of mutual trust, things could have been very difficult indeed.

We used our barely functioning network to try and download a huge amount of data. We should have done this offsite, at one of our other locations. We should have also immediately requested a physical drive to be sent to us. I offered multiple times to get on plane and courier the drive personally, however, this was turned down but did add to the pressure on the cloud backup company to get their act together.

Try to be calm. After all was said the done the total loss of business for the week that we were unable to either take care of that week or squeeze into the following week was estimated at 4% – well within the normal variation from week to week. Not even close to the amount to bother our insurance company with. Clients will understand. Deal with what you can, improvise, and communicate as much as possible with everyone.

The major lesson that I learned, however, was one of humility. Anything can he hacked. All it takes is time and a willingness to spend that time. There was, in the heat of the moment, a number of times when the blame game reared its head. I made the decision to not allow that from anyone, feeling that if there was any blame it was the wrong time to even talk about it. What I ultimately realized that what is important was not in preventing a hack, but our resilience if the face of that attack. It is not a matter of if, but when. We lost no data – I consider that a great victory. We lost little to no business – I also consider that a victory. We also came out the other side of the ransomware attack with a much stronger awareness and agreement on the importance of cyber security.

Humility is not an excuse, or a reason to not try everything possible to prevent issues. But it helps with the realization that all systems are vulnerable. That the very things that make I.T. systems so great and useful, are also the things that can lead to vulnerabilities.

Humility is directly related to resilience. When bad things happen what is important is that we can recover from them as quickly and easily as possible- not to pretend that there are no bad things or that we are immune to them.

%d bloggers like this: