Archives for posts with tag: cloud

For all the time that I have been managing veterinary hospitals I have also had oversight responsibility for the computers and technological systems (I.T.) employed at those hospitals. Coming from a technical background in the entertainment lighting world, this just seemed to be a natural extension of my existing skill sets – the things that make me good (hopefully) at what I do.

One of the things that I have always been passionate about is data security. I’ve written articles and had them published on the subject. I made sure my hospitals had proper backup procedures, good anti-virus protection, updates ran regularly, and great firewalls.  I’d always felt that there is always an element of carelessness, or lack of understanding of the risks, when hearing about those that have fallen foul of hardware failures or ransomware. Nothing in my years of experience did anything to dissuade me from this impression.

Until one of my hospitals was hacked.

The story starts first thing on a Sunday morning when I get a call from the office manager on site saying that they are unable to get their practice management software up and running. I try logging in remotely and get nowhere so I make the five-minute drive to take a look. What meets me on the screen of the server is a message that runs my blood cold and leads to feeling of despair sinking into my stomach.

“Your System has been hacked. All of your data has been encrypted. To release your, data payment must be made via Bitcoin….”

We have backups, we have a replication server, we’ll be fine.

We were not.

The ransomware attack had been possible due to one the connections that we used to allow doctors to write up their medical records from home. In addition, there had been an old user with administrator rights that had somehow been overlooked and led to the hacker being able to access the server. The height of irony was that we were in the process of moving over to a more secure system to allow remote access when this attack took place. If we had been a couple of weeks further along this attack would not have been possible as it unfolded.

Our server was encrypted, our replication server was encrypted, our daily incremental backup drive was encrypted, our weekly full backup drive was encrypted, and several workstations were also encrypted. We had no internet, no practice management software, and so no access to medical records, schedule, email, or files.

Paying was not an option on general principles.

Our last hope was offsite cloud backup.

This backup had been fully protected and within 24 hours we were able to have remote access to this so we could access schedule and records.

We tried for three days to download the massive database onto a drive to allow us to restore the server. After three days of failure, in part cause by file size, an inopportune Windows automatic update, and network / computer stability issues, our cloud backup vendor arranged for a physical drive to be sent to us. Once the drive arrived (at 8PM at night several days later) the instructions were unclear as to how to access what had been sent to us and the cloud backup provider did not have anyone on staff late at night who knew how the drive had been prepared. We finally restored functionality to most of the hospital on the seventh day after the attack.

Lessons learned

Our I.T. vendor had, for the most part, been great. They understood the position we were in and I, in turn, protected them from the owners and staff who were rightly upset and frustrated. I had several moments of frustration myself , particularly when it came to getting a physical drive from the cloud backup vendor which turned into a comedy of errors. But both sides were able to work on the problem and maintain a professional atmosphere. We’ve had a long relationship with our I.T. support vendor and they have been very good to us in turn. They understood our need to go with other suppliers for things such as phone systems and servers but were still being prepared to help support those items and the overall health of our networks. Without that long term relationship, and atmosphere of mutual trust, things could have been very difficult indeed.

We used our barely functioning network to try and download a huge amount of data. We should have done this offsite, at one of our other locations. We should have also immediately requested a physical drive to be sent to us. I offered multiple times to get on plane and courier the drive personally, however, this was turned down but did add to the pressure on the cloud backup company to get their act together.

Try to be calm. After all was said the done the total loss of business for the week that we were unable to either take care of that week or squeeze into the following week was estimated at 4% – well within the normal variation from week to week. Not even close to the amount to bother our insurance company with. Clients will understand. Deal with what you can, improvise, and communicate as much as possible with everyone.

The major lesson that I learned, however, was one of humility. Anything can he hacked. All it takes is time and a willingness to spend that time. There was, in the heat of the moment, a number of times when the blame game reared its head. I made the decision to not allow that from anyone, feeling that if there was any blame it was the wrong time to even talk about it. What I ultimately realized that what is important was not in preventing a hack, but our resilience if the face of that attack. It is not a matter of if, but when. We lost no data – I consider that a great victory. We lost little to no business – I also consider that a victory. We also came out the other side of the ransomware attack with a much stronger awareness and agreement on the importance of cyber security.

Humility is not an excuse, or a reason to not try everything possible to prevent issues. But it helps with the realization that all systems are vulnerable. That the very things that make I.T. systems so great and useful, are also the things that can lead to vulnerabilities.

Humility is directly related to resilience. When bad things happen what is important is that we can recover from them as quickly and easily as possible- not to pretend that there are no bad things or that we are immune to them.

I recently installed a new VOIP (Voice Over Internet Protocol) based phone system in two veterinary hospitals. I consider myself a reasonably technical person who had a grasp of the issues and drawbacks of such as system, as well as the benefits. I learned a lot during the process. While I am ultimately happy with our system, and how the installation process went, there were multiple things that I wish I had known before getting into the project. This is an attempt to pass on some of those lessons.

It should be noted that some of these lessons can also be applied to cloud-based mission critical software, such as cloud-based practice management software in the veterinary world; however, I do not have enough experience with such systems to make them a feature of this article.

First things first…

Are you the right person?

If you do not have a good understanding of how the business concerned works, at a process and protocol level, you are the wrong person to be purchasing a VOIP phone system for that business. It is very easy for people, even those who deal day in and day out with phones, to completely misunderstand the needs of a business and its phone system. Modern IP based phone systems can be very flexible and yet still have limitations. If you are the right person, don’t be afraid to get input from others; you are not perfect. You are about to radically affect how your colleagues work each and every day. Getting things right, and getting people on board, is critical.

Understanding Workflow

Map out exactly how the phone system is to work on paper with a schematic for call flow with all the relevant parties. For example, veterinary hospitals are very different from a lot of other businesses. They can have very high call volumes, few users will have dedicated extensions, and the way calls are answered can vary dramatically from other businesses.

Tackling Phone Trees

IP Phone system vendors love phone trees. They cover a multitude of sins. You may also love phone trees. Your business may also be right for using phone trees. Don’t, however, be bullied into using phone trees if you don’t want to use them. There is nothing to say that just because a phone system is capable of having a phone tree, that they have to be used.

Phone trees can work great if a business can guarantee that an employee will be a particular extension 90% of the time that it is rang, and is able to perform a particular function. If employees are constantly in flux, and rarely at a specific extension, phone trees may not be a good solution.     

Recruit Allies

Spend way more time figuring out who is installing and configuring the phone system, than the company that the phones are to be purchased from. Simply put, the installer will make or break a new phone system.

Yes, it is possible for you to configure your own system with phone based technical support.

Yes, this is a very bad idea and you will be miserable.    

In addition, get your IT vendor, or person who looks after business’s network, on board. You are about to make their lives much more complicated. They have to be on board or the installer and IT will be at locker heads from day one and setup will be hell.

Your Internet Sucks, You Just Don’t Know It

Obviously, internet speed is a potential issue with IP based phone system; however, reliability is often overlooked. When browsing the web, having the internet drop out, or have significant latency or packet loss, for 30 seconds to a couple of hours, does not often come to a user’s attention. With an IP based phone system, however, four minutes of internet down time, which will mean that a business will have no incoming or outgoing calls, can be an eternity.

The only way to find out if there are internet issues, with a current internet service, is to use a tool that looks for them. A tool such as Multi-Ping, can monitor the internet constantly for days and weeks, and send alerts about outages. This is not a complicated tool to use, or setup, however, getting some input from both your phone system installer and your IT vendor is probably sensible.

The solution to some internet issues may be to move from cable internet to having a dedicated fiber connection. This can be significantly more expensive, or may not even be available in your area. IP phone systems usually mean significant savings over traditional line-based telephones; however, the need for fiber can put a significant dent in those savings, or wipe them out entirely. It is worth looking at this issue during the initial planning stages rather than once you have an IP phone system and are dealing with multiple outages.

Choosing A System

Identify key new features that are needed in the new phone system, and features from the old system that need to be kept. Make the demonstration of new phone systems address each of these issues in detail – take nothing for granted. Have each potential vendor go through the training process on how the phones work before a purchase is made.  Don’t just settle for a demonstration. Irksome functionality, or lack of features, will only come up during training and are two easy to overlook during a sales demonstration.

Things to look out for:

  • How can a call be parked and picked up by other users?
  • How can multiple phones be paged so that users know a call is parked for them?
  • Are there different rings for internal or external calls?
  • What happens when a call is made to an extension that is in the process of dialing out?
  • How are incoming calls routed?
  • What happens when incoming calls are not answered?

Call the technical support line for the new phone system and ask some dumb questions. Do you like what you hear? How long does it take to get through?

Visit a business that has your potential new phone system already installed and has been using it for a while – even if that business is in different field to your own. It will provide valuable insight into the system working in the real world.

Signing the Contract

Get a guarantee about getting out of a new contract.

Usually, companies offer a 30-day money back guarantee. That is probably the minimum amount of time that it will take to setup and configure all but the simplest of systems. Try to get at least 60 days and agree with your installer and the phone vendor on date to go live within this period. That way, if major issues arise during the first month there are options, and leverage.

Phone Lines and Phone Numbers

In a traditional phone system, every incoming and outgoing call takes up a phone line. Each line has a phone number associated with it. With IP based phone systems there are no telephone lines and does your business want to keep these phone numbers? What will happen when a client calls one of these numbers when the new phone system is in place?

Moving numbers can take a significant amount of time and will almost certainly dictate the date and time of the new phone system going live. This is also a process that can go wrong. The disconnection of lines that are no longer needed invariably does go wrong. Ensuring that the correct lines have been disconnected, and the correct lines have been transferred is an important area to double check.

Ye Oldie Fax Machine

Faxes are pretty old school these days; however, here are plenty of businesses that continue to use them. If this is your business think long and hard before turning over this piece of phone technology to the IP phone system’s solution. There is a reason that your business has not moved away from the humble fax machine, and it is almost guaranteed that the new phone system’s fax solution is going to look a lot like email.

Consider keeping your fax machine as is until the new phone system is in place and settled. It is a change that can be made at a later date without too much trouble. In a worst-case scenario, it also gives you a backup form of communication should there be issues on day one of going live with the new system.

The Human Element

Have cheat sheets, extension lists, and phone maps ready before the system goes live. If users have to make their own it can be difficult to stop bad habits from developing. Give your team the tools to succeed.

Be prepared to make changes. Field Marshal Helmuth Karl Bernhard Graf von Moltke, a 19th century Chief of Staff of the Prussian General Staff, is famously quoted as saying; “no plan of operations extends with any certainty beyond the first contact with the main hostile force.” This is often paraphrased as; “No battle plan ever survives contact with the enemy.”

Employees, and colleagues, are not the enemy, but the concept is the same. There will be things that have not been thought of in the planning stage, even if you have involved as many people as possible in the design of the phone system. Be prepared to make changes, and adapt to make a new phone system a success for everyone.

Preparing for Disaster

What happens in an emergency, such as a complete loss of internet, or power? It is easy to leave the planning for emergencies, until all the kinks have been resolved in the new system.

This is a mistake.

Have those plans already worked out, and the kinks in the emergency plans worked out, before the new system goes live. By making the emergency plan part of the main plan it will mean that you are not scrambling when there is an issue sooner than you had hoped.

Make testing your emergency solution part of the going live process. Also make sure that the emergency procedures are written down and easy to follow. Staff are going to absorbing a lot of new information when dealing with a new phone system. It is unlikely that they are going to remember how to switch over to the back plan, weeks or even months after it was explained to them.

All the Shiny New Toys

The aim of rolling out a new phone system should be to replace the existing phone system and address some of its shortcomings. Don’t be in too much of a rush to show off just how powerful and “cool” this new toy is. Get the basics sorted and stable. Adding new features to your workflow, and foisting large amounts of change all at once, while being unable to perform key functions of the business can easily back fire and cause hospitality. There is nothing wrong with rolling out features in stages to make managing change more, well, manageable.   

Final Notes   

VOIP phone systems are tools. They should not dictate how a business functions, unless that business considers the change a benefit. It is the job of the tool to change to suit the needs of the business. For this reason, VOIP phone systems can be complicated beasts. It is therefore to be expected that installing a new phone system is a collaborative effort. Stick to your guns about what you want from a phone system, because it will be you who will suffer if it does not work how you want it to.

It is a cliché, but an ounce of prevention really is worth a pound of cure.  

%d bloggers like this: