I know you can do it, because I have done it multiple times, for many clients! Every scenario is different; however, I have saved $150k+ (and much higher) annually in some instances. Enormous savings are achieved due to the high expense of a using a third party vendor for dedicated seats (as opposed to shared seats) and the compressing of RTO to minutes rather than hours or days – depending on a vendor’s service level agreement.
In one instance building an in-house hot-site recovery trading floor was a big win for me in my business continuity career. I patterned much of my future thinking and many of my subsequent successful projects on this type of innovative low cost/high return on investment solution. You can extend this thinking to any type of very time-sensitive process you need quickly recovered.
Here are some suggestions based on my success in building in-house recovery solutions:
Processes, such as customer service and securities trading, require extremely aggressive RTO’s and RPO’s. If you are down even a couple of hours it can mean missing very large financial opportunities, reputation impacts and regulatory implications. The goal is to build recovery capability with a less-than-1-hour RTO. That would mean critical systems must be available and butts need to be in recovery seats ready to trade in less than one hour. So, let’s push the envelope. By the way, I have achieved this scenario and ultimately surpassed our goal and delivered customer service and trading rooms that had a <15 minute recovery time!
Your alternative is to have a hot site with a third-party vendor. I have used this solution as well and it can be very good. You will incur a significant expense, but it may make sense if the impact of the process not being available is significant. I have also used a combination of an in-house hot-site recovery solution and a vendor solution to attain maximum resilience.
Here are some tips that may help you if you are considering an in-house hot site recovery solution:
Tip – Bring together a formal project team to map out the details. You will need the backing of your senior leadership, as it is a big project. You will also need the business process you are building the recovery solutions for, IT, facilities and other professionals working as an integrated team. In my experience, it takes a great team and a couple of intense months to build the solution. I come from a project management background integrating data centers and building enterprise software solutions so it was easy for me to build a comprehensive very detailed plan. You may want to include someone from you project management office for this project, otherwise if they do not have a resource you can do it.
Tip – Consider using a conference room or similar work-space. I have utilized conference rooms and training rooms at sites located an appropriate distance (not too far or close) from the production site.
Tip – Use cloud computing solutions where appropriate. I use a mixture of cloud and internal infrastructure where it makes sense from a capability and security perspective. The cloud solution simplifies recovery from any location.
Tip – Spend as much time as necessary to get the telecommunications part right. If you are re-routing customer facing toll free numbers make sure you have captured them all and built routing tables. You will then be in a resilient position to re-route your toll-free numbers anywhere in minutes. I go into more detail in the post, ‘How Time Sensitive Call Re-Routing Can Keep Customers Happy AND Add New Customers/ Revenue’.
Tip – Partner with IT and regularly image and patch the desktops and laptops. This should be an automated process, if possible. Test aggressively. Having the solution in-house reduces travel time and any hassles using the vendor’s infrastructure – so there are no excuses not to tested.
Tip- Read the post ‘Reduce Expenses and Improve Resilience with The Laptop Re-use Project’, about utilizing retired laptops as recovery machines. I have found ‘not so old’ retired laptops or inexpensive Chromebooks can be great recovery machines. With more and more systems going to the cloud you can often use laptops with ‘lighter’ configurations as opposed to systems that require heavy powerful ‘client machine’ configurations. The best way to determine if all systems are working is to partner with IT and test – test – test. You will discover if there is any latency or system conflicts. Perhaps, you are not expecting business as usual during a crisis so a small amount of latency might be acceptable to keep the business continuing.
Tip – The secret to getting from <1 hour to <15 minutes is in the people! If you can, train shadow staff at the hot-site recovery location to assume recovering the process until the primary recovery staff can arrive at the location. I have done this on numerous occasions for customer service and trading. The business process managers have to take ownership of the cross training; however it will be worth the effort to compress any potentially very expensive down-time impacts.
Tip – Success is in the details. Do tabletops and workstation recovery exercises based on having to activate your in-house hot-site. Think about forms, special equipment, headsets, batteries, stamps… the devil is truly in the detail.
Tip –In my experience, management just loves and values this type of win-win solution.