In this post we will discuss a variety of business recovery options. It does not have to be all-or-nothing. You can mix-and-match strategies to bake maximum resilience into your overall response capabilities.
Internal Recovery – ‘Sister-Sites’ Option:
I always try to leverage internal recovery sites and seats into my strategies, whenever practical. Internal seats have many advantages and can be quite valuable and cost effective compared to 3rd party vendor seats. There are several case studies detailed on Ultimate Business Continuity where I effectively recovered time-sensitive processes internally.
In-house recovery advantages:
- You control the recovery environment. I have used training rooms, meeting rooms, conference rooms, lunch rooms, ‘quiet rooms’ (these are neat little rooms) and ‘bouncing’ (in nice ways of course) employees with less time-sensitive process requirements to accommodate employees with more time-sensitive recovery needs. Walk around your locations. Is it practical to recover in-house? Remember, be creative and think ‘outside the box!
- Cost savings – In most cases you can still use the designated in-house recovery areas for production meetings with the understanding that you will commandeer them during a crisis. Get this agreed upon in writing with management, facilities, IT and other critical partners. Make sure they understand the in-house rooms will be used to recover processes during a disruptive event. Otherwise, at time of a crisis some big head SVP may make a fuss on giving up his/her daily BS coffee-klotch meeting. Trust me, it could happen.
- Testing – You can test internal recovery as often as you want to. With a 3rd party vendor you will have limited hours specified in your contract during which you can test.
- If you are using shared space with a vendor you will be subject to ‘first come-first serve’ and may not get the seats you need in the most convenient location.
- You can train ‘shadow staff’ in the internal recovery location to ‘stand up’ the process within minutes. That can enable you to meet even the most aggressive RTO’s. I have done this successfully with Customer Serve and Trading and describe it in another post.
Internal recovery seat challenges:
- You will need equipment. Employees may have their laptops available during a disruptive event, but in a scenario such as a fire during the work day (evacuation) they would need recovery equipment. If you are using a training room and it already has computers and phones, then it is not as much an issue. Otherwise, think it through. Also, read the post, ‘Reduce Expenses and Improve Resilience with The Laptop Re-use Project’ on Ultimate Business Continuity on leveraging retired laptops for recovery purposes.
- You must keep the equipment and software up-to-date. You must apply patches. That is very important. You should consider an automated tool that pushes the patches and updates to the laptops and desktops to make this manageable.
- The recovery location must be a safe distance from the production site. This is a concern for multiple reasons. You will want to be out of harm’s way. You will want to be on another electrical grid, although a campus environment can work if only one building is impacted and the sister building is functional. I have used campus recovery effectively when the scenario dictated it. In addition, there could be regulatory rules stating a recovery site must be at least 50 miles or 100 miles from the production site.
- Tip – When planning internal recovery seats at sister sites plan closely with the recovery location Incident Commander and facilities person. Communicate your needs. Learn what space they have available and how many seats you can use. Remember, they may have already promised seats to someone else. You may also have to partner on building out conference or meeting rooms to meet the expectations of the recovering employees. Communicate with the sister site regularly to determine if anything has changed in the way of available space.
- Tip – Store special equipment and supplies at the recovery location or a central location.
Work-from-Home Recovery Option:
If you have been building recovery strategies for a while you have probably noticed a trend for processes to work-from-home as their primary recovery strategy rather than going to a vendor site or even a sister site.
I believe work-from-home can be a great recovery strategy in many scenarios, especially when travel is difficult or impossible, schools are closed or employees have responsibilities to care for their parents. In a real crisis, sometimes people that said during planning they would recover 50 miles away in actuality cannot or will not do it. Sometimes I cannot blame them.
There are many office processes where people can effectively recover at home, although not so for those in the factory, warehouse, retail, health or delivery processes, to name a few. In fact, when practical, telecommuting on a regular basis has a lot of advantages for employees and organizations.
I worked primarily from home for six years as a business continuity senior analyst. Working-from-home actually improved my productivity. I had less distractions than being in the office and a 6 second commute from by bedroom to my home office. I also received a stipend for supplies and equipment. Not too shabby, right? My employer was a Fortune 100 Company. They saved a lot of money on rent and utilities. Multiply that by thousands of employees and the bottom line popped, in a positive way! Plus, the most talented people wanted to work for that company as work-from-home is a great perk.
Tip – Stress test your VPN. Otherwise you may find out the hard way that you cannot support enough work-from-home employees simultaneously during a widespread disruption. I have experienced it whereby employees incur latency issues and the inability to log onto VPN. If you think you will need to support a maximum of 12,000 simultaneous users leave room for growth and plan to support 24,000+.
Tip – If employees are using hard-tokens to get onto VPN make sure they have re-certified their VPN chip according to your security policies. You do not want to find out during a crisis that employees have old chips and cannot access the network. Work with IT to make sure you have ‘all your ducks in a row’.
Establish a work-from-home policy:
If you have employees that are primarily work-from-home, as I was, your company should have a well-documented work-from-home policy in place that every work-from-home employee must agree to and sign-off on. This can address security issues and having a separate room or area dedicated to work. HR will have information on this.
If your company has work-from-home, a smart thing to do is to establish ‘hoteling’ areas in your major locations.
These are work areas where work-from-home employees can book a seat when they need to be in the office for meetings. These areas also make excellent workstation recovery strategies in the event of a disaster. They are very useful when people do not have power at home, are impacted by flood, fire… you get it. For business continuity, these hoteling areas make us more resilient. Hoteling is a win-win scenario in my opinion!
Compliance:
Depending on your industry you should check about any compliance issues with employees working from home. For example, securities trading is highly regulated but during a crisis when people cannot safely travel, regulatory rules may be adjusted. Determine that in advance with the regulatory agencies and document it in your plans.
Security:
Another big consideration is if you will allow people to use their home computers (not corporate issued) to access your network. If this is the plan you should be thinking about security, security and oh yeah, security. You must work closely with your Information Technology and Cyber Security experts to implement this properly.
- Who has access to the home computer?
- What are the base requirements for the hardware and OS?
- How will you provide technical support when the inevitable questions arise?
One idea is to control the environment by creating virtual machines. Employees can use their home computer but they will be limited to a ‘sandbox’ terminal environment. I have done this and it works well. Think about it.
Tip – Work-from-home employees must be in-scope for recovery exercises:
Even if in-office processes indicate they can work-from-home, they should still send representatives to the workstation area recovery exercises at the 3rd party locations, sister sites or mobile trailers.
In the event there is a power outage in their home, they will need a place to work. In an evacuation scenario during which they leave their laptop in their office, they will need a place to work. It is critical they validate they can work from an alternate recovery location. You do not want to identify gaps at time of disaster. We never want that!
I suggest you have a small percentage of people test from home on a regular basis. Rotate people during normal business periods so that during the course of a year everyone who is expected to work-from-home in a crisis validates they have no issues. Having people test from home is one of the easiest requests you will make in business continuity. I have never had a complaint about testing from home.
Survey work-from-home employees after they test. Ask questions and ask for comments. Below is a starter set of questions. Please add your own:
- Was the test a success? Why and why not?
- Could you access our network from home through VPN?
- Were you able to reach all of your critical applications?
- Did you have any ergonomic issues?
- Did you have any phone issues?
- Did you have any supply issues?
- Did you have any equipment issues?
- Did you have any compliance/oversight issues?
- What could be improved?
3rd Party Vendor Work-Area-Recovery Seat Option
Using a 3rd party vendor for recovery seats can be a viable recovery solution especially if you do not have internal sites you can leverage. The advantage of a 3rd party vendor is you do not have to dedicate space or maintain up-to-date computers and software.
The downside is limited test time and if you are using shared seats, you will most likely be competing for seats with other companies on a first to declare basis. If you are using dedicated seats in a vendor hot site it can get expensive but you have great control over the environment and test time.
I have had great success using 3rd party vendors for testing and recovery during real events. I discuss details on using a 3rd party vendor in the Interchangeable Work-Area-Recovery Exercise (IWARE) post. Below are some tips:
Tip – Build relationships with hotels near the vendor recovery site. This may be critical for recovery employees so they do not have to shuttle great distances every day. Consider shift work – 3 days on and 3 days off. At time of crisis many companies may be vying for limited hotel space. If you have high impact highly time sensitive processes, it may make sense to have an agreement in place with hotels to insure you can accommodate your recovery employees.
Tip – If you are recovering multiple processes in one large vendor workstation area recovery room consider any “Chinese Wall” separation of processes regulations. In the financial securities industry, it can a be concern that trading is not commingled with certain other processes. Research this in advance.
Tip – During the BIA it should be determined if processes can work during regular, mid and night shifts. Shift work may be difficult with customer facing processes but there are other processes that may be able to do it, thereby enabling you to share recovery seats. For example, one person during the day, another during the second shift and possibly a third during the third shift. Using only one seat for all the shifts (3 people) can save significant money and produce more work in a 24-hour cycle.
Tip – Build relationships with transportation companies. You may want to use buses or vans to transport groups of employees to the recovery site. That will make it easier for them. Group transport is also valuable during winter storms or transit strikes. Sometimes it is preferable to have a professional driving.
3rd Party Vendor Mobile Trailer Recovery Option
Using 3rd party vendor trailers can be a good strategy for certain scenarios. I have implemented this solution for management, customer service and many operations teams. Depending on the vendor you use, each trailer seats approximately 20 and 40 employees. You can daisy-chain multiple trailers for larger staff recovery requirements. I have done setups for 100+ employees. The nice thing about trailers is people do not have to travel long distances to a physical recovery location. As you know, during certain scenarios, people will not want to be away from their family for an extended period.
Modern vendor recovery trailers are very impressive. Their features include:
- Comfortable seating (but watch the overhead credenzas – I used to hit my head, but fortunately the credenzas were padded)
- Bathrooms
- Kitchen space
- Televisions
- Computers
- Most important (well the bathrooms are equally as important) they have satellite connectivity for telecommunications. Therefore, critical customer facing teams can answer incoming and make outgoing calls.
Challenges:
- You probably cannot meet aggressive less-than 24-hour RTO requirements as the trailers must travel and be set up
- You need space to park them
- You might need permits due to their size
- There is not as much seating space and ‘elbow room’ as in an office
- Possible latency issues may be encountered when bouncing the calls off the satellites. You should test to get used to it.
- Although testing may be pricey, I suggest you do at least a couple of rounds of tests from the trailers. It is also important you have your IT and telecom folks closely involved and in the trailer as they will need to image the computers and work on any telecommunication issues that arise.
As I mentioned I have implemented all of the solutions described above with great success and so can you. Have fun and contact me with any questions!