You can’t test absolutely everything; it’s fact that rapidly becomes obvious when you start to put together business continuity test scenarios. Common sense dictates that as a priority you should test the scenarios that have higher risk and that do more damage, all part of the risk and business impact analysis that goes into BC planning. It’s interesting to see that at a recent FICO-hosted (FICO does credit scoring, among other things) meeting in London, financial services organisations considered cyber-attacks to be a major threat as the 2012 Olympic Games in London draw closer. Is that conclusion based on feeling? Or is it a prediction based on scientific analysis?
Archive for the ‘Business Continuity Testing’ Category
Business Continuity Test Scenarios and Predicting What to Test
Friday, April 20th, 2012Fail Early – a Mantra for Business Continuity Test Scenarios
Monday, March 26th, 2012In what used to be a business climate orientated firmly towards success, the notion of constructive failure has changed attitudes, hopefully opening up new possibilities for progress by liberating organisations from the notion that all failure was bad. There’s a message for business continuity test scenarios as well – it’s the “fail early, fail cheaply” mantra of entrepreneurs and innovators, who know that it takes a few false starts to home in on the winning formula. One company, Netflix, has taken this to heart with an in-house tool called the “Chaos Monkey”.
Business Continuity Test Scenarios – the Game!
Tuesday, February 14th, 2012There’s nothing like making something fun to get people involved and interested, and the same applies to business continuity test scenarios. What makes them fun depends. For some, it’s the intellectual challenge of figuring out the right way to test scenarios to cover the right proportion of all the possible outcomes. For others, it’s a competition to see how well they can do in the face of a test situation that’s put to them. Satisfying both groups of people at the same time can be a challenge, but a recent online simulation game may indicate some possibilities for the future.
The telecommunications industry and business continuity
Wednesday, October 19th, 2011The National Emergency Communications Plan drawn up by the US Government in 2008 makes interesting reading. In its introductory section, it states that “during the last three decades, the nation has witnessed how inadequate emergency communications capabilities can adversely affect response and recovery efforts”.
Business continuity test scenarios – do you have to “pull the plug”?
Saturday, October 8th, 2011Business continuity test scenarios are an integral part of good Business Continuity planning, on two conditions: they test for the right things; and that they are realistic in how they test. It’s important to keep the end goal in mind. A simple definition of business continuity can be helpful here, such as the one from the US Department of Homeland Security – “the ability of an organisation to take a lickin’ and keep on tickin’”. A good test scenario therefore has to mirror a situation where an organisation is under real pressure or in a real crisis, rather than just running your finger down a checklist of “if this, then that” line items. (more…)
How do you get the organisation to recognise the importance of Business Continuity?
Friday, September 16th, 2011The road to hell is paved with good intentions. Sure, people in an organisation want business to go on successfully. Their jobs, families and futures depend on it. If you ask them what would happen if systems suddenly crashed, if access to their workspace was blocked, they’ll probably agree it would be a disaster.
Getting real with CBCP – Certified Business Continuity Professional
Tuesday, July 12th, 2011The real power of information is when it’s applied. When theory turns into practice, or to use another expression, where the rubber meets the road. Business continuity is no exception. An organisation looking for someone with the right capabilities for a business continuity management role will want two things: in-depth understanding of the principles of BC and the trends in the industry; and demonstrable experience. However, whereas you can show how much you understand by answering questions or taking exams, how can you prove that you’re also a seasoned professional who has the practice as well as the theory under control?
That’s where the CBCP comes in – the Certified Business Continuity Professional. It’s a qualification originating from DRI International and it tackles the tricky question of certifying practical experience. This makes it of interest to, among others, professionals who have already applied their knowledge and skills in the BC industry for good results, and who now merit the recognition that certification like this brings. CBCP qualification uses references to confirm the performance of a professional in selected domains pertaining to business continuity management. While there is a exam to be passed as part of the process, what sets the CBCP qualification apart is the condition specifying a minimum of two years practical experience, which must also have been gained within the last ten years.
BC professionals who don’t yet have two years experience have other options for qualification. A number of well thought out certifications exist to encourage professionals to build a strong theoretical base, which they can then apply to gain valuable practical experience. With that experience and by keeping up to date in the field of business continuity management, CBCP is then a natural next level to aim for. CBCP is also reasonably flexible about the number of references, allowing you to put forward two references who can vouch for you across the board, or more than two if you have different references for different BC subject matters.
OpsCentre is running the Business Continuity Planning course with the CBCP exam in Melbourne from 18-22nd July. A Sydney course will be held in Sept and Canberra in November.
Contact us to express interest in joining us for one of our CBCP courses.
Sydney Round Table Event – Testing Business Continuity Plans
Thursday, May 26th, 2011OpsCentre’s Managing Director, Rod Crowder, will be facilitating a round table event on behalf of Continuity Forum, to be held 29th Jun 2011. The topic is Business Continuity Testing/Exercising.
For more information and registration details please go to the Continuity Forum website here.
We hope to see you there.
Embedding Business Continuity Management into your organisation’s culture
Thursday, November 25th, 2010Your business continuity and IT disaster recovery plans are living documents that need to continually evolve otherwise they will stagnate. If you maintain and exercise your plan it will evolve along with your organisation, helping you to be prepared should a business interruption strike.
Here are OpsCentre’s top 5 tips on how to keep the Business Continuity Plan alive in your organisation.
- Business Continuity needs a senior sponsor that has the authority and influence to establish the priority of BCP for the organisation. Get BCP on the agenda for Road Shows and Strategic Planning sessions that the Executive presents.
- Ensure that impacts upon Business Continuity Strategy are considered when assessing the business case for all new projects. Not just IT projects but business ones as well such as implementing new products, services or business locations. Ensure that any changes required to the business continuity strategy, for example extra seats at a recovery site, are included when you cost out the new project. You can also include reviewing the BIA and updating the recovery procedures for the affected business units as activities in the project. Update your business case templates and change request templates to prompt for these considerations up front.
- Include a BCP awareness package in the induction training for all new staff.
- Include business continuity ‘roles’ in position descriptions, workplans and KPI’s.
- Exercise and Test your plan every year at a minimum. Testing is not a pass or fail exercise, it helps to refine your plan and provides an excellent opportunity for staff to gain familiarity with their business continuity roles and the continuity strategies. It doesn’t have to be boring, business continuity can be an interesting, fun, team building event.
Sydney’s F3 Traffic Debacle has lessons for us in Business Continuity Management
Thursday, April 15th, 2010This week, motorists were stranded for up to 9 hours on Sydney’s F3 Motorway due to a traffic incident. Emergency plans to implement ‘contra-flow’ arrangements to get the traffic moving again were not implemented until many hours into this incident whilst people endured hours waiting in their cars with no water being distrubuted to them and no way out.
While the facts of the matter are yet to fully emerge and the reasons behind this failure to successfully execute the traffic emergency plan are not yet published, we can consider how this type of scenario can happen to any organization, even if they have business continuity plans in place if they are not thoroughly tested.
Often an organization will have a plan outlined on paper about how a given scenario will be handled. The reality, with all of the real life complications and human factors, is often quite different. This is why we exercise and test the plans.
Real life complexities are difficult to capture in your paper plans because you cannot always envisage the multiple factors that may impact on your recovery processes.
Consider factors that may affect how your recovery plan is executed and how your organization would handle it:
1. An evolving status report
Initially you are told that the incident is not too severe and will be rectified within the hour but then as time progresses it worsens in severity and time frame estimates keep gradually creeping out.
Do you know what your ‘drop dead point’ is, how long can elapse before invoking your plan?
What is your ‘maximum tolerable outage’? How long can the ‘estimated incident recovery time’ be before it is worthwhile to invoke.
Are you getting your updates from a well informed primary source? Do they understand the need for an accurate estimate?
2. Delegation of authority
What if the CEO or appointed Business Continuity Command Team is un-contactable during this incident?
Is there a backup person nominated who is definitely going to be available in their place?
Does the backup person have the complete authority to make decisions which may involve the major ramifications and expenditure?
Has this backup person been trained in how to co-ordinate the communication and oversight of recovery from an incident?
3. Communication Protocols
Imagine the chaos created if various staff members were contacted by different media outlets. Because they have not been given clear guidelines that only the ‘Communications Manager’ may issue any statements to any external parties these well meaning staff members offer their understanding of where the current situation is at. Conflicting or incorrect information is then released to the public.
How will staff react in an incident if they have not had their expectations set about who will communicate what to them?
In a state of confusion people will try and contact their supervisor, their co-workers, whomever they can get a hold of to find out what they should be doing. Just like Chinese Whispers, various accounts of what is going on and what should be done are spreading throughout the organization.
Consider the alternative. All staff have been trained in your business continuity protocols and understand how communication will occur in an incident. There are clear roles for who will co-ordinate recovery efforts and known backup persons should the nominated person be unavailable.
All staff know that there is a communication tree whereby the status updates and requirements will be communicated to them by their business continuity team leader. They know there is a hotline number and an intranet site they can log onto where the ‘Communication Manager’ will post regular updates of information that staff need to know.
Testing your plans thrashes out the finer details, highlights shortcomings and also gets all of the parties involved familiar with the plan and their role.
It is during this process that chain of communication and authority issues can be uncovered and resolved before the plan needs to be enacted in real life.
Business Continuity Test Scenarios
Monday, March 29th, 2010Testing BC and DR planning is an essential component of any “healthy” continuity management program and as such, should be undertaken on a regular basis. While this is generally “good practice,” organisations are often under internal and external compliance and governance pressures to complete additional and more complex or mature testing regimes.
There is a broad range of testing options across the spectrum depending upon the maturity of the organisation’s planning. If this is the first time that a test has been undertaken – (“green fields”) planning can start with a plan walk through (Table Top or White Board) test. These are paper based scenario workshops with business and/or technical personnel attending. The test is generally a few hours long and should question the information and logical sequence of priorities contained in the planning documentation.
At OpsCentre, we often are engaged by a client to assist with an upcoming test and use our experience to add complexity and interest to the activity. The client organisation has successfully completed tested (often the same test) on a number of occasions and would like us to provide more in-depth rigor around the process in general.
Other than experiencing a full blown disaster (which by the way is the best form of test – although not recommended on an annual basis) we have orchestrated testing workshops to assist our clients as detailed below:
1. Applications Functional Testing
- Technical failover of applications or services from the primary production facility to the alternate recovery site
- Insure that the test is isolated from production and that no “cheating” occurs whereby test attendees liaise with production resources or documentation that would not be available in a disaster
2. BC and DR End to end process flow testing
- Complete testing of the recovery facilities by business and technical units including up and downstream application restoration in the disaster recovery environment
- This can be an expensive and resource intensive exercise. The results are extensive and recommended to establish detailed baselines for all aspects of BC/DR planning
3. Denial of Access Testing
- Business site-wide tests for recovery personnel to perform a normal day’s work from their alternate recovery sites with applications/systems pointed to normal production services
- Try this test at 3:00AM – convening disaster personnel and timing their response. Disasters can occur at any time and if not possible to physically attempt this type of test, logically the process flow should include “out of hours” scenarios.
4. Facility Power Downs
- During essential mechanical and electrical maintenance activities at key facilities. Contingency plans are executed/tested concurrently
- If the production infrastructure is going to be off-line due to maintenance that is predetermined, use this opportunity to test your planning and response mechanisms to their fullest.
Completing any of the scenarios illustrated will take a fair amount of project planning and management buy-in. Considerations should be thought out well in advance of the test/audit/governance/compliance schedule so that the test exercises run as smooth as possible and the best results are achieved.
7 Habits of Highly Effective Business Continuity
Friday, January 29th, 20101. The Senior Executive actively supports Business Continuity
The CEO\Director\General Manager that believes in and wants a functional Business Continuity program in place is a critical success factor.
To have a senior Executive that is responsible for setting the priorities and vision for the organisation to stand behind BCP and communicate this to the staff is a powerful change motivator.
2. A Whole of Business Approach
A business continuity program that prioritises the organisation from the Executive’s birdseye perspective as well as analysing business impacts across all business functions in a consistent manner will lead to a better informed business continuity strategy being proposed. It allows the Executive to see the requirements of the business in a single snapshot and make a cost benefit justified decision on the level of continuity required.
3. A Single Point of Business Continuity Management
Someone needs to be responsible for BCP at an organisational level. It needs to be in their job description and a priority for them, otherwise it runs the risk of falling between the cracks. With one person accountable for co-ordinating, aggregating, monitoring the overall Business Continuity program and reporting to the Executive, the program is more likely to stay visible and maintain momentum.
4. Testing, Testing, Testing
Business Continuity should be viewed as an ongoing continuous improvement program. And as such testing is vital. It highlights flaws and validates assumptions in your business continuity plans, giving opportunity to improve them. Testing builds confidence and competence within the business continuity team as it brings home how the strategy would actually work in a variety of scenarios and how the roles will interrelate. An untested Business Continuity Plan cannot be considered viable.
5. Embedding BCP into job descriptions and procedures
The various BCP roles such as BCP Manager, Command Team Leader, Business Unit Leader, etc should be written into position descriptions so that it is very clear that is a part of the responsibilities of the staff members. Procedures for new projects, business changes and IT changes should include provision for ensuring the change has BCP/ IT Disaster Recovery aspects taken into account. All changes should have an impact analysis conducted that includes impact on BCP/IT Disaster Recovery procedures.
6. Starting on the right foot
An induction training package that briefs new employees on the Business Continuity and Emergency Management strategies and plans in place is a great way to start them off on the right foot, highlighting the importance of this to the organisation.
7. Maintenance
The person responsible as BCP Manager should be tasked with ensuring maintenance of the documentation occurs on a regular basis. Outputs from changes and testing sessions all need to be fed into the plans. Periodically the BIA should be revisited and organisation’s prioritisations and maximum tolerable outages reviewed.
Business Continuity Software Benefits
Wednesday, December 23rd, 2009Many organisations utilise software to create, support, maintain, distribute and test their Business Continuity Plans and ensure business survival in any emergency. Regardless of size, most companies can benefit greatly from the use of Business Continuity software and many options exist for its implementation and plan maintenance strategies. Some of the direct benefits that Business Continuity software can provide an organisation are as follows:
• Conducting and automating the business impact analysis (BIA) process
• Applying relational database architectures to manage plan updates quickly and efficiently, keep documentation “alive” and synchronize it with interfacing applications (e.g. automatically updating plan emergency contact lists with employees’ latest contact information when the corporate employee database changes).
• Distributing Business Continuity Plans to each business unit for training, testing and other implementation events
• Providing document-format questionnaires to ensure thorough analysis and response planning.
• Prompt notification to employees of emergency actions to take, according to corporate protocol.
Business continuity software can provide for risk and business impact assessment tools, plan-building tools, databases and collaborative planning tools, emergency notification and incident management tools. A number of vendors offer integrated modules from which to choose. A company’s BCP project may require only one or all types, depending on its current level of BC maturity and the features and scope of its proposed plan. By using such tools, even first-timers can take advantage of the planning methodologies of experienced business continuity planners.
Does your Business Continuity Plan rely on teleworking?
Friday, December 11th, 2009CSO Online article highlights the results from a recent Telework Exchange research report, finding organization’s expect staff to work from home in a pandemic but do not provide adequate resources for them to be able to do this.
Full article at CSO Online
The teleworking provisions in your Business Continuity Plans must be included as part of your test regime to ensure that this aspect of your plan will work as anticipated.
Choosing a Business Continuity Recovery Site
Wednesday, November 25th, 2009If an organization experiences a ‘denial of access’ or ‘loss of premises’ due to incidents such as extended power outage, flood or fire, an alternate location for critical business processes and staff needs to be established.
An Alternate Site is the premises to which a business unit may transfer its operations in the event of a business continuity incident. This is sometimes also known by the name Fallback Site or Recovery Site.
There are a number of different options that can be used as an Alternate Site depending on organization’s overall BCP strategy, recovery time frame requirements, budget etc. These are:
Commercial Recovery Site
In most capital cities there are organizations that provide both dedicated and shared recovery seats and some provide IT recovery infrastructure as well. Annual leasing fees are paid based on the number and type of seats required as well as for any IT equipment, storage of your IT equipment and other related services.
Internal Property Assets
Sometimes organizations may have other property assets which have vacant, underutilized or lower priority business functions housed there. These could be designated as an Alternate Site for a higher priority business function should the BCP need to be invoked. This is why it is important to have a clear prioritization of your business functions from the BIA as it will ensure lower priority business functions are vacated in the event of a significant business disruption to enable operations of a higher criticality to continue. It is also vital to have a displacement plan in place for the regular staff of the Alternate Site so everyone knows where they are going. Other considerations when planning how to use the displaced Alternate Site are transport, parking, seating, security access and IT requirements.
Telecommuting
Often staff are already geared up to telecommute and this does offer a low cost solution that suits many business functions. However there still needs to be a clear plan around which business functions are expected to telecommute and to ensure they have the resources such as IT equipment and remote access in order to do their jobs.
Vacant seats or displaced seats at a partner \third party organization
On some occasions there is a partner\third party organization that have capacity to house additional staff should the need arise. This may be a reciprocal arrangement. If an organization needs to rely on this type of arrangement it should be formalized and reviewed on a regular basis to ensure the seats are able to be made available should they be needed and to outline any commercial terms.
Commercial Serviced Offices
A commercial serviced office will certainly have the meeting room, seating and internet access required to get many people up and running initially. However, as this is a first-in first-served arrangement it is not recommended that this be relied upon as the sole recovery site for critical functions. If the serviced office is likely to be subject to increased demand from other organizations affected by the incident, you may not be able to get in as expected. It is still a useful contingency to have the contact details for some serviced offices both near the office and geographically separate as well. Hotels are also another option as they will typically have a business centre and meeting rooms.
In all instances it is best practice to maintain geographic distance between your primary site and your Alternate Site(s) in case there is a widespread incident affecting the general area of your primary site, for example, a large power outage. If your Alternate Site is too close, it may be affected as well.
Whichever type of Alternate Site is selected it is vital to include this as part of your regular Business Continuity and IT Disaster Recovery testing exercises to build staff familiarity and ensure that they can activate and function as you planned.