In this lesson,
we're going to cover resilience and recovery testing.
Now, resilience testing is used to assess a system's ability
to withstand and adapt to disruptive events.
Recovery testing, on the other hand,
will evaluate a system's capacity
to restore normal operations after a disruptive event.
Now, at their core, these types of resilience
and recovery testing will serve as a form of fire drill
for our enterprise networks
and our organization's operations.
These tests can occur using either a tabletop exercise,
a failover test, a simulation,
or parallel processing to determine how your organization
can survive a potentially disruptive event,
like a power loss, a natural disaster,
a ransomware attack, or even a data breach.
Now first, we need to talk about tabletop exercises.
A tabletop exercise is a simulated scenario-based discussion
among your key stakeholders to assess
and improve in organization's preparedness
and response to a specific emergency
or crisis situation without the need
for the actual deployment of its resources.
So what does a tabletop event look like?
Well, it's kind of like storytelling with a twist.
Imagine that you're sitting around a table
with a bunch of other key stakeholders in your organization,
and each of you is handed a basic script.
Then a facilitator walks into the room
and declares that something bad just happened.
For example, they might say
that your security operation center just detected
that one of your domain controllers has been compromised
by a nation-state actor,
and they're then going to ask you
what your organization is going to do
based upon that piece of information
about the suspected attack,
which we call an exercise inject.
At this point,
all the stakeholders in the room are now thrown
into this hypothetical disaster,
and they're going to take turns discussing
what actions each of their teams is going to conduct
in response to this detected malicious activity.
Now, the great thing about a tabletop exercise is
that it lets each stakeholder
and their team figure out
how they're going to respond effectively to the given inject,
and this is a fairly low-cost option to use
while still providing an extremely engaging environment
for those involved in this tabletop exercise.
Now, as each stakeholder lays out their planned actions,
the other stakeholders can identify gaps
or seams in that plan that could cause issues
during a real response too.
So these plans will then get adjusted and captured as part
of your organization's future standard operating procedures
and your playbooks for a future incident response
that you can use.
Tabletops are also considered to be a form of team building
because all the stakeholders
and their teams are going to work together
to try and solve the hypothetical disaster
or malicious activities
that are being injected into that given scenario.
Second, we have a failover test.
Now, a failover test is a controlled experiment
that's designed to verify the seamless transition
of a system or application
from a primary component to a backup or secondary component
in the event of a failure
to ensure that uninterrupted functionality can be achieved
during a disaster or other incident.
For example, if your organization has already decided
that if you have a large-scale disaster
at your East Coast Data Center,
that you're going to shift all your operations
to an alternative hot site
that you've located on the West Coast
in your data center,
this can actually be done using a failover test,
where we can actually attempt to do this cutover
from the East Coast to the West Coast.
Now, since we're actually going to be taking all these actions,
this does involve more resources,
more time, and more energy to perform.
But by performing a failover test,
you're going to be able to verify
that your planned actions will actually work
when a real disaster strikes.
For example, I used to work
for a large organization located near Washington, D.C.,
and if our facility was affected by a disaster,
we had a plan where we could shift our operations
to an alternative hot site located about 1,500 miles away.
To ensure we could execute this plan during an emergency,
we actually tested this using a failover test
at least one time per year.
In preparation for the failover test,
we actually would fly out a small team to the hot site
to ensure our operations could continue smoothly.
After any issues, we always had in place a rollback plan,
where we could shift operations back to our main facility
while we figured out
why our failover didn't work out as expected,
which only happened once or twice
over the several years I was there.
Third, we have simulations.
Now, a simulation is a computer generated
or artificial representation of a real-world system
or scenario that's used to mimic various conditions,
interactions, or failures, to assess how a system
or process will respond in order to refine its performance.
Now, unlike a tabletop exercise
where the threat is simply a scenario presented
on a piece of paper,
a simulation actually allows your incident responders
and system administrators
to perform their response actions inside of a virtualized
or simulated environment.
For example, with our modern cloud-based infrastructures,
we can actually spin up a virtual
or simulated version of our corporate network
inside of the cloud,
and then we can have a red team attack that network,
while our defenders, who are known as the blue team,
are trying to detect that red team's attacks
and utilize their proper incident response techniques
to isolate the attackers from the network
and remove any infected systems from that simulated network.
With a simulation, your personnel are able to respond
to the simulated crisis in real time,
which helps test your reactions
in a more realistic environment
that you can then evaluate not just the plan itself
but also your staff members
and their ability to work
during the unpredictable chaos of a cyber attack.
After the simulation is over,
all the observers are going to provide feedback
on the attacker's performance and the defender's performance
so that each side can learn from their mistakes
and improve their skills.
Now, fourth and finally, we have parallel processing.
Parallel processing involves replicating your data
and systems processes onto a secondary system,
and then you run both the primary
and secondary systems together at the same time.
Our goal when we conduct parallel processing
is to check the reliability
and stability of your secondary setup
to make sure it can handle processing data
without disrupting your day-to-day operations
during a disaster.
Now, parallel processing does require meticulous planning,
flawless execution, and an eagle eye for detail
to ensure that there is zero disruption
to your ongoing operations.
So it is something you have to do very carefully.
Now, in resilience testing, we often use parallel processing
to help test the system's ability
to handle multiple failure scenarios simultaneously,
such as network outages,
hardware failures, or data corruption.
When it comes to recovery testing,
we use parallel processing
to evaluate how efficiently a system can recover
from multiple failure points.
So remember, when it comes to tabletop exercises,
failover tests, simulations, and parallel processing,
our goal here is to plan for the worst
and learn how to overcome any obstacles
that we might potentially face.
Resilience and recovery testing
is not just about creating a plan,
but it's about creating a plan, testing that plan,
and assuring ourselves and our stakeholders
that our plan will really work when a disaster
or incident does eventually occur to us.
Each of these methods, including tabletop exercises,
failovers, simulations, and parallel processing,
are going to offer us with different insights,
the ability to pinpoint various weaknesses,
and they'll help us to prepare our teams
for just about anything that an attacker
or the environment might throw at us.
With all these tests, though,
this isn't going to be a one-time thing.
Instead, we have to keep doing it routinely.
You should regularly test, adapt, and improve your plans
to ensure that your plans are evolving
in response to the rapidly changing threat environment
and any new vulnerabilities that may have emerged over time.
After all, your organization's resilience
is not just a check in the box.
It is an ongoing, ever-continuing journey
towards continual improvement.