-: Recovery sites. 2 2
In this lesson, we're going to discuss the concept 3 3
of recovery sites. 4 4
After all things are going to break and your networks 5 5
are going to go down. 6 6
This is just a fact of life. 7 7
So what are you going to do when it comes time 8 8
to recover your enterprise network? 9 9
Well, that's what we're going to discuss in this lesson. 10 10
When it comes to designing redundant operations 11 11
for your company, 12 12
you really should consider a recovery site. 13 13
And with recovery sites, you have four options. 14 14
You see, you can have all the software and hardware 15 15
redundancy you want. 16 16
But at the end of the day, 17 17
sometimes you need to actually recover your site too. 18 18
Now this could be because there's a fire that breaks out 19 19
in your building or a hurricane or earthquake. 20 20
All of these things might require you to relocate 21 21
and if you do, you're going to have to choose 22 22
one of four options. 23 23
This could be a cold site, a warm site, a hot site 24 24
or a cloud site. 25 25
Now when we deal with cold sites, 26 26
this means that you have a building that's available 27 27
for you to use, 28 28
but you don't have any hardware or software in place. 29 29
And if you do, those things aren't even configured. 30 30
So you may have to go out to the store and buy routers 31 31
and switches and laptops and servers 32 32
and all that kind of stuff. 33 33
You're going to bring it to a new building, configure it 34 34
and then restore your network. 35 35
This means that while recovery is possible, 36 36
it's going to be slow and it's going to be time consuming. 37 37
If I have to build you out a new network and a cold site, 38 38
that means I'm going to need you to bring everything in 39 39
after the bad thing has already happened, 40 40
such as your building catching fire. 41 41
And this can take me weeks or even months 42 42
to get you fully backing up and running. 43 43
Now, the biggest benefit of using a cold site 44 44
is that it is the cheapest option 45 45
that we're going to talk about. 46 46
The drawbacks are that it is slow and essentially 47 47
this is just going to be an empty building 48 48
that's waiting for you to move in and start rebuilding. 49 49
Now next, we have a warm site. 50 50
A warm site means you have the building available 51 51
and it already contains a lot of the equipment. 52 52
You might not have all your software installed 53 53
on these servers or maybe you don't have the latest security 54 54
patches or even the data backups from your other site 55 55
haven't been recovered here yet. 56 56
But you do already have the hardware 57 57
and the cabling in place. 58 58
With a warm site, 59 59
we already have a network that's running the facility. 60 60
We have switches and routers and firewalls. 61 61
But we may not maintain it fully 62 62
each and every day of the year. 63 63
So, when a bad event happens 64 64
and you need to move into the warm site, 65 65
we can load up our configurations on our routers 66 66
and switches, install the operating systems on the servers, 67 67
restore the files from backup 68 68
and usually within a couple of days, 69 69
we can get you back up and running. 70 70
Normally with a warm site, 71 71
we're looking to restore the time between 24 hours 72 72
and seven days. 73 73
Basically, under a week. 74 74
Recovery here is going to be fairly quick, 75 75
but not everything from the original site 76 76
is going to be there and ready for all employees 77 77
at all times. 78 78
Now, if speed of recovery is really important to you, 79 79
the next type of site is your best choice. 80 80
It's known as a hot site. 81 81
Now hot site is my personal favorite. 82 82
But it's also the most expensive to operate. 83 83
With a hot site, you have a building, you have the equipment 84 84
and you have the data already on site. 85 85
That means everything in the hot site is up and running 86 86
all the time. 87 87
Ready for you to instantly switch over your operations 88 88
from your primary site to your hot site 89 89
at the flip of a switch. 90 90
This means you need to have the system and network 91 91
administrators working at that hut site every day 92 92
of the year, keeping it up and running, secured 93 93
and patched and ready for us to take over operations 94 94
whenever we're told to. 95 95
Basically, your people are going to walk out of the old site, 96 96
get in their car, drive to the new site, login 97 97
and they're back to work as if nothing ever happened. 98 98
This is great because there's very minimal downtime. 99 99
And you're going to have nearly identical levels of servers 100 100
at the main site in the hut site. 101 101
But as you can imagine, this costs a lot of money. 102 102
Because I have to pay for the building, 103 103
two sets of equipment, two sets of software licenses 104 104
and all the people to run all this stuff. 105 105
You're basically running two sites at all times. 106 106
Therefore, a hot site gets really expensive. 107 107
Now a hot site is very critical 108 108
if you're in a high availability type of situation. 109 109
Let's say you work for a credit card processing company. 110 110
And every minute they're down cost them millions of dollars. 111 111
They would want to have a hot site, right. 112 112
They don't want to be down for three or four weeks. 113 113
So they have to make sure they have their network up 114 114
and available at all times. 115 115
Same thing if you're working for the government 116 116
or the military, 117 117
they always need to make sure they're operating 118 118
cause otherwise people could die. 119 119
And so they want to make sure that is always up and running. 120 120
That's where hot sites are used. 121 121
Now if you can get away from those type of criticality 122 122
requirements though, which most organizations can. 123 123
You're going to end up settling on something like a warm site, 124 124
because it's going to save you on the cost of running 125 125
that full recovery hot site. 126 126
Now the fourth type of site we have 127 127
is known as a cloud site. 128 128
Now a cloud site isn't exactly a full recovery site, 129 129
like a cold warm or hot site is. 130 130
In fact, there may be no building for you to move 131 131
your operations into. 132 132
Instead, a cloud site is a virtual recovery site 133 133
that allows you to create a recovery version 134 134
of your organization's network in the cloud. 135 135
Then if disaster strikes, you can shift all your employees 136 136
to telework operations by accessing that cloud site. 137 137
Or you can combine that cloud site with a cold or warm site. 138 138
This allows you to have a single set of system 139 139
administrators and network administrators 140 140
that run your day to day operational networks 141 141
and they can also run your backup cloud site. 142 142
Because they can operate at all 143 143
from wherever they're sitting in the world. 144 144
Now cloud sites are a good option to use, 145 145
but you are going to be paying a cloud service provider 146 146
for all the compute time, the storage 147 147
and the network access required to use that cloud site 148 148
before, during and after the disastrous event. 149 149
So, which of these four options should you consider? 150 150
Well, that really depends on your organization. 151 151
It's recovery time objectives, the RTO 152 152
and its recovery point objectives, RPO. 153 153
Now the recovery time objective or RTO 154 154
is the duration of time and service level 155 155
within which a business process has to be restored 156 156
after disaster happens in order to avoid unacceptable 157 157
consequences associated with a breaking continuity. 158 158
In other words, our RTO is going to answer our question, 159 159
how much time did it take for the recovery to happen 160 160
after the notification of a business process disruption? 161 161
So, if you have a very low RTO, 162 162
then you're going to have to use either a hot site 163 163
or a cloud site because you need to get up and running 164 164
quickly. 165 165
That is the idea of a low RTO. 166 166
Now on the other hand, we have to think about our RPO. 167 167
Which is our recovery point objective. 168 168
Now RPO is going to be the interval of time that might pass 169 169
during the disruption before the quantity of data loss 170 170
during that period exceeds the business continuity plans 171 171
maximum allowable threshold or tolerance. 172 172
Now RPO is going to determine the amount of data 173 173
that will be lost or will have to be re-entered 174 174
during network operations in downtime. 175 175
It symbolizes the amount of data that can be acceptably lost 176 176
by the organization. 177 177
For example, in my company we have an RPO of 24 hours. 178 178
That means if all of our servers crashed right now, 179 179
I as the CEO have accepted the fact that I can lose no more 180 180
than the last 24 hours worth of data and that would be okay. 181 181
To achieve this RPO, 182 182
I have daily backups that are conducted every 24 hours. 183 183
So, we can ensure we always have our data backed up 184 184
and ready for restoral at any time. 185 185
And that means we will lose at most 24 hours worth of data. 186 186
The RTO that recovery time objective is going to be focused 187 187
on the real time that passes during a disruption. 188 188
Like if you took out a stopwatch and started counting. 189 189
For example, can my business survive 190 190
if we're down for 24 hours? 191 191
Sure. 192 192
It would hurt, we would lose some money, but we can do it. 193 193
How about seven days? 194 194
Yeah, again, we would lose some money, 195 195
we'd have some really angry students, 196 196
but we could still survive. 197 197
Now, what about 30 days? 198 198
No way. 199 199
Within 30 days all of my customers and students, 200 200
they would have left me. 201 201
They would take their certifications 202 202
through some other provider out there 203 203
and I would be out of business. 204 204
So I had to figure out what my RTO someplace between one 205 205
and seven days to make me happy. 206 206
So that's the idea of operational risk tolerance, 207 207
we start thinking about this from an organizational level. 208 208
How much downtime are you willing to accept? 209 209
Based on my ability to accept seven days, 210 210
I could use a warm site instead of a hot site. 211 211
But if I currently accept 24 hours of downtime 212 212
or five minutes of downtime, 213 213
then I would have to use a hot site instead. 214 214
RTO is used to designate that amount of real time 215 215
that passes on the clock before that disruption 216 216
begins to have serious and unacceptable impedances 217 217
to the flow of our normal business operations. 218 218
That is the whole concept here with RTO. 219 219
Now when we start talking about RPO and RTO, 220 220
you're going to see this talked about a lot in backups 221 221
and recovery as well. 222 222
When you deal with backups and recovery, 223 223
you a few different types of backups. 224 224
We have things like full backups, incremental backups, 225 225
differential backups and snapshots. 226 226
Now a full backup is just what it sounds like. 227 227
It's a complete backup of every single file on a machine. 228 228
It is the safest and most comprehensive backup method, 229 229
but it's also the most time consuming and costly. 230 230
It's going to take up the most disk space 231 231
and the most time to run. 232 232
This is normally going to be run on your servers. 233 233
Now another type of backup we have 234 234
is known as an incremental backup. 235 235
With an incremental backup, I'm going to back up the data 236 236
that changed since the last backup. 237 237
So, if I did a full backup on Sunday 238 238
and I go to do an incremental backup on Monday, 239 239
I'm only going to back up the things that have changed 240 240
since doing that full backup on Sunday. 241 241
Now another type we have is known as a differential backup. 242 242
A differential backup is only going to back up the data 243 243
since the last full backup. 244 244
So, let's go back to my example 245 245
of Sunday being a full backup 246 246
and then I did an incremental backup on Monday. 247 247
Then that backup is going to copy everything since Sunday. 248 248
But if I do an incremental on Tuesday, it's only going to do 249 249
the difference between Monday and Tuesday. 250 250
Cause Monday was the last backup on the incremental backup. 251 251
When I do it Wednesday, 252 252
I'm going to get from Tuesday to Wednesday. 253 253
And so when I do these incrementals, 254 254
I now have a bunch of smaller pieces 255 255
that to put back together when I want to restore my servers. 256 256
Now at differential on the other hand is going to be 257 257
the entire difference since the last full backup. 258 258
So if on Wednesday I did a differential backup, 259 259
I'm going to have all the data that's different from Sunday, 260 260
the last full backup all the way up through Wednesday. 261 261
This is the difference between the differential 262 262
and an incremental. 263 263
So if I do a full backup on Sunday 264 264
and then I do a differential on Monday. 265 265
Monday I did an incremental and the differential, 266 266
they're going to look the exact same. 267 267
But on Tuesday the incremental is only going to include 268 268
the stuff since Monday. 269 269
But the differential will include everything since Sunday. 270 270
This includes all of Monday and Tuesdays changes. 271 271
And so you can see how this differential is going to grow 272 272
throughout the week until I do another full backup 273 273
on the next Sunday. 274 274
Now I do an incremental, it's only that last 24 hour period. 275 275
Now the last type of backup we have is known as a snapshot. 276 276
Now if you're using virtualization 277 277
and you're using virtual machines, 278 278
this becomes a read only copy of your data frozen in time. 279 279
For example, I use snapshots a lot when I'm using virtual 280 280
machines or I'm doing malware analysis. 281 281
I can take a snapshot on my machine, 282 282
which is a frozen instant time. 283 283
And then I can load the malware and all the bad things 284 284
I need to do. 285 285
And then once I'm done doing that, 286 286
I can restore back to that snapshot which was clean 287 287
before I installed all the malware. 288 288
This allows me to do dynamic analysis of it. 289 289
Now if you have a very large Sand array or storage area 290 290
or network array, 291 291
you can take snapshots of your servers 292 292
and your virtual machines in a very quick and easy way 293 293
and then you'll be able to restore them exactly back 294 294
to the way they were at any given moment in time. 295 295
Now when we use full, incremental and differential, 296 296
most of the time those are going to be used with tape backups 297 297
and offsite storage. 298 298
But if you're going to be doing snapshots, 299 299
that's usually done to a disc like a storage area array. 300 300
Now, in addition to conducting your backups of your servers, 301 301
it's also important to conduct backups 302 302
of your network devices. 303 303
This includes their state and their configurations. 304 304
The state of a network device contains all the configuration 305 305
and dynamic information from a network device 306 306
at any given time. 307 307
If you export the state of a network device, 308 308
it can later be restored to the exact same device 309 309
or another device of the same model. 310 310
Similarly, you can backup just the configuration information 311 311
by conducting a backup of the network device configuration. 312 312
This can be done using the command line interface 313 313
on the device or using third-party tools. 314 314
For example, one organization I worked for 315 315
had thousands of network devices. 316 316
So we didn't want to go around and do a weekly configuration 317 317
backup for all those devices individually. 318 318
Instead, we configure them to do that using the tool 319 319
known as SolarWinds. 320 320
Now once a week, the SolarWinds tool would back up 321 321
all the configurations and store them 322 322
on a centralized server. 323 323
This way, if we ever had a network device that failed, 324 324
we could quickly install a spare from our inventory, 325 325
restore the configurations from SolarWinds 326 326
back to that device and we will be back online 327 327
in just a couple of minutes.