-: High availability. 2 2
In this lesson, 3 3
we're going to talk all about high availability. 4 4
Now, when we're talking about high availability, 5 5
we're really talking about making sure our systems are up 6 6
and available. 7 7
Availability is going to be measured in what we call uptime 8 8
or how many minutes or hours you're up and available 9 9
as shown as a percentage. 10 10
Usually, you're going to take the amount of minutes 11 11
you were up, 12 12
divided by the total amount of minutes in a period, 13 13
and that gives you a percentage known as uptime. 14 14
Now, we try to maintain what is known as the five nines 15 15
of availability in most commercial networks. 16 16
This is actually really hard because that's 99.999%. 17 17
That means I get a maximum of about five minutes 18 18
of downtime per year, 19 19
which is not a whole lot of downtime. 20 20
In some cloud based networks, 21 21
they aim for six nines of availability or 99.99999%. 22 22
This equates to just 31 seconds of downtime 23 23
each and every year. 24 24
Now, as you imagine, 25 25
I need more than 31 seconds of downtime 26 26
or even five minutes of downtime 27 27
fueled up patch my servers and install a new hard drive 28 28
or put in a new router or switch when one fails. 29 29
So, how do I maintain that high level of availability? 30 30
Well, I'm going to do that, 31 31
by designing my networks to be highly available. 32 32
Now, there are two terms you need to understand 33 33
and be familiar with, 34 34
when we talk about high availability. 35 35
There is availability and reliability, 36 36
and these are different things. 37 37
When I'm talking about availability, 38 38
this is concerned with being up and operational. 39 39
When I talk about reliability, 40 40
I'm concerned with not dropping packets 41 41
inside of my network. 42 42
If your network is highly available, 43 43
but it's not reliable, 44 44
it's not a very good network 45 45
because it's dropping things all the time 46 46
and isn't doing what it's supposed to. 47 47
But conversely, 48 48
you can have a really highly reliable network, 49 49
but if it's not a highly available one, 50 50
nobody can use it either because it's down all the time. 51 51
So that wouldn't be good either. 52 52
So, let's say I had the most highly reliable network 53 53
in the entire world, 54 54
but it's only up 20 minutes a year. 55 55
That's not going to be any good, right? 56 56
So, we want to make sure we balance these two things. 57 57
We have to aim for good enough in both areas 58 58
to meet our business needs based on the available resources 59 59
and the amount of money we have to build our networks. 60 60
So, when we measure our different network components, 61 61
we have to determine how highly available they are. 62 62
And we do that through measurement of MTTR and MTBF. 63 63
Now, MTTR is the mean time to repair. 64 64
This measure is the average time it takes to repair 65 65
a network device when it breaks. 66 66
After all, 67 67
everything is going to break eventually. 68 68
So, when a device breaks, 69 69
how long does it take for you to fix it? 70 70
And how much downtime are you going to experience? 71 71
That is what we're trying to measure 72 72
when we deal with the mean time to repair. 73 73
Now, the mean time between failures or MTBF, 74 74
is going to measure the average time 75 75
between when a failure happens on a device 76 76
and the next failure happens. 77 77
Now, these two terms can often be confusing. 78 78
So, let me display it on a timeline and explain a little bit 79 79
about what they look like in the real world. 80 80
Now, let's say I had a system failure 81 81
at this first stop sign here on the left side. 82 82
Then we resume normal operations because we fix things. 83 83
That amount of time, was the time to repair. 84 84
Now, if I average all the times to repair 85 85
over the entire year for that type of device, 86 86
that's going to give me my MTTR, 87 87
my mean time to repair, the average time to repair. 88 88
Now, on the failure side of things, 89 89
we want to measure the failure time from one failure 90 90
thus using it, fixing it, 91 91
and then another failure happens. 92 92
This becomes the time between the failures. 93 93
If I average all those together, 94 94
I get the average time between failures 95 95
or the mean time between failures, MTBF. 96 96
Hopefully, you can see the difference here. 97 97
Remember, when we're dealing with mean time to repair, 98 98
we want this to be a very small number. 99 99
When we deal with the mean time between failures, 100 100
we want this to be a very large number. 101 101
This means, 102 102
and for the very small number for mean time to repair, 103 103
we can fix things really quickly 104 104
and get ourselves back online. 105 105
So, the lower the mean time to repair is, 106 106
the better the network availability. 107 107
Now, on the other hand, 108 108
we start talking about mean time between failures, 109 109
we want a really long time 110 110
because this means that the device has stayed up 111 111
and operational for a very long time before they fail. 112 112
This is going to give us better network availability, 113 113
and overall, it should give us better reliability too. 114 114
Now, we don't want a lot of failures here. 115 115
And so the more time in between failures, 116 116
the better that is for our network. 117 117
So, how do we design these networks 118 118
to be highly reliable and highly available? 119 119
Well, we're going to add redundancy to our networks 120 120
and their devices. 121 121
Now, redundancy can be achieved through a single device 122 122
or by using multiple devices. 123 123
If you're using a single device, 124 124
you're still going to have single points of failure 125 125
in your network, 126 126
but it is cheaper than being fully hardware redundant. 127 127
Let's take a look at this concept for a moment. 128 128
Here you could see a single point of failure in my network. 129 129
Even though I have two switches and multiple connections 130 130
between those switches, 131 131
which gives me additional redundancy, 132 132
that router is not giving me additional redundancy. 133 133
It's a single point of failure 134 134
because it's the only router I have. 135 135
So, even if the router has internal hardware redundancy, 136 136
like two power supplies and two network cards, 137 137
I still only have one router chassis and one circuit board 138 138
running in that router. 139 139
So, if that router goes down, 140 140
this entire network is going to stop. 141 141
Therefore, this is considered a single point of failure. 142 142
Now instead, 143 143
I could redesign the network 144 144
and I can increase its redundancy by doing this. 145 145
Notice, I now have two PCs that want to talk to each other. 146 146
And each of them has dual network interface cards 147 147
talking to two different switches. 148 148
And each of those switches talks to two different routers. 149 149
Everything is connected to everything else 150 150
in a mesh topology for these network devices. 151 151
This gives me multiple connections between each device 152 152
and provides me with link redundancy, component redundancy, 153 153
and even inside those devices, 154 154
I may have two network cards, two power supplies, 155 155
and to every other internal network component there is, 156 156
so that I have a very redundant 157 157
and highly available network. 158 158
Now, if one of those routers needs to be upgraded, 159 159
I can take it offline and update its firmware, 160 160
and then the entire time that second router 161 161
is still on the network, 162 162
maintaining the load and providing service to all the users. 163 163
Then I can put the first router back on the network, 164 164
take off the second router and then do its upgrades. 165 165
By doing this and taking turns, 166 166
I still am able to have network functions run, 167 167
and I have no downtime associated with this network. 168 168
This is how you keep a network highly available. 169 169
Now, let's talk a little bit more about hardware redundancy. 170 170
Inside these routers and other network devices, 171 171
we can have hardware redundancy or the devices themselves 172 172
could be hardware redundant. 173 173
Now, if I have two routers and they're both 174 174
serving the same function, 175 175
this is considered hardware redundancy. 176 176
I could also have hardware redundancy in the components 177 177
by having two network cards or two hard drives 178 178
or two internal power supplies on a single device. 179 179
That way, if one of them fails, the second one takes over. 180 180
Usually, you're going to find this 181 181
in strategic network devices, 182 182
things like your switches, your routers, your firewalls, 183 183
and your servers, 184 184
because you can't afford a failure 185 185
in any one of those devices, 186 186
because they would take down large portions 187 187
of your network or its services. 188 188
On the other hand, if I'm considering my laptop, 189 189
I only have one hard drive in it. 190 190
If that laptop fails or that hard drive fails, 191 191
I would just deal with that downtime. 192 192
I might buy a new laptop or a new hard drive 193 193
and then restore from an old backup. 194 194
That would get me back up and running. 195 195
Now, when we're working with end-user devices 196 196
like workstations and clients, 197 197
we often don't deal with redundancy. 198 198
But when you start getting to servers and routers 199 199
and switches and firewalls, 200 200
you need to start having hardware 201 201
and component level redundancy 202 202
because these serve lots of end-users. 203 203
We deal with this redundancy, 204 204
we can then cluster our devices and have them work either 205 205
an active-active, 206 206
or active-passive configuration. 207 207
All right. 208 208
Let's assume I have this one computer 209 209
and it has two network interface cards 210 210
that are connected to the network. 211 211
Do I want to talk to both routers at the same time? 212 212
Well, if I'm active-active, 213 213
then both of those network interface cards 214 214
are going to be active at the same time, 215 215
and they each are going to have their own Mac address, 216 216
and they're going to be talking at the same time 217 217
to either of these two routers. 218 218
This can then be done to increase the bandwidth 219 219
of this computer and load balance 220 220
across both network interface cards. 221 221
This is known as Network Interface Card teaming, 222 222
or NIC teaming, 223 223
where a group of network interface cards, 224 224
is used for load balancing and failover for a server 225 225
or another device like that. 226 226
Now, on the other hand, 227 227
we can use active-passive, 228 228
and this is going to have a primary 229 229
and a backup network interface card. 230 230
Now, one of these cards is going to be active 231 231
and being used at all times. 232 232
And when it fails, 233 233
the other card is going to go from standby and take over. 234 234
In this case, 235 235
there is no performance increased by having two cards, 236 236
but I have true redundancy and failover capabilities. 237 237
In an active-passive configuration, 238 238
both NICs are going to be working together 239 239
and they're going to have a single Mac address 240 240
that they're going to display to the network, 241 241
so they look like they're a single device. 242 242
Now, when you start looking at redundancy at layer three, 243 243
we're going to start talking about our routers. 244 244
Now here, 245 245
our clients are getting configured with a default gateway, 246 246
which is our router by default. 247 247
But, if the default gateway went down, 248 248
we wouldn't be able to leave the sub-net, 249 249
and so it'd be stuck on the internal network. 250 250
Now, we don't want that. 251 251
So instead, 252 252
we want to add some redundancy 253 253
and we'll use layer three redundancy 254 254
using a virtual gateway. 255 255
To create a virtual gateway, 256 256
we need to use either 257 257
the First Hop Redundancy Protocol, FHRP, 258 258
or the Virtual Router Redundancy Protocol, VRRP. 259 259
Now, the most commonly used First Hop Redundancy Protocol 260 260
is known as HSRP or the Hot Standby Router Protocol. 261 261
This is a layer three redundancy protocol 262 262
that's used as a proprietary First Hop Redundancy Protocol 263 263
in Cisco devices. 264 264
HSRP is going to allow for an active and a standby router 265 265
to be used together. 266 266
And instead, 267 267
we get a virtual router that's defined 268 268
as our default gateway. 269 269
The client devices like the workstations and servers 270 270
are then going to be configured to use the virtual router 271 271
as its gateway. 272 272
When the PC communicates to the IP of the virtual router, 273 273
the router will determine which physical router is active 274 274
and which one is standby. 275 275
And then, it forwards the traffic to that active router. 276 276
If the active router goes down, 277 277
the standby router will pick up the responsibility 278 278
for that active router 279 279
until the other router comes back online 280 280
and takes over its job again. 281 281
Now, with VRRP, 282 282
the Virtual Router Redundancy Protocol, 283 283
this is one that was created 284 284
by the internet engineering task force. 285 285
Is an open standard variant 286 286
of the Hot Standby Router Protocol or HSRP. 287 287
VRRP allows for one master or active router, 288 288
and the rest can then be added in a cluster as backups. 289 289
Unlike HSRP, 290 290
where you can only have one router as active, 291 291
and one as standby. 292 292
VRRP is going to allow you to have multiple standby routers. 293 293
Just like HSRP, 294 294
you're going to configure the VRRP 295 295
to create a virtual router 296 296
that's going to be used as a default gateway 297 297
for all of your client devices. 298 298
Now, in order to provide load balancing on your networks 299 299
and to increase both redundancy 300 300
and performance of your networks, 301 301
you can use GLBP, 302 302
which is the Gateway Load Balancing Protocol, 303 303
or you can use LACP the Link Aggregation Control Protocol. 304 304
Now, GLBP or the Gateway Load Balancing Protocol 305 305
is a Cisco protocol, 306 306
and it's another proprietary First Hop Redundancy Protocol. 307 307
Now, GLBP will allow us to create a virtual router 308 308
and that virtual router will have two routers 309 309
being placed behind it, 310 310
in active and standby configuration. 311 311
The virtual router or gateway will then forward traffic 312 312
to the active or standby router 313 313
based on which one has the lower current loading, 314 314
when the gateway receives that traffic. 315 315
If both can support the loading, 316 316
then the GLBP will send it to the active 317 317
since it's considered the primary device. 318 318
By using GLBP, 319 319
you can increase the speeds of your network 320 320
by using load balancing between two routers or gateways. 321 321
Now, the second thing we can use, 322 322
is LACP or Link Aggregation Control Protocol. 323 323
This is a redundancy protocol that's used at layer two. 324 324
So we're going to be using this with switches. 325 325
LACP is going to achieve redundancy by having multiple links 326 326
between the network devices, 327 327
where load balancing over multiple links can occur. 328 328
The devices are all going to be considered 329 329
part of a single combined link 330 330
even when we have multiple links. 331 331
This gives us higher speeds and increases our bandwidth. 332 332
For example, 333 333
let's pretend I have four Cat5 cables. 334 334
Each of these are connected to the same switch. 335 335
Now, each of those cables has 100 megabits per second 336 336
of bandwidth. 337 337
Now, if I use the Link Aggregation Control Protocol, 338 338
I can bind these altogether and aggregate them 339 339
to give me 400 megabits per second 340 340
of continuous bandwidth by creating 341 341
this one single combined bandwidth, 342 342
from those four connections. 343 343
Now, let's consider what happens 344 344
when that is trying to leave our default gateway 345 345
and get out to the internet. 346 346
Now in your home, 347 347
you probably only have one internet connection, 348 348
but for a company, 349 349
you may wish to have redundant paths. 350 350
For example, at my office, 351 351
we have three different internet connections. 352 352
The first is a microwave link that operates 353 353
at 215 megabits per second for uploads and downloads. 354 354
The second, is a cable modem connection. 355 355
It operates at 300 megabits per second for downloads 356 356
and 30 megabits per second for uploads. 357 357
Now the third is a cellular modem, 358 358
and that gives me about 100 megabits per second 359 359
for downloads and about 30 megabits per second for uploads. 360 360
Now, the reason I have multiple connections, 361 361
is to provide us with increased speed and redundancy. 362 362
So, to achieve this, 363 363
I take all three of these connections 364 364
and connect them to a single gateway 365 365
that's going to act as a load balancer. 366 366
If all the connections are up and running, 367 367
they're going to load balance my traffic 368 368
across all three of those connections 369 369
to give me the highest speeds at any given time. 370 370
But, if one of those connections drops, 371 371
the load balancer will remove it from the pool 372 372
until it can be returned to service. 373 373
By doing this, 374 374
I can get a maximum speed of about 615 megabits per second 375 375
for a combined download. 376 376
And on the upload, 377 377
I can get about 310 megabits per second, 378 378
when using all three connections 379 379
and they're all functioning in online. 380 380
Similarly, 381 381
you might be in an area where you can get fiber connections 382 382
to your building. 383 383
Now, in those cases, 384 384
you may purchase a primary and a backup connection. 385 385
And if you do, 386 386
you should buy them from two different providers. 387 387
If both of your connections are coming from the same company 388 388
and they go down, 389 389
well, guess what? 390 390
You just lost both of your connections 391 391
because the upstream ISP might be down. 392 392
For this reason, 393 393
it's always important to have diversity in your path 394 394
when you're creating link redundancy, 395 395
just like I did in my office. 396 396
I have a microwave connection through one ISP. 397 397
I have a cable modem through another ISP, 398 398
and I have a cellular modem through a third ISP. 399 399
That way, 400 400
if anyone goes down, 401 401
I still have two other paths I can use. 402 402
Now, the final type of redundancy that we need to discuss, 403 403
is known as multipathing. 404 404
Multipathing is used in our storage area networks. 405 405
Multipathing is used to create more than one physical path 406 406
between the server and its storage devices. 407 407
And this allows for better fault tolerance 408 408
and performance enhancements. 409 409
Basically, think of multipathing 410 410
as a form of link aggregation, 411 411
but instead of using it for switches, 412 412
we're going to use it for our storage area networks.