-: Traffic spikes, in this lesson, 2 2
we're going to talk about traffic spikes 3 3
and some of the things that may cause them 4 4
because this could be an indicator 5 5
or compromise within your network. 6 6
Now, when I talk about a traffic spike, 7 7
this is really any sharp increase in connection request 8 8
in comparison with a given baseline. 9 9
Now, I know this is a very generic statement 10 10
but it's important that we start out generically 11 11
because by default, 12 12
a traffic spike doesn't have to be a bad thing. 13 13
For instance, let's take a look here. 14 14
Let's say you're looking at your logs 15 15
and you start seeing that traffic was around 100 16 16
or 200 or 300 for the first 10 or 15 minutes after 1:00 AM. 17 17
But then it jumps up at around 1:15 to 1:25 18 18
and during that time you reach a peak 19 19
of about 800 megabits per second. 20 20
Then it drops down again to 200, 100, 21 21
kind of stays there through the rest of the night. 22 22
Now, is this malicious, are you under attack? 23 23
Well, maybe, we don't have enough information. 24 24
All we know right now is there is something anomalous. 25 25
This could be normal. 26 26
For instance, on my server, we run our backups 27 27
that go from our servers to offsite cloud storage 28 28
between 1:00 and 2:00 AM. 29 29
And so if I saw this, I would then look 30 30
and see where was the traffic going, 31 31
and if it was going to my backup server, 32 32
this would be normal 33 33
because it's going from my servers to a backup server 34 34
which shows a large export of data 35 35
between 1:00 and 2:00 AM on my servers. 36 36
Now, if it happened at 1:00 in the afternoon, 37 37
that would be different, but I expected it this time. 38 38
And so it really does depend 39 39
what is causing that traffic spike 40 40
and that's an important thing to consider. 41 41
Now, one of the things that could cause one 42 42
of these traffic spikes that would be malicious in nature 43 43
would be something like a denial of service attack 44 44
or a Distributed Denial of Service attack, 45 45
which is much more likely these days. 46 46
Now, when you deal 47 47
with a Distributed Denial of Service attack, 48 48
this is when you're going to have lots 49 49
of different hosts all trying to connect 50 50
to your server at once to waste your resources, 51 51
whether that's network resources, 52 52
processor resources, or memory. 53 53
Essentially, a Distributed Denial of Service attack or DDoS 54 54
is an attack that uses multiple compromised hosts, 55 55
usually bots or zombies inside of a botnet, 56 56
to overwhelm a service with request 57 57
or with response traffic. 58 58
And by doing that, 59 59
we can essentially take your machine offline 60 60
by attacking it and overwhelming it. 61 61
Now, one of my favorite examples to use 62 62
with a denial of service 63 63
or Distributed Denial of Service attack 64 64
is thinking about kids. 65 65
For instance, if you're a parent, 66 66
you may have one or two or three kids. 67 67
When I had one kid and they would look up and say, 68 68
mommy or daddy, I need help, it wasn't that big of a deal. 69 69
You could stop what you're doing, help the child 70 70
and then go back to what you're doing. 71 71
Now over time, if you have more kids, 72 72
say you have two kids, it becomes a little harder. 73 73
And so mommy might get a little more overwhelmed 74 74
because now she's got two kids 75 75
who are trying to get her attention at all times, 76 76
going, mommy, mommy, mommy and that can become overwhelming. 77 77
Now, if I add three or four or five kids, or say 20 kids 78 78
'cause you're an elementary school teacher, 79 79
you can really get overwhelmed quickly. 80 80
Now, that's the difference. 81 81
When you have one kid, that's a denial of service. 82 82
Yes, you had to stop what you're doing 83 83
and take care of the child, but as soon as you're done, 84 84
you can go right back to what you're doing 85 85
and you can recover quickly. 86 86
Now, if you had two or three or five 87 87
or 10 or 20 kids doing this, you can become overwhelmed 88 88
and it would take you a lot more time 89 89
to satisfy all of their needs 90 90
and you basically can just shut down 91 91
because there's too much going on. 92 92
Well, that's what our servers do as well. 93 93
Now, A DDoS can really overwhelm 94 94
even the most well defended networks 95 95
through the sheer volume of traffic 96 96
that they're going to be exposing you to. 97 97
This means you have to come up 98 98
with good mitigation strategies and be prepared for them 99 99
because even if you have all the best security measures, 100 100
most DDoS traffic comes at you 101 101
and it looks like legitimate traffic. 102 102
So your servers want to try to answer it. 103 103
Now, one of the other big problems we have 104 104
with DDoSs these days is that anyone can do them. 105 105
So it used to be you had to actually be a developer 106 106
and create this botnet and you'd have to buy resources 107 107
or hack other people's machines and add them to your botnet. 108 108
Well, nowadays, people actually sell botnets. 109 109
So you can actually go in 110 110
for the low price of 34.99 per month, 111 111
you can have access to the one month Diamond Plan 112 112
which will give you 3,600 seconds, 113 113
which is about six minutes of time to use this DDoS. 114 114
You can have two concurrent streams going at once, 115 115
you can get a total network traffic 116 116
of about 220 gigabits per second 117 117
and all the tools are included 118 118
and you get 24, seven support. 119 119
I mean, who wouldn't want that deal 120 120
for 34.99 per month, right? 121 121
This is the idea of how cheap 122 122
and inexpensive these botnets have gotten 123 123
and it really takes no skill at all to use them. 124 124
So anyone can. 125 125
So your network has to be prepared. 126 126
Now, as I said before, not all surges 127 127
in traffic mean you have a botnet, 128 128
but if you do have a large unexpected surge in traffic 129 129
from internet hosts, this could be the indication 130 130
of an ongoing DDoS attack against you using a botnet. 131 131
Now, it's not a clear indication, but it is an indication 132 132
and so you'd have to back that up with other factors. 133 133
Now, what are some of those other factors? 134 134
Well, you might start seeing an excessive number 135 135
of time weight connections in your load balancer 136 136
or inside your web server state table. 137 137
If you see that, plus you have high numbers 138 138
of HTTP 503 errors, 139 139
which is service unavailable log events, 140 140
this could indicate that a DDoS attack is occurring. 141 141
Now, those are great indications that you are the victim 142 142
of a DDoS attack that people are pointing at you 143 143
but how do you know you're not part of the problem 144 144
and you don't have hosts compromising your network 145 145
that are part of this problem 146 146
that are attacking somebody else? 147 147
Well, if you happen to see a large amount 148 148
of outbound traffic from your network, this could indicate 149 149
that your network contains victimized hosts 150 150
that are being used in a DDoS against other people. 151 151
And so you may want to figure out where those hosts are 152 152
and get them cleaned up 153 153
because you don't want victims on your network, right? 154 154
So now that we've talked 155 155
about DDOS attacks and what they look like coming at us 156 156
or going against somebody else, 157 157
we have to talk about how do you measure a DDoS attack? 158 158
Well, the most common way to measure it 159 159
is how much bandwidth is being consumed. 160 160
So you'll do it in megabytes per second 161 161
or gigabytes per second, or even terabytes per second. 162 162
Now, when we do this, this is all known 163 163
as bandwidth consumption. 164 164
And bandwidth consumption can be measured 165 165
as the value of bites that were sent or received 166 166
or as a percentage of your link utilization. 167 167
For example, if you're reading a news article 168 168
about a recent Distributed Denial of Service attack, 169 169
usually it'll say something like this 170 170
was a 1.5 terabit per second attack, and that's talking 171 171
about the amount of bites sent are received. 172 172
Now if you're looking at internally on your organization, 173 173
you'll know what your total link size is. 174 174
And so if you have a one gigabit per second connection 175 175
from your server out to the internet 176 176
and you see that you've been using 800 megabits per second, 177 177
you could say that was an 80% link utilization 178 178
based on this attack. 179 179
And so both those are ways you can measure it 180 180
from the outside looking in or the inside looking out. 181 181
In addition to a standard DDoS attack, 182 182
there's another kind out there called a DRDoS, 183 183
and this is a distributed reflection denial 184 184
of service attack. 185 185
Now, back in security, plus you probably learned 186 186
about reflection attacks or amplification attacks 187 187
and that's really what we're talking about here. 188 188
A DRDoS attack is a network-based attack 189 189
where the attacker dramatically increases the bandwidth sent 190 190
to the victim during the attack 191 191
by implementing an amplification factor. 192 192
A DRDoS attack can occur 193 193
when the adversary spoofs the victim's IP address 194 194
and tries to open up connections with multiple servers. 195 195
Now, by doing that, 196 196
all those servers try to respond back to them. 197 197
We talked about this back in security plus as well 198 198
when we talk about the three-way handshake that occurs. 199 199
When you're trying to connect to a server, 200 200
you send a SYN packet. 201 201
The server then will hold a space for you 202 202
and send back a SYN ACK. 203 203
It now has resources reserved to handle your request, 204 204
and then you acknowledge that by sending an ACK message. 205 205
And then you have a two-way communication 206 206
that will occur after that three-way handshake. 207 207
So SYN, SYN ACK, ACK, and then start communicating. 208 208
Well, it's like I said hi to you and you said hi to me 209 209
and then I ignored you 210 210
and you're waiting for me to say something back. 211 211
You won't hang up the phone until I say something 212 212
but I give you silence. 213 213
That's essentially what's happening inside of this attack. 214 214
Now, you don't have to just use TCP to do this, though. 215 215
There's lots of different protocols you can use. 216 216
For instance, one of the most common ones 217 217
that used to be used was the ICMP attack 218 218
where you would flood a server with ICMP requests, 219 219
which are ping requests. 220 220
Essentially, the attacker would send a ping request 221 221
to the broadcast of a subnet. 222 222
Then that subnet has all of the machines on it 223 223
responding back to that server. 224 224
This way, I send out one ping, I get back five, 10, 50, 225 225
100 computers responding to that server. 226 226
Notice here, the ping request, the source IP is spoofed. 227 227
It's saying, Hey, I'm coming from 10.1.1.2 228 228
but that's actually the IP of the server. 229 229
So when all those hosts on the subnet try to respond 230 230
to that ping request using an echo reply, 231 231
all of those replies go back 232 232
to the server instead of to the original attacker. 233 233
This is the idea of an amplification and this one uses ICMP. 234 234
Now, like I said, 235 235
there are lots of different amplification attacks out there 236 236
using different protocols. 237 237
You might use DNS. 238 238
Now, DNS is a good one that attackers use as well 239 239
because they can send out a bogus DNS query 240 240
and then that is only a small request for information 241 241
but the server has to give back a lot of information. 242 242
For instance, if the request was, 243 243
tell me what you know about diontraining.com, 244 244
the DNS server can come back and say, 245 245
I know lots about that. 246 246
I know what its mail server is, and here is its IP. 247 247
I know that it has this CNAME and that CNAME 248 248
and this SPF record and this TXT record 249 249
and it will give you all that information back, 250 250
and that can overwhelm the server 251 251
if I send out a lot of these DNS queries 252 252
and all those responses are coming back to it 253 253
and we're just taking up resources, that's the idea here. 254 254
Another amplification attack can be used 255 255
with Network Time Protocol. 256 256
Now, the reason why using NTP is so effective 257 257
is because a single NTP request will generate a response 258 258
from the server for the last 600 machines 259 259
that that server has contacted. 260 260
And so if I make a request to the NTP server, 261 261
it's going to send back 600 times what I asked for. 262 262
And so that can really overwhelm people very quickly. 263 263
Now remember, as I said before, all of this is indications. 264 264
It doesn't prove there's a DDoS attack 265 265
but it is good indications of it. 266 266
Now, when we start dealing with bandwidth consumption 267 267
and traffic spikes, this can indicate a DDoS attack 268 268
but it can also be indicative of many other types too. 269 269
And we'll talk about those as we go through this section 270 270
and the rest of the course. 271 271
So I want you to keep that in mind as we go through. 272 272
Just because you see a traffic spike 273 273
doesn't mean it's a Distributed Denial of Service attack. 274 274
Now, the other part of this is just 275 275
because you see large amounts of bandwidth 276 276
or traffic spikes being used doesn't mean it 277 277
was malicious to begin with. 278 278
For example, if you're running a website, 279 279
your website can crash 280 280
under the normal unexpected server load 281 281
if that load increases 282 282
because your website becomes popular too quickly. 283 283
Now, I've seen this happen to a lot of smaller companies 284 284
but I've also seen this happen to big companies. 285 285
A couple of years ago, I was working 286 286
at a security operations center, and at the time, 287 287
United Airlines had reported they were having issues 288 288
with their servers 289 289
and they were having problems all over the US. 290 290
Now, when this happened, I don't remember 291 291
if they actually were under attack or not 292 292
but at the same time, a couple hours later, 293 293
we started seeing the wallstreetjournal.com 294 294
and it dropped offline. 295 295
Now everyone started thinking, 296 296
oh no, Wall Street Journal's being attacked. 297 297
And they all thought there was a denial of service going on 298 298
or Distributed Denial of Service. 299 299
And when they went through and started looking at it, 300 300
that isn't what happened. 301 301
What happened was 302 302
because the United Airlines had this system crash, 303 303
which was some kind of a configuration issue, 304 304
all the news outlets started talking about it 305 305
and Wall Street Journal was one of those outlets. 306 306
Because of that, people were going 307 307
to Wall Street Journal to learn more 308 308
about this issue with United 309 309
because if it was really that they were hacked or attacked, 310 310
that was going to be a huge deal 311 311
because this was grounding flights all over America. 312 312
And so this became a big news story. 313 313
Well, it became such a big news story 314 314
that so many people were going to the wallstreetjournal.com 315 315
at the same time that it actually took down their servers. 316 316
And so it wasn't a malicious denial of service attack 317 317
or a malicious DDoS but it was a huge traffic spike 318 318
that was unexpected. 319 319
And their site became way too popular way too quickly 320 320
and actually took them down for about an hour. 321 321
Now, there's actually a name for this effect 322 322
and it's called the Slashdot Effect. 323 323
This is also known as slashdotting. 324 324
Now, this is causing a website to crash 325 325
when a smaller website can become very popular very quickly 326 326
due to exposure on some kind of a social sharing site, 327 327
like Slashdot, Reddit or Twitter, or in the case 328 328
of the Wall Street Journal, at the time, it was being shown 329 329
on things like Fox News and CNN 330 330
and people were going to that website 331 331
to try to learn more information about it. 332 332
Regardless, the effect was the same. 333 333
Way too many legitimate users all going to the site 334 334
at the same time, making it very popular very quickly 335 335
and the server couldn't handle it, and it ended up crashing. 336 336
Now, this usually happens with smaller websites 337 337
'cause their architecture isn't designed 338 338
in an elastic manner to handle an exceptionally high load. 339 339
For instance, if my company became very popular overnight 340 340
and we went from the couple of hundred thousand users 341 341
that we have on a daily basis up 342 342
to a couple of million users on a daily basis, 343 343
that could actually shut us down 344 344
because we would grow too quickly. 345 345
And so we are always looking at our load 346 346
and expanding as needed based on our user loads, 347 347
but we don't have a server 348 348
that can handle millions and millions of users 349 349
because that would be prohibitively expensive 350 350
for a small company like us. 351 351
And so we keep an eye and we expand as we grow 352 352
but that can also be a downfall for us 353 353
because if we had 10 million people show up 354 354
at our site tomorrow, it would shut us down. 355 355
And so we put different mitigations and protections in place 356 356
to be able to help prevent us 357 357
from getting taken down by a huge surge 358 358
of traffic from something like a DDoS attack. 359 359
Now, those precautions won't help us 360 360
from legitimate traffic. 361 361
So if we became popular overnight like that, 362 362
we would still get taken down. 363 363
So we have to keep an eye on that 364 364
and we're constantly looking at our logs 365 365
to make sure we grow with our user base. 366 366
So that brings us to the concept of mitigation. 367 367
So how can we mitigate a DDoS attack? 368 368
Well, there are five different ways 369 369
we're going to talk about. 370 370
The first, you want to conduct real-time log analysis 371 371
to identify patterns of suspicious traffic 372 372
and redirect it to a black hole or a sinkhole. 373 373
Now, we've talked about black holes 374 374
and sinkholes previously in this course, 375 375
and the whole idea here 376 376
is we have all this malicious traffic coming in 377 377
from a DDoS, instead of it trying to go 378 378
into our network and process it, we just want to dump it 379 379
to this black hole so we don't have to respond to it 380 380
and we don't waste processing time on it. 381 381
Next, we want to make sure 382 382
that we're using geolocation and IP reputation data 383 383
so we can redirect or ignore suspicious traffic. 384 384
Let's pretend I rent a restaurant here in the local area. 385 385
If I run a local restaurant 386 386
and I have a website to take orders for that, 387 387
do I need to allow traffic 388 388
from Russia and China and the Middle East 389 389
and Europe and any place outside of my local area? 390 390
Well, no, so in that case, I can use geolocation, say, well, 391 391
this person's trying to connect to me from California, 392 392
and since California 393 393
is way too far out of my delivery radius, 394 394
they probably don't really want to order food from me 395 395
so I can just drop that person 396 396
because they're not my real customer anyway. 397 397
So that's the idea of redirecting 398 398
or ignoring any suspicious traffic using geolocation 399 399
or IP reputational data. 400 400
People who have already been known to do bad things 401 401
with that IP address, we can block them as well. 402 402
Now, the third thing we can do 403 403
is we can aggressively close slower connections 404 404
by reducing the timeouts on the affected servers. 405 405
So let's say my server was getting under load 406 406
from this DDoS attack, 407 407
one of the things I could do is actually go in there 408 408
and change my timeouts, which by default 409 409
is generally around two to three minutes. 410 410
So if I have a three minute timeout or 180 seconds, 411 411
I can actually reduce that maybe down 412 412
to 60 seconds or 30 seconds or even 10 seconds. 413 413
And that way if you can't connect within 10 seconds, 414 414
the connection will be closed. 415 415
And if you're a legitimate person, 416 416
you'll probably come back. 417 417
If you are a Distributed Denial of Service and you're a bot, 418 418
you might go away and go attack somebody else 419 419
and hopefully we can ride out that DDoS. 420 420
The fourth item is we want to use caching 421 421
and backend infrastructure to offload processing 422 422
to other servers. 423 423
If I have one server trying to do all of the load, 424 424
it can quickly become overwhelmed. 425 425
So instead, we want to use things like proxies and caches 426 426
and other backend infrastructure 427 427
to help offload a lot of that processing to other servers 428 428
or even use a CDN where we can distribute that load 429 429
across multiple servers 430 430
and that way we can suffer through that attack 431 431
and keep serving our legitimate customers. 432 432
And then our fifth and final one 433 433
is we can utilize enterprise DDoS protection services, 434 434
such as things like CloudFlare or Alchemy. 435 435
These are commercial providers 436 436
and they sit in front of your site. 437 437
So when you actually request to go to my site, 438 438
it goes first to CloudFlare. 439 439
CloudFlare checks you, sees if you're part of a DDoS, 440 440
and if you are they black hole and sinkhole you. 441 441
If you're not, they redirect you into our site 442 442
and you get your service. 443 443
That's the way these type of services work. 444 444
Now, these services are commercial services. 445 445
There is a monthly fee for them, and depending 446 446
on how big your site is and how much protection you need, 447 447
you'll pay more or less money. 448 448
They are really good to use though 449 449
especially if you're a smaller site or a large enterprise 450 450
because as a smaller site, 451 451
you can't afford to run your own DDoS protection. 452 452
And as a larger site, 453 453
you may not want to have the headache of it. 454 454
And so you'd rather outsource this 455 455
to the experts like CloudFlare and Alchemy. 456 456
So with all that said about Distributed Denial of Service 457 457
and traffic spikes, what is our goal as a company? 458 458
Well, your goal as a company and a network defender 459 459
should be to survive the DDoS attack. 460 460
Most of the time a DDoS is going to be short-lived. 461 461
They're not sustained for long periods of time, 462 462
for hours or days or weeks. 463 463
Generally, they're just a few minutes. 464 464
As you saw earlier when I talked about buying a DDoS 465 465
and DDoS for hire, they were giving you something 466 466
like a six minute DDoS for that $35 per month. 467 467
And so generally you're going to see these things last five, 468 468
10, 15, 20 minutes and then they'll move on. 469 469
They'll go on to another target or something else. 470 470
Now, in some cases, 471 471
there have been sustained DDoS that have gone on 472 472
for several hours or even as long as a day. 473 473
But in general, you can just survive 474 474
by getting through it, mitigating what you can, 475 475
closing down those connection cycles, 476 476
offloading things to other servers, 477 477
utilizing other protections, using black holes and sinkholes 478 478
and other things like that to at least mitigate it down, 479 479
try to keep serving your users the best you can 480 480
until the attack stops and you can go on with your business.