Instructor: URL analysis. 2 2
Now, at this point in your career, 3 3
you should know what a URL is. 4 4
A URL is a uniform resource locator. 5 5
It's that little address that you type 6 6
at the top of your web browser 7 7
to get to diontraining.com 8 8
or comptia.org 9 9
or whatever other website you want to go to. 10 10
Now, a major portion of a cybersecurity analyst job 11 11
is looking at different URLs 12 12
in a proxy log or other security logs 13 13
to determine what websites have been visited 14 14
and what type of data has been passed to those websites. 15 15
Now URLs have the ability 16 16
to encode some action or some data, 17 17
and that can then be submitted 18 18
to the server during the request. 19 19
When you conduct a URL analysis, 20 20
this is basically just an activity 21 21
that's performed to identify 22 22
whether a link is already flagged 23 23
on an existing reputation list, 24 24
and if not, 25 25
to identify what malicious script 26 26
or activity might be coded within it. 27 27
Now, this is important 28 28
because there's some encoding 29 29
that can happen inside the URL 30 30
and so you need to use the right tools 31 31
to be able to resolve that percent encoding 32 32
that might be there, 33 33
assess what sort of redirection 34 34
the URL might be performing, 35 35
and showing the source code of any scripts 36 36
that are called by the URL without executing them. 37 37
And you want to do all of this inside a sandbox environment 38 38
so you don't infect your own machine. 39 39
Now let's take a look here 40 40
at a simple URL and I say simple 41 41
but it's not really that simple. 42 42
You can see how long it is here on the screen. 43 43
It's diontraining.com/upload.php?post= 44 44
and then this long thing. 45 45
Now, as we go through this lesson, 46 46
we're going to come back to this URL quite a few times, 47 47
but first, before we start analyzing it, 48 48
we really have to understand some basics of URL analysis. 49 49
That way we can better understand this URL 50 50
and see if it's really good or bad 51 51
or if there's some kind of malicious thing going on. 52 52
Now there are three things 53 53
we're going to talk about in this lesson. 54 54
We're going to talk about HTTP methods, 55 55
we're going to talk about HTTP response codes 56 56
and we're going to talk about percent encoding. 57 57
First, what is an HTTP method? 58 58
Well, this is a set of request methods 59 59
that indicates the desired action 60 60
to be performed for a given resource. 61 61
Now, when I talk about a resource, 62 62
essentially think about that 63 63
as something on the server. 64 64
Could be a webpage, an image, a file 65 65
or something like that. 66 66
Now a request is going to contain a method, 67 67
a resource, a version number, 68 68
the header and the body of the request. 69 69
We put all that information together 70 70
when we send something out through a URL. 71 71
So when you go to diontraining.com, 72 72
you are actually sending an HTTP method 73 73
saying please, go to this website, get the page, 74 74
and send it back to me 75 75
with the body and the header 76 76
and that's what our server does. 77 77
Now there are many different methods 78 78
that we're going to talk about here. 79 79
The first one is get. 80 80
A get is the principle method 81 81
that's used when you do an HTTP request 82 82
and it's used to retrieve a resource. 83 83
So again, when you go to diontraining.com, 84 84
you are using a get saying, get me that page, 85 85
get me the index page of that website. 86 86
The second one we have is a post 87 87
and this is what you saw 88 88
in the URL I showed you earlier. 89 89
A post is used to send data to the server 90 90
for processing by the requested resource. 91 91
So in the example I had, 92 92
it was the upload.php file 93 93
and the action was post. 94 94
So we were sending data to the upload.php file 95 95
which is a script 96 96
that would then be run on that server. 97 97
Next we have the put method 98 98
and put is going to create 99 99
or replace the requested resource. 100 100
So if I put a file on the server, 101 101
I'm going to upload it there 102 102
and overwrite what's existing. 103 103
The next one we have is delete. 104 104
This is used to remove the required resource. 105 105
So if I wanted to delete a file, 106 106
I could use the delete command. 107 107
Then we have head. 108 108
Head is going to retrieve the headers for the resource only 109 109
and it's going to ignore the body. 110 110
So if I send out the head command to the server, 111 111
I'm only going to get the head portion of the HTTP request. 112 112
This is often used 113 113
as a penetration tester 114 114
when you're doing a banner grab 115 115
because all you want to know is information 116 116
about the server and about the page, 117 117
like the title of the page. 118 118
Now data can be submitted via a URL 119 119
and it's going to be deliminated 120 120
by the question mark character. 121 121
You saw earlier when I had my upload.php, 122 122
there was a question mark there 123 123
that then said post= 124 124
and that was saying, for this resource, 125 125
I want to do this next thing, I want to post. 126 126
And that's the idea here of submitting data via a URL 127 127
using that question mark character. 128 128
Now you also have query parameters 129 129
and query parameters are usually going to be formatted 130 130
as one or more name equals value pairs 131 131
and they're going to use an ampersand delineating each pair. 132 132
So if you think back to our Google hacking lesson, 133 133
we used the ampersand 134 134
to connect different value pairs together 135 135
as we sent them to Google to do a specific search. 136 136
Now the next thing we want to talk about is a hashtag 137 137
and a hashtag is used to indicate a fragment 138 138
or an anchor ID 139 139
and it's not actually processed by your web server. 140 140
If you've ever coded html, 141 141
you'll know that you can use a hashtag 142 142
to say this is a particular part of the page. 143 143
For instance, if a page is longer 144 144
than what can fit on the screen, 145 145
maybe you have a very long book that's in one webpage, 146 146
I can actually have a hashtag 147 147
to show where those anchor points are for each chapter. 148 148
And then when you click a link at the top of the page, 149 149
it would take you directly 150 150
to that portion of the page and jump down. 151 151
Again, it's not processed by the server, 152 152
but you should be on the lookout for these 153 153
as part of your URL analysis 154 154
because sometimes people will misuse that hashtag 155 155
and inject JavaScript into your page 156 156
by using that as part of the URL. 157 157
So let's look at the URL one more time. 158 158
Here you can see the post method is being used 159 159
like we talked about. 160 160
This is going to indicate that we're sending data 161 161
to the server for processing by the resource. 162 162
In this case, the upload.php script 163 163
which is sitting on diontraining.com. 164 164
Also, you'll see there's that question mark there. 165 165
That is being used to indicate 166 166
that data being submitted to that page 167 167
is going to use the post command 168 168
and its value is after the equal sign. 169 169
Now, all that stuff after the equal sign, 170 170
we're going to talk about that in a little bit 171 171
but we have some more stuff to cover first. 172 172
Let's go ahead and talk about HTTP response codes. 173 173
Now when you deal with an HTTP response code, 174 174
the header value is going to be returned 175 175
by the server 176 176
when the client requests a URL. 177 177
And these different codes are numerical codes, 178 178
usually three digits, 179 179
that are going to tell you some information 180 180
that the server wants you to know. 181 181
For example, if you get a code 200, 182 182
this indicates a successful get or post request, 183 183
meaning you went and got the page 184 184
and it came back to you successfully. 185 185
If you get a response code of 201, 186 186
this indicates that your put request has been successful 187 187
in creating a resource. 188 188
If you have a response code of 300, 189 189
anything in the 300 range, 190 190
this is going to indicate 191 191
a redirect has occurred by the server. 192 192
For example, there's a 301 redirect that we use 193 193
when we move a webpage 194 194
from our site to a different site. 195 195
So that way we tell Google that we want 196 196
all that information they know about that old page 197 197
to be transferred to the new page. 198 198
That's a 301 redirect. 199 199
You don't need to know all the codes for the 300 series 200 200
but you should know that 300 and anything is going to mean 201 201
that a redirect has occurred by your server. 202 202
The next series we have is the 400 series 203 203
and this indicates 204 204
that you're going to have an error 205 205
in the client request 206 206
and there's some specific ones here that you should know. 207 207
The 400 itself is going to indicate 208 208
that a request could not be parsed by the server. 209 209
If you have a 401, 210 210
this indicates the request 211 211
did not supply the authentication credentials. 212 212
If you have 403, this indicates 213 213
that the request did not have sufficient permissions 214 214
to access that thing. 215 215
If you have a 404, this is very common. 216 216
It indicates 217 217
that the client has requested a non-existent resource. 218 218
The next series we want to talk about is the 500 series 219 219
and a 500 series has any code in that area 220 220
means that there is some kind of a server side issue. 221 221
Everything up to this point 222 222
was really a client not getting what they wanted. 223 223
Now we have some server side issues. 224 224
A 500 is going to indicate a general error 225 225
on the server side of the application. 226 226
A 502 indicates a bad gateway, 227 227
and you'll often see this 228 228
when the server is acting as a proxy. 229 229
A 503 is going to indicate 230 230
there's an overloading in the server 231 231
and that's going to be causing service unavailability. 232 232
And a 504 indicates 233 233
that a gateway timeout has occurred 234 234
and this means there's some kind of an issue 235 235
with the upstream server. 236 236
Now for the exam, 237 237
you do not have to memorize all of these error codes 238 238
but you should be aware of them in your role 239 239
as a cybersecurity analyst 240 240
inside the real world 241 241
because you're going to be dealing with these, 242 242
especially if you're dealing with web applications 243 243
and web application testing. 244 244
Now, let's go ahead 245 245
and look at our URL one more time here. 246 246
Now, does knowing those codes help 247 247
with identifying anything in this URL? 248 248
Well, not really, 249 249
but if we tried opening this URL on your web browser, 250 250
you would get one of these codes back. 251 251
You would get a 404 error. 252 252
Now what does that mean? 253 253
It means the page doesn't exist. 254 254
Why is that? 255 255
Well, it's because the requested resource 256 256
which was the upload.php, can't be found. 257 257
Now why can't it be found? 258 258
Well, because I made up this URL. 259 259
We don't actually have a page 260 260
on our website called upload.php on our server 261 261
but there's still a lot in this URL 262 262
that we don't know. 263 263
Everything after the equal sign 264 264
we haven't even looked at yet. 265 265
And that's going to bring us 266 266
to the concept of percent encoding. 267 267
Now, percent encoding is a mechanism 268 268
to encode eight bit characters 269 269
that have a specific meaning 270 270
in the context of a URL. 271 271
This is also known as URL encoding. 272 272
Now, a URL can contain unreserved 273 273
and reserved characters from the ASCII set. 274 274
Unreserved characters are things like letters and numbers, 275 275
A through Z, zero through nine. 276 276
It's the dash, the period, 277 277
the underscore, and the tilde. 278 278
Now we have some reserved characters as well. 279 279
These are special characters like the colon, 280 280
the back slash, the question mark, 281 281
the hashtag, the brackets, the at sign, 282 282
exclamation, dollar sign, ampersand, 283 283
single quote, parenthesis, 284 284
star, plus, comma, semicolon and equals. 285 285
All of these are reserved characters. 286 286
You can use these inside of a URL. 287 287
There's nothing wrong with using them 288 288
but they do have a special meaning. 289 289
For instance, we talked about the question mark already. 290 290
We've talked about the ampersand, 291 291
we've talked about the equal sign. 292 292
All of these are special characters 293 293
or reserved characters 294 294
that can be used inside your URL. 295 295
Now, a URL cannot contain any unsafe characters. 296 296
Now what's an unsafe character? 297 297
Well, these are things like null string termination, 298 298
carriage returns, line feed, 299 299
end of line, tab, space, 300 300
and things like the forward slash, 301 301
the less than sign, the greater than sign 302 302
and the curly brackets. 303 303
All of these are characters 304 304
that are considered unsafe 305 305
and cannot be used in a URL. 306 306
Now, percent encoding is going to allow a user agent 307 307
to submit any safe or unsafe character 308 308
or any binary data for that matter 309 309
to the server within a URL. 310 310
So if I want to send a space, 311 311
I can do that if I encode it 312 312
with one of these percent encodings. 313 313
Now here's where I need to bring up the warning. 314 314
Warning, warning, warning, 315 315
percent encoding can be misused 316 316
to obfuscate the nature of a URL. 317 317
This allows them to encode unreserved characters 318 318
and submit malicious input as a script 319 319
or binary or some other method like that 320 320
to perform directory traversals 321 321
and other bad things. 322 322
So when you see percent encoding, 323 323
you probably should think to yourself, 324 324
there's probably something being hidden here. 325 325
I need to look further into this. 326 326
As an analyst, whenever I see that 327 327
or on the exam, anytime you see that, 328 328
you need to have the hair 329 329
on the back of your neck standing up 330 330
and going I need to look further into this. 331 331
There's probably something here. 332 332
Now let's take a look at percent encoding. 333 333
And here on the screen, you're going to see a chart. 334 334
On the left, you'll see the characters 335 335
and on the right, you'll see the code. 336 336
Now, I'm not going to read them all to you 337 337
but you do have them here on the screen 338 338
and you do have them in your notes 339 339
in the PDF study guide. 340 340
Now, the thing I want to point out here is 341 341
that this is not a list of all of the percent encoding. 342 342
These are just some of the most common ones 343 343
that you should know and be familiar with. 344 344
There are many, many others. 345 345
When you're an analyst in the real world, 346 346
I do recommend 347 347
that you print out a PDF 348 348
that has the entire list of these 349 349
because you will use them in your URL analysis. 350 350
For the exam, you do not need to memorize all of these 351 351
but you do need to know a couple of them 352 352
and we'll talk more about that later. 353 353
Now, some really tricky attackers have started to do 354 354
what's called double encoding 355 355
and they can double encode the URL 356 356
by encoding that percent sign too. 357 357
So when you're looking at the URL, 358 358
and so you won't see something like %20 359 359
even though that thing will be percent encoded. 360 360
Now for the exam, when you see a URL, 361 361
if they want to talk about URL analysis 362 362
and they want to talk about percent encoding, 363 363
they will use the percent sign. 364 364
They won't do the double encode, 365 365
but I want you to be aware of that 366 366
because in the real world, 367 367
this is something you have to be aware of. 368 368
All right, one more time, 369 369
let's go back to our URL. 370 370
Here again, we have our URL, 371 371
diontraining.com/upload.php?post=. 372 372
We've talked about that. 373 373
This is the website we're going to, 374 374
the file on that website 375 375
and the action we're going to do, 376 376
we're going to post something. 377 377
Now, let's get into what we are going to post. 378 378
So if we look at that URL again, 379 379
we have %3Cscript%3E. 380 380
Now what is 3C and 3E in percent encoding? 381 381
Well, if we go back to our chart, 382 382
you would know 383 383
that %3C is the less than sign 384 384
and %3E is the greater than sign. 385 385
So essentially this is saying post equals bracket, 386 386
script bracket in HTML. 387 387
Now if you know anything about HTML, 388 388
you know that bracket script bracket 389 389
means we're calling a JavaScript. 390 390
And so that is something 391 391
that we need to think about 392 392
of what script is being called 393 393
and that may be something bad. 394 394
So let's go a little bit further into this. 395 395
The next area we have, %27http%3A%2F%2Fabc123.com, 396 396
%2Frat%2Ejs. 397 397
What does that all translate to? 398 398
Well %27 is a single quote mark, 399 399
%3A is a colon, 400 400
%2F is a backslash, and we have that twice. 401 401
And then percent 2E is a dot. 402 402
So if I was going to read that 403 403
without the percent encoding, 404 404
I would read that as single quote, 405 405
http://abc123.com/rat.js. 406 406
And so now you can see what this URL really is saying. 407 407
It's saying go to diontraining.com, 408 408
access the file upload.php and post 409 409
meaning upload send this file of script, 410 410
this website address with a link to rat.js. 411 411
So we're trying to send this file, 412 412
this JavaScript file to this site, 413 413
and if it was vulnerable, 414 414
we can then upload malicious code. 415 415
That's how we do this URL analysis. 416 416
And for the exam, you are going to be asked 417 417
to do some URL analysis. 418 418
Now let me give you an exam tip 419 419
for this area of the exam 420 420
when we talk about URL analysis. 421 421
You don't need to necessarily remember the values 422 422
for all those percent encodings 423 423
but you should know what it is 424 424
if you see percent encoding. 425 425
It is some kind of a URL analysis question 426 426
and you're probably going to be asked 427 427
to try to figure out what they're hiding in there. 428 428
Now, they're not going to be very specific 429 429
in the fact of asking you exactly what's going on, 430 430
but using this, 431 431
you should be able to see certain things. 432 432
For example, are they trying to do an SQL inject? 433 433
If so, they're probably using that single quote 434 434
and they're using an equal sign. 435 435
If they're trying to do something like XML, 436 436
you'll see some XML being passed. 437 437
If they're trying to load a JavaScript, 438 438
they're going to do something like 439 439
what I just showed you here in the URL. 440 440
These are the kind of things you'll have to do. 441 441
Now, what do you do 442 442
if you don't remember all those codes 443 443
because that's a lot of percent encoding, 444 444
all those different codes of remembering %3F 445 445
and %2E, and things like that. 446 446
Well, on the exam, 447 447
it's all multiple choice, right? 448 448
And so when you get a question, 449 449
you may not remember what %3C and %3E were 450 450
because that was the brackets 451 451
for the script on either side, 452 452
but you did see the word script. 453 453
And so if you see the word script, 454 454
you might think, hey, 455 455
they might be trying pass an HTML script 456 456
or a JavaScript there. 457 457
And as you saw the rest of it, 458 458
you could read things like abc123.com, 459 459
which is a web link. 460 460
And so you're seeing that you're going to Dion Training 461 461
but now you're sending another web link 462 462
as part of this URL 463 463
and that should flag to you 464 464
that there's something wrong here. 465 465
So even if you don't remember all of the colons 466 466
and the back slashes and the dots, 467 467
you can probably put together most of this 468 468
based on what you're reading in that URL 469 469
as you're doing your analysis for the exam. 470 470
In the real world though, 471 471
I do recommend having one of these charts handy 472 472
because then you can read exactly what it is 473 473
and really put it back together.