Instructor: Searching and Piping Commands. 2 2
In the last lesson 3 3
we started talking about SIEM correlation rules 4 4
and we talked about SIEM queries. 5 5
Now, I mentioned the idea 6 6
of searching for some kind of information, 7 7
and we left it very generic, 8 8
but in this lesson we're going to dive 9 9
much deeper into that concept. 10 10
Now, when you create a SIEM correlation rule, 11 11
this is usually going to involve 12 12
searching with some kind of a string. 13 13
Now, to do this more effectively, 14 14
we can use regular expressions. 15 15
Now, regular expressions or regex, as you'll hear it called, 16 16
is a group of characters 17 17
that describe how to execute a specific search pattern 18 18
on a given set of text. 19 19
Now, regex is really important to learn 20 20
as a good cybersecurity analyst, 21 21
because if you can use regex efficiently, 22 22
you can search through mountains of log data 23 23
and find what you're looking for, 24 24
whether that's a threat, an incident, or an attack. 25 25
But for the CySA Plus exam, 26 26
you just need to know the basics. 27 27
They don't expect you to be an expert, 28 28
but you may get a question 29 29
that asks you to look at regular expression 30 30
and figure out what the response would be, 31 31
coming from that tool. 32 32
So we're going to cover the basics here 33 33
and if you want to learn more, 34 34
I'm going to give you a handy website to go 35 35
and practice this concept and learn a lot more about regex, 36 36
because again, in the real world, 37 37
it's really helpful to know this stuff. 38 38
Now, the first commonly used element 39 39
inside a regex syntax is the brackets. 40 40
Now, I have it here as bracket dot dot dot bracket, 41 41
but really, it's what we put 42 42
in the brackets that's important. 43 43
When we look at the brackets, 44 44
this is going to match a single instance 45 45
of a character within those brackets. 46 46
So for example, if I use bracket A dash Z, 47 47
that's going to give me a lowercase letter. 48 48
If I have capital A dash Z, it'll gimme an uppercase letter. 49 49
Zero through nine gives me a number. 50 50
If I have something written like A dash Z, 51 51
capital A dash capital Z, zero dash nine, 52 52
it's going to find me an uppercase, lowercase, or a number 53 53
as a single digit or single character from that range. 54 54
If I want to use a white space, I'm going to use slash S. 55 55
If I want to use a digit, it's slash D. 56 56
That would give me any single digit. 57 57
These are some of the examples of what you can search for 58 58
when you're using the brackets. 59 59
The next thing we can use is a plus sign. 60 60
And a plus sign is going to match one or more occurrence 61 61
and this is also known as a quantifier. 62 62
Now, for example, if I use slash D, that's a single digit, 63 63
but if I use a plus, that's going to match one or more digits 64 64
because I have the plus after the slash D. 65 65
Next, I have a star, 66 66
and it's going to match zero or more occurrences. 67 67
So if I have slash D star, 68 68
this matches zero digits or any number of digits 69 69
that I might find in the search string. 70 70
The difference between the star and the plus 71 71
is that the star starts at zero 72 72
where the plus starts at one. 73 73
And next, we have a question mark. 74 74
A question mark is another type of quantifier. 75 75
It's going to match one or none 76 76
of the times that defines that thing. 77 77
So in a slash D question mark scenario, 78 78
I'm going to match zero or one digits. 79 79
Next, I have my curly braces. 80 80
And usually you'll see a number written inside of them. 81 81
This is going to match the number of times 82 82
within the curly braces. 83 83
For example, going back to my digit example, 84 84
slash D curly brace three and curly brace. 85 85
This says I want to match three digits, 86 86
and if I use something like slash D brace 87 87
seven dash 10 brace, 88 88
that would match anything from seven to 10 digits. 89 89
And so for instance, if I was looking for a phone number, 90 90
I know in the United States 91 91
those are seven digits or 10 digits, 92 92
I could use something that looks like this. 93 93
Next, I have parentheses. 94 94
And parentheses are going to define 95 95
a matching group with a regex sequence 96 96
placed between those parentheses. 97 97
And then each group can subsequently 98 98
be referred to by slash one for the first group, 99 99
slash two for the second group, and so on. 100 100
This way I can set up lots of different groups 101 101
and be able to find any of those matching groups. 102 102
The next one we have is the OR logical operator, 103 103
which looks like the pipe character. 104 104
This is going to be used to match conditions 105 105
such as this or that. 106 106
The next one we have is the caret, 107 107
and this is an anchor or boundary character. 108 108
This is going to say only return the matches 109 109
that start at the beginning of a line 110 110
when you're searching with regex. 111 111
Now, this is as opposed to the dollar sign. 112 112
The dollar sign is the opposite. 113 113
It's only going to match 114 114
at the end of a line when you're searching. 115 115
And so you can use the caret for the beginning 116 116
and the dollar sign for the end. 117 117
This way you can find the beginning or end of any line. 118 118
Now, as I said, this was just 119 119
a very brief introduction to regex, 120 120
but if you want to learn more about regular expressions 121 121
and how to build them 122 122
and practice building them with an interactive tool, 123 123
I highly recommend you go to regexr.com. 124 124
This is a free site and you can go there 125 125
and start playing around with this 126 126
and getting better using the hands-on tools there 127 127
because it really will help you in the field. 128 128
The next thing we want to talk about 129 129
is a couple of commands that rely on regex. 130 130
The first one we want to talk about is grep. 131 131
Grep is a command on Unix, Linux and macOS systems 132 132
that involves simple string matching or regex syntax 133 133
to search text files for a specific string. 134 134
Now, as we look at grep, 135 135
let's take a look at some examples here. 136 136
First, grep dash F, 192.168.1.10 access.log. 137 137
What is this telling me? 138 138
Well, grep is the command. 139 139
The dash F says, I want to do a simple search. 140 140
I want to treat the next thing 141 141
as a literal that I want to search. 142 142
And that thing is an IP address, 192.168.1.10. 143 143
And then where do I want to search it? 144 144
Inside this file called access.log. 145 145
That's a very simple way to use grep. 146 146
Next we have grep, quote 192.168.1.10 quote star. 147 147
Now, the fact that we have grep there 148 148
tells us we're going to grep something. 149 149
What's inside the quotes is the literal. 150 150
So instead of using dash F, 151 151
we can replace that with the quote marks. 152 152
And then the star says, 153 153
I'm going to search all files for this particular IP address. 154 154
Here's another example. 155 155
Grep slash R 192 slash dot 168 slash dot one slash dot 156 156
bracket slash D bracket curly brace one comma three 157 157
curly brace dot. 158 158
Wow, that's a long one. 159 159
It looks pretty complex, right? 160 160
But if we take it step by step, it's really not that hard. 161 161
So first in this line, we have grep, 162 162
which says, we're going to issue the grep command. 163 163
Then we have dash R. 164 164
And dash R means recursion. 165 165
We are going to look inside this file or directory 166 166
and all of the subsequent files 167 167
and directories underneath it. 168 168
So if the command is giving it a directory, 169 169
in this case, we are with the dot means present directory, 170 170
I'm going to look for any files in the present directory 171 171
or in it's sub directories, 172 172
and I'm going to search for them using this regular expression. 173 173
Now, the reason I know this is a regular expression 174 174
is because I don't have dash F and I don't have quote marks. 175 175
So this is a regular expression. 176 176
Now, how do I read this regular expression? 177 177
Well, 192 is a number, so I'll just read it as 192. 178 178
The slash is an escape character, 179 179
and that means the next thing that's being shown 180 180
should be treated as a literal character. 181 181
So we have to put that slash there 182 182
because otherwise the dot has it's own meaning inside regex. 183 183
And so we want to make a slash, 184 184
so it treats it as a dot that we use in an IP address, 185 185
and not as the special character dot 186 186
that might be used in regex. 187 187
Then we have 168. 188 188
Again, a literal number. 189 189
Slash is the escape character. 190 190
Again, we're showing the dot. 191 191
One slash dot. 192 192
And then we get to this other part. 193 193
Now, bracket slash D says, I want any single digit. 194 194
Then I have the curly braces. 195 195
If you remember from earlier in this lesson, 196 196
it says, I want a number that has this many digits. 197 197
So the fact that we have one comma three says, 198 198
show me anything that has one digit, 199 199
two digits or three digits. 200 200
And that is what's being shown here 201 201
inside this grep expression. 202 202
So what I'm searching for is 192.168.1 dot something. 203 203
Any digit that is either one, 204 204
two, or three digits in length. 205 205
That's what we're looking for here. 206 206
Let's take a look at another one. 207 207
This one is grep dash R. 208 208
So again, recursion. 209 209
Again, the target is dot, 210 210
which is the present working directory. 211 211
And then you'll see 192 escape character dot 212 212
168 escape character dot 213 213
one escape character dot, 214 214
and then I have these bracketed of zero to 255. 215 215
What am I looking for here? 216 216
Here, I'm looking for a specific IP address, 217 217
a valid IP address. 218 218
Now, in the one before that I just used, the slash D one, 219 219
I looked for any one, two, or three digit number. 220 220
So I could have had 192.168.1.999. 221 221
Well, 999 is not a valid IP address. 222 222
So instead, by using this zero dash 255, 223 223
I'm saying show me all the numbers from zero to 255, 224 224
which are all the numbers that could exist as an IP address 225 225
because anything from 256 and beyond 226 226
is not going to be a valid IP address. 227 227
So this is just another way to search 228 228
for an IP address by using that syntax. 229 229
All right. 230 230
So the last little bit we need to talk about here with grep 231 231
is some of the command line options. 232 232
You'll notice I threw in that dash R for recursion, 233 233
but there's other ones too. 234 234
For instance, you have Dash I, 235 235
which says ignore case sensitivity. 236 236
By default, grep is going to be case sensitive. 237 237
So if I have uppercase password versus lowercase password, 238 238
those are two different names, 239 239
but by using Dash I, 240 240
I will find both of those regardless of the case. 241 241
The next one is dash V. 242 242
Dash V is going to reverse grep's default behavior. 243 243
So instead of finding me things that match, 244 244
it's going to find me everything that doesn't match. 245 245
This is really helpful, for instance, 246 246
if I wanted to look at somebody's machine 247 247
and find all the cases where somebody logged in 248 248
who wasn't the person who's assigned to that computer. 249 249
For instance, I'm the only person who logs into my computer. 250 250
So if anybody logs in and they're not Jason, 251 251
I want to know about it 252 252
and I can run dash V with a grep search 253 253
to find anyone who is not Jason. 254 254
Next we have dash W, 255 255
and dash W is going to treat your search strings as words. 256 256
Now, by default, when you search for something, 257 257
for instance, let's say 258 258
I was searching for the word no, N-O. 259 259
Well, it's going to find all the instances of the word no, 260 260
but it's also going to find the word nano and nowhere 261 261
and anything else that has N-O inside of it. 262 262
And I might not want that. 263 263
I might only want to find the word no. 264 264
So by using dash W in front, 265 265
I'm only going to find the word N-O 266 266
where N-O was by itself, no. 267 267
Next, we have dash C, 268 268
which is going to return 269 269
the total count of matching strings only. 270 270
So if I'm searching for the word no, 271 271
in this case, I'm not going to get the words no back, 272 272
I'm only going to get the number of times 273 273
it found the word no inside that file. 274 274
So for instance, if I wanted to look 275 275
for a particular IP address 276 276
and find out how many times 277 277
it called out to a particular server, 278 278
I can do that by searching for that IP address 279 279
and then finding all those instances. 280 280
Next, I have dash L and this is a lowercase L. 281 281
This is going to return the names of files with matching lines. 282 282
So if I'm going to search multiple different files, 283 283
for instance, I use the search 284 284
of the present working directory with the recursion, 285 285
I'm going to have lots of files I'm looking at. 286 286
Well, instead of finding the lines 287 287
that have whatever I'm searching for, 288 288
I can just find the files that have that. 289 289
So for example, let's say I was searching 290 290
for something that is a password 291 291
that I know has been compromised. 292 292
I can search all the files on a server 293 293
for that particular password string, 294 294
and it will return to me all those file names 295 295
that have that password. 296 296
And so now I can go in and figure out 297 297
who created those files and who gave away our passwords. 298 298
And the last one we have is dash L. 299 299
This one is a capital L this time. 300 300
This is going to return the names of the files 301 301
without matching the lines. 302 302
In this case, it's going to work like the dash V option 303 303
and give you just the names of the files 304 304
that contain the non-matching lines. 305 305
So if I'm looking for all the files 306 306
that don't contain this thing, 307 307
it's going to go ahead and return those for me. 308 308
Now, grep is an awesome tool, 309 309
but you're only going to find it on Unix, Linux and macOS. 310 310
You don't find it on Windows by default. 311 311
Instead for Windows, we have two commands. 312 312
We have the find command for basic strings, 313 313
and findstr for regex searching. 314 314
Both of these will work a lot like grep. 315 315
Now, the next command we want to talk about is cut. 316 316
And cut is a command that enables the user to specify 317 317
which text on a line they want removed from the results. 318 318
This makes it easier for you to read. 319 319
So for example, if I have have cut dash c5 syslog.txt, 320 320
this is going to return only the fifth character in each line 321 321
from the syslog.txt file. 322 322
Now, I know that sounds kind of weird, right? 323 323
But if you think about it, 324 324
all our logs come in a standard format usually, 325 325
based on whatever program you're using. 326 326
So maybe the fifth character inside of a log file 327 327
might tell you if it was an inbound for I 328 328
or outbound O as the character 329 329
to tell you which direction traffic went. 330 330
And you just want to get a quick display 331 331
of all the inbound and outbound traffic, 332 332
so you could see how many things were in 333 333
and how many were out. 334 334
It's a silly example, 335 335
but that would work by using this command. 336 336
Now, in addition to this, 337 337
you can actually modify this command just a little bit 338 338
by adding another number. 339 339
For instance, if I did cut dash c5 dash five syslog.txt, 340 340
this is going to return only the fifth through 10th characters 341 341
in each line in the syslog.txt file. 342 342
This would allow you maybe to just get 343 343
a column of information that you wanted, 344 344
like usernames or passwords or IP addresses 345 345
or whatever it was that you needed 346 346
and you knew exactly where it was in the file, 347 347
how many characters in 348 348
and how many characters that thing was going to be. 349 349
Another way you can use cut is like this. 350 350
Cut dash D quote space quote dash f1 dash four syslog.txt. 351 351
Now, this is going to return 352 352
the first four entries of each line 353 353
as delimited by the space character. 354 354
So if you have a format like that, 355 355
you can then say, give me the first four pieces on each line 356 356
as you get the first four words separated by spaces. 357 357
The next command we're going to talk about is sort, 358 358
and sort is a command that can be used to change the order 359 359
of something is going to be outputted in. 360 360
So for instance, if I had a text file, 361 361
and I wanted to sort it in different ways, 362 362
I can do that using the sort command. 363 363
For instance, if I use sort syslog.txt, 364 364
this will give me the contents of the syslog.txt file 365 365
in alphabetical order, going from A down to Z. 366 366
Now, if I wanted to do it backwards, 367 367
I could use sort dash R syslog.txt. 368 368
This returns the contents of the syslog file 369 369
in reverse alphabetic order, going from Z down to A. 370 370
Now, if I want to do this with numbers, 371 371
I can do that using dash N. 372 372
Sort dash N syslog.txt 373 373
will return the contents of the syslog.txt file 374 374
in numerical order from zero to nine. 375 375
Now, another way I can sort things is based on columns. 376 376
If I use sort dash K and then a number like dash K two, 377 377
this will allow me to sort that file 378 378
and return the contents of the screen 379 379
based on the column specified. 380 380
In this case, the second column. 381 381
Now, another way I can use sort 382 382
is by using the dash T command. 383 383
So in this case, I can use sort dash T 384 384
quote comma quote dash K two syslog.txt. 385 385
And this is going to return the contents of the syslog file 386 386
in order based on the column specified. 387 387
In this case, the second column. 388 388
Now, how does it determine a column? 389 389
By default, it uses spaces, but because I used dash T, 390 390
I said I want you to use these columns based on commas. 391 391
So this might be a CSV or comma separated value file, 392 392
and that's why I'm going to go through it that way. 393 393
Again, there's lots more to learn about cut and sort, 394 394
and if you use the man pages on your Linux system, 395 395
you can learn all about them. 396 396
For the CySA Plus exam though, 397 397
you should just know the basics 398 398
of what cut is and what sort is. 399 399
The next command we want to talk about is head, 400 400
and head is going to be a command 401 401
that gives you the first 10 lines of any file specified. 402 402
For example, if I did head syslog.txt, 403 403
the first 10 lines of that file 404 404
are going to be displayed to my screen. 405 405
Now, this isn't nearly as useful 406 406
when you're dealing with logs though, 407 407
because logs tend to write things 408 408
at the bottom of the file with the most current stuff. 409 409
So in that case, we would use tail. 410 410
Tail is going to give you the last 10 lines of a file. 411 411
So again, if I use tail syslog.txt, 412 412
you're going to get the last 10 lines of that file. 413 413
This is really useful when dealing with logs, 414 414
because you're always going to get 415 415
the 10 most recent log entries in that particular file. 416 416
The last thing we're going to talk about here is piping, 417 417
and piping is a character on your keyboard 418 418
which is right over your return or enter key. 419 419
If you hit the shift button over the backslash, 420 420
this will give you that character. 421 421
This basically looks like a straight up and down line. 422 422
Now when you use piping, this is the process 423 423
of using the output of one command 424 424
as the input for a second or third or fourth command, 425 425
and you can pipe lots of things together. 426 426
Now, when we deal with piping, it looks like this. 427 427
For instance, I had this command, 428 428
grep "NetworkManager" slash var slash log slash syslog 429 429
pipe cut dash D quote space quote 430 430
dash f1 dash five pipe sort dash T 431 431
quote space quote dash K three. 432 432
Now, what is this telling me to do? 433 433
Well, we're always going to read it from left to right, 434 434
and on a Linux system, this would all be on one line. 435 435
I just put it on two lines here 436 436
so we could read it a little bit easier. 437 437
Now, we're going to start out with grep. 438 438
What are we doing? 439 439
We are going to search for the word, 440 440
the literal word NetworkManager 441 441
inside the file var slash log slash syslog. 442 442
So anytime I find NetworkManager in that syslog file, 443 443
I'm going to then pipe it and send it to the next command. 444 444
So I've gone through and I've done the grep 445 445
and I found all that information. 446 446
I now, instead of putting it onto the screen, 447 447
have passed it to cut, 448 448
and cut is going to do something with it. 449 449
It's going to use Dash D quote space quote, 450 450
which basically says I'm going to delimitate based on spaces 451 451
and then I'm going to use dash f1 dash five. 452 452
Now, what is that dash f1 dash five? 453 453
Well, that is going to cut the characters 454 454
based on the delimiters. 455 455
So every time we have those delimiters, 456 456
we are going to see the first five columns 457 457
for each line inside of that syslog. 458 458
Then we are going to sort, 459 459
we're going to pass all that stuff 460 460
that we just cutted into sort, 461 461
and sort is then going to use dash T quote space quote, 462 462
which, again, is the delimiter, 463 463
we're doing this based on spaces, 464 464
and we're going to look for the third column. 465 465
So how we're sorting this file that we've now created 466 466
is based on that third column. 467 467
And now that we've done all three of those steps, 468 468
we are then going to push it to the screen and show it to you. 469 469
So at this point, 470 470
instead of just getting a big long list 471 471
of all the times we found NetworkManager inside of syslog, 472 472
we are now going to find 473 473
that we only have the first five columns 474 474
and they're all sorted by the third column inside of syslog. 475 475
That gives me something that's a lot more useful 476 476
as an analyst to go through.