Instructor: Application logs. 2 2
In this lesson we're going to talk about finding 3 3
indicators of compromise inside your application logs. 4 4
Now remember, the simple idea of unexpected growth of a log 5 5
could itself be an indicator of compromise 6 6
but in addition to that, there is contents inside these logs 7 7
that we can use as signatures 8 8
as we're searching for indicators or compromise in a system. 9 9
Now most applications can be configured 10 10
to log different events. 11 11
As we go through this lesson 12 12
we're going to start out by covering four different log types. 13 13
These are DNS event logs, HTTP access logs, FTP access logs 14 14
and SQL event logs. 15 15
Let's start with DNS event logs. 16 16
First, what is a DNS event log? 17 17
Well, this is a log that contains a log 18 18
of all the different events for each time 19 19
the DNS server handles a request to convert 20 20
between a domain name and an IP address. 21 21
So if I pull that up on a Window system, 22 22
it looks something like this. 23 23
Notice this is an information level event 24 24
and it's showing it is a DNS client event. 25 25
As I look at the details, I can see that a DNS query 26 26
was called for this particular domain name, 27 27
the type of query it was, the options it was given, 28 28
and all the other information about this request. 29 29
I also know what user did this and which computer did this. 30 30
This is helpful information as we're trying to figure out 31 31
where malicious information is 32 32
on our different systems within our network. 33 33
The next type of log we have is an HTTP access log, 34 34
and this is a log containing HTTP traffic 35 35
that encountered an error 36 36
or traffic that matches some predefined rule set. 37 37
So as we start looking through that information 38 38
we can find relevant information in that log. 39 39
And this information is recorded 40 40
in the common log format, CLF 41 41
or W3C extended log file format. 42 42
There is no standard of which one of these 43 43
it's going to use. 44 44
That is going to depend on the HTTP server that you're running 45 45
but it will usually be one of these two formats. 46 46
Now, as you look at it, 47 47
you're going to get all these different status codes in there 48 48
and we've talked before about the HTTP status codes. 49 49
These status codes of responses 50 50
will indicate if there was an error 51 51
and whether it was caused by the client or the server. 52 52
If it's a client-based error code, 53 53
it's going to be something in the 400 range. 54 54
If it's a server-based error code, 55 55
it's going to be something in the 500 range. 56 56
Now, some web server software logs 57 57
are going to have the HTTP header information 58 58
for both the requests and the responses. 59 59
And this is really helpful for us 60 60
because we can then get even more information 61 61
that we can look through it. 62 62
Now, as you're looking through these logs, 63 63
one of the fields you're going to see is a user agent field. 64 64
This identifies the type of application 65 65
that's making the request, such as the web browser version 66 66
or the client's operating system. 67 67
Now, a quick word of warning here for you, 68 68
the user agent field is very helpful 69 69
but it is not a reliable indicator 70 70
of the client's environment. 71 71
It can be easily spoofed or simply misreported 72 72
because a lot of browsers use the same underlying code. 73 73
So let's take a look at a couple of examples here. 74 74
The first example of an HTTP access log looks like this. 75 75
Here you can see three entries on the screen 76 76
and we're going to work through them together 77 77
so we can understand what's going on. 78 78
The first entry has 10.1.0.102. 79 79
In this first entry, we see an unknown user. 80 80
Notice the second dash after the IP address 81 81
indicates this is an unknown user. 82 82
And then we can see this unknown user 83 83
is trying to receive a file from the downloads directory 84 84
which is why the command get was being sent. 85 85
And then we get a response code of 401, 86 86
which indicates that the user is being challenged 87 87
to provide a username and password by the server. 88 88
Then we're going to get some information 89 89
about the client browser and the operating system. 90 90
In this case, it's identified 91 91
as a Windows 64 bit operating system 92 92
running Windows NT 10.0 93 93
which is just some version of Windows 10, 94 94
like Windows 10 Home, windows 10 Pro, Windows 10 Education 95 95
or Windows 10 Enterprise. 96 96
Now, from this line alone, we're not sure which one 97 97
but we do know it's some version of Windows 10. 98 98
Now, once we see that 99 99
we can also see the browser's been identified here 100 100
as either Chrome, Safari, or Edge 101 101
because they all share a common code base 102 102
and it makes it really hard to identify positively 103 103
which of the three it really is. 104 104
The next entry we have 105 105
we can see that the user was identified as jason. 106 106
This is because the user has provided 107 107
their username and password 108 108
in response to the status code 401 from the first entry. 109 109
Here we see a status code of 200 110 110
which indicates that the resource was successfully accessed 111 111
and this case that means the file requested 112 112
in the first entry was now downloaded. 113 113
Then we get to our third entry. 114 114
This third entry, we can see another user trying. 115 115
And again, this is an unknown user 116 116
by showing that second dash there after the IP address. 117 117
Again this user is trying to download that same file 118 118
from the server, but from a different computer. 119 119
This time, from the 10.1.0.103 IP. 120 120
Notice we don't see another entry 121 121
with the proper credentials being accepted 122 122
with a status code 200. 123 123
So we can assume that this unknown user 124 124
has provided the incorrect credentials 125 125
and received an error message instead 126 126
and they weren't able to download that file. 127 127
Now again, this assumes that these were 128 128
the only three entries we had, 129 129
but if we went further in the logs 130 130
we may see that status code 200 131 131
and a successful attempt later on. 132 132
All right, so let's go ahead 133 133
and take a look at a second example. 134 134
Here we're going to go into another HTTP access log example 135 135
and here I have a single entry for you. 136 136
In this log entry, 137 137
we can see something suspicious is going on. 138 138
Notice this time it's from an unknown user 139 139
and we have a get command being issued, 140 140
but inside that query we can see 141 141
that there is that question mark, and then variable equals. 142 142
This is trying to send that variable 143 143
that's listed here on the screen. 144 144
Now, what is this variable? 145 145
Well, it's really just a snippet of JavaScript 146 146
and what we're seeing here is that we can see this snippet 147 147
being sent by the user agent of Nikto, 148 148
which is a web application vulnerability scanner. 149 149
This entry comes from a vulnerability scan 150 150
against our web server, 151 151
and this scan is actually running the test number 000816 152 152
which is attempting to see if this web server 153 153
is vulnerable to a code injection attack 154 154
through a query string. 155 155
Also, we see status code 302 was returned 156 156
inside this entry. 157 157
This is used to indicate that the server 158 158
found the requested object, 159 159
but it redirected the browser to a login page 160 160
if it wants you to be able to access that. 161 161
For the exam you need to be able to read 162 162
and understand these types of logs 163 163
and then identify if something is suspicious or not. 164 164
Notice in the second example, this is suspicious 165 165
because we see someone trying to perform a code injection 166 166
against your web application. 167 167
Now, if this wasn't your security team, 168 168
how would you be able to mitigate this type of attack? 169 169
Well, you could install a web application firewall or a WAF 170 170
because this will help you analyze the inbound request 171 171
and strip out things like code injection 172 172
to be able to prevent this type of attack. 173 173
For the exam this is the level of detail you need 174 174
to be able to see. 175 175
They were trying to inject something and you can stop that 176 176
by doing what thing, what is your mitigation? 177 177
If you can know those two things, 178 178
you're going to do well on the exam. 179 179
Let's move on to our FTP access logs. 180 180
Now for FTP access logs, these are logs 181 181
containing FTP traffic in a W3C extended log format. 182 182
This might look something like this. 183 183
Here you can see a couple of different codes 184 184
that we have. 185 185
You see 331 and 530 and 230 for example. 186 186
Now, 530 indicates that a user needs to log in 187 187
before a command is processed. 188 188
So in this case we have a username 189 189
and a password being submitted. 190 190
Then you see further down we have code 230. 191 191
Code 230 indicates that a user 192 192
has successfully logged in with their credentials 193 193
and has access to perform the command. 194 194
In this case, they're probably trying to download a file 195 195
because it says the file transfer protocol. 196 196
Now if we look in this example, 197 197
you can see five requests for credentials 198 198
like a username and password 199 199
and only one of those was accepted at the end. 200 200
So what does this mean? 201 201
Are we under attack? 202 202
Well, maybe. 203 203
Maybe we're under attack from somebody attempting 204 204
a brute force attempt to crack our password 205 205
or maybe somebody just forgot their password 206 206
and is trying their top five most common passwords 207 207
until they found the one that was right. 208 208
We really can't be sure here, 209 209
but we do know that they are trying to log in 210 210
four times unsuccessfully 211 211
before being successful on that fifth attempt. 212 212
So if you were worried about a brute force password attack, 213 213
how could you mitigate it? 214 214
Well, you could set up a global policy 215 215
that added a 15 minute or 30 minute lockout 216 216
for any account after three failed login attempts. 217 217
This is a common mitigation against this type of attack 218 218
because it extends the amount of time needed 219 219
to successfully crack a password by an attacker. 220 220
And this is all you're trying to do here 221 221
is delay the timeout, make it take longer for the attacker 222 222
'cause if you can do that, hopefully they'll give up 223 223
and they'll go to an easier target. 224 224
Our next type of log is an SSH access log 225 225
and this is an unstandardized type of log 226 226
that can provide basic client server session information. 227 227
Now, when you're dealing with SSH, this is secure shell. 228 228
Here on the screen is an example of an SSH log. 229 229
This is showing somebody 230 230
who successfully is logging in as root. 231 231
They went in and they gave their password 232 232
and it was accepted, and so a session was open. 233 233
How do we know it was root? Notice the UID. 234 234
UID equals zero. 235 235
The root of a Linux system 236 236
is always going to have user ID of zero, so keep that in mind. 237 237
Now, let's say we had a bunch of failed logins. 238 238
What will we do to be able to identify 239 239
all the failed root logins on a server 240 240
as part of our threat hunting or our analysis? 241 241
Well, we could do something like this. 242 242
We can go through here and use egrep, 243 243
which is a grep command to search the authentication log 244 244
at the /var/log/auth.log. 245 245
And when we do that 246 246
we can display any instances of the word failed or failure. 247 247
Here, for example, you can see 15 instances on December 5th 248 248
from 9:39 to 9:40 at night when somebody tried to log in 249 249
and they failed to do that using the SSH service. 250 250
Now again, for the exam 251 251
you need to be able to read logs like these 252 252
and identify what type of attack might be going on. 253 253
In this case, it's probably some kind of password guessing 254 254
since we have repeated login attempts that are all failing. 255 255
Also, notice in this log you can see exactly 256 256
what IP was attempting these connection requests. 257 257
This way you could use this as part of your analysis 258 258
and see if that client machine may have been compromised. 259 259
That device may be something that was compromised 260 260
by an attacker and now they're using it 261 261
and trying to pivot onto another machine 262 262
using the root credentials. 263 263
This is a way to identify what actions have been taken 264 264
and what possible root cause there is. 265 265
The fourth type of log we need to talk 266 266
about is an SQL event log. 267 267
This is an event or error log 268 268
that records events with fields like date, time 269 269
and the action taken, such as server startup, 270 270
individual database startup, database cache clearing 271 271
and databases not starting or shutting down unexpectedly. 272 272
Now, when we're dealing with this 273 273
we are dealing with SQL servers. 274 274
And SQL servers can log individual query strings 275 275
that are being sent to the database 276 276
so we can actually look at everything that's been sent 277 277
to the database by a particular user. 278 278
Now, as you go through 279 279
and start looking at your SQL server logs, 280 280
you're going to find a lot of information there 281 281
other than the date time and the user who sent the query. 282 282
For example, you might see things 283 283
like the query operation that was performed, 284 284
the schema associated with that operation 285 285
and the object of the query. 286 286
By retrieving all this information 287 287
you're going to be able to get information 288 288
from the database on what has occurred 289 289
and identify what data may have been breached 290 290
if you're investigating a data breach. 291 291
The particular format of those logs 292 292
and how you access them is going to vary based 293 293
on the type of SQL database you're running. 294 294
For example, if you're using MySQL, 295 295
you can go into the MySQL workbench 296 296
and use your audit inspector to go through those audit logs. 297 297
Notice here you'll see things like the record ID, 298 298
the timestamp, the type which is a query or a ping, 299 299
the connection ID, the user, the host IP, the status, 300 300
the command class that was being used 301 301
and any information about that particular command.