Narrator: Data normalization. 2 2
Data normalization is really important to us, 3 3
especially when we're collecting everything 4 4
into a central repository. 5 5
When we collect everything into a central repository 6 6
like a SIEM, we have to realize that our security data 7 7
comes from numerous sources across our organization 8 8
and all of these use different formats 9 9
and different ways of storing data. 10 10
So what we have to do is we have to normalize that. 11 11
When we talk about normalization, 12 12
this is a process where the data is going to be reformatted 13 13
or restructured to facilitate the scanning 14 14
and analysis process later on during the intelligence cycle. 15 15
Now, when you start thinking about your SIEM data 16 16
where does this data come from? 17 17
Well, it can come from three different places 18 18
across our network. 19 19
For instance, here on the screen, you can see 20 20
all sorts of different places that data is coming from. 21 21
We have it coming from a HIDS or an EDR. 22 22
We have it coming from agents. 23 23
We have it coming over the syslog protocol. 24 24
We have it coming from span ports and tap ports. 25 25
And we aggregate all that information into a SIEM. 26 26
Now, we're going to break that down 27 27
as we go through this lesson. 28 28
First, we have agent-based information. 29 29
Now, agent-based collection comes from systems 30 30
like host-based intrusion detection systems, or EDRs. 31 31
It comes from agents that are actually installed 32 32
on a host or on a server, 33 33
and all that data can then be sent back to a SIEM. 34 34
When dealing with agent-based collection, 35 35
we have an agent service that's installed on each host 36 36
or server to log, filter, aggregate, and normalize the data 37 37
on that host or server before we send it up 38 38
to the SIEM server for analysis and storage. 39 39
Now, the second way we can deal with things 40 40
is when we deal with a listener or a collector. 41 41
When we have a listener or collector, 42 42
this is going to send information over the syslog protocol. 43 43
And as you can see here, 44 44
we have syslog being used inside our extranet and our VPN 45 45
to send that data back to the SIEM server. 46 46
Now, as we look at this, a listener/collector 47 47
is going to be a host that's configured to push updates 48 48
to the SIEM server using a protocol like syslog or SNMP. 49 49
It can use either of these protocols or another protocol. 50 50
Either way, it's still considered a listener and collector. 51 51
In my diagram I just showed you, we were using syslog. 52 52
Now, the next one we have is sensors. 53 53
And here you see I have those TAPs 54 54
and the SPAN ports across the switch. 55 55
That's going to allow me to collect network data. 56 56
Now, when I'm dealing with the sensor, 57 57
this is going to allow my SIEM to collect packet capture 58 58
and traffic flow data from different sniffers 59 59
and sensors that are positioned across your network. 60 60
And by doing this, we get a good mixture 61 61
across all of our different devices because now we have 62 62
all of these things working together 63 63
pulling data from hosts, from network equipment, 64 64
from packet capture, from servers, 65 65
and it brings all that data 66 66
to the centralized server where it's aggregated. 67 67
That gives me all the data aggregated 68 68
so I can look through it. 69 69
This data is aggregated across the network 70 70
from multiple sources in multiple formats. 71 71
Now, because of that, that does lead to a problem though 72 72
because it does come to us 73 73
in all those different multiple formats. 74 74
Now, we have formats like proprietary binary formats, 75 75
tab-separated formats, comma-separated values, 76 76
database log storage, syslog format, SNMP format, 77 77
XML format, JSON format, or text-based format. 78 78
All sorts of data from all different sensors 79 79
and systems are all coming 80 80
to this one SIEM to aggregate it all. 81 81
So we need to parse it and normalize it. 82 82
Parsing and normalization is used to interpret the data, 83 83
which is what we call parsing, 84 84
from all these different formats, and then standardize them 85 85
into a single format for later analysis and processing. 86 86
And this is one of the major functions of your SIEM 87 87
in addition to just collecting all that data 88 88
is making sure it's all in a format that we can look at. 89 89
Now, the way it does this is 90 90
by using connectors or plug-ins. 91 91
Now, a connector or plug-in is a piece of software 92 92
that's designed to provide parsing 93 93
and normalization functions to a particular SIEM. 94 94
As I said previously when talking about Splunk, 95 95
there are tons of different connectors 96 96
that are built into the Splunk framework, 97 97
so you can pretty much get data from everything. 98 98
And depending on the system you're dealing with, 99 99
that can be one of your deciding factors 100 100
on which SIEM you're going to use. 101 101
Which one has the widest variety of plug-ins and connectors 102 102
that will work with all the other systems 103 103
you have on your network? 104 104
Now, the last problem we have to deal with 105 105
is what we call synchronization. 106 106
So now that we've figured out 107 107
the problem of parsing and normalization, 108 108
there is one more problem that we have to solve, 109 109
and that's time. 110 110
There is time coming from all these different systems. 111 111
And depending on your organization, 112 112
you may not just be located in one time zone. 113 113
For instance, in my company, 114 114
we operate in four different time zones. 115 115
We operate in the east coast of the United States, 116 116
we operate in Puerto Rico, which is Atlantic Standard Time, 117 117
we operate in the Philippines, 118 118
and then we also operate in Italy. 119 119
So right now we have four different time zones 120 120
that we are synchronizing across our entire staff 121 121
to make sure we're all working together. 122 122
Now, that's not such a big deal 123 123
when we're working together. 124 124
We can send files asynchronously, not a big deal. 125 125
I sent an email, somebody will work on it later today. 126 126
But it is a big issue 127 127
if we're trying to synchronize our logs 128 128
because all of these different systems 129 129
are having data coming in from different time zones 130 130
and that can become a big challenge for us. 131 131
So especially if we're trying to correlate an event 132 132
because we had an intruder come in 133 133
or some kind of a data breach 134 134
and we need to reconstruct a timeline, 135 135
this can become really difficult 136 136
without synchronizing all of our dates and times. 137 137
So one of the things I like to do with my SIEMS 138 138
is I like to use a standard format for all of our time, 139 139
and every system is going to work together 140 140
inside that time format. 141 141
And that way we all have the same time 142 142
regardless of where we are in the world. 143 143
Now, that doesn't mean we all decide 144 144
to use Eastern Standard Time. 145 145
No, instead, we like to use Coordinated Universal Time, 146 146
which is UTC. 147 147
Now, let's take a look at a couple of times real quick 148 148
and see if we can understand the problem a little better. 149 149
First, what time is it? 150 150
Well, this is something in UTC. 151 151
It very clearly tells me what time it is. 152 152
It is January 1st, 2021 153 153
at 00:00 and zero one seconds. 154 154
It is one second after midnight in UTC. 155 155
Now what time is this one? 156 156
2020-12-31 157 157
158 19:00.01-05:00. 158 159
Well, this is based on the Eastern time zone 159 160
of the United States, which includes places 160 161
like Miami, Florida, Washington, D.C., and New York. 161 162
Essentially, this is actually the exact same time 162 163
as the one I just showed you. 163 164
The difference is it's minus five 164 165
because it's five time zones to the left of UTC. 165 166
So in this case, we're actually talking about 166 167
one second past 7:00 p.m. on New Year's Eve. 167 168
This is the exact same time as we talked about 168 169
with UTC being one second after midnight 169 170
because there's a five-hour difference between the two. 170 171
And so when we do this, 171 172
what we like to do is coordinate all these by using UTC. 172 173
And by using Coordinated Universal Time, 173 174
this uses a time standard as opposed to a time zone. 174 175
Everything can be referenced back to UTC very easily 175 176
and so most of the time you'll see your servers 176 177
are going to be listed in UTC for their times. 177 178
So now that we've solved the problem 178 179
of big data coming to us in all sorts of format 179 180
by parsing it, we've normalized that data, 180 181
it's all stored in the right time using UTC, 181 182
we have one more problem we have to solve. 182 183
And that's secure logging. 183 184
We have all this log data 184 185
and logs contain a lot of good information, 185 186
but it is information that needs to be secured as well. 186 187
Now, in a large organization, 187 188
you can generate a lot of data, 188 189
in fact, gigabytes or terabytes 189 190
of log data every single hour. 190 191
Now, all that data has to be stored someplace 191 192
and that storage needs to be secure. 192 193
And so when you're storing your log data, 193 194
you have to make sure you're securing it 194 195
using the principles of confidentiality, integrity, 195 196
and availability. 196 197
For confidentiality, you want to make sure 197 198
you're encrypting that data 198 199
when you're storing it on the SIEM server. 199 200
For integrity, you want to make sure 200 201
you're doing hashes to make sure the data 201 202
isn't being modified or changed. 202 203
Availability, you want to make sure you're doing backups 203 204
and you have redundancy. 204 205
All the things you think about when you deal with CIA, 205 206
you need to consider that for your log data as well.