Instructor: Data enrichment. 2 2
In this lesson, we're going to talk about data enrichment 3 3
and how we can use machine learning to help us with that. 4 4
One of the best things that machine learning 5 5
can assist us with is data correlation, 6 6
because there is so much data across all of our systems, 7 7
and if we throw it all into our SIEM, 8 8
it's still a lot of data for us as an analyst to go through. 9 9
So by using machine learning, 10 10
we can have it surface up to the top 11 11
what it thinks is the most important things 12 12
for us to look at as an analyst. 13 13
Now, when we do this, one of the things 14 14
that it can also do to help us is use data enrichment. 15 15
Now, data enrichment is a process 16 16
of incorporating new updates and information 17 17
to an organization's existing database 18 18
to help improve its accuracy. 19 19
So if I have my SIEM with all the things I saw 20 20
in my network, but I don't have any threat information 21 21
from other third-party sources, I may not catch something. 22 22
But by taking machine learning and doing data enrichment 23 23
and bringing all that data together, 24 24
I can have different open source feeds. 25 25
I can have other partner systems 26 26
and my own systems telling me what I'm seeing 27 27
and that can help me figure it out. 28 28
Now, AI-based systems can help combine 29 29
all of these indicators from multiple threat feeds 30 30
to help reduce our false positives 31 31
and our false negatives inside of our systems. 32 32
So it makes our systems even better 33 33
by using this as an enrichment technique. 34 34
Again, our goal here isn't to eliminate the person. 35 35
It's to make sure that the person has the best information 36 36
to make the best human decisions. 37 37
Now, let me give you an example. 38 38
Let's say that I need to create malware signatures. 39 39
That was my job. 40 40
I'm a reverse malware analyst, and I'm going to go through 41 41
and take apart those binaries, decompile them, 42 42
look at the hex code and figure out 43 43
exactly what they're doing. 44 44
And, from that, I can create a signature. 45 45
So, here on the screen, you can see an example 46 46
of an old piece of malware called the I Love You Bug. 47 47
Now, in this one, there were certain pieces of code 48 48
that you can pull out as your signature. 49 49
Now that's great, but what happens 50 50
if somebody changes one of those bites of code? 51 51
Well, now it has a different signature, 52 52
and that means I can miss it. 53 53
And so if somebody keeps doing that by changing the code, 54 54
it's going to make my job as a malware analyst 55 55
really, really hard because I'm going to to keep going back 56 56
and creating new signatures all the time. 57 57
If I use AI, I can give it a basic signature 58 58
and say, "This is what I'm looking for. 59 59
Now, if you find any variations that look similar to this, 60 60
go ahead and flag those as well 61 61
because somebody might have changed one bit here 62 62
or one bit there." 63 63
And by doing that, we can integrate AI 64 64
and machine learning to help us still identify 65 65
what is malicious, even if they change the source code. 66 66
Now, AI-based systems can really help us 67 67
as we're trying to identify this malware 68 68
that's been jumbled up, and it can do it 69 69
a lot better than our human counterparts. 70 70
People like me just can't do it nearly as fast 71 71
as a computer can. 72 72
So these AI-based systems, though, they do have struggles. 73 73
They're not really good when they're trying 74 74
to identify things like administrative actions. 75 75
So if somebody went in there and did something malicious 76 76
as an administrator, that's really hard for them to detect 77 77
because what is malicious about somebody changing a password 78 78
or creating a new user account or changing a policy setting 79 79
or moving files around? 80 80
All of those are normal actions we do on a daily basis. 81 81
What makes them malicious is the intent behind them. 82 82
And so if somebody's creating a new user account 83 83
to be able to have an extra admin account 84 84
so if they get kicked off the system, 85 85
they can log back in, that's malicious. 86 86
But just creating a new admin account, 87 87
that's not necessarily malicious. 88 88
That might happen just because I hired 89 89
a new system administrator. 90 90
And so you have to think about these things, 91 91
and this is where humans are better 92 92
at identifying things than AI-based systems. 93 93
Now, when you're dealing with your machine learning, 94 94
remember as I said before, machine learning is only as good 95 95
as the datasets that you use during training. 96 96
If you give it really bad datasets, 97 97
you're going to have a lot of false positives. 98 98
If you have really good datasets, 99 99
you're going to be able to catch a lot more malware 100 100
and a lot more threats. 101 101
So keep that in mind. 102 102
And this is a place where using the right training 103 103
and the right datasets 104 104
really does pay off for your machine learning.