-: Lessons Learned. 2 2
In this lesson, we are going to talk about lessons learned 3 3
which is a key part of your post-incident activities. 4 4
Now, lessons learned are an analysis 5 5
of the events that can provide us insight 6 6
into how to improve response processes in the future. 7 7
The lessons learned process is a formalized method 8 8
for us to document the things we experienced 9 9
during the incident, what went right, what went wrong, 10 10
what could we do better next time? 11 11
All of these things are things that should be recorded 12 12
and our internal organizational processes should be improved 13 13
so that the same issues don't occur again 14 14
during the next incident. 15 15
For example, maybe the change management board was too slow 16 16
to approve the security fix that we needed to implement 17 17
during the incident response 18 18
as we tried to secure the network. 19 19
One lesson learned here might be captured 20 20
as the need for us to decrease the approval times 21 21
for emergent change requests during an incident. 22 22
Now, we don't have to redevelop the change process 23 23
during the lessons learned session, though, 24 24
we just need to identify what could have been improved. 25 25
To capture the lessons learned, 26 26
I recommend you conduct a meeting 27 27
with the people directly involved with the incident response 28 28
at various levels of the organization. 29 29
This will allow you to determine exactly what happened 30 30
and at what times during that specific incident. 31 31
You'll also want to focus on the relationships 32 32
within the organization during that response. 33 33
Was communication up and down the organization effective? 34 34
How about the documented notification procedures? 35 35
Were those followed correctly? 36 36
And if so, did they work? 37 37
All of these are things you want to capture 38 38
during your lessons learned. 39 39
Now, lessons learned meetings can be structured 40 40
using what we call the six questions. 41 41
And these are the things you really want to base your meeting 42 42
around to be able to capture the most data from your teams. 43 43
First, who is the adversary? 44 44
Essentially, was this an insider threat, 45 45
an external threat, or both? 46 46
Second, why was the incident conducted? 47 47
Exactly what were the motives of the adversary 48 48
and what type of assets were they trying to go after? 49 49
All of this goes to the why, the motive behind it. 50 50
Our third thing we'll look at, 51 51
when did the incident occur? 52 52
Now, this is important because we need to know 53 53
when did this thing happen? 54 54
Did it happen during a busy shopping period? 55 55
Did it happen during the middle of summer? 56 56
Did it happen overnight? 57 57
We have to figure that out 58 58
because that can tell us when we're being targeted. 59 59
In addition to that, 60 60
we might want to figure out when did we detect it? 61 61
Because if it happened at three in the morning 62 62
and our people didn't come to work till eight in the morning 63 63
and it took them three more hours to figure it out, 64 64
that was a long time that we didn't detect this incident. 65 65
And so these are things you want to think about. 66 66
Also, we want to think about where. 67 67
Where did this incident occur? 68 68
Did it occur inside of a host system or a server 69 69
or a network segment? 70 70
What was being affected as part of this? 71 71
Our fifth thing we want to consider is how. 72 72
How did the incentive occur? 73 73
What tactics and techniques and procedures 74 74
did the adversary use? 75 75
How did they break into the system? 76 76
Did they have a good knowledge base 77 77
of their adversary kill chain 78 78
and know all about your systems 79 79
and what all your weaknesses were? 80 80
All of that goes back to how, how did this happen? 81 81
Which then can tell us how we could stop 82 82
in the future, right? 83 83
And then six, what controls could have mitigated it? 84 84
Now, this is important 85 85
because now that we've captured all this information 86 86
about who and why and when and where, and how, 87 87
we need to think about what. 88 88
What can we do about it? 89 89
And that's a big part of lessons learned. 90 90
We identify all these problems 91 91
and we want to learn the lessons 92 92
and incorporate those into our future network design. 93 93
Now, another thing I want to point out, 94 94
is that when we are dealing with a lessons learned meeting, 95 95
it is important that we are focused on 96 96
what we can do better, not who do we blame for this issue. 97 97
In a lessons learned workshop 98 98
we are never trying to find blame, only improvements. 99 99
So I like to pull out my can of blame remover 100 100
and remove all blame from the situation. 101 101
We want to figured out what worked and do more of that. 102 102
We want to figured out what didn't work and do less of that. 103 103
This is what's important when dealing 104 104
with a lessons learned meeting. 105 105
We want to provide a culture of safety 106 106
where your employees feel safe to tell 107 107
what they think went wrong without being blamed. 108 108
They don't want to be fearful of losing their job here. 109 109
They want to make sure we all are okay 110 110
with the fact that something bad happened 111 111
and now we're trying to work together 112 112
to prevent something bad happening again in the future. 113 113
Now, another part of this is we're going to conduct a report 114 114
and this is often called an After-Action Report or an AAR, 115 115
or a Lessons Learned Report, an LLR. 116 116
Now, this report is going to provide insight 117 117
into the specific incident 118 118
and how to improve response processes in the future. 119 119
You might see here that we actually write 120 120
a lot of reports as cybersecurity analysts. 121 121
We talked in the last lesson 122 122
about our incident summary reports 123 123
and our executive summaries. 124 124
Now we're talking about After-Action Reports 125 125
and Lessons Learned Reports. 126 126
These reports are going to allow us to bring up to light 127 127
all of those issues that we found 128 128
inside that working group after the incident, 129 129
so we can solve these things for the long term. 130 130
Now, there are a lot of benefits 131 131
of using Lessons Learned and After-Action Reports. 132 132
These include Incident Response Plan Updates, 133 133
IoC Generation and Monitoring, and Change Control Processes. 134 134
Now, when we talk about Incident Response Plan Updates, 135 135
this is going to be updated based on the lessons learned 136 136
that we had from the real world. 137 137
Oftentimes, we write up all these plans in theory, 138 138
what we think is going to happen 139 139
and how we think we will best handle the situation. 140 140
But then when the rubber meets the road, 141 141
and we actually have an incident 142 142
and we execute those plans, things go wrong. 143 143
So we have those scars 144 144
and we want to take those scars as lessons learned 145 145
and incorporate them and modify those plans. 146 146
I always tell my team, 147 147
"I don't care if we make mistakes, 148 148
but we just shouldn't make the same mistakes 149 149
over and over again. 150 150
We need to learn from those mistakes, 151 151
update our plans and processes, 152 152
and then incorporate those changes into the way we work. 153 153
Our second thing we want to talk about here 154 154
is indicators of compromise. 155 155
One of the things you can generate 156 156
from your lessons learned is indicators of compromise. 157 157
If the response team feels 158 158
that it didn't receive enough actionable information 159 159
during an incident, they can also verify 160 160
that the security monitoring 161 161
and logging services are up to par. 162 162
During the incident, 163 163
analysts may have developed new filters and query statements 164 164
and they have different scripts they've created 165 165
to discover and correlate 166 166
these different indicators of compromise. 167 167
The incident response team may be able 168 168
to detect new or variant malware code based on that 169 169
and they create signatures specifically to identify it. 170 170
These new detection rules 171 171
and binary signatures can be added back 172 172
into your security systems to provide ongoing monitoring 173 173
as a way to identify these new IoCs you've developed, 174 174
based on these lessons learned. 175 175
And finally, the change control 176 176
or change management process. 177 177
We want to use our lessons learned 178 178
to improve the change control process in our organization. 179 179
For example, a lot of organizations are very tight held 180 180
in their change management process, 181 181
and this can slow things down. 182 182
Well, in the middle of an incident response, 183 183
that can be a really bad thing 184 184
because if it takes you three weeks to get approval 185 185
to put a new router or switch on the network, 186 186
that is going to be way too much time 187 187
and you're going to be way too slow 188 188
to get the adversary out of your network. 189 189
Instead, you need to have a change management 190 190
and change control process that can work very fluidly 191 191
and very rapidly in an agile manner 192 192
to help you get the changes you need through the system. 193 193
And lessons learned is a great way to point 194 194
out issues with your change management process 195 195
and get those changes back 196 196
into the process for future iterations.