1 1 00:00:00,330 --> 00:00:02,130 Lessons Learned. 2 2 00:00:02,130 --> 00:00:05,190 In this lesson, we are going to talk about lessons learned 3 3 00:00:05,190 --> 00:00:08,490 which is a key part of your post-incident activities. 4 4 00:00:08,490 --> 00:00:10,470 Now, lessons learned are an analysis 5 5 00:00:10,470 --> 00:00:12,390 of the events that can provide us insight 6 6 00:00:12,390 --> 00:00:15,960 into how to improve response processes in the future. 7 7 00:00:15,960 --> 00:00:18,600 The lessons learned process is a formalized method 8 8 00:00:18,600 --> 00:00:20,460 for us to document the things we experienced 9 9 00:00:20,460 --> 00:00:23,640 during the incident, what went right, what went wrong, 10 10 00:00:23,640 --> 00:00:25,470 what could we do better next time? 11 11 00:00:25,470 --> 00:00:27,780 All of these things are things that should be recorded 12 12 00:00:27,780 --> 00:00:30,810 and our internal organizational processes should be improved 13 13 00:00:30,810 --> 00:00:32,880 so that the same issues don't occur again 14 14 00:00:32,880 --> 00:00:34,620 during the next incident. 15 15 00:00:34,620 --> 00:00:37,680 For example, maybe the change management board was too slow 16 16 00:00:37,680 --> 00:00:40,320 to approve the security fix that we needed to implement 17 17 00:00:40,320 --> 00:00:41,670 during the incident response 18 18 00:00:41,670 --> 00:00:43,710 as we tried to secure the network. 19 19 00:00:43,710 --> 00:00:45,990 One lesson learned here might be captured 20 20 00:00:45,990 --> 00:00:48,270 as the need for us to decrease the approval times 21 21 00:00:48,270 --> 00:00:51,210 for emergent change requests during an incident. 22 22 00:00:51,210 --> 00:00:53,550 Now, we don't have to redevelop the change process 23 23 00:00:53,550 --> 00:00:55,260 during the lessons learned session, though, 24 24 00:00:55,260 --> 00:00:58,230 we just need to identify what could have been improved. 25 25 00:00:58,230 --> 00:00:59,730 To capture the lessons learned, 26 26 00:00:59,730 --> 00:01:01,170 I recommend you conduct a meeting 27 27 00:01:01,170 --> 00:01:03,750 with the people directly involved with the incident response 28 28 00:01:03,750 --> 00:01:05,940 at various levels of the organization. 29 29 00:01:05,940 --> 00:01:08,520 This will allow you to determine exactly what happened 30 30 00:01:08,520 --> 00:01:11,430 and at what times during that specific incident. 31 31 00:01:11,430 --> 00:01:13,260 You'll also want to focus on the relationships 32 32 00:01:13,260 --> 00:01:16,260 within the organization during that response. 33 33 00:01:16,260 --> 00:01:19,440 Was communication up and down the organization effective? 34 34 00:01:19,440 --> 00:01:21,540 How about the documented notification procedures? 35 35 00:01:21,540 --> 00:01:22,830 Were those followed correctly? 36 36 00:01:22,830 --> 00:01:24,780 And if so, did they work? 37 37 00:01:24,780 --> 00:01:26,400 All of these are things you want to capture 38 38 00:01:26,400 --> 00:01:28,260 during your lessons learned. 39 39 00:01:28,260 --> 00:01:30,810 Now, lessons learned meetings can be structured 40 40 00:01:30,810 --> 00:01:33,240 using what we call the six questions. 41 41 00:01:33,240 --> 00:01:35,100 And these are the things you really want to base your meeting 42 42 00:01:35,100 --> 00:01:38,160 around to be able to capture the most data from your teams. 43 43 00:01:38,160 --> 00:01:40,530 First, who is the adversary? 44 44 00:01:40,530 --> 00:01:42,630 Essentially, was this an insider threat, 45 45 00:01:42,630 --> 00:01:44,550 an external threat, or both? 46 46 00:01:44,550 --> 00:01:47,250 Second, why was the incident conducted? 47 47 00:01:47,250 --> 00:01:49,680 Exactly what were the motives of the adversary 48 48 00:01:49,680 --> 00:01:52,290 and what type of assets were they trying to go after? 49 49 00:01:52,290 --> 00:01:55,800 All of this goes to the why, the motive behind it. 50 50 00:01:55,800 --> 00:01:57,240 Our third thing we'll look at, 51 51 00:01:57,240 --> 00:01:59,160 when did the incident occur? 52 52 00:01:59,160 --> 00:02:01,020 Now, this is important because we need to know 53 53 00:02:01,020 --> 00:02:02,760 when did this thing happen? 54 54 00:02:02,760 --> 00:02:04,830 Did it happen during a busy shopping period? 55 55 00:02:04,830 --> 00:02:06,630 Did it happen during the middle of summer? 56 56 00:02:06,630 --> 00:02:08,040 Did it happen overnight? 57 57 00:02:08,040 --> 00:02:09,030 We have to figure that out 58 58 00:02:09,030 --> 00:02:11,310 because that can tell us when we're being targeted. 59 59 00:02:11,310 --> 00:02:12,300 In addition to that, 60 60 00:02:12,300 --> 00:02:14,520 we might want to figure out when did we detect it? 61 61 00:02:14,520 --> 00:02:16,350 Because if it happened at three in the morning 62 62 00:02:16,350 --> 00:02:18,270 and our people didn't come to work till eight in the morning 63 63 00:02:18,270 --> 00:02:20,400 and it took them three more hours to figure it out, 64 64 00:02:20,400 --> 00:02:22,977 that was a long time that we didn't detect this incident. 65 65 00:02:22,977 --> 00:02:25,440 And so these are things you want to think about. 66 66 00:02:25,440 --> 00:02:27,450 Also, we want to think about where. 67 67 00:02:27,450 --> 00:02:29,280 Where did this incident occur? 68 68 00:02:29,280 --> 00:02:32,520 Did it occur inside of a host system or a server 69 69 00:02:32,520 --> 00:02:33,870 or a network segment? 70 70 00:02:33,870 --> 00:02:36,570 What was being affected as part of this? 71 71 00:02:36,570 --> 00:02:38,850 Our fifth thing we want to consider is how. 72 72 00:02:38,850 --> 00:02:40,380 How did the incentive occur? 73 73 00:02:40,380 --> 00:02:42,810 What tactics and techniques and procedures 74 74 00:02:42,810 --> 00:02:44,250 did the adversary use? 75 75 00:02:44,250 --> 00:02:45,930 How did they break into the system? 76 76 00:02:45,930 --> 00:02:47,610 Did they have a good knowledge base 77 77 00:02:47,610 --> 00:02:48,930 of their adversary kill chain 78 78 00:02:48,930 --> 00:02:50,340 and know all about your systems 79 79 00:02:50,340 --> 00:02:52,230 and what all your weaknesses were? 80 80 00:02:52,230 --> 00:02:55,290 All of that goes back to how, how did this happen? 81 81 00:02:55,290 --> 00:02:56,910 Which then can tell us how we could stop 82 82 00:02:56,910 --> 00:02:58,380 in the future, right? 83 83 00:02:58,380 --> 00:03:01,740 And then six, what controls could have mitigated it? 84 84 00:03:01,740 --> 00:03:02,700 Now, this is important 85 85 00:03:02,700 --> 00:03:04,980 because now that we've captured all this information 86 86 00:03:04,980 --> 00:03:08,340 about who and why and when and where, and how, 87 87 00:03:08,340 --> 00:03:09,870 we need to think about what. 88 88 00:03:09,870 --> 00:03:11,400 What can we do about it? 89 89 00:03:11,400 --> 00:03:13,140 And that's a big part of lessons learned. 90 90 00:03:13,140 --> 00:03:14,940 We identify all these problems 91 91 00:03:14,940 --> 00:03:16,320 and we want to learn the lessons 92 92 00:03:16,320 --> 00:03:19,830 and incorporate those into our future network design. 93 93 00:03:19,830 --> 00:03:21,570 Now, another thing I want to point out, 94 94 00:03:21,570 --> 00:03:24,000 is that when we are dealing with a lessons learned meeting, 95 95 00:03:24,000 --> 00:03:25,710 it is important that we are focused on 96 96 00:03:25,710 --> 00:03:29,610 what we can do better, not who do we blame for this issue. 97 97 00:03:29,610 --> 00:03:31,050 In a lessons learned workshop 98 98 00:03:31,050 --> 00:03:34,200 we are never trying to find blame, only improvements. 99 99 00:03:34,200 --> 00:03:36,330 So I like to pull out my can of blame remover 100 100 00:03:36,330 --> 00:03:38,970 and remove all blame from the situation. 101 101 00:03:38,970 --> 00:03:41,640 We want to figured out what worked and do more of that. 102 102 00:03:41,640 --> 00:03:44,370 We want to figured out what didn't work and do less of that. 103 103 00:03:44,370 --> 00:03:45,810 This is what's important when dealing 104 104 00:03:45,810 --> 00:03:47,280 with a lessons learned meeting. 105 105 00:03:47,280 --> 00:03:49,470 We want to provide a culture of safety 106 106 00:03:49,470 --> 00:03:51,900 where your employees feel safe to tell 107 107 00:03:51,900 --> 00:03:54,540 what they think went wrong without being blamed. 108 108 00:03:54,540 --> 00:03:56,820 They don't want to be fearful of losing their job here. 109 109 00:03:56,820 --> 00:03:58,950 They want to make sure we all are okay 110 110 00:03:58,950 --> 00:04:00,720 with the fact that something bad happened 111 111 00:04:00,720 --> 00:04:02,100 and now we're trying to work together 112 112 00:04:02,100 --> 00:04:05,220 to prevent something bad happening again in the future. 113 113 00:04:05,220 --> 00:04:08,160 Now, another part of this is we're going to conduct a report 114 114 00:04:08,160 --> 00:04:11,670 and this is often called an After-Action Report or an AAR, 115 115 00:04:11,670 --> 00:04:14,280 or a Lessons Learned Report, an LLR. 116 116 00:04:14,280 --> 00:04:16,380 Now, this report is going to provide insight 117 117 00:04:16,380 --> 00:04:18,000 into the specific incident 118 118 00:04:18,000 --> 00:04:21,330 and how to improve response processes in the future. 119 119 00:04:21,330 --> 00:04:22,860 You might see here that we actually write 120 120 00:04:22,860 --> 00:04:25,260 a lot of reports as cybersecurity analysts. 121 121 00:04:25,260 --> 00:04:26,700 We talked in the last lesson 122 122 00:04:26,700 --> 00:04:28,260 about our incident summary reports 123 123 00:04:28,260 --> 00:04:29,550 and our executive summaries. 124 124 00:04:29,550 --> 00:04:31,410 Now we're talking about After-Action Reports 125 125 00:04:31,410 --> 00:04:33,090 and Lessons Learned Reports. 126 126 00:04:33,090 --> 00:04:36,030 These reports are going to allow us to bring up to light 127 127 00:04:36,030 --> 00:04:37,530 all of those issues that we found 128 128 00:04:37,530 --> 00:04:39,930 inside that working group after the incident, 129 129 00:04:39,930 --> 00:04:42,360 so we can solve these things for the long term. 130 130 00:04:42,360 --> 00:04:43,560 Now, there are a lot of benefits 131 131 00:04:43,560 --> 00:04:46,170 of using Lessons Learned and After-Action Reports. 132 132 00:04:46,170 --> 00:04:48,690 These include Incident Response Plan Updates, 133 133 00:04:48,690 --> 00:04:52,440 IoC Generation and Monitoring, and Change Control Processes. 134 134 00:04:52,440 --> 00:04:55,350 Now, when we talk about Incident Response Plan Updates, 135 135 00:04:55,350 --> 00:04:57,540 this is going to be updated based on the lessons learned 136 136 00:04:57,540 --> 00:04:59,280 that we had from the real world. 137 137 00:04:59,280 --> 00:05:02,340 Oftentimes, we write up all these plans in theory, 138 138 00:05:02,340 --> 00:05:03,690 what we think is going to happen 139 139 00:05:03,690 --> 00:05:06,120 and how we think we will best handle the situation. 140 140 00:05:06,120 --> 00:05:08,100 But then when the rubber meets the road, 141 141 00:05:08,100 --> 00:05:09,300 and we actually have an incident 142 142 00:05:09,300 --> 00:05:12,000 and we execute those plans, things go wrong. 143 143 00:05:12,000 --> 00:05:13,890 So we have those scars 144 144 00:05:13,890 --> 00:05:16,290 and we want to take those scars as lessons learned 145 145 00:05:16,290 --> 00:05:19,200 and incorporate them and modify those plans. 146 146 00:05:19,200 --> 00:05:20,227 I always tell my team, 147 147 00:05:20,227 --> 00:05:21,840 "I don't care if we make mistakes, 148 148 00:05:21,840 --> 00:05:23,940 but we just shouldn't make the same mistakes 149 149 00:05:23,940 --> 00:05:25,200 over and over again. 150 150 00:05:25,200 --> 00:05:26,880 We need to learn from those mistakes, 151 151 00:05:26,880 --> 00:05:28,620 update our plans and processes, 152 152 00:05:28,620 --> 00:05:32,040 and then incorporate those changes into the way we work. 153 153 00:05:32,040 --> 00:05:33,870 Our second thing we want to talk about here 154 154 00:05:33,870 --> 00:05:35,850 is indicators of compromise. 155 155 00:05:35,850 --> 00:05:37,080 One of the things you can generate 156 156 00:05:37,080 --> 00:05:40,320 from your lessons learned is indicators of compromise. 157 157 00:05:40,320 --> 00:05:41,490 If the response team feels 158 158 00:05:41,490 --> 00:05:43,470 that it didn't receive enough actionable information 159 159 00:05:43,470 --> 00:05:45,660 during an incident, they can also verify 160 160 00:05:45,660 --> 00:05:46,740 that the security monitoring 161 161 00:05:46,740 --> 00:05:49,260 and logging services are up to par. 162 162 00:05:49,260 --> 00:05:50,250 During the incident, 163 163 00:05:50,250 --> 00:05:53,220 analysts may have developed new filters and query statements 164 164 00:05:53,220 --> 00:05:54,960 and they have different scripts they've created 165 165 00:05:54,960 --> 00:05:56,190 to discover and correlate 166 166 00:05:56,190 --> 00:05:58,320 these different indicators of compromise. 167 167 00:05:58,320 --> 00:06:00,120 The incident response team may be able 168 168 00:06:00,120 --> 00:06:03,090 to detect new or variant malware code based on that 169 169 00:06:03,090 --> 00:06:06,060 and they create signatures specifically to identify it. 170 170 00:06:06,060 --> 00:06:07,290 These new detection rules 171 171 00:06:07,290 --> 00:06:09,510 and binary signatures can be added back 172 172 00:06:09,510 --> 00:06:12,600 into your security systems to provide ongoing monitoring 173 173 00:06:12,600 --> 00:06:15,840 as a way to identify these new IoCs you've developed, 174 174 00:06:15,840 --> 00:06:17,820 based on these lessons learned. 175 175 00:06:17,820 --> 00:06:19,410 And finally, the change control 176 176 00:06:19,410 --> 00:06:21,270 or change management process. 177 177 00:06:21,270 --> 00:06:22,800 We want to use our lessons learned 178 178 00:06:22,800 --> 00:06:26,190 to improve the change control process in our organization. 179 179 00:06:26,190 --> 00:06:29,460 For example, a lot of organizations are very tight held 180 180 00:06:29,460 --> 00:06:31,350 in their change management process, 181 181 00:06:31,350 --> 00:06:33,120 and this can slow things down. 182 182 00:06:33,120 --> 00:06:34,770 Well, in the middle of an incident response, 183 183 00:06:34,770 --> 00:06:36,510 that can be a really bad thing 184 184 00:06:36,510 --> 00:06:38,370 because if it takes you three weeks to get approval 185 185 00:06:38,370 --> 00:06:40,680 to put a new router or switch on the network, 186 186 00:06:40,680 --> 00:06:42,780 that is going to be way too much time 187 187 00:06:42,780 --> 00:06:43,950 and you're going to be way too slow 188 188 00:06:43,950 --> 00:06:45,990 to get the adversary out of your network. 189 189 00:06:45,990 --> 00:06:48,270 Instead, you need to have a change management 190 190 00:06:48,270 --> 00:06:51,480 and change control process that can work very fluidly 191 191 00:06:51,480 --> 00:06:54,000 and very rapidly in an agile manner 192 192 00:06:54,000 --> 00:06:56,850 to help you get the changes you need through the system. 193 193 00:06:56,850 --> 00:06:58,710 And lessons learned is a great way to point 194 194 00:06:58,710 --> 00:07:00,930 out issues with your change management process 195 195 00:07:00,930 --> 00:07:02,490 and get those changes back 196 196 00:07:02,490 --> 00:07:05,013 into the process for future iterations.