1 00:00:00,240 --> 00:00:01,080 In this lesson, 2 00:00:01,080 --> 00:00:04,140 we're going to discuss NetFlow and Flow Analysis. 3 00:00:04,140 --> 00:00:05,580 Now, as you've probably guessed, 4 00:00:05,580 --> 00:00:08,189 packet captures can take up a lot of storage space 5 00:00:08,189 --> 00:00:10,080 and they can quickly grow in size. 6 00:00:10,080 --> 00:00:12,750 For example, if I wanted to conduct full packet capture 7 00:00:12,750 --> 00:00:14,280 on my home network each day, 8 00:00:14,280 --> 00:00:16,320 I would need several gigabytes of storage 9 00:00:16,320 --> 00:00:17,730 just for my small family 10 00:00:17,730 --> 00:00:20,520 because every single packet that goes in and out of my house 11 00:00:20,520 --> 00:00:22,950 would have to be captured by that network sniffer. 12 00:00:22,950 --> 00:00:24,960 Every video game my son is playing online, 13 00:00:24,960 --> 00:00:26,670 every YouTube video he's streaming, 14 00:00:26,670 --> 00:00:28,920 every Netflix show that my wife is binging, 15 00:00:28,920 --> 00:00:30,210 all of that is going to be captured 16 00:00:30,210 --> 00:00:32,310 frame by frame and bit by bit. 17 00:00:32,310 --> 00:00:35,100 This is known as a full packet capture. 18 00:00:35,100 --> 00:00:38,610 Now, with full packet capture, also known as FPCs, 19 00:00:38,610 --> 00:00:40,920 you're going to have to capture the entire packet, 20 00:00:40,920 --> 00:00:43,050 including its header and its payload. 21 00:00:43,050 --> 00:00:44,400 Because full packet captures 22 00:00:44,400 --> 00:00:46,350 take up so much storage space though, 23 00:00:46,350 --> 00:00:48,660 there are many times that we don't want to collect it all 24 00:00:48,660 --> 00:00:51,570 because this can be a waste of a lot of resources for us. 25 00:00:51,570 --> 00:00:54,120 Instead, we can opt to use flow analysis 26 00:00:54,120 --> 00:00:55,620 to collect the most important details 27 00:00:55,620 --> 00:00:57,870 about the traffic being sent. 28 00:00:57,870 --> 00:01:00,570 Now, flow analysis relies on a flow collector, 29 00:01:00,570 --> 00:01:03,120 which is a means of recording metadata and statistics 30 00:01:03,120 --> 00:01:04,470 about network traffic 31 00:01:04,470 --> 00:01:06,300 rather than recording each and every frame 32 00:01:06,300 --> 00:01:07,950 that's passing over the network. 33 00:01:07,950 --> 00:01:10,680 The flow collector can then sample the network traffic 34 00:01:10,680 --> 00:01:13,110 and provide statistics to our flow analysis tools 35 00:01:13,110 --> 00:01:14,640 for us be able to observe. 36 00:01:14,640 --> 00:01:17,370 By collecting this metadata about the data itself, 37 00:01:17,370 --> 00:01:19,080 we can save a lot of storage space 38 00:01:19,080 --> 00:01:21,510 and minimize the processing costs as well. 39 00:01:21,510 --> 00:01:23,550 Now, the downside of using flow analysis 40 00:01:23,550 --> 00:01:25,290 instead of full packet capture though 41 00:01:25,290 --> 00:01:26,670 is that we won't have the contents 42 00:01:26,670 --> 00:01:28,380 of what we sent over the network, 43 00:01:28,380 --> 00:01:30,960 we only have information about that data. 44 00:01:30,960 --> 00:01:32,670 For example, if I had a list 45 00:01:32,670 --> 00:01:34,410 of every phone call you made today, 46 00:01:34,410 --> 00:01:36,990 I would know who you called, when the call started, 47 00:01:36,990 --> 00:01:39,420 how long you talked for, and when the call finished, 48 00:01:39,420 --> 00:01:40,680 but I wouldn't have any idea 49 00:01:40,680 --> 00:01:42,720 of what you said during that call. 50 00:01:42,720 --> 00:01:44,460 That's the idea of metadata 51 00:01:44,460 --> 00:01:46,860 when we collect things about the information, 52 00:01:46,860 --> 00:01:48,540 and so this kind of metadata we collect 53 00:01:48,540 --> 00:01:50,160 when dealing with flow analysis 54 00:01:50,160 --> 00:01:52,560 is going to be dealing with what kind of traffic was sent, 55 00:01:52,560 --> 00:01:56,370 if it was HTTP, HTTPS, FTP, SSH, 56 00:01:56,370 --> 00:01:59,010 or some other protocol, and how much of it. 57 00:01:59,010 --> 00:02:01,290 Now, all the flow analysis information that's gathered 58 00:02:01,290 --> 00:02:03,090 can then be stored inside of a database 59 00:02:03,090 --> 00:02:04,740 and queried using different tools 60 00:02:04,740 --> 00:02:06,720 to produce reports and graphs. 61 00:02:06,720 --> 00:02:09,090 Flow analysis is wonderful for highlighting trends 62 00:02:09,090 --> 00:02:11,039 and patterns in the traffic observed, 63 00:02:11,039 --> 00:02:12,000 and it can create alerts 64 00:02:12,000 --> 00:02:13,260 based on anomalies observed 65 00:02:13,260 --> 00:02:15,450 in those traffic patterns as well. 66 00:02:15,450 --> 00:02:18,450 Flow analysis can also quickly create visualizations 67 00:02:18,450 --> 00:02:20,220 to map out the different network connections, 68 00:02:20,220 --> 00:02:21,060 the traffic types, 69 00:02:21,060 --> 00:02:23,610 and the volume of each session or connection. 70 00:02:23,610 --> 00:02:24,810 This can be really useful 71 00:02:24,810 --> 00:02:26,670 in identifying different traffic patterns 72 00:02:26,670 --> 00:02:28,170 that might reveal bad behavior, 73 00:02:28,170 --> 00:02:30,660 such as malware in transit, data exfiltration, 74 00:02:30,660 --> 00:02:33,540 or other suspicious or malicious activity. 75 00:02:33,540 --> 00:02:35,340 NetFlow is a Cisco-developed means 76 00:02:35,340 --> 00:02:37,170 of reporting network flow information 77 00:02:37,170 --> 00:02:38,790 to a structured database. 78 00:02:38,790 --> 00:02:40,800 This was actually one of the first ones out there, 79 00:02:40,800 --> 00:02:42,540 and eventually it became the standard 80 00:02:42,540 --> 00:02:43,770 that everybody started using 81 00:02:43,770 --> 00:02:48,660 under the name IPFIX, or IP Flow Information Export. 82 00:02:48,660 --> 00:02:50,430 Now, IPFIX is going to be used 83 00:02:50,430 --> 00:02:52,380 to define a particular traffic flow 84 00:02:52,380 --> 00:02:53,610 based on different packets 85 00:02:53,610 --> 00:02:55,470 that share the same characteristics. 86 00:02:55,470 --> 00:02:57,720 For example, we could set up a traffic flow 87 00:02:57,720 --> 00:03:00,030 based on the same source and destination IP 88 00:03:00,030 --> 00:03:01,470 that's being used to signify 89 00:03:01,470 --> 00:03:03,870 there is a session between two different hosts. 90 00:03:03,870 --> 00:03:05,940 Now, this is the data flow between two hosts, 91 00:03:05,940 --> 00:03:08,160 and we can gain some information about it. 92 00:03:08,160 --> 00:03:10,200 Then, we might have a second data flow 93 00:03:10,200 --> 00:03:12,450 for anything communicating with a particular domain name 94 00:03:12,450 --> 00:03:15,270 and we can collect information about that data flow too. 95 00:03:15,270 --> 00:03:17,670 Now, as we look at the data collected by NetFlow, 96 00:03:17,670 --> 00:03:18,780 we're going to find information 97 00:03:18,780 --> 00:03:20,340 about the packets that was sent, 98 00:03:20,340 --> 00:03:22,830 including the network protocol interface being used, 99 00:03:22,830 --> 00:03:24,990 the version and type of IP being used, 100 00:03:24,990 --> 00:03:27,570 the source and destination IP addresses being used, 101 00:03:27,570 --> 00:03:29,820 the source and destination ports being used, 102 00:03:29,820 --> 00:03:32,100 or the type of service being used. 103 00:03:32,100 --> 00:03:33,810 All this information can be gathered 104 00:03:33,810 --> 00:03:36,990 and then analyzed visually using these different tools. 105 00:03:36,990 --> 00:03:38,730 For example, here on the screen 106 00:03:38,730 --> 00:03:40,500 you can see how SolarWinds tools 107 00:03:40,500 --> 00:03:43,620 are used to display the collected NetFlow data of a network. 108 00:03:43,620 --> 00:03:46,440 Similarly, you can look at it in a more text-based manner 109 00:03:46,440 --> 00:03:48,360 by listing out the data flows as 1 110 00:03:48,360 --> 00:03:50,340 and going all the way down to flow 15 111 00:03:50,340 --> 00:03:51,900 in this little snippet. 112 00:03:51,900 --> 00:03:53,610 Here, if you wanted to see what each one 113 00:03:53,610 --> 00:03:54,960 had inside of its flow, 114 00:03:54,960 --> 00:03:58,110 you would click the plus sign like I did next to flow 15, 115 00:03:58,110 --> 00:04:00,360 and you could see the source and destination addresses, 116 00:04:00,360 --> 00:04:02,160 the source port and destination port, 117 00:04:02,160 --> 00:04:05,430 and some basic information about this particular data flow. 118 00:04:05,430 --> 00:04:06,780 Notice though, you're not getting 119 00:04:06,780 --> 00:04:08,760 any of the actual content of the packets 120 00:04:08,760 --> 00:04:10,470 that were inside of that traffic flow 121 00:04:10,470 --> 00:04:13,080 because NetFlow only shares with you metadata 122 00:04:13,080 --> 00:04:14,400 about the data. 123 00:04:14,400 --> 00:04:17,370 This is because you didn't collect the full packet captures. 124 00:04:17,370 --> 00:04:19,709 Now, if you want the best of both worlds, 125 00:04:19,709 --> 00:04:21,510 you can use a tool like Zeek. 126 00:04:21,510 --> 00:04:22,890 Zeek is a hybrid tool 127 00:04:22,890 --> 00:04:25,230 that passively monitors your network like a sniffer 128 00:04:25,230 --> 00:04:27,660 and then it can log full packet captures 129 00:04:27,660 --> 00:04:29,970 based on data of potential interest. 130 00:04:29,970 --> 00:04:33,330 Essentially, Zeek is going to sample the data just like NetFlow 131 00:04:33,330 --> 00:04:35,460 and it's going to use a flow collector to do that, 132 00:04:35,460 --> 00:04:38,310 but then Zeek is going to log the full packet capture 133 00:04:38,310 --> 00:04:39,780 whenever it finds something interesting 134 00:04:39,780 --> 00:04:42,090 based on its filters or signatures. 135 00:04:42,090 --> 00:04:43,770 This allows us to go back and analyze 136 00:04:43,770 --> 00:04:46,140 the full packet capture for those items collected 137 00:04:46,140 --> 00:04:48,480 without wasting a ton of storage and processing 138 00:04:48,480 --> 00:04:50,790 on other data that's passing over the network. 139 00:04:50,790 --> 00:04:52,050 Zeek also does a great job 140 00:04:52,050 --> 00:04:54,090 of normalizing all this data for us 141 00:04:54,090 --> 00:04:56,490 by storing everything in either a tab-delimited 142 00:04:56,490 --> 00:05:00,330 or JavaScript object notation or JSON-formatted text file. 143 00:05:00,330 --> 00:05:02,490 Now, once you have this normalized data file, 144 00:05:02,490 --> 00:05:04,140 you can then import it into other tools 145 00:05:04,140 --> 00:05:06,870 for visualization, searching, or analysis. 146 00:05:06,870 --> 00:05:09,090 For example, here I've taken the Zeek logs 147 00:05:09,090 --> 00:05:10,860 and I've imported them into Splunk, 148 00:05:10,860 --> 00:05:13,110 and now I can search for specific information 149 00:05:13,110 --> 00:05:15,750 during my analysis of a potential incident. 150 00:05:15,750 --> 00:05:18,060 Another useful tool when dealing with flow analysis 151 00:05:18,060 --> 00:05:21,720 is the Multi Router Traffic Grapher, or MRTG. 152 00:05:21,720 --> 00:05:23,460 Now, this tool is used to create graphs 153 00:05:23,460 --> 00:05:24,960 that show network traffic flows 154 00:05:24,960 --> 00:05:26,250 through the network interfaces 155 00:05:26,250 --> 00:05:27,840 of different routers and switches 156 00:05:27,840 --> 00:05:30,780 by pulling those appliances using SNMP, 157 00:05:30,780 --> 00:05:32,940 the Simple Network Management Protocol. 158 00:05:32,940 --> 00:05:34,350 This is extremely useful 159 00:05:34,350 --> 00:05:35,820 because we can see patterns emerging 160 00:05:35,820 --> 00:05:38,970 based on the volume of traffic being sent or received. 161 00:05:38,970 --> 00:05:42,030 For example, in the top graph of the router firewall, 162 00:05:42,030 --> 00:05:44,250 where do you see a big spike in traffic? 163 00:05:44,250 --> 00:05:46,020 Quickly, you should be able to identify 164 00:05:46,020 --> 00:05:49,620 there is a big spike in traffic between 2:00 am and 4:00 am 165 00:05:49,620 --> 00:05:53,670 Now, the question is, is this normal or is this suspicious? 166 00:05:53,670 --> 00:05:55,710 Well, maybe and maybe not, 167 00:05:55,710 --> 00:05:58,110 but it's something we should probably go and analyze further 168 00:05:58,110 --> 00:06:00,450 to see why are we seeing this big traffic spike 169 00:06:00,450 --> 00:06:02,460 between 2:00 am and 4:00 am? 170 00:06:02,460 --> 00:06:03,600 This could be something normal 171 00:06:03,600 --> 00:06:05,670 like doing offsite backups to the cloud, 172 00:06:05,670 --> 00:06:07,440 that would be a reasonable explanation. 173 00:06:07,440 --> 00:06:09,450 So, if I was responsible for this network, 174 00:06:09,450 --> 00:06:11,130 I would check with our offsite backups 175 00:06:11,130 --> 00:06:13,200 and figure out when are those performed. 176 00:06:13,200 --> 00:06:16,200 But if I don't do offsite backups at 2:00 to 4:00 am, 177 00:06:16,200 --> 00:06:17,070 then I need to figure out 178 00:06:17,070 --> 00:06:19,500 why is there this big traffic spike there? 179 00:06:19,500 --> 00:06:22,110 Maybe it's because our database is being exfiltrated 180 00:06:22,110 --> 00:06:23,580 due to a malware infection 181 00:06:23,580 --> 00:06:25,740 and it's sending off a copy of our entire database 182 00:06:25,740 --> 00:06:27,990 at 2:00 to 4:00 am each morning. 183 00:06:27,990 --> 00:06:28,823 And that way, 184 00:06:28,823 --> 00:06:30,570 when the system administrator's at home sleeping 185 00:06:30,570 --> 00:06:32,670 all this data is being stolen from us. 186 00:06:32,670 --> 00:06:34,380 Now, by just looking at this graph, 187 00:06:34,380 --> 00:06:35,460 we don't know what the reason 188 00:06:35,460 --> 00:06:37,470 is for this large spike in data transfer, 189 00:06:37,470 --> 00:06:39,930 but we do know we should investigate it 190 00:06:39,930 --> 00:06:41,610 because it could be anomalous, 191 00:06:41,610 --> 00:06:44,070 as you're seeing here the rest of the traffic for the day. 192 00:06:44,070 --> 00:06:46,020 Now, another thing we can do based on this spike 193 00:06:46,020 --> 00:06:48,360 is if we determined it wasn't an offsite backup 194 00:06:48,360 --> 00:06:49,710 and it is suspicious, 195 00:06:49,710 --> 00:06:51,480 we might want to set up a network sniffer 196 00:06:51,480 --> 00:06:52,740 in front of this server 197 00:06:52,740 --> 00:06:55,620 and see what traffic is leaving and where it's going. 198 00:06:55,620 --> 00:06:56,460 Based on that, 199 00:06:56,460 --> 00:06:59,040 we can then dive deeper to see exactly what is happening 200 00:06:59,040 --> 00:07:01,140 and identify other indicators of compromise 201 00:07:01,140 --> 00:07:02,310 based on the PCAP data 202 00:07:02,310 --> 00:07:04,210 and the flow analysis that we collect.