In this lesson,
we're going to discuss NetFlow and Flow Analysis.
Now, as you've probably guessed,
packet captures can take up a lot of storage space
and they can quickly grow in size.
For example, if I wanted to conduct full packet capture
on my home network each day,
I would need several gigabytes of storage
just for my small family
because every single packet that goes in and out of my house
would have to be captured by that network sniffer.
Every video game my son is playing online,
every YouTube video he's streaming,
every Netflix show that my wife is binging,
all of that is going to be captured
frame by frame and bit by bit.
This is known as a full packet capture.
Now, with full packet capture, also known as FPCs,
you're going to have to capture the entire packet,
including its header and its payload.
Because full packet captures
take up so much storage space though,
there are many times that we don't want to collect it all
because this can be a waste of a lot of resources for us.
Instead, we can opt to use flow analysis
to collect the most important details
about the traffic being sent.
Now, flow analysis relies on a flow collector,
which is a means of recording metadata and statistics
about network traffic
rather than recording each and every frame
that's passing over the network.
The flow collector can then sample the network traffic
and provide statistics to our flow analysis tools
for us be able to observe.
By collecting this metadata about the data itself,
we can save a lot of storage space
and minimize the processing costs as well.
Now, the downside of using flow analysis
instead of full packet capture though
is that we won't have the contents
of what we sent over the network,
we only have information about that data.
For example, if I had a list
of every phone call you made today,
I would know who you called, when the call started,
how long you talked for, and when the call finished,
but I wouldn't have any idea
of what you said during that call.
That's the idea of metadata
when we collect things about the information,
and so this kind of metadata we collect
when dealing with flow analysis
is going to be dealing with what kind of traffic was sent,
if it was HTTP, HTTPS, FTP, SSH,
or some other protocol, and how much of it.
Now, all the flow analysis information that's gathered
can then be stored inside of a database
and queried using different tools
to produce reports and graphs.
Flow analysis is wonderful for highlighting trends
and patterns in the traffic observed,
and it can create alerts
based on anomalies observed
in those traffic patterns as well.
Flow analysis can also quickly create visualizations
to map out the different network connections,
the traffic types,
and the volume of each session or connection.
This can be really useful
in identifying different traffic patterns
that might reveal bad behavior,
such as malware in transit, data exfiltration,
or other suspicious or malicious activity.
NetFlow is a Cisco-developed means
of reporting network flow information
to a structured database.
This was actually one of the first ones out there,
and eventually it became the standard
that everybody started using
under the name IPFIX, or IP Flow Information Export.
Now, IPFIX is going to be used
to define a particular traffic flow
based on different packets
that share the same characteristics.
For example, we could set up a traffic flow
based on the same source and destination IP
that's being used to signify
there is a session between two different hosts.
Now, this is the data flow between two hosts,
and we can gain some information about it.
Then, we might have a second data flow
for anything communicating with a particular domain name
and we can collect information about that data flow too.
Now, as we look at the data collected by NetFlow,
we're going to find information
about the packets that was sent,
including the network protocol interface being used,
the version and type of IP being used,
the source and destination IP addresses being used,
the source and destination ports being used,
or the type of service being used.
All this information can be gathered
and then analyzed visually using these different tools.
For example, here on the screen
you can see how SolarWinds tools
are used to display the collected NetFlow data of a network.
Similarly, you can look at it in a more text-based manner
by listing out the data flows as 1
and going all the way down to flow 15
in this little snippet.
Here, if you wanted to see what each one
had inside of its flow,
you would click the plus sign like I did next to flow 15,
and you could see the source and destination addresses,
the source port and destination port,
and some basic information about this particular data flow.
Notice though, you're not getting
any of the actual content of the packets
that were inside of that traffic flow
because NetFlow only shares with you metadata
about the data.
This is because you didn't collect the full packet captures.
Now, if you want the best of both worlds,
you can use a tool like Zeek.
Zeek is a hybrid tool
that passively monitors your network like a sniffer
and then it can log full packet captures
based on data of potential interest.
Essentially, Zeek is going to sample the data just like NetFlow
and it's going to use a flow collector to do that,
but then Zeek is going to log the full packet capture
whenever it finds something interesting
based on its filters or signatures.
This allows us to go back and analyze
the full packet capture for those items collected
without wasting a ton of storage and processing
on other data that's passing over the network.
Zeek also does a great job
of normalizing all this data for us
by storing everything in either a tab-delimited
or JavaScript object notation or JSON-formatted text file.
Now, once you have this normalized data file,
you can then import it into other tools
for visualization, searching, or analysis.
For example, here I've taken the Zeek logs
and I've imported them into Splunk,
and now I can search for specific information
during my analysis of a potential incident.
Another useful tool when dealing with flow analysis
is the Multi Router Traffic Grapher, or MRTG.
Now, this tool is used to create graphs
that show network traffic flows
through the network interfaces
of different routers and switches
by pulling those appliances using SNMP,
the Simple Network Management Protocol.
This is extremely useful
because we can see patterns emerging
based on the volume of traffic being sent or received.
For example, in the top graph of the router firewall,
where do you see a big spike in traffic?
Quickly, you should be able to identify
there is a big spike in traffic between 2:00 am and 4:00 am
Now, the question is, is this normal or is this suspicious?
Well, maybe and maybe not,
but it's something we should probably go and analyze further
to see why are we seeing this big traffic spike
between 2:00 am and 4:00 am?
This could be something normal
like doing offsite backups to the cloud,
that would be a reasonable explanation.
So, if I was responsible for this network,
I would check with our offsite backups
and figure out when are those performed.
But if I don't do offsite backups at 2:00 to 4:00 am,
then I need to figure out
why is there this big traffic spike there?
Maybe it's because our database is being exfiltrated
due to a malware infection
and it's sending off a copy of our entire database
at 2:00 to 4:00 am each morning.
And that way,
when the system administrator's at home sleeping
all this data is being stolen from us.
Now, by just looking at this graph,
we don't know what the reason
is for this large spike in data transfer,
but we do know we should investigate it
because it could be anomalous,
as you're seeing here the rest of the traffic for the day.
Now, another thing we can do based on this spike
is if we determined it wasn't an offsite backup
and it is suspicious,
we might want to set up a network sniffer
in front of this server
and see what traffic is leaving and where it's going.
Based on that,
we can then dive deeper to see exactly what is happening
and identify other indicators of compromise
based on the PCAP data
and the flow analysis that we collect.