In this lesson,
we're going to discuss how you can troubleshoot network issues
that you may come across in the field.
By having a basic understanding of network troubleshooting,
you can determine whether or not the issue is caused by the
network, your own Linux workstation or server,
or something else.
We're going to cover the basics of network troubleshooting via
the command line inside of Linux inside of this lesson
to help you understand how to diagnose most networking
issues and how to solve them faster
and using a more consistent method
when you find these different network problems.
Things like latency, disconnections,
unresolved host names, and much more.
A lot of the commands that you're going to see here are things
that we've already talked about
back in your A plus or network plus studies.
So I'm going to be sure to cover them all here,
but I won't necessarily dive deep into some of those
commands because you should already know them.
If you need more information in any of these commands,
remember, you can always use man and the command name
to learn more about it from within your Linux system.
Now, network troubleshooting should always begin with a
series of general steps before
you move into more complex techniques.
First,
you want to make sure you check the basics,
like is your device actually plugged in and powered on?
Next,
you want to verify that the network interfaces
are actually configured correctly.
Remember,
they have to have an appropriate IP address configuration,
subnet mass, default gateway value,
and name resolution settings inside of those
network interfaces, otherwise, they're not going to work.
Also,
you want to check that the network interface is actually
detected by your Linux operating system.
And you can use the ETH tool command to gather driver and
other information about that particular device.
Now you can also troubleshoot name resolution issues
using several different techniques
if you're having trouble reaching certain websites.
The first way to do this is to ping a destination
by both its host name and then by its IP address.
If you can ping it using the IP address,
but you can't ping it using the host name,
that means your system is not properly resolving network
names and there is a DNS issue.
At this point,
you're going to use tools like host or nslookup
or dig to test your system's ability to perform name
resolution by querying that DNS server.
If it's your own DNS server,
you might have to go look at that DNS server
and check its configurations as well.
The network itself can also experience issues,
including things like high traffic,
which can cause latency and saturation issues
at various points within your network.
If you use netstat on your Linux system,
you can then determine the number and types of connections
that are being made to your system and see if the latency is
too high or if you're starting to become overly saturated.
Another issue is that your network interface
card itself could be failing
and this can cause data packets
to be dropped on the network,
which causes a loss in data, timeout messages,
or reduced network throughput.
If any of these issues are occurring,
you're going to need to look at whether you need to replace
your network card or check to make sure that the network
itself isn't having a problem
on the other side of the connection,
such as the network switch,
to ensure that's not what's at fault.
Now,
when a developer is trying to troubleshoot application
performance on a Linux system,
they actually have to decide which way
the communication is going to occur.
This can be done using either a local host or Unix sockets
to be able to send data to and from the network
when they design their programs.
Now the first method known as local host
is going to create a full network connection,
and this includes all the TCP error-checking,
so you're going to be able to troubleshoot
the connection much easier.
This method, though,
is significantly slower than our second method,
which relies on using Unix sockets,
also known as Unix domain sockets.
This approach has a much faster performance because it
removes the TCP overhead from this transaction,
but it does result in less detailed error-checking.
Now, unless you're a developer,
you're really not going to mess too much with the local host or
Unix socket's methods because
that's going to be coded into the applications
you're already using.
Now, as you might know, network adapters,
which are also called network interface cards or NICs,
are those devices that provide the
physical connectivity to your network.
These cards typically are going to have an RJ45 connector
if your network relies on UTP or STP copper wiring.
These network interface cards are often found
integrated onto your motherboard
or they can be added to the system using an expansion slot.
If your network card is not being recognized
by your Linux system,
verify that the appropriate driver has been installed so
that Linux can detect the NIC and
take advantage of its features.
All right.
With some of the basics out of the way,
let's go ahead and look at some of the different network
utilities that you can use for troubleshooting.
This includes ping, traceroute, tracepath,
netstat, SS, dig, nslookup, host,
IP, route, nmap, wireshark, tcpdump, netcraft,
iftop, iperf, MTR, ARP, and whois.
One of the first parts of network troubleshooting
is sending test packets between two systems.
This is done using a TCP/IP utility known as ping.
The ping command is going to generate
a response request from the sending computer
and it should receive a reply from the destination computer.
Possible outcomes from the ping command include
receiving a reply from host,
which means the connection was successful.
But if you get a reply as destination unreachable,
this means that your source computer
can't find the path to the destination.
This often indicates there's a problem with your source
computer or the connection along the way
between you and the distant end.
Timeout is another thing you might see and timeout means
that the request reached the destination computer,
but a response did not return to the source computer because
the source computer timed out.
This often indicates that the problem
is with the destination computer or some really bad
slowdowns in your network.
Although using ping is one of the
earliest steps in the troubleshooting process,
it's only going to tell you if something is wrong,
but not exactly what is wrong.
To run the ping command,
you're going to simply type in ping, the options,
and the destination you want to test.
One of the most common ping options
you're going to use is dash C.
Dash C is used to tell you exactly how many ping attempts
you're going to use when trying to connect to the destination.
The dash V option is another one
that you'll commonly see used
and this is to specify that you want to see
the output in verbose mode,
which means you're going to see
a lot more information coming back
that's going to help you in your troubleshooting process.
When you're using the ping command,
you're also going to get a sequence number for the ICMP
sequence for each ping attempt.
The host that sends the ping can use this number to match up
each request with the response it gets back.
If you get mismatched sequence numbers,
this could indicate that there's a drop packet going on
and that means there's instability in the network
or with your network interface card.
The next utility we have is known as traceroute.
The traceroute command is used to report the network path
between the source and the destination computers,
including any routers that the connection uses in between.
The process of a packet traveling across
from one router to the next is known as a hop
and the traceroute command is going to give
an output for each hop along the path.
This is really effective in troubleshooting
your internet connectivity issues
because you're going to be able to figure out exactly which
router things are falling off at.
To run traceroute,
simply type in traceroute, options, and the destination.
Now, similar to traceroute,
there's another utility known as tracepath.
It does much of the same functionality
and it's used in the same way.
To use it, you're going to type in tracepath,
options, and the destination.
A lot of your routing issues are also the result of
misconfigured routing tables.
These issues can usually be fixed by
updating the routing tables themselves.
However,
you first have to identify what's causing the issue.
By using commands like traceroute and tracepath,
you're going to be able to reveal routing issues like routing
loops in which traffic is continuously routed back and forth
between multiple nodes and
never gets to its final destination.
For example,
let's say that node A uses node B as a path to node C,
but node B uses node A as a path to node C.
If the traffic is bound for node C,
nodes A and B are endlessly going to send
traffic between themselves because
each one thinks the other one knows the way to get to node C
and this causes a routing loop.
To fix this,
you're going to have to go into the routing table
of either A or B to be able to fix those issues
and send them along a straight path
instead of this circular path.
Next, we have netstat.
Now the netstat command is used to gather information about
TCP connections on your system.
Depending on the options used,
netstat can inform the user of existing connections,
ports that are set in a listening state on a server,
NIC information, and other things like that.
Some common options for the netstat command
include dash V,
which is going to be used to activate verbose mode,
dash I and interface,
which is used to display information about the network
interfaces or specified interface.
Dash C is an option to continuously print information every
second to continually refresh
what it's seeing inside of netstat.
And the dash L option is going to be used to show only what
ports are set to listening mode on that given server.
To run the netstat command, type netstat,
the options, and then hit enter.
The default output of netstat is going to be presented in a
column-based format with the protocol used by the socket,
the number of processes attached to that socket,
the flies that get further information
about the socket status,
the type of socket access, the state of the socket,
the idea of the process attached to the socket,
and the path of the process attached to that socket.
For example,
you might see something like Unix two
bracket bracket stream connected 472 slash run slash
dbus slash system underscore bus underscore socket.
Now it's important to mention that netstat is installed by
default on most Linux distributions,
but it has been considered deprecated for a little while
now, and instead, it's been replaced with SS.
So, let's talk about SS.
The SS command, which stands for socket state,
is an information gathering utility
that's very similar to netstat and actually replaced it.
The nice thing about SS is it provides simpler output
and syntax than using netstat.
The SS tool can provide information about the established
TCP connections or which ports the
system may be listening on for inbound connections.
This can also be used to help you diagnose problems related
to clients and servers that are
unable to communicate with one another
over a desired protocol.
A missing socket could mean that the service isn't running
and a closed socket could mean that either the client or the
server is prematurely terminating that connection.
Some common options you should be aware of using the SS
utility include dash L.
Dash L is used to show currently listening sockets.
The DST host option is another option you should be aware of
and it's used to show whether the specified host is
connected and what the connection statistics are.
And the dash I option is going to be used to show only what
ports are being listened on on that given server.
To run the SS command,
simply type in SS and the options you want to use.
Now one of the most important network services
that we have is name resolution.
If systems aren't able to reach a DNS server
or conduct name resolution,
they're likely not going to be able to access
a lot of their needed network services
because most things are referred to by their host name or
their domain name and not by their IP address by default.
There are three different tools that we can use
to test and troubleshoot name resolution.
This includes dig, nslookup, and host.
First, we have dig.
The dig command is a powerful tool for
gathering information and testing name resolution
and it's installed on most Linux distributions by default.
The output is going to be displayed in an answer section
and so when you want to do this and figure out what is the
IP address for a given domain name,
you simply type in dig and the domain name.
The command dig at IP address domain name
will resolve that domain name against a given DNS server
that you specify using an IP address.
This can be helpful if you trying to test
if the issue is with your DNS server or DNS at large.
For example,
if I did dig diontraining.com
and it came back as unresolved,
that can indicate a problem with my DNS server.
So, I can then type in dig at 8.8.8.8 diontraining.com
and test my domain name against
the Google DNS server located 8.8.8.8,
which is highly reliable.
If that comes back successfully,
I know it's my DNS server that's having an issue.
The next command we have is known as nslookup
and this is a tool for gathering name
resolution information and testing name resolution.
This is available on most Linux distributions
as well as on Windows and Mac systems.
This command has a noninteractive mode in which you can
provide a domain name in a single command
or you can use an interactive mode
in which you issue the command by itself
and then you enter into a text-based user interface where
you can provide domain names on
separate consecutive prompts to get information.
If you have a lot of work to do with
a domain name server and testing,
I like to go into this interactive mode.
To enter this interactive mode,
simply type in nslookup and hit enter.
The third command we have for DNS
is known as the host command.
The host command is capable of gathering information and
testing name resolution, and again,
it's installed on most Linux distributions.
To use it, simply type in host,
the domain name you want to test,
and the IP address of the DNS server
you want to use to look up that name.
Now, like most things in Linux,
there's lots of different tools to use
to do name resolution, but here,
we're just going to cover the three big ones
which were dig, nslookup, and host.
Next, we have the IP command.
The IP command has replaced the older IF config command for
interacting with your network interface card.
This means the IP command can be used for troubleshooting
as well as for network configuration.
The first steps in the troubleshooting network connectivity
process is to verify that all your settings are correct,
including your IP address.
To check this,
simply type in IP space ADDR, or IP address,
as a way to enable the administrator
to see the configuration that is
currently set for that adapter.
When you do this,
you're going to be able to check the IP address configuration,
the status of the network interface card,
and the usage of the network interface card.
There's lots of other sub commands you can use with IP,
but again, for your basic troubleshooting,
IP addr is going to be your first step.
Next, we have the route command.
This is used to view the routing table on a given machine.
This command is going to be used to manipulate the routing
table and enable the administrator to configure desired
routes statically inside of the system if they need to.
To use the route command,
simply type in route and hit enter
to display the current routing table on the system.
If you want to add a gateway,
you can do that by typing route space add space
default space GW space IP address
and this will configure a default gateway
by its IP address into your routing table.
If you need to add another host into your routing table,
you can do this by typing route space
add space dash host space and the IP address.
If you want to reject or filter traffic
that's destined to a specific address,
you can do that by using the reject sub command.
This will enable an administrator to control the connections
to any given host that's going through this system,
and that way, it will block that route.
Now, to run the route command,
you're going to use the basic format of route
and then the options.
The next tool we have is known as
the network mapper or nmap.
Nmap is a powerful tool for exploring a network environment
and it can be used to identify nodes and be able to report
back on the available services, operating system versions,
host names, IP addresses, Mac addresses,
network devices like switches and routers, network printers,
and other things that it finds when you run nmap.
Nmap is a great command line utility
that is used heavily by cybersecurity analysts
as they're doing vulnerability scans across a network.
To run nmap,
simply type in nmap, the options,
and the targets that you want to scan.
Next, we have wireshark.
Wireshark is a very popular packet sniffer
and network analyzer.
Network analyzers are basically used
to intercept and potentially read network traffic
and these tools can be used for
eavesdropping attacks by attackers,
but it can also be used by us for network troubleshooting.
When network traffic is being intercepted,
we're going to get information about it such as the source and
destination Mac addresses at layer two,
the source and destination IP addresses at layer three,
the port numbers and the packet payload data
and all of that can be seen inside of wireshark.
Now an advantage of a tool like wireshark is our ability to
see exactly what packets are moving through a given network
segment or across a given network interface card
and what packets aren't.
And so this can be really helpful
when you're doing your troubleshooting.
Wireshark is a graphical user interface-based tool,
so you have to have a graphical environment
to be able to run it.
On the other hand,
if you're using a command line environment only,
you can use the network analyzer known as tcpdump.
Tcpdump was created back in 1987
and it still remains one of the most
popular packet sniffers available today.
It's installed by default on many Linux distributions, too,
making it an easy go-to tool for you to use
in your command line.
Now, users can determine the exact traffic type and content
they want to see using this command
and setting up different filters.
It works very similar to wireshark,
but you can use it in the command line environment.
Now, some common options you're going to see with tcpdump
include dash I,
which specifies the interface you want to listen on,
dash N,
which is going to be used to not resolve host names and show
everything as IP addresses,
and dash V,
which is used to tell the system to operate in verbose mode
and give you a lot more errors and details
that you can go through.
To run the tcpdump command, simply type in tcpdump,
the options dash I and the
interface you want to listen on, and host,
and the IP address that you want to look at.
Next, we have netcat.
The netcat command can be used to test
connectivity and send data across network connections.
This command may be spelled out as netcat
or abbreviated as NC depending on
the distribution of Linux you're using.
Systems may also be identified by their
IP address or their host name,
depending on how you want to connect to them.
When you're troubleshooting,
you can use netcat to listen on the destination computer
and then attempt a connection from a source computer
in order to verify that network functionality exists between
those two systems.
Netcat can also be used to send files to
and from a remote machine.
To use netcat, simply type Netcat and the options
or NC and the options.
Next, we have iftop or if top.
The iftop command displays bandwidth usage information
for the system and helps to identify
whether a particular network interface card
or protocol is consuming a lot of bandwidth.
This command can be used to identify why a particular link
may be slowing down by showing you
the traffic on that connection.
And you can use it to check to see
what's consuming the most bandwidth on a given interface.
Network slowness is often a symptom of bandwidth saturation
and this occurs when a network's link capacity
is being exceeded.
For example,
all the bandwidth is being used up because you're streaming
too many videos over a small internet connection.
This can lead to degraded network performance
or even service outages.
With the iftop command,
you can investigate any NIC on a network link that you
suspect may be sending or receiving excessive amounts of
traffic across that link.
To use iftop, type iftop, options, dash I,
and the interface you want to interact with.
Next, we have iperf.
The iperf command is used to test the maximum
throughput of an interface across a network.
To use this, though,
you have to install it on two systems.
One is going to be the sending machine
and one's going to be the receiver.
Essentially, we're going to have one acting as a server
and the other one as a client.
Then, the iperf client is going to be sending
data over to the server and the server
is going to calculate how much and how fast that
data has been getting there.
This will help you ensure that the throughput that you're
expecting is actually what you're receiving.
To run the iperf command,
type in iperf dash C if you're running it on the client
or iperf dash S if you're setting it up on the server.
Now, both the iftop and iperf utilities
are really here to measure throughput,
and there is a difference between throughput and bandwidth.
Bandwidth is a potential amount of data that you could move
through a network connection in a given amount of time.
It's theoretical.
But throughput is the amount of data that is actually moving
through the network.
And that's why we use utilities
like iftop and iperf to measure that actual throughput to
see where the slowdowns exist on your given network.
For example,
let's say you just went and ran all new cabling for your
network using one gigabit per second Cat6 cabling.
That's great,
but if you're only getting 100 megabits per second,
that means something is wrong, and by using these tools,
you can identify where that's happening
inside of your network,
between different network segments
or different network devices.
Next, we have MTR.
Now, the MTR utility is really
a combination of ping and traceroute,
but it has additional improvements that enable testing the
quality of a network connection, too.
The MTR command can also help you identify if there's
packet losses or packet drops in your network.
This can happen if one or more packets sent from a source
aren't able to reach their intended destination.
And this packet loss can actually cause latency because
those packets are being queued
for retransmission using the TCP protocol.
If you're using UDP,
that data will not be resent
and that means the data is not
going to be successfully transmitted at all.
If you have a large number of lost packets,
this is really an indicator that you have a network issue
along your path.
By identifying that the issue exists,
we can then figure out where in the path it exists by
using MTR, or traceroute, or other things like that.
MTR will enable an administrator to find
all those potential failures of those networking components
and then figure out what percentage of packets are being
dropped along the way at which points in the network.
This will help us identify exactly which router
or switch we need to look at first.
To be able to run MTR, simply type in MTR,
the options, and the host name that you want to reach.
Next, we have ARP.
ARP is the address resolution protocol
and this is used to relate IP addresses to Mac addresses and
do the conversion from layer two to layer three
or layer three to layer two,
depending on if you're doing ARP or reverse ARP.
There's also an ARP command that
administrators are going to run
and that will help you discover information
about known Mac addresses on a given system.
If a computer has cached incorrect or outdated information,
connectivity problems can happen and you can lose your
connectivity to particular nodes.
The ARP cache can be cleared
as part of the troubleshooting process too
using the ARP command.
For example,
if you want to delete a single address from your ARP table,
you can do that by running ARP dash D and the IP address to
clear the entry for that IP address
that matches a given Mac address.
Then, you could try pinging the host again
and seeing if that fixes it.
To be able to see your entire cache,
simply type in ARP dash A
and this will display all of the things in your cache.
Anytime you want to run ARP,
simply type in ARP and the options you want to use.
Next, we have whois.
The whois command is going to provide information
on the internet DNS registrations
for different organizations.
This is useful if you're trying to learn or verify
information regarding the ownership of a given domain name,
including the contact information for an organization.
To run whois, simply type in whois,
the options, and the domain name,
such as whois deontraining.com.
Remember, when you're troubleshooting networking issues,
you need to narrow the scope of the problem
and then verify the IP address configuration on your system
using the IP addr command.
If the system has an IP address beginning with a 169.254
dot something dot something,
that means it failed to get a lease from the DHCP server
and it doesn't have a valid IP address
because that is an IP pop.
In this case,
you need to go troubleshoot your DHCP server or your
connectivity between your system and the DHCP server.
After that,
you can attempt to renew your IP address using DHCP to see
if you can then connect to the DHCP server and get a new IP.
If you're using a static IP address,
always make sure your static IP is properly typed in
and that you don't have any typos or errors in it.
If you want to test your latency, you can use tools like ping.
If you want to test the connectivity from end to end,
you can use ping or traceroute.
If you need to test your bandwidth versus your throughput,
you can use tools like iftop and iperf to test that
connection from end to end and verify that you're actually
getting the throughput you'd expect based on the bandwidth
your system was designed for.
If you're not,
it could be an issue with the interface itself.
Maybe you're only using a 100 megabit per second
network interface card,
but you're expecting to get
one gigabit per second of traffic
because you're on a Cat6 network.
Those two things aren't going to work.
To be able to make it go to one gigabit per second,
you're going to have to replace that network interface card
with a one gigabit adapter.
This is the idea of how to use these general commands that
we talked about in this lesson to test your connectivity,
your latency, your bandwidth and throughput,
as well as your name resolution
to solve the issues you're seeing
in a given troubleshooting scenario.