In this lesson,
we're going to discuss how you can troubleshoot process issues
within the Linux operating system.
While running processes within the Linux operating system,
your users may experience technical issues.
To provide uninterrupted services to your users,
you may need to be able to solve problems
that are going to come up during operations.
As an administrator managing multiple systems on a network,
you're going to have installed and managed
various services and packages
that are required for your users to do their job.
However, there's sometimes going to be instances
where applications and services
do not function as they're supposed to.
As a system administrator,
you're going to be expected to determine
and resolve those problems.
Now there are many different issues that could affect
or be caused by a process of running instance of software
or something else like that.
These issues can cause instability when processes hang.
They can consume resources
that should be allocated to other processes
and it can create a general sluggishness to your system
and other issues like that.
Processes move through a life cycle
from creation until their eventual termination.
Now there are five different states that a process can be in
during this life cycle.
Each one defines how the system or other applications
can interact with that given process.
Knowing a process's state
can help you determine what to do with that process
or it can help you diagnose problems
concerning that process.
The running state means the process is currently executing
in the user space or the kernel space.
The interruptible sleep state means the process
relinquishes access to the CPU
and is waiting to be reactivated by the scheduler.
The interruptible sleep state
means that the process will only wake up
when its resources that it's waiting for
become available to it.
Otherwise, it's going to stay in a sleep state.
Then, we have the zombie state
and this indicates that a process was terminated
but it has not yet been released by its parent process
so it cannot accept a kill signal.
The final state is a stop state
and this state indicates that a process
was stopped by a debugger or through a kill signal.
Now every process on your system
has a unique process ID known as a PID
and it gets this when it started up by the system,
and that way, users and the system
can identify that process.
The PID is a non-negative integer that increases
for each new process that's being started on the system.
Now the init daemon always has a PID of one
because it's the first process to start up on the system
and it becomes the parent for all other processes
on that system.
After this one is started,
then processes are going to be able to be started up
by the system or the user
by that init daemon,
and they're going to be assigned a higher available number
because they're going to be something higher than one
because they're started by the init daemon.
When you're troubleshooting something,
you need to know what the process's PID is
or that identification number
because this helps you understand how to terminate it,
change its priority or perform other management tasks
on that given process.
Now if you want to find that,
you can use the pgrep command.
The pgrep command
looks through the currently running processes
and lists out the process IDs
which match the selection criteria
to your standard output,
similar to how grep uses it
to find matching patterns inside of a text file.
If you use pgrep,
you're going to be able to identify a process
based on multiple different factors
even if you don't know its exact PID.
This will help you identify the PID
and identifying the PID is typically your first step
in managing a process.
To run the pgrep command,
simply type in pgrep, the options,
and the pattern you're searching for.
Another way to find the process idea of a running program
is to use the pidof command,
pidof, spelled P-I-D-O-F,
is going to allow you to find the process ID
of the named program that you give it.
To use it, you're simply going to type pidof,
and then the name of the program.
For example, if you are running something
like the ls command,
you could run pidof space ls, Enter,
and then find what was the PID
associated with the ls command.
Another command you can use is known as ps.
The ps command invokes the process table
which is a record that summarizes
the current running processes on a given system.
When the command is run without any options,
it's going to display all the processes
that are run by the current shell with the details.
This includes things like the PID,
the terminal associated with that process,
the accumulated CPU time,
and the command that started that exact process.
However, different options may be used
along with that command
to filter out the displayed fields or processes too.
The a option, for example,
will list all user-triggered processes.
The -e option is going to be used to list
all of the processes.
The -l option will list the processes
using a long-listing format.
And the u option is going to be used to list out processes
along with the username and the start time.
The r option is going to be used to exclude processes
that are not running currently.
And the x option is used to include processes
without a terminal.
The T option is going to be used to exclude processes
that were started by any terminal
other than the current one.
And the dash -U option with the username
is going to be used to display all the processes
based on a specified user.
The -p and PID option will display only the process
associated with the specific PID.
The -C command option is going to display all the processes
by their command name
while the --tty internal number option
is going to be used to display all the processes
running on a specified terminal.
Like ps, the top command is going to list
all the processes running on a Linux system.
It acts as a process management tool
by enabling you to prioritize, sort,
or terminate processes interactively.
It's going to display a dynamic process status
and reflect real-time changes.
Different keystrokes within this tool are going to execute
various process management actions as well.
Some of the frequently used command keys
include the Enter key.
And when you press that,
it's going to refresh the status of all your processes.
If you press Shift + N,
you're going to sort the processes
in decreasing order of their PID.
If you press the M key,
it's going to sort the processes by their memory usage.
If you press the P key,
it's going to sort the processes by CPU usage.
If you press U, it's going to display the processes
belonging to the user specified at the prompt.
If you press K,
it's going to terminate the process
for which you specify the PID for.
If you press R,
it's going to alter the scheduling priority
of that process and you have to specify this by its PID.
And if you press Q,
it's going to exit the process list.
To run the top command, simply type in top,
and your options at the command prompt.
Now in addition to top,
there's a newer version called htop,
htop is also an interactive, system monitor,
process viewer and process manager
but it really does a lot more
and in a lot easier way of using it than does top.
This is because htop was developed back in 2004
where top was developed all the way back in 1984.
Now the one drawback is that htop
is not installed by default on all Linux distributions.
So if you want to use htop
and it's not already installed on your distribution,
you're going to have to install it using a package manager
or compiling it from the source code
and installing it on your Linux server.
Now really, when it comes down
to some of the big differences between top and htop,
it really comes down to how it presents the data.
When you're using top,
it's a very basic black and white interface
and it's text only.
But when you start using htop,
you're going to get a more colorful interface
with some text-based graphics
that'll depict exactly what's going on.
For example, at the top of the screen in an htop display,
you're going to see a graphical representation of the CPU,
memory, and swap space
and how much is being used and how much is free.
Then, it will go on to show you all the different processes,
the users that are running those processes,
and all the information you're going to want to know
about those processes on your given system,
and it does this in a color-coded way,
so it becomes really easy
to quickly pick out the information you're looking for
when you're using htop instead of top.
Another command we want to talk about is systemd-analyze.
Now the systemd-analyze command
is used to retrieve performance statistics
for boot operations.
The command takes one or more sub commands
that determine what type of information is going to be printed
and how it's going to be printed.
Now for process management and troubleshooting,
blame is going to be the most relevant sub command.
This sub command will print a list of all the systemd units
that were executed at boot
along with the time it took for each unit to execute.
You can then use the systemd-analyze blame function
to identify services and other units
that make the system really slow during the bootup process
To run the systemd-analyze command,
simply type in systemd-analyze,
the options and the sub command.
The list of open files or lsof command
is going to print out a list of all the files
that are currently open
to all your active processes on the system.
This can include everything from a text file
to a device file.
Basically, any object on that system
can be parsed as a file.
Now sometimes, you're going to be prevented
from modifying a file
if it's already open by another process.
So you can use the lsof command
to identify the offending process
that you need to terminate so you can take over that file.
You can also use the lsof command
to analyze how a process uses files
which can be really helpful
if you're identifying malicious processes
or processes that have unwanted side effects
in a given system.
The lsof command or list of open files
will print each file that's opened by a process
to its own line on the screen.
It's going to print information
such as the name of the command or process,
the PID, the invoking user,
the file descriptor,
and what kind of permissions the file has been opened with,
as well as the type of file and the name of the file.
To run this command,
simply type in lsof
and the options at the command prompt.
Now another thing that the operating system does for us
is it schedules out those processes
to give each one some CPU time.
This is known as the scheduler.
Now the scheduler is usually really effective
at assigning CPU time
but not every decision it makes
is going to be considered optimal.
In some cases, you may need to manually tell your CPU
to prioritize certain processes over others.
Now processes are going to be prioritized
based on a given number that's associated with it,
anywhere from -20 to 19.
This value is known as the nice value or the niceness value.
If you have a lower number,
it means you have a higher priority.
For example, if you have a process
with a nice value of -18,
it's going to get more CPU time
than a process with a nice value of 15
because -18 is much smaller than 15.
Now a process is going to inherit a nice value
from its parent as well.
And by default, the value it's always going to get is zero.
The nice command will enable you to run a command
with different nice values than the default one
it's going to be getting.
To do this, you're going to use the -n option,
and then specify a number.
If you don't specify an integer,
it's going to go ahead and assume you wanted a 10 increment.
Now by running nice without any options,
you're going to see a default nice value.
You must be the root user or have root user authority
to run this command at a higher priority
if you want to be able to change people's values.
To use this, type in nice -n,
the nice value increment you want
and the command you want to change.
Now the nice command is usually going to be used
to start a new process at a given priority
but there's another command out there
if you want to change an existing process
and its nice value,
and this is known as renice,
renice is the command to enable you
to alter the scheduling priority
of an already running process.
You're going to use the renice command
with the -n option to specify your new nice value
that you want that process to have.
Now if you want to, you can also renice
not just a single process but an entire process group,
and you do this using the -g option.
This will cause all the processes in a given process group
to have their nice values altered
based on what you told it.
Also, if you want to do a renice
against everything with a user,
you can do that too by using the -u option.
This will alter the nice values of all the processes
that are owned by a given user.
To use renice,
all you have to do is enter renice -n,
the nice value, the options,
and the identifier on the command line interface.
Now when we're working in the command line,
we're usually working in Bash,
and with Bash, most of your commands
are executed in the foreground by default.
The problem with this is that your command prompt
can become consumed and unusable
until that command finishes processing.
Now for simpler commands
like the date command or the copy command,
or the list command, this isn't really a problem,
but if you're running a longer script or a longer command,
this can start becoming a real issue for you
and it can become possible where you have these commands
that are just taking over your command prompt
and you can't get a new command in
because you're waiting for it too long.
So one of the things you can do
is you can actually run the commands in the background,
and this way, you can execute them
but not consume your shell.
To do this, you can use the fg or foreground command
or the bg, background command,
and this will move the processes
either into or out of view.
Now there are a lot of different options you can use
with both the foreground and background processes.
The foreground or fg can be used with percent,
and then a job ID to bring a job back to the foreground.
If you use Ctrl + Z,
this will halt a job temporarily
so you can use the background command
or bg and a percent sign with a job ID
to push that job to the background.
And if you use the ampersand sign,
this will actually start the command
running in the background
when it's added at the end of the command.
For example, if I use the date command,
I could put date, ampersand and hit Enter,
and it would run the date command in the background.
Now the foreground and background command
are actually part of a larger job control command function.
The main function for this is known as jobs.
If you enter jobs with a command prompt,
it's going to list out all of those jobs you have
either in the foreground or the background.
This gives you a master list,
and then you can then select which ones you want
by using the fg or bg commands to bring them forward.
If you want to stop a foreground job
and place it in the background when you're working in it,
you can simply type in Ctrl + Z.
If you're working in the command line environment
and you're running a program that you want to stop,
you can force quit it by using Ctrl + C
at the command line.
If you want to log out of your current user session,
you can do that by using Ctrl + D.
All of these are shortcuts
for you to be able to go ahead
and move in or out of a process
as you need to when you're working in the jobs
or the command line interface.
Now some other commands we have that are important
are things like nohup and no hangup.
Now nohup or no hangup
is a command that's going to prevent a process
from ending when the user logs off.
For example, let's say an administrator
launched a backup script,
and then logs off the system.
By default, that script would actually stop running.
That would be bad because it means our backup
is going to stop running.
But if we use nohup in front of the normal command,
like nohup.backup script,
that script will continue to run
even after the administrator logged off.
So to run this,
simply type in nohup and the command or script name.
Different commands are also going to be used
to send signals to processes
to either terminate or kill them
when you don't need them anymore.
This is necessary when a process hangs,
which causes a system to become unstable,
or it fails to relinquish control over a file
that you're trying to modify.
A simple way to get this taken care of is by issuing
the kill command.
The kill command is going to send any specified signal,
or by default determination signal,
to one or more processes.
To run this, type in kill,
the options and the process ID.
Another variant of this is known as pkill.
Now the pkill command is going to send a specified signal,
or by default determination signal,
to a process based on a pattern.
Now for example, let's say you start top in one terminal,
and then you issue pkill top in another terminal,
you're going to see that top terminates.
This is because the word top match the pattern of T-O-P
or the top process.
So if you want to run a command like this,
simply enter pkill, the options,
and the pattern you want to terminate.
On the other hand, the killall command
will send any specified signal,
or again by default determination signal,
to all processes that are matching the name specified.
To run this command,
simply type in killall,
the options and the process name.
Now you can also use the kill command
with the process table to end processes
and you can do this by entering kill,
followed by the PID
for that specific process that you want to terminate.
However, the process table may display processes
that do not belong to you.
And as a user, you can only kill things that you own.
As root though, you can kill
any process you want on the system
and there are many different ways to kill a process,
like I just mentioned,
and each one of them will be mapped to a signal.
Now the signal determines how to actually kill your process.
By default, we use the termination signal.
For example, if you want to terminate a process gracefully,
you can give it time to clean up any last minute activities.
However, some processes will ignore this signal
or they'll simply fail to terminate
in a reasonable amount of time.
For those processes,
you may have to issue a more aggressive signal.
Now there are many different types
of kill signals out there.
Each signal has a name
and one or more corresponding number values
and you can use them with the kill command.
The SIGHUP signal has a value of one,
and it's sent to a process
when its controlling terminal is closed.
If that terminal is closed or disconnected,
it's said to have hung up,
which is why this signal is called SIGHUP,
or S-I-G-H-U-P, which stands for signal hang up.
This SIGINT signal is going to have a value of two
that interrupts a process from the terminal
and it enables it to end gracefully.
The signal can be caught or ignored.
This is basically the same thing
as pressing Ctrl + C at a terminal.
A process might change a shortcut behavior over,
and in those cases, you can use SIGINT
Now, the SIGKILL is going to be a signal with a value of nine
and it kills the process immediately.
The signal cannot be caught or ignored.
This is typically used as a last resort.
Another one is SIGTERM.
This is a signal with a value of 15
and it terminates a process
and enables it to end gracefully.
This signal can be caught or ignored
and is typically sent by the kill command
or another program and not from the terminal.
The SIGSTOP signal is going to have a value
of 17, 19 and 23,
and it pauses a process.
This signal cannot be caught or ignored by the program
and this is typically sent from kill or another program
and not by the terminal itself.
The SIGSTP signal has a value of 18, 20 and 24,
and it pauses a process from the terminal.
This signal can be caught and it can be ignored.
This is the same as pressing Ctrl + Z at your terminal
and a process might change this shortcut behavior, however,
in which case you would use
the sigstop instead or SIGSTP.
So let me give you a couple of examples
to show you how this works.
Let's say I had a process that had a process ID of 921.
Now if I wanted to kill that,
I could send the kill signal
by using kill 15 921 at the command prompt.
And this would then send the signal of 15
to kill process 921
which is the signal termination signal.
Now if I wanted to use signal 17,
I could enter kill 17 921
and this would pause that process
instead of removing the process entirely.
Now most of the time,
people who use Linux prefer to do their own troubleshooting
and solving problems becomes an art and a science.
Solving technical problems,
such as those that occur with computers
requires a great deal of specialized knowledge too.
We may use guidelines
when troubleshooting our process issues
and troubleshooting begins
with the identification of a problem
and it doesn't end until the services have been restored
and the problem no longer is adversely affecting our users.
Remember, you want to gather information about a process
including its process ID or PID and the state
when you start to troubleshoot these process issues.
You can use the ps command to print information
about all your running processes
including CPU utilization.
For example, if you have a process
that's consuming over 50% of the CPU,
it may cause performance issues across the entire system
and it becomes worth investigating.
If you're using the top command,
you can retrieve a dynamic and interactive list
of all your running processes.
If you use the systemd-analyze blame command,
you can determine what startup processes
are slowing down the boot operations.
If you use lsof,
you can identify which processes
have files that are open right now and in use.
If you use nice and renice,
you can prioritize certain processes over others.
If you use fg and bg,
you can measure foreground and background processes.
And if you use nohup,
you can keep a command running even after you log off.
Finally, if you use kill and its associated commands,
you can terminate problem processes
and restore the system disability.
Each time you solve a problem,
you're going to increase your troubleshooting skills
by gaining more experience.
And as you start to learn
how to combine these different steps or skip steps,
you can start reaching a solution much more quickly.