In this lesson, you're going to be introduced
to some common CPU and memory issues
that occur within Linux,
as well as important command tools
that you can use to troubleshoot issues
that you may find along the way.
After all, sometimes unexpected behavior
on the server side can occur
and this can be caused
by resource limitations on your system.
Linux, by design,
aims to use all of the available physical memory
as efficiently as possible.
But CPU and memory are two fundamental computing concepts
and you need to be able to manage these crucial elements
of a Linux system to ensure optimal performance
while supporting the best user experience.
There are many possible issues that could affect
or be caused by the CPU.
These issues may include an underperforming CPU
for the desired level of speed and responsiveness.
An overloaded CPU caused by too many requests resulting
in a delay of response, or you can have non-functional cores
or other things that are causing problems.
You can use the information
from /proc/cpuinfo to identify characteristics
about your CPU that might indicate issues related
to performance or a lack of support
for specific features you might need.
Some useful fields in the /proc/cpuinfo file
include the processor,
which refers to the number of logical cores,
the vendor ID, which refers to the CPU manufacturer,
the model name, the CPU megahertz,
which refers to the logical core's clock speed
and is measured out to the thousands decimal place.
The cache size and the flags
which refer to the characteristics
about the CPU as well as the supported features.
As you've seen,
the system CTL command enables you
to view kernel parameters at runtime.
The uptime command will display the time
from when the system started running as well.
The output of the uptime command gives information
about the current time,
how long the system has been running,
and how many users are currently logged in.
Most relevant to the CPU troubleshooting, though,
is the load average field.
A CPU's load is going to be expressed
as the number of processes
that are either using or waiting to use that CPU.
It can also include the number of processes
in queue for storage input/output.
Using uptime, you're going to be able to find the average load
over three different periods of time.
You can use these values to determine
when your CPU has become overloaded,
which may lead to performance issues.
For example, if you have four logical cores
and the load average of the last 15 minutes is 4.0,
then that means your CPU was being fully utilized.
If you're load average were 8.0,
that means your CPU would be 100% overloaded.
Now, one of the commands we can use
to look at our system reports is known as sar, S-A-R.
sar is going to display system usage reports based
on data collected from system activity.
These reports contain various different sections,
and each of those sections consists of the type of data
and the time at which that data was collected.
The default mode of the sar command
is going to display the CPU usage
in various time increments for each category
of resource that's accessed by the CPU.
This includes your users, the system,
input/output scheduling, and other things.
It also displays the percentage of the CPU
that was idle at any given time.
At the bottom of the report is an average
of each of these data points across the listed time periods.
By default, sar reports are going to have data collected
every 10 minutes.
Though, you can use different options to filter
and shape these reports as well.
To use this command,
simply type sar and the options you want.
The next command we're going to talk
about is sysctl or system control.
The system control command can be used
to troubleshoot CPU issues as well
by retrieving CPU-based kernel parameters at runtime.
One useful set of parameters concerns scheduling domains.
Now scheduling domains are a method
by which the kernel groups can take logical cores together
and then share scheduling policies
and other properties between them.
These parameters will usually take the format
of kernel.sched_domain.cpu#
.domain#.param.
Now another way to get information
about your CPU and its architecture is
to use the lscpu command.
lscpu is a command line utility
that's going to read the CPU architecture information
from the sysfs and /proc/cpuinfo files
and then print that to your terminal.
The information here is going to include things
like the number of CPUs, the threads, the cores,
the sockets, and non-uniform memory access, or NUMA nodes.
It's also going to be able to display CPU cache information
and cache sharing as well as family, model, bogomips,
byte order, and stepping of the CPU.
Now, in addition to all the issues you can have
with your CPU, you can also have issues caused by memory.
Now, memory issues may include not having enough
total memory to service all your processes at once
or not having enough free memory
to service new processes that need to be launched
or processes that are unable to access memory,
despite there being memory available
or processes accessing too much memory
and basically draining your system
or you can have things
that are leaving other processes without memory
because they're sucking up all that extra memory.
Or we can have a system
that cannot quickly access files from a cache
or a buffer because there's not enough physical memory
and things are being put into a cache or buffer instead.
And lots of other memory issues like this.
Now to troubleshoot some of these issues,
you can use information from the /proc/meminfo file.
Now this will ensure
that the system's RAM modules are performing
to the specification you expect,
and that the OS is consuming memory
at the right rate that you expect
and that your system has enough overall available memory
to perform the tasks you want.
Now, some useful fields
in those /proc/meminfo file include things like memtotal,
which refers to the total amount
of physical memory in the system
or memfree, which is the total amount
of free memory that's available in that system
at this time.
Cached, this is going to refer
to the total amount of physical memory
that's being used as cache memory.
Swaptotal, this refers to the total amount
of swap space on your system.
And swapfree, which refers to the total amount
of swap space that is currently unused on your system.
Dirty refers to the total amount
of memory that is waiting to be written to storage
and writeback refers to the total amount
of memory currently being written to storage.
Now the free command is another command we can use
to look at our memory
and free is going to parse the /proc/meminfo file
for easier analysis for those memory usage statistics.
Its default behavior is to display the information
about system memory and the swap space
as well and include the total memory,
the total used, the total free,
the total shared, the total buffered and cached,
and the total available for starting new apps.
There are several different command options
that you can use with the free command.
Things like -b, -k, -m,
g, or -tera.
And all those options will display the memory
in either bytes, kilobytes, megabytes, gigabytes,
or terabytes, respectively.
If you use -s and the number of seconds you want,
it will tell you how quickly
you want to update the memory statistics based
on the number of seconds you tell it to.
So you might say -s10 to do it every 10 seconds.
o is going to disable the display
of the buffered or cached information.
t is going to display the total line
that's going to combine physical memory
with swap space as one combined memory area.
And -h will give you output that is more human readable.
Now memory can also be cached
and this means it's going to be stored temporarily
so that the data can then be accessed more quickly
in the future.
Now, the buffers field
inside the /proc/meminfo file indicates memory
that has been assigned to a specific block device,
like a hard drive.
This memory can then be used to cache file system metadata
like directory contents, permissions,
and other things like that.
The cached memory is going to be similar to this
but instead of storing file metadata,
it's going to store the actual contents of the file.
So the buffers store metadata
but cache is going to store the actual files.
Now, when you use the free command,
we can combine these two values together
inside of our output as well.
Another way to look at the information
about your memory is to use the lsmem command.
The lsmem command is going to list the ranges
of available memory with our online status.
This will show you all the different memory blocks
that correspond to the memory blocks inside of the sysfs.
This command will also show the memory block size,
the device size,
and the amount of memory in online or offline statuses.
Another command we need to talk about is vmstat.
Now vmstat is a command that displays various statistics
about virtual memory.
This includes things like our processes, our CPU,
and our IO statistics, as well as our virtual memory
when we use vmstat.
Now by default, the report that you're going to get
from vmstat is going to provide you with averages
for each statistic since the last system boot up.
But you can also specify a delay value
to sample from a period of time.
For instance, every 60 seconds, you want to get a new report,
you can tell it to do that.
Now, some memory-based statistics here
are going to include total virtual memory available.
The total virtual memory
that's free for us to use,
the total memory used in the buffers and cache,
and the total memory that's used in swap spaces.
Some CPU-based statistics
are going to include time spent running user space,
the time spent running in kernel space,
the time spent idle, and the time spent waiting
for input and output.
To run the vmstat command, type in vmstat,
the options, the delay with the counter, and then hit enter.
It's recommended that you supply vmstat
with a delay for more accurate report as well.
For example, vmstat five five will run the command
on a five-second delay for five different intervals,
giving you a good average over that time.
Another thing we want to talk about is OOM,
the out of memory killer or OOM is a feature
of the Linux kernel that determines
what processes need to be killed
when the system becomes extremely low on memory.
The out of memory killer will continue to kill processes
until enough free memory is made
so that the kernel
and the system can begin running smoothly again.
Rather than killing processes at random though,
the out of memory killer will leverage an algorithm
that assigns each process an OOM score.
The higher the score,
the higher the chance that process has of being killed
during an out of memory event.
The assignment algorithm is going to consider
what processes will free up the greatest amount of memory
when killed as well as what processes are least important
to system stability.
Then it'll assign a score based
on what it determines to be the most optimal targets
for termination.
Although this mechanism is mostly automated by the kernel,
you do have the ability to control some of its behavior.
You can mount the OOM control group
as a desired mount point.
And if, for example
you want to designate a group of processes to be killed first,
you can create a directory at this mount point
and create a task file in this directory
then list out the PIDs of the relevant processes.
Then you can create an OOM.priority file
with a higher integer value like 256
to give the processes a higher priority
for the OOM killer to take care of.
Or to prevent a process from being killed entirely,
you can assign it zero as a priority value.
Now the configuration of swap space is another thing we need
to think about
because your swap space
can help alleviate memory-related issues,
especially when your system
and applications request more memory
than your system physically has.
Now systems with a low amount
of memory are particularly vulnerable
to these issues of swapping.
Swap space is not a replacement for adding more memory
but it can help minimize system
and application sluggishness, unresponsiveness, and crashes.
Swap space can be of one of three types.
A device swap space is configured
when you partition the storage device
to allow you to have some swap space on that disc.
This will be used by the operating system
to run larger applications.
File system swap space is configured primarily
when you install Linux.
It's going to be used by the operating system
as an emergency resource
when the available swap space runs out.
On the other hand,
pseudo swap space enables large applications
to run on computers with limited RAM.
Swap files are created for storing data
that's going to be transferred
from a system's memory to a storage device.
These things are dynamic and they change in size
when the data's moved in and out of memory.
Swap files are going to be used as a medium
to transfer data from RAM into that storage device.
A swap partition is an area of virtual memory
on a storage device that complement the physical RAM
in the computer.
Swap partitions are used by Linux
because they perform better than a swap file system.
Now, if you want to create a swap space
on a storage partition,
you're going to use the make swap command
which is M-K-S-W-A-P.
The make swap command is going to be used to create swap space
on a storage partition and is typically used
when you want to move swap space to a different partition
than the one it was originally created
on during system installation.
For example, let's say you wanted to save space
on a low capacity boot drive.
You could provide an option to be able
to move that over to a new swap space.
If you use -c,
this option verifies that the device is free
from bad sectors before mounting the swap space.
Essentially it's a check disc.
p is going to set the page size
that's going to be used by the make swap command.
And -l will activate the swap space using labels applied
to partitions or file systems.
Another command we have is known as swap on.
The swap on command is used to activate a swap partition
in a specified device.
While the swap off command is used
to deactivate the swap space on a device.
Some of the frequently used swap on
and swap off command options are swapon-e,
which is used to skip devices that do not exist
or swapon-a, which is used to activate all of the swap space
or swapoff-a, which is just to deactivate all
of your swap space.
All right, so now that you're familiar with some
of the basic command tools that you can use,
including what they do and how they can help you,
you're now going to be able to troubleshoot CPU
and memory issues.
When you're troubleshooting CPU and memory issues,
remember, identify key information about the CPU
and its logical cores using the /proc/cpuinfo file.
Use the uptime command to identify CPU load averages
and use the sar command
to see what components are causing heavy load on your CPU.
When you want to look at memory,
you can identify key information about your memory usage
by looking at the /proc/meminfo file,
or you can use the free command
to more easily analyze memory usage information
and the vmstat command to retrieve more information
on both your CPU and memory usage,
as well as your virtual memory.
Also, you may want to consider tweaking your OOM killer
to spare or sacrifice specific processes
when you're running low on memory.
And finally,
remember that you can always create more swap space
if you need to because you ran out
of physical memory by using your swap commands.