In this lesson, we're going to discuss
how you can troubleshoot storage issues
using common storage tools available
in the Linux environment to solve these issues
you may come across.
Now organizations of all size and industries
face data storage challenges.
And as an administrator, you're going to face a variety
of issues with local storage devices.
A key aspect of server management
includes ensuring that storage devices are being recognized
by the system, and that they're available to your users.
There are a lot of local storage issues
including missing devices, missing volumes,
missing mount points, degraded storage, performance issues,
resource exhaustion, storage integrity, and many more.
You're going to need to investigate, diagnose,
and suggest solutions
that can affect your storage solutions.
If you want to limit the system resources
for a user in a Linux based server
then you're going to need to use the ulimit command.
Now ulimit is going to be able to set a limit
for the maximum number of open file descriptors.
For example, if I wanted to set that to 512,
I could type in ulimit -n 512.
This will then put our limit of 512 maximal open files
for particular user.
You can also display all of your current limits
by issuing the command ulimit -a.
If you want to learn more about ulimit,
go ahead and type in man space u-limit
in your command prompt.
If you want to use it, you're going to type
in ulimit the options and the amount you want to limit.
Now, there's also two commands
to confirm excessive storage space consumption
that might be causing system issues for you.
These are df/du.
The df and du commands are going to facilitate
storage space tracking.
df is going to enable you to view the device's free space,
the file system, the total size,
the space used, the percentage value of space used,
and the mount point.
The du is going to display the device that's being used
including the size of the directory trees
and files within it.
Also enables you to track space hogs.
Those people or directories
that are consuming large amounts of space
on your storage drive.
To run these, you're going to type either du,
the options and the object name
or df, the options and the object name.
Another thing you need to think about is I/O scheduling.
Now, I/O scheduling stands for input/output scheduling,
and it's the process by which the operating system
determines the order of input and output operations
as they pertain to block storage devices.
Scheduling is really important,
because compared to CPU and memory operations,
block storage operations are pretty darn slow
especially if you're using disc based technologies
like old traditional hard discs.
Now there are some different schedulers
that are available for modern Linux kernel versions.
The deadline scheduler performs sorting
of input/output operations, using three different queues.
We have a standard pending request queue,
a read first in, first out queue or FIFO queue,
and a write first in, first out or FIFO queue.
These last two, these two FIFO queues
for reading and writing can be sorted
by the submission time and have expiration values as well.
When a request is submitted, it's going to be sorted
into the standard queue and placed at the end
of the appropriate FIFO queue either for reading or writing.
If the request gets the top of the FIFO queue,
and it becomes older than the queue's expiration,
the schedule will actually stop working
with the standard queue and start servicing requests
from the top of the FIFO queue.
Other words, it wants to handle the oldest request first.
The CFQ scheduler refers
to the Complete Fair Queueing or CFQ scheduler.
Now this scheduler is going to have a process
that's given to each queue and each queue
has an interval by which it's being accessed,
or it's given time slice.
Now, the scheduler's going to use a round robin system
to access each queue and service request
from each of these queues
until their time slices or requests are fully exhausted.
When this happens, the CFQ is going to wait 10 milliseconds
until it sees a new request on the queue.
And if it doesn't, it's going to move on to another queue
so it doesn't waste time.
Now, like the deadline scheduler,
this helps to minimize request starvation.
Another way we can do this
is by using the NOOP Scheduler, N-O-O-P.
Now the NOOP scheduler can be ideal in situations
where the device or its storage controller
performs its own sorting operations.
This can benefit devices
that don't have mechanical components
that require seek time.
Things like SSD, solid state devices or USB flash drives
because this scheduler doesn't expend much effort
in reducing seek time.
Now you, as a system administrator, can actually set
the scheduler that you want to use for particular device
by modifying the scheduler file located
at /sys/block/ whatever the device is /queue/scheduler.
So for example, if I want to set the NOOP scheduler
as my schedule for my SDA device,
I can do this by setting it into the text file,
and I can do it simply by using the echo command.
I would type echo, space noop, space redirector,
or the carrot to the right,
space /sys/block/sda/queue/scheduler.
This sets the scheduler for runtime only
and this setting will revert next time I reboot the system.
If I want to make persistent changes, though,
I have to modify the system's boot loader configuration.
Another utility we need to talk about
is iostat or input/output statistics.
The iostat utility generates reports on CPU
and device usage.
For storage, it's going to provide input and output statistics
for blocked devices and partitions.
Oftentimes, you'll hear input/output operations
per second use, which is I-O-P-S or IOPS.
When you have low IOPS, this means that you have
a very low amount of input and output
operations happening per second,
and there's a slowdown in your system.
To identify this, you can use a tool like iostat
to see what are you currently generating
in terms of input/output,
and then you can identify ways to speed that up
and increase that number
of input/output operations per second.
There are lots of different options when using iostat.
For example, you can use the -D option
to specify the device information only.
The iostat command in this case will list
the following statistics for each storage device,
including its transfers or I/O requests per second,
as tps, the number of blocks read per second
in kilobytes read per second.
The number of blocks written per second
in kilobytes written per second.
The number of total blocks read
which is kilobytes read,
and the total number of blocks written,
which is kilobytes written.
You can use this report to monitor how the storage drive
is being used and to identify any potential bottlenecks.
For example, you might have a faulty drive,
and it can have lower read speeds
or lower write speeds per second, than you're expecting.
If you use this report, it can help you decide
how to best distribute the input/output load
between the different available devices you have.
To run the iostat command,
you're simply going to type in iostat,
the options and the device name.
Now, another command we need to talk about
is ioping, the ioping command generates
a report of device input/output latency in real time.
It's going to continuously ping the specified device
with a request and then print the information
about each request at your command line.
By default, this information tracks
how long it took for an input/output request to finish.
Aside from specifying a device to test,
you can also specify a path name to test
whatever device is associated with that path,
so you can consider using ioping
to troubleshoot latency issues with storage devices,
especially if you believe your read or right speeds
are slower than they really should be.
To do this simply type in ioping,
the options and the file directory or device name.
Now there's some options you can use
with the ioping command.
Things like -c and the count.
This will be used to specify the number
of input/output requests that you want to perform
before stopping automatically.
So you might send 10 requests and see what that looks like.
You can also use -I and time.
This is used to set the time interval
between those input/output requests.
You might also use -t and the time.
This is used to set a minimum valid request time
so requests faster than this are going to be ignored.
If you use dash capital T and time,
this is going to be used to set the maximum valid request time,
so any request that are slower than this will be ignored.
And finally, if you use -s and size,
this is going to set the size of the request
that you're sending using the ioping command.
Now, when we talk about storage,
we need to talk about the idea of storage quotas.
Now, a storage quota is the storage space
that's allocated to a user for file storage
on a computer.
Storage quotas are going to be configured on a per user basis.
And if you use these, it can actually prevent
a lot of issues for you.
If you start doing this, you're going to be able
to prevent issues with people using excessive storage space
and eating up all of the storage
so the system can't operate.
You can use these quotas to ensure
that your users are not consuming all of your drive space
and leaving none for any other users on the system.
Remember quota management is an effective allotment
and monitoring of all the quotas for all of your users.
Linux has a lot of different commands
that can help you ease the job of quota management
as a system administrator.
For example, we have quotacheck,
and if you use quotacheck -cug and the mount point,
this command will actually create quota-based
database files for the file system,
and then check the users and group quotas against it.
Now, if you use edquota -u, username,
this is going to be used to edit quotas for a specific user.
If you use edquota -g and the group name,
this will be used to edit these quotas for a specific group.
If you use setquota -u and the username,
this is going to be used to set quotas for specific user.
If you use setquota -g group name,
this will be able to set quotas for a specific group.
Now, before you can use any of these commands,
you have to activate user and group quotas
on the file system.
And to do this, you'll edit your fstab file
and add the options, usrquota
or grpquota to the relevant file system.
Now, you can also use your xfs_admin utility
to configure quotas if you're using an XFS file system.
This utility can run
in both interactive and non interactive mode.
If you're running it non interactively,
you can use the -c option to specify
which commands to run,
and the -x options to enable expert mode
which is required for many of the administrative tasks.
Tasks inside the XFS admin utility include things
like setting limits on writing blocks and iNodes,
setting warning limits, generating quarter reports,
and much more.
Now quarter reports are going to be created by the system
so you can view storage space that's being used
by each person.
These reports enable you to check which user
is taking up the most disk space.
This can also help you troubleshoot issues
with the quotas themself.
For example, quotas may be too restrictive
or too permissive.
A quota report is going to contain the name of the user
or group, the total number of blocks in kilobytes
that are being used by that user or group
in the file system, the user or group storage soft limit,
the user or group storage hard limit, the grace period,
the total number of iNodes that have been used
by the file system by that user or group,
the soft limit on those iNodes,
and the hard limit on those iNodes.
Now, there are several commands available
for the generation of effective quota reports.
This includes things like rep quota.
Now the rep quota -a command is going to be used
to display the reports for all the file systems indicated
as read or write with quotas in the mtab file.
repquota -u, and the username will display
the quota reports for particular user.
If you use quota -uv, and the username,
this command will be used to display the quota report
for a particular user with verbose output.
If you use warnquota -u,
this command will be used to check
if a user are not exceeding the allocated quota limit.
And if you use warnquota -g, this'll check if any groups
are not exceeding the allocated quota limit.
Now when troubleshooting storage issues,
you should always start out with the simple things
and then move towards the complex things.
Make sure your devices are physically connected
to the system when your troubleshooting storage issues.
I've seen a lot of times where the device simply
isn't even connected or powered on.
So make sure you've checked that it's powered,
verify the device is recognized by the system
by checking the slash proc directory.
Confirm that the configuration files
don't contain any typographical errors.
Also ensure the configuration files have been loaded
if you made changes to them
and confirm there's enough storage capacity,
as well as confirming the input/output workload
is not overwhelming your devices,
and you can always use the part probe command
to enable the system to scan for new storage devices
and partitions and load those files back into the kernel.
This is all some of the basic stuff you need to check
to make sure your storage devices are working optimally.