In this lesson,
we're going to discuss the processing of text files
by using a number of command line tools
to perform operations on text files
and then write those results to the standard output.
By the end of this video,
you're going to be expected to process information
in powerful ways
by restructuring output to generate useful reports,
modifying text in the files,
and many other system administration tasks.
To do this,
we're first going to talk about the echo command.
The echo command is a built-in Linux feature
that prints out arguments as the standard output.
The echo command is commonly used
to display text strings or command results
as messages to the screen.
To run echo,
you simply type in echo and a string.
For example, let's say I typed in
echo "Hello, World!"
The output that's going to be printed to the screen
is Hello, World!
So that means using the echo command
returns the provided string as the output
with absolutely no changes.
The printf command is similar to echo
but it provides the user with a lot more control
over how the output is going to be formatted.
You can supply various format characters
within the text you want to output
by using things like the backslash
to indicate where they're going to be used.
For example,
if I type printf "Hello.\nWhat's your name?",
this will print Hello. What's your name?
because that \n is a new line format character.
It's automatically going to add a new line
wherever it's placed,
in this case after the hello.
Whenever you need to perform operations
like removing repeated characters,
converting uppercase to lowercase,
and basic character replacing and removing,
you might want to look at the tr command.
The tr command can be run
by typing tr, the first character,
and then the second character.
This is where the first character
is the character you want to replace.
So for example, let's say I typed in linuxize,
L-I-N-U-X-I-Z-E,
and then I pipe that over to the tr command,
and I want to remove l,i,n
and replace it with r,e,d.
Each time I see an occurrence of L,
it's going to be replace with the R,
every time I see an I,
it's going to replace with an E,
every time I see that N,
it's going to be replace with a D,
so I'm replacing those three characters
with the r,e,d characters.
This will give me the output of R-E-D-U-X-E-Z-E.
So now instead of linuxize,
I'm getting reduxeze.
Another command we can use is known as wc.
wc is going to allow you to count the number of lines
the words, the characters,
and the bites of any given file or standard input,
and then print the result out.
If multiple files are specified,
the command will display the counts for each file
and the total count for all of the files.
To run this,
simply type in wc, the options, and the file names.
Now, there is a lot of different options you can use
within the wc command.
For example,
you can -c if you want to display the byte count,
if you use -m,
it's going to display the character count,
if you do -l,
it's going to display the new line count,
and if you do -w,
it's going to display the word count.
Another command we have is known as sort.
The sort command is a command line utility
for sorting lines of text files.
The sort command supports sorting alphabetically,
in reverse order,
by number, by month,
and it can also remove duplicates for you.
There are a few options for the sort command
you should be familiar with
including the -k option and the column number.
When you do this,
it's going to specify the field value that you want to use.
So if I use -k2,
this indicates the second field.
If you use the -n option,
this compare and sorts lines
based on a string's numerical value,
if you use the -r option,
it's going to sort the fields in descending order
because the fields are going to be sorted
in ascending order by default,
and so by using -r,
we're going to reverse that.
Another option you can use is -t,
which allows you to separate one field from another.
To run the sort command,
simply type in sort, your options, and your file name.
Another command we need to talk about is the cut command.
The cut command extracts specific lines of text
from a file.
Common cut command options include -c.
The -c option is going to specify
the number of characters to cut from each line.
If you use the -d and a delimiter,
it's going to separate one field from another
by using that delimiter.
For instance, a comma for comma-separated values.
If you use -f and the field number,
this is going to specify the field numbers to cut
as separated by the delimiter.
For example, -f2 will indicate the field
between the first and second instances of the delimiter
or those commas in a comma-separated value file.
And finally,
s is an option we can use to suppress a line
if a delimiter is not found.
To run the cut command,
simply type in cut, the options, and the file name.
Another one we have is the paste command.
The paste command is used to merge lines
from text files horizontally,
each line of an initial file in a row
is going to be in the first column.
In using paste, you specify a second file,
and every line of the second file
becomes a row in a new second column.
By default,
the paste command uses a tab space deliminator
to separate each column.
So you can use the -d option
to specify a different deliminator,
like a comma if you wanted to,
but by default, it is going to use a tab.
So, for example, let's say I have two files:
the first file is a name of cities
and I have New York, Tokyo, London, and Lima in it,
then I have another file called countries
and it has United States, Japan, England, and Peru.
So if I wanted to combine those two files into one list,
I could do that by running the paste command.
I would type paste -d , cities countries,
and hit Enter.
This then creates the output
of New York, United States;
Tokyo, Japan;
London, England;
and Lima, Peru
because I'm using that comma I specified
as my delineator.
Another command for us to look at is the diff command.
Now the diff command is used to compare text files
and it's going to find the difference between them.
This command is going to display the two files
and the differences between them.
It's really useful when you're trying to do version control
across different code bases.
If you're using various symbols,
the output can actually suggest
how you can change one file to another
to make it identical to each other.
Each symbol now has a special meaning.
If you have a less than symbol with a line after it,
this means the line should be removed from the first file
because it doesn't appear in the second file.
If you have a greater than symbol with the line after it,
this means that line should be added to the second file
because it doesn't exist there.
In addition to this,
the diff command can also denote the line numbers
for each line in the file
that would be affected by the deletion, addition,
or change operations.
To run the diff command,
type diff, file name 1 and file name 2.
Now the diff command also has some various options
that we're going to talk about:
b is the option that ignores spacing differences,
i ignores case differences,
t is going to expand tab characters in the output files,
w is going to ignore spacing differences in tabs,
c is going to display a list of all the differences
with three lines of context,
while -u is going to output the results in a unified mode,
which presents a more streamlined format.
The next command we're going to talk about is grep,
and grep in its most basic form is a search tool.
But unlike find or locate,
it's not limited to finding file names.
Instead, it's often used to search the contents of the file
for a particular string of text
By default,
grep is going to display each full line of the file
that your search pattern is found inside of.
In this way,
you can use grep to both process a text file
and read the contents that are most pertinent to you,
and we use grep a lot in the cybersecurity world
when we're looking through log files
and trying to find specific instances of things.
For example,
you may want to audit the user's login events
by looking at an access log.
Instead of reading the entire log
and stepping through a search term in a text editor,
you can instead simply print
all the relevant lines to the screen
by using the grep command.
The proper way to use the grep command
is to type in grep, the options, the search pattern,
and the file name.
The grep command has a couple of options
we need to talk about,
the first one is -E and a pattern.
This is used to use pattern matching
using an extended regular expression
instead of just using a standard search term.
If you use -F in a pattern,
this is used to match a pattern
as a list of fixed strings.
The option -f and a file name
will be used to match patterns
that are contained in a specified file.
The option -i is going to be used to ignore casing,
and the option -v is used to output lines
only if those lines don't match the provided pattern,
so essentially,
show me everything that doesn't match
what I'm looking for.
The option -c is going to be used to print
only the numbers of the matching lines
and not the actual lines themself,
and the option -l is going to print only the files
that have matching lines,
not the lines themself.
This is helpful if you're trying to find
is there this keyword across all the files on my system
and it'll tell you all the files that match that keyword
inside of them,
and the option -o is going to print
only the matching part of a line
instead of the entire line.
Now, in addition to searching the contents of the files,
you can also use grep to search a directory
in order to locate a certain file
that contains those keywords or patterns.
For example, if I wanted to find files
with the keyword audit in them,
I can use the ls -l command
and then pipe that into grep with the word audit.
This would return a long listing of any files
in the current directory
whose name contains audit.
The next command we're going to talk about
is the awk command.
The awk command performs pattern matching on files.
It's based on the AWK programming language.
The awk keyword is going to be followed by the pattern,
the action to be performed,
and the file name.
Now, the action to be performed
is given within curly braces,
and the pattern and the action to be performed
should be specified within a single quote.
If the pattern is not specified,
the action is performed on all input data.
However, if the action is not specified,
the entire line is going to be printed.
The awk command can be executed from the command line
or from within an awk script file.
The awk command can be used to process text
in a variety of different ways,
including extracting text that matches a certain pattern,
deleting text that matches a certain pattern,
adding text that matches a certain pattern,
and much more.
To run awk,
simply type in awk, options, patterns, actions,
and file names.
In awk scripts,
you can also provide patterns along with blocks of code.
If pattern matching any line in the input file,
the code blocks in the script can then be executed.
The pattern can include regular_expressions as well
that will retrieve all the records that match it.
For example, if I use / bracket abc bracket /,
it's going to find any record that begins with a, b, or c.
If you use the pattern relational_expressions,
this will retrieve all the records
whose first field contains the value abc.
For example, $1 == "abc".
The pattern pattern_1 && pattern_2
is going to retrieve all the records
whose first field contains the value abc
and the second field contains the value 01.
So for example,
if I have $1 == "abc" && $2 == "01",
this creates that pattern.
If I want to create an OR condition,
I can do that by using two pipe characters.
So if I use the pattern of pattern_1 || pattern_2,
this will retrieve all the records
that satisfy the conditions for the first field
that contains the values of pattern 1 or abc,
or the second field that contains the value of 01,
which was my second pattern,
or even both of them.
So for example,
I might have somebody that looks like
$1 == "abc" || $2 == "01",
and this would fulfill this OR condition.
Now, another pattern we can use
is basically an if-then,
and we do this by having
pattern_1 ? pattern_2 : pattern_3.
So what's going to happen
is it's first going to evaluate pattern 1,
and if it matches pattern 1,
it's going to go to pattern 2,
and if it matches pattern two,
it's going to print it to the screen.
But if it evaluates pattern 1 and it doesn't match,
it's going to then evaluate pattern 3,
and if it matches pattern 3,
it'll display that to the screen.
So what does this look like?
Well, let's say I have something
like $1 == "10" ? $5 == "20" : $9 == "30".
This means that if field 1 equals 10,
I'm going to evaluate field 5 and see if it equals 20.
If it does, I'll print that line to the screen.
But if I evaluated field 1 and it didn't equal 10,
I'm then going to evaluate field 9
and see if it equals 30,
and if it does, I'll print it to the screen.
In the case that any of those are not going to happen,
we are then going to print nothing to the screen
and evaluate the next line.
The last pattern we need to cover
is known as pattern_1, pattern 2,
and this will print a range of records
starting from the record whose first field
contains the value of pattern 1
and going until it finds the field of pattern 2.
So for example,
if I have $1 == "01", $1 == "02",
this tells me I'm going to check the first field
for any values of 01
until I find ones that have 02.
The next thing I want to talk about
is known as a stream editor or sed command.
The sed command is a program that you can use
to modify text files according to various parameters.
The sed command can also be used
for global search and replace actions.
Some common options here are going to be option d,
which is used to delete lines
that match a specific pattern or line number,
option -n,p will be used to print only the lines
that contain the pattern,
option s is going to be used to substitute
the first occurrence of the string in the file,
and the option s,g is going to be used
to globally substitute the original string
with the replacement string
for each occurrence in the file.
The proper way to use this command
is to type sed, the option, address, or action,
and then the file names.
Another command we need to talk about is the link command.
This is known as ln.
The ln command is used to create a link to a file.
Linking enables a file in one directory, the link,
to point to a file in another directory, the target.
A link does not contain data on its own,
only a reference to the target file.
Any changes to the link
will reflect into that target file as well.
If you don't specify the link name,
the ln command will create the link
in your current working directory.
The syntax of this command
is ln, the options, the target name, and the link name.
The ln command has various different options
that you can use as well including -backup,
which is used to back up existing destination files,
F, which will be used to remove
existing destination files,
s, to make a symbolic link instead of a hard link,
and -i, to prompt to remove destination files,
and finally, -v, to print the name of a file before linking.
Using the ln command,
you can create two different types of links:
these are called hard links and symbolic links.
Now a hard link is a reference to another file
and it enables the files data to have more than one name
in different locations in the same file system.
If the original file is deleted
after the hard link is created,
all of its contents will still be available
in the linked file.
A symbolic link also called a soft link
is a reference to a file or directory
that can span multiple different file systems.
But if the original file or directory
is deleted after the symbolic link is created,
the original content is lost.
In order to create a hard link,
you would simply type in ln, the file you want to do,
and then the link itself
because the ln command uses hard links by default
when creating a new link.