In this lesson,
we're going to talk about manipulation of a file's output
by using piping and redirection to create powerful workflows
that will automate your work, saving you time and effort.
Now many things in Linux are done by knowing the behavior
of certain commands and aspects of the system
and then using them in creative ways
to achieve your desired outcome.
We're going to explore the command line
as we provide you with a series of building blocks here.
Now you're going to be free to use these building blocks
in any way you like,
but to really be effective at doing this,
you have to understand not only how to do their function
but why you do those functions.
Everything in Linux revolves around streams of data,
particularly text streams.
Now text streams is a sequence of one or more lines of text
that applications can leverage to read from
or write to a particular device or system component.
This enables the application to interface with components
like the command line interface, the files,
the network sockets and much more
while hiding those components details
from the overall applications.
In most Linux shells, there are three types of streams,
standard input, standard output and standard error.
The standard input, also known as stdin
is a text stream that acts as the source for command input.
Standard input for a Linux command line
is usually generated from your keyboard.
In the case of a graphical user interface,
the standard input can also come from the mouse.
The standard output, or stdout, is a text stream
that acts as the destination for command output.
By default, the standard output for a Linux command
is directed to the command line interface.
Now the third thing we have
is the standard error, or stderr.
This is a text stream that's used as a destination
for any error messages.
By default, the standard error stream
is going to print out error messages
at your command line interface.
Technically, redirection happens inherently
whenever you interact with the computer.
Input gets read from the standard in,
output goes into the standard out
and any errors gets sent to the standard error.
It is possible, though, to redirect the input
and output of various commands to and from different files.
There are various operators that are used
to redirect input or output.
For example, the operator is the greater than sign
is going to be used to redirect
the standard output to a given file.
For example, if you type ls greater than file1.txt,
the output of the ls command will redirect itself
into a file called file1.txt.
If you use the operator greater than greater than,
it's going to be used to append the standard output
to the end of the destination file
instead of overwriting it.
So you could type ls greater than greater than file1.txt
and the output of the ls command
will be appended to the file named file1.txt.
The operator two greater than is going to be used
to redirect the standard error messages to a file.
For example, let's assume that file3.txt does not exist yet
and you type ls file3.txt two greater than errorfile.txt.
The resulting error from the ls command
for not finding file3.txt
will not be displayed on your screen,
but instead, it'll be redirected to the file that we created
called errorfile.txt.
If we want to append it instead of overwriting it,
we'll use the operator two greater than greater than,
and that way we append the standard error message
to the end of the destination file
instead of overwriting it.
So again, assuming that file3.txt
does not exist on your system
and you type ls file3.txt
two greater than greater than errorfile.txt,
the resulting errors will not be displayed on the screen
but they'll be redirected and appended to errorfile.txt.
Now if you use the pattern ampersand greater than,
it's going to be used to redirect both the standard output
and the standard error messages to a file.
So let's assume that file1.txt exists
and file3.txt does not,
and then we type ls file1.txt file3.txt
ampersand greater than errorfile.txt onto the command line.
Now the resulting output and errors
will not be displayed on the screen,
but instead, they're being redirected
into that file called errorfile.txt.
If we use the pattern of a less than sign,
it's going to be used to read input from a file
instead of doing it from the keyboard or mouse.
So let's say you typed mail user@address
less than myletter.txt and you put that on the command line.
Well, it's going to take the file myletter.txt
and take that as the input
and attach it to that email message.
Finally, if you use the operator less than less than string,
it's going to be used to provide input data
from the current source
and it's only going to stop
when the line containing the provided string you listed
actually is seen.
When you place this in a script,
it's called the here document,
which refers to a special block of code
that contains multi-line strings
that'll be redirected to a command.
For example, let's say you typed in cat
less than less than EOF
and then you typed in a document,
something like This is a here document. EOF.
Well, when it reads that in and it gets to EOF,
it's going to stop, so the cat command is going to use
the rest of the lines in this file as the input.
And it'll stop accepting that input
whenever it reaches the string EOF,
which is the end of file.
This string can really be named anything you want.
I just used EOF for end of file to make my point.
Now when you're doing the output of the cat command,
it would then show up as This is a here document
because that's all I wrote there before the EOF was found.
Another command we have is known as the pipe.
Now the pipe is a command in Linux
that lets you use two or more commands
such as the output of one command
that becomes serving as the input for another.
And so, we create this chain of events.
In short, the output of each process directs itself
as the input to the next one in the pipe,
so this becomes a chained pipeline.
Now a pipe looks like a long up and down line
and this is the symbol we call a pipe.
Pipes help you mash together two or more commands
at the same time and then run them consecutively.
For example, earlier, when we talked about grep,
I said that I can use ls dash l pipe grep audit,
and this would actually search the directory,
list them out and then run the grep command
on that list of files that we found.
This is the example of using a pipe.
The standard output from ls dash l
was fed into grep as the input for it
and then searched against the pattern of audit.
This is the idea of using pipes
and be able to chain things together.
Now by learning all the standard
and common commands in Linux,
you're going to be able to start chaining them together
to create powerful new commands
or pipelines by using this pipe.
A great example of this is the xargs command.
Now x a-r-g-s is a great command
that read streams of data from the standard input
and then it generates and executes command lines,
meaning it can take the output of a command
and pass it as an argument into another command
and if no command is specified,
the x a-r-g's are going to execute echo by default
and display them to the screen.
To run the xargs command,
you actually run it a little bit backwards
because you're first going to type in the command
you want to use, the options and arguments,
then a pipe, then xargs, its options
and its command inside of the command line.
So let me show you how this works.
Let's say, for example, you want to delete all the files
in the slash foo directory that have a .pdf extension.
You could use xargs to automate this process.
So you're going to use
find /foo -type f -name "*.pdf" l xargs rm.
So what we're doing is we're running the find command first
to find all those PDFs.
Then we take that and pass it to the xargs command
and it's going to run the remove command
on all the results we found.
Now because the results from the find command
are going to be deliminated by a space,
the xarg command is going to run the rm, or remove command,
for each file in the results,
removing all the PDF files in that directory.
This command has a lot of different options available,
but we're going to cover just a few of 'em here
that are really important.
For example, you have the dash l and a replacement string.
You can use this if you want to consider each line
in the standard input as a single argument.
If you use dash capital L and a line number,
it'll read a specified number of lines
from the standard input
and concatenate them into one large string.
If you use dash p, this is going to be used
to prompt the user before each command.
If you use dash n in the number of arguments,
this is going to allow you to read the maximum number
of arguments from the standard input
and insert them at the end of the command template.
The option dash capital E in end of string
is going to represent the end of the standard input.
Similar to what we used earlier
when we talked about the operator of less than less than
with the string being provided.
If we use the option dash t,
this is going to be used to write each command
to the standard error output before executing the command.
And if we use dash s in the maximum size,
this can be used to set the maximum allowable size
of an argument list to the specified number of characters.
The next command I want to talk about is tee.
Tee is spelled t-e-e like a golf tee.
Now the tee command reads the standard input
and sends the output to the default output device,
which is the command line interface,
but it also copies that output to each specified file.
This command enables you to verify
the output of a command immediately
by seeing it on the screen
but also storing that to a file for later reference.
When used with the dash a option,
tee will append the output to the output file
instead of overwriting it.
To be able to run the tee command,
simply type in the command you want to run,
the options and the arguments, the pipe and then tee,
the options for tee
and the file names you want to write it to.
For example, let's say you wanted to check
the contents of a directory
and also output those contents to a file
so you can process them later.
You could issue separate commands to do this
or you can use the tee command and do it all at once.
So you can do this by typing ls -l pipe tee listing.txt,
and that way, it'll display the list to the screen
and save a copy in the listing.txt file.
Now another interesting thing about Linux
is that it hosts a lot of virtual devices
for a lot of different purposes.
As far as the programs are concerned
that are running on this system,
these virtual devices act as if they're real files,
but tools can request and feed data
from these different sources by accessing these files.
The data generated by the operating system,
instead of reading them from a disc,
is actually done as part of these files.
For example, we have one virtual interface
known as /dev/null.
It's a special file that's present
on every single Linux system you'll ever work on.
However, unlike most virtual files, instead of reading,
this one is used for writing.
Now whatever you write to /dev/null
is actually going to be discarded and forgotten.
Essentially, it's a black hole.
So if you have a lot of data coming in off an interface
and it may be causing a denial of service to you
or something like that,
you can actually redirect that to /dev/null
and it'll go off into the void and just be forgotten.
Now a running process in Linux
can also be controlled by the command line environment
inside the terminal
and multiple terminals can be run at once.
When you're doing this,
you have to have a way to control these different terminals
and we do that by assigning an identifier.
The identifier takes the form of /dev/tty with a number.
And that number is unique to that terminal.
You can then redirect standard input and output
to another controlling terminal if you want to
by referencing its /dev/tty number.
This can be really useful
if you need to redirect text streams
between different running processes
and other things like that as well.