1 00:00:00,750 --> 00:00:01,830 In this lesson, 2 00:00:01,830 --> 00:00:05,160 we're going to talk about manipulation of a file's output 3 00:00:05,160 --> 00:00:08,850 by using piping and redirection to create powerful workflows 4 00:00:08,850 --> 00:00:12,120 that will automate your work, saving you time and effort. 5 00:00:12,120 --> 00:00:15,000 Now many things in Linux are done by knowing the behavior 6 00:00:15,000 --> 00:00:18,060 of certain commands and aspects of the system 7 00:00:18,060 --> 00:00:20,070 and then using them in creative ways 8 00:00:20,070 --> 00:00:22,200 to achieve your desired outcome. 9 00:00:22,200 --> 00:00:24,150 We're going to explore the command line 10 00:00:24,150 --> 00:00:27,270 as we provide you with a series of building blocks here. 11 00:00:27,270 --> 00:00:29,610 Now you're going to be free to use these building blocks 12 00:00:29,610 --> 00:00:31,050 in any way you like, 13 00:00:31,050 --> 00:00:33,630 but to really be effective at doing this, 14 00:00:33,630 --> 00:00:36,300 you have to understand not only how to do their function 15 00:00:36,300 --> 00:00:38,640 but why you do those functions. 16 00:00:38,640 --> 00:00:41,610 Everything in Linux revolves around streams of data, 17 00:00:41,610 --> 00:00:43,740 particularly text streams. 18 00:00:43,740 --> 00:00:47,010 Now text streams is a sequence of one or more lines of text 19 00:00:47,010 --> 00:00:49,470 that applications can leverage to read from 20 00:00:49,470 --> 00:00:52,950 or write to a particular device or system component. 21 00:00:52,950 --> 00:00:55,890 This enables the application to interface with components 22 00:00:55,890 --> 00:00:58,620 like the command line interface, the files, 23 00:00:58,620 --> 00:01:00,840 the network sockets and much more 24 00:01:00,840 --> 00:01:02,760 while hiding those components details 25 00:01:02,760 --> 00:01:05,040 from the overall applications. 26 00:01:05,040 --> 00:01:08,460 In most Linux shells, there are three types of streams, 27 00:01:08,460 --> 00:01:12,390 standard input, standard output and standard error. 28 00:01:12,390 --> 00:01:16,140 The standard input, also known as stdin 29 00:01:16,140 --> 00:01:19,680 is a text stream that acts as the source for command input. 30 00:01:19,680 --> 00:01:21,990 Standard input for a Linux command line 31 00:01:21,990 --> 00:01:24,330 is usually generated from your keyboard. 32 00:01:24,330 --> 00:01:26,610 In the case of a graphical user interface, 33 00:01:26,610 --> 00:01:29,700 the standard input can also come from the mouse. 34 00:01:29,700 --> 00:01:34,080 The standard output, or stdout, is a text stream 35 00:01:34,080 --> 00:01:37,290 that acts as the destination for command output. 36 00:01:37,290 --> 00:01:40,170 By default, the standard output for a Linux command 37 00:01:40,170 --> 00:01:42,870 is directed to the command line interface. 38 00:01:42,870 --> 00:01:44,160 Now the third thing we have 39 00:01:44,160 --> 00:01:47,013 is the standard error, or stderr. 40 00:01:48,390 --> 00:01:50,970 This is a text stream that's used as a destination 41 00:01:50,970 --> 00:01:52,830 for any error messages. 42 00:01:52,830 --> 00:01:55,140 By default, the standard error stream 43 00:01:55,140 --> 00:01:56,700 is going to print out error messages 44 00:01:56,700 --> 00:01:58,890 at your command line interface. 45 00:01:58,890 --> 00:02:01,680 Technically, redirection happens inherently 46 00:02:01,680 --> 00:02:04,020 whenever you interact with the computer. 47 00:02:04,020 --> 00:02:06,180 Input gets read from the standard in, 48 00:02:06,180 --> 00:02:07,920 output goes into the standard out 49 00:02:07,920 --> 00:02:10,919 and any errors gets sent to the standard error. 50 00:02:10,919 --> 00:02:13,470 It is possible, though, to redirect the input 51 00:02:13,470 --> 00:02:17,190 and output of various commands to and from different files. 52 00:02:17,190 --> 00:02:18,810 There are various operators that are used 53 00:02:18,810 --> 00:02:21,090 to redirect input or output. 54 00:02:21,090 --> 00:02:24,060 For example, the operator is the greater than sign 55 00:02:24,060 --> 00:02:25,590 is going to be used to redirect 56 00:02:25,590 --> 00:02:28,380 the standard output to a given file. 57 00:02:28,380 --> 00:02:33,380 For example, if you type ls greater than file1.txt, 58 00:02:33,510 --> 00:02:36,330 the output of the ls command will redirect itself 59 00:02:36,330 --> 00:02:39,660 into a file called file1.txt. 60 00:02:39,660 --> 00:02:42,300 If you use the operator greater than greater than, 61 00:02:42,300 --> 00:02:44,550 it's going to be used to append the standard output 62 00:02:44,550 --> 00:02:46,200 to the end of the destination file 63 00:02:46,200 --> 00:02:47,910 instead of overwriting it. 64 00:02:47,910 --> 00:02:52,530 So you could type ls greater than greater than file1.txt 65 00:02:52,530 --> 00:02:54,300 and the output of the ls command 66 00:02:54,300 --> 00:02:58,410 will be appended to the file named file1.txt. 67 00:02:58,410 --> 00:03:01,020 The operator two greater than is going to be used 68 00:03:01,020 --> 00:03:04,230 to redirect the standard error messages to a file. 69 00:03:04,230 --> 00:03:08,880 For example, let's assume that file3.txt does not exist yet 70 00:03:08,880 --> 00:03:13,880 and you type ls file3.txt two greater than errorfile.txt. 71 00:03:15,120 --> 00:03:17,370 The resulting error from the ls command 72 00:03:17,370 --> 00:03:19,920 for not finding file3.txt 73 00:03:19,920 --> 00:03:21,810 will not be displayed on your screen, 74 00:03:21,810 --> 00:03:25,170 but instead, it'll be redirected to the file that we created 75 00:03:25,170 --> 00:03:27,510 called errorfile.txt. 76 00:03:27,510 --> 00:03:30,000 If we want to append it instead of overwriting it, 77 00:03:30,000 --> 00:03:33,090 we'll use the operator two greater than greater than, 78 00:03:33,090 --> 00:03:35,280 and that way we append the standard error message 79 00:03:35,280 --> 00:03:37,050 to the end of the destination file 80 00:03:37,050 --> 00:03:38,730 instead of overwriting it. 81 00:03:38,730 --> 00:03:41,340 So again, assuming that file3.txt 82 00:03:41,340 --> 00:03:43,050 does not exist on your system 83 00:03:43,050 --> 00:03:45,840 and you type ls file3.txt 84 00:03:45,840 --> 00:03:49,080 two greater than greater than errorfile.txt, 85 00:03:49,080 --> 00:03:51,810 the resulting errors will not be displayed on the screen 86 00:03:51,810 --> 00:03:56,640 but they'll be redirected and appended to errorfile.txt. 87 00:03:56,640 --> 00:03:59,310 Now if you use the pattern ampersand greater than, 88 00:03:59,310 --> 00:04:01,980 it's going to be used to redirect both the standard output 89 00:04:01,980 --> 00:04:04,710 and the standard error messages to a file. 90 00:04:04,710 --> 00:04:08,160 So let's assume that file1.txt exists 91 00:04:08,160 --> 00:04:10,530 and file3.txt does not, 92 00:04:10,530 --> 00:04:15,530 and then we type ls file1.txt file3.txt 93 00:04:15,540 --> 00:04:20,220 ampersand greater than errorfile.txt onto the command line. 94 00:04:20,220 --> 00:04:22,740 Now the resulting output and errors 95 00:04:22,740 --> 00:04:24,480 will not be displayed on the screen, 96 00:04:24,480 --> 00:04:26,550 but instead, they're being redirected 97 00:04:26,550 --> 00:04:30,060 into that file called errorfile.txt. 98 00:04:30,060 --> 00:04:32,580 If we use the pattern of a less than sign, 99 00:04:32,580 --> 00:04:35,040 it's going to be used to read input from a file 100 00:04:35,040 --> 00:04:37,590 instead of doing it from the keyboard or mouse. 101 00:04:37,590 --> 00:04:41,640 So let's say you typed mail user@address 102 00:04:41,640 --> 00:04:45,990 less than myletter.txt and you put that on the command line. 103 00:04:45,990 --> 00:04:49,560 Well, it's going to take the file myletter.txt 104 00:04:49,560 --> 00:04:50,940 and take that as the input 105 00:04:50,940 --> 00:04:53,370 and attach it to that email message. 106 00:04:53,370 --> 00:04:57,000 Finally, if you use the operator less than less than string, 107 00:04:57,000 --> 00:04:58,770 it's going to be used to provide input data 108 00:04:58,770 --> 00:05:00,030 from the current source 109 00:05:00,030 --> 00:05:01,230 and it's only going to stop 110 00:05:01,230 --> 00:05:03,720 when the line containing the provided string you listed 111 00:05:03,720 --> 00:05:05,250 actually is seen. 112 00:05:05,250 --> 00:05:06,990 When you place this in a script, 113 00:05:06,990 --> 00:05:08,670 it's called the here document, 114 00:05:08,670 --> 00:05:10,860 which refers to a special block of code 115 00:05:10,860 --> 00:05:12,600 that contains multi-line strings 116 00:05:12,600 --> 00:05:14,580 that'll be redirected to a command. 117 00:05:14,580 --> 00:05:17,370 For example, let's say you typed in cat 118 00:05:17,370 --> 00:05:19,560 less than less than EOF 119 00:05:19,560 --> 00:05:20,820 and then you typed in a document, 120 00:05:20,820 --> 00:05:25,350 something like This is a here document. EOF. 121 00:05:25,350 --> 00:05:27,870 Well, when it reads that in and it gets to EOF, 122 00:05:27,870 --> 00:05:30,360 it's going to stop, so the cat command is going to use 123 00:05:30,360 --> 00:05:33,120 the rest of the lines in this file as the input. 124 00:05:33,120 --> 00:05:34,830 And it'll stop accepting that input 125 00:05:34,830 --> 00:05:37,020 whenever it reaches the string EOF, 126 00:05:37,020 --> 00:05:39,210 which is the end of file. 127 00:05:39,210 --> 00:05:41,310 This string can really be named anything you want. 128 00:05:41,310 --> 00:05:45,180 I just used EOF for end of file to make my point. 129 00:05:45,180 --> 00:05:47,580 Now when you're doing the output of the cat command, 130 00:05:47,580 --> 00:05:50,610 it would then show up as This is a here document 131 00:05:50,610 --> 00:05:54,450 because that's all I wrote there before the EOF was found. 132 00:05:54,450 --> 00:05:57,150 Another command we have is known as the pipe. 133 00:05:57,150 --> 00:05:58,890 Now the pipe is a command in Linux 134 00:05:58,890 --> 00:06:00,990 that lets you use two or more commands 135 00:06:00,990 --> 00:06:02,700 such as the output of one command 136 00:06:02,700 --> 00:06:04,950 that becomes serving as the input for another. 137 00:06:04,950 --> 00:06:07,290 And so, we create this chain of events. 138 00:06:07,290 --> 00:06:10,590 In short, the output of each process directs itself 139 00:06:10,590 --> 00:06:13,320 as the input to the next one in the pipe, 140 00:06:13,320 --> 00:06:15,480 so this becomes a chained pipeline. 141 00:06:15,480 --> 00:06:18,780 Now a pipe looks like a long up and down line 142 00:06:18,780 --> 00:06:21,210 and this is the symbol we call a pipe. 143 00:06:21,210 --> 00:06:23,730 Pipes help you mash together two or more commands 144 00:06:23,730 --> 00:06:26,820 at the same time and then run them consecutively. 145 00:06:26,820 --> 00:06:29,490 For example, earlier, when we talked about grep, 146 00:06:29,490 --> 00:06:33,900 I said that I can use ls dash l pipe grep audit, 147 00:06:33,900 --> 00:06:35,940 and this would actually search the directory, 148 00:06:35,940 --> 00:06:38,490 list them out and then run the grep command 149 00:06:38,490 --> 00:06:40,860 on that list of files that we found. 150 00:06:40,860 --> 00:06:43,080 This is the example of using a pipe. 151 00:06:43,080 --> 00:06:45,510 The standard output from ls dash l 152 00:06:45,510 --> 00:06:48,210 was fed into grep as the input for it 153 00:06:48,210 --> 00:06:50,790 and then searched against the pattern of audit. 154 00:06:50,790 --> 00:06:52,530 This is the idea of using pipes 155 00:06:52,530 --> 00:06:54,510 and be able to chain things together. 156 00:06:54,510 --> 00:06:56,070 Now by learning all the standard 157 00:06:56,070 --> 00:06:57,720 and common commands in Linux, 158 00:06:57,720 --> 00:06:59,730 you're going to be able to start chaining them together 159 00:06:59,730 --> 00:07:01,170 to create powerful new commands 160 00:07:01,170 --> 00:07:03,750 or pipelines by using this pipe. 161 00:07:03,750 --> 00:07:06,480 A great example of this is the xargs command. 162 00:07:06,480 --> 00:07:09,000 Now x a-r-g-s is a great command 163 00:07:09,000 --> 00:07:11,910 that read streams of data from the standard input 164 00:07:11,910 --> 00:07:14,970 and then it generates and executes command lines, 165 00:07:14,970 --> 00:07:17,460 meaning it can take the output of a command 166 00:07:17,460 --> 00:07:19,947 and pass it as an argument into another command 167 00:07:19,947 --> 00:07:22,140 and if no command is specified, 168 00:07:22,140 --> 00:07:25,530 the x a-r-g's are going to execute echo by default 169 00:07:25,530 --> 00:07:27,330 and display them to the screen. 170 00:07:27,330 --> 00:07:28,980 To run the xargs command, 171 00:07:28,980 --> 00:07:30,990 you actually run it a little bit backwards 172 00:07:30,990 --> 00:07:32,700 because you're first going to type in the command 173 00:07:32,700 --> 00:07:35,070 you want to use, the options and arguments, 174 00:07:35,070 --> 00:07:37,950 then a pipe, then xargs, its options 175 00:07:37,950 --> 00:07:40,560 and its command inside of the command line. 176 00:07:40,560 --> 00:07:42,180 So let me show you how this works. 177 00:07:42,180 --> 00:07:45,060 Let's say, for example, you want to delete all the files 178 00:07:45,060 --> 00:07:49,110 in the slash foo directory that have a .pdf extension. 179 00:07:49,110 --> 00:07:52,170 You could use xargs to automate this process. 180 00:07:52,170 --> 00:07:53,520 So you're going to use 181 00:07:53,520 --> 00:07:58,520 find /foo -type f -name "*.pdf" l xargs rm. 182 00:08:02,730 --> 00:08:05,460 So what we're doing is we're running the find command first 183 00:08:05,460 --> 00:08:07,290 to find all those PDFs. 184 00:08:07,290 --> 00:08:10,350 Then we take that and pass it to the xargs command 185 00:08:10,350 --> 00:08:11,910 and it's going to run the remove command 186 00:08:11,910 --> 00:08:13,950 on all the results we found. 187 00:08:13,950 --> 00:08:16,170 Now because the results from the find command 188 00:08:16,170 --> 00:08:17,940 are going to be deliminated by a space, 189 00:08:17,940 --> 00:08:21,330 the xarg command is going to run the rm, or remove command, 190 00:08:21,330 --> 00:08:23,070 for each file in the results, 191 00:08:23,070 --> 00:08:26,460 removing all the PDF files in that directory. 192 00:08:26,460 --> 00:08:29,190 This command has a lot of different options available, 193 00:08:29,190 --> 00:08:30,780 but we're going to cover just a few of 'em here 194 00:08:30,780 --> 00:08:32,190 that are really important. 195 00:08:32,190 --> 00:08:35,909 For example, you have the dash l and a replacement string. 196 00:08:35,909 --> 00:08:38,400 You can use this if you want to consider each line 197 00:08:38,400 --> 00:08:41,250 in the standard input as a single argument. 198 00:08:41,250 --> 00:08:44,370 If you use dash capital L and a line number, 199 00:08:44,370 --> 00:08:46,530 it'll read a specified number of lines 200 00:08:46,530 --> 00:08:47,910 from the standard input 201 00:08:47,910 --> 00:08:50,940 and concatenate them into one large string. 202 00:08:50,940 --> 00:08:53,250 If you use dash p, this is going to be used 203 00:08:53,250 --> 00:08:55,920 to prompt the user before each command. 204 00:08:55,920 --> 00:08:58,590 If you use dash n in the number of arguments, 205 00:08:58,590 --> 00:09:00,690 this is going to allow you to read the maximum number 206 00:09:00,690 --> 00:09:02,910 of arguments from the standard input 207 00:09:02,910 --> 00:09:05,850 and insert them at the end of the command template. 208 00:09:05,850 --> 00:09:08,970 The option dash capital E in end of string 209 00:09:08,970 --> 00:09:11,430 is going to represent the end of the standard input. 210 00:09:11,430 --> 00:09:12,990 Similar to what we used earlier 211 00:09:12,990 --> 00:09:15,510 when we talked about the operator of less than less than 212 00:09:15,510 --> 00:09:17,160 with the string being provided. 213 00:09:17,160 --> 00:09:19,170 If we use the option dash t, 214 00:09:19,170 --> 00:09:21,030 this is going to be used to write each command 215 00:09:21,030 --> 00:09:24,570 to the standard error output before executing the command. 216 00:09:24,570 --> 00:09:27,360 And if we use dash s in the maximum size, 217 00:09:27,360 --> 00:09:29,730 this can be used to set the maximum allowable size 218 00:09:29,730 --> 00:09:33,450 of an argument list to the specified number of characters. 219 00:09:33,450 --> 00:09:36,120 The next command I want to talk about is tee. 220 00:09:36,120 --> 00:09:39,480 Tee is spelled t-e-e like a golf tee. 221 00:09:39,480 --> 00:09:42,030 Now the tee command reads the standard input 222 00:09:42,030 --> 00:09:44,760 and sends the output to the default output device, 223 00:09:44,760 --> 00:09:46,470 which is the command line interface, 224 00:09:46,470 --> 00:09:51,000 but it also copies that output to each specified file. 225 00:09:51,000 --> 00:09:52,590 This command enables you to verify 226 00:09:52,590 --> 00:09:54,540 the output of a command immediately 227 00:09:54,540 --> 00:09:56,010 by seeing it on the screen 228 00:09:56,010 --> 00:09:59,490 but also storing that to a file for later reference. 229 00:09:59,490 --> 00:10:01,590 When used with the dash a option, 230 00:10:01,590 --> 00:10:03,990 tee will append the output to the output file 231 00:10:03,990 --> 00:10:06,000 instead of overwriting it. 232 00:10:06,000 --> 00:10:07,260 To be able to run the tee command, 233 00:10:07,260 --> 00:10:09,900 simply type in the command you want to run, 234 00:10:09,900 --> 00:10:13,980 the options and the arguments, the pipe and then tee, 235 00:10:13,980 --> 00:10:15,300 the options for tee 236 00:10:15,300 --> 00:10:17,520 and the file names you want to write it to. 237 00:10:17,520 --> 00:10:19,470 For example, let's say you wanted to check 238 00:10:19,470 --> 00:10:20,910 the contents of a directory 239 00:10:20,910 --> 00:10:23,220 and also output those contents to a file 240 00:10:23,220 --> 00:10:25,020 so you can process them later. 241 00:10:25,020 --> 00:10:27,090 You could issue separate commands to do this 242 00:10:27,090 --> 00:10:30,450 or you can use the tee command and do it all at once. 243 00:10:30,450 --> 00:10:35,450 So you can do this by typing ls -l pipe tee listing.txt, 244 00:10:36,210 --> 00:10:38,790 and that way, it'll display the list to the screen 245 00:10:38,790 --> 00:10:42,540 and save a copy in the listing.txt file. 246 00:10:42,540 --> 00:10:44,640 Now another interesting thing about Linux 247 00:10:44,640 --> 00:10:46,920 is that it hosts a lot of virtual devices 248 00:10:46,920 --> 00:10:48,810 for a lot of different purposes. 249 00:10:48,810 --> 00:10:50,430 As far as the programs are concerned 250 00:10:50,430 --> 00:10:51,900 that are running on this system, 251 00:10:51,900 --> 00:10:55,440 these virtual devices act as if they're real files, 252 00:10:55,440 --> 00:10:57,750 but tools can request and feed data 253 00:10:57,750 --> 00:11:01,050 from these different sources by accessing these files. 254 00:11:01,050 --> 00:11:03,210 The data generated by the operating system, 255 00:11:03,210 --> 00:11:05,100 instead of reading them from a disc, 256 00:11:05,100 --> 00:11:07,500 is actually done as part of these files. 257 00:11:07,500 --> 00:11:09,990 For example, we have one virtual interface 258 00:11:09,990 --> 00:11:12,600 known as /dev/null. 259 00:11:12,600 --> 00:11:14,340 It's a special file that's present 260 00:11:14,340 --> 00:11:17,430 on every single Linux system you'll ever work on. 261 00:11:17,430 --> 00:11:21,270 However, unlike most virtual files, instead of reading, 262 00:11:21,270 --> 00:11:23,460 this one is used for writing. 263 00:11:23,460 --> 00:11:26,100 Now whatever you write to /dev/null 264 00:11:26,100 --> 00:11:28,380 is actually going to be discarded and forgotten. 265 00:11:28,380 --> 00:11:30,510 Essentially, it's a black hole. 266 00:11:30,510 --> 00:11:32,880 So if you have a lot of data coming in off an interface 267 00:11:32,880 --> 00:11:34,560 and it may be causing a denial of service to you 268 00:11:34,560 --> 00:11:35,790 or something like that, 269 00:11:35,790 --> 00:11:38,880 you can actually redirect that to /dev/null 270 00:11:38,880 --> 00:11:41,760 and it'll go off into the void and just be forgotten. 271 00:11:41,760 --> 00:11:43,290 Now a running process in Linux 272 00:11:43,290 --> 00:11:46,110 can also be controlled by the command line environment 273 00:11:46,110 --> 00:11:47,430 inside the terminal 274 00:11:47,430 --> 00:11:50,100 and multiple terminals can be run at once. 275 00:11:50,100 --> 00:11:51,090 When you're doing this, 276 00:11:51,090 --> 00:11:53,520 you have to have a way to control these different terminals 277 00:11:53,520 --> 00:11:56,070 and we do that by assigning an identifier. 278 00:11:56,070 --> 00:12:01,070 The identifier takes the form of /dev/tty with a number. 279 00:12:01,200 --> 00:12:03,870 And that number is unique to that terminal. 280 00:12:03,870 --> 00:12:06,000 You can then redirect standard input and output 281 00:12:06,000 --> 00:12:08,340 to another controlling terminal if you want to 282 00:12:08,340 --> 00:12:12,240 by referencing its /dev/tty number. 283 00:12:12,240 --> 00:12:13,320 This can be really useful 284 00:12:13,320 --> 00:12:14,940 if you need to redirect text streams 285 00:12:14,940 --> 00:12:16,740 between different running processes 286 00:12:16,740 --> 00:12:18,490 and other things like that as well.