1 00:00:00,020 --> 00:00:00,990 In this lesson, 2 00:00:00,990 --> 00:00:02,910 we're going to talk about logging services, 3 00:00:02,910 --> 00:00:04,890 including what Linux logs are, 4 00:00:04,890 --> 00:00:06,180 where you can find these logs 5 00:00:06,180 --> 00:00:07,980 and how you can interpret them. 6 00:00:07,980 --> 00:00:09,671 Now, security is not just a process 7 00:00:09,671 --> 00:00:11,970 of designing a hardened system, 8 00:00:11,970 --> 00:00:14,312 but you also may be able to identify malicious behavior 9 00:00:14,312 --> 00:00:17,460 or misconfigurations when those things happen, 10 00:00:17,460 --> 00:00:18,660 and the best way to do that 11 00:00:18,660 --> 00:00:21,690 is by generating and maintaining logs of those events 12 00:00:21,690 --> 00:00:24,240 and going through those logs to understand that. 13 00:00:24,240 --> 00:00:26,280 Operating system logs provide a wealth 14 00:00:26,280 --> 00:00:28,680 of diagnostic information about your computer 15 00:00:28,680 --> 00:00:30,870 and Linux is no exception. 16 00:00:30,870 --> 00:00:33,360 Everything from kernel events to user actions 17 00:00:33,360 --> 00:00:34,890 is being logged by Linux. 18 00:00:34,890 --> 00:00:36,900 And this allows you to see almost any action 19 00:00:36,900 --> 00:00:40,290 that's been performed on your servers or workstations. 20 00:00:40,290 --> 00:00:41,940 Now, system logs are a record 21 00:00:41,940 --> 00:00:43,320 of system activities and events 22 00:00:43,320 --> 00:00:46,590 that are tracked and maintained by the syslogd daemon. 23 00:00:46,590 --> 00:00:48,090 Now, system logs are going to use 24 00:00:48,090 --> 00:00:50,070 the syslog standard by default 25 00:00:50,070 --> 00:00:52,650 and this facilitates a centralized logging server 26 00:00:52,650 --> 00:00:55,050 that can then receive and process syslog data 27 00:00:55,050 --> 00:00:57,870 from many different system systems across your network. 28 00:00:57,870 --> 00:00:59,820 This is called remote logging. 29 00:00:59,820 --> 00:01:03,060 Now, the syslog standard also supports local logging 30 00:01:03,060 --> 00:01:05,099 where these logs are going to be stored on the same system 31 00:01:05,099 --> 00:01:07,740 that generates them, but for best security, 32 00:01:07,740 --> 00:01:09,180 you should set up a syslog server 33 00:01:09,180 --> 00:01:12,300 and offload your logs over to that syslog server 34 00:01:12,300 --> 00:01:14,010 for centralization. 35 00:01:14,010 --> 00:01:15,750 Now, in most Linux distributions, 36 00:01:15,750 --> 00:01:18,090 system logs are going to be located by default, 37 00:01:18,090 --> 00:01:21,120 in your /var/log directory. 38 00:01:21,120 --> 00:01:23,670 Inside this directory are the logs themselves 39 00:01:23,670 --> 00:01:26,070 and each file corresponds to a different service, 40 00:01:26,070 --> 00:01:29,040 application or feature of the operating system. 41 00:01:29,040 --> 00:01:32,370 Now, /var/log/syslog for example, 42 00:01:32,370 --> 00:01:34,350 is going to contain all the system events 43 00:01:34,350 --> 00:01:36,720 except for the authentication messages. 44 00:01:36,720 --> 00:01:41,460 Those are going to be kept in /var/log/auth.log, 45 00:01:41,460 --> 00:01:43,980 which is going to contain all your authentication messages, 46 00:01:43,980 --> 00:01:46,860 such as your login successes and failures. 47 00:01:46,860 --> 00:01:49,050 Now, both of these are going to be used primarily 48 00:01:49,050 --> 00:01:50,700 by Debian-based distros. 49 00:01:50,700 --> 00:01:52,650 So if you're using something like Kali Linux 50 00:01:52,650 --> 00:01:53,801 or another Debian-based distro, 51 00:01:53,801 --> 00:01:55,860 that's where you'll find them. 52 00:01:55,860 --> 00:01:56,700 Now, on the other hand, 53 00:01:56,700 --> 00:01:58,650 if you're using Red Hat Enterprise Linux 54 00:01:58,650 --> 00:02:01,110 or CentOS like we are in this class, 55 00:02:01,110 --> 00:02:02,700 you're going to find these messages kept 56 00:02:02,700 --> 00:02:06,180 in the /var/log/messages 57 00:02:06,180 --> 00:02:08,460 for all of your non-critical system events. 58 00:02:08,460 --> 00:02:10,199 And all of your authentication messages 59 00:02:10,199 --> 00:02:14,070 are going to be kept in /var/log/secure, 60 00:02:14,070 --> 00:02:15,780 where all of your login successes 61 00:02:15,780 --> 00:02:17,760 and failures are going to be kept. 62 00:02:17,760 --> 00:02:18,779 Now, in both of these systems, 63 00:02:18,779 --> 00:02:21,540 if you want to find out information about the kernel, 64 00:02:21,540 --> 00:02:26,540 you can look at the kernel log in /var/log/kern.log 65 00:02:27,870 --> 00:02:30,420 and that file is going to have all of your kernel messages, 66 00:02:30,420 --> 00:02:31,950 such as all the output you're going to get 67 00:02:31,950 --> 00:02:34,650 from the DMESG command. 68 00:02:34,650 --> 00:02:35,729 Now, another thing you want to look at, 69 00:02:35,729 --> 00:02:38,490 is for any of your application files. 70 00:02:38,490 --> 00:02:42,480 Most of these are going to be kept in /var/log/ 71 00:02:42,480 --> 00:02:44,130 and the name of the application. 72 00:02:44,130 --> 00:02:46,020 For example, if you're looking for logs 73 00:02:46,020 --> 00:02:49,260 about cron or firewalld or mailog, 74 00:02:49,260 --> 00:02:54,260 you can find those in /var/log/firewalld for example. 75 00:02:54,600 --> 00:02:57,450 Now, log rotation is another thing we need to think about. 76 00:02:57,450 --> 00:03:00,000 Log rotation is the practice creating new versions 77 00:03:00,000 --> 00:03:03,600 of a log file to maintain a minimum log file size. 78 00:03:03,600 --> 00:03:05,610 Now, there's a utility to help us with this 79 00:03:05,610 --> 00:03:08,070 and perform automatic rotation of our logs. 80 00:03:08,070 --> 00:03:10,350 It's known as log rotate. 81 00:03:10,350 --> 00:03:12,420 Now, log rotate, when you execute it, 82 00:03:12,420 --> 00:03:15,180 is going to add a 0.1 to the end of the file name 83 00:03:15,180 --> 00:03:17,580 of the current version of the log file. 84 00:03:17,580 --> 00:03:20,576 Previously rotated files are suffixed with 0.2, 85 00:03:20,576 --> 00:03:22,620 0.3 and so on. 86 00:03:22,620 --> 00:03:23,955 Using this automatic rotation, 87 00:03:23,955 --> 00:03:27,720 all versions of a log file will be maintained on that system 88 00:03:27,720 --> 00:03:30,090 until you specify otherwise. 89 00:03:30,090 --> 00:03:32,790 Now, log rotation can actually be configured as well 90 00:03:32,790 --> 00:03:34,530 by changing the behavior of it. 91 00:03:34,530 --> 00:03:36,726 To be able to configure this log rotation behavior, 92 00:03:36,726 --> 00:03:38,940 you're going to set up configuration files 93 00:03:38,940 --> 00:03:43,260 inside the /etc/logrotate.d directory. 94 00:03:43,260 --> 00:03:45,780 When you do this, you'll be able to define the logs 95 00:03:45,780 --> 00:03:48,360 for that file and how large they should be, 96 00:03:48,360 --> 00:03:50,730 what permissions those log files should have 97 00:03:50,730 --> 00:03:52,616 and how many log files should you keep 98 00:03:52,616 --> 00:03:55,530 before deleting them to make space. 99 00:03:55,530 --> 00:03:56,880 The next thing we're going to talk about 100 00:03:56,880 --> 00:04:00,240 is the configuration for the rsyslogd service. 101 00:04:00,240 --> 00:04:01,655 You can find the configuration file 102 00:04:01,655 --> 00:04:05,460 at /etc/rsyslog.conf 103 00:04:05,460 --> 00:04:07,830 and be able to edit this configuration file. 104 00:04:07,830 --> 00:04:09,180 Now, this file is going to determine 105 00:04:09,180 --> 00:04:11,310 how you're going to handle syslog messages 106 00:04:11,310 --> 00:04:13,380 through a variety of rules that you can modify 107 00:04:13,380 --> 00:04:14,820 to suit your needs. 108 00:04:14,820 --> 00:04:17,430 The file takes up a two column format. 109 00:04:17,430 --> 00:04:21,209 The first column lists the message facilities or severities. 110 00:04:21,209 --> 00:04:23,610 The severities are defined in word format 111 00:04:23,610 --> 00:04:26,760 rather than the normal numbers of zero through seven. 112 00:04:26,760 --> 00:04:28,350 The second column is going to define 113 00:04:28,350 --> 00:04:30,750 what action should be taken for those messages 114 00:04:30,750 --> 00:04:33,960 that correspond to that facility or severity. 115 00:04:33,960 --> 00:04:35,310 The next thing we're going to talk about 116 00:04:35,310 --> 00:04:37,470 is the new replacement for syslogd, 117 00:04:37,470 --> 00:04:42,090 which is known as syslog-ng, or next generation. 118 00:04:42,090 --> 00:04:45,450 Now, syslog-ng is a replacement for syslogd 119 00:04:45,450 --> 00:04:47,310 and it offers similar functionality 120 00:04:47,310 --> 00:04:50,790 to the old syslog or the rsyslogd daemon. 121 00:04:50,790 --> 00:04:53,640 Now, syslog-ng does have a different syntax 122 00:04:53,640 --> 00:04:56,370 than the former syslog or rsyslog service, 123 00:04:56,370 --> 00:04:58,890 so keep that in mind when you're configuring it. 124 00:04:58,890 --> 00:05:00,090 Now, another thing to think about 125 00:05:00,090 --> 00:05:02,460 is your centralized administration capabilities 126 00:05:02,460 --> 00:05:04,650 and when you do the syslog ability 127 00:05:04,650 --> 00:05:06,270 by grabbing all of that data, 128 00:05:06,270 --> 00:05:08,070 it really helps centralize your ability 129 00:05:08,070 --> 00:05:09,570 to manage your systems. 130 00:05:09,570 --> 00:05:10,800 This gives you greater control 131 00:05:10,800 --> 00:05:12,510 over your logging and your network 132 00:05:12,510 --> 00:05:14,880 and it really is a great way to do it. 133 00:05:14,880 --> 00:05:18,120 Unfortunately though, syslog is not universally supported 134 00:05:18,120 --> 00:05:19,890 by all platforms. 135 00:05:19,890 --> 00:05:23,730 Windows, for example, by default uses a proprietary format 136 00:05:23,730 --> 00:05:25,890 known as the Windows event log format 137 00:05:25,890 --> 00:05:28,170 to record all of its system messages. 138 00:05:28,170 --> 00:05:30,420 So if you want to facilitate interaction 139 00:05:30,420 --> 00:05:33,360 between syslog and non-syslog platforms, 140 00:05:33,360 --> 00:05:35,760 you're going to have to use third party agents. 141 00:05:35,760 --> 00:05:37,244 Now, an agent is a software program 142 00:05:37,244 --> 00:05:40,470 that acts on behalf of some other program or service. 143 00:05:40,470 --> 00:05:42,960 Essentially, you're going to install a syslog agent 144 00:05:42,960 --> 00:05:44,850 on that platform and that way, 145 00:05:44,850 --> 00:05:47,130 even if it doesn't support that standard natively, 146 00:05:47,130 --> 00:05:49,524 like Windows, you can then still collect that information 147 00:05:49,524 --> 00:05:52,320 and send it back in a syslog format. 148 00:05:52,320 --> 00:05:53,790 Now, which agent you're going to install 149 00:05:53,790 --> 00:05:55,950 is going to depend on the platform you want to monitor, 150 00:05:55,950 --> 00:05:58,350 as well as the feature sets that you're trying to target. 151 00:05:58,350 --> 00:06:01,320 For example, rsyslog and syslog-ng 152 00:06:01,320 --> 00:06:03,600 both require their own agent software 153 00:06:03,600 --> 00:06:05,430 and it's not the same one. 154 00:06:05,430 --> 00:06:07,020 Once that agent is installed though, 155 00:06:07,020 --> 00:06:09,210 you'll be able to configure it to capture messages 156 00:06:09,210 --> 00:06:11,670 in a syslog format and send those messages 157 00:06:11,670 --> 00:06:14,370 over to your centralized syslog server. 158 00:06:14,370 --> 00:06:16,080 Now, another command you need to be aware of 159 00:06:16,080 --> 00:06:18,270 is known as journalctl. 160 00:06:18,270 --> 00:06:20,250 Now, journalctl is going to enable you 161 00:06:20,250 --> 00:06:22,050 to view and query log files 162 00:06:22,050 --> 00:06:23,760 that are created by the journal component 163 00:06:23,760 --> 00:06:25,710 of the systemd suite. 164 00:06:25,710 --> 00:06:27,690 Log information here is going to be collected 165 00:06:27,690 --> 00:06:30,900 and stored via the systemd journald service. 166 00:06:30,900 --> 00:06:32,790 And you can use the journalctl 167 00:06:32,790 --> 00:06:34,860 to print the entire journal log 168 00:06:34,860 --> 00:06:37,620 or you can issue various options to that command 169 00:06:37,620 --> 00:06:40,260 to filter that log in a number of different ways, 170 00:06:40,260 --> 00:06:43,200 such as matching a service name or only printing messages 171 00:06:43,200 --> 00:06:45,540 that match a specified severity level. 172 00:06:45,540 --> 00:06:47,610 The journald service is often used 173 00:06:47,610 --> 00:06:50,040 in conjunction with a traditional syslog daemon, 174 00:06:50,040 --> 00:06:52,293 such as syslogd or rsyslogd. 175 00:06:53,220 --> 00:06:55,200 The settings for journald are configured 176 00:06:55,200 --> 00:07:00,200 in the /etc/systemd/journald.conf file. 177 00:07:00,300 --> 00:07:02,250 To run the journalctl command, 178 00:07:02,250 --> 00:07:04,080 simply type in journalctl, 179 00:07:04,080 --> 00:07:06,360 the options and the matches. 180 00:07:06,360 --> 00:07:08,970 Now, as I said, the journalctl utility provides 181 00:07:08,970 --> 00:07:12,690 a lot of ways for you to query the journald log data. 182 00:07:12,690 --> 00:07:14,400 Some of the most frequently used options 183 00:07:14,400 --> 00:07:17,430 are things like -n and the number of lines. 184 00:07:17,430 --> 00:07:19,710 This option is used to specify the number of lines 185 00:07:19,710 --> 00:07:22,140 of the journal log you want to display. 186 00:07:22,140 --> 00:07:24,420 The -o and the output format option 187 00:07:24,420 --> 00:07:27,480 is going to be used to specify the format of the output. 188 00:07:27,480 --> 00:07:29,310 The -f option is going to be used 189 00:07:29,310 --> 00:07:31,410 to display the most recent journal entries, 190 00:07:31,410 --> 00:07:34,260 so I like to think about it as fresh for F 191 00:07:34,260 --> 00:07:35,910 and then you can continually update 192 00:07:35,910 --> 00:07:37,290 the display with new entries 193 00:07:37,290 --> 00:07:40,530 as they're added to the journal in this interactive mode. 194 00:07:40,530 --> 00:07:43,170 Then we have -p and -p is going to be used 195 00:07:43,170 --> 00:07:46,020 to filter out the log output by severity. 196 00:07:46,020 --> 00:07:48,420 If you use -u, this option is going to be used 197 00:07:48,420 --> 00:07:51,360 to filter journal log output by their specified unit, 198 00:07:51,360 --> 00:07:53,010 such as the name of the service 199 00:07:53,010 --> 00:07:55,290 and the -b [boot ID] option 200 00:07:55,290 --> 00:07:57,330 is going to be used to show log messages 201 00:07:57,330 --> 00:07:59,280 that are only from the current boot session 202 00:07:59,280 --> 00:08:01,620 or from the boot ID that you specified, 203 00:08:01,620 --> 00:08:03,960 since that is an optional characteristic. 204 00:08:03,960 --> 00:08:05,372 Now, in its default configuration, 205 00:08:05,372 --> 00:08:08,700 the systemd journal only stores logs in memory 206 00:08:08,700 --> 00:08:10,170 and these logs are going to be cleared out 207 00:08:10,170 --> 00:08:11,880 on each system reboot. 208 00:08:11,880 --> 00:08:14,160 You can also have those journald logs persist 209 00:08:14,160 --> 00:08:19,140 after a reboot by creating a /var/log/journal directory 210 00:08:19,140 --> 00:08:20,970 and that way they can be stored there. 211 00:08:20,970 --> 00:08:22,830 The systemd service is configured 212 00:08:22,830 --> 00:08:25,320 to automatically maintain logs in this directory 213 00:08:25,320 --> 00:08:26,850 if that directory exists. 214 00:08:26,850 --> 00:08:29,730 So to enable journaling and logging of your journal, 215 00:08:29,730 --> 00:08:34,200 simply create a directory at /var/log/journal. 216 00:08:34,200 --> 00:08:37,110 The next command we're going to talk about is the last command. 217 00:08:37,110 --> 00:08:39,102 Now, the last command displays the running history 218 00:08:39,102 --> 00:08:41,789 of the user login and logout events, 219 00:08:41,789 --> 00:08:43,950 along with the actual time and date. 220 00:08:43,950 --> 00:08:46,200 This also has various options that you can enable 221 00:08:46,200 --> 00:08:48,840 to filter the results, such as filtering by the users 222 00:08:48,840 --> 00:08:51,030 who have logged in through a specific terminal. 223 00:08:51,030 --> 00:08:53,400 For example, last 1 will display 224 00:08:53,400 --> 00:08:55,740 all the details of the users who logged in 225 00:08:55,740 --> 00:08:59,160 using the first terminal known as tty 1. 226 00:08:59,160 --> 00:09:01,350 The last command will also retrieve information 227 00:09:01,350 --> 00:09:05,700 from the /var/log/wtemp file. 228 00:09:05,700 --> 00:09:07,260 Now, to run the last command, 229 00:09:07,260 --> 00:09:10,620 simply type in last and the options you want to use. 230 00:09:10,620 --> 00:09:13,680 Finally, let's talk about the lastlog command. 231 00:09:13,680 --> 00:09:16,440 The lastlog command is similar to the last command, 232 00:09:16,440 --> 00:09:19,680 but instead of listing out the most recent login events, 233 00:09:19,680 --> 00:09:21,000 it's going to list all the users 234 00:09:21,000 --> 00:09:22,980 and the last time they logged in. 235 00:09:22,980 --> 00:09:24,450 This command retrieves information 236 00:09:24,450 --> 00:09:28,470 from the /var/log/lastlog file. 237 00:09:28,470 --> 00:09:30,930 So as you can see, there's lots of different ways 238 00:09:30,930 --> 00:09:33,450 to log information on your Linux system. 239 00:09:33,450 --> 00:09:36,030 It's really important that you have good logging in place, 240 00:09:36,030 --> 00:09:37,140 because if you're the victim 241 00:09:37,140 --> 00:09:39,090 of a data breach or cyber attack, 242 00:09:39,090 --> 00:09:41,220 your instant responders are going to use those logs 243 00:09:41,220 --> 00:09:42,930 to figure out what happened, 244 00:09:42,930 --> 00:09:45,000 how the bad person got into your network 245 00:09:45,000 --> 00:09:46,230 and what kind of things they did 246 00:09:46,230 --> 00:09:47,520 while they were in your network. 247 00:09:47,520 --> 00:09:49,530 So always ensure you have good logging 248 00:09:49,530 --> 00:09:52,050 and remember to set up a centralized administration server 249 00:09:52,050 --> 00:09:54,090 using syslog or something similar 250 00:09:54,090 --> 00:09:56,370 to collect all those logs in one place. 251 00:09:56,370 --> 00:09:58,670 Your instant responders will thank you for it.