1 1 00:00:00,390 --> 00:00:01,710 Proxy logs. 2 2 00:00:01,710 --> 00:00:04,950 In this lesson we're going to discuss proxy logs. 3 3 00:00:04,950 --> 00:00:07,440 Now going back to your earlier certification studies, 4 4 00:00:07,440 --> 00:00:09,240 you should remember that a proxy server 5 5 00:00:09,240 --> 00:00:12,120 acts as a gateway between you and the internet. 6 6 00:00:12,120 --> 00:00:14,430 Basically, it's an intermediary server, 7 7 00:00:14,430 --> 00:00:15,840 that separates the end users 8 8 00:00:15,840 --> 00:00:18,000 from the websites they're trying to browse. 9 9 00:00:18,000 --> 00:00:20,160 Proxy servers are going to provide varying levels 10 10 00:00:20,160 --> 00:00:22,530 of functionality, security, and privacy, 11 11 00:00:22,530 --> 00:00:24,420 depending on your use case, your needs, 12 12 00:00:24,420 --> 00:00:26,520 and your company policy. 13 13 00:00:26,520 --> 00:00:28,410 Now, the first type of proxy server we have 14 14 00:00:28,410 --> 00:00:30,900 is what's known as a forward proxy. 15 15 00:00:30,900 --> 00:00:33,030 Now, a forward proxy is a server that mediates 16 16 00:00:33,030 --> 00:00:35,850 the communications between a client and another server. 17 17 00:00:35,850 --> 00:00:38,070 And it can filter or modify communications, 18 18 00:00:38,070 --> 00:00:41,070 and provide caching services to improve performance. 19 19 00:00:41,070 --> 00:00:43,710 Now basically, a forward proxy is going to act on behalf 20 20 00:00:43,710 --> 00:00:45,810 of your internal host or workstation, 21 21 00:00:45,810 --> 00:00:50,070 and forward their HTTP request to the intended destination. 22 22 00:00:50,070 --> 00:00:53,520 Basically, when I'm sitting at work, I'm on my computer, 23 23 00:00:53,520 --> 00:00:55,950 it's going to go from me to the proxy server, 24 24 00:00:55,950 --> 00:00:57,300 and then from the proxy server, 25 25 00:00:57,300 --> 00:00:59,550 out to the website I want to connect to. 26 26 00:00:59,550 --> 00:01:02,490 Now, proxies can be classified in one of two ways. 27 27 00:01:02,490 --> 00:01:05,430 They can either be non-transparent, or transparent. 28 28 00:01:05,430 --> 00:01:07,830 When I talk about a non-transparent proxy, 29 29 00:01:07,830 --> 00:01:09,660 this is a server that redirects requests 30 30 00:01:09,660 --> 00:01:11,460 and responses from the client's configured, 31 31 00:01:11,460 --> 00:01:14,100 with the proxy address and their port. 32 32 00:01:14,100 --> 00:01:16,080 Now, these non-transparent proxies 33 33 00:01:16,080 --> 00:01:18,450 are ones where you know there's a proxy being there. 34 34 00:01:18,450 --> 00:01:20,640 If you go into your internet settings of your browser, 35 35 00:01:20,640 --> 00:01:22,740 you're going to see that there is a proxy configured, 36 36 00:01:22,740 --> 00:01:25,530 and that your traffic is going from you to that proxy, 37 37 00:01:25,530 --> 00:01:28,050 and from that proxy out to the internet. 38 38 00:01:28,050 --> 00:01:30,480 Now on the other hand, we have transparent proxies. 39 39 00:01:30,480 --> 00:01:33,240 These are forced, or intercepting proxies. 40 40 00:01:33,240 --> 00:01:36,030 This is when a server redirects requests and responses, 41 41 00:01:36,030 --> 00:01:39,300 without the clients being explicitly configured to use it. 42 42 00:01:39,300 --> 00:01:41,460 Now, this is often used if you want to make sure 43 43 00:01:41,460 --> 00:01:43,770 your employees aren't turning off your proxy, 44 44 00:01:43,770 --> 00:01:45,930 because this is done at a network layer, 45 45 00:01:45,930 --> 00:01:47,820 so all the machines are going to go through 46 46 00:01:47,820 --> 00:01:49,350 this transparent proxy, 47 47 00:01:49,350 --> 00:01:51,900 not just the ones that are configured for it. 48 48 00:01:51,900 --> 00:01:53,880 Now, an analysis of your proxy logs 49 49 00:01:53,880 --> 00:01:55,800 can reveal a lot of information for you, 50 50 00:01:55,800 --> 00:01:58,800 about the exact nature of these different web requests. 51 51 00:01:58,800 --> 00:02:00,960 This includes websites that the users visit, 52 52 00:02:00,960 --> 00:02:03,030 and the contents of each request. 53 53 00:02:03,030 --> 00:02:06,060 By using a proxy server and logging all this information, 54 54 00:02:06,060 --> 00:02:07,080 you can actually understand 55 55 00:02:07,080 --> 00:02:09,030 exactly what your users are doing, 56 56 00:02:09,030 --> 00:02:12,240 which websites are going to, and how long they spent there. 57 57 00:02:12,240 --> 00:02:14,610 Now, a proxy is going to use different log formats 58 58 00:02:14,610 --> 00:02:16,170 depending on the proxy you're using, 59 59 00:02:16,170 --> 00:02:18,960 but one of the most common is the common log format. 60 60 00:02:18,960 --> 00:02:21,000 This is the same one used by web servers. 61 61 00:02:21,000 --> 00:02:22,110 We took a look at this earlier 62 62 00:02:22,110 --> 00:02:25,380 when we looked at the Microsoft logs on the firewall lesson. 63 63 00:02:25,380 --> 00:02:26,970 Here again, you see the information 64 64 00:02:26,970 --> 00:02:28,830 in that last couple of lines. 65 65 00:02:28,830 --> 00:02:31,650 This tells you the date, the time, the destination, 66 66 00:02:31,650 --> 00:02:34,230 the type of request it was, what port it went to, 67 67 00:02:34,230 --> 00:02:36,630 what the action was, and things like that. 68 68 00:02:36,630 --> 00:02:38,040 This is the same type of log file 69 69 00:02:38,040 --> 00:02:40,830 that we talked about when we talked about Windows firewalls. 70 70 00:02:40,830 --> 00:02:42,510 Inside this, you'll see information 71 71 00:02:42,510 --> 00:02:43,890 like the user ID of the client 72 72 00:02:43,890 --> 00:02:45,750 when it's authenticated to the proxy, 73 73 00:02:45,750 --> 00:02:47,250 the request method that was being used, 74 74 00:02:47,250 --> 00:02:49,170 like a get or a post request. 75 75 00:02:49,170 --> 00:02:50,610 It'll tell you the status code, 76 76 00:02:50,610 --> 00:02:54,570 such as status code 200, or 403, or 404, 77 77 00:02:54,570 --> 00:02:55,740 and the size in bytes, 78 78 00:02:55,740 --> 00:02:58,200 and the type of information that was returned to the client. 79 79 00:02:58,200 --> 00:02:59,490 Whether it was a text file, 80 80 00:02:59,490 --> 00:03:01,530 or a picture, or something like that. 81 81 00:03:01,530 --> 00:03:03,930 Now, proxies that are set to intercept or block traffic, 82 82 00:03:03,930 --> 00:03:06,450 can also record the rule that the request matched, 83 83 00:03:06,450 --> 00:03:08,310 to determine the employee's intent. 84 84 00:03:08,310 --> 00:03:10,920 So if somebody is going out and trying to access something, 85 85 00:03:10,920 --> 00:03:12,960 you can determine was there malicious intent, 86 86 00:03:12,960 --> 00:03:14,280 or harmless intent. 87 87 00:03:14,280 --> 00:03:16,470 Let's take a look at an example here. 88 88 00:03:16,470 --> 00:03:19,500 Here on the screen you can see a Squid access table. 89 89 00:03:19,500 --> 00:03:21,090 This comes from a Squid logging program, 90 90 00:03:21,090 --> 00:03:22,590 which is a proxy server. 91 91 00:03:22,590 --> 00:03:24,660 Now here, if we start from the bottom up, 92 92 00:03:24,660 --> 00:03:27,420 we go from the oldest entries to the newest entries. 93 93 00:03:27,420 --> 00:03:28,710 As we look through these logs, 94 94 00:03:28,710 --> 00:03:30,510 what do we think is happening here? 95 95 00:03:30,510 --> 00:03:33,000 Well, as we start looking, we start seeing this website, 96 96 00:03:33,000 --> 00:03:37,410 515web.net, and you can see there was a code 200. 97 97 00:03:37,410 --> 00:03:39,750 If you remember back to our HTTP lesson, 98 98 00:03:39,750 --> 00:03:42,240 code 200 means successful attempt. 99 99 00:03:42,240 --> 00:03:45,420 So this was an okay, permitted access action, 100 100 00:03:45,420 --> 00:03:47,370 at this particular time. 101 101 00:03:47,370 --> 00:03:49,380 Now, about seven minutes later, they tried again, 102 102 00:03:49,380 --> 00:03:53,280 and instead they went to www.515web.net. 103 103 00:03:53,280 --> 00:03:55,140 And again, this was allowed. 104 104 00:03:55,140 --> 00:03:57,030 Now about 90 seconds after that, 105 105 00:03:57,030 --> 00:03:59,250 we see they tried to go back to that same website, 106 106 00:03:59,250 --> 00:04:04,250 www.515web.net, and you'll notice here, there's a 403 error. 107 107 00:04:04,890 --> 00:04:06,420 What happened that's different? 108 108 00:04:06,420 --> 00:04:08,670 Well, between that 90 seconds, 109 109 00:04:08,670 --> 00:04:10,770 somebody released a policy update, 110 110 00:04:10,770 --> 00:04:12,780 and it told the proxy server, for now on, 111 111 00:04:12,780 --> 00:04:15,510 that website is considered something that's not allowed, 112 112 00:04:15,510 --> 00:04:17,460 and so it's being blocked by the proxy. 113 113 00:04:17,460 --> 00:04:21,420 And you can see that there with TCP_denied as the status, 114 114 00:04:21,420 --> 00:04:23,160 and a 403 error. 115 115 00:04:23,160 --> 00:04:25,200 This is just some of the information you can find 116 116 00:04:25,200 --> 00:04:27,180 when looking through these proxy logs. 117 117 00:04:27,180 --> 00:04:29,040 This particular snippet is only showing 118 118 00:04:29,040 --> 00:04:32,190 a couple of the columns, you're not seeing all the content, 119 119 00:04:32,190 --> 00:04:34,530 and the mime data inside of those requests, 120 120 00:04:34,530 --> 00:04:36,690 I just showed you the status and the address, 121 121 00:04:36,690 --> 00:04:40,041 and the IP address and the date for this particular example. 122 122 00:04:40,041 --> 00:04:41,910 Now, another type of proxy we have, 123 123 00:04:41,910 --> 00:04:43,650 is known as a reverse proxy, 124 124 00:04:43,650 --> 00:04:46,260 and this is a type of proxy server that protects servers 125 125 00:04:46,260 --> 00:04:48,930 from direct contact with the client request. 126 126 00:04:48,930 --> 00:04:50,340 The idea with a reverse proxy 127 127 00:04:50,340 --> 00:04:53,760 is to provide for protocol specific inbound traffic. 128 128 00:04:53,760 --> 00:04:55,560 When we get a request from the public internet, 129 129 00:04:55,560 --> 00:04:57,630 it goes to the proxy server first. 130 130 00:04:57,630 --> 00:05:00,870 Then, that proxy server can create the appropriate request 131 131 00:05:00,870 --> 00:05:03,330 to the internal server, whether it's a mail server, 132 132 00:05:03,330 --> 00:05:06,840 or a web server, or something like that, get that request, 133 133 00:05:06,840 --> 00:05:09,180 and then send it back to the external client. 134 134 00:05:09,180 --> 00:05:10,830 This means the external client 135 135 00:05:10,830 --> 00:05:12,810 never directly touches your servers, 136 136 00:05:12,810 --> 00:05:15,000 it only touches your proxy server, 137 137 00:05:15,000 --> 00:05:17,940 and this can help protect things from malicious traffic. 138 138 00:05:17,940 --> 00:05:19,560 Now the other great thing about doing this, 139 139 00:05:19,560 --> 00:05:21,780 is you get a great source of logs 140 140 00:05:21,780 --> 00:05:23,610 right at that reverse proxy, 141 141 00:05:23,610 --> 00:05:26,640 that you can analyze for indicators of attack or compromise, 142 142 00:05:26,640 --> 00:05:30,600 such as malicious code and HTTP request headers or URLs. 143 143 00:05:30,600 --> 00:05:33,720 This way, you have a single point to check all those logs, 144 144 00:05:33,720 --> 00:05:36,300 because everything is going in through that proxy, 145 145 00:05:36,300 --> 00:05:39,420 anything from the outside internet coming into your network 146 146 00:05:39,420 --> 00:05:41,490 to be able to touch one of your internal servers, 147 147 00:05:41,490 --> 00:05:43,560 so this is a great way to do this. 148 148 00:05:43,560 --> 00:05:45,750 Now here on the screen, you could see another example 149 149 00:05:45,750 --> 00:05:48,000 of a Squid log from a proxy server. 150 150 00:05:48,000 --> 00:05:50,370 This one is coming from a reverse proxy. 151 151 00:05:50,370 --> 00:05:54,270 Notice, everything coming in from the IPs are external IPs, 152 152 00:05:54,270 --> 00:05:56,610 every destination is an internal server, 153 153 00:05:56,610 --> 00:05:58,050 and the proxy here is in the middle, 154 154 00:05:58,050 --> 00:05:59,700 and it's the one that's seeing the information, 155 155 00:05:59,700 --> 00:06:01,920 and passing those requests back and forth, 156 156 00:06:01,920 --> 00:06:04,950 from the external side over to the internal server. 157 157 00:06:04,950 --> 00:06:08,100 That way your internal servers only touch your proxy server, 158 158 00:06:08,100 --> 00:06:10,620 they do not touch the external client directly, 159 159 00:06:10,620 --> 00:06:11,970 and this gives us that single point 160 160 00:06:11,970 --> 00:06:14,400 where we can start looking at all of the status messages, 161 161 00:06:14,400 --> 00:06:16,140 and start doing statistical analysis 162 162 00:06:16,140 --> 00:06:17,760 of those different response codes, 163 163 00:06:17,760 --> 00:06:19,680 to be able to identify any suspicious trends, 164 164 00:06:19,680 --> 00:06:22,180 or anomalous deviations from the baseline traffic.