Instructor: Audits and assessments. 2 2
In this lesson, we are going to talk all about audits 3 3
and assessments and all of the different components 4 4
that are comprised as part of this auditing and assessments. 5 5
Now this is going to include things like quality control 6 6
and quality assurance, verification and validation, 7 7
assessments and evaluations, audits themselves 8 8
and schedule reviews and continual improvement. 9 9
As we go through this lesson, 10 10
we're going to define each of these terms 11 11
and talk about how they differ. 12 12
First, quality control. 13 13
Now quality control is the process of determining 14 14
whether a system is free from defects or deficiencies. 15 15
For example, if you're putting together a car, 16 16
you want to go back and look at that car 17 17
and make sure it meets the quality control standards. 18 18
This is how manufacturing plants work on daily basis. 19 19
Now working with that 20 20
is something known as quality assurance. 21 21
Now quality assurance is the process that analyzes 22 22
what constitutes quality 23 23
and how it can be measured and checked. 24 24
So quality control is the actions that happen 25 25
at the time of putting something together. 26 26
So if you're building a new piece of software, for instance, 27 27
you might go through a quality control of that software. 28 28
Quality assurance is the program around it 29 29
that includes things like quality control 30 30
to make sure we reach a certain level of quality. 31 31
Now when you're dealing with QC and QA, 32 32
a lot of times this will take the form 33 33
of something known as verification and validation or V&V. 34 34
This is especially true 35 35
within the software development world. 36 36
So when we talk about this, 37 37
we have to first start with verification. 38 38
Verification is a compliance testing process 39 39
to ensure that the security system meets the requirements 40 40
of a framework or regulatory environment 41 41
or that the product or system meets its design goals. 42 42
Essentially, verification is a form of quality control 43 43
but we call it verification inside the digital world. 44 44
Now when we start dealing with validation, 45 45
this is the process of determining 46 46
whether the security system is fit for purpose. 47 47
Does it actually meet the set designs we have 48 48
and has it been installed correctly? 49 49
That's really what we talk about with validation. 50 50
Verification, we might look at a piece of code and say, 51 51
yes, this is doing what it's supposed to. 52 52
Validation is once we install that code, 53 53
has it been installed properly? 54 54
Has it been configured properly 55 55
and does it do what it's meant to do? 56 56
Is it fit for purpose? 57 57
Now fit for purpose is kind of a weird term, 58 58
it comes from the ITIL world. 59 59
In the ITIL framework, 60 60
the term fit for purpose is known as utility. 61 61
And utility really is just defined 62 62
as meeting the design needs of a software or a service 63 63
for a particular use case. 64 64
So if I have something like a word processor, 65 65
it needs to be able to do those functions 66 66
that I want a word processor to do. 67 67
If it can do those things, it has good utility, 68 68
it is fit for purpose, 69 69
I can use it for the thing I want to use it for. 70 70
Now the next two terms we're going to talk about 71 71
are assessment and evaluations. 72 72
And these also sound very similar 73 73
but there is a distinction between them. 74 74
When we talk about assessment, 75 75
this is the process of testing the subject 76 76
against a checklist of requirements 77 77
in a highly structured way for measurement 78 78
against an absolute standard. 79 79
Now that's the key here when we talk about an assessment, 80 80
it is an absolute standard that we're going against. 81 81
So often you'll see these used with a checklist approach. 82 82
Now an evaluation on the other hand 83 83
is a less methodical process that's used for testing, 84 84
that is aimed at examining the outcomes 85 85
or providing usefulness of a subject being tested. 86 86
So when I start talking about evaluation, 87 87
evaluation is much more likely to use 88 88
comparative measurements as more likely to depend 89 89
on the judgment of the evaluator 90 90
than on a checklist or a framework. 91 91
So a lot of times if you're evaluating something, 92 92
you're saying, "I think this looks good." 93 93
When you're assessing it, 94 94
you're going to have it against a checklist. 95 95
So if you ever bought a used car, 96 96
one of the things they sell you now 97 97
with the certified pre-owned used cars 98 98
is we have this 83-point checklist. 99 99
That's an assessment. 100 100
They're going to go down and check all 83 things. 101 101
Whereas if you took that same car to your local mechanic, 102 102
he might do an evaluation for you 103 103
and just look over the car, test a couple of things and say, 104 104
"Yep, looks good to me. 105 105
"I think this is a good car." 106 106
That's the difference 107 107
between an assessment and an evaluation. 108 108
The evaluation is much more fluid. 109 109
It is much more of an opinion-based thing 110 110
than a checklist based thing. 111 111
Now the next term we want to talk about is audit. 112 112
And an audit is a much more rigid process 113 113
than an assessment or an evaluation. 114 114
In this case, the auditor is going to compare the organization 115 115
against a predefined baseline to identify any areas 116 116
that require remediation. 117 117
Now audits are generally going to be required 118 118
in regulated industries. 119 119
So if you're dealing with the payment card industry 120 120
or healthcare data processing, 121 121
all of these things are highly regulated 122 122
and so you're going to go through either quarterly 123 123
or annual audits. 124 124
Now when you go through those, 125 125
they are going to do it not based on an evaluation, 126 126
not based on their opinion, 127 127
they may not even do it based on an assessment 128 128
or a checklist, they're going to have something much more rigid 129 129
and they're going to compare you against a known standard. 130 130
Everybody needs to meet this standard 131 131
whereas an assessment everyone needs to meet this standard 132 132
for your organization. 133 133
It's more tailored to you. 134 134
So that's one of the difference 135 135
there between audits and assessments. 136 136
Now we start talking about scheduled reviews. 137 137
These are similar to a lessons learned review 138 138
except that it occurs at a regular interview 139 139
such as quarterly or annually. 140 140
Now having a scheduled review is really important 141 141
and when you go through a scheduled review, 142 142
there's going to be some things you're going to be thinking about. 143 143
You're going to be thinking about the previous major incidents 144 144
you had in the last period. 145 145
You might think about the trend and analysis 146 146
of threat intelligence that might affect your company 147 147
based on the cybersecurity posture. 148 148
You might think about changes and additions 149 149
to your security controls or your systems 150 150
that may increase or decrease risk. 151 151
And you want to think about the progress you made 152 152
during the last period as you've adopted 153 153
or updated different compliance within your frameworks. 154 154
All of this is things you have to consider 155 155
as part of your scheduled reviews to see if you're on track 156 156
to make progress towards your goals. 157 157
Now speaking of making progress towards your goals, 158 158
one of the best ways to do this 159 159
is through continual improvement. 160 160
Now continual improvement is the process 161 161
of making small incremental gains to products and services 162 162
by identifying defects and inefficiencies 163 163
for further refinement. 164 164
Now what this means is we are going to do small changes 165 165
over and over and all those small changes 166 166
add up to big changes. 167 167
Now this is important because we can't take on 168 168
some big project and think we're going to get it all done 169 169
in a week, it's just not going to happen. 170 170
But if I could take that big project and say, 171 171
I have this goal of being here in a year, 172 172
I can then break that down into what do I need each week 173 173
to happen to get to that final outcome? 174 174
And this is the idea of continual improvement. 175 175
By making small changes over time, 176 176
we can make big changes over a longer period of time. 177 177
Now there are many different continual improvement methods 178 178
out there, including the Six Sigma Method, the Deming Cycle, 179 179
the Lean Method and even the Seven Step Improvement Method 180 180
within ITIL. 181 181
For the CYSA exam though, 182 182
you don't need to know any of these specific methods, 183 183
just know that it's important to be continually improving 184 184
in your organization. 185 185
All of these methods do follow a similar concept 186 186
of defining your goals, establishing a baseline 187 187
through measurement, and then analyzing it, 188 188
putting it into practice 189 189
with some kind of improvement action, 190 190
monitoring that action results and then measuring again, 191 191
making more changes and continuing through this 192 192
incremental loop over and over and over again 193 193
until you get to the final place you want to be 194 194
which ultimately is a great system that works perfectly 195 195
every single time. 196 196
Now that's the hope and the dream 197 197
but a lot of times we never get there 198 198
but we do get close by doing all these single improvements, 199 199
over time they add up to big change.