Information Security Blog

November 10th, 2016

Penetration Test vs Vulnerability Assessment

Some say Potato, some say Patato. The term "Penetration Test" has been thrown around a lot in the Information Security industry. Some vendors and institutions use the term Penetration Test interchangeably with "Vulnerability Scan" (or Assessment), when in fact, the two define very different scopes, methodologies, and deliverables. The recently updated FFIEC Information Security Booklet discusses these types of tests and offers definitions and expectations of what is required of financial institutions in these areas. The short story is that yes, both are different, and yes, both are needed as part of an effective audit program.

Without wandering too far into the weeds, this post will attempt to shed some light on these two testing types.

Section IV.A.2(b) of the booklet released in September 2016 defines a Penetration Test as a test that "subjects a system to real-world attacks" and should "demonstrate a potential for loss." Penetration Tests should simulate the methods and goals of an attacker that is determined to gain access to your systems. It is designed to show how all of an organization's layered controls worked together (or did not work) to defend against a hacker. It is performed with "shields up" and often internal staff will respond to any detected attack as if it were real. The scope will generally not only consist of technical attacks, but can also include social engineering and even physical penetration attempts. The resulting deliverable reads like an after-action report, detailing from start to finish how the "attackers" found vulnerable systems, if they breached the network, and what they would have accomplished if it were a real-world scenario. The report should then list what defensive controls could have detected and stopped each attack. This helps the institution identify gaps and weaknesses that need to be addressed.

On the flip side, Section IV.A.2(c) describes a Vulnerability Assessment as a process that "identifies, and classifies the vulnerabilities in a computer, network, or communications infrastructure." These assessments are limited in scope and methodology, and are done with full access and knowledge of an organization's network. The goal with this test is not to simulate an attack, but to identify all vulnerabilities or weaknesses in a given system or environment. The subsequent report is an exhaustive list of systems, the vulnerabilities identified on each, risk classifications, and the recommended remediation steps. It does not take into account other mitigating controls or real world consequences of exploitation, it simply helps administrators and management identify vulnerabilities that need to be remediated.

Both of these are essential, useful tools that help an institution obtain a clear determination of their resilience against cyber-attack, and both paint a different part of an overall picture of the security posture of your network. If you think of your network as a medieval castle, the Vulnerability Assessment will identify all of the cracks and weaknesses in the wall, whether the drawbridge and gates were installed correctly, and whether the tower walls are tall enough to prevent climbing. A Penetration Test, on the other hand, will show how real attackers performed reconnaissance from the tree line, jumped the moat, found a hole in a wall, slipped past the guards undetected, and found the keys to the tower door (thereby eliminating the need for climbing). It will also demonstrate in a very real way how the attackers were able to successfully make off with crown jewels, showing the King and Queen just how important it is to fix the weaknesses that were exploited (and they get their jewels back!)

FFIEC guidance and examination procedures indicate that while institutions can determine the frequency and types of Penetration Tests and Vulnerability Assessments, it does not say you can pick one or the other. Both are needed to help assure that your Information Security Program is providing adequate protection from cyber-threats. Additionally, recent feedback 10-D has seen from examiners appears to confirm that all institutions, regardless of size, need to include both methodologies in their audit and testing program.

Another thing to keep in mind is that the terms are still confused by many vendors when offering proposals. Some scopes may say Penetration Test, but organizations must look closely at methodology and deliverables to determine what is actually going to be done. True Penetration Tests are conducted by skilled Security Professionals with experience tailoring available tools and techniques to attack each unique environment. They are more manual and time intensive, and thus will generally cost more than a Vulnerability Assessment. Some key components of a Penetration Test scope will be:

Vulnerability Assessments are generally a simpler scope, but methodology matters here as well. When evaluating Vulnerability Assessment scopes, a few things to look for:


FFIEC Information Security Booklet IV.A.2, Types of Tests and Evaluations:

Authored By: Jeremy Johnson, CISSP

PDF Icon ImageBlog PDF Download