Penetration Test vs the Vulnerability Assessment

Some say Potato, some say Patato. The term"Penetration Test" has been thrown around a lot in the Information Security industry. Some vendors and institutions use the term Penetration Test interchangeably with"Vulnerability Scan" (or Assessment), when in fact, the two define very different scopes, methodologies, and deliverables. The recently updated FFIEC Information Security Booklet discusses these types of tests and offers definitions and expectations of what is required of financial institutions in these areas. The short story is that yes, both are different, and yes, both are needed as part of an effective audit program.

Without wandering too far into the weeds, this post will attempt to shed some light on these two testing types.

Penetration Test

Section IV.A.2(b) of the booklet released in September 2016 defines a penetration test as a test that"subjects a system to real-world attacks" and should"demonstrate a potential for loss." Penetration Tests should simulate the methods and goals of an attacker that is determined to gain access to your systems. It is designed to show how all of an organization's layered controls worked together (or did not work) to defend against a hacker. It is performed with"shields up" and often internal staff will respond to any detected attack as if it were real. The scope will generally not only consist of technical attacks, but can also include social engineering and even physical penetration attempts. The resulting deliverable reads like an after-action report, detailing from start to finish how the"attackers" found vulnerable systems, if they breached the network, and what they would have accomplished if it were a real-world scenario. The report should then list what defensive controls could have detected and stopped each attack. This helps the institution identify gaps and weaknesses that need to be addressed.

Vulnerability Assessment

On the flip side, Section IV.A.2(c) describes a Vulnerability Assessment as a process that"identifies, and classifies the vulnerabilities in a computer, network, or communications infrastructure." These assessments are limited in scope and methodology, and are done with full access and knowledge of an organization's network. The goal with this test is not to simulate an attack, but to identify all vulnerabilities or weaknesses in a given system or environment. The subsequent report is an exhaustive list of systems, the vulnerabilities identified on each, risk classifications, and the recommended remediation steps. It does not take into account other mitigating controls or real world consequences of exploitation, it simply helps administrators and management identify vulnerabilities that need to be remediated.

Importance of Both

Both of these are essential, useful tools that help an institution obtain a clear determination of their resilience against cyber-attack, and both paint a different part of an overall picture of the security posture of your network. If you think of your network as a medieval castle, the Vulnerability Assessment will identify all of the cracks and weaknesses in the wall, whether the drawbridge and gates were installed correctly, and whether the tower walls are tall enough to prevent climbing. A Penetration Test, on the other hand, will show how real attackers performed reconnaissance from the tree line, jumped the moat, found a hole in a wall, slipped past the guards undetected, and found the keys to the tower door (thereby eliminating the need for climbing). It will also demonstrate in a very real way how the attackers were able to successfully make off with crown jewels, showing the King and Queen just how important it is to fix the weaknesses that were exploited (and they get their jewels back!)

Frequency and Scope

FFIEC guidance and examination procedures indicate that while institutions can determine the frequency and types of Penetration Tests and Vulnerability Assessments, it does not say you can pick one or the other. Both are needed to help assure that your Information Security Program is providing adequate protection from cyber-threats. Additionally, recent feedback 10-D has seen from examiners appears to confirm that all institutions, regardless of size, need to include both methodologies in their audit and testing program.

Choosing the Right Approach

Another thing to keep in mind is that the terms are still confused by many vendors when offering proposals. Some scopes may say Penetration Test, but organizations must look closely at methodology and deliverables to determine what is actually going to be done. True Penetration Tests are conducted by skilled Security Professionals with experience tailoring available tools and techniques to attack each unique environment. They are more manual and time intensive, and thus will generally cost more than a Vulnerability Assessment. Some key components of a Penetration Test scope will be:

• Goals or"Flags" to capture. A Penetration Test is targeted, meaning that just like an attacker, the test has a goal to accomplish. This can be collecting customer information, compromising a particular server, or just gaining Domain Admin level access.
• Rules. Laying out what is in-scope and what is not in-scope is important. The rules should define what, if anything, is off limits.
• Methodology. Again, the methodology for a Penetration Test is completely different. Beware of scopes that contain"vulnerability scanning." While a Pen Tester will look for vulnerabilities, they generally will not perform full-scale vulnerability scanning. If this is involved, the scope may be more of a Vulnerability Assessment than a true Penetration Test.

Vulnerability Assessments are generally a simpler scope, but methodology matters here as well. When evaluating Vulnerability Assessment scopes, a few things to look for:

• What is scanned? It is highly recommended that all network devices be scanned. Do not scan just servers and workstations, include entire subnets. This can find issues and even devices that have fallen through the cracks.
• Authentication. As a Vulnerability Assessment does not simulate an attacker, and is more of an administrative security test, these scans should be run with Domain Administrator level credentials when assessing Windows systems, and Root credentials when scanning Unix based boxes wherever possible. This will allow the scanning software to scan each system for installed software, determine exactly what versions are present, and will be able to find vulnerabilities that un-authenticated scans cannot. Remember, admin level credentials are needed because you are scanning all systems remotely from a central location. This does not mean that an attacker would need the same privilege level to exploit the issue. A good example of this would be out of date Java. Any user that can log into the system can open Java and see the version, but if you want to check the version from a remote system on the network, you need administrative level access to perform that remote query. An attacker exploiting this issue merely needs to entice a standard user to click on link in a phishing email to exploit it and compromise the system.

Reference:

FFIEC Information Security Booklet IV.A.2(b), Penetration Tests: https://ithandbook.ffiec.gov/it-booklets/information-security/iv-information-security-program-effectiveness/iva-assurance-and-testing/iva2-types-of-tests-and-evaluations/iva2(b)-penetration-tests.aspx

FFIEC Information Security Booklet IV.A.2(c), Vulnerability Assessments: https://ithandbook.ffiec.gov/it-booklets/information-security/iv-information-security-program-effectiveness/iva-assurance-and-testing/iva2-types-of-tests-and-evaluations/iva2(c)-vulnerability-assessments.aspx

  Download Blog

Authored By: Jeremy Johnson, CISSP