Some reflections on security assurance, how it can be achieved and verified, from the view of an evaluation lab.
Security assurance is usually hard to grasp and sometimes we have seen there is the misconception how it can be achieved. One of the early milestones in understanding assurance came with the vulnerability analysis of Multics operating system:
The internal controls of current computers repeatedly have been shown insecure though numerous penetration exercises on such systems as […]. This insecurity is a fundamental weakness of contemporary operating systems and cannot be corrected by “patches”, “fix-ups”, or “add-ons” to those systems.
Rather, a fundamental re-implementation using an integrated hardware/software design which considers security as a fundamental requirement is necessary. In particular, steps must be taken to ensure the correctness of the security related portions of the operating system.
It is not sufficient to use a team of experts to “test” the security controls of a system. Such a “tiger team” can only show the existence of vulnerabilities but cannot prove their non-existence.
– Paul Karger in the Multics Vulnerability Analysis 1974
It identified the need for secure development to achieve assurance and a systematic approach to very assurance. But what does secure development means approach and what would such an evaluation method look like? Of course these methods have to be practical, which means cost effective. They should also be mutually supportive, meaning that they should have the same interpretation of what assurance and security is and actually contribute so that any developer assurance measures are recognized and used when doing the evaluations. It sounds obvious, but not always the case when looking into the methods used today.
Secure development
We need to be able to specify what we mean with security, so that we know what type of security the product should provided. Products usually only protect against certain threats, under certain conditions and when using the product in a certain way.
A system without requirements cannot fail; it merely presents surprises.
– Young, Boebert and Kain, “Proving a computer system secure”.
Scientific Honeyweller, 6(2):18–27, July 1985.
There are few standards for specifying security. The best known are the Protection Profiles and Security Targets in the CC. This has to do that CC cannot work without an ST since the CC standard by itself doesn’t state the specific requirements for a certain evaluation in the CC ISO/IEC 15408. It’s not easy do do, but there are also guides for this.
Even a big standard like ISO/IEC 15408 is not a substitute for thinking, and complex matters like IT security cannot be reduced to one sentence descriptions, no matter how hard you try.
– Mike Nash, “Guide for the production of Protection Profiles and Security Targets”
ISO/IEC TR 15446:2009(E)
So what are the secure development principles? There are design and architecture principles, such as those described as the Saltzer-Schroeder principles from 1975 that also grew directly out of the Multics experience. These and other principles are documented by Peter G. Neumann in his report “Principled Assuredly Trustworthy Composable Architectures” from 2004.
However, there is also a Technical Report from ISO ISO/IEC PDTS 19249 providing a good overview of Architectural Principle and Design Principles, stating:
Building a secure product, system, or application requires not only the implementation of functional requirements but also an architecture that allows for the effective enforcement of specific security properties the product, system, or application is supposed to enforce. The ability to withstand attacks the product, system, or application may be face in its intended operational environment is highly dependent on an architecture that prohibits those attacks or – if they cannot be prohibited – allows for detection of such attacks and/or limitation of the damage such an attack can cause.
– Helmut Kurth, Technical Report from ISO, ISO/IEC PDTS 19249
But there is more to secure development than just design principles. The importance of development processes, both for the initial development as well as for the maintenance is getting more attention. There are many different security standards that focuses on that such as GSMA NESAS and OpenSAMM (but not CC).
Note: There is a change in focus that largely has to do with the complexity of modern products, the frequent updates made to them (new features, bug fixes and security fixes) and of course the cost of evaluating the products.
Security evaluations
What about evaluations? What should an evaluation do to demonstrate that a product meets its security requirements? On a high level, there are two aspects that could (or should) be looked at. Assessment of the (1) secure development processes and (2) the assessment of the product itself, i.e. the result of these processes. The assessment of a product, may include a review of design (documentation), security (functional) testing and vulnerability analysis followed by penetration testing. It may give very high confidence for a certain version of the product, but may say very little about the future releases. On the other hand, the assessment of processes gives confidence that the developer has the security development processes under control, also for future products, but may say less about a specific product.
Then we have the aspect of security analysis vs compliance testing. Some standards, such as FIPS 140 used for crypto modules is focused on compliance testing, while other standard such as Common Criteria relies more on open-ended security analysis and search for potential vulnerabilities. All Common Criteria evaluations done in the US uses NIAP approved Protection Profiles that are more or less require to perform compliance testing only. Compliance testing is usually more objective than an open-ended analysis because of its nature. But this means that it is less flexible when it comes to product types and functionality. On the other hand, the open-ended analysis can also verify the absence of exploitable vulnerabilities, i.e. an active search for vulnerabilities and an analysis if they could be exploited. This approach provides more flexibility, but also requires more evidence from the vendor and more skills from the evaluator. There is just no no simple answers.
For every complex problem, there is a simple solution. And it’s always wrong.
– H.L. Mencken, US author and journalist
In summary, there is always a mixture between process and product assessment, and also between compliance and analysis. Unfortunately, many security assessment standards today have very little to do with secure development and almost none of them takes secure design principles into considerations. NESAS is here one exception.
atsec is an evaluation lab for FIPS-140 (with NIST), Common Criteria (in Germany, US, Sweden, Italy, Singapore). We are also NESAS auditors and a SCAS test lab. We are involved with national accreditation in Europe and have done audits according to OpenSAMM. We have seen benefits and drawbacks with all these different approaches, but we think that the assurance measures of developers should be better recognized and so assessments should promote development assurance.
References
- Paul Karger in the Multics Vulnerability Analysis 1974.
https://www.acsac.org/2002/papers/classic-multics-orig.pdf - J.H. Saltzer, M.D. Schroederer, “The Protection of Information in Computer Systems”, Proc. IEEE, vol. 63, no. 9, 1975, pp. 1278–1308. https://www.cs.virginia.edu/~evans/cs551/saltzer/
- Young, Boebert and Kain, “Proving a computer system secure”,
Scientific Honeyweller, 6(2):18–27, July 1985. - ISO/IEC TR 15446:2009(E)
“Guide for the production of Protection Profiles and Security Targets”. - Peter G. Neumann, “Principled Assuredly Trustworthy Composable Architectures”, 2004.
- ISO ISO/IEC PDTS 19249, “Catalogue of architectural and design principles for secure products, systems, and applications”, Technical Specification, 2017.
