Mission Possible: Building Critical Systems Using Third Party Software and SOUP

Whether you’re a C-level executive, an engineering director, or an engineer — supplying mission critical software can be stress-inducing. A poorly timed software crash or an exploited vulnerability can create a highly public failure that can hurt reputations, destroy expensive equipment – and in the worst case, harm people. That’s true across many functional safety domains, whether you work on robotic surgery machines, nuclear power plants, military helicopters, self-driving cars, high-speed rail engines, assisted care cobots, or communications satellites.

These events are the stuff of risk-and-compliance officer nightmares. But the danger of devastating surprises has been greatly compounded by the nature of today’s advanced technology. Let’s discuss why this risk has substantially grown and what you can do about it.

Why You Can’t Avoid SOUP

Companies building mission critical systems once specified and created entire systems from scratch to ensure their dependability and suitability for the task. While that’s one way to avoid using uncertified off-the-shelf or open-source software (in other words, Software of Unknown Provenance or “SOUP”), modern software has generally become too complex to make this approach practical.

Modern software development incorporates substantial amounts of third-party software, primarily to avoid “wheel reinvention,” to allow engineers to build a system within a reasonable timeframe. SOUP can also be used to take advantage of hard-won experience within components developed over many years, often making them large and extremely complicated. Components like cryptography services, advanced math libraries, or graphics frameworks are highly impractical to replicate. Another way SOUP enters a product is through easily overlooked places (such as device drivers, file systems, and runtime libraries), which are often unavoidable. Finally, a typical supply chain may involve many parties in creating software, greatly complicating the burden of tracking what SOUP enters a product.

How Can You Assess Software You Can’t See?

However expedient it may be to use SOUP, it introduces unknown risks and software assurance issues that can impact mission-critical systems. Unfortunately, it’s usually impractical or impossible to use software audits to review SOUP source code. This can result from commercial arrangements that limit source access, insufficient in-house domain expertise, or the sheer size of the code base. The result is that your product likely contains massive amounts of source code that nobody in your company has reviewed or even knows about.

In the past, incorporating SOUP from suppliers within your products involved a leap of faith. Now, however, intelligent tools have become available that scan a system’s compiled binaries. These tools, such as  BlackBerry^® Jarvis^®, can provide much the same level of comfort as having expert programmers examining source code, while also providing a wealth of information for risk mitigation and license-compliance purposes.

Let’s look at some SOUP-related issues and how they can be managed with automated code inspection in a fast, cost effective, and repeatable manner. In these use cases, we’ll be using the features of BlackBerry Jarvis as a specific example of what these tools can achieve.

Determining Software Composition

The first step to ensuring the security of your product’s software is to understand its composition. A software bill of materials (SBOM) identifies software packages, authors, and versions, providing software composition transparency. Having an awareness of your product’s subcomponents mitigates risks from cybersecurity vulnerabilities, IP disputes, and regulatory non-compliance. And that’s not all. The volume of files you can uncover within your systems can be significant. For example, within components such as vehicle domain controllers, you can find more than 120,000 files, over 80 unique file types, and 800 instances of open-source software. Even in “simpler” components such as a near field sensing unit, you’ll find around 2,000 files, 60 unique types, and 70 instances of open-source software. How can you present all this information in such a way that you can organize and analyze it?

An SBOM allows your organization to identify your product’s suppliers – often a client requirement for mission-critical systems. It can be used to double-check software license terms are being met, and can allow your organization to answer questions quickly and easily about software vulnerabilities as they’re discovered. It can also help procurement specialists as they negotiate quality metrics and pricing with your suppliers.

Complying with Software Licensing Agreements

There are a huge number of open source software licenses in common use, and they all specify distinct contractual terms for software: copyright terms, redistribution and sublicensing, source code accessibility, required mentions and text, and more. Additionally, commercial software always has licensing terms that extend beyond these – most notably pertaining to the number of copies in use and payment terms.

Ensuring that a company understands and complies with all the software licensing agreements contained within their product can be a Herculean task. It is made much simpler when contained software components and their associated licensing can be accessed in one place. 

Meeting Software Quality Metrics

Without access to source code, is it possible to see if software is of high quality and written using best practices? Yes – the binary provides clues to the quality of its contained software. To get an idea of how this is accomplished, imagine decompiling sources from the executable image and performing static code analysis on the result. Note that whether it literally occurs this way or not is immaterial; by looking at several attributes of the executable code, you can determine the level of care used in creating it.

For example, BlackBerry Jarvis examines the machine code within a system’s binaries to determine:

• Use of platforms and frameworks (QNX® Neutrino® RTOS, AutoSAR)
• Improper or dangerous API calls
• Race conditions
• Uninitialized variables
• Dereferenced null pointers

These factors contribute to understanding the quality and craftsmanship of each module within the executable image. With this knowledge, a company can confirm if their software is adhering to appropriate standards of code complexity and development practices.

Finding and Fixing Known Vulnerabilities

Does your software contain any cybersecurity vulnerabilities? Since new problems are constantly being discovered, it is impossible to be certain there are no vulnerabilities. However, understanding what known vulnerabilities may already be present is critical. This is true across your entire code base – not just within the SOUP hiding in your binaries.

Vulnerabilities can be divided into three distinct types. The first is explicit problems found in specific components, like an open source DHCP client with an exploitable weakness. The second type is insecure software patterns such as dangerous API calls or buffer overruns. The third is intentionally added software backdoors, usually cleverly hidden in plain sight. 

These three types of problems are cataloged in Mitre’s common vulnerabilities and exposures (CVE), a massive database of software vulnerabilities contributed to and maintained by the software industry. The best automated code inspection tools will find and flag possible CVE issues within your code.

Meeting Regulations

Regulations are one tool that governments use to control rampant cybersecurity problems and as such, regulations continue to multiply. A brief sampling of recent activity:

• The UN created WP29 R155 to harmonize vehicle cybersecurity regulations, forcing automakers to deeply understand their exposure and develop a consistent connected car cybersecurity approach.
• Nationally, the U.S. is addressing cybersecurity issues within both government and the private sector through regulations such as the Executive Order on Improving the Nation’s Cybersecurity.
• Data privacy and ownership regulations are being put in place throughout the world, notably in Europe, China, and California. They protect an individual’s data and may include mandatory security measures needed to handle personal data or grant consumers legal rights in case of data breaches.
• Various regulations are mandating quality for products that have safety implications, which indirectly enforce cybersecurity best practices, such as the FDA’s Quality System regulation for medical devices.
• Standards, while not technically regulations, are also being used to enforce software development processes, such as ISO/SAE 21434:2021, a recently released standard for cybersecurity within the automotive sector.

Ensuring that your company is meeting these and other regulations can be impossible if you don’t understand what’s in your software. Using an automated code inspection tool such as BlackBerry Jarvis can help your organization meet the requirements that these regulations embody.

Ensuring Build Integrity

Even when source is available, automated code analysis tools can be used to ensure the integrity of the build process. That means verifying that the development tool chain consists of compilers, linkers, and other build tools that are properly patched and up to date, so that they won’t inadvertently create vulnerable code. (These code analysis tools can also discover if your compiler has been hacked to silently insert backdoors into compiled binaries.)

Summary – Yes, SOUP for You

Unless your company writes every line of code, you have little control or insight into the reliability and hack resistance of your product. However, all non-trivial software today – even mission critical software – cannot be built without leveraging pre-existing components, aka SOUP. Even if you had all the software’s source code, it doesn’t make sense to manually search for vulnerabilities, compliance, or quality issues. Using months of an engineering team’s time versus minutes of computer analysis is an ROI no-brainer.

Tools like BlackBerry Jarvis can help companies understand their SBOM, vulnerability exposure, regulatory standing, licensing compliance, and more. If developing software under strict confidentiality requirements, you also want to ensure the software you use for the task isn’t cloud-based. That prevents confidential source going to a centralized server for analysis where it could be saved or stolen, and is a crucial requirement for the U.S. Department of Defense and other government agencies in the U.S. and abroad, and also applies to certain specific products across a diverse number of market sectors.

To learn more, view the on-demand webinar, Don’t Fear the Black Box: Ensuring Reliable and Secure Software in the Modern Age, presented by Brandon Bailey, Senior Cybersecurity Project Leader, Cyber Assessments and Research Department at The Aerospace Corporation, and myself.