The need for DevSecOps in the embedded world

Have you ever wondered why the embedded development industry is behind others when it comes to security? Or how web application developers improved their security maturity over the years? In this blog post, I want to talk about what happened in other parts of the development world in terms of security and how the embedded world can learn from it.

As I started gaining experience in the embedded security industry, I started seeing patterns emerge in the vulnerabilities I discovered. These vulnerabilities were in embedded devices, ranging from home routers to Telematics boxes within heavy vehicles. There were some common classes of vulnerabilities in the automotive, home connectivity and industrial control system devices. Some of these were used by me and Ramiro Pareja Veredas to attack thousands of vehicles remotely. We presented this at ESCAR Europe 2022.

Most of these were vulnerabilities are common knowledge in the regular application security community. For example, look at how the OWASP Top 10 has changed from 2013 to 2020 . They have identified and mitigated the same vulnerabilities almost a decade back. The question that comes to my mind every time I find a command injection or simple privilege escalation vulnerability in these devices is this: What causes this resurgence in vulnerabilities?

In my opinion, there’s not one answer to this question. It could be due to the IT industry fragmentation, developers not being educated about age old security issues or even the rush to production to minimize time to market. However, the effect of this is quite visible in the security space. Let’s take a look at some of the examples within the embedded world.

The Sisyphean struggle of security

Example 1: Vulnerabilities in Baseboard Management Controllers:

Researchers at Nozomi Networks Labs identified multiple vulnerabilities such as command injection and buffer overflows in the Baseboard Management Controllers of a major vendor. Using the vulnerabilities found, an attacker could get root access to the BMC without any authentication. Since BMCs can control things such as memory, processing and storage on a server, this allows an attacker to get full access to everything running on that server!

Example 2: Shellshock in embedded devices

The Shellshock vulnerability was initially found in 2014 on systems running Bash and were quickly patched in servers due to the severity. However, the embedded world was slower in uptake as many vendors did not realize the bash shell versions in their environments were also vulnerable.

Example 3: Remote vulnerabilities in Telematics boxes

A set of T-Boxes from Ovarro had vulnerabilities ranging from hardcoded credentials to path traversal. This could be used by an attacker to gain control of the T-Box remotely which in turn could compromise the car these T-Boxes are connected to.

One of the common themes here is command and well known vulnerabilities such as command injection, simple buffer overflows and path traversals which are getting more difficult to find in the web or mobile application world are still to be found in embedded systems. So the question remains, why has some industries (such as payment, content protection and regular web applications) found ways to mitigate certain classes of vulnerabilities but others have not?

Learning from our past mistakes

So now we can see a pattern of where these vulnerabilities pop up. Can we do something about this? Maybe we can learn from history, figure out what the other industries did and then try to implement the same for the embedded devices world.

For example, it’s quite common in the embedded industries to use outdated libraries and binaries that contain known vulnerabilities. The SBOM or Software Bill of Materials discussion connects directly to this issue. One of the most common feedback I get from developers is that it’s difficult to go for the latest bleeding edge release when working with embedded devices. However, that’s not a requirement for being reasonably secure and it might even be counter productive. A recent stable release is usually good enough for avoiding vulnerabilities that have been disclosed a few months ago. If the latest stable version does not have major security patches, it’s also understandable to update to a few versions before the latest stable. The issue is when firmware in the field has components that’s years or in some cases a decade out of date.

Vulnerabilities in the BMCs described earlier could have been mitigated with proper input sanitization but this requires the developers to have caught it in the first place. Maybe an SAST tool that identifies such issues and breaks the builds would have caught it earlier.

The issues in embedded devices due to ShellShock could have been identified with dependency checkers that would have alerted on known CVEs in the components used by the build system.

The vulnerabilities in Ovaro’s T-Boxes could have been spotted by using either an SAST or DAST integration in the build pipeline.

The web application security industry has learned things the hard way and some classes of vulnerabilities are getting rarer within that space. What does this look like? If you look at the OWASP Top 10 over the last few years, you would notice that the vulnerability classes have changed priorities over the years. This is primarily because of a few reasons:

• Developers learning from previous (usually public) mistakes.
• Using third party libraries that have security defaults that prevent quite a few vulnerabilities (or guardrails as my friend Martijn Bogaard puts it).
• Tooling in the pipeline that identifies security misconfigurations and vulnerabilities early in the development process.
• Additional security measures in the infrastructure such as web application firewalls and secrets management systems.

All of the lessons that the web application developers learned cannot be applied directly and as is to other industries but I believe that there are still take-aways that are useful.

Why do you need DevSecOps in embedded devices industry?

We briefly discussed learning from the past. What does this mean in the context of the DevSecOps movement?

Learning from the past does not just mean educating the developers. That’s a very important step but you can take it further by implementing security tooling, using continuous integration/development pipelines and ensuring that you have detection and monitoring systems in place. This is not something that I’ve seen implemented very well in the embedded development industry until now.

Concretely, what would this look like?

Tooling in the pipeline to automatically catch issues and mitigate issues

• SAST to identify vulnerabilities in source code.
• DAST to dynamically test built firmware.
• Secrets Scanning to detect cryptographic/ API keys and default passwords.
• Dependency checks for vulnerability management and for licensing.

Culture: Educate developers about security issues and make them part of the detection process

• Have regular security awareness and training sessions focusing on the tooling used.
• Send Developers to conferences.
• Schedule talks in house with security experts.

Automated detection and monitoring

• How good is your detection mechanisms?
• Can it identify anomalous traffic?
• Can it differentiate between a very chatty edge device and a hacker trying to exfiltrate data?

Metrics

• What’s the average API response time?
• What’s the size of an average response?

How would this look like?

Some parts of this process are easier than the others. For example, setting up some basic tooling in the pipeline to detect secrets and vulnerabilities could be a good first step. In the next series in the blog post we will see how setting up SAST, DAST and secrets detection in the pipeline would look like for an embedded development project.