Simplify debugging to reduce the complexity of embedded system development

The complexity associated with the development of embedded systems is increasing rapidly. For instance, it is estimated that the average complexity of software projects in the automotive industry has increased by 300% over the past decade.

Today, every piece of hardware is driven by software and most hardware is composed of multiple electronic boards running synchronized applications. Devices have more and more features, but adding features means increasing development and debugging complexity. A University of Cambridge report found that developers spend up to 50% of their programming time debugging.

Thankfully, there are practical ways to reduce the complexity of debugging embedded systems. Let’s take a look.

Earlier is better

Bugs will pop up during the entirety of a product’s lifetime: in development, testing and in the field. Resolving a bug later down the road can increase costs by as much as 15 times and lead to user frustration, in addition to creating challenges associated with updating embedded devices that are in production.

However, identifying bugs at the early stages of your product’s development will allow you to track them while prioritizing them by severity. This will allow you to debug before other dependencies and variables are introduced later in the lifecyle, which makes bugs easier and cheaper to resolve.

Manage versioning

To properly replicate a bug, you must be able to have a device in the exact same state it was when the bug first appeared. With embedded devices, there are three distinct variables to look at when issues crop up:

1. The software version. This is the version of each feature. This applies to the code you build as well as to potential dependencies, such as imported libraries.
2. The board version. Specifically, the design of the board. Board design changes constantly as components are added/removed or moved around.
3. Manufacturing. Which assembly line made the board and at what time? A unique serial number is usually used for every card.

When designing your code, anything that references one of these three elements must be made a variable. To manage this versioning granularity, you need a registry. Open source tool PlatformIO is a great way to do this.

Operationalize replicability

Once you can fully define a given device’s state, you must be able to replicate that state on a local device so that you can debug. To do that, you’ll need a script that will pull the required version, compile the right binaries and push them to your product.

Here is a code snippet containing a script I use for one of my projects.

Additionally, when you have a bug, you must find the simplest configuration that lets you reproduce it easily so you can limit the area of code you need to inspect. By managing your features independently, you can easily enable and disable each of them on your code.

The best way to achieve this is to compile every feature as a library, where each feature has an init and loop function — Arduino-style — that can be called from the main file.

Trace everything

Debugging will only be efficient if you have the right information. Traces, which log the low-level event of a program’s execution, are a must here. Both hardware and software features must generate traces for everything they do. Tools like Memfault (open source) or Freedom Robotics can help you get there.

While your device should constantly be generating traces, only when an issue occurs should traces be automatically saved and sent back to you so that you can analyze them. To be able to properly capture as many anomalies as possible, you must anticipate their types.

Anomalies come in different shapes and sizes with embedded devices — it might be a software issue, but it could also be hardware issues such as overheating, water damage or broken components. The sky’s the limit with embedded systems. For example, one of our customers is building articulated arm robots that perform sensitive maintenance operations in nuclear facilities, exposing the hardware to high levels of radiation, which can impact the hardware and software in random ways.

Ensure timeline consistency

Another key component of traces is timing. Because embedded devices are often made of multiple cards that have multiple inputs, such as sensors and user input, and outputs, such as engine and screen, it’s key to track timing so you can reconstruct a timeline of what happened.

The tracking needs to happen at the millisecond-level (sometimes even at the nanosecond-level), and each timing needs to be precisely aligned with other components. Timelines across components can drift because a device can have different microcontroller units (MCUs) that are started at different times, cadenced at different frequencies and have different temperatures.

There are two ways to ensure traces can be timed correctly.

One way is to synchronize time across different cards — to get a coherent timestamp of data across different nodes — by sending specific update messages. Depending on how much the timeline is drifting, you will need to adjust the frequency of those messages. But since this synchronization message needs a predictable latency to guarantee the accuracy of the date, devices generally need to stop every operation in the network to ensure that the latency will always be the same. This can be problematic for some products.

The new way of doing it, pioneered by a paper from the University of California, Berkeley, is to embrace latency and compute timelines based on it. Latency is a sum of delays, so by measuring delays across the product, latency can be calculated and a timeline can be reconstructed.

• sourceLatency = communicationDate – dataGenerationDate
• targetLatency = dataConsumingDate – communicationDate
• networkLatency = propagationTime + IRQraise
• totalLatency = sourceLatency + targetLatency + networkLatency

The advantage of this method is that it constantly produces consistent results without having to worry about the frequency of synchronization messages and without having to stop every other operation in the network. I’ve written a detailed paper on how to implement this methodology for embedded purposes.

Look for bug trends

Finally, with embedded projects, issues can often come from a specific part or assembly of the implementation. That is why keeping track of your bug history is important, as it allows you to identify trends of problematic areas or a set of devices that have a specific set of versions as quickly as possible.

Tools like Luos (open source) or Freedom Robotics can help you to accurately monitor the events that occur in your embedded system so you can resolve them more easily and quickly.