Dan Lorenc
SolarWinds and Log4j have made software supply chain security issues a topic of intense interest and scrutiny for businesses and governments alike.
SolarWinds was a terrifying example of what can go wrong with the integrity of software build systems: Russian intelligence services hijacked the software build system for SolarWinds software, surreptitiously adding a backdoor to a piece of software and hitching a ride into the computer networks of thousands of customers. Log4J epitomizes the garbage-in, garbage-out problem of open source software: If you’re grabbing no-warranties code from the internet, there are going to be bugs, and some of these bugs will be exploitable.
What’s less talked about, though, is that these attacks represent only a fraction of the different types of software supply chain compromises that are possible.
Let’s take a look at some of the lesser-known, but no less serious, types of software supply chain attacks.
Unauthorized commits
This class of attacks describes an unauthorized user compromising a developer laptop or a source code management system (e.g., GitHub) and then pushing code.
A particularly famous example occurred when an attacker compromised the server hosting the PHP programming language and inserted malicious code into the programming language itself. Although discovered quickly, the code, if not corrected, would have enabled widespread unauthorized access across large swaths of the internet.
Fortunately, recently developed tools like Sigstore and gitsign reduce the probability of this type of attack and the damage if such an attack does occur.
Publishing server compromise
Recently an attacker, potentially the Chinese intelligence services, hacked the servers that distribute the Chinese messaging app MiMi, replacing the normal chat app with a malicious version. The malware allowed the attackers to monitor and control the chat software remotely.
This attack stems from the fact that the software industry has failed to treat critical points in the software supply chain (like publishing servers or build systems) with the same care as production environments and network perimeters.
Open source package repository attacks
From the Python Package Index, which houses Python packages, to npm, the world’s software now literally depends on vast stores of software packages, the open source software programmer’s equivalent of the Apple App Store.
Sadly, there has been an explosion of security incidents related to these repositories, putting at risk the security of the world’s digital infrastructure. The particulars of repository attacks vary, but recent examples include tricking developers into downloading malware by using “typosquatting” techniques and good maintainers “gone bad,” who abuse the trust their users have placed in them. The vastness of this category of attacks should cause those who blindly trust open source software to reconsider their attitude.
Credentials theft hijacking
There’s a meme that your security is as secure as the least secure laptop owned by any open source maintainer that you depend on.
Anytime an open source software developer’s credentials are stolen, there’s an opportunity for an attacker to pretend to be them and push malicious code or publish malicious packages. This is not hypothetical: in one recent attack, someone phished Python open source software maintainers, stealing the credentials of at least a couple, and published malicious code.
OK, so what am I supposed to do?
Locking down production environments and network perimeters has become second nature for most companies.
Every developer now knows to never pick up a random thumb drive off the ground and plug it into a computer — this has become security common sense. But for decades, developers have been downloading open source packages with no way to verify that what they are installing is safe.
Bad actors have started to capitalize on this attack vector, because it is the new low hanging fruit. They’ve realized that they can gain access through these holes, and once inside, can pivot to all the other systems that have dependencies on whatever insecure artifact they gained entry through.
Here’s where you need to start to develop a strategy for making your software supply chain secure by default.
First, snap out of the scanner and SCA pipe dream
The security vendor landscape is selling a pipe dream that “scanners” and “software composition analysis” wares can detect all of the critical vulnerabilities at the software artifact level. They don’t. Scanners tend to be very reliant on the package managers supported by popular Linux distributions, but they cannot scan the integrity of all the software downloaded by your dev team outside of those distributions.
The problem with scanners is made clear by the recent push for software vendors to provide SBOMs (software bills of materials — a “list of ingredients” of what’s inside your software) alongside their software. SBOMs would require vendors to provide a detailed inventory of the dependencies included in a software package so end users can manage their own vulnerability posture.
Unfortunately, many of the early approaches to generating SBOMs operate in a very similar manner to SCA tools. Scanners are not going to solve this problem for you holistically, and neither are SBOMs alone, so don’t think either is a silver bullet. They are both tools and not singular solutions for complete software supply chain security.
Think zero trust, but for software artifacts
Zero trust tends to refer to people inside and outside of a firewall. In software supply chains, we’re dealing with code, build systems or artifacts that are built. These are binaries, container images, Python, Javascript and Java packages.
Right now, most organizations place controls at the perimeter of their source code or artifact repositories, implicitly trusting everything that makes its way in. This is the opposite of zero trust! Zero-trust supply chains trust artifacts based on how they were built, by whom and what code is inside them.
So if you want to get rid of firewalls, allow lists and trusted registries, you have to be able to trace artifacts back to where they came from. Artifacts don’t have names, they’re not people and they can’t tell you who they are, so you must be able to create policies based on where these artifacts come from, how they were built, who reviewed the source code, etc.
Ultimately, your goal is a zero trust software supply chain — meaning, every artifact can be verifiably traced back to the source code and hardware it was built on.
Lock down your build systems
You must start treating your build environments like production environments. You spend all this time protecting your run times, then deploy to them using amateur tooling — unpatched Jenkins boxes sitting under someone’s desk that no one is looking after. It’s like one of those movies where they build a big expensive vault and then someone sneaks in through the food cart.
As an industry, we’ve gotten so good at protecting our infrastructure that attackers are looking at the next easiest way, which happens to be through the supply chain — the doorways we’ve left open. Instead of attacking one company, they can find a vendor, an open source dependency or a library, and then pivot to all of their customers.
Your build system should be at least as secure as your production environment. SLSA, a best practice framework for software supply chain security, will get you started with critical steps for securing your build process.
Sign your base images and binary code
Your next step is embracing signing and verifying your code and artifacts. This is a cornerstone in the guidance that the National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA) and Office of the Director of National Intelligence (ODNI) recently emphasized in their “Securing the Software Supply Chain for Developers” report. Open source projects like Sigstore are bringing chain-of-custody records for software artifacts into build systems and programming language registries.
Now you can start thinking about “provenance”
The term “provenance,” in physical and software supply chains, represents where something has come from. Now that you’re signing your base images and binary code, you can base your security policies on provenance.
Instead of basing policy on where an image is or where an artifact is, you want to base your policy on where it came from and how it was produced. You want to capture this at every step of the build. Builds are recursive and have multiple steps, and this applies to dependencies as well, so you have to do it recursively.
This applies to every single step and piece — the source code, the build process, the publication process, everything. When you sum all this up, you get a system where you can trust what the artifact is and where it came to be, not just where it is in your environment.
Editor’s note: After publishing this article, we received the following statement on behalf of SolarWinds:
“While it’s accurate that thousands downloaded the malware, SolarWinds announced ‘the actual number of customers who were hacked through SUNBURST to be fewer than 100.’ This number is consistent with estimates previously released by the White House.”
Comment