In the last few weeks we've learned a lot about the Linux xz vulnerability. We won't rehash those details in this article (here’s a great one that does), rather we'll focus on how we minimize this type of deep supply chain attack in the future, and how we recover from attacks when they do happen. A few of the details that we've learned about the xz attack should be carried forward into this conversation.

Those are:

• The preparation was long term took years to integrate into the Linux repository,
• The attackers were very advanced in their methodology (even though their code was garbage),
• This was most likely a nation state level effort.

In this article we'll discuss what we should learn from the xz supply chain attack and how we can better prepare ourselves to defend or detect these actions in the future. We'll talk about how engineering risk approaches in the automotive and other industrial spaces during early 2000s is a great way to help us attack this type of problem. We need to be more informed going forward – we’ll talk about what data and methodologies can help us succeed, and how using engineering risk principles can at least reduce our exposure.

“Isn’t this just a nothing burger? I mean, we caught this, right?”

Let's address this right off the bat - even though xz never got widely deployed, and we never lost a single system for it, and there was no financial impact, this is not a security success story. The developer who discovered xz, as we mentioned in our first article, was because of due diligence on his part and a massive amount of luck. This was not a result of a security audit, a code scan, or any other security process that we have in place to stop malicious code. This was absolutely due to one man's good find and follow on effort. We also have to give the malicious code author a little bit of credit here too, as the code was so bad that its performance annoyed someone enough to take a second look at what was going on. Had the exploit been written just a little better, it would have passed this first smell test and easily gotten into our environment. The big “So What?” from this attack is how the threat actor got this code into a mainline Linux repository which would have been published and signed by Red Hat prior to deploying into our environments. Had it executed, the vulnerability would have led to compromised login systems for millions of Linux machines that protect our critical infrastructure and daily business. It purposefully targeted an OS – RedHat Linux – that we consider more secure, and hijacked operation of what we would consider one of the most secure features within Linux.

The uniqueness of this attack was that it was a complete subversion of the software supply chain. It’s the first of its kind we’ve publicly seen, and it’s not likely to be the only one. It took years to plant and execute, showing a great amount of dedication and skill. SolarWinds was more of a watering hole attack. Log4j was a widespread bad component in which a vulnerability was found and exploited. Fully compromising the development supply chain as xz did really is the worst type of supply chain attack.

“Ok Debbie Downer – so what do we do from here?”

Why should you keep reading? Haven’t we already lost? Absolutely not. The idea that the battle against cybersecurity threats is already lost is simply not true. The reality is, no magic bullet exists to stop this kind of attack; there's no single tool you can install, there's no product you can buy that can eliminate cyber risk on its own. We need a complete and fundamental rethinking of how we create software in this industry, and we should learn from other industries who have gone through similar pains. I believe it will require us to take a more measured and engineering-style approach to coding, to ensure that the products we produce have the same safety expectations as other products we use in our daily lives.

A great example of this approach is the safety culture that was borne out of Chernobyl, matured over the 1990s, and really came into force in the mid-2000s. Many industries, including power generation, automotive, heavy machinery, and ICS, redefined their daily business practice and emphasized safety as a part of every day and every conversation. We need that type of dedication to security in software development. By ingraining a security culture into our software development from the early stages of teaching in colleges, universities, and online code workshops, and making it a daily discussion in all of our development groups, scrum groups, and agile methodologies, we can see the same types of advancements in security that will be required to defeat these types of attacks. This approach, although it does not eliminate risks entirely, can significantly enhance the security of our products and substantially reduce our vulnerability.

I started out as a systems and software engineer in heavy industry, and every meeting started with a safety moment. It was geared towards physical safety back then, but the idea can be easily applied to software security, and we need that type of intervention in our coding ethos. In addition to the base change in software engineering principles, we also need the data and intelligence that will help us find and address risk early and continuously throughout our development processes and engineering. This was failure mode analysis in engineering; I believe this will be SBOMs (Software Bill of Materials) and continuous monitoring as part of Cyber Informed Engineering (CIE) for security.

How Do SBOMs move the needle?

SBOMs provide three critical functions in security. First they give you a component list of what makes up your software, second they show links and dependencies between components and 3rd they provide versioning information for all the components. In development, software teams can see when third party software changes, becomes obsolete, or has newly discovered vulnerabilities. This way you don’t end up releasing vulnerable code on day one, and you spend less time and money fixing security issues during pen-testing or other stage gates. Being able to see dependencies in your software also allow developers the insight to ask the pertinent CIE questions – Is there a real need for the services to be linked? Does this component really need access to this high-risk service? We’ve gotten good at limiting permissions when asked on our phones – SBOMs brings that practice into code development. Once released into production, continuous monitoring of SBOMs can provide alerting if your product has a vulnerable component, saving a great amount of time in discovery and response to third party vulns. This is how we recover faster – knowing definitively what’s in our software stops the hair on fire drills when inevitable vulnerabilities are discovered. We can be more confident, quicker, and more through in our response like in traditional engineering by having the appropriate documentation and understanding of our products.

Looking for more on SBOMs? This blog discusses Seven Risk Areas Addressed by SBOMs

What can you do now and how can Fortress help?

This is a large problem, and not an easy ship to turn, so let’s start with the basics.

• For both Developers and End Users: Admit we have a problem. Be the evangelist in your company - start talking about CIE and gaining internal support for the resources needed.
• If you’re on a development team – include third party code reviews and risk mapping as part of your process and show the financial befit of early discovery in time to remediate.
• As a user, get SBOM information on your software and embedded firmware products from your vendors, and work the component tracking into your asset profiles and SOC tracking. Run a tabletop that shows the difference in response time for an unknown component vs a known component.

As an OEM, make commitments for secure development. As a End User, require evidence of secure development before you sign your next contract. Fortress provides risk intelligence, including SBOMs, HBOMs, and Continuous Monitoring for teams both in development phases and in post-production monitoring. We also provide this service to end users/ purchasers as part of risk management – as most vendors don’t offer this level of risk tracking maturity, we can help you gain the same benefit to your company SOC and security response team by giving them the same level of insight.

To learn more about how Fortress can help, speak to an expert.