Software supply chain attacks over the holidays have once again left developers picking up the pieces in January. So how can the industry finally turn the tide?
By Joe Fay
After Christmas 2021 was ruined by the Log4J debacle, security pros were no doubt hoping for a quiet holiday period in 2022.
It didn't quite work out like that. On New Year's Eve the team behind the PyTorch machine learning framework revealed an unknown attacker had uploaded a malicious dependency package, that could exfiltrate sensitive data, to the Python Package Index repository.
Days later, CI/CD vendor CircleCI warned users to rotate secrets after an attacker used malware on an employee's laptop to execute session cookie theft, ultimately escalating access to "a subset of our production systems" and getting access to data. This situation left customers scrambling to locate and rotate "secrets" such as account credentials embedded in their projects.
Both incidents illustrate how the complexity of the modern software supply chain presents a security challenge. Developers might be using dozens, hundreds, even thousands of open source or third party components and services, any of which could contain vulnerabilities at best, malicious code at worst. SaaS providers are embedded in development pipelines and infrastructure more generally, meaning customers are potentially importing their suppliers' security failings.
Sonatype's eight annual State of the Software Supply Chain report, released in November, stated that 1.2 billion vulnerable dependencies are downloaded every month.
More worryingly, Sonatype said the number of “malicious, next-generation attacks” it detected had increased from 12,000 instances in 2021 to 88,000 in 2022. These attacks exploit automated build or dependency management systems, and range from dependency confusion and typo squatting attacks, to malicious code injections. Sonatype describes this figure – a 633 per cent increase - as “conservative”, relying as it does on just its own ecosystem.
Are things better in the non-open-source world? That’s hard to say. As Andy Chakraborty, former executive director for product security at Chase UK, pointed out, while Log4j ruined Christmas 2021, “before that there was SolarWinds”.
“It's a combination of both issues in the supply chain,” he said. “There's potential vulnerabilities versus companies not really staying on top of fixes that these companies might be putting out. The supply chain itself is always going to be vulnerable.”
And even when you’re paying for software or a service that’s not explicitly open source, that doesn’t mean you’re entirely protected.
“You're dealing with a lot of companies handling a lot of information on your behalf, where you don't have direct control over the environment, or how they're managing things, and so on.”
Chakraborty recalled software startups looking to sell into financial services firms often being surprised by the depth of information they are asked for around issues like auditing, encryption, or data access.
“We can mitigate the risk on our side by at least understanding how our vendor was managing on their end. I think the supply chain has definitely changed from just what software you're ingesting to a more general supplier management situation.”
Dropping the SBOM
Chakraborty said increased risk is the price of the shorter development cycles opensource promises. “Avoiding open source would mean development cycles would increase in length considerably, because you’d be creating functionality that somebody else has already created.”
David Wheeler, director of Open Source Supply Chain Security at the Open Source Security Foundation, added, “It's actually easier to answer questions about the open source world than the proprietary world…they won't talk.”
The risks to the software supply chain have certainly caught the attention of governments, with U.S., U.K. and E.U. authorities all issuing guidance on how organizations can protect their supply chains.
The U.S. government has demanded that Federal government suppliers produce a software bill of materials for their products to assure their integrity, and the White House is expected to sign off a new cybersecurity policy imminently. The E.U. recently brought new rules into force covering cyber reporting, while the E.U. Cyber Resilience Act is currently working its way through the system. The U.K. is expected to unveil further cyber resiliency proposals soon.
In the meantime, government agencies, such as the U.K.’s National Cyber Security Centre (NCSC), have been issuing guidelines on how to avoid supply chain attacks. Though Chakraborty suggested “Those sorts of programs are definitely more useful for smaller organizations that don't have their own threat intelligence teams and can't stay on top of it. If you look at larger organizations, they tend to have a lot of these functions internally.”
But the open-source world has been taking action of its own and Wheeler said 2023 will see a “maturation of existing efforts”.
This includes Supply chain Levels for Software Artifacts (SLSA), moving to 1.0. This is a Google-originated framework for “ensuring the integrity of software artifacts throughout the software supply chain” which is primarily focused on build processes.
Microsoft contributed its Secure Supply Chain Consumption Framework (S2C2F) to the OSSF last year. As the name suggests, this is focused on the consumption side of the pipeline. Wheeler explained, “I’m looking at software, I’m worried about its supply chain and my process bringing it in, how do I evaluate that.”
This is not quite as advanced as SLSA, he added. “I don’t know if we’ll come up with a 1.0, but we’ll certainly come up with a release.”
At the same time, the OpenSSF has a new working group bringing together repos, registries, and package managers to share experiences and compare notes, Wheeler said: “I think we're going to see more borrowing of security ideas from one ecosystem to another about how do we protect the packages that people are downloading and using? How do we harden up things?”
A very concrete advance will come from GitHub mandating multifactor authentication for all code contributors by the end of 2023. “That’s going to cause a sea change,” he added. There’s also pressure to roll out MFA in package ecosystems too.
This doesn’t solve all problems, Wheeler noted – the vast majority of attacks are typo squatting and dependency confusion after all. But arguably, hackers taking over the account of a respected developer or project is perceived as more pernicious. Once GitHub starts enforcing MFA, “We're going to significantly reduce the attacks on open-source software through subverted accounts.”
And the organizations using open-source software, particularly big commercial concerns such as those in the financial services world have their own part to play. Financial firms have been criticized in the past for being heavy users of open source but not necessarily great community members.
Who’s stepping in?
In November, James Holland, director of application security at Citi told the Kubernetes Community Days Conference UK that it would open source a project called Continuous Secure Software Ingestion, designed with Control Place, to "automate the ingestion and grooming of the libraries."
Chakraborty suggested the industry was at “an inflection point” and that “You will see more financial services organizations having things like bug bounty programs available.”
However, despite the best efforts of government and the OpenSSF, not every organization will be able or willing to dive into the source code of every component they are using.
So, commercial software composition analysis and other tools will remain an important part of the scene.
Chakraborty said “most places” rely on third party tooling to manage the risk. “I think that's key … understanding exactly what you have there, where it's deployed, and what potential risks you're facing”
Wheeler said that ultimately diving deep into every component is the better approach, but also constitutes a bigger change, “Some organizations are going to be basically saying, ‘I need to know what's in there. I need to know right now, I'm not willing to spend a lot of time to analyze, to change, please give me a tool that makes the pain go away.’”
This is all happening against a backdrop of upheaval in the broader tech industry, with major tech firms making large scale layoffs. Could this have a long-term effect on open-source security as organizations and developers alike review their commitments to projects?
Wheeler said it’s too early to make firm predictions. “You may see some retrenching in terms of prioritization, to go more focused on things that are more important to them than others.”
But while the largest technology firms might be casting off some engineers, other organizations have been suffering from a dearth of developers. “They’re not going to stay unemployed.” Wheeler added.
The biggest firms are the ones that have been taking the lead in securing development pipelines and running DevSecOps, and those practices will likely percolate down to other companies along with their former employees.
“I think we're going to get a dissemination of the approaches that the high-tech companies have been using, but other organizations haven't,” said Wheeler. “And we're going to see a very interesting education moment.”