SOFTWARE SUPPLY CHAIN

Hardening software development environments 101

Eran Medan
September 21, 2022

TL;DR

The development ecosystem has changed dramatically over the last 5 years and has given way to the concept of software supply chain security. In this piece, we cover the different components of software supply chain security and how a zero trust approach can help you secure your development environments more effectively.

Arnica is now live in Beta!
Arnica is now live. Come see how Arnica is mitigating software supply chain risks.
Get Started Today

{{arnica-top-signup-banner="/template-pages/try-arnica-banner"}}

Before the pandemic, most companies used perimeter security, locking down all on-premises machines and funneling all inbound traffic through firewalls. But today, when remote work is still widespread, they have to control access from the outside by employees who might bring their own device (BYOD).

Developers and their development environments are part of the software supply chain, so if their accounts get compromised, attackers get control over parts of this chain. Nowadays, many developers are working on these environments from their homes and can end up as entry points for malicious code or let attackers steal credentials to production services. Therefore, while “hardening” used to mean securing a developer’s local computer, it now also means bolstering the security of the tools they need to do their work. These include source code management (SCM) tools, binary artifacts, and build/CI/CD pipelines.

This article will explain what a zero trust approach is and how it helps to secure development environments in times of remote work.

Zero Trust to the Rescue

Zero trust is a security model with the default assumption that everyone can be an attacker. It tries to fight the risk of account takeover (for example, stolen credentials) and insider threat (for example, a disgruntled employee). A critical element of zero trust is enforcing explicit permissions over implicit ones. Instead of giving your internal users the benefit of the doubt, you verify them any time they try to access a resource. Multi-factor authentication is a part of this.

This concept works very well for workstations that connect to corporate systems, but for SCM, it is a bit more restricted. Slowing a developer down too much with overburdened security requirements can impact the software delivery process.

Principle of Least Privilege

Another vital part of zero trust is the principle of least privilege. After you verify that your user is who they are, you have to permit them to execute their task. These permissions should be as minimal as possible to not keep a developer from doing their work. But they also must be strict enough so that they can’t make code changes that aren’t part of their present task.

Following this principle, the focus is on resource protection—never grant permissions implicitly, and continuously check these permissions. While this might sound tedious, it shouldn’t mean every user has to wait long to get their permissions. Everyone gets their permissions as quickly as possible, but they have to request them explicitly; and if they don’t use them for long, they will be revoked automatically.

The goal is to make people think deliberately about their permissions while not slowing them down. After all, negatively impacting development velocity is a recipe for failure.

What Does Zero Trust Mean for SCM?

Developers have to authenticate and should only get the permissions they need to do their job. Fine-grained permissions are required to secure source code, such as GitHub repository permissions. If a developer needs to simply contribute code, they don’t need admin access to the whole repository; thus, write permissions should suffice. Also, permissions should be revoked if unused after an extended period of time.

Most application and production security tools only look at the cloud and not the specifics of the services running on that cloud. This means they treat Git like a web server and ignore repository and branch security, which can lead to excessive unchecked permissions for too long. This in turn increases the risk of, for example, pull requests getting approved by people who aren’t at the company anymore.

Modern SCMs offer a way to grant repository permissions to a user; some SCMs even offer branch protections. GitHub, for example, provides the CODEOWNERS file to manage who can approve pull requests on protected branches.

Zero trust also entails anomaly detection, that is, checking if users behave in a way they didn’t before. The reasons for this behavior change can come from a malicious account takeover, spoofed Git credentials (git config --user.email "fake@example.com"), or simply a disgruntled employee. If someone doesn’t act as they usually do, it’s worth checking out the cause.

Another critical part of development environments is CI/CD pipelines. They’re responsible for checking out the code, building the binaries, testing, and deployment. This means they have access to the code repositories and the production environment. So don’t just run every CI/CD pipeline process with root privileges, but ensure processes run by these pipelines also have minimal permissions. For those that need to run on well-known TCP ports (under 1024), minimal permissions can still mean root access is required; for most other processes, lower permissions are enough.

Anomaly detection should check if CI scripts have been altered unknowingly.

Software supply chain security

The software supply chain consists of the steps taken from planning to delivery. This includes development environments because they’re responsible for managing code, building, testing, etc. So, secure development environments sit at the intersection of these areas. Let’s look at some of the different methods to help secure this chain.

Endpoint security entails the practice of securing the actual devices used to access your network. This goes from basic antivirus and malware detection software and disk encryption to full-fledged endpoint security platforms—software suites that include everything needed to secure an end-user device and manage its security from a central location. Every computer a developer uses to access code for development purposes should at least implement minimal endpoint security.  Additionally, git clients need to be properly configured on the endpoints. For example, if the developer uses SSH/GPG keys to authenticate to the SCM, the use of these keys should be protected by a password or biometric mechanism. Otherwise, a malware on the endpoint can use git without limitations.

The Software Bill of Materials (SBOM) is a list of software tools, libraries, and frameworks that are all part of the software to deliver. It gives you an overview of everything used to get an application to production. But you can also use it to verify software components at different steps along the supply chain, ensuring that nobody tampered with any items on the list.

Access management should be a priority; find out who has permissions, what they include, and how long they keep them. Perform regular user entitlement reviews so your information doesn’t get stale. The code developers write code that sooner or later ends up in production, and this can be a liability. If a developer who has been fired can still write code that winds up in production, because those responsible forgot to remove the former employee’s permissions, you could get into trouble (we have seen it with GitHub organizations where bring-your-own-identity is in use and SAML is not properly enforced).

Anomaly detection across the supply chain will notify you about unexpected changes. Did someone push code without a Pull Request review? Did they access resources they’d never accessed before? Did they suddenly delete resources without an apparent reason? All of this can imply an attack is happening, either from stolen credentials or an employee gone rogue.

And last but not least, misconfigurations. You must configure all the software and services you use correctly; sometimes, it might not be obvious what a bad configuration is, especially if it’s a default. Scanning your configuration files can prevent issues here.

Summary

Securing development environments in today’s software industry has changed over the last few years. Talent is scattered worldwide, and engineers have become quite comfortable working from home. This means perimeter security won’t cut it anymore.

Zero trust is the new paradigm, which means consistently verifying users before granting access and following the principle of least privilege.

The number of software supply chain attacks has been rising in recent years, and development environments touch many parts of these chains. Developers’ code will end up in production, so it’s crucial to verify their permissions continuously. Only give them access to the resources they need for as long as they need them. Get as fine-grained as practically possible, and focus your security strategy on branch permissions instead of entire repositories.

Keep in mind that obtaining permissions has to be simple and quick. It’s not about slowing developers down—it’s about making permission grants explicit so nothing slips under your radar.

With Arnica’s behavior-based software supply chain security, you get easy access management for your development environments with all the features discussed in this post. Arnica checks for unused permissions or anomalies in developer behavior. It additionally removes permissions that aren’t in use and notifies you if behavior changes. All this while making it quick and easy for developers to get their permissions back when needed, for example, using Arnica’s Slack bot.

Arnica is now live in Beta!
Arnica is now live. Come see how Arnica is mitigating software supply chain risks.
Get Started Today

{{arnica-bottom-signup-banner="/template-pages/try-arnica-banner"}}

Four takeaways from the NSA's software supply chain security recommendations
Four takeaways from the NSA's software supply chain security recommendations
September 10, 2022
How to protect stale source code repositories on GitHub
How to protect stale source code repositories on GitHub
July 18, 2022
Hacking Upstream: Finding a 0-Day in an OpenSSH Key Parser Library
Hacking Upstream: Finding a 0-Day in an OpenSSH Key Parser Library
July 6, 2022