DEVOPS

13 considerations for choosing GitHub-hosted or self-hosted runners

Eran Medan
November 8, 2022

TL;DR

In order to ensure efficiency and quality, modern software development relies heavily on automation tools, such as code repositories and continuous integration/continuous deployment (CI/CD) pipelines.

Of all the different solutions available today, one of the most popular for code repositories is GitHub, which comes with a CI/CD solution known as GitHub Actions. In this blog we explore GitHub Actions, runners, and the merits and pitfalls of self-hosting GitHub Actions runners.

Arnica is now live in Beta!
Arnica is now live. Come see how Arnica is mitigating software supply chain risks.
Get Started Today

{{arnica-top-signup-banner="/template-pages/try-arnica-banner"}}

What is GitHub Actions and its runners?

GitHub Actions is a CI/CD platform that lets you run pipelines to build, test, lint, and deploy your code, among other commands. Runners are the apps that run a pipeline (also called a job). In other words, when you have a pipeline building your code, it is being built by a GitHub Actions runner. 

When a pipeline is triggered by an event, such as a push to a branch, a runner will be assigned the job and execute it. A GitHub Actions setup can have multiple runners so that jobs can be executed in parallel. 

Runners can be self-hosted or cloud-based (GitHub-Native)

Having self-hosted runners means that the runner app is hosted in your environment of choice—an on-premises server or hosted via your selected cloud provider. For example, you could be running your runner on AWS EC2 machines, Azure Virtual Machines, or GCP compute engines, directly in your company’s AWS/GCP account.

On the other hand, if you choose to use a GitHub-hosted runner, your workflow will run on the virtual machines offered by GitHub.

In the end, a tradeoff needs to be made between the pros and cons of each method. We’ll discuss the benefits of both in the following sections.

Benefits of self-hosted runners

Having self-hosted runners brings many benefits that generally revolve around customization and control for users. For starters, when using a GitHub-hosted solution, you will be limited to using GitHub’s VM for running CI jobs, but there are other advantages to consider as well. 

Ability to select your OS and machine

A big issue with running your pipeline on a GitHub-hosted machine is that you are locked into the OS and hardware selection that GitHub provides for its virtual machines. At the moment of writing this article, the available hardware can still be limiting for many users as it:

  • Is limited to only 2 core CPUs
  • Can only run on X86_64 CPU (no ARM compatibility)
  • Is limited to 7 GB of RAM on a RAM-hungry Windows and Linux server
  • Has other limitations (for RAM and SSD space), although these are less of a concern for most users

For the OS selection, Windows Server (2022 and 2019), Linux (Ubuntu 22.04 and 20.04), and macOS (12 and 11 ) are available. Linux only comes with the Ubuntu distribution, which can be limiting for those who want to use things like Pacman for packages or other tools not shipped with Ubuntu.

Finally, if you want to run a pipeline for a legacy application that can’t run on these latest OS, you might reconsider and go for a self-hosted type. For example, think of a legacy Windows application using an x86 32-bit CPU library that can hardly compile on these latest Windows servers, at least natively.

The complete list of OS and hardware for GitHub-hosted runners is available here.

While writing this post, Github announced larger action runner machines, up to 64-cores for Windows and Linux and up to 256 GB of RAM. While this still does not appear to address the OS issues, it certainly addresses the hardware limitation that causes slow build time and is a major advantage for GitHub-hosted runner users. 

Note that the larger action runner machines feature is still in beta and not yet available to the general public.

Faster CI/CD pipeline and build time

As seen in the previous section, CPU cores for GitHub-hosted runners are limited to two. For multithreaded compilation and testing, this can become a bottleneck. Many compilation tools use the advantage of multiple CPUs to speed things up. 

Having a shorter pipeline completion time possibly means less time wasted waiting by your team members. 

Better observability

Another benefit of having self-hosted runners is that you have better observability. By having Actions running on your self-hosted machine, you have better visibility into and control over logs and debug tooling. For example, you can log in via SSH to the machine and troubleshoot any issues (hardware, networking, etc.).

When using GitHub-hosted runners, you will have some visibility via logs in the Actions panel. This can help you understand why a pipeline has failed, is stuck, or is having any sort of trouble, but it is also more limiting.

Less downtime

GitHub-hosted runners are virtual machines hosted on Microsoft’s Azure cloud service. This means your Actions are dependent on the health of Azure’s servers, so when they have downtime, you might be unable to run your pipeline.

Depending on your self-hosting strategy and redundancy, this can mean less or more downtime for your Actions runners. 

Not hosted on internet-facing machines

Depending on your setup, your self-hosted runner servers could be only accessible via a VPN or even only local area for in-office usage. This can provide more security, as the machine is not—or at least less—exposed to hackers and malicious intruders and requests. This kind of setup can be seen as a way to implement the least privilege principle when it comes to accessing source code.

Downsides of self-hosted runners

Having self-hosted runners brings some disadvantages as well. These primarily involve more manual work and self-management, which can result in higher costs.

More setup and configuration 

When using a GitHub-hosted runner, all you need to do is point out what type of instance (OS) your pipeline needs to run on, and GitHub performs its magic behind the scenes.

But when you are using a self-hosted runner, you will need to spend time provisioning and setting up the servers (cloud-based or on-premises). Then you need to install the runner application and configure it. This includes desired behavior for autoscaling, monitoring, possibly establishing a proxy server, etc.

These things provide a high level of control and customization but also take time and expertise.

More management and maintenance

As with any self-managed server, your team will have to maintain and manage self-hosted runners. This includes but is not limited to:

  • Third-party package updates
  • OS updates
  • Runner application updates
  • Handling of hardware failures
More expensive

Running with your own server can be more costly, although this very much depends on your selected hardware and software.

Also, since self-hosted runners require more management and maintenance, the total cost can be higher since you need to pay engineers and IT and DevOps specialists to do these jobs.

Tips and recommendations

Choosing between self-hosting or using GitHub’s hosting service for your GitHub Actions runners is a matter of tradeoffs and depends on your specific use case. Below are some tips and recommendations that can help you decide which solution best fits your needs.

Large workload and specific OS

If your code base takes a long time to build and/or test and involves processes that are very multithreaded (e.g., you build your code with 12 CPU cores with make -j 12), or if you need a specific Linux distribution or even an old Windows version, you would be better off with a self-hosted solution. This is because you will otherwise be blocked by GitHub’s limited hardware and OS selection.

Sensitive workload and security

If your pipeline and code include sensitive data or workloads, having a self-hosted runner will let you have more control over the network and security. That can be a plus for your team or it can be a risk if you don’t know what you are doing.

Small or large team

If you are working in a small team of developers, with possibly no dedicated DevOps engineer, you should consider a GitHub-hosted solution, as it simplifies a lot of the work, giving your team more time to work on software development. 

Usage of spot instances for cloud runners

If you would prefer using a self-hosted cloud solution (e.g., installing the GitHub runner on an EC2 instance), you should consider using spot instances to lower your cost since your runner will not be busy running jobs most of the time.

Usage of EKS/ECS and AWS Fargate

Another way to simplify self-hosted cloud deployments is to use a managed container service such as ECS/EKS (optionally with Fargate, a serverless compute engine that runs containerized workloads on ECS/EKS). Using these containerized services can minimize maintenance work on the runners and may reduce costs.

Conclusion

Choosing between GitHub-hosted runners and self-hosted runners is mostly a matter of tradeoff between customization and simplicity. 

On the one hand, a self-hosted runner lets you (with minimal limitation) choose the OS, hardware, networking solution, redundancy setup, and observability solution. These can lead to faster pipeline time, better observability, and less downtime, amongst other benefits.

On the other hand, a GitHub-hosted pipeline will save you a lot of configuration and maintenance work by your team.

The decision must be made relative to the needs and requirements of a company and the team using the service, but general guidelines suggest that a self-hosted solution is better suited for teams with large or sensitive workloads and teams with specific/non-standard OS and hardware requirements.

It also suggests that a small team with no dedicated DevOps/IT engineer should probably go for a GitHub-hosted solution. 

Arnica is now live in Beta!
Arnica is now live. Come see how Arnica is mitigating software supply chain risks.
Get Started Today

{{arnica-bottom-signup-banner="/template-pages/try-arnica-banner"}}

What to consider before enforcing MFA on GitHub
What to consider before enforcing MFA on GitHub
October 19, 2022
What every developer should know about GitHub CODEOWNERS
What every developer should know about GitHub CODEOWNERS
July 23, 2022
Hacking HackerNews for fun and profit
Hacking HackerNews for fun and profit
January 4, 2022