Own Your CI with Self-Hosted GitHub Actions

There are a lot of reasons you might want to own your CI pipeline. You might have done the math and figured out it would cost you $115/day to continuously build your iOS apps on pay-per-minute runners. Or maybe you need to sign your company's Electron app using the absurd extended validation physical USB dongle. Or maybe you're sick of debugging CI failures with iterative logging improvements that take forever to validate and wish you could just remote desktop into the machine. Or maybe you just want to learn something new and put that old dusty rig in your closet to use. Whatever your reason, I love it! Let's see how it's done.

WARNING: Before going any further, ask yourself if you trust everyone that has the ability to create a pull request on your repository as they will be able to execute arbitrary code on build machines that reside in your network. If you have a public repo, the answer should automatically be no. Use cloud infrastructure instead.

GitHub Actions Primer

In this post, I'll walk through how to setup on an always-on, self-hosted GitHub Actions runner for both macOS and Windows from start to finish. We'll start with hardware purchasing and carry on all the way through to monitoring and debugging.

Before we dive into setup steps though, you'll want to familiarize yourself with GitHub Actions workflows. It's free for open source projects and simple to setup. The rest of this guide will assume you have a working knowledge of basic GitHub Actions concepts like workflows, runners, and continuous integration. I'll go grab a (sweet) tea while you familiarize yourself...done? Good, let's get to it.

Hardware

I wanted two small, dedicated machines for my buildbots, so I purchased new hardware specifically for the task, but you can use pretty much anything that's got a functioning CPU, a few GB of RAM, and an HDMI port (HDMI port will be important later). I've linked to Amazon for the non-obvious components so you can see exactly what I'm talking about but I encourage you to try to get these parts at a local brick-and-mortar electronics store.

Refurbished Mac Mini (Late 2014) - $292
Refurbished Lenovo ThinkCentre M93P - $210
4K HDMI Fake Display Emulator (2 pack) - $9
DisplayPort to HDMI Adapter* - $12
5-port Ethernet switch (optional) - $16
3 3-ft Ethernet cables (optional) - $9

* = only because I was an idiot and didn't read the description of the ThinkCentre carefully enough to notice it only had DisplayPort and no HDMI 🤦‍♂️ Do yourself a favor and use a rig with built-in HDMI

You will also need temporary access to...

Monitor or TV with HDMI
HDMI cable
USB keyboard
USB mouse

I say temporary because this is only for setup and installation. Once your runner is live it will be running with the headless displays so you can stick it in your closet. I'm also assuming you have access to the internet or you wouldn't be reading this :)

Once you have all your gear ready, move on to the setup your platform of choice.

Initial Setup

These steps will require your machine to be powered on, connected to a monitor/keyboard/mouse, and

Windows 10

Not interested? Skip to macOS

Factory Reset & OS Setup

I'm setting my machine up from scratch to make sure there's nothing left over from prior usage. If you're dealing with used hardware, I'd recommend doing a factory reset to install a fresh copy of Windows. During generic OS setup the only note you'll need to make is the user and password used. I picked a generic username like "builder" so I wouldn't get confused I'm on a personal machine. I also skipped any sort of Microsoft account creation for the same reasons (Microsoft really tries to force you to create one but you can bypass by using an "Offline Account", the "Limited Experience", and declining any personalization or Cortana features during setup).

System Configuration

In Control Panel > System and Security > System, set the name of the computer to something unique for your network. I used buildbot-win. This is the hostname you'll use later to remote desktop.
In Control Panel > System and Security > System > Allow remote access, allow remote connections to this computer. This is what enables remote desktop.
In Control Panel > System and Security > Power Options, set the plan to "High performance" and edit plan settings to never turn the display off or sleep "Never"/"Never".
Using Run... (Windows+R), Launch netplwiz, while selecting your "builder" user uncheck "Users must enter a user name and password...", and hit "Apply". This will automatically login the selected account on boot.

Install Dev Toolchain

First we'll want to setup a package manager to help us install our dependencies. There are a couple competing ones to choose from for Windows including a new one from Microsoft themselves, but we'll use chocolately for this example.

In PowerShell with administrative privileges (right-click on PowerShell to start one with admin rights), run these commands to install chocolatey and a few commands.

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
choco install -y git git-lfs zip hub nvm nano
Add-Content $profile '$env:Path += ";C:\Program Files\Git\usr\bin"' # add git tools to your default path

At this point we'll have access to some familiar bash tools thanks to the Git Bash for Windows we just installed. If like me you're more familiar with bash, pop open a bash window instead otherwise feel free to continue in PowerShell.

git lfs install
nvm install 10.21.0
nvm use 10.21.0

Configure the GitHub Actions Runner

Now that we've got our dependencies straightened out, this next bit is the magic sauce. We'll add a self-hosted runner to our account, download and install the GitHub Actions runner, and configure it as a service to run in the background. These steps are basically exactly what's displayed in the GitHub UI for your repo at the link https://github.com/--USER--/--REPO--/settings/actions/add-new-runner. Feel free to follow that advice instead for the most up-to-date instructions.

IMPORTANT: When prompted if you want to configure it as a service, respond "Y" and then use the user account you created during the OS setup instead of the default.

cd C:\
mkdir actions-runner; cd actions-runner
Invoke-WebRequest -Uri https://github.com/actions/runner/releases/download/v2.263.0/actions-runner-win-x64-2.263.0.zip -OutFile actions-runner-win-x64-2.263.0.zip
Add-Type -AssemblyName System.IO.Compression.FileSystem ; [System.IO.Compression.ZipFile]::ExtractToDirectory("$PWD/actions-runner-win-x64-2.263.0.zip", "$PWD")
Set-ExecutionPolicy RemoteSigned
./config.cmd --url https://github.com/XXXXX/XXXXX --token XXXX
Get-Service "actions.runner.*"

macOS

Factory Reset & OS Setup

System Configuration

Once you get through generic OS setup, you'll want to pop open the System Preferences to make the Mac behave a little more server-like. These settings all assume anyone with network access to the machine can be completely trusted. If you're working in some large office with carefully controlled access, you'll want to consult your IT department on how to not mess this up :)

In Sharing, set the "Computer Name" to something descriptive and unique to your network. I used buildbot-mac. This is roughly the hostname you'll use later to remote desktop into the machine.
In Sharing > Remote Login, allow access for all users. This is how you'll SSH into the machine later.
In Sharing > Remote Management, allow access for all users. This is how you'll remote desktop into the machine later.
In Security & Privacy > General, enable automatic login on startup and disable the password requirement. This will allow the Mac to boot directly into your builder user just by pressing the power button without needing to login manually if it ever shuts down.
In Energy Saver, set the screensaver, shut off display, and sleep settings to "Never". We'll be using a fake display that doesn't use any power anyway and you want your Mac mini awake to handle builds.*
In Spotlight, uncheck every box for suggestions and add your home folder to the excluded folders in "Privacy". Spotlight indexing is a monster CPU hog when it comes to build artifacts and node_modules folders. It's not like you'll ever be using Spotlight on this machine anyway.

Install Dev Toolchain

Next we'll want to setup all the dependencies for building and running our Github Actions workflows. Feel free to add additional packages here you might need in your projects. The beauty of this is that you're not limited to whatever you can install through package managers, add bespoke dependencies, configure the environment however you'd like, it's YOUR build machine!

Pop open the Terminal.app and throw these commands in there.

# Install homebrew for package management, this also installs XCode Command Line Tools
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

# Install utilities
brew update
brew install git git-lfs hub nvm
git lfs install

# Setup nvm to be loaded by default
mkdir -p ~/.nvm
cat >> ~/.profile <<'EOF'
export NVM_DIR="$HOME/.nvm"
export NVM_INSTALL_DIR="/usr/local/opt/nvm"
[ -s "$NVM_INSTALL_DIR/nvm.sh" ] && \. "$NVM_INSTALL_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_INSTALL_DIR/bash_completion" ] && \. "$NVM_INSTALL_DIR/bash_completion"  # This loads nvm bash_completion
EOF

# Setup node
nvm install v10

Configure the GitHub Actions Runner

cd ~
mkdir actions-runner && cd actions-runner
curl -O -L https://github.com/actions/runner/releases/download/v2.263.0/actions-runner-osx-x64-2.263.0.tar.gz
tar xzf ./actions-runner-osx-x64-2.263.0.tar.gz
./config.sh --url https://github.com/<your user>/<your repo> --token XXXX
./svc.sh install
./svc.sh start

Configure the Repository

Now that you've got the self-hosted actions runners going it's time to try out a CI job on them! Configure a new GitHub Actions workflow .github/workflows/ci.yml that runs-on your new self-hosted infrastructure.

name: Self-Hosted CI
on: [push]
jobs:
  windowsbuild:
    runs-on: [self-hosted, windows]
    steps:
      - run: npm run build:windows
  macbuild:
    runs-on: [self-hosted, macOS]
    steps:
      - run: npm run build

Push a commit with this sucker in the repo you configured for your runner and you should be off to the races.

Monitoring, Debugging, and Troubleshooting

Windows Remote Desktop

Microsoft offers a first-party remote desktop client for pretty much every platform imaginable. Check out their docs for how to setup the client on your platform.

Once you've got the client installed, setup a new connection with buildbot-win as the PC name and "Add User Account" to use the "builder" user you created during OS setup.

This worked for me immediately but later on after subsequent restarts I had some intermittent trouble with RDP connections being rejected that I solved with using ethernet instead of WiFi and a script to run on startup that pings a local network IP. For some reason even when the firewall was completely disabled and direct wired connection I couldn't remote desktop into the machine until it made at least one outbound connection to a local device, network sysadmins feel free hit me up to let me know what I'm doing wrong 🤷‍♂️

Mac Remote Desktop

Mac is super easy to remote desktop into from another Mac assuming you've already followed the system configuration instructions above. Open the Screen Sharing.app already installed on your client machine and enter buildbot-mac.local in the prompt. That's it! I never had any problems even after the machine went headless with this method.

GitHub Actions Logs

Once you have a few jobs running you might want to follow along with the build logs. The log structure is a little odd, but it's pretty simple to follow your jobs by navigating to actions-runner/_diag/pages and running tail -f * to follow all the log files.

Remember that the actions-runner folder is at C:\actions-runner on Windows and ~/actions-runner on Mac.

Inspecting Failures

GitHub Actions has another very nice feature in that it leaves the repo intact after a job completes. That means when something fails you can remote desktop into the machine and poke around the files, retry individual commands, edit the source files and log extra details, all without restarting the entire workflow! This has saved me hours of debugging time and is one of my favorite parts of self-hosting your runners.

GitHub keeps the repo in actions-runner/_work/<repo>/<repo>, so just navigate there once you're remoted in to poke around. Remember that the actions-runner folder is at C:\actions-runner on Windows and ~/actions-runner on Mac.

Serverifying

Once you've got the basic setup down, you can remote into the machine comfortably, and you've run a few GitHub actions jobs just fine, it's time to serverify everything. This phase will let you stick your build machines into a closet and never think about them again.

Unplug monitor/keyboard/mouse.
Plug in the 4K HDMI Fake Display Emulator. I've seen anecdotal evidence this might not be necessary but having a display kept my GPU build tasks working as intended, color reproduction similar to headful screenshots, and it was only a few dollars.
Use wired ethernet instead of WiFi (optional but highly, highly recommended).
Restart.
That's pretty much it...

Conclusions & ROI

All told I spent around ~$550 and about a full day getting it all up and running. Maintenance thusfar has been minimal with the biggest annoyance being manually rebooting after a power outage that outlasts my UPS.

You might be thinking "What a goof, lean startups don't waste their time building infrastructure, pay some IaaS to do it for you, rah-rah I'm swimming in VC money." If you've got millions to blow, hate accountability, and don't like fun ;) then more power to you, but by any business standard this was a pretty solid investment.

Given that I need to run about ~5 hours of builds a day on both Mac and Windows for Eris Ventures apps, I should breakeven in about ~3 weeks compared to paying GitHub per-minute (5 hours of Mac builds ~ $24, 5 hours of Windows builds ~$ 5, $550 /$ 29 per day ~= 19 days).

Add to that the fact that I can remote desktop into a machine after a failure to directly poke around the code and retry with edits as if it were local development (which I've already done twice in the first 24 hours!), and I've saved hours of development time already.