Slash CI/CD Bills (Part 2): Using Hetzner Cloud GitHub Runners for Your Repository

Welcome to our second article on using Hetzner Cloud GitHub runners to control costs for Altinity Stable Builds, which are Altinity’s builds of ClickHouse. To control our CI/CD bills, we moved to using the TestFlows Hetzner GitHub Runners project to benefit from autoscaling GitHub Actions runners using the thrifty Hetzner Cloud for our ClickHouse CI/CD pipeline. It turned out to be great, and in Part 1, Using Hetzner Cloud GitHub Runners for ClickHouse Builds, we compared the costs for a basic Linux, 2 vCPU runner instance:

Name	Price per hour
GitHub instances	$0.48
Hetzner Cloud	$0.022
AWS on-demand hourly rate	$0.0672
AWS spot instance	$0.0307

We showed that our massive pipeline, which needs four different server types, 156 jobs, and almost 5 days and 10 hours of compute time, costs us from €17.56 in the worst case to €9.77 in the best case for each pipeline execution. That gives us an average per-hour cost of €0.1353 to €0.0753.

In this second part, I will show you how to set up and try the TestFlows Hetzner GitHub Runners project for your own GitHub repository. All you need is a GitHub and Hetzner Cloud account. You can quickly sign up if you still need a Hetzner account. We’ll go through a simple installation, prepare a GitHub repository, set up the Hetzner Cloud project, and deploy the Hetzner Cloud GitHub runners service on a cloud instance. Once done, we’ll see our first jobs executing on our self-hosted runners and quickly learn to switch between different server types, including ARM64 instances. We’ll also peek at day-two tasks like monitoring logs and estimating costs. Log in to your GitHub account and follow along!

Features and limitations

Before we install github-hetzner-runners, reviewing the main features and limitations is useful. Here are some highlights:

on-demand autoscaling runners
server recycling to minimize costs
simple configuration, no Webhooks, no need for AWS lambdas, and no need to set up any GitHub application
custom runner server types, images, and locations using job labels
self-contained program that you can use to deploy, redeploy, and manage the service on a cloud instance
x64 (x86) and ARM64 (arm) runners
efficient GitHub API usage using HTTP caching and conditional requests
cost estimator

The project has a good set of documentation you can reference by browsing its wiki. The limitations include the fact that group runners are not supported, so you need to set up a dedicated service for each repository. Another limitation is that a unique Hetzner Cloud project must be used for each repository, but this limitation has a benefit: dedicated projects allow for easy tracking of runner costs per repository.

On the implementation side, the service is stateless. It uses a simple polling technique to detect when a job is queued up and waiting for a runner. This avoids problems with losing events and jobs never getting a runner assigned. It consumes GitHub API calls but optimizes their usage by employing HTTP caching and conditional requests. The stateless nature of the service means that it can be restarted and recovered even if interrupted. If you want to see the details, head to GitHub and browse the code.

Enough with the introduction. Let’s install and play around with it so you can see what it is all about.

Installation

The installation process is pretty simple, given that the github-hetzner-runners is just a Python program that you can install using the pip3 command. I’ll be using an Ubuntu 22.04 machine.

pip3 install testflows.github.hetzner.runners==1.7.240319.1151947

Check that the installation was successful and that the github-hetzner-runners executable is accessible.

github-hetzner-runners -v

The executable gets installed into ~/.local/bin/github-hetzner-runners, so if you get an error that the github-hetzner-runners command is not found, then check that the ~/.local/bin is in your $PATH.

After the installation, we only need to set three environment variables or specify the equivalent options on the command line to run the service on the local machine. Running locally is suitable for testing and debugging. We recommend deploying it as a cloud service in production.

export GITHUB_TOKEN=ghp_...
export GITHUB_REPOSITORY=...
export HETZNER_TOKEN=GJzdc...
github-hetzner-runners

Instead of using environment variables, you could also use command-line options as follows:

github-hetzner-runners --github-token <GITHUB_TOKEN> --github-repository <GITHUB_REPOSITORY> --hetzner-token <HETZNER_TOKEN>

In theory, you are all set if your GitHub token has the correct permissions for the specified repository and you have a valid Hetzner token. The service will be looking for any jobs that have a self-hosted label, and as soon as it sees one, it will create a default instance, CX11, with the default image, ubuntu-22.04.

However, I will walk you through how you can set everything up from scratch and get the values for the GITHUB_TOKEN and GITHUB_REPOSITORY by creating a GitHub repository and the corresponding token, then for the HETZNER_TOKEN, we’ll set up a Hetzner Cloud project for which we’ll generate an API token.

Preparing GitHub repository

You can either create or pick one of your existing repositories. I will make a new one called demo-testflows-github-hetzner-runners.

Given that my username on GitHub is vzakaznikov, the full repository name for me will be:

export GITHUB_REPOSITORY=vzakaznikov/demo-testflows-github-hetzner-runners

However, more is needed to just having a repository; we also need to add a workflow with at least one job. I will create .github/workflows/demo.yml using an example from the Quickstart for GitHub Actions guide.

If you already have a workflow. Just update your job’s runs-on attribute to include a self-hosted label. I’ve also specified a custom server type using the type-cpx21 label. In general, you can specify custom server types, server locations, and server images using labels.

name: GitHub Actions Demo
run-name: ${{ github.actor }} is testing out GitHub Actions 🚀
on: [push]
jobs:
  Explore-GitHub-Actions:
    runs-on: [self-hosted, type-cpx21]
    steps:
      - run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."
      - run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!"
      - run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
      - name: Check out repository code
        uses: actions/checkout@v3
      - run: echo "💡 The ${{ github.repository }} repository has been cloned to the runner."
      - run: echo "🖥️ The workflow is now ready to test your code on the runner."
      - name: List files in the repository
        run: |
          ls ${{ github.workspace }}
      - run: echo "🍏 This job's status is ${{ job.status }}."

Finally, you need to create a GitHub API token with the workflow privileges, save it, and set it as an environment variable.

export GITHUB_TOKEN=ghp_V7...

Here is how I did it.

We’re done on the GitHub side. Commit your demo.yml, and in the Actions, you should see a queued-up job. It will have no runner as we haven’t yet started the github-hetzner-runners service.

Setting up the Hetzner Cloud project

We must create a new Hetzner Cloud project and an API token with Read & Write permissions. Here are the steps.

Once you get the token, we will set the final HETZNER_TOKEN environment variable.

export HETZNER_TOKEN=5Up04IHu...

Starting the service

We can start the service with the GitHub repository and the Hetzner project prepared. For testing and debugging, you can execute github-hetzner-runners directly after setting GITHUB_TOKEN, GITHUB_REPOSITORY, and HETZNER_TOKEN environment variables.

export GITHUB_TOKEN=ghp_...
export GITHUB_REPOSITORY=...
export HETZNER_TOKEN=GJzdc...
github-hetzner-runners

You should see the output messages similar to the ones below.

16:51:43 🍀 Enabling HTTP cache at /tmp/tmpfoct_b59/http_cache
16:51:43 🍀 Logging in to Hetzner Cloud
16:51:43 🍀 Logging in to GitHub
16:51:43 🍀 Getting repository vzakaznikov/demo-testflows-github-hetzner-runners
16:51:45 🍀 Checking if default image exists
16:51:46 🍀 Checking if default location exists
16:51:46 🍀 Checking if default server type exists
...

If you want to stop the local service, press Ctrl-C. It will interrupt the program. For a clean shutdown, wait until the program exits.

Alternatively, we could also deploy the service on the remote server using the github-hetzner-runners cloud deploy command as described in the cloud service section of the documentation. Again we first need to make sure the GITHUB_TOKEN, GITHUB_REPOSITORY, and HETZNER_TOKEN environment variables are set, and then instead of running the github-hetzner-runners command, execute github-hetzher-runners cloud deploy instead.

export GITHUB_TOKEN=ghp_...
export GITHUB_REPOSITORY=...
export HETZNER_TOKEN=GJzdc...
github-hetzner-runners cloud deploy

The output will be similar to the following:

$ github-hetzner-runners cloud deploy
17:03:56 🍀 Logging in to Hetzner Cloud
17:03:56 🍀 Checking if SSH key exists
17:03:57 🍀 Checking if default image exists
17:03:57 🍀 Checking if default location exists
...
17:05:13    > 21:05:03 🍀 Installing /etc/systemd/system/github-hetzner-runners.service
17:05:13    > 21:05:03 🍀 Reloading systemd
17:05:13    > 21:05:04 🍀 Enabling service
17:05:13    > Created symlink /etc/systemd/system/multi-user.target.wants/github-hetzner-runners.service → /etc/systemd/system/github-hetzner-runners.service.
17:05:13    > 21:05:04 🍀 Starting service

The cloud deploy will create a new server in your Hetzner project named github-hetzner-runners where the github-hetzner-runners service will be installed and started.

If you want to delete the cloud service, execute the github-hetzner-runners cloud delete command.

Running first jobs

Given that we had just added the .github/workflows/demo.yml file, which created a pending GitHub Actions job, you should see the following messages related to the new runner being created.

16:52:54 🍀 Creating new server for WorkflowJob(url="https://api.github.com/repos/vzakaznikov/demo-testflows-github-hetzner-runners/actions/jobs/23504873126", id=23504873126)
16:52:54 🍀 Validating server github-hetzner-runner-8412316197-23504873126 labels
16:52:54 🍀 Creating server github-hetzner-runner-8412316197-23504873126 with labels {'type-cpx21', 'self-hosted'}
16:52:54 🍀 Waiting to finish creating server github-hetzner-runner-8412316197-23504873126
16:52:55 🍀 Waiting for server github-hetzner-runner-8412316197-23504873126 to be ready
16:52:55    github-hetzner-runner-8412316197-23504873126 initializing
...
16:53:30    > + ./config.sh --unattended --replace --url https://github.com/vzakaznikov/demo-testflows-github-hetzner-runners --token AJ6ABQEBOMMWHKM5QIODJHTGCBZMM --name github-hetzner-runner-8412316197-23504873126-cpx21-hel1 --runnergroup Default --labels type-cpx21,self-hosted --work _work --ephemeral
16:53:30    > 
16:53:30    > --------------------------------------------------------------------------------
16:53:30    > |        ____ _ _   _   _       _          _        _   _                      |
16:53:30    > |       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
16:53:30    > |      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
16:53:30    > |      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
16:53:30    > |       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
16:53:30    > |                                                                              |
16:53:30    > |                       Self-hosted runner registration                        |
16:53:30    > |                                                                              |
16:53:30    > --------------------------------------------------------------------------------
..

If you now go and see Actions in your GitHub repository, you should see the pending action completed. Voilà, we got the first job running on our thrifty self-hosted runner!

Picking server types and running on ARM64

Having our first job executed on our self-hosted runner. We could easily run our job either on a different server type or even on ARM64 architecture. This can be quickly done by changing the type-cpx31 label, which has the type-<server type name> format, to specify a different server type. For example, let’s now use CPX41, which has 8 vCPUs, 16GB RAM, and costs €0.0409/hr. We can do this by using the type-cpx41 label.

name: GitHub Actions Demo
run-name: ${{ github.actor }} is testing out GitHub Actions 🚀
on: [push]
jobs:
  Explore-GitHub-Actions:
    runs-on: [self-hosted, type-cpx41]
    steps:
        ...

Update the .github/workflows/demo.yml and wait for the job to complete.

If we want an ARM64 runner, we must change our type label to specify one of the ARM server types, such as CAX11, giving us the type-cax11 label. We must also specify the image label since the default image is for the x86. Let’s use Ubuntu 22.04 by adding the image-arm-system-ubuntu-22.04 label.

name: GitHub Actions Demo
run-name: ${{ github.actor }} is testing out GitHub Actions 🚀
on: [push]
jobs:
  Explore-GitHub-Actions:
    runs-on: [self-hosted, type-cax11, image-arm-system-ubuntu-22.04]
    steps:
        ...

Again, commit your changes and see your job now executed on an ARM64 runner.

Checking logs

The log messages are printed to stdout if you run a service locally. However, if you’ve started a cloud service, you can quickly check the logs on the github-hetzner-runners server using the github-hetzner-runners cloud log command. There is no need to SSH into the machine directly.

For example,

$ github-hetzner-runners cloud log 
Using config file: /home/ubuntu/.github-hetzner-runners/config.yaml
21:05:05 http_cache     INFO     🍀 Enabling HTTP cache at /tmp/tmplh9qrbbl/http_cache
21:05:05 main           INFO     🍀 Logging in to Hetzner Cloud
21:05:05 main           INFO     🍀 Logging in to GitHub
...

We can also tail the log by specifying the -f option, which can be combined with the -n option, similar to the standard tail command.

$ github-hetzner-runners cloud log -f -n 10
Using config file: /home/ubuntu/.github-hetzner-runners/config.yaml
22:05:23 scale_down     INFO     🍀 Marking powered off server github-hetzner-runner-8576187015-23506726225 used 0d0h2m for
                                 recycling with 58m of life
22:05:28 api_watch      INFO     🍀 Logging in to GitHub
22:05:28 api_watch      INFO     🍀 Checking current API calls consumption rate
22:05:28 api_watch      INFO     🍀 Consumed 1 calls in 60 sec, 4993 calls left, reset in 3442 sec
22:06:28 api_watch      INFO     🍀 Logging in to GitHub
22:06:28 api_watch      INFO     🍀 Checking current API calls consumption rate
22:06:28 api_watch      INFO     🍀 Consumed 0 calls in 60 sec, 4993 calls left, reset in 3382 sec
22:07:28 api_watch      INFO     🍀 Logging in to GitHub
22:07:28 api_watch      INFO     🍀 Checking current API calls consumption rate
...

Such a convenient way to check logs helps monitor any issues or unexpected behaviors you might encounter. All can be done using the same github-hetzner-runners command that we’ve used to deploy our cloud runners service.

Estimating costs

In addition to a convenient way to check logs, we can estimate costs for different runs using the github-hetzner-runners estimate command. This command can estimate costs for a job, a run, or a set of recent runs. For example, let’s check the estimates for my recent runs.

$ github-hetzner-runners estimate runs     
18:13:23 🍀 Logging in to Hetzner Cloud
18:13:23 🍀 Logging in to GitHub
18:13:23 🍀 Getting repository vzakaznikov/demo-testflows-github-hetzner-runners
18:13:23 🍀 Getting current server prices
18:13:24 🍀 Getting workflow runs
18:13:24 🍀 Getting jobs for the workflow run id 8576187015
- name: vzakaznikov is testing out GitHub Actions 🚀
  id: 8576187015
  attempt: 1
  jobs:
  ...
  estimate:
    servers:
    - type: cpx41
      location: hel1
      price: 0.041700
      duration: 00:00:10
      worst: 0.041700
      best: 0.000116
    worst: 0.041700
    best: 0.000116
✋ Press any key to continue (Ctrl-D to abort)...

As we can see, my recent run used the CPX41 server type, and the estimated range is between €0.0417, the worst-case, and €0.000116, the best-case. The worst-case scenario is taken when the server is not re-used; therefore, the cost is the time the job has taken rounded to the nearest hour. This is because Hetzner servers are billed per hour. The best case is when other jobs actively re-use the server, and you get a cost close to per-minute billing. You can read more about estimating costs in the Estimating Costs section.

Our example usage

Altinity’s builds of ClickHouse have extra test coverage provided by our clickhouse-regression tests, for which we also run CI/CD separately. Below is one of our active clickhouse-regression runs that has 47 jobs and uses tens of active runners to speed up overall execution time.

One of the complete runs can be found here, which took just 3h 21m to execute with overall compute usage of 1d 8h 45m.

The handy cost estimator shows that this run costs us between €4.06 and €2.35.

$ github-hetzner-runners estimate run 8484350712
...
total:
    jobs: 47
    duration: '32:45:05'
  known:
    jobs: 47
    duration: '32:45:05'
  unknown:
    jobs: 0
    duration: 00:00:00
  estimate:
    servers:
    - type: cpx41
      location: hel1
      price: 0.041700
      duration: '11:18:56'
      worst: 1.334400
      best: 0.471859
    - type: cpx51
      location: hel1
      price: 0.088000
      duration: '21:26:09'
      worst: 2.728000
      best: 1.886353
    worst: 4.062400
    best: 2.358212

This is why having a cost-effective solution for CI/CD runners is critical in enabling us to push the limit of our ever-growing GitHub Actions pipelines.

Conclusion

Now you can run CI/CD jobs using the thrifty Hetzner Cloud. Again, all you need to get started is a GitHub and Hetzner Cloud account. The installation and deployment of TestFlows GitHub Hetzner Runners is simple, given that most common tasks can be accomplished with the same github-hetzner-runners command. We’ve seen that deploying to a remote cloud instance is trivial, but day-two tasks such as checking logs and estimating costs for your runs are also straightforward. Combined with an easy control of server type and images using labels, it provides a setup that is easy to manage. We are actively using these runners in production for our ClickHouse as well as the clickhouse-regression projects where close to a hundred jobs could sometimes be running simultaneously. Give it a try! If you have any issues, please ping us on GitHub. We’ll be happy to help you get started.

Get in touch with ClickHouse experts.

Get in touch with ClickHouse experts.

Slash CI/CD Bills (Part 2): Using Hetzner Cloud GitHub Runners for Your Repository

Features and limitations

Installation

Preparing GitHub repository

Setting up the Hetzner Cloud project

Starting the service

Running first jobs

Picking server types and running on ARM64

Checking logs

Estimating costs

Our example usage

Conclusion

Related:

Leave a Reply Cancel reply

Get in touch with ClickHouse experts.

Get in touch with ClickHouse experts.

Features and limitations

Installation

Preparing GitHub repository

Setting up the Hetzner Cloud project

Starting the service

Running first jobs

Picking server types and running on ARM64

Checking logs

Estimating costs

Our example usage

Conclusion

Related:

Low-Cost ClickHouse clusters using Hetzner Cloud with Altinity.Cloud Anywhere

Slash CI/CD Bills (Part 1): Using Hetzner Cloud GitHub Runners for ClickHouse Builds

Build a Low-Cost, High-Performance Analytic Platform with Kubernetes and Open Source

Leave a Reply Cancel reply