Nsryjdtyk

I have installed and configured:

1. an on-premises GitLab Omnibus on ServerA running on HTTPS


2. an on-premises GitLab-Runner installed as Docker Service in ServerB
   

ServerA certificate is generated by a custom CA Root

The Configuration

I've have put the CA Root Certificate on ServerB:

/srv/gitlab-runner/config/certs/ca.crt

Installed the Runner on ServerB as described in Run GitLab Runner in a container - Docker image installation and configuration:

docker run -d --name gitlab-runner --restart always 

           -v /srv/gitlab-runner/config:/etc/gitlab-runner 

           -v /var/run/docker.sock:/var/run/docker.sock 

           gitlab/gitlab-runner:latest

Registered the Runner as described in Registering Runners - One-line registration command:

docker run --rm -t -i 

            -v /srv/gitlab-runner/config:/etc/gitlab-runner 

           --name gitlab-docker-runner gitlab/gitlab-runner register 

           --non-interactive 

           --executor "docker" 

           --docker-image alpine:latest 

           --url "https://MY_PRIVATE_REPO_URL_HERE/" 

           --registration-token "MY_PRIVATE_TOKEN_HERE" 

           --description "MyDockerServer-Runner" 

           --tag-list "TAG_1,TAG_2,TAG_3" 

           --run-untagged 

           --locked="false"

This command gave the following output:

Updating CA certificates...

Runtime platform arch=amd64 os=linux pid=5 revision=cf91d5e1 version=11.4.2

Running in system-mode.

Registering runner... succeeded runner=8UtcUXCY

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

I checked with

$ docker exec -it gitlab-runner bash 

and once in the container with

$ awk -v cmd='openssl x509 -noout -subject' '

/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt

and the custom CA root is correctly there.

The Problem

When running Gitlab-Runner from GitLab-CI, the pipeline fails miserably telling me that:

$ git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}@ServerA/foo/bar/My-Project.wiki.git

Cloning into 'My-Project.wiki'...

fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@ServerA/foo/bar/My-Project.wiki.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

ERROR: Job failed: exit code 1

It does not recognize the Issuer (my custom CA Root), but according to The self-signed certificates or custom Certification Authorities, point n.1, it should out-of-the-box:

Default: GitLab Runner reads system certificate store and verifies the GitLab server against the CA’s stored in system.

I've then tried the solution from point n.3, editing

/srv/gitlab-runner/config/config.toml:

and adding:

[[runners]]

tls-ca-file = "/srv/gitlab-runner/config/certs/ca.crt"

But it still doesn't work.

How can I make Gitlab Runner read the CA Root certificate?

While I've still not got why it doesn't work out-of-the-box, I've found the Egg of Columbus:

Gitlab-Runner configuration:

[[runners]]

  name = "MyDockerServer-Runner"

  url = "https://MY_PRIVATE_REPO_URL_HERE/"

  token = "MY_TOKEN_HERE"

  executor = "docker"

  ...

  [runners.docker]

    image = "ubuntu:latest"



  # The trick is the following:

    volumes = ["/cache","/srv/gitlab-runner/config:/etc/gitlab-runner"]

    ...

Gitlab-ci.yml pipeline:

MyJob:

    image: ubuntu:latest



    script:

      - awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt

      - git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}@ServerA/foo/bar/My-Project.wiki.git

      - wget -O foo.png https://ServerA/foo/bar/foo.png 



    before_script:

      - apt-get update -y >/dev/null

      - apt-get install -y apt-utils dialog >/dev/null

      - apt-get install -y git >/dev/null

      - apt-get install -y wget >/dev/null



    # The trick is the following:

      - cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt

      - update-ca-certificates

That's it:

• Mount the volume once (per Docker executor)


• Update the CA certificates once (per job)

And everything will work as expected: git clone, wget https, etc...

A great workaround, until someone at GitLab will fix it or explain me where I'm wrong (be my guest!)

You have two options:

Ignore SSL verification

Put this at the top of your .gitlab-ci.yml:

variables:

  GIT_SSL_NO_VERIFY: "1"

Point GitLab-Runner to the proper certificate

As outlined in the official documentation, you can use the tls-*-file options to setup your certificate, e.g.:

[[runners]]

  ...

  tls-ca-file = "/etc/gitlab-runner/ssl/ca-bundle.crt"

  [runners.docker]

  ...

As the documentation states, "this file will be read every time when runner tries to access the GitLab server."

Other options include tls-cert-file to define the certificate to be used if needed.

From the output you provided i think that the certificate might be OK but you are lacking the CRL file : server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

The CRL file is used to verify that even if the certificate is valid is hasn't been revoked by the CA owner. You shoudl then need to :

1) Generate a CRL file based on your CA:

openssl ca -gencrl -keyfile ca.key -cert ca.crt -out crl.pem

source: https://blog.didierstevens.com/2013/05/08/howto-make-your-own-cert-and-revocation-list-with-openssl/

2) Instruct the runner to use it :

[[runners]]

  ...

  tls-ca-file = "/etc/gitlab-runner/ssl/ca-bundle.crt"

  crl-file = "/etc/gitlab-runner/ssl/ca.crl"

3) Of course setting GIT_SSL_NO_VERIFY will work but you will be more sensitive to man-in-the-middle attacks