Continuous integration process consists in automatically building software sources, run tests, and publish documentation and binaries of packages.
This page describes how to deal with continuous integration, based on the use of a gitlab server:
For gitlab administrators, how to put in place continuous integration process.
For package developpers, how to configure project to enable continuous integration for a project
Install continuous integration
In following subsections we explain de process that administrator have to follow to put in place contiunous integration. This process is based on gitlab so we suppose that projects repositories are available on a gitlab server. In gitlab, continuous integration is already integrated, but is divided into two parts: coordinator server and ci-runners. The coordinator server is the server hosting the gitlab application so you have no more to do except deploying a gitlab server like gite.lirmm.fr. The ci-runners are the application that trully run the build/test process depending on the coordinator ask them, and then they can give back resulting artefacts to the coordinator once the build/test process is finished. The following subsection will mainly speak about how to deploy and configure a runner (either on a server or on a workstation, this does not really matter even if it is preferable to have a runner always running and accessible on a server). We will call the machine that host the ci-runner the build server.
Install docker on the build server
Docker engine is required to execute the ci-runner with docker container management. You can follow the gitlab help page to kno how to installa runner or follow these instructions, considering you are working on a debian like system:
- update the server packages database:
- Add the GPG key for the official Docker repository to the system:
- Add the Docker repository to APT sources:
- Update the package database with the Docker packages from the newly added repo:
- You should see output similar to the follow:
Notice that docker-engine is not installed, but the candidate for installation is from the Docker repository for Ubuntu 16.04. The docker-engine version number might be different.
- Install the cadidate package:
- Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it’s running:
- The output should be similar to the following, showing that the service is active and running:
- If you want to avoid typing sudo whenever you run the docker command, add your username to the docker group:
If you want to uninstall docker do:
Install gitlab-ci runner on the server
- Add GitLab’s official repository:
- install the runner:
update runner version
Create a dedicated docker image for PID in gitlab-ci
You need to setup a specific docker image that contains all the required dependencies in order for PID CI system to work:
- Find the base image corresponding to the kernel in use. For instance on a ubuntu server we will use the ubuntu image:
The output gives you all available images for ubuntu on docker hub, something like:
Choose the adequate image, for now we use the basic
ubuntu image. To select an adeqaute version of ubuntu use docker tags like
ubuntu:18.04 to get the corresponding version of the distribution.
- Installing this image locally:
Now we can use it to create a new docker image.
- Create the new image by using an interactive approach:
Now you enter in a bash session.
- configure the container in order to manage ssh and generate ssh keys WARNING: do not enter any passphrase when creating keys otherwise the runner would prompt for it when running tests !!
- To find out the hostkeys of your gitlab server, put the result of the
ssh-keyscan YOUR_SERVERcommand into your ssh known hosts:
- adding required dependencies for a full featured “PID ready” runner:
- configure the runner git account:
Here simply use same name and email as those used when the PID GitlabRunner identity has been created in gitlab. Content generated by this runner will so be tagged as written by
- adding git lfs dependency:
- adding sphinx dependency (only useful when runner must build the pid framework, to optimize a bit the configuration process):
- exitting the interactive mode :
simple type ‘exit’ in the shell.
- Registering the container as a reusable image:
- Getting the id of the container you just exitted
The output gives you the id of this container, something like:
- Create the new image:
You have to replace names and IDs with the adequate informations.
The image is now cached, to see it look into the local repository:
Do not worry if the image is quite big is normal because we have a fully functional development environment.
Note: You can install more package to get a more complete environment. This may be useful if you want your runner to reflect the configuration of a given computer/server.
Register the runner in gitlab
In gitlab create a dummy account for the runner (e.g. with username pid-gitlab-runner), then add the public ssh key to this account by copy/pasting the content of the public generated in the docker container. You can otherwise add an existing PID runner.
As a final step, in gitlab add the new PID GitlabRunner user to the pid-workspace project. Go to Settings > Member and add it as a Developper of the project. Another way to do that is to set the PID GitlabRunner user as a member of the group containing the official pid-workspace project.
Executing the runner
This step consists in running the gitlab runner service and in regitering this runner into the gitlab instance.
You need to collect some information prior to executing the runner:
- the URL of your gitlab instance. At LIRMM the URL of our gitlab server that coordinates runner is https://gite.lirmm.fr
- go into Admin area, in menu Overview click on the Runners tab. The pages provide you the token to use got registering the runner.
- the name of the docker image to use. This is the image generated at previous step, that we called pid-runner-x86_64 (use the good name depending on your server specification).
- the platform you are currently targetting with the container. It depends on: the architecture type in use (x86, arm), the architecture bits (32 or 64), the OS (without greater customization by default linux is the only possible solution with docker), and the abi in use, depending on your compiler version (if your compiler is gcc with version greater than 5.0 or clang with version greater than 3.0, the ABI is abi11, otherwise it is abi98 (the legacy ABI for C/C++). From these information you can build a platform description string like: x86_64_linux_stdc++11 (put underscores between each term).
Now you can execute the runner by setting adequate information for options –url, –registration-token (given by gitlab administrator interface), –docker-image (pointing to the docker image we just created) and –tag-list (add a tag describing the host platform). For this later option keep the term
pid into the list to tell the system it is configured adequately with PID dependencies.
The program prompts the user to enter values, simply press enter to keep default values passed as parameters.
Configuring the runner in gitlab (required admin rights in gitlab)
Go to the runners page of gitlab admin area: the new gitlab runner should appear.
One last configuration may be required depending on version of gitlab you are using:
Activeoption should be checked.
Protectedmust be unchecked.
Lock to current projectoption must be unchecked.
Run untagged jobsoption should be unchecked, in order to avoid non PID build to interfere with PID jobs.
Final configuration of the build server
On the server edit the file /etc/gitlab-runner/config.toml. This file is used to configure the behavior of runners (those registered by root superuser). This file contains a section for each registered runner something that looks like.
If you have multiple runner registered just try to find the adequate section by simply looking at
image = property. When done add to this section the parameter
pull_policy = "if-not-present":
Enable continuous integration for a package
From package developer perspective there are few steps to follow to enable continuous integration.
In gitlab go into your project general settings and activate pipelines. The CI/CD pipelines menu now appears in project settings.
In this menu choose the git clone strategy in the section Git strategy for pipelines, otherwise builds will fail (due to a BUG in gitlab-ci relative to the management of git LFS).
In order to optimize a bit the CI process you should also check the Auto-cancel redundant, pending pipelines option and uncheck the Public pipelines.
Finally add the available runners gitlab users to your gitlab project. Ask the PID administrators for the name of users to set as member. You can choose the Developper for these users (so that you are sure they will not push to master branch).
That’s it your project is ready to be automatically built and deployed.
These steps are the same wether the considered gitlab project is:
- a package project
- a framework project
- a package static site project
Additional configuration for static sites projects
Static site project require one additionnal step : you have to set runners gitlab user that are member of the project as Master (instead of Developpers), so that they will be able to update their content.