GitLab CI

Rationale

GitLab CI  is the system that orchestrates all the CI/CD  workflows within our company. Such workflows are the backbone of our entire development cycle . By using it, we become capable of:

• Running automated processes on every commit 
• Automating application testing 
• Automating application deployment 
• Automating every QA test we can think of 

By having highly automated workflows, we become capable of deploying applications many times a day  without sacrificing quality or security.

The main reasons why we chose GitLab CI  over other alternatives are:

• It is Open source .
• Built-in support for GitLab : As GitLab is the platform we use for our product repository , it represents an advantage for us to be able to easily integrate our CI  solution with it. All GUI -related capabilities like pipelines , jobs , CI/CD variables , environments , schedules , and container registries  are a consequence of such integration.
• It supports pipelines as code : It allows us to write all our pipelines as code .
• It supports horizontal autoscaling : In order to be able to run hundreds of jobs  for many developers, all in real time, our system must support horizontal autoscaling .
• It supports directed acyclic graphs (DAG) : Such a capability allows us to make our integrations as fast as possible, as jobs  exclusively depend on what they really should. It is a must when implementing a monorepo  strategy like ours .
• Highly versatile configurations : As every piece of software usually has its own needs when it comes to building, testing and deploying, GitLab CI  offers a vast set of configurations  that range from parallelism , static pages , and services  to includes , workflows  and artifacts .
• Highly versatile infrastructure : The AWS autoscaler  allows configurations for S3 cache , machine type , max number of machines , spot instances , C5d instances, SSD disk usage , EBS disks , off-peak periods , tagging , max builds before destruction , among many others. The importance of having a highly versatile CI  comes from the fact that our development cycle  completely depends on it, making us expect clockwork-like responsiveness and as-fast-as-possible computing speed.

Alternatives

The following alternatives were considered but not chosen for the following reasons:

• Jenkins : It did not support pipelines as code  at the time it was reviewed.
• TravisCI : It required licensing for private repositories at the time it was reviewed.
• CircleCI : It did not support GitLab , was very expensive, and was not as parameterizable.
• Buildkite : It is still pending for review.

Usage

We use GitLab CI  for:

• Running all our CI/CD  jobs.
• Managing all our CI pipelines as code .
• Configuring our AWS autoscaler  as code .
• Implementing a Continuous Delivery  approach for our development cycle . This means that although the whole process is automated, including deployments for both development and production, a manual merge request approval  from a developer is still required in order to be able to deploy changes to production.

We do not use GitLab CI  for highly time-consuming schedules that take longer than six hours, like Analytics ETLs , Machine learning  training, among others. We use AWS Batch  instead. The reason for this is that the GitLab CI  is not meant to run jobs  that take that many hours, often resulting in jobs  being terminated before they can finish, mainly due to disconnections between the worker running the job and its GitLab CI Bastion .