Batch

Rationale

We use Batch  for running batch processing  jobs in the cloud . The main reasons why we chose it over other alternatives are the following:

• It is SaaS  (software as a service), so we do not need to manage any infrastructure directly.
• It is free , so we only have to pay for the Elastic Compute Cloud (EC2 ) machines we use to process workloads.
• It complies with several  certifications from ISO  and CSA . Many of these certifications are focused on ensuring that the entity follows best practices regarding secure cloud-based  environments and information security.
• We can monitor  job logs using CloudWatch .
• The jobs are highly resilient , which means they rarely go unresponsive. This feature is very important when jobs take several days to finish.
• It supports EC2 spot instances , which considerably decreases EC2 costs.
• All its settings can be written as code  using Terraform .
• We can use Nix  to queue jobs easily .
• It supports priority-based queuing , which allows us to prioritize jobs by assigning them to one queue or another.
• It supports automatic retries  of jobs.
• It integrates  with Identity and Access Management (IAM ), allowing us to keep a least privilege  approach regarding authentication and authorization .
• EC2 workers running jobs can be monitored using CloudWatch.

Alternatives

GitLab CI

We used GitLab CI  before implementing Batch . We migrated because GitLab CI is not intended to run scheduled jobs that take many hours, often resulting in jobs becoming unresponsive before they could finish, mainly due to disconnections between the worker running the job and the GitLab CI Bastion . On top of this, GitLab CI has a limit on the number of schedules per project, and running thousands of jobs puts a lot of pressure on the GitLab coordinator and the GitLab CI Bastion.

Buildkite

Pros:

• Buildkite  handles submission of duplicated jobs.
• Gives us logging, monitoring, and stability measurements out of the box.
• We can separate costs by having different queues (associated with different deployments).
• Notifications out-of-the-box to email and others.
• Support pipelines out-of-the-box.
• They have an API to query information about past jobs on a pipeline and trigger new builds, which is much more flexible than Batch’s API.

Cons: It is much more expensive.

Kubernetes Jobs

See Jobs in Kubernetes documentation .

Pros: It allows better separation of costs.

Cons: It requires manually kick-starting a build, because it doesn’t listen automatically to the queue as Batch does.

Usage

We use Batch  for running

• production background schedules  for all our components, and
• ARM background tasks , like cloning roots and refreshing targets of evaluation.