Skip to main content

Batch

Rationale#

We use Batch for running batch processing jobs in the cloud.

The main reasons why we chose it over other alternatives are:

  1. It is SaaS, as no infrastructure needs to be directly managed.
  2. It is free, as we only need to pay for the EC2 machines used to process workloads.
  3. It complies with several certifications from ISO and CSA. Many of these certifications are focused on granting that the entity follows best practices regarding secure cloud-based environments and information security.
  4. Job logs can be monitored using CloudWatch.
  5. Jobs are highly resilent, meaning that they rarely go irresponsive. This feature is very important when jobs take several days to finish.
  6. It supports EC2 spot instances, which considerably decreases EC2 costs.
  7. All its settings can be written as code using Terraform.
  8. We can use Nix for easily queueing jobs.
  9. It supports priority-based queues, allowing to prioritize jobs by assigning them to one queue or another.
  10. It supports job automatic retries.
  11. It integrates with IAM, allowing to keep a least privilege approach regarding authentication and authorization.
  12. EC2 workers running jobs can be monitored using CloudWatch.

Alternatives#

Gitlab CI: We used it before implementing Batch. We migrated because Gitlab CI is not meant to run scheduled jobs that take many hours, often resulting in jobs becoming irresponsive before they could finish, mainly due to disconnections between the worker running the job and the Gitlab CI Bastion.

Usage#

We use Batch for:

  1. Running Observes ETLs.
  2. Running Skims scans.
  3. Running Skims OWASP Benchmark.
  4. Running ASM Users to Entity reports.

Guidelines#

  1. You can access the Batch console after authenticating on AWS.
  2. Any changes to Batch infrastructure must be done via Merge Requests.
  3. You can queue new jobs to Batch by using the compute-on-aws module.
  4. If a scheduled job takes longer than six hours, it generally should run in Batch, otherwise it can use the Gitlab CI.
  5. To learn how to test and apply infrastructure via Terraform, visit the Terraform Guidelines.