Step Functions

Rationale

We use Step Functions  for orchestrating workflow  execution in the cloud . The main reasons why we chose it over other alternatives are the following:

• It is SaaS  (software as a service), so we do not need to manage any infrastructure directly.
• It is inexpensive , so we only have to pay for the state transitions and the underlying compute resources (like Lambda  functions, EC2  instances, or Batch  tasks) we use to execute workflow steps.
• It complies with several certifications  from ISO  and CSA . Many of these certifications are focused on ensuring that the entity follows best practices regarding secure cloud-based  environments and information security.
• We can monitor  workflow execution and state transitions using CloudWatch .
• The workflows are highly resilient , with built-in error handling and retry mechanisms. This feature is very important when workflows orchestrate long-running processes that take several hours or days to finish.
• It supports visual workflow design  and provides a visual representation of workflow execution, making it easier to understand and debug complex workflows.
• All its settings can be written as code  using Terraform .
• It supports parallel execution  of workflow branches, allowing us to run multiple tasks concurrently and improve overall workflow performance.
• It supports conditional branching  and dynamic routing , enabling complex workflow logic based on data conditions.
• It supports automatic retries  with configurable retry policies for handling transient failures.
• It integrates  seamlessly with over 200 AWS services, including Lambda, ECS, Batch, DynamoDB, SNS, SQS, and many others.
• It integrates with Identity and Access Management (IAM ), allowing us to keep a least privilege  approach regarding authentication and authorization.
• It maintains execution history  for up to 90 days, providing visibility into workflow execution and making it easier to troubleshoot issues.

Alternatives

AWS Batch

We use Batch  for running batch processing jobs. While Batch is excellent for executing individual jobs, Step Functions provides superior orchestration capabilities when we need to coordinate multiple steps, handle complex branching logic, or integrate with various AWS services in a workflow.

We use Step Functions instead of Batch:

• Orchestrating multi-step workflows with dependencies
• Needing conditional logic and dynamic routing
• Integrating multiple AWS services in a single workflow
• Requiring visual workflow representation and debugging

We use Batch instead of Step Functions:

• Executing single, long-running batch processing jobs
• Compute resource management as primary need rather than workflow orchestration

Apache Airflow

Apache Airflow  is a popular open-source workflow orchestration platform.

Pros:

• Highly flexible and extensible
• Rich ecosystem of operators and integrations
• Strong community support

Cons:

• Requires infrastructure management and maintenance
• More complex setup and configuration
• Higher operational overhead
• Less native integration with AWS services compared to Step Functions

Temporal

Temporal  is a modern workflow orchestration platform.

Pros:

• Excellent developer experience with SDKs in multiple languages
• Strong support for long-running workflows
• Good observability and debugging tools
• Can handle complex workflow patterns

Cons:

• Requires infrastructure management (unless using Temporal Cloud, which is paid)
• Less native integration with AWS services
• Higher learning curve for teams unfamiliar with Temporal concepts

Custom orchestration logic

Before implementing Step Functions, we used a custom orchestration approach built with Python code, DynamoDB , and Batch . This approach involved writing imperative Python code to determine workflow steps, manually managing dependencies, and using DynamoDB as a job queue.

Pros:

• Can implement complex business logic directly in code
• No additional service costs for orchestration (only DynamoDB and Batch costs)
• Works well with existing Python codebase and patterns
• Can easily integrate with existing data loaders and business logic

Cons:

• Workflow logic is imperative and scattered across code, making it harder to understand end-to-end workflows
• No visual representation of workflows, making debugging and onboarding more difficult
• Manual dependency management requires careful code maintenance
• Harder to reason about complex multi-step workflows
• No built-in retry policies or error handling at the orchestration level
• Workflow changes require code changes and deployments
• Limited observability into workflow execution state and history
• Workflow logic difficult to test in isolation

Usage

We use Step Functions  for orchestrating:

• Repository synchronization  for our clients’ repositories, including cloning, post-processing and vulnerability reporting.
• Production background schedules  for some of our components.