AWS Batch and AWS Step functions broken by terraform apply

@Link512 wrote:

A short simplified description of my setup:

Let’s say I have a SFN State machine with 2 states, A,B. Each of them use an AWS Batch job with a certain revision like so:

A -> jobA:3
B -> jobB:4

This State machine is being run on a cron schedule and might take a few hours to complete.

When a deployment via CI/CD happens, the following things may occur:

1. the docker image for jobB changes, thus the container_properties for the job change
2. terraform will force a new resource, causing jobB:4 to be marked as INACTIVE and a new revision jobB:5 to be submitted
3. The sfn state machine now is:

   A -> jobA:3
   B -> jobB:5

If a SFN is running while this deployment is being made:

• the SFN definition includes the old revision of jobB
• state A finishes, SFN will try to queue jobB:4 causing an error

Is there a way of preventing this? I know that terraform’s default behaviour for forcing a new resource is “delete old resource, create a new one”, but in the case of AWS Batch where you have revisions, it would be nice if there were a way to preserve the old revisions of a job without marking them as inactive

Posts: 4

Participants: 2

Read full topic