AWS Batch Job Definition with Fargate via resourceRequirements

Create a Job Definition using Fargate setting resourceRequirements instead computeResources

I was creating with a AWS Batch Job Definition via Boto3 to set the environment/resources (i.e. set the ECR image, environment variables, instance type, memory, vCPUs, etc.) that would be used in a Machine Learning batch processing routine and I found out that it’s documentation was quite confusing to provide an actual example of how to do it via AWS Fargate.

The Errors

Initially, I tried to use the vanilla example presented in the docs and I started to get the following error:

Exception : vcpus is not applicable for Fargate. Use resource requirements instead

With that message, I just created a new section called resourceRequirements and included the vcpus attribute. Executing again the script I received a new error:

"Cannot use a combination of containerProperties vcpus/memory and respective resourceRequirements"

The solution was to remove the current memory attribute from the containerProperties and include that as a new value in the resourceRequirements.

I decided to read those parameters in the docs to check what I missed and why I got those errors…

The Issue

The current documentation has the vcpus and memory as attributes in the main JSON, however, the same documentation says that those two attributes are deprecated and instead should be passed via resourceRequirements instead to be set in the computeResources or in the containerProperties.

I confess that I was a bit puzzled by those errors and with the documentation. I say that because if you have in some documentation example two or more distinct ways to set some attributes it should be clear that distinction and separate examples should be provided. Especially if there’s some attribute inconsistency during the API calls and some of those attributes are actually deprecated but still working (It works using AWS EC2 instead AWS Fargate).

Solution

The code containing the working solution I found for this issue is in the Gist below: