AWS Batch Job Definition with Fargate via resourceRequirements2022 Jan 05
Create a Job Definition using Fargate setting resourceRequirements instead computeResources
I was creating with a AWS Batch Job Definition via Boto3 to set the environment/resources (i.e. set the ECR image, environment variables, instance type, memory, vCPUs, etc.) that would be used in a Machine Learning batch processing routine and I found out that it’s documentation was quite confusing to provide an actual example of how to do it via AWS Fargate.
Initially, I tried to use the vanilla example presented in the docs and I started to get the following error:
Exception : vcpus is not applicable for Fargate. Use resource requirements instead
With that message, I just created a new section called
resourceRequirements and included the
attribute. Executing again the script I received a new error:
"Cannot use a combination of containerProperties vcpus/memory and respective resourceRequirements"
The solution was to remove the current
memory attribute from the
include that as a new value in the
I decided to read those parameters in the docs to check what I missed and why I got those errors…
The current documentation has the
memory as attributes in the main JSON,
however, the same documentation says that those two attributes are deprecated and
instead should be passed via
resourceRequirements instead to be set in the
computeResources or in the
I confess that I was a bit puzzled by those errors and with the documentation. I say that because if you have in some documentation example two or more distinct ways to set some attributes it should be clear that distinction and separate examples should be provided. Especially if there’s some attribute inconsistency during the API calls and some of those attributes are actually deprecated but still working (It works using AWS EC2 instead AWS Fargate).
The code containing the working solution I found for this issue is in the Gist below: