I have set up a new box for one of my side projects and am trying to make it as container-friendly as possible. I started from scratch with a new version of TrueNAS that no longer uses kubernetes. For my setup, this is fine since I only have two containers that don’t need any fancy orchestration.

One recent change I made was to have my ASP.NET Core container authenticate against AWS using IAM Roles Anywhere. With this method of authentication, you can have your container assume an IAM role, similar to how a container in ECS assumes role. I also don’t have to manage credentials on the box itself.

How does the container authenticate against AWS?

The container uses a certificate and private key to get AWS access keys which are actually what are used to authenticate against the AWS services. You also need to be in control over the CA (certificate authority) that issued the certificate and private key because you need to tell AWS how to verify clients (like my container). By providing the CA information to AWS, IAM Roles Anywhere can use the CA’s public certificate to verify the container’s certificate was actually generated by that same CA.

If you don’t have a CA, AWS has a simple solution: AWS Private CA.

Setting up IAM Roles Anywhere

Once I had my CA, I used it to generate a certificate with a private key, which will be solely used by my container.

The next step was to then tell AWS about the CA. To do this, you need to go to IAM in the AWS console => Roles => and then look for Roles Anywhere section at the bottom. I am not sure why it is so hidden because I always forget how to get to it when I go to the IAM console.

In the IAM Roles Anywhere manager, you then need to create:

1. Trust anchor - I pasted my CA’s certificate here (not the certificate for your container) in here so that AWS knows to trust your container certificate.
2. Profile - The profile is how you tell AWS which roles you want to be usable by whichever entity is trying to authenticate with a certificate (in my case, the ASP.NET Core container)

Configuring the ASP.NET Core app to authenticate using IAM Roles Anywhere

In order for the verification process to happen between the AWS trust anchor and the app, the client creates a signature for AWS to verify using the certificate and private key. From my research, it doesn’t look like the .NET AWSSDK itself can do the signing. So instead, I used the official AWS tool: aws_signing_helper.

Rather than having my ASP.NET Core app call this executable to generate the credentials, I saw you could configure your ~/.aws/credentials file with a profile that called this executable itself.

[profile roles_anywhere]
    credential_process = ./aws_signing_helper credential-process --certificate /path/to/certificate --private-key /path/to/private-key --trust-anchor-arn arn:aws:rolesanywhere:region:account:trust-anchor/TA_ID --profile-arn arn:aws:rolesanywhere:region:account:profile/PROFILE_ID --role-arn arn:aws:iam::account:role/role-name-with-path
    region = region

And then you simply use the .NET AWSSDK to read credentials as if it were reading from the ~/.aws/credentials file:

var chain = new CredentialProfileStoreChain();
AWSConfigs.AWSProfileName = "roles_anywhere";
if (!chain.TryGetAWSCredentials("roles_anywhere", out var credentials))
{
    throw new Exception("Missing AWS credentials profile");
}

return new AmazonS3Client(credentials);

With just that, my container was able to assume the role I specified in the IAM Roles Anywhere profile.

How to give the container access to the certificate and private key

When I was learning about this process, I almost forgot to consider how my container would get access to the certificate and private key. In the same way you want to avoid baking in credentials to a container image, you definitely don’t want to do the same thing with the private key.

In my case, this was simple since I am in control of the box. I just mounted a volume to the container where the certificate and private key are stored.

Configuring the container to run the signing helper

Because the certificate and private key are just files on the file system from the perspective of my app, I only needed to make sure my container image could dynamically get these locations along with the ARNs of the trust anchor, profile, and corresponding role. The ARNs and file paths to the certificate and private key alone are not sensitive, so it was fine to inject them as ENV vars.

To accomplish this, I copied the following ~/.aws/credentials template file into my container image:

[profile roles_anywhere]
    credential_process = /home/app/aws_signing_helper credential-process --certificate __CERT_PATH__ --private-key __PK_PATH__ --trust-anchor-arn __TRUST_ARN__ --profile-arn __PROFILE_ARN__ --role-arn __ROLE_ARN__
    region = __REGION__

And then my Dockerfile’s ENTRYPOINT simply runs a script that replaces the placeholders surrounded by double underscores with the corresponding values from the ENV vars. After, it starts the ASP.NET Core app as a standard ASP.NET Core image would:

#!/bin/sh

perl -0777 -pe "
    s|__CERT_PATH__|$CERT_PATH|g;
    s|__PK_PATH__|$PK_PATH|g;
    s|__TRUST_ARN__|$TRUST_ARN|g;
    s|__PROFILE_ARN__|$PROFILE_ARN|g;
    s|__ROLE_ARN__|$ROLE_ARN|g;
    s|__REGION__|$REGION|g;
    " /home/app/.aws/credentials_template > /home/app/.aws/credentials

dotnet DotSync.Apps.WebApp.dll

You can see my full Dockerfile here.