AWS Fargate Spring Boot CI-CD - AWS and serverless

AWS Fargate Spring Boot Continuous integration and deployment (CI-CD)

In last few weeks I was working on AWS CodePipeline/Commit/Build products. In result, I finished a small application that includes CDK CI-CD stack for Spring Boot webapp running inside Fargate cluster.

It can be found on Github:

https://github.com/stokilo/aws-spring-cdk-ci-cd

CDK stacks included in the infrastructure project provision CodeCommit repository, ECR repository, CodePipeline, and Fargate cluster. CodePipeline is executed on each git commit. It rebuilds the Docker Spring Boot application and redeploys it on the ECS.

Configuration details can be found on Github in the README.MD file.

Cross-account stacks.

I've decided to use IaC tools to model my organization. I found an interesting project:

https://github.com/org-formation/org-formation-cli

It is possible to dump your existing organization into a Cloudformation like file called organization.yml. Once it is done, you can edit it and add new accounts, change SCP, and many more.

I've modeled a sample organization unit with the name 'STC'

Root
  -> STC
    -> INFRASTRUCTURE (account 111111111111)
       ---> CodeCommit
       ---> ECR
    -> CICD (account 222222222222)
       ---> CodePipeline
       ---> Fargate cluster
    -> WORKLOADS
      -> DEV (account 333333333333)
      -> PROD ...
      -> TEST ...

INFRASTRUCTURE org unit holds single account 111111111111. I store there my CodeCommit and ECR repositories.

CICD org unit is for the CodePipeline stack. There I also run the Fargate cluster. CodePipeline checkout source code and fetch Docker images from 111111111111 account.

This requires to configure cross-account stacks and appropriate permissions. CDK helps a lot in this area. However, it was not easy to find out how to do it properly.

I've created a role that is allowed to be assumed by CICD account (222222222222) only. This role was created in the CodeCommit account (111111111111)

SSO configuration

I configured my development machine to use SSO short term credentials.

This project includes a sample helper script (sso.sh) to refresh tokens and update ~/.aws/credentials file. You first should run 'aws configure sso' from the command line and configure all your SSO profiles under ~/.aws/config. Each profile has a name, region, auth link, and permission set associated. This is enough for sso.sh to execute the AWS CLI command to refresh access keys under credentials file. The browser window is open to input credentials and MFA token.

I store account configuration together with region and profile names under env.properties. CDK deployment is wrapped with another shell script cdk.sh . This script is nothing more than a set of helper functions to pass the correct profile name to the deployment target. Additionally, it performs SSO token refresh.

I found this process quite ok and decided to migrate my organization to use SSO. More technical details on how to configure SSO are included in the Github repository.

Deployment order

All stacks must be deployed in order. You start with the infrastructure stack. Once this is done you have CodeCommit and ECR repositories up and running.

The next step is manual. You push initial files to the repository including Dockerfile. Then you build a base Docker image. I've decided to do this manually to keep things simple. Otherwise, I would have to pull from the Docker registry using AWS IP. Such Docker pull requests are often rate-limited and the CodeBuild stage fails. You can log in to the Docker with your credentials but this requires you to store your credentials in the SecretManager. To keep this project simple I've decided to build a base image locally. Then this image is inserted manually into Dockerfile. The CodeBuild is fetching images from AWS ECR only.

Once initial data is present in both repositories, you can deploy CICD stack. After this step, CodePipeline is executed on each next commit.

Testing

The Fargate cluster is created under the CICD stack. It is updated with a new image every time you push into CodeCommit repository. This process takes around 10 minutes. CDK provision Fargate cluster with public application load balancer. You can copy DNS name and test application from your side.

Lessons learned

1. AWS CDK heavy lifting

AWS CDK is doing a lot for you in the background. This can be misleading in some cases, sometimes following AWS documentation for resource setup is not required at all. For example, cross-account setup for CDK requires attaching an imported role from another account. CDK magically attached required policies and resolved CodeCommit repository ARN. Without that, you would end up with an unusable stack where AWS Console shows you the wrong account number in the CodeCommit ARN

=> Bookmark AWS CDK Github, join slack, search existing tickets and ask questions

2. AWS CDK pipelines library

AWS CDK list some issues i.e. links on the Console won't work in a cross-account scenario. You should be aware that it is a limitation, but the stack works perfectly fine.

=> Always check AWS CDK typescript documentation

3. SSO

Background token refresh on the web AWS Console is not consistently refreshed. You have to reload browser tab manually. Fortunately, you are not logged out.

Session duration is set to 4 hours, it is ok for regular work use cases.

=> Use SSO whenever possible regardless of small issues and requirement to refresh ~/.aws/credentials with sso.sh script

4. Cross account browser workspace

I found the best for my use case to use the same browser with one regular window and one in private mode.

=> Working with single browser allows accessing your private bookmarks. Don't trust browser extensions for manage isolated sessions, this is too risky.

5. ECR repository permission

Docker introduced rate-limiting for unauthorized pull requests. This affects code pipeline workflows, it is easy to get rate limited on Docker pull because requests are executed from AWS network. The suggested workaround is to configure Docker credentials and use them to log in. This requires managing secrets. I've decided to push my base images and use them for Dockerfile FROM clause. However, this resulted in a requirement to add to my policy ARN with /repository path (see the code).

=> Keep it simple for testing stacks.

6. AWS CDK source code

Checkout AWS CDK source code. It is implemented in a typescript language. It is useful for tracing more complex issues.

=> Check the source code before you ask the question.

Slawomir Stec