Slawomir Stec

User profile with file upload feature

This is a post about the new functionality added to the sample application. It is a user settings page with an option to upload a profile picture. Every new account gets assigned a default picture. After the login, the user can see it on the right side of the navigation bar. In the utils submenu, user can access a form to change his name and a file upload to update his picture. That is a description of the feature from the user's perspective.

My motivation to implement this feature was the lack of S3 integration in the sample application. This is probably one of the most important services in AWS, I wanted to add it to the CDK provisioning process and integrate with the UI.

What I quickly realized was that I shouldn't process upload through the Lambda function. My Lambda sits behind API Gateway, and there is a limit for the ApiGateway payload size set to 10 MB. I wanted to reuse the code to upload larger files in the future. And I already know that there is a cheaper option described below.

The solution for this problem was uploading to S3 directly. AWS optimized file upload to S3 buckets and Lambda execution is not required for it. That is why I mentioned there exists a cheaper solution.

Files stored in the S3 buckets are private by default. For file upload, there is a solution to generate a pre-signed URL to the S3 bucket and allow any user to write to it. It is really *any* user. In my case, it is restricted to the authenticated user of the sample application. Pre-signed URL has defined timeout value (i.e. 5 minutes) and after that becomes invalid.

This was what I needed but I didn't want to expose the S3 bucket to the public and allow reads for all users. I decided to add Cloudflare CDN to the mix. I allow all authenticated users to do the uploads which are bucket write action. However, I restricted users to forbid them to see other user's uploads. Additionally, I wanted to have images served by Cloudflare CDN to save some money here.

I succeed almost in all cases except Cloudflare CDN assets protection, more details below.

Cloudflare setup

All new accounts will get assigned a default profile picture. URL looks like below:

https://img.slawomirstec.com/default_assets/default_profile_image.png

I created single S3 bucket with name: img.slawomirstec.com and copied default image /default_assets/default_profile_image.png there. Bucket and default image deployment is provisioned by CDK, I automatically create it during infrastructure setup. However I don't serve images from a bucket directly, I don't serve it via AWS Cloudfront either. I decided to use the Cloudflare CDN.

Cloudflare cache validity is set to 24 hours. I configured S3 to allow reads from Cloudflare IP addresses only. To have this setup working I had to change my domain DNS and add CNAME record

CNAME: img
value: img.slawomirstec.com.s3.{my-region}.amazonaws.com

where my-region is replaced with the region where I host the app.

As a result, the content of my S3 bucket is served via CDN. This is pretty simple. Images are served via CDN, S3 read can be done by Cloudflare only, and it is a cached read, this saves me some money and I don't have to setup Cloudfront on the AWS side. Bucket writes are allowed for all authenticated users and performed directly on the S3 bucket pre-signed URL.

S3 CDK setup

I started with provisioning the S3 bucket. Bucket name must match CNAME set on Cloudflare side, in this case img.slawomirstec.com

I construct this dynamically using environment variables because I support different domains as deployment targets. The subdomain is configurable and by default is equal to the string 'img'.

The next step is the creation of the bucket itself. I set it as non-public and delete it when the infrastructure is destroyed. Additionally, I set CORS rules. When you compare it with the creation of the bucket using AWS Console, this is very similar, here I can do in the code directly. I found one issue with CDK and S3, it is not possible to delete a non-empty bucket. I had to do it from AWS CLI, luckily I do it already with PyAWS CLI so this was not a big issue to add this step.

After that, I set a policy to allow reads only for Cloudflare IP addresses. This is set to AnyPrincipal() that access my configured image domain. Effectively bucket becomes public on the S3 Console! AWS documentation states that public buckets are not recommended, they should be accompanied by access restriction condition. I defined the condition for CDN access and for s3:GetObject action only.

The last step copies default images and deploys them into a newly created bucket.

I did one more change that is not visible in the code below. I had to assign AWS managed policy 'AmazonS3FullAccess' to the Lambda role. It was required for the generation of pre-signed URLs from Lambda. I could not find why I needed to assign full s3 access to make it work, it worked locally with Chalice. I guess this is boto3 specific. When boto3 was executed locally, AWS credentials were used from the user directory. On AWS premise this policy was required.

Client changes

On the client side, I implemented two functions. First one getSignedUploadUrl, calls the backend to request a pre-signed URL. In the response, I receive an upload object with a URL to call (POST) and fields that I have to include in the form data. Second function uploadS3Post post form data directly to the S3 URL.

Model:

My fronted stack consists of Vue+TypeScript and Nuxt framework. I converted fields received with the pre-signed URL to form data object. In Typescript it can be easily done using DOM API and File/FormData interfaces.

Backend

Upload service autogenerates path for each stored profile picture. I stored year/month/day/hour in the path for easier debugging. The filename gets a random UUID name assigned.

For the pre-signed URL, I defined the maximum size limit to 1MB. Timeout is set to 60 seconds and the bucket name is generated from environment variables.

As expected, there is no code required for S3 upload.

Cloudflare token authentication

I wanted to implement Cloudflare token authentication as described here:

but I could not get it in a Cloudflare free plan that I use. I wanted to use it to ensure that users can't see other user's uploads. For time being, I had to rely on UUID and HTTPS protocol to hide this information. Cloudflare token authentication looks like a great addition, by defining shared secret on Cloudflare and AWS side, I can create a token that is validated on each GET request. It may look like nothing special but please note that in this scenario, I don't have to run any Lambda function to validate user JWT or implement new Lambda on the Cloudfront Edge to check access rights. Anyway, I will check this in the future and configure it just to play around with it, documentation is very good so it should not take much time to do it.