Reuse CodeBuild projects across different CodePipeline

AWS CodeBuild is a fully managed service for continuous integration that compiles code, executes tests and generates artefacts ready to be deployed to your environments. Since it is a managed service there is no need to provision or operate servers as this is taken care of entirely by AWS and you pay only for what you use.

AWS CodePipeline is a continuous delivery service that manages application and infrastructure code releases in a rapid and consistent manner. CodePipeline stitches together source control stages with services like AWS CodeCommit or GitHub, build stages with AWS CodeBuild and code with AWS CodeDeploy or AWS CloudFormation. AWS CodePipeline is also a fully managed service, which means no server provisioning or operating, and you pay only for what you use.

How Codepipeline and Codebuild work together

At a high level, CodePipeline acts as the orchestrator, lining up stages and passing artefacts as inputs and outputs between them. Artefacts are passed between stages with the help of an S3 bucket where the zip files are written and read from.

In this sample diagram CodePipeline polls a GitHub repo with our application code. Once it detects changes it pulls the code, zips it, pushes it to S3 and passes to CodeBuild a reference to the artefact. CodeBuild downloads the zip file from S3 and runs the build project. Say we are writing a Java app, the output would be a jar file ready to be deployed that, again, zipped and pushed to S3. CodePipeline takes control back and passes to CodeDeploy a reference to the artefact in S3. CodeDeploy downloads the artefact and proceeds to deploy as required to the compute target.

When it comes to CodePipeline and CodeBuild, the only information you could pass was the artefact. CodeBuild has Environment Variables that could not be set from CodePipeline. That made any given pipeline and build project tightly coupled, since the environment variables would have the same values even if the project was used from different pipelines.

A good example could be building an artefact, then run the same automated tests on that artefact with a different database connection string for different environments. The connection string could be stored in a different SSM Parameter Store parameter per environment and the CodeBuild integration with SSM Parameter Store would facilitate fetching the value for you. This could all be abstracted in a way that the buildspec doesn’t have a reference to the environment variable and what parameter store parameter stores the value. This was not possible though, since the Pipeline couldn’t set the Environment Variables when interacting with CodeBuild. The way to solve that was to create a separate CodeBuild project per pipeline, and ‘hard code’ the environment variables to the right parameter store parameter per environment,

or add the environment variables declaration in the builspec.yml file and move that to the git repo together with the application code. The problem with that is that you would have to manage different buildspec.yml files per environment (branch) and the reference to the SSM Parameter Store parameters are visible to the developer. Potentially those could be changed and pull values from a different environment.

We can easily see that none of these are ideal.

What’s new

On October the 14th 2019, AWS announced that CodePipeline can now set environment variables on CodeBuild jobs.

What this means for the example we went through earlier is that we can now share the CodeBuild project with 2 or more Pipelines, and let the Pipelines set the Environment Variables as required. CodeBuild will take it from there, using the Environment Variables as usual, with no change to the project.

e can now separate those environment variables declarations and leave that responsibility to each of the Pipelines and leave the actual buildspec.yml file the instructions to run your tests and builds without exposing where the environment variables values come from.

Let’s see it in action

We are going to deploy some resources to test this new feature. You can clone this repo and provision the resources yourself using CloudFormation and Sceptre.

Here is what we are going to build:

• 2 SSM Parameter Store Parameters, 1 per environment. Since CloudFormation does not support the SecureString type we will use the aws cli for this.
• 1 CodeBuild project with no Environment Variables or buildspec. The Environment Variables will come from the pipelines, and the buildspec from the git repo.
• 2 pipelines, 1 per environment. Both are identical, only difference will be the reference to the SSM Parameter Store parameter.

What we are looking for is simple. The CodeBuild project won’t do anything real. All it will do is print out in the logs the values for those plain text variables and SSM Parameter Store parameters so that we can see that each build run, depending on the Pipeline that triggered it, uses different values. A very simple test.

Both pipelines have completed successfully, and if we check the details for each build action, we confirm that, while it is the same build project, the Environment Variables are different for each build run.

Looking at the logs we confirm that the retrieved values from SSM Parameter Store are different as expected. Pipeline 1 retrieves some_secret_for_pipeline_1 from Parameter Store, which we populated via the aws cli at the start of the demo, and Pipeline 2 retrieves some_secret_for_pipeline_2.

Conclusion

Being able to set environment variables in CodePipeline is a nice new feature that can facilitate a few use cases in your CI/CD pipelines, reusing the same CodeBuild project across many pipelines.

As we saw in this post, previously there were some cases when we could end up provisioning copies of CodeBuild projects with each pipeline. Even in an infrastructure as code world this was not ideal, since we would have to ensure changes were made across all projects.

Now you can provision one CodeBuild project, update it separately from the Pipelines that make use of it, and keep consistency across all those pipelines.

This article was written by Jesus Rodriguez, Cloud Solutions Architect at AC3. Jesus has over 8 years of experience in IT, Certified AWS Solutions Architect Professional, Certified AWS DevOps Professional, and Certified AWS Big Data - Specialty, with a strong focus on implementing, deploying and provisioning secure, high available, scalable and cost optimized applications utilizing the best AWS services for each use case.