The number of Infrastructure as Code tools for provisioning public cloud resources has grown significantly over the last few years. What new tools do better than the old ones? What are they suitable for, what not? It is confusing even for cloud veterans, but definitely for those who have just started the journey. This article will not explain why IaC is used; my colleague Janne Kuha has already done it in his article about Google Cloud & Terraform. In turn, I aim to describe the ten-year path of tooling and some of those that are adding value to more specific areas, serverless microservices development or application development in general.
Declarative and imperative
Traditionally, Infrastructure as Code has been divided into declarative and imperative approaches. The declarative approach is about describing the eventual target; it defines the desired state of your resources. The declarative approach answers the question of what needs to be created - for example, I need two virtual machines. Imperative way responds to question how the infrastructure needs to be changed to achieve the goal, usually by a sequence of different commands. Ansible playbooks are an excellent example of an imperative approach.
A brief history of cloud infrastructure as code
“You have to know the past to understand the present.” ‑Carl Sagan
The foundation of IaC in the public clouds is these three cloud vendor-specific tools: AWS CloudFormation in AWS, Azure Resource Manager (ARM) in Azure, and Cloud Deployment Manager in GCP. These are YAML or JSON based, declarative tools and have been in cloud toolboxes for a long time. In the next month, February 2021, AWS CloudFormation will celebrate its 10th birthday! Azure Resource Manager was launched in April 2014 (at Build 2014 with the new Azure portal) and Cloud Deployment Manager in July 2015.
AWS S3 bucket example with AWS CloudFormation:
1AWSTemplateFormatVersion: 2010-09-09
2Resources:
3 S3Bucket:
4 Type: 'AWS::S3::Bucket'
5 DeletionPolicy: Retain
6 Properties:
7 BucketName: DOC-EXAMPLE-BUCKETWith the declarative approach, over time in practice, templates accumulate, and copy-pasting becomes more common. What if you have used the very same template or snippet in ten projects and you notice some improvement? Yes - the same fix is needed for each separately.
From third-party tools, Terraform by HashiCorp has been a popular cloud-agnostic choice. Terraform has also been there for a while. It was published in June 2014. It uses its domain-specific language HCL (Hashicorp Configuration Language), which allows a bit more complex business logic than the aforementioned tools. There is also a demonstration of use with GCP in Kuha’s excellent writing referenced at the beginning of the article.
Tools designed for Serverless Applications - the first wave
These classic declarative tools require a fair amount of markup code. Tools with shortcuts or “conventions over configuration” were developed to boost productivity and make distributed microservice applications seem more like a traditional monolithic application or a framework. These tools provide best-practice defaults and enable building and testing your serverless applications locally on your machine. The first and still prevalent tool was the cloud-agnostic Serverless Framework by Serverless Inc, released in October 2015. The deployment process of Serverless Framework is based on template generation so that native IaC templates are compiled during the deploy command.
Serverless Framework - AWS Lambda + S3 example:
1org: my-company
2app: my-app
3service: image-processor
4
5functions:
6 imageResizer:
7 runtime: node.js8.10
8 handler: index.imageResizer
9 event:
10 - s3:
11 bucket: images
12 event: s3:ObjectCreated:*Shortly after, in November 2016, AWS launched their own native Serverless Application Model or AWS SAM in November 2016. AWS SAM extends AWS CloudFormation with specific AWS::Serverless namespace types that are transformed to CloudFormation syntax. These types are shortcuts for defining standard serverless application components like AWS Lambda functions or API Gateway endpoints.
AWS SAM example:
1AWSTemplateFormatVersion: '2010-09-09'
2Transform: AWS::Serverless-2016-10-31
3Description: >
4 sam-app
5
6Parameters:
7 AppBucketName:
8 Type: String
9 Description: "REQUIRED: Unique S3 bucket name to use for the app."
10
11Resources:
12 S3JsonLoggerFunction:
13 Type: AWS::Serverless::Function
14 Properties:
15 Handler: src/handlers/s3-json-logger.s3JsonLoggerHandler
16 Runtime: nodejs12.x
17 MemorySize: 128
18 Timeout: 60
19 Policies:
20 S3ReadPolicy:
21 BucketName: !Ref AppBucketName
22 Events:
23 S3NewObjectEvent:
24 Type: S3
25 Properties:
26 Bucket: !Ref AppBucket
27 Events: s3:ObjectCreated:*
28 Filter:
29 S3Key:
30 Rules:
31 - Name: suffix
32 Value: ".json"
33 AppBucket:
34 Type: AWS::S3::Bucket
35 Properties:
36 BucketName: !Ref AppBucketNameNote the first two rows, SAM is a CloudFormation extension and transformed into CloudFormation. You can use CloudFormation types (row 34) or SAM specific Serverless types under AWS::Serverless namespace (row 13).
Serverless Infrastructure as programming language code - the second wave
Declarative language has always boundaries when you need to do more complex business logic than their parameters, conditions, mappings, and loops (Terraform only) allows. Sometimes, you might need to do some tricks by using external scripting, etc. A programming language could get you around these boundaries and limitations. This generation of tools generates the declarative markup code with the programming language code or bypasses it and utilizes cloud APIs. Tools with programming language support is a rising and trending approach at the moment.
An example of a generative tool is the AWS Cloud Development Kit or AWS CDK (Jul 2019). AWS CDK uses the term synthesizing instead of generation. Speakers on behalf of the template generation say it’ll bring benefits in both declarative and imperative approaches. Exciting additions to AWS CDK, an open-source framework, in AWS re:Invent 2020 were alpha releases of CDK for Terraform (cdk tf) and CDK for Kubernetes (cdk8s). The CDK for Terraform provides the CDK constructs for defining Terraform HCL files in TypeScript or Python and cdk8s for defining Kubernetes configuration in TypeScript, Python, or Java.
AWS CDK example:
1import * as cdk from '@aws-cdk/core';
2import * as s3 from '@aws-cdk/aws-s3';
3
4export class HelloCdkStack extends cdk.Stack {
5 constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
6 super(scope, id, props);
7
8 new s3.Bucket(this, 'MyFirstBucket', {
9 versioned: true
10 });
11 }
12}
Examples of non-generative cloud-agnostic tools in this programming language genre are Pulumi (Sep 2019) and Serverless Components by HashiCorp (Apr 2020).
Pulumi AWS S3 bucket example:
1import * as pulumi from "@pulumi/pulumi";
2import * as aws from "@pulumi/aws";
3import * as awsx from "@pulumi/awsx";
4
5// Create an AWS resource (S3 Bucket)
6const bucket = new aws.s3.Bucket("my-bucket");
7
8// Export the name of the bucket
9export const bucketName = bucket.id;
Serverless Components even states that it bypasses the restraints of AWS CloudFormation and builds reusable templates that are inspired by component-based React. The actual usage of Serverless Components is a YAML declaration.
Creation of a Serverless Component example:
1const { Component } = require('@serverless/core')
2
3class MyBlog extends Component {
4 async default(inputs) {
5 this.context.status('Deploying a serverless blog')
6 // Load a component
7 const website = await this.load('@serverless/website')
8 // Deploy it
9 const outputs = await.website({ code: { src: './blog-code' } })
10 this.state.url = outputs.url
11 return outputs
12 }
13}
14
15module.exports = MyBlog
Tools specialized in more narrow use
There are also plenty of others. AWS offers more native specialized tools than other vendors, even overlapping ones, for various infrastructure purposes. Good examples are AWS Amplify (Nov 2018) and AWS Copilot (Jul 2020). Both are wizard-like command-line interfaces to handle the infrastructure.
AWS Amplify offers a holistic model (in short, infrastructure code in the same repository as the application and DevOps pipelines for both) and its own Amplify admin web interface for developing web and mobile applications with their serverless services. For example, a ready registration and login process on top of AWS Cognito, including UI components and code examples to start with. AWS Amplify CLI generates the CloudFormation code. Previously it was in JSON, but nowadays YAML.
AWS Amplify CLI commands:
AWS Copilot is for launching and managing container applications on Amazon ECS and AWS Fargate. It has a way to export the CloudFormation template with the “copilot svc package” command. It is not mandatory, and you can use it in a more imperative style.
AWS Copilot commands:
The pros of purpose-built IaC tools are immediate benefits, but on the other hand, models and frameworks tend to bind hands. For example, in AWS Amplify, you don’t have much control over AWS CloudFront. I still have great personal experiences with AWS Amplify, both in native mobile applications and web applications.
Conclusion
To summarise, there is the traditional division into two approaches - declarative and imperative. Modern tooling with programming language support combines them, while some tools are bypassing the template generation. There are pros and cons in each, and I wouldn’t say that, for example, the CDK is always a better choice than Terraform, SAM, or AWS Amplify. It depends on the case and is also a bit of a matter of taste. However, the CDK approach is a step forward in IaC code reusability. It is an easier way to share solutions starting points both in house via code artifact repository and with the community with AWS Construct and AWS Solutions Construct libraries.
The programming style needs a different mindset and programming skills and can be a problematic extra layer without. Roughly the division could be programming language-based tools for software development to avoid context switching between markup language and programming language, declarative tools for others, and third-party cloud-agnostic tools for hybrid cloud use cases. Reality is not that black and white.
We need to remember that all these tools are continually evolving. For example, the ten-year-old CloudFormation got a nice new feature last year - AWS CloudFormation modules. Experiment with what suits you, your team, and the purpose the best. You can always mix and match and cherry-pick the best ones. The most important - stop clicking and start writing the infrastructure!
Mentioned tools sorted by GA release date:
- AWS CloudFormation (Feb 2011)
- Azure Resource Manager (Apr 2014)
- Terraform (Jun 2014)
- GCP Cloud Deployment Manager (Jul 2015)
- Serverless Framework (Oct 2015)
- AWS SAM (Nov 2016)
- AWS Amplify (Nov 2018)
- AWS CDK (Jul 2019)
- Pulumi (Sep 2019)
- Serverless Components (Apr 2020)
- AWS Copilot (Jul 2020)