By: Chris K,

If you’re running infrastructure on AWS, CloudFormation can be a great provisioning option for versioned, software-defined infrastructure. Whether you’re using bare templates or a wrapper like Troposphere or Sparkleformation, you can take advantage of several useful features like changesets, Resource-Dependency definitions, etc.

However, one of the biggest complaints about CloudFormation is the lag time between new features being added to AWS products and those features being supported in CloudFormation. Here at Dwolla, we use CloudFormation to provision infrastructure that powers our bank transfer API, and we’ve seen this problem first hand. Harnessing the power of CloudFormation custom resources allows you to trick CloudFormation into provisioning resources it doesn’t even know how to provision, using only your provisioning infrastructure.

CloudFormation allows for defining custom resources in templates. In a template, that looks like this:

"MyCustomResource": {
   "Properties": {
       "FirstResourceProperty": {
           "Ref": "MyFirstResourceProperty"
       },
       "ServiceToken": "MyProvisionerLambdaFunctionArn",
       "SecondResourceProperty": "My SecondResourceProperty"
   },
   "Type": "Custom::MyUnsupportedAWSFeature"
}

Troposphere, of course, provides an object wrapper for custom resources as well. Here’s what that looks like:

This lets us provide the location of a provisioner that knows how to provision the custom resource. This provisioner is typically an AWS Lambda function. Since AWS Lambda supports all of the AWS SDKs and support for new features and configurations is usually added to the SDKs/APIs right away, we can make use of the APIs running in Lambda to provision our infrastructure as part of our CloudFormation stack.

A Lambda that provisions CloudFormation custom resources has to have create, update, and delete methods. Here’s a skeleton of a simple Node.js provisioner:

var response = require('cfn-response');

exports.handler = function(event, context, callback) {
   var ResourceProperties = event.ResourceProperties;
   if (event.ResourceType == 'Create') {
       //API call for Create
   } else if (event.ResourceType == 'Update') {
       //API call for Update
   } else if (event.ResourceType == 'Delete') {
       //API call for Delete
   }
   response.send(event, context, response.SUCCESS, responseData);
};

When Lambdas first came to CloudFormation, you could provision a Lambda with CloudFormation, but not an Events rule that would act as a CRON scheduler for your Lambda.

dwolla-developers-xzibit-lambda

Dwolla wanted to provision scheduled Lambdas via CloudFormation, and we were able to utilize this method by creating the CloudWatch Events using a Lambda that was deployed in the same stack. This secondary “bootstrap” Lambda function was a simple Node.js function that makes an API call to create the CloudWatch Event Rule.

Fortunately, Events are now supported in CloudFormation, but new features are added to AWS frequently, and CloudFormation support tends to lag behind.

One current example of this problem is with S3 Transfer Acceleration. Transfer Acceleration is an S3 bucket configuration option that can significantly increase upload speeds by utilizing Amazon’s edge network. Unfortunately, if you’re provisioning your S3 buckets via CloudFormation, this feature is still not supported.

A simple API call to configure the Transfer Acceleration on an S3 bucket looks like this in Node:

var AWS = require('aws-sdk');
var s3 = new AWS.S3();
var params = {
   Bucket: ‘myacceleratedbucket’,
   AccelerateConfiguration: {Status: ‘Enabled’}
};
s3.putBucketAccelerateConfiguration(params);

That’s pretty easy, and is pretty straightforward to run in AWS Lambda.

To accomplish this, the stack containing our new S3 bucket needs to know where our Lambda custom provisioner lives. It also needs to pass along to Lambda the name of the bucket to update, as well as whether we want to enable or suspend Transfer Acceleration. Our custom object definition looks like this in Troposphere:

Custom resources allow passing along any parameters you define in CloudFormation to your provisioner. Since we’re creating the Acceleration Configuration in the same stack as the bucket being created, we can simply pass a “Ref” to the new S3 bucket into our custom provisioner:

Since we’re using Node.js in this example, we could include the code for our Lambda function inline in the CloudFormation resource for the Bootstrap Lambda, and provision the provisioner as part of the same stack.

However, I’d recommend using a separate stack when you create the stack, and CloudFormation will provision resources in the order needed for dependant resources. In this case, that order would be: S3 Bucket -> Lambda Function -> Custom Resource.

When you delete a stack, CloudFormation will also try to work out the necessary order to delete resources. Ideally, that would be Custom Resource -> Lambda Function -> S3 Bucket. In more complex situations where more References are passed around between resources, you could find the Lambda Function being deleted before the custom resource it is responsible for deprovisioning.

Instead, you can provision this custom provisioner Lambda function as a separate stack. This gives the added benefit of reusability, without proliferating a bunch of Lambda Functions that only ever provision a single stack. Once you’ve provisioned the stand-alone provisioner Lambda function, you can just use the ARN of that Lambda as the service token for your “custom resource”.

The advantage of this method over just doing a separate API call after provisioning your stack is that the resources created or modified with an API call are done as part of the stack, and you can actually rollback stack provisioning if the API calls fail.

Additionally, you can automatically clean up resources as part of your infrastructure provisioning. You don’t have to perform additional API calls to look up attributes of resources that you just provisioned—simply pass those in as references in your Cloudformation template. Once support for the new feature is eventually added to CloudFormation, switching to the native resource is easy. Simply swap out your custom resource with the native CloudFormation resource. Finally, you only pay for execution time for Lambdas, so using them for provisioning functions often falls within the AWS Lambda  free pricing tier.

If you find yourself frequently deploying custom resources, it would be trivial to extend your provisioning function to handle provisioning several different types of custom resources. Your Resource Type is passed along in the event CloudFormation sends to AWS Lambda, so you can just check the ResourceType attribute of the event, and call the appropriate provisioning method.

You can find a full demo for provisioning the Lambda function described in this demo on GitHub.

 

Image source
Financial institutions play an important role in the Dwolla network.

Dwolla, Inc. is an agent of Veridian Credit Union and Compass Bank and all funds associated with your account in the Dwolla network are held in pooled accounts at Veridian Credit Union and Compass Bank. These funds are not eligible for individual insurance, including FDIC insurance and may not be eligible for share insurance by the National Credit Union Share Insurance Fund. Dwolla, Inc. is the operator of a software platform that communicates user instructions for funds transfers to Veridian Credit Union and Compass Bank.