At Dwolla, we believe infrastructure is code and we build tooling enabling our developers to treat it as such.

We’ve open-sourced several tools and libraries in the past, along with tips for custom CloudFormation resources. We also have an internal domain-specific language used to define the infrastructure requirements for each of the components that make up our Access and Transfer APIs.

Until recently, defining our public-facing services in DNS lagged behind the automation we had put in place elsewhere. Any time a new service came online, the responsible team had to coordinate with another team so they could make the necessary changes manually.

We use Cloudflare for DNS and DDoS protection and—like Dwolla—Cloudflare has an excellent API for managing your use of their products. Our team decided that we should stop making manual changes to our DNS settings and integrate Cloudflare’s API into our infrastructure automation.

Because we use AWS CloudFormation to manage our infrastructure, this meant creating a custom resource. We’ve had success writing custom resources in Scala, using our CloudFormation sbt plugin and abstract custom resource projects, so I started there.

The input, as a CloudFormation resource definition, looks something like this:

{
  "PublicDns": {
    "Type": "Custom::CloudflareDnsRecord",
    "Properties": {
      "Name": "example.dwolla.com",
      "Content": {
        "Fn::GetAtt": [
          "ElasticLoadBalancer",
          "DNSName"
        ]
      },
      "Proxied": false,
      "TTL": 1,
      "Type": "CNAME",
      "CloudflareEmail": "encrypted-email-address",
      "CloudflareKey": "encrypted-api-key",
      "ServiceToken": {
        "Fn::ImportValue": "CloudflarePublicHostnameLambda"
      }
    },
    "DependsOn": [
      "ElasticLoadBalancer"
    ]
  }
}

As you can see, it is a Custom::CloudflareDnsRecord resource with several properties defined:

  • Name defines the public hostname that should be published in DNS
  • Content defines the value to which the hostname should resolve; in this case, the DNS name of the Elastic Load Balancer handling the service’s traffic
  • Type defines the type of DNS record to create or update at Cloudflare
  • CloudflareEmail and CloudflareKey are Amazon KMS-encrypted values used to authenticate with the Cloudflare API
  • ServiceToken tells CloudFormation what Lambda Function or SNS Topic to use to service the custom resource
  • TTL is the time to live: the duration (in seconds) other DNS servers can safely cache the record value. One second indicates Cloudflare should automatically manage the TTL.
  • Proxied is a Cloudflare setting indicating whether traffic should flow directly to the origin server or be proxied via Cloudflare for DDoS protection

The custom resource also depends on ElasticLoadBalancer, a resource defined elsewhere in the CloudFormation template, to ensure that the ELB is active and healthy before updating anything in Cloudflare.

How to Use

The source code for the lambda is available on GitHub. You’ll need an S3 bucket where you can store code artifacts. Clone the repository, then set the s3Bucket and s3Key according to your needs, and run
sbt publish stack/deploy
to publish the code as a JAR on S3 and create a CloudFormation stack that defines the lambda.

The CloudFormation stack assumes two IAM roles exist:

  • DataEncrypter is a role that will be able to encrypt values using the KMS key assigned to the lambda. Use this role to encrypt your Cloudflare email and API access key using temporary AWS credentials:
aws sts assume-role --role-arn arn:aws:iam::123456789012:role/DataEncrypter --role-session-name "DataEncrypter" > /tmp/assume-role-output.txt
export AWS_ACCESS_KEY_ID=…       # values from /tmp/assume-role-output.txt
export AWS_SECRET_ACCESS_KEY=…
export AWS_SESSION_TOKEN=…
aws --region us-west-2 kms encrypt --key-id {key ID from CloudFormation} --plaintext {your Cloudflare email address}
aws --region us-west-2 kms encrypt --key-id {key ID from CloudFormation} --plaintext {your Cloudflare API key}
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN

 

  • The KMS commands will return encrypted values you can include in your CloudFormation templates as resource properties for the custom resource.
  • cloudformation/deployer/cloudformation-deployer is a role that we use to create and modify CloudFormation stacks; it has permission to do things like create Lambda functions or ECS task definitions. In this case, it will be granted access to do management tasks on the Lambda’s KMS key to avoid the default KMS key policy being applied.

Understanding the Implementation

The abstract custom resource requires the definition of a handleRequest: CloudFormationCustomResourceRequest ⇒ Future[HandlerResponse] method:

def handleRequest(input: CloudFormationCustomResourceRequest): Future[HandlerResponse] = {
  val resourceProperties = input.ResourceProperties.getOrElse(throw MissingResourceProperties)
  val dnsRecord = parseDtoFrom(input.PhysicalResourceId, resourceProperties)
  decryptCloudflareCredentials(resourceProperties).flatMap { implicit cloudflare ⇒
    input.RequestType.toUpperCase match {
      case "CREATE" | "UPDATE" ⇒ handleCreateOrUpdate(dnsRecord, input.PhysicalResourceId)
      case "DELETE" ⇒ handleDelete(dnsRecord, input.PhysicalResourceId.get)
    }
  }
}

 

This method first extracts the required data from the Properties section of the resource, decrypts the Cloudflare credentials, and then determines if the resource should be created, updated, or deleted.

private def handleCreateOrUpdate(dnsRecordDto: DnsRecordDTO, cloudformationProvidedPhysicalResourceId: Option[String])
                                  (implicit cloudflare: DnsRecordClient): Future[HandlerResponse] = {
  for {
    existingRecord ← cloudflare.getExistingDnsRecord(dnsRecordDto.name)
    updateableId = existingRecord.map(_.physicalResourceId)
    createOrUpdate ← updateableId.fold[Future[CreateOrUpdate[IdentifiedDnsRecord]]](cloudflare.createDnsRecord(dnsRecordDto).map(Create(_))) { physicalResourceId ⇒
      val newRecordState = dnsRecordDto.identifyAs(physicalResourceId)
      assertRecordTypeWillNotChange(existingRecord.get.recordType, newRecordState.recordType)
      cloudflare.updateDnsRecord(newRecordState).map(Update(_))
    }
  } yield {
    warnIfProvidedIdDoesNotMatchDiscoveredId(cloudformationProvidedPhysicalResourceId, updateableId, dnsRecordDto.name)
    warnIfNoIdWasProvidedButDnsRecordExisted(cloudformationProvidedPhysicalResourceId, existingRecord)
    val dnsRecord = createOrUpdate.value
    val data = Map(
      "dnsRecord" → dnsRecord,
      "created" → createOrUpdate.create,
      "updated" → createOrUpdate.update,
      "oldDnsRecord" → existingRecord
    )
    HandlerResponse(dnsRecord.physicalResourceId, data)
  }
}

Creating and Updating Resources

CloudFormation will tell us if, from its perspective, the custom resource is being created or updated, but we treat these cases as equivalent.

There are existing DNS records defined in Cloudflare—we want to create CloudFormation resources that start to manage those existing records without having to first remove them from Cloudflare. (Also, despite our best intentions, it will still be possible to make changes in Cloudflare directly, so the Lambda needs to handle that reality).

For this reason, handleCreateOrUpdate first checks to see if an existing DNS record exists for given fully-qualified name, and if so, obtains its Cloudflare record ID:

existingRecord ← cloudflare.getExistingDnsRecord(dnsRecordDto.name)
updateableId: Option[String] = existingRecord.map(_.physicalResourceId)

 

It then folds over the optional ID:

createOrUpdate ← updateableId.fold[Future[CreateOrUpdate[IdentifiedDnsRecord]]](cloudflare.createDnsRecord(dnsRecordDto).map(Create(_))) { physicalResourceId ⇒
  val newRecordState = dnsRecordDto.identifyAs(physicalResourceId)
  assertRecordTypeWillNotChange(existingRecord.get.recordType, newRecordState.recordType)
  cloudflare.updateDnsRecord(newRecordState).map(Update(_))
}

 

That’s a lot to unpack!

First, createOrUpdate will be type CreateOrUpdate[IdentifiedDnsRecord], which is a union type of either a created or updated IdentifiedDnsRecord. We specify the type at the beginning of the fold so we don’t have to cast Create(_)CreateOrUpdate[IdentifiedDnsRecord]—if we skipped both, createOrUpdate would take the type Create[IdentifiedDnsRecord] instead, and the second part of the fold wouldn’t compile.

If updateableId is None, the record will be created via cloudflare.createDnsRecord.

If updateableId is Some, the method builds a new DTO to use for updating, assert that its type won’t change, and then update the record. (Right now, Dwolla doesn’t have a need for changing record types once they’ve been defined. If the need arises, we will modify this code to add the input validation necessary to support changes.)
At this point, createOrUpdate will contain the record that was either created or updated.

The method warns us if the record’s physical ID differs from the one that was passed in, or if no physical ID was provided but the fully-qualified name already exists. These situations indicate differences between the CloudFormation stack’s description of the world and the world as it actually is. In either case, it’s fine, because the end-state is in sync, but it’s good to have a log of what happened.

Finally, CloudFormation allows us to return arbitrary data, so the method outputs the full record, whether it was created or updated, and any existing record that existed prior to the changes.

Both the cloudflare.createDnsRecord and cloudflare.updateDnsRecord methods return Future[IdentifiedDnsRecord], with IdentifiedDnsRecord.physicalResourceId set to the API URL of the Cloudflare record (e.g. https://api.cloudflare.com/client/v4/zones/{zone-id}/dns_records/{record-id}), so the Physical Resource ID of the CloudFormation response will be the Cloudflare API URL.

Deleting Resources

Deleting records is much simpler, thanks in part to a design assumption that only resources actually managed by CloudFormation should be deleted.

This means a delete request with a hostname that actually exists (e.g. www.dwolla.com) will not be deleted unless it comes with a valid Physical Resource ID.

private def handleDelete(physicalResourceId: String)
                          (implicit cloudflare: DnsRecordClient): Future[HandlerResponse] = {
  for {
    deleted ← cloudflare.deleteDnsRecord(physicalResourceId)
  } yield {
    val data = Map(
      "deletedRecordId" → deleted
    )
    HandlerResponse(physicalResourceId, data)
  }
}.recover {
  case ex: DnsRecordIdDoesNotExistException ⇒
    logger.error("The record could not be deleted because it did not exist; nonetheless, responding with Success!", ex)
    HandlerResponse(physicalResourceId, Map.empty[String, AnyRef])
}

 

If the record could not be deleted because it did not exist, the method returns success to CloudFormation anyway, because from CloudFormation’s perspective, the record being gone is the desired outcome.

Next Steps

The integration between CloudFormation and Cloudflare allows our team to quickly roll out new public-facing properties with very low overhead and ensures that the configuration of those properties stays up to date as the components backing them change over time.

We coordinate multiple vendors with configuration and code that is ultimately checked into version control and managed and controlled using our normal pull request process.

For many of our properties, there is another wrinkle: the addition of Distil Networks’ bot detection service. Stay tuned for a future blog post describing the integration of CloudFormation, Distil, and an Akka actor system that manages that integration and waits for activation of DNS and nginx resources at Distil before returning success to CloudFormation.

If this sounds like the kind of problem you’d be interested in helping us solve, Dwolla is hiring in Des Moines, Iowa! Please reach out!

Follow Brian on Twitter: @bpholt

Talk With Our Integration Experts

Contact Sales