By: Brian Holt,

At Dwolla, we believe infrastructure is code, and we build tooling enabling our developers to treat it as such. We’ve open-sourced several tools and libraries in the past, along with tips for custom CloudFormation resources. We also have an internal domain-specific language used to define the infrastructure requirements for each of the components that make up our Access and Transfer APIs.

Until recently, defining our public-facing services in DNS lagged behind the automation we had put in place elsewhere. Any time a new service came online, the responsible team had to coordinate with another team so they could make the necessary changes manually.

We use Cloudflare for DNS and DDoS protection and—like Dwolla—Cloudflare has an excellent API for managing your use of their products. Our team decided that we should stop making manual changes to our DNS settings and integrate Cloudflare’s API into our infrastructure automation.

Because we use AWS CloudFormation to manage our infrastructure, this meant creating a custom resource. We’ve had success writing custom resources in Scala, using our CloudFormation sbt plugin and abstract custom resource projects, so I started there.

The input, as a CloudFormation resource definition, looks something like this:

{
  "PublicDns": {
    "Type": "Custom::CloudflareDnsRecord",
    "Properties": {
      "Name": "example.dwolla.com",
      "Content": {
        "Fn::GetAtt": [
          "ElasticLoadBalancer",
          "DNSName"
        ]
      },
      "Proxied": false,
      "TTL": 1,
      "Type": "CNAME",
      "CloudflareEmail": "encrypted-email-address",
      "CloudflareKey": "encrypted-api-key",
      "ServiceToken": {
        "Fn::ImportValue": "CloudflarePublicHostnameLambda"
      }
    },
    "DependsOn": [
      "ElasticLoadBalancer"
    ]
  }
}

As you can see, it is a Custom::CloudflareDnsRecord resource with several properties defined:

The custom resource also depends on ElasticLoadBalancer, a resource defined elsewhere in the CloudFormation template, to ensure that the ELB is active and healthy before updating anything in Cloudflare.

How to Use

The source code for the lambda is available on GitHub. You’ll need an S3 bucket where you can store code artifacts. Clone the repository, then set the s3Bucket and s3Key according to your needs, and run

sbt publish stack/deploy

to publish the code as a JAR on S3 and create a CloudFormation stack that defines the lambda.

The CloudFormation stack assumes two IAM roles exist:

aws sts assume-role --role-arn arn:aws:iam::123456789012:role/DataEncrypter --role-session-name "DataEncrypter" > /tmp/assume-role-output.txt
export AWS_ACCESS_KEY_ID=…       # values from /tmp/assume-role-output.txt
export AWS_SECRET_ACCESS_KEY=…
export AWS_SESSION_TOKEN=…
 
aws --region us-west-2 kms encrypt --key-id {key ID from CloudFormation} --plaintext {your Cloudflare email address}
aws --region us-west-2 kms encrypt --key-id {key ID from CloudFormation} --plaintext {your Cloudflare API key}
 
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN

Understanding the Implementation

The abstract custom resource requires the definition of a handleRequest: CloudFormationCustomResourceRequest ⇒ Future[HandlerResponse] method:

def handleRequest(input: CloudFormationCustomResourceRequest): Future[HandlerResponse] = {
  val resourceProperties = input.ResourceProperties.getOrElse(throw MissingResourceProperties)
  val dnsRecord = parseDtoFrom(input.PhysicalResourceId, resourceProperties)

  decryptCloudflareCredentials(resourceProperties).flatMap { implicit cloudflare 
    input.RequestType.toUpperCase match {
      case "CREATE" | "UPDATE"  handleCreateOrUpdate(dnsRecord, input.PhysicalResourceId)
      case "DELETE"  handleDelete(dnsRecord, input.PhysicalResourceId.get)
    }
  }
}

This method first extracts the required data from the Properties section of the resource, decrypts the Cloudflare credentials, and then determines if the resource should be created, updated, or deleted.

private def handleCreateOrUpdate(dnsRecordDto: DnsRecordDTO, cloudformationProvidedPhysicalResourceId: Option[String])
                                  (implicit cloudflare: DnsRecordClient): Future[HandlerResponse] = {

  for {
    existingRecord  cloudflare.getExistingDnsRecord(dnsRecordDto.name)
    updateableId = existingRecord.map(_.physicalResourceId)
    createOrUpdate  updateableId.fold[Future[CreateOrUpdate[IdentifiedDnsRecord]]](cloudflare.createDnsRecord(dnsRecordDto).map(Create(_))) { physicalResourceId 
      val newRecordState = dnsRecordDto.identifyAs(physicalResourceId)
      assertRecordTypeWillNotChange(existingRecord.get.recordType, newRecordState.recordType)
      cloudflare.updateDnsRecord(newRecordState).map(Update(_))
    }
  } yield {
    warnIfProvidedIdDoesNotMatchDiscoveredId(cloudformationProvidedPhysicalResourceId, updateableId, dnsRecordDto.name)
    warnIfNoIdWasProvidedButDnsRecordExisted(cloudformationProvidedPhysicalResourceId, existingRecord)

    val dnsRecord = createOrUpdate.value
    val data = Map(
      "dnsRecord"  dnsRecord,
      "created"  createOrUpdate.create,
      "updated"  createOrUpdate.update,
      "oldDnsRecord"  existingRecord
    )

    HandlerResponse(dnsRecord.physicalResourceId, data)
  }
}

Creating and Updating Resources

CloudFormation will tell us if, from its perspective, the custom resource is being created or updated, but we treat these cases as equivalent. There are existing DNS records defined in Cloudflare—we want to create CloudFormation resources that start to manage those existing records without having to first remove them from Cloudflare. Also, despite our best intentions, it will still be possible to make changes in Cloudflare directly, so the Lambda needs to handle that reality!

For this reason, handleCreateOrUpdate first checks to see if an existing DNS record exists for given fully-qualified name, and if so, obtains its Cloudflare record ID:

existingRecord  cloudflare.getExistingDnsRecord(dnsRecordDto.name)
updateableId: Option[String] = existingRecord.map(_.physicalResourceId)

It then folds over the optional ID:

createOrUpdate  updateableId.fold[Future[CreateOrUpdate[IdentifiedDnsRecord]]](cloudflare.createDnsRecord(dnsRecordDto).map(Create(_))) { physicalResourceId 
  val newRecordState = dnsRecordDto.identifyAs(physicalResourceId)
  assertRecordTypeWillNotChange(existingRecord.get.recordType, newRecordState.recordType)
  cloudflare.updateDnsRecord(newRecordState).map(Update(_))
}

That’s a lot to unpack!

First, createOrUpdate will be type CreateOrUpdate[IdentifiedDnsRecord], which is a union type of either a created or updated IdentifiedDnsRecord. We specify the type at the beginning of the fold so we don’t have to cast Create(_)CreateOrUpdate[IdentifiedDnsRecord]—if we skipped both, createOrUpdate would take the type Create[IdentifiedDnsRecord] instead, and the second part of the fold wouldn’t compile.

If updateableId is None, the record will be created via cloudflare.createDnsRecord.

If updateableId is Some, the method builds a new DTO to use for updating, assert that its type won’t change, and then update the record. (Right now, Dwolla doesn’t have a need for changing record types once they’ve been defined. If the need arises, we will modify this code to add the input validation necessary to support changes.)

At this point, createOrUpdate will contain the record that was either created or updated. The method warns us if the record’s physical ID differs from the one that was passed in, or if no physical ID was provided but the fully-qualified name already exists. These situations indicate differences between the CloudFormation stack’s description of the world and the world as it actually is. In either case, it’s fine, because the end-state is in sync, but it’s good to have a log of what happened.

Finally, CloudFormation allows us to return arbitrary data, so the method outputs the full record, whether it was created or updated, and any existing record that existed prior to the changes.

Both the cloudflare.createDnsRecord and cloudflare.updateDnsRecord methods return Future[IdentifiedDnsRecord], with IdentifiedDnsRecord.physicalResourceId set to the API URL of the Cloudflare record (e.g. https://api.cloudflare.com/client/v4/zones/{zone-id}/dns_records/{record-id}), so the Physical Resource ID of the CloudFormation response will be the Cloudflare API URL.

Deleting Resources

Deleting records is much simpler, thanks in part to a design assumption that only resources actually managed by CloudFormation should be deleted. This means a delete request with a hostname that actually exists (e.g. www.dwolla.com) will not be deleted unless it comes with a valid Physical Resource ID.

private def handleDelete(physicalResourceId: String)
                          (implicit cloudflare: DnsRecordClient): Future[HandlerResponse] = {
  for {
    deleted  cloudflare.deleteDnsRecord(physicalResourceId)
  } yield {
    val data = Map(
      "deletedRecordId"  deleted
    )

    HandlerResponse(physicalResourceId, data)
  }
}.recover {
  case ex: DnsRecordIdDoesNotExistException 
    logger.error("The record could not be deleted because it did not exist; nonetheless, responding with Success!", ex)
    HandlerResponse(physicalResourceId, Map.empty[String, AnyRef])
}

If the record could not be deleted because it did not exist, the method returns success to CloudFormation anyway, because from CloudFormation’s perspective, the record being gone is the desired outcome.

Next Steps

The integration between CloudFormation and Cloudflare allows our team to quickly roll out new public-facing properties with very low overhead and ensures that the configuration of those properties stays up to date as the components backing them change over time. We coordinate multiple vendors with configuration and code that is ultimately checked into version control and managed and controlled using our normal pull request process.

For many of our properties, there is another wrinkle: the addition of Distil Networks’ bot detection service. Stay tuned for a future blog post describing the integration of CloudFormation, Distil, and an Akka actor system that manages that integration and waits for activation of DNS and nginx resources at Distil before returning success to CloudFormation.

If this sounds like the kind of problem you’d be interested in helping us solve, Dwolla is hiring in Des Moines, Iowa! Please reach out!

Follow Brian on Twitter: @bpholt

Get started with ACH transfers

We'll help you design your ideal payments experience.

Loading...

Thanks!

We’ve sent you a message to kick off the conversation with our team. Please check your inbox.

Sorry

There was an error and the form was not submitted.

Financial institutions play an important role in the Dwolla network.

Dwolla, Inc. is an agent of Veridian Credit Union and Compass Bank and all funds associated with your account in the Dwolla network are held in pooled accounts at Veridian Credit Union and Compass Bank. These funds are not eligible for individual insurance, including FDIC insurance and may not be eligible for share insurance by the National Credit Union Share Insurance Fund. Dwolla, Inc. is the operator of a software platform that communicates user instructions for funds transfers to Veridian Credit Union and Compass Bank.