Skip to content

Instantly share code, notes, and snippets.

@afloesch
Last active October 15, 2024 22:20
Show Gist options
  • Save afloesch/dc7d8865eeb91100648330a46967be25 to your computer and use it in GitHub Desktop.
Save afloesch/dc7d8865eeb91100648330a46967be25 to your computer and use it in GitHub Desktop.

Terraforming API Gateway to SQS queue

Example of a bare-minimum terraform script to setup an API Gateway endpoint that takes records and puts them into an SQS queue.

SQS

Start by creating the SQS queue.

resource "aws_sqs_queue" "queue" {
  name                      = "my-sqs-queue"
  delay_seconds             = 0              // how long to delay delivery of records
  max_message_size          = 262144         // = 256KiB, which is the limit set by AWS
  message_retention_seconds = 86400          // = 1 day in seconds
  receive_wait_time_seconds = 10             // how long to wait for a record to stream in when ReceiveMessage is called
}

IAM

Now we need to define an IAM role so API Gateway has the necessary permissions to SendMessage to our SQS queue. These perms will give API Gateway the ability to create and write to CloudWatch logs, as well as the ability to put, read, and list data to the SQS queue.

resource "aws_iam_role" "api" {
  name = "my-api-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "apigateway.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_policy" "api" {
  name = "my-api-perms"

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:PutLogEvents",
          "logs:GetLogEvents",
          "logs:FilterLogEvents"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "sqs:GetQueueUrl",
          "sqs:ChangeMessageVisibility",
          "sqs:ListDeadLetterSourceQueues",
          "sqs:SendMessageBatch",
          "sqs:PurgeQueue",
          "sqs:ReceiveMessage",
          "sqs:SendMessage",
          "sqs:GetQueueAttributes",
          "sqs:CreateQueue",
          "sqs:ListQueueTags",
          "sqs:ChangeMessageVisibilityBatch",
          "sqs:SetQueueAttributes"
        ],
        "Resource": "${aws_sqs_queue.queue.arn}"
      },
      {
        "Effect": "Allow",
        "Action": "sqs:ListQueues",
        "Resource": "*"
      }      
    ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "api" {
  role       = "${aws_iam_role.api.name}"
  policy_arn = "${aws_iam_policy.api.arn}"
}

API Gateway

Now define an API Gateway REST API. We will host the POST method at the root / of the API.

resource "aws_api_gateway_rest_api" "api" {
  name        = "my-sqs-api"
  description = "POST records to SQS queue"
}

Create a validation template to validate the POST request body. You can leave this out, but it comes in handy for transforming and customizing service responses, in addition to validating the actual request payload.

This template defines our payload should be of Content-Type: application/json and the json needs to contain an object with two keys; id, and docs, and docs should be an array of objects greater than or equal to 1. See the AWS docs on creating request validators with OpenAPI.

resource "aws_api_gateway_request_validator" "api" {
  rest_api_id           = "${aws_api_gateway_rest_api.api.id}"
  name                  = "payload-validator"
  validate_request_body = true
}

resource "aws_api_gateway_model" "api" {
  rest_api_id  = "${aws_api_gateway_rest_api.api.id}"
  name         = "PayloadValidator"
  description  = "validate the json body content conforms to the below spec"
  content_type = "application/json"

  schema = <<EOF
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "required": [ "id", "docs"],
  "properties": {
    "id": { "type": "string" },
    "docs": {
      "minItems": 1,
      "type": "array",
      "items": {
        "type": "object"
      }
    }
  }
}
EOF
}

Create an API Gateway POST method for the API. The request does not require the request_models or request_validator_id key, but as mentioned above it's useful for both validating the message body format and formatting our service responses, which will be demonstrated shortly.

resource "aws_api_gateway_method" "api" {
  rest_api_id          = "${aws_api_gateway_rest_api.api.id}"
  resource_id          = "${aws_api_gateway_rest_api.api.root_resource_id}"
  api_key_required     = false
  http_method          = "POST"
  authorization        = "NONE"
  request_validator_id = "${aws_api_gateway_request_validator.api.id}"

  request_models = {
    "application/json" = "${aws_api_gateway_model.api.name}"
  }
}

Now we can define the API gateway integration that will forward records into the SQS queue. The request_templates defined is required in order to take the message body content and send it on to S3 (assuming the content type to be handled is json), as well as the request_parameters defined, since the SQS endpoint requires the data to be form encoded.

resource "aws_api_gateway_integration" "api" {
  rest_api_id             = "${aws_api_gateway_rest_api.api.id}"
  resource_id             = "${aws_api_gateway_rest_api.api.root_resource_id}"
  http_method             = "POST"
  type                    = "AWS"
  integration_http_method = "POST"
  passthrough_behavior    = "NEVER"
  credentials             = "${aws_iam_role.api.arn}"
  uri                     = "arn:aws:apigateway:${var.region}:sqs:path/${aws_sqs_queue.queue.name}"

  request_parameters = {
    "integration.request.header.Content-Type" = "'application/x-www-form-urlencoded'"
  }

  request_templates = {
    "application/json" = "Action=SendMessage&MessageBody=$input.body"
  }
}

We should define a basic 200 handler for successful requests with a custom response message. Layer on more responses as needed.

Notice the response_templates value below, which is what the service will return as a 200 status code and message. The selection_pattern is nothing more than a regex pattern to match any 2XX status codes that come back from SQS, which will then return a 200 and the json message {"message": "great success!"} to the client from the API Gateway request.

resource "aws_api_gateway_integration_response" "200" {
  rest_api_id       = "${aws_api_gateway_rest_api.api.id}"
  resource_id       = "${aws_api_gateway_rest_api.api.root_resource_id}"
  http_method       = "${aws_api_gateway_method.api.http_method}"
  status_code       = "${aws_api_gateway_method_response.200.status_code}"
  selection_pattern = "^2[0-9][0-9]"                                       // regex pattern for any 200 message that comes back from SQS

  response_templates = {
    "application/json" = "{\"message\": \"great success!\"}"
  }

  depends_on = ["aws_api_gateway_integration.api"]
}

resource "aws_api_gateway_method_response" "200" {
  rest_api_id = "${aws_api_gateway_rest_api.api.id}"
  resource_id = "${aws_api_gateway_rest_api.api.root_resource_id}"
  http_method = "${aws_api_gateway_method.api.http_method}"
  status_code = 200

  response_models = {
    "application/json" = "Empty"
  }
}

And lastly, create the API Gateway deployment.

resource "aws_api_gateway_deployment" "api" {
  rest_api_id = "${aws_api_gateway_rest_api.api.id}"
  stage_name  = "main"

  depends_on = [
    "aws_api_gateway_integration.api",
  ]
}

Create and Test

With everything defined we are good to run.

terraform init
terraform apply

Which will output something like below.

Apply complete! Resources: 12 added, 0 changed, 0 destroyed.

Outputs:

test_cURL = curl -X POST -H 'Content-Type: application/json' -d '{"id":"test", "docs":[{"key":"value"}]}' https://4h8qo0bdd1.execute-api.us-east-1.amazonaws.com/main/

Run it yourself and give the outputted test cURL command a try to see that the record ended up in the SQS queue. You can view all the example terraform code together below.

provider "aws" {
region = "${var.region}"
}
variable "region" {
default = "us-east-1"
type = "string"
}
output "test_cURL" {
value = "curl -X POST -H 'Content-Type: application/json' -d '{\"id\":\"test\", \"docs\":[{\"key\":\"value\"}]}' ${aws_api_gateway_deployment.api.invoke_url}/"
}
resource "aws_sqs_queue" "queue" {
name = "my-sqs-queue"
delay_seconds = 0 // how long to delay delivery of records
max_message_size = 262144 // = 256KiB, which is the limit set by AWS
message_retention_seconds = 86400 // = 1 day in seconds
receive_wait_time_seconds = 10 // how long to wait for a record to stream in when ReceiveMessage is called
}
resource "aws_iam_role" "api" {
name = "my-api-role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "apigateway.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
resource "aws_iam_policy" "api" {
name = "my-api-perms"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents",
"logs:GetLogEvents",
"logs:FilterLogEvents"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"sqs:GetQueueUrl",
"sqs:ChangeMessageVisibility",
"sqs:ListDeadLetterSourceQueues",
"sqs:SendMessageBatch",
"sqs:PurgeQueue",
"sqs:ReceiveMessage",
"sqs:SendMessage",
"sqs:GetQueueAttributes",
"sqs:CreateQueue",
"sqs:ListQueueTags",
"sqs:ChangeMessageVisibilityBatch",
"sqs:SetQueueAttributes"
],
"Resource": "${aws_sqs_queue.queue.arn}"
},
{
"Effect": "Allow",
"Action": "sqs:ListQueues",
"Resource": "*"
}
]
}
EOF
}
resource "aws_iam_role_policy_attachment" "api" {
role = "${aws_iam_role.api.name}"
policy_arn = "${aws_iam_policy.api.arn}"
}
resource "aws_api_gateway_rest_api" "api" {
name = "my-sqs-api"
description = "POST records to SQS queue"
}
resource "aws_api_gateway_request_validator" "api" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
name = "payload-validator"
validate_request_body = true
}
resource "aws_api_gateway_model" "api" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
name = "PayloadValidator"
description = "validate the json body content conforms to the below spec"
content_type = "application/json"
schema = <<EOF
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"required": [ "id", "docs"],
"properties": {
"id": { "type": "string" },
"docs": {
"minItems": 1,
"type": "array",
"items": {
"type": "object"
}
}
}
}
EOF
}
resource "aws_api_gateway_method" "api" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
resource_id = "${aws_api_gateway_rest_api.api.root_resource_id}"
api_key_required = false
http_method = "POST"
authorization = "NONE"
request_validator_id = "${aws_api_gateway_request_validator.api.id}"
request_models = {
"application/json" = "${aws_api_gateway_model.api.name}"
}
}
resource "aws_api_gateway_integration" "api" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
resource_id = "${aws_api_gateway_rest_api.api.root_resource_id}"
http_method = "POST"
type = "AWS"
integration_http_method = "POST"
passthrough_behavior = "NEVER"
credentials = "${aws_iam_role.api.arn}"
uri = "arn:aws:apigateway:${var.region}:sqs:path/${aws_sqs_queue.queue.name}"
request_parameters = {
"integration.request.header.Content-Type" = "'application/x-www-form-urlencoded'"
}
request_templates = {
"application/json" = "Action=SendMessage&MessageBody=$input.body"
}
}
resource "aws_api_gateway_integration_response" "200" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
resource_id = "${aws_api_gateway_rest_api.api.root_resource_id}"
http_method = "${aws_api_gateway_method.api.http_method}"
status_code = "${aws_api_gateway_method_response.200.status_code}"
selection_pattern = "^2[0-9][0-9]" // regex pattern for any 200 message that comes back from SQS
response_templates = {
"application/json" = "{\"message\": \"great success!\"}"
}
depends_on = ["aws_api_gateway_integration.api"]
}
resource "aws_api_gateway_method_response" "200" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
resource_id = "${aws_api_gateway_rest_api.api.root_resource_id}"
http_method = "${aws_api_gateway_method.api.http_method}"
status_code = 200
response_models = {
"application/json" = "Empty"
}
}
resource "aws_api_gateway_deployment" "api" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
stage_name = "main"
depends_on = [
"aws_api_gateway_integration.api",
]
}
@youjenli
Copy link

youjenli commented Nov 8, 2023

Thank you so much for sharing this. Going home earlier becomes possible.😂

@wolframite
Copy link

I wish I found that earlier! I still have one problem though: How can I pass on HTTP headers from the request to SQS message attributes? Creating attributes in the request template works like this: Action=SendMessage&MessageBody=$input.body&MessageAttribute.1.Name=X-TEST&MessageAttribute.1.Value.DataType=String&MessageAttribute.1.Value.StringValue=test. What I don't know is how I can assign the value from the actual header...

@wolframite
Copy link

Maybe it helps someone: Action=SendMessage&MessageBody=$input.body&MessageAttribute.1.Name=X-Github-Event&MessageAttribute.1.Value.DataType=String&MessageAttribute.1.Value.StringValue=$input.params("X-Github-Event")&MessageAttribute.2.Name=X-Github-Delivery&MessageAttribute.2.Value.DataType=String&MessageAttribute.2.Value.StringValue=$input.params("X-Github-Delivery")&MessageAttribute.3.Name=X-Hub-Signature-256&MessageAttribute.3.Value.DataType=String&MessageAttribute.3.Value.StringValue=$input.params("X-Hub-Signature-256")

@chris-albers
Copy link

We were dealing with an error caused by an ampersand in the body of the message, which caused SQS to return this cryptic error:

"Error": {
    "Code": "MalformedQueryString",
    "Message": "Keys may not contain  ",
    "Type": "Sender"
},

This can be fixed by URL encoding the contents of $input.body with $util.urlEncode like so:

request_templates = {
  "application/json" = "Action=SendMessage&MessageBody=$util.urlEncode($input.body)"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment