Problem of Not Queueing in SSE SQS by CloudWatch Events

Problem of Not Queueing in SSE SQS by CloudWatch Events

Takahiro Iwasa
(岩佐 孝浩)
Takahiro Iwasa (岩佐 孝浩)
3 min read
CloudWatch Events SQS

I noticed that CloudWatch Events did not queue messages in SSE SQS. After some investigation, it seemed that CloudWatch Events did not support SSE SQS as target. This may be caused by an SQS policy. I’ll leave it as a note.

Creating AWS Resources

Use the following CloudFormation template. CloudWatch Events will be configured to start every minute using a cron expression (line 16).

AWSTemplateFormatVersion: "2010-09-09"
Description: Example of CloudWatch Events not queueing to SSE SQS
Resources:
  SQS:
    Type: AWS::SQS::Queue
    Properties:
      KmsDataKeyReusePeriodSeconds: 86400
      KmsMasterKeyId: alias/aws/sqs
      MessageRetentionPeriod: 1209600
      QueueName: sse-sqs-for-cloudwatch-events

  Events:
    Type: AWS::Events::Rule
    Properties:
      Name: cloudwatch-events-test
      ScheduleExpression: 'cron(0/1 * * * ? *)'
      State: ENABLED
      Targets:
        - Arn: !GetAtt SQS.Arn
          Id: cloudwatch-events-test

  QueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      PolicyDocument:
        Version: '2012-10-17'
        Id: !Sub
          - ${SqsArn}/SQSDefaultPolicy
          - {SqsArn: !GetAtt SQS.Arn}
        Statement:
          - Sid: !Sub
            - AWSEvents_${SqsName}_Id123456789012
            - {SqsName: !GetAtt SQS.QueueName}
            Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: sqs:SendMessage
            Resource: !GetAtt SQS.Arn
            Condition:
              ArnEquals:
                aws:SourceArn: !GetAtt Events.Arn
      Queues:
        - !Ref SQS

Deploy the CloudFormation stack with the following command.

aws cloudformation deploy --template template.yaml --stack-name cloudwatch-events-test

Testing

CloudWatch Events

Check the CloudWatch Events rule with the following command.

aws events describe-rule --name cloudwatch-events-test
{
    "Name": "cloudwatch-events-test",
    "Arn": "arn:aws:events:<YOUR_REGION>:<YOUR_ACCOUNT_ID>:rule/cloudwatch-events-test",
    "ScheduleExpression": "cron(0/1 * * * ? *)",
    "State": "ENABLED",
    "EventBusName": "default",
    "CreatedBy": "<YOUR_ACCOUNT_ID>"
}

SQS

Confirm the available message count. It should consistently show zero.

aws sqs get-queue-url --queue-name sse-sqs-for-cloudwatch-events
{
    "QueueUrl": "https://sqs.<YOUR_REGION>.amazonaws.com/<YOUR_ACCOUNT_ID>/sse-sqs-for-cloudwatch-events"
}
aws sqs get-queue-attributes \
  --queue-url https://sqs.<YOUR_REGION>.amazonaws.com/<YOUR_ACCOUNT_ID>/sse-sqs-for-cloudwatch-events \
  --attribute-names ApproximateNumberOfMessages
{
    "Attributes": {
        "ApproximateNumberOfMessages": "0"
    }
}

Updating SSE SQS to SQS

Comment out KmsDataKeyReusePeriodSeconds and KmsMasterKeyId in the CloudFormation template, and then update the stack.

--- 	Sun Oct 10 17:31:16 2021 UTC
+++ 	Sun Oct 10 17:31:16 2021 UTC
@@ -4,8 +4,8 @@
   SQS:
     Type: AWS::SQS::Queue
     Properties:
-      KmsDataKeyReusePeriodSeconds: 86400
-      KmsMasterKeyId: alias/aws/sqs
+#      KmsDataKeyReusePeriodSeconds: 86400
+#      KmsMasterKeyId: alias/aws/sqs
       MessageRetentionPeriod: 1209600
       QueueName: sse-sqs-for-cloudwatch-events

After successfully updating the stack, check the available message count again. It should consistently increase.

aws sqs get-queue-url --queue-name sse-sqs-for-cloudwatch-events
{
    "QueueUrl": "https://sqs.<YOUR_REGION>.amazonaws.com/<YOUR_ACCOUNT_ID>/sse-sqs-for-cloudwatch-events"
}
aws sqs get-queue-attributes \
  --queue-url https://sqs.<YOUR_REGION>.amazonaws.com/<YOUR_ACCOUNT_ID>/sse-sqs-for-cloudwatch-events \
  --attribute-names ApproximateNumberOfMessages
{
    "Attributes": {
        "ApproximateNumberOfMessages": "1"
    }
}

Cleaning Up

Clean up the provisioned AWS resources with the following command.

aws cloudformation delete-stack --stack-name cloudwatch-events-test

Conclusion

If you intend to configure an SSE SQS as a target of CloudWatch Events, additional settings may be required. Unfortunately, I could not find information on that issue in the AWS official documentation.

Takahiro Iwasa
(岩佐 孝浩)

Takahiro Iwasa (岩佐 孝浩)

Software Developer at iret, Inc.
Architecting and developing cloud native applications mainly with AWS. Japan AWS Top Engineers 2020-2023