cloudonaut Posted August 11, 2022 Share Posted August 11, 2022 cfn-init is a little helper to install and configure EC2 instances managed with CloudFormation. Lately, I was running into issues when starting a more significant amount of EC2 (let’s say 50) during an auto scaling event. This blog post will teach you why the error happens and how to avoid it. /images/2022/08/throttling-error.jpg Introducing cfn-init cfn-init configuration is added as metadata to a resource using the AWS::CloudFormation::Init key. The following example configures cfn-init to create/update the file /etc/sample.conf. enable & start the service sample (also restarts the service if /etc/sample.conf is changed). cfn-init is usually executed in the user data script. VirtualMachine: Type: 'AWS::EC2::Instance' Metadata: 'AWS::CloudFormation::Init': config: files: '/etc/sample.conf': content: !Sub | [main] region=${AWS::Region} mode: '000400' owner: root group: root services: sysvinit: sample: enabled: true ensureRunning: true files: - '/etc/sample.conf' Properties: # [...] UserData: 'Fn::Base64': !Sub | #!/bin/bash -ex /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource VirtualMachine --region ${AWS::Region} The pitfall The way cfn-init is implemented is this: Call the CloudFormation API DescribeStackResource to read the metadata. Validate and parse the metadata. Apply the configuration to the EC2 instance. Unfortunately, the CloudFormation API has notorious low API rate limits, and cfn-init does not retry in the case of a rate exceeded error. Therefore, when many EC2 instances run cfn-init more or less at the same time, you will see the following error: 2022-06-02 07:13:14,838 [DEBUG] Response: 400 https://cloudformation.us-east-1.amazonaws.com/?Action=DescribeStackResource&LogicalResourceId=ScanLaunchTemplate&ContentType=JSON&StackName=***&Version=2010-05-15 [headers: {'x-amzn-RequestId': '***', 'Content-Type': 'application/json', 'Content-Length': '124', 'Date': 'Thu, 02 Jun 2022 07:13:14 GMT', 'Connection': 'close'}]2022-06-02 07:13:14,838 [DEBUG] Response error: b'{""Error"":{""Code"":""Throttling"",""Message"":""Rate exceeded"",""Type"":""Sender""},""RequestId"":""***""}' Solving the issue How can we solve the issue? Do not use cfn-init at all. Load the metadata from a file. To load the metadata from a file and not the CloudFormation API, create a file (e.g., metadata.json) like this: { "AWS::CloudFormation::Init": { "config": { "files": { "/etc/sample.conf": { "content": "[main]\nregion=eu-west-1\n", "mode": "000400", "owner": "root", "group": "root" } }, "services": { "sysvinit": { "sample": { "enabled": true, "ensureRunning": true, "files": [ "/etc/sample.conf" ] } } } } }} And invoke cfn-init like this: /opt/aws/bin/cfn-init -v metadata.json I hope this article will help you avoid the pitfall. View the full article Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.