Amazon EC2 Auto Scaling lifecycle hooks to Export Instance Logs After Marked For Terminate

When an Auto Scaling group needs to scales down it terminates the EC2 Instances based on Auto Scaling termination policy. The data gets lost in the EC2 Instances that gets terminated, and any ongoing tasks get interrupted.
We can use Auto Scaling lifecycle hooks feature to solve this. Auto Scaling lifecycle hooks give you more control over timing after an instance the marked for termination.
Here I have written the detailed step by step guide to back up your data automatically before the Instance gets terminated.

The AWS Services used.
1. Amazon EC2 Auto Scaling Lifecycle Hooks
2. AWS Lambda
3. AWS SSM(Run command)
4. AWS SNS
5. AWS AWS Identity and Access Management
6. AWS Cloud Watch Rules

Step 1 — Create the SNS topic to get the notifications after backup.
In this step, we need to create the SNS topic in the same region where Amazon EC2 Auto Scaling created.
Create the SNS topic and subscribe your email address in the SNS Topic to receive the email notification. To confirm your email address to the SNS topic, you get the confirmation email to your email address.
Save the SNS topic ARN which we need in the ongoing setup.

SNS Topic and Email subscription

Step -2 — Create IAM roles for instance and lambda function.
Here I used the AWS Console to create IAM roles for EC2 Instances and AWS Lambda function to enable to access to run SSM commands and upload files to S3 bucket.
You need to create a custom policy to allow your EC2 Instances and AWS Lambda function to complete Auto Scaling lifecycle hooks and publish to the SNS topic created above.

1. Login to AWS Console and open the IAM service.
2. Goto Policies and select the Create Policy to create your policies.
3. Paste the following policy document into the JSON. It allows full lifecycle hook actions and SNS.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "autoscaling:CompleteLifecycleAction",
        "sns:Publish"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

4. Give the policy name as your convenient and create the policy.

Iam Policy

Create the EC2 Instance Role:
1. In the left navigation, choose Roles and click on Create New Role.
2. Provide the role name and click on Next Step.
3. Select Amazon EC2 Service and click on Next Step.
4. Add the policies AmazonEC2RoleforSSM (default policy by AWS) and the Policy which created above.
5. Click on Next Step and then click on Create the Role.

Ec2 Instance Role

Create the IAM Role for AWS Lambda function:
1. In the left navigation, choose Roles and click on Create New Role.
2. Provide the Role name and select the Next Step.
3. Choose AWS Lambda Service and click on Next Step.
4. Add AmazonSSMFullAccess, AWSLambdaBasicExecutionRole policies and the Policy which created above.
5. Choose Next Step and click on Create the Role.

Lambda Role

Step -3 Create lifecycle hook on autoscaling group.

1. Choose Auto Scaling Groups from the navigation panel.
2. Select Auto Scaling group and then select the lifecycle hook from the configuration panel.
3. To Create lifecycle hook, select the Create lifecycle hook.
4. Provide the lifecycle hook name, select lifecycle transition as Instance terminate, keep the default Heartbeat Timeout value which is 600 and click Continue.
5. Click on Create the lifecycle hook.

Lifecycle hook creation

Step -4 — Create the S3 bucket for backup files.

Create S3 bucket to store the backup files, and S3 bucket name should be unique across AWS. If you already have the S3 bucket, you can use that.Open the s3 console and select the create bucket.

1. Open the S3 console and select the Create Bucket.
2. Provide the bucket name and the should be unique.
3. Click on Create to create the bucket.

The SSM document which does archive the files in the server and archived backup data to S3 bucket using AWS CLI. Once backup procedure gets complete, it sends the notification to the email which subscribed in the setup
You can get the SSM document from GitHub repo.
File Name: SSM-Document

GogineniManojkumar/aws-lifecycle-hook

1. From the AWS EC2 console, go to System Manager Shared Resources,
2. Choose Documents then click on Create document
3. Enter the document name
4. Document Type, keep the default Command
5. For Content, you copy the JSON document provided on my GitHub repo.
6. Click on Create document.

Step -6— Create the Lambda function.
The AWS Lambda function uses modules included in the Python 2.7 and the Python module (boto3). The function performs the following:

A. Checks whether the SSM document exists, which has the script that runs on your EC2 Instances.
B. Sends the command to the EC2 Instance that marked for termination.
C. Checks for the status of EC2 Run Command, and if it fails, the Lambda function completes the lifecycle hook.

1. Log in to the AWS Lambda console, select Create Lambda function.
2. You can skip Select blueprint and click on Next.
3. For Name, type any name and for Runtime, choose Python 2.7.
4. For Lambda function code, paste the Lambda function from the [link] GitHub repository.
   From the code you copied, you need to make changes to the autoscaling group names and the directory where you want to take the backup.

Need modifications on ASG name and backup path

https://github.com/GogineniManojkumar/aws-lifecycle-hook/blob/master/Lambda-code

5. Select Choose an existing role and choose the lambda-role which created above.

6. Create the following Environment variables in the AWS Lambda function

S3BUCKET : Bucket which is created for the archive files in step-3.
SNSTARGET : SNS topic ARN to get notified after archive created in Step-1.
SSM_DOCUMENT_NAME: Document which created in step-4.

Lambda environment variables

7. In Advanced settings, configure Timeout for 5 minutes.
8. Choose Next, Create function.

Step -7 — Create Cloud Watch Event To Trigger the Lambda Function

1. Go to CloudWatch Console, choose Events and click Create Rule.
2. Select Event Source as Auto Scaling
3. Select AWS Lambda function ad Target
4. Select the AWS Lambda function that you previously created in step Choose Configure details.
5. In Rule definition, type a name and click on Create Rule.
6. Whenever Auto Scaling group (mentioned in the AWS Lambda function which creates a lifecycle hook) starts terminating an EC2 Instance, the Lambda function gets triggered.

Step- 8 — Testing The Environment.
From the Auto Scaling Console, you can change the Desired and the Minimum for your testing Auto Scaling group to 0, then AWS EC2 Auto Scaling group marks EC2 Instances for termination, that Instance lifecycle status changed to Termination: Wait, which you can see from Instances tab. Then the AWS Lambda function gets triggered by AWS CloudWatch Event.

You can review your CloudWatch logs to see the Lambda function output. From the CloudWatch Console, choose Logs and select /aws/lambda/{function_name} to see the execution output.

You can go to your S3 bucket and check the uploaded files. You can also check AWS SSM Command history from the EC2 Console to see if the command executed correctly.

https://github.com/GogineniManojkumar/aws-lifecycle-hook.git