Aws Automating Cloudformation

About Me

DevOps@RajeshKumar.XYZ

“Amazon Web Servicesis a platform of web services offering solutions for computing, storing, and networking, at different layers of abstraction.”

Amazon Web Services in Action, Manning 2015

Manage Infrastructure Manually
- Good way to start
- Perfect for Proof-of-Concept and Prototyping
Automate Infrastructure
- Improve quality and efficiency
- Major benefit of using AWS

Background Knowledge

AWS CloudFormation

Automate almost every part of AWS with CloudFormation
Learn how to automate:
- EC2 Instance
- VPC
- Security Group
- RDS

Use CloudFormation to create stacks
- Managment Console
- Command Line Interface
Create CloudFormation templates

Let’s start!

Exploring AWS CloudFormation

Overview

Automating infrastructure
- Lower costs
- Improve quality
- Improve flexibility
Describing infrastructure in code
- No scripting or programming needed
- Perfect tool for the job
AWS CloudFormation
- Describing AWS infrastructure in code
- Highly integrated with AWS

Automation vs. Manual Work

Managing AWS Manually

Adam, DevOps engineer at Globomantics
Frustrated with current situation
Goals
- Lower costs
- Improve quality
- Improve flexibility

Advantages of Automating Infrastructure

What is the biggest advantage of Amazon Web Services?

Every part of AWS is controllable via an API

AWS API Enables Automation

Controlling AWS with an API

API Usage Examples

Launch EC2 instance
Create Security Group
Update RDS database configuration
Create networking infrastructure
Add new IAM user
....

Tools to Access AWS API

Resolve dependencies
- RDS database needs to be created before EC2 instances are created.
- Security Groups needs to be created before EC2 instance is created.

Update infrastructure
- Which steps are necessary to update existing infrastructure?
- Hard coded transformation not very flexible.

Describing Infrastructure in Code

Describing infrastructure in code
- Resolves dependencies automatically
- Creates, updates or deletes infrastructure
- Performs idempotent and predictable changes to infrastructure

AWS CloudFormation

Features of CloudFormation


	"Type": "AWS::EC2::Instance",
	"Properties": {
	   "ImageId": "ami-bff32ccc",
	   "InstanceType": "t2.nano”
	}

Template

Written in JSON describing the target state of the infrastructure.

Stack, Instance of a Template

1 Template

=

Multiple Stacks

Simple WordPress environment
- EC2 instance running Web Server
- RDS instance with MySQL engine
- Security Groups
Have a look at template defining target state
Create stack based on template

Text Editor
IDE Integration
- Eclipse
- Visual Studio
AWS CloudFormation Designer
- Graphical tool for creating templates
- Part of AWS Management Console

Summary

Automating infrastructure
- Lower costs
- Improve quality
- Improve flexibility
Describing infrastructure in code
- No scripting or programming needed
- Perfect tool for the job
AWS CloudFormation
- Describing AWS infrastructure in code
- Highly integrated with AWS

Creating a CloudFormation Template

Overview

Creating a template from scratch
Anatomy of a template
Define target state of your infrastructure
Explore resource types and their properties
Describe dependencies between resources

Adam, DevOps engineer at Globomantics
Starting with a manageable scope
Creating his first CloudFormation template

SSH Bastion Host

Needed Parts for Bastion Host

Anatomy of a CloudFormation Template

Minimal Template


	{
	 "AWSTemplateFormatVersion": "2010-09-09",
	 "Description": "SSH Bastion Host",
	 "Resources": {
	    "EC2Instance": {...},
	    "ElasticIp": {...},
	    "SecurityGroup": {...} 
	 }
	}


	{
	  "AWSTemplateFormatVersion": "2010-09-09",
	  "Description": "SSH Bastion Host",
	}

Template Format Version and Description

Version of the template anatomy: only valid value: 2010-09-09

A description helps you to organize and document your template


	"Resources": {
		"EC2Instance": {...},
		"ElasticIp": {...},
		"SecurityGroup": {...}
	}

Resources Section

Includes the description of your infrastructure

The target state of all involved resources is described here

Working with Resources

Resource Types

AWS::EC2::Instance
AWS::EC2::SecurityGroup
AWS::EC2::EIP
AWS::RDS::DBInstance
AWS::IAM::Role
AWS::EC2::VPC
AWS::ElastiCache::CacheCluster
AWS::DynamoDB::Table
...

Resource Definition


 {
	...
	"Resources": {
		"EC2Instance": {
		   "Type": "AWS::EC2::Instance",
		   "Properties": {...}
		}
	}
 }


	"Type": "AWS::EC2::Instance",
	"Properties": {
		"ImageId": "ami-bff32ccc",
		"InstanceType": "t2.nano"
	}

EC2 Instance

Depending on the Resource Type, there are different properties available

It is possible to specify AMI and Instance Type for an EC2 instance

EC2 Instance


 "EC2Instance": {
	"Type": "AWS::EC2::Instance",
	"Properties": {
		"ImageId": "ami-bff32ccc",
		"InstanceType": "t2.nano",
		"NetworkInterfaces": [{"SubnetId": “..."}]
    }
 }


	"Type": "AWS::EC2::EIP",
	"Properties": {
	    "InstanceId": "i-d0f2e055",
		"Domain": "vpc"
	}}

Elastic IP

Properties of resource of type AWS::EC2::EIP

An Elastic IP address needs to be linked to an EC2 instance

Domain vpc as we are operating in a VPC


	"Type": "AWS::EC2::SecurityGroup",
	"Properties": {
		"GroupDescription": "ssh-bastion-host",
		"VpcId": "vpc-6b53320e",
		"SecurityGroupIngress": [{...}]
	}

Security Group

Resource describing a Security Group

Includes link to VPC and nested ingress rules


	"SecurityGroupIngress": [{
		"CidrIp": "0.0.0.0/0",
		"FromPort": 22,
		"IpProtocol": "tcp",
		"ToPort": 22
	}]

Security Group Ingress

Nested ingress rules of Security Group

Allowing incoming TCP traffic from everywhere on port 22

Dependencies Between Resources

CloudFormation is able to resolve dependencies between resources automatically
But how do you define dependencies within the template

Dependencies Between Resources

Referencing Security Group


 "EC2Instance": {
	"Properties": {
		"NetworkInterfaces": [{
		  "GroupSet": [{"Ref": "SecurityGroup"}]
		}]
	}
 },
 "SecurityGroup": {...}

Referencing EC2 Instance


 "EC2Instance": {...},
 "ElasticIP": {
    "Type": "AWS::EC2::EIP",
    "Properties": {
	   "InstanceId": {"Ref": "EC2Instance"},
	   "Domain": "vpc"
   }
 }

Walk-through template
- EC2 Instance
- Security Group
- Elastic IP
- References between Resources
Create stack out of template by using Command Line Interface (CLI)

Install and Configure AWS Command Line Interface (CLI)

https://aws.amazon.com/cli/

Summary

Minimal CloudFormation template
- Format Version
- Description
- Resources Section
Target state is defined within resources section
Describing resources
- Name
- Resource Type
- Parameters

Using Parameters and Outputs to Customizea Stack

Overview

Reuse templates for multiple slightly different stacks
Add parameters to a template
Integrate stacks into existing environments
Use outputs to access results from a running stack

Adam, DevOps engineer at Globomantics
Already done
- CloudFormationtemplate including a SSH bastion host environment
To do
- Customize stack for slightly different use cases

Input and Output

Parameters to Specify Input

Adam wants to reuse his template for multiple SSH bastion host setups
Small changes are needed depending for different setups
- Key Pair
- Instance Type

Reuse Template

Parameters Section


  {
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "SSH Bastion Host",
	"Parameters": {...},
	"Resources": {...}
  }

Defining a Parameter


  "Parameters": {
     "KeyPair": {
	  "Description": "A SSH key pair.",
	  "Type": "AWS::EC2::KeyPair::KeyName"
     }
  }


  "InstanceType": {
	"Description": “The instance type.",
	"Type": "String"
  }

Description

Describe your parameters to simplify filling them out later

Defining a Parameter


  "Parameters": {
     "KeyPair": {
	  "Description": "A SSH key pair.",
	  "Type": "AWS::EC2::KeyPair::KeyName"
	 }
  }


   String: "string"
   Number: 1
   List<Number>: [1,2,3]
   CommaDelimitedList: "1,2,3"

Parameter Type

General parameter types


	AWS::EC2::Image::Id: "ami-b43503a4"
	AWS::EC2::Instance::Id: "i-47cad982"
	AWS::EC2::SecurityGroup::Id: "sg-6246b70c"
	AWS::Route53::HostedZone::Id: "Z2VHZWU704GYI1"

Parameter Type

AWS parameter types

Referencing Parameters


 "Parameters": {
   "KeyPair": {...}
 },
 "Resources": {
   "EC2Instance": {
      "Properties": {
         "KeyName": {"Ref": "KeyPair"}
       }
   }}


	AllowedValues: ["a", "b", "c"]
	AllowedPattern: "[a-z0-9]*"
	MaxLength/ MinLength: 1
	MaxValue/ MinValue: 2

Input Validation

Software development 101 : validate your inputs

Validating Instance Type


 "InstanceType": {
	"Description": "The instance type.",
	"Type": "String",
	"AllowedValues": ["t2.nano", "t2.micro", "t2.small"]
  }

Default Value


 "InstanceType": {
	"Description": "The instance type.",
	"Type": "String",
	"Default": "t2.nano"
	}

Walk-through template
- Input Parameters
- Validation
Create stack out of template by using Command Line Interface (CLI)
Specify input parameters for stack

Outputs to Access Results from Stack

Adam wants to integrate SSH bastion server stacks into existing environment
He needs to access the results from a stack to be able to
- Update DNS record with a Public IP
- Add a Private IP to firewall configuration

Access Results of Stack

Outputs Section


  {
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "SSH Bastion Host",
	"Parameters": {...},
	"Resources": {...},
	"Outputs": {...}
  }

Defining an Output


  "Outputs": {
	"SSHBastionHostPublicIp": {
	  "Description": "Public IP address...",
	  "Value": {"Ref": "ElasticIP"}
	}
  }


	"SSHBastionHostPublicIp": {
	  "Description": "Public IP address...",
	  "Value": {"Ref": "ElasticIP"}
	}

Default Resource Return Value

Accessible via built -in Ref function

Return Value depends on the Resource Type


  "SSHBastionHostPrivateIp": {
	"Description": "Private IP address...",
	"Value": {"Fn::GetAtt": ["EC2Instance", "PrivateIp"]}
  }

Specific Resource Return Value

Accessible via built-in Fn::GetAtt function

Different return values depending on the Resource Type

Returning Public IP


    "Resources": {
	  "ElasticIP": {...}
	}
	"Outputs": {
	  "SSHBastionHostPublicIp": {
	    "Description": "Public IP address...",
	    "Value": {"Ref": "ElasticIP"}
	  }
	}

Returning Private IP


    "Resources": {
      “EC2": {...}
   }
   "Outputs": {
     "SSHBastionHostPrivateIp": {
	    "Description": "Private IP address...",
	    "Value": {"Fn::GetAtt": ["EC2", "PrivateIp"]}
     }
    }

Walk-through template
- Defining Outputs
- Accessing return values from resources
Get outputs from a running stack via CLI

Summary

Inputs
- Allows you to reuse a template in slightly differnt use cases
- Template defines possible Parameters
- Paramters have to be specified when creating/updating a stack

Outputs
- Allows you to access results from a stack
- Template defines outputs
- Outputs can be accessed after stack was created/updated successfully

Building More Powerful Templates with Built in Helpers

Overview

Creating multi-region template
- Mappings
- Fn::FindInMap
Using Pseudo Parameters
- AWS::AccountId, AWS::Region, …
- Fn::GetAZs
Provisioning an EC2 Instance
- User Data
- Fn::Join and Fn::Base64

Common problems when creating a CloudFormationtemplate …
- Multi-region support
- Install and configure software on EC2 instances automatically
- Reduce number of parameters
But how to do that in JSON?
- CloudFormation offers built in helpers for common problems

Adam, DevOps engineer at Globomantics
Create CloudFormation template
- Networking configuration
- EC2 Instance running web server
- Latest Amazon Linux AMI
Create stacks in the following regions
- us-east-1
- eu-west-1
- ap-northeast-1

Independent AMI per Region

Introducing Mappings

“A map is an object that maps keys to values.”

What is a map?

Mappings Section


  {
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "SSH Bastion Host",
	“Mappings": {...},
	"Resources": {...}
  }

Defining a Mapping


  "Mappings" : {
	"Mapping" : {
	   "Key" : {
		"Name" : "Value"
	   }
	}
   }


    "Mapping" : {
	  "KeyA" : {},
	  "KeyB" : {}
	}

Unique Key

A key has to be unique within a mapping


    "Key" : {
	   "NameA" : "ValueA",
	   "NameB" : "ValueB",
	}

Named Values

A key can contain multiple named values

A name has to be unique within a key

Map AMIs per Region


 "RegionAMI": {
	"us-east-1": {
	   "AmazonLinux": "ami-8fcee4e5",
	   "Ubuntu": "ami-fce3c696"
	},
	"eu-west-1": {
	   "AmazonLinux": "ami-e1398992",
	   "Ubuntu": "ami-f95ef58a"
 }}


   "Fn::FindInMap": ["RegionAMI", "eu-west-1", "AmazonLinux"]

Fn:: FindInMap

Allows to access values from a Mapping

Accepts Mapping name, Key, and Name as parameters

CloudFormation template
- Networking configuration
- EC2 Instance
- Security Group
Mapping contains AMI IDs for multiple regions
Create stack in multiple regions with the help of AWS CLI

Using Pseudo Parameters

Adam, DevOps engineer at Globomantics
CloudFormation template reusable for multiple regions
How to define the Availability Zone for a Subnet?

Define AZ for Multi-Region Template


 "Subnet": {
	"Type": "AWS::EC2::Subnet",
	"Properties": {
	"AvailabilityZone": "???",
	"CidrBlock": "10.0.0.0/24",
	"VpcId": {"Ref": "VPC"},
	"MapPublicIpOnLaunch": "true"
 }}


   {"Ref": "AWS::Region"}

Accessing a Pseudo Parameter

Use built in Ref function to access pseudo parameters


   {"Fn::GetAZs": {"Ref": "AWS::Region"}}

Fn::GetAZs

Returns all Availability Zones for a region

Uses current region of stack in this example


   {"Fn::Select": ["0", ["a", "b", "c"]}

Fn::Select

Selects a value from a list

Selects first value of list in this example

Define AZ for Multi-region Template


	"Type": "AWS::EC2::Subnet",
	"Properties": {
	  "AvailabilityZone": {
		"Fn::Select": [
		  "0",
		  {"Fn::GetAZs": {"Ref": "AWS::Region"}}
		]
	  }, ...
	}

Template Reference

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-reference.html

CloudFormationTemplate
- Pseudo Parameters
- Fn::GetAZs
- Fn::Select

Provision EC2 Instance Automatically

Adam, DevOps engineer at Globomantics
CloudFormation template includes EC2 Instance
Launch Web Server on EC2 Instance automatically

Necessary Steps During Bootstrap

User Data of EC2 Instance


   "Type": "AWS::EC2::Instance",
	  "Properties": {
	  "UserData": "..."
	}

User Data Property

User Data can be described as property of an EC2 Instance

But User Data needs to be encoded in base64


   {"Fn::Base64": "A String"}

Fn::Base64

There’s a built in function for that

Encodes a String in base64


   {"Fn::Join": [";", ["a", "b", "c"]]}

Fn::Join

Allows you to concatenate Strings

Accepts a delimiter and a list of Strings as input

User Data Containing Shell Script


   "UserData": {"Fn::Base64": {"Fn::Join": ["\n", [
	  "#!/bin/bash-ex",
	  "yum install-y httpd",
	  "cd /var/www/html",
	  "echo '<html><body>...</body></html>' > index.html",
	  "service httpd start"
   ]]}}

CloudFormation Template
- EC2 Instance
- User Data containing Shell Script
Create stack with the help of AWS CLI and access the created Web Server

Summary

Mappings section
- Map between region and AMI
Pseudo Parameters
Built in functions
- Fn::GetAZs
- Fn::Select
- Fn::Join
- Fn::Base64
User Data
- Provision EC2 Instance automatically

Automating AWS with CloudFormation

INTRODUCTION