Let's go back a few years. I'd just joined a new team in a new company, all the infrastructure was in AWS, and I had very little IaC experience at the time.
Some background
CloudFormation is AWS's Infrastructure as Code service by default and many other internal services use it behind curtains to perform certain configuration tasks and deploy their own components. It is a very powerful tool, it is fully integrated with a lot of AWS APIs and services, it is a no-brainer if you are (and you really should) managing your infrastructure via code, not using 3rd party tools.
But CloudFormation can be a bit too much when it comes to creating big stacks (as in infrastructure resources grouped in a logical way that makes sense application-wise. E.g. a load balancer, security groups, EC2 autoscaling groups, RDS databases and S3 buckets all serving a single application).
As CloudFormation defines the resources either in a JSON or YAML file, the template file will grow as you add more AWS objects. This, as you may imagine, will get messy and prone to time-wasting, eventual errors and most of all: difficult to read.
Of course, one could argue, why not decouple resources into function stacks? Sure, valid point - but everything depends on the use case. Even though you have the perfect tool for 98% of infrastructure topologies and ways of managing almost all examples you read about, there is always something a bit different. And that's where the fun begins.
It is worth mentioning, these were pre-CDK times and we had not fully implemented Terraform because..reasons.
So how could we define resources in a programmatic, readable (short!) and maintainable way? Yes, maybe you already guessed from the article title, Troposphere.
Troposphere
This is a Python library that creates AWS CloudFormation definitions (templates). These will then be used as descriptors sent to CloudFormation API to create and manage resources.
In short, you define a resource using a given class and it creates a definition (template) out of it that you can then use to pass to a CloudFormation API method (like create stack). Sound pretty familiar huh? CDK vibes?
A couple of great things about Troposphere:
Active community.
Property and type check built in (meaning it will output errors if a resource is malformed or badly initialized).
It is written in (and uses) Python.
There are a lot of implementation examples.
But not everything is roses: this is not an official AWS tool, which means it depends on community contribution to support any AWS service updates - so it can get behind, as expected.
Quick walkthrough
Prerequisites
An AWS account with at least a VPC in any region.
AWS CLI configured with IAM credentials allowing to create/delete EC2 resources (security groups, subnets and instances).
(option) An existing EC2 KeyPair, if you want to test out a SSH connection to the created instances.
Python >= 3.11.2 (that's the one I tested this with).
Installation
I always recommend using a (Python) virtual environment to avoid messing around with any other locally installed version and dependencies you may have..but yeah, it's a personal preference, not required.
python3 -m venv new_env
source new_env/bin/activate
pip install troposphere
Example template(s)
I added a couple of example scripts in this repo to showcase the functionalities.
git clone https://github.com/marianogg9/troposphering.git
There are two folders:
.
├── README.md
├── instances
│ ├── instances.py
│ └── instances_input
└── subnets
├── subnets.py
└── subnets_input
Each folder contains a script and an input file, where you will add your current AWS account values, such as:
Once you added the corresponding values, change the input files names to be: subnets_input.json and instances_input.json.
One important detail: "instances" stack is dependent on "subnets" stack as it references the subnet names from the latter.
Run the scripts to generate both templates:
cd subnets
python3 subnets.py
cd instances
python3 instance.py
The first script will create a template along these lines (defining a set of subnets in a given input VPC + exporting the values to be used by the second template later on):
Outputs:
a:
Export:
Name: !Sub '${AWS::StackName}-a'
Value: !Ref 'a'
b:
Export:
Name: !Sub '${AWS::StackName}-b'
Value: !Ref 'b'
Resources:
a:
Properties:
AvailabilityZone: a
CidrBlock: some-cidr
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: subnet-a
- Key: Description
Value: Playing around with CloudFormation and Troposphere
- Key: CommongTag2
Value: Just to add one more
VpcId: vpc-abcdefgh
Type: AWS::EC2::Subnet
b:
Properties:
AvailabilityZone: b
CidrBlock: 172.30.124.80/28
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: subnet-b
- Key: Description
Value: Playing around with CloudFormation and Troposphere
- Key: CommongTag2
Value: Just to add one more
VpcId: vpc-abcdefgh
Type: AWS::EC2::Subnet
And this will be the second generated file (creating an EC2 instance, in one of the above to-be-created subnets, and a set of security groups):
Outputs:
firstIntanceInstanceID:
Value: !Ref 'firstIntance'
firstIntancePrivateIP:
Value: !GetAtt 'firstIntance.PrivateIp'
firstIntancePublicIP:
Value: !GetAtt 'firstIntance.PublicIp'
instanceNumberTwoInstanceID:
Value: !Ref 'instanceNumberTwo'
instanceNumberTwoPrivateIP:
Value: !GetAtt 'instanceNumberTwo.PrivateIp'
instanceNumberTwoPublicIP:
Value: !GetAtt 'instanceNumberTwo.PublicIp'
Resources:
defaultSG:
Properties:
GroupDescription: This is the default Security Group
SecurityGroupEgress:
- CidrIp: someOtherCidr
FromPort: 80
IpProtocol: tcp
ToPort: 80
SecurityGroupIngress:
- CidrIp: someCidr
FromPort: 22
IpProtocol: tcp
ToPort: 22
- CidrIp: someOtherCidr
FromPort: 123
IpProtocol: tcp
ToPort: 123
Tags:
- Key: Name
Value: defaultSG
- Key: Description
Value: Playing around with CloudFormation and Troposphere
- Key: CommongTag2
Value: Just to add one more
VpcId: vpc-abcdefgh
Type: AWS::EC2::SecurityGroup
firstIntance:
Properties:
ImageId: ami-123
InstanceType: t2.micro
KeyName: YourExistingKeyPair
SecurityGroupIds:
- !Ref 'defaultSG'
SubnetId: !ImportValue 'AddingSubnetsWithTroposphere-someregiona'
Tags:
- Key: Name
Value: firstIntance
- Key: Description
Value: Playing around with CloudFormation and Troposphere
- Key: CommongTag2
Value: Just to add one more
Type: AWS::EC2::Instance
instanceNumberTwo:
Properties:
ImageId: ami-123
InstanceType: t2.micro
KeyName: YourExistingKeyPair
SecurityGroupIds:
- !Ref 'defaultSG'
SubnetId: !ImportValue 'AddingSubnetsWithTroposphere-someregionb'
Tags:
- Key: Name
Value: instanceNumberTwo
- Key: Description
Value: Playing around with CloudFormation and Troposphere
- Key: CommongTag2
Value: Just to add one more
Type: AWS::EC2::Instance
These YAML templates use CloudFormation ImportValue, GetAtt, Ref intrinsic functions to fetch and reference values either defined externally (like Subnet Names from the first stack) or locally.
Now use those templates to create the resources:
$ cd subnets
$ python3 subnets.py
$ aws cloudformation create-stack --stack-name AddingSubnetsWithTroposphere --template-body file://subnets_template.yaml
The stack name AddingSubnetsWithTroposphere is part of the instances_input.json, so if you want to use a different stack name, please remember to update it in the values before running:
$ cd instances
$ python3 instances.py
$ aws cloudformation create-stack --stack-name AddingInstancesWithTroposphere --template-body file://instances_template.yaml
Each of the above will output the CFN stack id, something like:
{
"StackId": "arn:aws:cloudformation:<your_region>:<your_aws_account_id>:stack/AddingInstancesWithTroposphere/<CFN_stack_id>"
}
(optional) After a few minutes (to give some time for the instances to be ready), you can test the SSH connection by:
ssh -i your_key_pair ec2-user@<InstanceXPublicIP>
Where <InstanceXPublicIP> is an output value available in the AddingInstancesWithTroposphere CFN stack. You can check all output values in CloudFormation console > AddingInstancesWithTroposphere stack > Outputs tab.
Don't forget to clean up!
Delete both CloudFormation stacks, either via the console or using the CLI:
Conclusion
This was my first proper IaC tool deep dive experience, and although my teammates were the ones who implemented it from scratch, it was a great first step.
Looking back I think Terraform and mostly CDK have taken over this approach, but again this is a very simple concept, pretty easy to use once you get a hold of it. The Community is super active and they are open to getting help. If you know Python and like the tool, don't think it twice and open an issue!
This implementation can use a couple more iterations like automating CFN stacks creation directly from a parent script and adding some more code reusing into the scripts regarding mapping regions or AMIs and instance types. I will be updating the repo, stay tuned!
Last but not least as a suggestion, if you are working with CloudFormation or IaC on AWS resources, remember to always check the CloudFormation reference documentation on a given resource, e.g. for an EC2 instance. It explains which attribute can be updated on the fly without interruption (replacement) - it has saved me more than once.
References
Thank you for stopping by! Do you know other ways to do this? Please let me know in the comments, I always like to learn how to do things differently.
