# Troposphere: make CloudFormation legible again Let's go back a few years. I'd just joined a new team in a new company, all the infrastructure was in AWS, and I had very little [IaC](https://glossary.cncf.io/infrastructure-as-code/) experience at the time. ![](https://media.giphy.com/media/oaRG0HAau2X1SU7BTw/giphy.gif align="center") ## Some background [CloudFormation](https://aws.amazon.com/cloudformation/) is AWS's Infrastructure as Code service by default and many other internal services use it behind curtains to perform certain configuration tasks and deploy their own components. It is a very powerful tool, it is fully integrated with a lot of AWS APIs and services, it is a no-brainer if you are (and you really should) managing your infrastructure via code, not using 3rd party tools. But CloudFormation can be a bit too much when it comes to creating big stacks (as in infrastructure resources grouped in a logical way that makes sense application-wise. E.g. a load balancer, security groups, EC2 autoscaling groups, RDS databases and S3 buckets all serving a single application). As CloudFormation defines the resources either in a JSON or YAML file, the template file will grow as you add more AWS objects. This, as you may imagine, will get messy and prone to time-wasting, eventual errors and most of all: difficult to read. Of course, one could argue, why not decouple resources into function stacks? Sure, valid point - but everything depends on the use case. Even though you have the perfect tool for 98% of infrastructure topologies and ways of managing almost all examples you read about, there is always something a bit different. And that's where the fun begins. > It is worth mentioning, these were [pre-CDK](https://aws.amazon.com/blogs/developer/introducing-the-aws-cdk-public-roadmap/) times and we had not fully implemented Terraform because..reasons. So how could we define resources in a programmatic, readable (short!) and maintainable way? Yes, maybe you already guessed from the article title, [Troposphere](https://github.com/cloudtools/troposphere). ## Troposphere This is a Python library that creates AWS CloudFormation definitions (templates). These will then be used as descriptors sent to CloudFormation API to create and manage resources. In short, you define a resource using a given class and it creates a definition (template) out of it that you can then use to pass to a CloudFormation API method (like `create stack`). Sound pretty familiar huh? CDK vibes? ![](https://media.giphy.com/media/L3ERvA6jWCd0qO4NdX/giphy.gif align="center") A couple of great things about Troposphere: * Active community. * [Property and type check built in](https://troposphere.readthedocs.io/en/latest/index.html#examples-of-the-error-checking-full-tracebacks-removed-for-clarity) (meaning it will output errors if a resource is malformed or badly initialized). * It is written in (and uses) Python. * There are a lot of implementation [examples](https://github.com/cloudtools/troposphere/tree/main/examples). But not everything is roses: this is not an official AWS tool, which means it depends on community contribution to support any AWS service updates - so it can get behind, as expected. ## Quick walkthrough ### Prerequisites * An AWS account with at least a VPC in any region. * [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) configured with IAM credentials allowing to create/delete EC2 resources (security groups, subnets and instances). * (option) An existing [EC2 KeyPair](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html), if you want to test out a SSH connection to the created instances. * Python >= 3.11.2 (that's the one I tested this with). ### Installation I always recommend using a (Python) [virtual environment](https://docs.python.org/3/library/venv.html) to avoid messing around with any other locally installed version and dependencies you may have..but yeah, it's a personal preference, not required. ```bash python3 -m venv new_env source new_env/bin/activate ``` ```bash pip install troposphere ``` ### Example template(s) I added a couple of [example scripts in this repo](https://github.com/marianogg9/troposphering) to showcase the functionalities. ```bash git clone https://github.com/marianogg9/troposphering.git ``` There are two folders: ```bash . ├── README.md ├── instances │ ├── instances.py │ └── instances_input └── subnets ├── subnets.py └── subnets_input ``` Each folder contains a script and an input file, where you will add your current AWS account values, such as: * VPC id. * Resources names. * CIDRs. * Region. * (optional) Common tags. * EC2 KeyPair name. * (optional) Your local IP CIDR (if you want to test out the SSH connection). * Etc. Once you added the corresponding values, change the input files names to be: `subnets_input.json` and `instances_input.json`. > One important detail: "instances" stack is dependent on "subnets" stack as it references the subnet names from the latter. Run the scripts to generate both templates: ```python cd subnets python3 subnets.py cd instances python3 instance.py ``` The first script will create a template along these lines (defining a set of subnets in a given input VPC + exporting the values to be used by the second template later on): ```yaml Outputs: a: Export: Name: !Sub '${AWS::StackName}-a' Value: !Ref 'a' b: Export: Name: !Sub '${AWS::StackName}-b' Value: !Ref 'b' Resources: a: Properties: AvailabilityZone: a CidrBlock: some-cidr MapPublicIpOnLaunch: true Tags: - Key: Name Value: subnet-a - Key: Description Value: Playing around with CloudFormation and Troposphere - Key: CommongTag2 Value: Just to add one more VpcId: vpc-abcdefgh Type: AWS::EC2::Subnet b: Properties: AvailabilityZone: b CidrBlock: 172.30.124.80/28 MapPublicIpOnLaunch: true Tags: - Key: Name Value: subnet-b - Key: Description Value: Playing around with CloudFormation and Troposphere - Key: CommongTag2 Value: Just to add one more VpcId: vpc-abcdefgh Type: AWS::EC2::Subnet ``` And this will be the second generated file (creating an EC2 instance, in one of the above to-be-created subnets, and a set of security groups): ```yaml Outputs: firstIntanceInstanceID: Value: !Ref 'firstIntance' firstIntancePrivateIP: Value: !GetAtt 'firstIntance.PrivateIp' firstIntancePublicIP: Value: !GetAtt 'firstIntance.PublicIp' # we will use this value to test SSH connection instanceNumberTwoInstanceID: Value: !Ref 'instanceNumberTwo' instanceNumberTwoPrivateIP: Value: !GetAtt 'instanceNumberTwo.PrivateIp' instanceNumberTwoPublicIP: Value: !GetAtt 'instanceNumberTwo.PublicIp' # we will use this value to test SSH connection Resources: defaultSG: Properties: GroupDescription: This is the default Security Group SecurityGroupEgress: - CidrIp: someOtherCidr FromPort: 80 IpProtocol: tcp ToPort: 80 SecurityGroupIngress: - CidrIp: someCidr # You can add your local public IP CIDR to test the SSH connection FromPort: 22 IpProtocol: tcp ToPort: 22 - CidrIp: someOtherCidr FromPort: 123 IpProtocol: tcp ToPort: 123 Tags: - Key: Name Value: defaultSG - Key: Description Value: Playing around with CloudFormation and Troposphere - Key: CommongTag2 Value: Just to add one more VpcId: vpc-abcdefgh Type: AWS::EC2::SecurityGroup firstIntance: Properties: ImageId: ami-123 InstanceType: t2.micro KeyName: YourExistingKeyPair SecurityGroupIds: - !Ref 'defaultSG' SubnetId: !ImportValue 'AddingSubnetsWithTroposphere-someregiona' Tags: - Key: Name Value: firstIntance - Key: Description Value: Playing around with CloudFormation and Troposphere - Key: CommongTag2 Value: Just to add one more Type: AWS::EC2::Instance instanceNumberTwo: Properties: ImageId: ami-123 InstanceType: t2.micro KeyName: YourExistingKeyPair SecurityGroupIds: - !Ref 'defaultSG' SubnetId: !ImportValue 'AddingSubnetsWithTroposphere-someregionb' Tags: - Key: Name Value: instanceNumberTwo - Key: Description Value: Playing around with CloudFormation and Troposphere - Key: CommongTag2 Value: Just to add one more Type: AWS::EC2::Instance ``` These YAML templates use CloudFormation `ImportValue`, `GetAtt`, `Ref` [intrinsic functions](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html) to fetch and reference values either defined externally (like Subnet Names from the first stack) or locally. Now use those templates to create the resources: ```bash $ cd subnets $ python3 subnets.py # to create the YAML template $ aws cloudformation create-stack --stack-name AddingSubnetsWithTroposphere --template-body file://subnets_template.yaml ``` The stack name `AddingSubnetsWithTroposphere` is part of the `instances_input.json`, so if you want to use a different stack name, please remember to update it in the values before running: ```bash $ cd instances $ python3 instances.py # to create the YAML template $ aws cloudformation create-stack --stack-name AddingInstancesWithTroposphere --template-body file://instances_template.yaml ``` Each of the above will output the CFN stack id, something like: ```bash { "StackId": "arn:aws:cloudformation:::stack/AddingInstancesWithTroposphere/" } ``` **(optional)** After a few minutes (to give some time for the instances to be ready), you can test the SSH connection by: ```bash ssh -i your_key_pair ec2-user@ ``` Where `` is an output value available in the `AddingInstancesWithTroposphere` CFN stack. You can check all output values in CloudFormation console > `AddingInstancesWithTroposphere` stack > **Outputs** tab. ![](https://media.giphy.com/media/MBVemoHuyw9Ik/giphy.gif align="center") ### Don't forget to clean up! Delete both CloudFormation stacks, either via the console or using the CLI: * `aws cloudformation delete-stack --stack-name ` * It does not print any output, but you can always check in the console. ## Conclusion This was my first proper IaC tool deep dive experience, and although my teammates were the ones who implemented it from scratch, it was a great first step. Looking back I think Terraform and mostly CDK have taken over this approach, but again this is a very simple concept, pretty easy to use once you get a hold of it. The Community is super active and they are open to getting help. If you know Python and like the tool, don't think it twice and [open an issue](https://github.com/cloudtools/troposphere#community)! This implementation can use a couple more iterations like automating CFN stacks creation directly from a parent script and adding some more code reusing into the scripts regarding mapping regions or AMIs and instance types. I will be updating [the repo](https://github.com/marianogg9/troposphering), stay tuned! Last but not least as a suggestion, if you are working with CloudFormation or IaC on AWS resources, remember to always check the CloudFormation reference documentation on a given resource, e.g. for an [EC2 instance](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html). It explains which attribute can be updated on the fly without interruption (replacement) - it has saved me more than once. ## References * [Troposphere](https://troposphere.readthedocs.io/en/latest/index.html) official docs. * And its [repository](https://github.com/cloudtools/troposphere). * CNCF glossary: [Infrastructure as Code](https://www.cncf.io/blog/2020/06/29/infrastructure-as-code-a-non-boring-guide-for-the-clueless/). * AWS [CDK](https://aws.amazon.com/cdk/). * [Boto](https://boto.cloudhackers.com/en/latest/) documentation. * AWS CloudFormation [intrinsic functions](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html). * [Example implementation repo](https://github.com/marianogg9/troposphering). --- Thank you for stopping by! Do you know other ways to do this? Please let me know in the comments, I always like to learn how to do things differently.