# Airflow in ECS with Redis - Part 3: docker compose Previously in [Deploying Airflow in ECS using S3 as DAG storage via Terraform](https://blog.mariano.cloud/airflow-in-ecs-with-redis-part-2-hands-on), I described how to deploy all components in AWS ECS using a hybrid EC2/Fargate launch type and S3 as DAG storage. Now let's do the same, but with three main differences: ![James Franco same same](https://media.giphy.com/media/C6JQPEUsZUyVq/giphy.gif align="center") * Using `docker compose` [integration with AWS ECS](https://docs.docker.com/cloud/ecs-integration/), which uses AWS CloudFormation behind curtains, instead of doing it via Terraform. * Fewer hybrid tools, you just need [Docker Desktop](https://docs.docker.com/desktop/) installed locally. * All components running on Fargate ECS launch type. * AWS EFS as DAG storage instead of S3. * No need to worry about S3 mount drivers, `docker compose` natively [integrates with AWS EFS](https://docs.docker.com/cloud/ecs-integration/#volumes). # Components ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1677836137173/7ac0d1cb-affc-4930-b27b-2764aa7b6a7a.png align="center") # Deploy ## Prerequisites * [Docker Desktop](https://docs.docker.com/desktop/). * [AWS CLI configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) in your local using a set of credentials with: * Baseline [required permissions](https://docs.docker.com/cloud/ecs-integration/#requirements). * Additional permissions: * `ec2:DescribeVpcAttribute` * `elasticfilesystem:DescribeFileSystems` * `elasticfilesystem:CreateFileSystem` * `elasticfilesystem:DeleteFileSystem` * `elasticfilesystem:CreateAccessPoint` * `elasticfilesystem:DeleteAccessPoint` * `elasticfilesystem:CreateMountTarget` * `elasticfilesystem:DeleteMountTarget` * `elaticfilesystem:DescribeAccessPoints` * `elasticfilesystem:DescribeMountTargets` * `elasticfilesystem:DescribeFileSystemPolicy` * `elasticfilesystem:DescribeBackupPolicy` * `logs:TagResource` * `iam:PutRolePolicy` * `iam:DeleteRolePolicy` * [VPC](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-getting-started.html) + [subnets](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-subnets.html#create-subnets). * Security Group to be used in this deployment. * Create the Security Group. ```bash aws ec2 create-security-group --group-name Airflow --description "Airflow traffic" --vpc-id "" ``` It creates a default egress rule to `0.0.0.0/0`. Take note of the output `GroupId`. ```bash { "GroupId": "take note of this value" } ``` * Add internal rules to it. * Self traffic. ```bash aws ec2 authorize-security-group-ingress --group-id "" --protocol all --source-group "" --ip-permissions IpProtocol=-1,FromPort=-1,ToPort=-1,IpRanges="[{CidrIp=,Description='Allow VPC internal traffic'}]" ``` * (optional) Add a rule for your public IP to access ports `5555` (Flower service) and `8080` (Webserver service). ```bash aws ec2 authorize-security-group-ingress --group-id "" --ip-permissions IpProtocol=tcp,FromPort=5555,ToPort=5555,IpRanges="[{CidrIp=,Description='Allow Flower access'}]" IpProtocol=tcp,FromPort=8080,ToPort=8080,IpRanges="[{CidrIp=,Description='Allow Webserver access'}]" ``` **NOTE:** there is [currently no way](https://github.com/docker/compose-cli/issues/1783) (natively) of avoiding CloudFormation to create a `0.0.0.0/0` rule in the SG for exposed ports in declared services. If you need to narrow down this access, you will have to delete the additional rules from the SG while `docker compose` creates the ECS services. ## Steps * Clone the repo: ```bash $ git clone https://github.com/marianogg9/airflow-in-ecs-with-compose local_dir $ cd local_dir ``` * Set required variables in `docker-compose.yaml`: ```yaml x-aws-vpc: "your VPC id" networks: back_tier: external: true name: "" ``` * (optional) If you want to use a custom password for the Webserver admin user (default user `airflow`): * This password will be created as an AWS Secrets Manager [secret](https://docs.docker.com/cloud/ecs-integration/#secrets) and its ARN will be passed as an environment variable with the following format: ```yaml secrets: name: _AIRFLOW_WWW_USER_PASSWORD valueFrom: ``` * Add a custom password in a local file: ```bash echo 'your_custom_password' > ui_admin_password ``` * Add a `secrets` definition block in `docker-compose.yaml`: ```yaml secrets: ui_admin_password: name: _AIRFLOW_WWW_USER_PASSWORD file: ./ui_admin_password.txt ``` * Add a `secrets` section in each service to mount to: ```yaml secrets: - _AIRFLOW_WWW_USER_PASSWORD ``` * Add the following required AWS Secrets Manager permissions to the IAM credentials you set `docker context` to use. * `secretsmanager:CreateSecret`. * `secretsmanager:DeleteSecret`. * `secretsmanager:GetSecretValue`. * `secretsmanager:DescribeSecret`. * `secretsmanager:TagResource`. * and please narrow down the above permissions to the secret ARN: `arn:aws:secretsmanager:::secret:AIRFLOWWWWUSERPASSWORD*` * Create a new `docker` context, selecting a preferred method of obtaining IAM credentials (either via environment variables, a named local profile or a set of `key:secret`): ```bash docker context create ecs new-context-name ``` * Use the newly created context: ```bash docker context use new-context-name ``` * (optional) Review CloudFormation template to be applied, via: ```bash docker compose convert ``` * Deploy: ```bash docker compose up ``` Once the deployment starts, `docker compose` will show updates on screen. You can also follow up on the resources creation in the (AWS) CloudFormation console. ### Web access * Get NLB name: ```bash aws elbv2 describe-load-balancers | grep DNSName | awk '{print$2}' | sed -e 's|,||g' ``` Or if you have `jq` installed: ```bash aws elbv2 describe-load-balancers | jq .LoadBalancers[].DNSName ``` * Webserver: `http://NLBDNSName:8080`. Login with `airflow:airflow` or `airflow:your_custom_created_password`. * Flower: `http://NLBDNSName:5555`. # Running an example pipeline I have included an [example DAG](https://github.com/marianogg9/airflow-in-ecs-with-compose/blob/main/example-dag/process-employees.py) (from [Airflow's examples](https://airflow.apache.org/docs/apache-airflow/stable/tutorial/pipeline.html)) in the repo. This DAG is also being fetched by `airflow-scheduler` task in startup time, so it will be available in Webserver UI. As a prerequisite, create a PostgreSQL connection (to be then used by the DAG). In Webserver UI > Admin > Connections > Add `+`. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1677755729394/6a0403a5-b50f-4495-9151-df48892e109f.png align="center") With the following parameters: * Connection Id: `tutorial_pg_conn`. * Connection Type: `postgres`. * Host: `postgres`. * Schema: `airflow`. * Login: `airflow`. * Password: `airflow`. * Port: `5432`. Then you can Test the connection, and if it passes, you can Save it. Back to DAGs list, unpause the DAG `process-employees` and it will automatically run: ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1677776807899/10b09f10-6bd8-4a1f-bfb1-8ec8af106782.png align="center") We can check the tasks being run (click in Last Run column link): ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1677776858558/96d71214-252c-4b23-bd11-93a8c5617840.png align="center") And have a look at Flower UI to followup on tasks vs workers: ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1677776894369/fef77f6c-8748-468f-9f7b-7b626b884fe2.png align="center") **FIN!** Now you can start adding/fetching DAGs from other sources as well, by modifying the fetch step in `airflow-scheduler` startup script and running `docker compose up` again to apply changes. # Clean up Once you are done, you can delete all created resources with: ```bash docker compose down ``` > **Important: remember to delete EFS volumes manually!!** > > `docker compose` integration creates EFS volumes with `retain` policy, so that whenever a new deployment occurs, it can reuse them. > > Please [see the official documentation](https://docs.docker.com/cloud/ecs-integration/#volumes) for more info. Also, remember to delete the SG and custom rules. * First the rules. ```bash aws ec2 revoke-security-group-ingress --group-id "" --security-group-rule-ids ``` ```bash aws ec2 revoke-security-group-ingress --group-id "" --ip-permissions IpProtocol=tcp,FromPort=5555,ToPort=5555,IpRanges="[{CidrIp=,Description='Allow Flower access'}]" IpProtocol=tcp,FromPort=8080,ToPort=8080,IpRanges="[{CidrIp=,Description='Allow Webserver access'}]" ``` ```bash aws ec2 revoke-security-group-ingress --group-id "" --ip-permissions IpProtocol=-1,FromPort=-1,ToPort=-1,IpRanges="[{CidrIp=,Description='Allow VPC internal traffic'}]" ``` * Then the SG. ```bash aws ec2 delete-security-group --group-id "" ``` # Gotchas & comments * `docker compose` outputs are not very descriptive and only show one error at a time -> to understand AWS access errors, I used CloudTrail (Warning! heavy S3 usage). * `docker compose` sometimes fails silently using `ecs` context. If you happen to face this, go back to default context with `docker context use default`, try to debug the errors (now they will be shown on screen) and then go back to `ecs` context `docker context use new-context-name`. See [this issue](https://github.com/docker/compose-cli/issues/2068) for more info. * `servicediscovery:*` permissions refer to CloudMap. * Workers and Webserver `.25 vCPU | .5 GB` is too few. Get it up to 2GB. * Don't forget to delete EFS volumes manually!! This `docker compose` ECS integration will define Docker volumes with `retain` policy, so they [will **not** be deleted automatically](https://docs.docker.com/cloud/ecs-integration/#volumes) with `docker compose down`. * `service.deploy.restart_policy` is [**not** supported](https://github.com/docker/compose-cli/issues/2153) even though the [documentation says it is](https://docs.docker.com/cloud/ecs-compose-features/). * You can configure POSIX permissions on the AWS EFS access points by adding `volumes.your-volume.driver_opts` as described in [volumes section](https://docs.docker.com/cloud/ecs-integration/#volumes). Also, trying to figure out how to mount the `/dags` directory from AWS EFS correctly on the containers, [setting rootdir and permissions](https://github.com/docker/compose-cli/issues/1459) is also allowed and supported. # Conclusion This `docker compose` ECS integration allows rolling updates as well. Since it relies on AWS CloudFormation, you could get a working baseline version (e.g. PostgreSQL, Redis, Scheduler) and then add the Webserver or Worker services without having to start from scratch, or modify existing resources. Simply updating your `docker-compose.yaml` locally and running `docker compose up` again will apply any new changes. This new run translates to a [CloudFormation Stack update](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks.html). This integration gives a lot of flexibility and offers an abstraction layer if you don't want to deal with external dependencies. In doing so, it will assume and configure a lot of settings for you, which for some use cases will not be ideal. If this is your case, then it might make more sense to tweak the underlying CloudFormation template being generated. Overall a great experience, with lots of unknowns and learnings. The documentation is not that extensive, it seems to be very practical as it won't go into deep details but gets the job done. Still some way to go, please check out this integration [issues and enhancements](https://github.com/docker/compose-cli/issues). # References * [Run Airflow in Docker](https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html). * [Docker compose ECS integration](https://docs.docker.com/cloud/ecs-integration/). * This implementation [repository](https://github.com/marianogg9/airflow-in-ecs-with-compose/blob/main/example-dag/process-employees.py). * [Airflow on Fargate (AWS blog)](https://aws.amazon.com/blogs/containers/running-airflow-on-aws-fargate/). * [Run apps in ECS using docker compose (AWS blog)](https://aws.amazon.com/blogs/containers/deploy-applications-on-amazon-ecs-using-docker-compose/). * [Part 1 of this series](https://blog.mariano.cloud/airflow-in-ecs-with-redis-part-1-overview). * [Part 2 of this series](https://blog.mariano.cloud/airflow-in-ecs-with-redis-part-2-hands-on). # **Improvements** * Add a reverse proxy [**https://airflow.apache.org/docs/apache-airflow/stable/howto/run-behind-proxy.html**](https://airflow.apache.org/docs/apache-airflow/stable/howto/run-behind-proxy.html). * HTTPS support. * Explore [Fargate Spot](https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/ec2-and-fargate-spot.html). * Deploy in Kubernetes. --- Thank you for stopping by! Do you know other ways to do this? Please let me know in the comments, I always like to learn how to do things differently.