I don't think we can automatically manage such a deployment, as we can't guess which subnets to attach to. At most, we could detect such a scenario and fail with an informative error message, vs wait for CloudFormation to report error.

Would it be possible to add a parameter to select subnets in the docker-compose file like you can with the VPC and the load balancer and such? We'd be okay with manually selecting the subnets, the issue is just we can't use this tool at all with our VPC setup.

Can you please clarify your use-case: why do you have to define multiple subnets per AZ in your VPC, and how do they differ? Maybe we could detect the "right one" load balancer should be attached to.

We need a private and a public subnet in each AZ so that's why we have two different subnets.

Take a look at this documentation here which describes a similar structure to our cloud network:
https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html

The private subnet is used for our internal database services to interact and the public subnet is for internet facing things like our web application.

I asked for some input from our devops who can comment better on this and here's what he said:

Any properly secure production workload would include both private and public subnets in it's VPC. A very simplistic workload might just have a single public subnet.

The main problem is that this tool is trying to use all subnets in the VPC which wouldn't be ideal. A good VPC design would have both private and public subnets. We also would need the ability to select which AZs to use.

I assume the LB would then be connected to the public subnets, right ?
Need to check if the API tells us about a subnet nature or if we need anoter API call...

Yeah in our case it would be the public subnets that we'd want the LB to connect to. However, I could see cases though where you might want to use the load balancer on a private subnet if you were doing load balancing on an internal service that doesn't face the internet.

AWS refers to load balancers on private subnets as "internal load balancers" in their documentation. I think selecting the public subnets would be a good default but I think it would be nice if this was something that could be configured (like selecting your VPC or existing load balancer is)

Ability to use private subnets will bring more issues as we would then not be able to pull the required sidecar images from DockerHub. By design, ECS integration for Docker compose do abstract away many implementation "details" to offer an opinionated mapping, and one can't expect to get full flexibility on AWS infrastructure

Hi there - I'm having the same issue as this ticket although I don't understand why it has been closed; It's very common to have multiple public subnets in a VPC. At a minimum it should just pick the first subnet from the list just to bypass this issue - it's a real roadblock!!!

i am also running into this issue. 😞😔😔

Hi, I am also facing the same issue. Hopefully this cool integration will also covers internal subnet in the future.
Currently I am canceling the use of compose-cli and back to old ecs-cli

Same issue. What about

x-aws-subnets:
  - subnet1
  - subnet2
  - ...

@spinlud I'm highly reluctant to introduce yet more x-aws-* extensions that don't make any sense in the compose model. I wish there's a reasonable way to guess which subnet a service should be attached to.

Since you are specifying vpc it makes sense to specify also subnets. It is something already used in serverless configuration

It's a common way to create multiple subnets within a VPC and span them accross the AZs. Without the possibility to specify the subnets, you will always run into issues because of having multiple subnets within one AZ. I like the idea from @spinlud intrducing x-aws-subnets: property. For me this feature lack is a showstopper which is really sad :/

I'm highly reluctant to introduce yet more x-aws-* extensions that don't make any sense in the compose model. I wish there's a reasonable way to guess which subnet a service should be attached to.

bruh, that ship has sailed when you guys added CF overlays...which AFAIK nobody asked for...which is, like, if i would go to such lengths...I would have stuck with CloudFormation.

to me, the purpose of docker-compose is to have a great level of abstraction from AWS deployment (i'm particularly not fond of CloudFormation in general...but that's topic for another day)...and i came to an understanding that there are limitations...but this limitation, just like the inability for Load Balancer to assign it at a service level...well, just unnecessarily cripples the tool's usability.

x-aws-overlays was introduced to avoid user being blocked by lack of support for x,y,z AWS feature. This let us "buy some time" until we get a correct understanding of adequate mapping to be proposed in the compose spec. Longer term I would expect nobody actually need them but for some highly-specific scenario.

the purpose of docker-compose is to have a great level of abstraction from AWS deployment

EXACT. And also to offer reasonable portability between local development and AWS deployment. This is why I don't like we introduce even more aws-specific extension to support various AWS networking corner cases. As a compose user, I should not have to know that AWS can define security groups accross subnets in various AZ, I just want my service to run.

@ndeloof i think we agree on what docker-compose is for, but disagree on how the tool should be extensible. i think this particular feature ask is relatively benign and reasonable...and honestly i'm confused with this pushback given the floodgates were basically opened with overlays.

My team also uses the common public/private subnet architecture across AZs. The lack of ability to specify subnets is also unfortunately looking like a deal breaker for us with this otherwise awesome tool. Feels like a no-brainer feature given the ability to specify VPC. Would love to see an implementation of one of the suggestions above.

Hey @ndeloof, would you consider supporting tags on Subnets in a similar way to Kubernetes cloud providers? Kubernetes will only attach ELBs to subnets tagged with kubernetes.io/role/elb or kubernetes.io/role/internal-elb.

Could a similar model be implemented here? That way we are not adding a new x-aws variable to the Compose spec, but instead Compose would discover the tags on existing subnets, and provision the ALB accordingly?

@ollypom that could be an option, but we need to be more fine-grained: user-facing services will be attached to public subnets, the others will be attached to private onces, you can have multiple private subnets, not to mention "Internal Load Balancer" feature requests...

I've proposed a distinct approach here: #1411

Yeah, this is an issue and probably should have been thought of a tad better. This is an awesome feature docker-compose can do for sure. It is common to have a 3 tier network stack as well or more so there should be some way to support subnets.

I ran into this today and it was really difficult to figure out. However, for those reading, a solution is to put this in docker-compose.yml:

x-aws-loadbalancer: YourLoadBalancersName

I did have to create a dummy target group to be allowed to create the NLB manually, but it worked once I had manually created an NLB and specified it.

We definitely need this to be able to place ECS tasks on private subnets, otherwise it's a blocked that doesn't allow to use this amazing tool...

Ability to use private subnets will bring more issues as we would then not be able to pull the required sidecar images from DockerHub. By design, ECS integration for Docker compose do abstract away many implementation "details" to offer an opinionated mapping, and one can't expect to get full flexibility on AWS infrastructure

in most cases private subnets have access to internet via NAT or Transit gateway in another vpc, so docker images can be pulled, the only issue still remains handling of the subnets, so we need to somehow specify on which subnet load balancer loads, and in which subnets are the services.. if load balancer is in the public subnet, it also have access to any other private subnets and the services in them

This is still an issue, providing the subnets manually in this very case makes absolutely sense. Please open the ticket again.

EDIT: I solved it using the documented overlay method. Now I understand @ndeloof comment. So for any other fellow googlers:

1. Read The docs about overlays
2. Get the cloudformation script with docker --context your_ecs_context compose convert
3. Add the overlay to the end of the docker-compose file, add the subnets you'd like to use to the script for each of the services:

x-aws-cloudformation:
  Resources:
    YourService:
      Properties:
        NetworkConfiguration:
          AwsvpcConfiguration:
            Subnets:
            - subnet-xxxxxxxxx
            - subnet-yyyyyyyyy
    Your2ndService:
      Properties:
        NetworkConfiguration:
          AwsvpcConfiguration:
            Subnets:
            - subnet-xxxxxxxxx
            - subnet-yyyyyyyyy
    LoadBalancer:
      Properties:
        Subnets:
        - subnet-xxxxxxxxx
        - subnet-yyyyyyyyy
...

It's a great solution, but it wasn't obvious to me when reading the issue ticket. I hope I made it clearer for everybody. Cheers!

hi Guys,

I know this is an old post but am I using the x-aws-cloudformation correctly?

version: '3.8'

x-aws-vpc: "vpc-0f64c8ba9cb5bb10f"

services:
  osticket:
    container_name: osticket-web
    image: osticket/osticket
    environment:
      MYSQL_HOST: db
      MYSQL_PASSWORD: secret
    depends_on:
      - db
    ports:
      - 80:80
  db:
    container_name: osticket-db
    image: mariadb
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: secret
      MYSQL_DATABASE: osticket
      MYSQL_USER: osticket
      MYSQL_PASSWORD: secret
    expose:
      - "3306"
      
x-aws-cloudformation:
  Resources:
    OsticketService:
      Properties:
        NetworkConfiguration:
          AwsvpcConfiguration:
            Subnets:
              - subnet-093223fe760e52016 #public subnet-1
              - subnet-08120f88feb55e3f1 #public subnet-2
    DbService:
      Properties:
        NetworkConfiguration:
          AwsvpcConfiguration:
            Subnets:
              - subnet-0c68a298227d9c2e8 #private subnet-1
              - subnet-042cae15125ba9b1b #private subnet-2

Thank you

Just wanted to mention that it seems impossible to use Compose with ECS unless you manually create your own load balancers. Here's my experience:

Deploying with a single-AZ VPC:
VPC vpc-xxxxxxxxxxxxx should have at least 2 associated public subnets in different availability zones

Deploying with a two-AZ VPC:
A load balancer cannot be attached to multiple subnets in the same Availability Zone (Service: AmazonElasticLoadBalancing; Status Code: 400; Error Code: InvalidConfigurationRequest; Request ID: redacted; Proxy: null)

Currently experiencing this. Commenting out the private subnets for the load balancer seems to work...