ERROR: Connection refused - integration test fails

[scurtis-disco]

I am having an issue running an integration test wherein, if I run the test as a standalone unit test, it works fine. But if I run it via the command line (mvn -P docker clean install), the application code that is running in one Docker container can't talk to the LocalStackContainer that I created in the test. I have combed the documentation and tried several approaches, but have not had any luck so far. I am in no way a Docker expert but all of the documentation makes it seem that my trivial setup should be enough.

The Details:
Spring Boot application
testcontainers version - 1.17.3
localstack version - 1.17.3

<!--test--> <dependency> <groupId>org.testcontainers</groupId> <artifactId>testcontainers</artifactId> <version>1.17.3</version> <scope>test</scope> </dependency> <dependency> <groupId>org.testcontainers</groupId> <artifactId>localstack</artifactId> <version>1.17.3</version> <scope>test</scope> </dependency>

The setup:

// I TRIED THE CUSTOM NETWORK SETUP HERE, BUT IT DID NOT WORK @Container private static LocalStackContainer container = new LocalStackContainer(DockerImageName.parse(LOCALSTACK_DOCKER)) // .withNetwork(NETWORK) // .withEnv("HOSTNAME_EXTERNAL", "localstack") .withEnv("TESTCONTAINERS_HOST_OVERRIDE", "localhost") .withServices(LocalStackContainer.Service.S3, SQS) .waitingFor(Wait.forHttps("/").forPort(4566).forStatusCode(200).withStartupTimeout(Duration.ofMinutes(2L))) .withClasspathResourceMapping("/localstack", "/docker-entrypoint-initaws.d", BindMode.READ_ONLY) .waitingFor(new LocalStackReadyLogWaitStrategy()) .withStartupTimeout(Duration.ofMinutes(3L));

Log output of container:
`2022-11-10 22:47:08.237 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mo.t.c.l.LocalStackContainer �[m : HOSTNAME_EXTERNAL environment variable set to 172.17.0.1 (to match host-routable address for container)
2022-11-10 22:47:08.242 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mo.t.c.GenericContainer �[m : Creating container for image: localstack/localstack:0.14.3
2022-11-10 22:47:08.243 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mo.t.u.RegistryAuthLocator �[m : Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: localstack/localstack:0.14.3, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
2022-11-10 22:47:08.943 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mo.t.c.GenericContainer �[m : Container localstack/localstack:0.14.3 is starting: 012ef87b96b68eba255ff7abfddb057caaf34dfad86bc0d4e399e54bbe4aae5b
2022-11-10 22:47:09.258 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : waitUntilReady called in LocalStackReadyLogWaitStrategy
2022-11-10 22:47:09.259 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : waitStrategyTarget client: LazyDockerClient, container ID: 012ef87b96b68eba255ff7abfddb057caaf34dfad86bc0d4e399e54bbe4aae5b
2022-11-10 22:47:09.281 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : Waiting for all LocalStack services to be ready

2022-11-10 22:47:09.457 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : 2022-11-10 22:47:09,456 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.

2022-11-10 22:47:09.468 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : 2022-11-10 22:47:09,459 INFO supervisord started with pid 15

2022-11-10 22:47:10.463 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : 2022-11-10 22:47:10,461 INFO spawned: 'infra' with pid 20

2022-11-10 22:47:11.468 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : 2022-11-10 22:47:11,463 INFO success: infra entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

2022-11-10 22:47:12.276 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : LocalStack version: 0.14.3.1

2022-11-10 22:47:12.276 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : 2022-11-10T22:47:12.162:INFO:hypercorn.error: Running on https://0.0.0.0:4566 (CTRL + C to quit)

2022-11-10 22:47:12.277 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : [2022-11-10 22:47:12 +0000] [20] [INFO] Running on https://0.0.0.0:4566 (CTRL + C to quit)

2022-11-10 22:47:12.279 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : Starting edge router (https port 4566)...

2022-11-10 22:47:12.283 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m :

2022-11-10 22:47:12.290 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : LocalStack build git hash: 496e6015

2022-11-10 22:47:12.297 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : LocalStack build date: 2022-05-25

2022-11-10 22:47:12.299 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : LocalStack Docker container id: 012ef87b96b6

2022-11-10 22:47:12.300 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m :

2022-11-10 22:47:13.276 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : 2022-11-10T22:47:12.449:INFO:bootstrap.py: Execution of "start_runtime_components" took 602.04ms

2022-11-10 22:47:13.278 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : Ready.

2022-11-10 22:47:17.424 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m :

2022-11-10 22:47:17.425 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : setting up test bucket...

2022-11-10 22:47:17.425 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initaws.d/init.sh

2022-11-10 22:47:17.426 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initaws.d/docker-compose-localstack.yaml

2022-11-10 22:47:17.891 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : "QueueUrl": "http://localhost:4566/000000000000/test-sqs-queue"

2022-11-10 22:47:17.892 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : }

2022-11-10 22:47:17.892 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : {

2022-11-10 22:47:18.480 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : {

2022-11-10 22:47:18.480 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : }

2022-11-10 22:47:18.481 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : "Location": "/test-s3-bucket"

2022-11-10 22:47:18.525 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m :

2022-11-10 22:47:18.525 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mi.u.LocalStackReadyLogWaitStrategy �[m : Initialized queue and bucket

2022-11-10 22:47:18.526 �[32m INFO�[m �[35m${sys:PID}�[m [ main] �[36mo.t.c.GenericContainer �[m : Container localstack/localstack:0.14.3 started in PT10.284415232S`

The Error:
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 172.17.0.1:51309 [/172.17.0.1] failed: Connection refused at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ... 120 more Caused by: java.net.ConnectException: Connection refused at java.base/sun.nio.ch.Net.pollConnect(Native Method) at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672) at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 121 more

[eddumelendez]

Can you share a sample project which reproduces the error, please?

[scurtis-disco]

@eddumelendez I have pushed updates that fully replicate the way my other code is running, and now I get the connection refused error. I also updated the README. I added the following:

• example-runner: executable for running the build
• runner.sh: executable that is copied into the container
• Makefile: a "wrapper" Makefile that includes the user-defined app.mk target(s) - there is only one defined at the moment, but you get the idea
• app.mk: currently defines the make target for the build_app step
• Dockerfile: builds the container within which the code will build and run tests
• docker-compose.yaml

This setup accurately reflects the way in which our code runs today. I even made sure that everything being installed on the image is the same in terms of libraries and environment setup so it should be as close to the code I am trying to fix as possible. Oh, and I mis-spoke in my original comment above - the internal tool we use is a lightweight wrapper around AWS SAM, not Docker, so that when we build and deploy resources, they have a consistent tagging taxonomy, naming patterns, etc., and developers can spin up stacks without having to repeat all of that copy-pasta for new projects. That has nothing to do with what we're solving here, but I felt like it makes clear why I added the additional wrapper functionality to this setup.

[eddumelendez]

Sorry but not able to run it. Also, the test is not helpful. Running mvn -P docker clean install was mentioned at the beginning and now a whole setup was provided which I can not run. If the test is trying to connect spring boot with SQS and Elasticsearch, all I can say is it works. Try a small example of a test class with a test method and property assertion such as sending a SQS message and read it, the same for elastic. Avoid using the host mode. Those are the suggestions I can provide so far.

[scurtis-disco]

@eddumelendez I appreciate your feedback, but saying "not able to run it" tells me nothing actionable - what kind of error is it causing? It is as generic as I can make it. I am running it from the command-line inside an Intellij terminal window, and most of the heavy lifting occurs inside the Docker containers.

As I took the time to explain above, this represents how the code I am trying to make functional is being built. I acknowledge that a simpler setup works, but I don't have the luxury of tearing down the current infrastructure. Any chance you could log the output of your local run and attach it here so maybe I can see what happens when you try the following from the root of the project?

./example-runner ENV=whatever APP_NAME=whatever-else

Is it possible that you need to make the files that should be executable, such as example-runner and runner.sh, as executable on your local machine, via something like chmod 755 <file>? (I doubt that's the issue but it's worth a try).

Also, the test is not helpful

I know that, the test itself is meaningless, because the code never gets to the point of running the test because the Spring Boot context fails to initialize due to the connection refused error from the SQS listener attempting to verify the existence of the queue.

Try a small example of a test class with a test method and property assertion such as sending a SQS message and read it, the same for elastic.

The original code has such tests but it doesn't get to that point.

I have beat my head against the wall for days trying to solve this problem, so please believe me that all I want is to figure out a solution. I also spent a solid day on trying to architect this example so it reflects the reality of my situation, I would love to see how it runs (or doesn't) for you.

What questions do you have for me so that I can help?

[scurtis-disco]

@eddumelendez anything new to add? I was out all of last week and the first of this week.

[scurtis-disco]

No one else wants to weigh in on this one? 😄