Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support running Fabric8IstioIT with colima #1602

Open
codefromthecrypt opened this issue Mar 17, 2024 · 7 comments
Open

support running Fabric8IstioIT with colima #1602

codefromthecrypt opened this issue Mar 17, 2024 · 7 comments

Comments

@codefromthecrypt
Copy link
Contributor

codefromthecrypt commented Mar 17, 2024

Describe the bug
I can see that Fabric8IstioIT runs in the workflow, but I'm having trouble executing it locally. First, I have to disable ryuk, then it still doesn't work on account of some resources. Are there some instructions I may be missing? I run install first. Note: I am running an M2 macbook (aarch64) and using colima.

Sample

ryuk enabled (default)

[INFO] Running org.springframework.cloud.kubernetes.fabric8.client.istio.Fabric8IstioIT
13:12:18.044 [main] INFO  org.testcontainers.images.PullPolicy - Image pull policy will be performed by: DefaultPullPolicy()
13:12:18.048 [main] INFO  org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
13:12:18.264 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///Users/adrian/.colima/default/docker.sock
13:12:18.267 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is 192.168.106.2
13:12:18.278 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 24.0.9
  API Version: 1.43
  Operating System: Ubuntu 23.10
  Total Memory: 7922 MB
13:12:18.282 [main] INFO  org.testcontainers.DockerClientFactory - Checking the system...
13:12:18.282 [main] INFO  org.testcontainers.DockerClientFactory - ✔︎ Docker server version should be at least 1.6.0
13:12:18.302 [main] INFO  tc.rancher/k3s:v1.25.4-k3s1 - Creating container for image: rancher/k3s:v1.25.4-k3s1
13:12:18.349 [main] INFO  org.testcontainers.utility.RegistryAuthLocator - Credential helper/store (docker-credential-osxkeychain) does not have credentials for https://index.docker.io/v1/
13:12:18.352 [main] INFO  tc.testcontainers/ryuk:0.6.0 - Creating container for image: testcontainers/ryuk:0.6.0
13:12:18.407 [main] INFO  tc.testcontainers/ryuk:0.6.0 - Container testcontainers/ryuk:0.6.0 is starting: f55a49f5bd6f328e0709ce8155b559c9a763563487be704527956ed724036f19
13:12:18.629 [main] INFO  tc.testcontainers/ryuk:0.6.0 - Container testcontainers/ryuk:0.6.0 started in PT0.276852S
13:12:18.642 [testcontainers-ryuk] WARN  org.testcontainers.utility.RyukResourceReaper - Can not connect to Ryuk at 192.168.106.2:32773
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:549)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:757)
	at org.testcontainers.utility.RyukResourceReaper.lambda$null$1(RyukResourceReaper.java:105)
	at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
	at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$2(RyukResourceReaper.java:101)
	at java.base/java.lang.Thread.run(Thread.java:1583)
13:12:18.910 [testcontainers-ryuk] WARN  org.testcontainers.utility.RyukResourceReaper - Can not connect to Ryuk at 192.168.106.2:32773

ryuk disabled

[INFO] Running org.springframework.cloud.kubernetes.fabric8.client.istio.Fabric8IstioIT
13:16:28.644 [main] INFO  org.testcontainers.images.PullPolicy - Image pull policy will be performed by: DefaultPullPolicy()
13:16:28.646 [main] INFO  org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
13:16:28.853 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///Users/adrian/.colima/default/docker.sock
13:16:28.854 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is null
13:16:28.863 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 24.0.9
  API Version: 1.43
  Operating System: Ubuntu 23.10
  Total Memory: 7922 MB
13:16:28.863 [main] WARN  org.testcontainers.utility.ResourceReaper - 
********************************************************************************
Ryuk has been disabled. This can cause unexpected behavior in your environment.
********************************************************************************
13:16:28.865 [main] INFO  org.testcontainers.DockerClientFactory - Checking the system...
13:16:28.865 [main] INFO  org.testcontainers.DockerClientFactory - ✔︎ Docker server version should be at least 1.6.0
13:16:28.884 [main] INFO  tc.rancher/k3s:v1.25.4-k3s1 - Creating container for image: rancher/k3s:v1.25.4-k3s1
13:16:28.930 [main] INFO  org.testcontainers.utility.RegistryAuthLocator - Credential helper/store (docker-credential-osxkeychain) does not have credentials for https://index.docker.io/v1/
13:16:28.940 [main] INFO  tc.alpine:3.16 - Pulling docker image: alpine:3.16. Please be patient; this may take some time but only needs to be done once.
13:16:32.664 [docker-java-stream-1037873389] INFO  tc.alpine:3.16 - Starting to pull image
13:16:32.681 [docker-java-stream-1037873389] INFO  tc.alpine:3.16 - Pulling image layers:  0 pending,  0 downloaded,  0 extracted, (0 bytes/0 bytes)
13:16:35.335 [docker-java-stream-1037873389] INFO  tc.alpine:3.16 - Pulling image layers:  0 pending,  1 downloaded,  0 extracted, (2 MB/2 MB)
13:16:35.442 [docker-java-stream-1037873389] INFO  tc.alpine:3.16 - Pulling image layers:  0 pending,  1 downloaded,  1 extracted, (2 MB/2 MB)
13:16:35.450 [docker-java-stream-1037873389] INFO  tc.alpine:3.16 - Pull complete. 1 layers, pulled in 2s (downloaded 2 MB at 1 MB/s)
13:16:35.450 [main] INFO  tc.alpine:3.16 - Image alpine:3.16 pull took PT6.510444S
13:16:36.941 [main] INFO  tc.rancher/k3s:v1.25.4-k3s1 - Container rancher/k3s:v1.25.4-k3s1 is starting: 2c4d57bd83fdb42c2112dcd3d726c8609c840546fa0b98a65ecacb88814e720a
13:16:55.749 [main] INFO  tc.rancher/k3s:v1.25.4-k3s1 - Container rancher/k3s:v1.25.4-k3s1 started in PT26.865397S
13:17:01.057 [awaitility-thread] INFO  org.springframework.cloud.kubernetes.integration.tests.commons.Commons - error is : ctr: open /var/folders/vd/1cf8zdb1721f4z5rjggy8bp40000gn/T/spring-cloud-kubernetes-fabric8-client-istio.tar: no such file or directory

cc @wind57 as seems you have been able to get this working. ps I have already tried upgrading the docker and testcontainers drivers. PS the istio, k3s, alpine, etc versions in this test are very out of date.

@wind57
Copy link
Contributor

wind57 commented Mar 17, 2024

From what I remember, this test gave me a lot of headache too a few years back when we were moving to test containers.

Let's try to go each at a time, cause there are more then one questions here.

  • I don't know how the ryuk issue is related to the test at all, may be it's your set-up different, I can't tell.
  • I would not call the k3s version we currently used "very" out of date :) current version we are using is 1.25.4 and the most recent version is 1.28.8. Yes, we can upgrade this part, it's a single line change, if everything goes well.
  • istio is indeed old, and may be needs to be upgraded indeed, but I will look closer to it when I reach that point (somewhere around this year)

The test failure for you locally: this is what bothers me the most, cause it works for me just fine. Let me give you my steps, which are as easy as it can get.

  • I'm using docker desktop

  • first, you need java-17 for this (we do not support 21 yet):

wind57@wind57s-MacBook-Pro ~/D/p/s/s/s/spring-cloud-kubernetes-fabric8-client-istio (main)> java -version
openjdk version "17.0.1" 2021-10-19 LTS
OpenJDK Runtime Environment (build 17.0.1+12-LTS)
OpenJDK 64-Bit Server VM (build 17.0.1+12-LTS, mixed mode, sharing)

then build the image for Mac M cpus (I'm on Mac M1):

mvn clean package -Dspring-boot.build-image.builder=dashaun/builder:tiny -DskipTests

and I am simply running the test from intellij.

P.S. That error ctr: open /var/folders/vd/1cf8zdb1721f4z5rjggy8bp40000gn/T/spring-cloud-kubernetes-fabric8-client-istio.tar: no such file or directory might be related to colima set-up, I can't tell for sure.

The process here is among these lines: create the image locally; archive it via docker image save to a temp directory (/var/folders/vd/1cf8zdb1721f4z5rjggy8bp40000gn/T), this directory is mounted into the k3s container; load that tar via ctr -i import so that if you issue crictl image into the k3s - you could see it there.

Because this test passes for me, it has to do with colima, I assume.

@codefromthecrypt
Copy link
Contributor Author

good point that k3s isn't as far behind as the others. nevermind that.

my machine is in process doing the dashaun thing, and hopefully we can stop this problem forever via #1603 I keep forgetting this, but it really isn't something people should have to memorize.

will report back if things progress now.

@codefromthecrypt
Copy link
Contributor Author

ps if only java 17 is supported, I would suggest adding maven-enforcer-plugin config, for a similar reason as adding a maven profile to fix the image. Basically, people shouldn't have to remember a lot of special rules. If we decide things don't work past 17, we ought to prevent it by default. Power users can always use a property to skip enforcer.

@codefromthecrypt
Copy link
Contributor Author

ok so downgrading to JRE 17 and also rebuilding with dashaun, same error

16:26:29.032 [awaitility-thread] INFO  org.springframework.cloud.kubernetes.integration.tests.commons.Commons - error is : ctr: open /var/folders/vd/1cf8zdb1721f4z5rjggy8bp40000gn/T/spring-cloud-kubernetes-fabric8-client-istio.tar: no such file or directory

My guess indeed is this about colima as it is running in a VM, and the test may be assuming the image it is pulling outside docker is actually on the same host. It is may be possible to redo this so that it can pull the image from k3s node to avoid this problem. I won't run docker desktop as it is a licensed product and also colima is generally useful for other things. I'll just rename this to an enhancement request.

@codefromthecrypt codefromthecrypt changed the title difficulty running Fabric8IstioIT locally support running Fabric8IstioIT with colima Mar 17, 2024
@wind57
Copy link
Contributor

wind57 commented Mar 17, 2024

If I find some time, I'll take a closer look at this one, but for me, personally, it's not going to be a priority (sorry about that). Mainly because this is how it runs on the pipeline, both here in github actions and on jenkins on the spring team...

Still, it was an interesting thing to look at this one again, after so much time.

@codefromthecrypt
Copy link
Contributor Author

fyi here's how I use colima

$ colima start --cpu 4 --memory 8 --network-address

Most of the time I also add --kubernetes so that it will also start k3s by default, but anyway to reproduce, I do colima delete then recreate per above.

@codefromthecrypt
Copy link
Contributor Author

codefromthecrypt commented Apr 11, 2024

The problem here is that the current code assumes the host running the testcontainers code is the same as the docker host. This is not true in colima (nor would be for a cloud host).

Here's a working version of image copying, but I ran into other problems so the whole IT isn't working, yet.

	/**
	 * Temporary folder where to load images.
	 */
	public static final Path HOST_IMAGE_PATH = Path.of(System.getProperty("java.io.tmpdir"));
	/**
	 * Container path corresponding to {@link #HOST_IMAGE_PATH}.
	 */
	public static final Path CONTAINER_IMAGE_PATH = Path.of("/images");

	private static final K3sContainer CONTAINER =
		new FixedPortsK3sContainer(DockerImageName.parse(Commons.RANCHER))
			.configureFixedPorts(EXPOSED_PORTS)
// here don't assume the path is the same, this helps keep things more sane later
			.withFileSystemBind(HOST_IMAGE_PATH.toFile().getAbsolutePath(),
				CONTAINER_IMAGE_PATH.toString(), BindMode.READ_WRITE)
			.withCommand(Commons.RANCHER_COMMAND);

--snip--

// different version of loadImage that doesn't assume the JVM's host is the same as the docker host
	public static void loadImage(String image, String tag, K3sContainer k3s)
		throws Exception {
		DockerClient client = k3s.getDockerClient();
		image = image + ":" + tag;
		String id = client.inspectImageCmd(image)
			.exec()
			.getId()
			.replace("sha256:", "");

		Path hostPath = HOST_IMAGE_PATH.resolve(id);
		if (!hostPath.toFile().exists()) {
			LOG.info("saving image " + image + " to " + hostPath);
			try (SaveImageCmd saveCmd = client.saveImageCmd(image);
				 InputStream in = saveCmd.exec();
				 OutputStream out = Files.newOutputStream(hostPath)) {
				in.transferTo(out);
			} catch (Exception ex) {
				hostPath.toFile().delete();
				throw ex;
			}
		}

		Path containerPath = CONTAINER_IMAGE_PATH.resolve(id);
		LOG.info("copying " + hostPath + " to " + containerPath);
		k3s.copyFileToContainer(MountableFile.forHostPath(hostPath), containerPath.toString());
		LOG.info("importing image " + image + " from " + containerPath);
		Container.ExecResult exec =
			k3s.execInContainer("ctr", "images", "import", containerPath.toString());
		if (exec.getExitCode() != 0) {
			throw new RuntimeException("Failed to load image " + image + ": " + exec);
		}
	}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants