A problem with a demo

[martynia]

Hi,

I follow the instruction and got stuck with a demo:
The output is below:

What could be a reason for this ?

cheers, JM

🦄 Installing DiracX...
W0927 13:03:44.631471 16664 warnings.go:70] unknown field "spec.template.spec.volumes[0].sizeLimit"
W0927 13:03:44.631573 16664 warnings.go:70] unknown field "spec.template.spec.volumes[2].empty-dir"
W0927 13:03:44.631593 16664 warnings.go:70] unknown field "spec.template.spec.volumes[2].sizeLimit"
W0927 13:03:44.631608 16664 warnings.go:70] unknown field "spec.template.spec.volumes[4].sizeLimit"
NAME: diracx-demo
LAST DEPLOYED: Wed Sep 27 13:03:35 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:

1. Get the application URL by running these commands:
   http:///
   🦄 Waiting for installation to finish...
   error: timed out waiting for the condition on pods/diracx-demo-cd88485cd-bwhmc
   ⚠️ Installation did not start sucessfully!

ℹ️ Press Ctrl+C to clean up and exit

[chrisburr]

You'll probably need to poke around with kubectl to figure out what's going on. See the README for a starting point:

https://github.com/DIRACGrid/diracx-charts#interacting-with-the-demo

At the moment the demo doesn't print the variables you need to set if something fails but if you're in the chart's repo they will be:

export KUBECONFIG="$PWD/.demo/kube.conf"
export HELM_DATA_HOME="$PWD/.demo/helm_data"
export PATH=${PATH}:$PWD/.demo

[martynia]

[chrisburr]

The errors above were caused by an incorrect version (apparently) of docker.

Why do you think this? If that is the case we should test the docker version put a more helpful error message.

The fact that I needed sudo was a bit worrying (docker is configured in such a way that docker commands don't require sudo).

There is no need to use sudo and the script has been working without it on Ubuntu/Fedora/Arch/macOS.

Since everything is cleaned up no much can be done with kubectl. Nothing is running in docker. Timeout (highlighted above) seems to be a problem.

Your original run of the demo seems to have been much more successful. I'd suggest trying to get back to that state and sharing the output of running kubectl get pods after set the three environment variables I mentioned. (It should now print them to the terminal if you update the repositories.

[martynia]

OK,
I realised that the installation of Kubernetes was a mistake. I went the original route:

1. sudo: this is caused by wrong permission in the cloned repository, i.e.
   [rocky@diracx-r9 diracx-charts]$ ls -l /home/rocky/diracx-charts/.demo/demo_cluster_conf.yaml
   -rw-r--r--. 1 root root 1040 Oct 3 10:16 /home/rocky/diracx-charts/.demo/demo_cluster_conf.yaml

(and I did not clone with sudo ...)

So, so far but only installing docker I'm reaching the same stage as in my first attempt:

`validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
🦄 Waiting for ingress controller to be created...
pod/ingress-nginx-controller-77f58bf6cd-x6lqp condition met
🦄 Installing DiracX...
W1003 10:21:26.735598 114666 warnings.go:70] unknown field "spec.template.spec.volumes[0].sizeLimit"
W1003 10:21:26.735686 114666 warnings.go:70] unknown field "spec.template.spec.volumes[2].empty-dir"
W1003 10:21:26.735711 114666 warnings.go:70] unknown field "spec.template.spec.volumes[2].sizeLimit"
W1003 10:21:26.735723 114666 warnings.go:70] unknown field "spec.template.spec.volumes[4].sizeLimit"
NAME: diracx-demo
LAST DEPLOYED: Tue Oct 3 10:21:17 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:

1. Get the application URL by running these commands:
   http:///
   🦄 Waiting for installation to finish...
   error: timed out waiting for the condition on pods/diracx-demo-f4b4dbf55-w62mf
   ⚠️ Installation did not start sucessfully!:`

From another terminal window I can explore the installation:
I've set the env. vars and:

(sorry for that sudo, it makes things look awful)

cluster info

[rocky@diracx-r9 diracx-charts]$ sudo /home/rocky/diracx-charts/.demo/kubectl cluster-info  --context kind-diracx-demo --kubeconfig /home/rocky/diracx-charts/.demo/kube.conf
Kubernetes control plane is running at https://127.0.0.1:43689
CoreDNS is running at https://127.0.0.1:43689/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

(I have done the dump and looked for some problematic places, but it is quite a lot of it, not sure what to search for.

Although there is a hint when describing pods, see below)

pods
sudo /home/rocky/diracx-charts/.demo/kubectl get pods --context kind-diracx-demo --kubeconfig /home/rocky/diracx-charts/.demo/kube.conf NAME READY STATUS RESTARTS AGE diracx-demo-dex-64856f8d5b-485cr 1/1 Running 0 64m diracx-demo-f4b4dbf55-w62mf 0/1 Init:CrashLoopBackOff 14 (4m37s ago) 64m diracx-demo-minio-5d87f67c7d-dl58t 1/1 Running 0 64m diracx-demo-mysql-0 0/1 Running 16 (90s ago) 64m diracx-demo-rabbitmq-0 1/1 Running 0 64m opensearch-cluster-master-0 1/1 Running 0 64m

Describe a problematic pod

sudo /home/rocky/diracx-charts/.demo/kubectl describe pod/diracx-demo-f4b4dbf55-w62mf --context kind-diracx-demo --kubeconfig /home/rocky/diracx-charts/.demo/kube.conf

(tail of the listing)

Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Normal   Pulling  59m                    kubelet  Pulling image "ghcr.io/diracgrid/diracx/server:latest"
  Normal   Pulled   59m                    kubelet  Successfully pulled image "ghcr.io/diracgrid/diracx/server:latest" in 1.204049421s (13.839721729s including waiting)
  Normal   Created  59m                    kubelet  Created container init-cs-user
  Normal   Started  59m                    kubelet  Started container init-cs-user
  Normal   Pulling  58m                    kubelet  Pulling image "ghcr.io/diracgrid/diracx/server:latest"
  Normal   Created  58m                    kubelet  Created container init-signing-key
  Normal   Pulled   58m                    kubelet  Successfully pulled image "ghcr.io/diracgrid/diracx/server:latest" in 777.47691ms (777.533585ms including waiting)
  Normal   Started  58m                    kubelet  Started container init-signing-key
  Normal   Pulled   58m                    kubelet  Successfully pulled image "ghcr.io/diracgrid/diracx/server:latest" in 766.966703ms (766.984454ms including waiting)
  Normal   Pulled   57m                    kubelet  Successfully pulled image "ghcr.io/diracgrid/diracx/server:latest" in 975.937695ms (975.99232ms including waiting)
  Normal   Created  57m (x3 over 58m)      kubelet  Created container create-sql-db-schema
  Normal   Pulled   57m                    kubelet  Successfully pulled image "ghcr.io/diracgrid/diracx/server:latest" in 743.452172ms (743.522113ms including waiting)
  Normal   Started  57m (x3 over 58m)      kubelet  Started container create-sql-db-schema
  Normal   Pulling  39m (x9 over 58m)      kubelet  Pulling image "ghcr.io/diracgrid/diracx/server:latest"
  Warning  BackOff  4m24s (x220 over 57m)  kubelet  Back-off restarting failed container create-sql-db-schema in pod diracx-demo-f4b4dbf55-w62mf_default(9be18492-c677-4fba-8c69-eec98a347548)

Is it something wrong with the DB schema ?

cheers, JM

[chrisburr]

From that point you can do kubectl logs diracx-demo-f4b4dbf55-w62mf_default -c create-sql-db-schema to see the logs and hopefully get some idea about the problem.

[martynia]

I went further w/o sudo, still a timeout, though.:

`ℹ️ To interact with the cluster directly using kubectl/helm you'll need to set the
first following environment variables:

export KUBECONFIG=/home/rocky/diracx-charts/.demo/kube.conf
export HELM_DATA_HOME=/home/rocky/diracx-charts/.demo/helm_data
export PATH=${PATH}:/home/rocky/diracx-charts/.demo

Then see the chart README for more information on how to use kubectl/helm.

ℹ️ To use the demo with the DiracX client set:

export DIRACX_URL=http://172.18.0.1:8000

ℹ️ To access the interactive API documentation visit:

http://172.18.0.1:8000/docs

ℹ️ When prompted to login the credentials are:

Username: admin@example.com
Password: password
🦄 Waiting for installation to finish...

error: timed out waiting for the condition on pods/diracx-demo-866c9cf6f-cvpv6
⚠️ Installation did not start sucessfully!`

Still timeout !

[chrisburr]

The path for debugging this is the same:

1. Find which pod is crashing (kubectl get pods)
2. Look at the logs (kubectl logs $POD_NAME)
3. If you don't get anything from 2 because the main container hasn't started, do kubectl logs $POD_NAME -c $CONTAINER_NAME (the available container names are printed when running kubectl logs $POD_NAME).

[chrisburr]

(Adding a centralised way of collecting the logs is being worked on, but if it's a problem starting up you still might need to fall back on looking at the pods explicitly.)

[martynia]

Running on my Mac:

ℹ️ When prompted to login the credentials are:

Username: admin@example.com
Password: password
🦄 Waiting for installation to finish...
pod/diracx-demo-79ddcb598c-9c8fx condition met
🎉 🎉 🎉 Pods are ready! 🎉 🎉 🎉

[martynia]

I looked around http://192.168.1.21:8000/docs

Will a client work on my Mac ?

[chrisburr]

Will a client work on my Mac ?

Yes, though be aware that everything is rapidly evolving and still very incomplete. We're still more focused on the foundations than any actual functionality.

[martynia]

I tried a demo again today. The installation stops at (Centos 7):

`networkpolicy.networking.k8s.io/ingress-nginx-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
🦄 Waiting for ingress controller to be created...
⚠️ Cluster is low on space (35GB free, 13%)
pod/ingress-nginx-controller-d9b6d7bf5-sxfsb condition met
🦄 Installing DiracX...
⚠️ Cluster is low on space (35GB free, 14%)
⚠️ Cluster is low on space (34GB free, 16%)
⚠️ Cluster is low on space (33GB free, 19%)
⚠️ Cluster is low on space (33GB free, 19%)
W1113 12:39:47.413773 109637 warnings.go:70] path /.well-known cannot be used with pathType Prefix
⚠️ Cluster is low on space (33GB free, 20%)
⚠️ Cluster is low on space (31GB free, 24%)
⚠️ Cluster is low on space (31GB free, 25%)
Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
* timed out waiting for the condition

⚠️ Error using helm DiracX

ℹ️ Press Ctrl+C to clean up and exit
⚠️ Cluster is low on space (31GB free, 25%)
⚠️ Cluster is low on space (31GB free, 25%)
⚠️ Cluster is low on space (31GB free, 25%)`

The "low on space" message is new, but the installation still stops with an error. Do I need any ports open or any other special settings on the node? (I doubt it since it works on my Mac w/o any adjustments)

[chaen]

you do not need anything special. Just re-run the script. Sometime it timesout (as per the message) for slowish network

[martynia]

I tried to reinstall the demo, the problem persists..

[martynia]

It installs on ubuntu 22.04.3 LTS . This was an OS problem, not the network. I tried an older Ubuntu and Centos7