Issue related to the External-IP and loadBalancer in a manged Kubernetes deployment #3
Comments
|
Just to add a bit more information, I tried to look into the log of my openldap pod in the real time. Following is the log, which seems okay and the user 'Phillip J. Fry' succeeded in signing in successfully. However, in reality, I ended up getting the same error 'undefined' in my abcdesktop browser window.
|
|
Hello, Thank you for using abcdesktop.io, i don't use the DigitalOcean Kubernetes cluster. It's the first time. Did you update the od.config file option ? You may have top replace http://localhost this your loadBalancer ip address or your FQDN ( if you set one) . Your ldap log is clear, thank you, you have to go in deep : can you read the pyos log file ? Get the pod name of pyos You should read Then read the log of a (or all) pyos pods The pod create process should hang with an error. Can you also check if your user pod is created ? I use the Thank you for your feedback |
|
Hello Vinay, I have installed a kubernetes cluster on Before using a Choose one of nginx front, and run the port forward command line For example Then open your web browser, you should get the home page, login using
Then login, and you get a pod user.
The user See you, Alexandre |
|
Hello Vinay, To use a LoadBalancer in digitalocean. Could you please update the nginx service section in After the a You get the ip address for your Then open your web browser to reach the abcdesktop service.
Login using
And you should get the
See you, Alexandre |
|
Dear Alexandre, Thank you very much for the help in figuring out the issue. I confirm that the DigitalOcean deployment of the abcdesktop works now perfectly fine as you illustrated in the above when I used the version-3.0(https://github.com/abcdesktopio/conf/blob/main/kubernetes/abcdesktop-3.0.yaml). I was somehow using an older version from (https://raw.githubusercontent.com/abcdesktopio/conf/main/kubernetes/abcdesktop.yaml). From the older version, I had two of the pods (as shown below in the Status column) failing to start. I have no moved on to the version-3, so I do not have those old logs from the
The next I am setting up a MinIO storage facility and aiming to map the ABCDesktop user's home directory to enable a dedicated (persistent?) data storage to make things meaningful with respect to my application that needs a data persistence and exchange. While looking into the abcdesktop installation information, I found some information regarding the persistentVolume . Does the 'pyos' version 3 support such thing? Your insightful suggestion on this side would be very much appreciated. thanks a lot. |
|
Hello Vinay, Thank you for your message, and for this feedback. 'pyos' version 3 only supports HostPath ( for a kubernetes limitation, subpath is not supported in ephemeral container, and ephemeral container need to know the user home directory path ) But i like your design with Minio to store user volume. Thank you again for your comments and for this new design request with S3/Minio Alexandre |
|
Dear Alexandre, Thanks again for considering to look into the minIO integration to enable persistentVolumeClaim. If it works, it will be very helpful feature. I will wait for this feature. After I make the changes in the od.conf, I delete and restart the configmap using the following three commands, my authmanager dictionary in the od.config looks like as below |
|
Hello, Some entries are missing in documentation ( I have to fix it, thank you ) Here is a provider dict with the
Thank you Regards, |
|
Great, thank you very much. regards, |
|
Hello Vinay, Thank you for your message. The last Using
|
| var | description |
|---|---|
| cn | Common Name |
| uid | user id |
| gid | group id |
| uidNumber | user id number |
| gidNumber | group id number |
| homeDirectory | homeDirectory |
| loginShell | loginShell |
| description | description |
| groups | groups |
| gecos | gecos |
| provider | provider |
| protocol | protocol |
| providertype | providertype |
| name | user name |
| userid | user id |
| locale | user's locale |
| template tag value | tag value set by auth rules |
Note:
hostPathsupports file permissions and the pod's init commandschownorchmodcan be used.
Using CSI for S3 k8s-csi-s3
I was looking for a csi S3 driver for kubernetes https://kubernetes-csi.github.io/docs/drivers.html but
I can't find an official AWS support for S3 as a CSI.
I've only found two fuse drivers :
- from ctrox here https://github.com/ctrox/csi-s3.git [ I can't make it work ]
- from Yandex-cloud here https://github.com/yandex-cloud/k8s-csi-s3
Please, let me know if you can find an official driver for S3 as a CSI.
Install a CSI for S3
I use the https://github.com/yandex-cloud/k8s-csi-s3 as a CSI for S3 with minio backend.
Follow https://github.com/yandex-cloud/k8s-csi-s3 setup guide and test with the sample pod to make sure that fuse mount the S3 file system.
Update storageclass.yaml
Add bucket: abcdesktop to use an existing bucket
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: csi-s3
provisioner: ru.yandex.s3.csi
parameters:
mounter: geesefs
# you can set mount options here, for example limit memory cache size (recommended)
options: "--memory-limit 1000 --dir-mode 0777 --file-mode 0666"
# to use an existing bucket, specify it here:
bucket: abcdesktop
csi.storage.k8s.io/provisioner-secret-name: csi-s3-secret
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
csi.storage.k8s.io/controller-publish-secret-name: csi-s3-secret
csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
csi.storage.k8s.io/node-stage-secret-name: csi-s3-secret
csi.storage.k8s.io/node-stage-secret-namespace: kube-system
csi.storage.k8s.io/node-publish-secret-name: csi-s3-secret
csi.storage.k8s.io/node-publish-secret-namespace: kube-systemUpdate od.config
In your od.config file, define the entry desktop.persistentvolumeclaimspec
desktop.homedirectorytype: 'persistentVolumeClaim' to use the persistentVolumeClaim features.desktop.persistentvolumespec: None to skip the persistent volume provisioning.desktop.persistentvolumeclaimspeccreate a new volume claim for the user's homeDir, the storageClassNamecsi-s3
# set to persistentVolumeClaim
desktop.homedirectorytype: 'persistentVolumeClaim'
desktop.persistentvolumespec: None
desktop.persistentvolumeclaimspec: {
'storageClassName': 'csi-s3',
'resources': {
'requests': {
'storage': '1Gi'
}
},
'accessModes': [ 'ReadWriteOnce' ] }
The mounter options
By default the storageclass use mounter: geesefs. geesefs does not store file permissions and the init commands chown or chmod exit with no zero value, then the pod does not start.
All files belongs to root, but with correct permissions options: "--memory-limit 1000 --dir-mode 0777 --file-mode 0666".
Update the 'init' in desktop.pod dict
'init': {
'image': 'busybox',
'enable': True,
'pullpolicy': 'IfNotPresent',
'securityContext': {
'runAsUser': 0,
'runAsGroup': 0
},
'acl': { 'permit': [ 'all' ] },
'command': [ 'sh', '-c', 'chown {{ uidNumber }}:{{ gidNumber }} ~ || true && chmod 750 ~ || true' ]
},
I can't make it works with another mounter like mounter: s3fs, but it should work too and support the file permissions.
kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-81a65ed9-b98e-462c-86c6-36c89c3d4f1b 1Gi RWO Delete Bound abcdesktop/github-12896316-96cb5 csi-s3 2m46skubectl get pvc -n abcdesktop
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
github-12896316-96cb5 Bound pvc-81a65ed9-b98e-462c-86c6-36c89c3d4f1b 1Gi RWO csi-s3 2m21sOn the minio console you get the objects
Feel free to give me your feedback. I hope that k8s-csi-s3 is stable enough.
Alexandre
|
Hi Vinay, The is an issue on k8s-csi-s3 yandex-cloud/k8s-csi-s3#45 We need to add |
|
Hi Vinay, Digitalocean provides a I share the od.config file, the # set to persistentVolumeClaim
desktop.homedirectorytype: 'persistentVolumeClaim'
desktop.persistentvolumespec: None
desktop.persistentvolumeclaimspec: {
'storageClassName': 'do-block-storage',
'resources': {
'requests': {
'storage': '1Gi'
}
},
'accessModes': [ 'ReadWriteOnce' ] }See you, Alexandre |
|
Dear Alexandre, Thank you for adding persistentVolumeClaim feature in the abcdesktopio/oc.pyos:3. This is excellent and works flawlessly. I struggled with hosting a S3 gateway, but the do-block-storage works very nicely. Thank you very much for the help in setting up the abcdesktop on cloud cluster. In the meantime, I also tried to enable ssl certificates using a rather cumbersome approach, which still has some issues, but I am debugging it. I may seek your advice if I do not succeed. I am using ingress-controller, cert-manager, nginx-ingress resource and in the end I use a networkpolicy to enable communication between the the two loadBalancers (ingress-controller and abcdesktop-nginx) pods to communicate to allow passing the security http01 challenge. This tool chain looks a bit complex and I am not sure if this is the right way to do this. thanks again. |
|
Dear Vinay, I've just run a test with an external domain, get a SSL certificat with Letsencrypt. Digital ocean does it for you. The SSL is established to the digital ocean (HTTPS) LoadBalancer, then incoming request goes into nginx HTTP. ---
kind: Service
apiVersion: v1
metadata:
name: nginx
namespace: abcdesktop
annotations:
service.beta.kubernetes.io/do-loadbalancer-protocol: "http"
service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
service.beta.kubernetes.io/do-loadbalancer-protocol: "https"
service.beta.kubernetes.io/do-loadbalancer-certificate-id: "xxxxxx-yyyy-zzzzz-aaaaa-bbbbbbbbbbb"
service.beta.kubernetes.io/do-loadbalancer-disable-lets-encrypt-dns-records: "true"
spec:
type: LoadBalancer
selector:
run: nginx-od
ports:
- protocol: TCP
port: 80
targetPort: 80
name: http
- protocol: TCP
port: 443
targetPort: 80
name: https
---See you |
|
Dear Alexandre,
regards, |
|
Hello Vinay, Thank you for your message, you've done a great job. The To by pass you can
Then run the curl command to reach the manager -- OR --
Example Then run the curl command with the X-API-Key header to reach the manager See you |
|
Dear Alexandre, Thank you for this very helpful note. I have been using a temporary workaround to run I am facing another issue with the a pod's description looks as below. I could not figure out what is happening. I had to delete my 'loadbalancer' from the digitalocean as a last resort, but that did not help. So, in the end, I pulled 'abcdesktopio/oc.nginx:3.0' and checked if there is regards, |
|
Hi Hi regards, |
|
Hello Vinay, Nice to get some news of your deployment, thank you. Why ?
ChangesCommandYou don't need to set the command parameter. I'm sorry for this update but abcdesktop requires lua script support to read the encrypted (RSA) jwt payload value. If you remove It should work. The complete nginx updated section RolesThe ClusterRole have been reduced to simple roles ( in the abcdesktop-3.0.yaml ) by default. DaemonSetDaemonSet have been replace by Deployment for Feel free to send me some feedback if nginx pod does not start Alexandre |
|
Dear Alexandre, Thank you for the note of confirmation to these changes. Niginx pods work fine now. On a different issue: |
|
Hello Vinay, I confirm this issue. octave does not start in gui mode and exit with code 0. A local command returns the octave prompt We need to add the parameter And octave runs, I'm fixing it Thank you for this issue Alexandre |
|
Hello Vinay, The octave issue is fixed. Please update
for pyos deploymentJust delete the for octaveThe new repo is abcdesktopio/images
The first start will pull the octave image, it can take a while 
This image is an ubuntu:22.04 with Thank you for your comments Alexandre |
|
Dear Alexandre, Thank you very much. This resolved octave issue, and it works fine now. regards, |
|
Dear Vinay, Thank you for your message, I really appreciate your feedbacks See you, |
|
Dear Alexandre, My example application is the non-standard application, my own python app, Case-2: I am logged-in from my GitHub account. The cluster node in this case is
To update my application container, I follow the following procedure. *Note: I am not using port-forwarding to add application. I have two versions of
Best regards, |
|
Hi Vinay, Your latest application has two folders
Your screenshots describe that applications image are not the same one on each node.
How to fix and make latest image pulled on each nodes ?Solution 1 : ManuallyPlease use this one first, it’s easiest. I suppose you are running containerd For each nodes ( in your case please choose
Solution 2 :Update od.config file to force the image policy for Then apply od.config changes and restart pyos pod Solution 3 :By default abcdekstop pyos account is based on a
Delete the default pyos rbac roles and update the new one with ClusterRoleAdd label to your working nodesRead the page on the doc website https://www.abcdesktop.io/3.0/config/desktop/#desktopnodeselector Change od.configAdd desktop.nodeselector option in od.config Restart pyos podThen you have to send your json file again, using Best regards, |
|
Dear Alexandre, Here's the JSON file I use to generate Dockerfile of promo application. |
|
Hello Vinay, Your promo application is executed as a container, and '/promo_app/' content comes from the default image.
If you can't, because you can't change the applications source code. There's is a workaround, but keep in mind that split application binary and application's data should be great.
To create
Example firefox application add a script in the directory /composer/init.d When firefox application starts, it creates the Download directory in the user's home dir.
The init file name suffix must be the same as your basename's application binary For firefox The basename command returns In your case, I suppose you get So you need to create a script file composer/init.d/init.run.py I hope this can help promo users to keep computed datas. Best regards, |
|
Thank you very much, Alexandre! On a slightly unrelated issue, I was thinking if there could be a way around to run a few applications from the terminal. For example, I have gnuplot and octave (currently I have Octave installed as an application in the ABCDesktop) that I was wondering to access from the command line. Let me be a bit more specific what I mean. In my promo application GUI, I have a small (rxrvt) terminal and from there I can for example run python, likewise I wish to run gnuplot and octave. I tried to add gnuplot and octave in as below, but this failed to build the docker. Is this correct way I am doing and is this possible to achieve? best regards, |
|
Vinay, Thank's your your reply. Only the user's home dir can keep your data persistent and accessible.
Look at your home directory, and check the content of the directory It's allow you to start another applications ( so another container ). I mean a container can ask the control plane to start another one. For example I'm running And the 2048 starts 
Of course you can build very big image with all in one (gnuplot, rxrvt, python, promo applications), but abcdesktop offers another design, each application is running as a container. I don't know why gnuplot and octave can't be build. I will check that point. Thank you for your comments Alexandre |
|
Dear Alexandre, I already have Octave GUI app and for other plotting related tasks I changed from the gnuplot to the the matplotlib and use Jupyterlab. thanks for the help. |
Dear Alexandre,
I was attempting to deploy the ABCDesktop (kubernetes-version-3.0) for testing my setup with a few of my own desktop applications. Everything worked fine when tested locally in my Ubuntu (22.04) machine. I then thought to deploy the setup in a DigitalOcean Kubernetes (DOK) cluster (I currently have a 3 nodes cluster). Because the DOK is a managed Kubernetes, so it was straight forward to use the context and deploy using the install.sh script (https://raw.githubusercontent.com/abcdesktopio/conf/main/kubernetes/install.sh), just for the testing purpose. To access the deployed ABCDesktop from an external IP, I changed the NGINX service type from nodePort to loadBalancer in the abcdesktop.yaml, as below. Saved and applied the chnages using 'kubectl apply -f abcdesktop.yaml'. I can now access the abcdesktop from 'http://<EXTERNAL-IP>', but I am unable to sign-in using login usernames and passwords from the OLDAP test samples. 'Sign-in Anonymously also fails. I get the error message:'undefined'. Any suggestions as to what I may be doing wrong here would be very much appreciated. Following are some of the details.
changes in the nginx service block within the abcdesktop.yaml
some additional information about the status of services and nodes in my deployment.
$kubectl get service -w$kubectl get nodes -o=wide$kubectl get services -n abcdesktopThe text was updated successfully, but these errors were encountered: