Skip to content
This repository has been archived by the owner on Jul 7, 2020. It is now read-only.

Error waiting for job 'heketi-storage-copy-job' to complete #648

Open
mewais opened this issue Apr 22, 2020 · 4 comments
Open

Error waiting for job 'heketi-storage-copy-job' to complete #648

mewais opened this issue Apr 22, 2020 · 4 comments

Comments

@mewais
Copy link

mewais commented Apr 22, 2020

As the title says, while using the gk-deploy script, it ends in failure with the following error: Error waiting for job 'heketi-storage-copy-job' to complete.

Here's the command:
./gk-deploy -g ../../storage_topology.json --admin-key admin --user-key user

And here's the output:

Do you wish to proceed with deployment?

[Y]es, [N]o? [Default: Y]: Y
Using Kubernetes CLI.
Using namespace "default".
Checking for pre-existing resources...
  GlusterFS pods ... not found.
  deploy-heketi pod ... not found.
  heketi pod ... not found.
  gluster-s3 pod ... not found.
Creating initial resources ... serviceaccount/heketi-service-account created
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view created
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view labeled
OK
node/storage-worker-01 labeled
node/storage-worker-02 labeled
node/storage-worker-03 labeled
daemonset.apps/glusterfs created
Waiting for GlusterFS pods to start ... OK
secret/heketi-config-secret created
secret/heketi-config-secret labeled
service/deploy-heketi created
deployment.apps/deploy-heketi created
Waiting for deploy-heketi pod to start ... OK
Creating cluster ... ID: 2f6eba4d13572a03c0b0af76cd519b79
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node storage-worker-01 ... ID: ed6fea5e1c5f9a7ccb1784e3f5126d4d
Adding device /dev/sdb ... OK
Creating node storage-worker-02 ... ID: 6f1db063cf2853b790156d749dcf1e1c
Adding device /dev/sdb ... OK
Creating node storage-worker-03 ... ID: 4d6ed79b2395317bf305ebb8f20babd2
Adding device /dev/sdb ... OK
heketi topology loaded.
Saving /tmp/heketi-storage.json
secret/heketi-storage-secret created
endpoints/heketi-storage-endpoints created
service/heketi-storage-endpoints created
job.batch/heketi-storage-copy-job created
Error waiting for job 'heketi-storage-copy-job' to complete.

I used the get event command to find out the errors: kubectl get ev --field-selector involvedObject.name=heketi-storage-copy-job-hbvv4

And the output was:

LAST SEEN   TYPE      REASON        OBJECT                              MESSAGE
<unknown>   Normal    Scheduled     pod/heketi-storage-copy-job-hbvv4   Successfully assigned default/heketi-storage-copy-job-hbvv4 to storage-worker-03
7m49s       Warning   FailedMount   pod/heketi-storage-copy-job-hbvv4   MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.3:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r174fd3d54bd2476ab9e056279ea42029.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-04-22 06:34:28.998556] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:28.998701] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x5556177d7e77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x5556177d1ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7fb786f0eeab] ) 0-graph: invalid argument: graph [Invalid argument]
7m49s       Warning   FailedMount   pod/heketi-storage-copy-job-hbvv4   MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.3:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r9787f39108aa4c99a177e556522e9f17.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-04-22 06:34:29.663496] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:29.663644] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55dc332fce77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55dc332f6ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f056a420eab] ) 0-graph: invalid argument: graph [Invalid argument]
7m48s       Warning   FailedMount   pod/heketi-storage-copy-job-hbvv4   MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.2:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r80514c65cb6647f6ac2697b5114f469b.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-04-22 06:34:30.779343] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:30.779493] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55f294288e77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55f294282ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f070e30beab] ) 0-graph: invalid argument: graph [Invalid argument]
7m45s       Warning   FailedMount   pod/heketi-storage-copy-job-hbvv4   MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.3:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r976c7b489f754d9caec29e1da0d1884d.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-04-22 06:34:33.011549] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:33.011731] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x560cd9c99e77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x560cd9c93ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f7dcd745eab] ) 0-graph: invalid argument: graph [Invalid argument]
7m41s       Warning   FailedMount   pod/heketi-storage-copy-job-hbvv4   MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.3:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-rdbd0b7e6802c414b813bd3b188076e4d.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-04-22 06:34:37.143898] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:37.144040] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55f51bd1ae77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55f51bd14ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7febafc8beab] ) 0-graph: invalid argument: graph [Invalid argument]
7m33s       Warning   FailedMount   pod/heketi-storage-copy-job-hbvv4   MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.3:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r867c9cb2f25a4f6ba066719ce4ca483d.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-04-22 06:34:45.293488] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:45.293683] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55bb171bfe77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55bb171b9ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f5d47a95eab] ) 0-graph: invalid argument: graph [Invalid argument]
7m17s       Warning   FailedMount   pod/heketi-storage-copy-job-hbvv4   MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.2:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-re7d2348c9f334f6eaed9a6216ef332bf.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-04-22 06:35:01.489830] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:35:01.489995] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55b7595fbe77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55b7595f5ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f50b6965eab] ) 0-graph: invalid argument: graph [Invalid argument]
6m45s       Warning   FailedMount   pod/heketi-storage-copy-job-hbvv4   MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.2:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r48261be86e40451bb4210c19286f31d8.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-04-22 06:35:33.693499] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:35:33.693662] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55be60e85e77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55be60e7fea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f20b8b7eeab] ) 0-graph: invalid argument: graph [Invalid argument]
3m30s       Warning   FailedMount   pod/heketi-storage-copy-job-hbvv4   Unable to attach or mount volumes: unmounted volumes=[heketi-storage], unattached volumes=[heketi-storage heketi-storage-secret default-token-t7qfx]: timed out waiting for the condition
76s         Warning   FailedMount   pod/heketi-storage-copy-job-hbvv4   (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[heketi-storage heketi-storage-secret default-token-t7qfx], unattached volumes=[heketi-storage heketi-storage-secret default-token-t7qfx]: timed out waiting for the condition

I'm using Ubuntu Server 18.04, running kubectl version yields this:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-14T16:45:05Z", GoVersion:"go1.13.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Any help would be appreciated.
@grzegorzgg
Copy link

I have the same problem
I also use Ubuntu 18.04

@mewais
Copy link
Author

mewais commented May 6, 2020

I moved on, I'm now using rook to install a ceph storage system. You can find a nice guide here

@aravindavk
Copy link
Member

I moved on, I'm now using rook to install a ceph storage system. You can find a nice guide here

If you are still interested in GlusterFS based solution, please try https://kadalu.io (Project page: https://github.com/kadalu/kadalu)

@mewais
Copy link
Author

mewais commented May 8, 2020

@aravindavk I will, thanks for bringing this up.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@aravindavk @grzegorzgg @mewais