Add replicaName argument to ReplicaRebuildVerify #2108

ejweber · 2023-08-17T15:07:46Z

I added the replicaName argument to instance-manager's ReplicaRebuildVerify client method (and the associated gRPC request) as a part of longhorn/longhorn#5845, but I did not create a way for longhorn-manager to provide it. This PR fixes the issue and allows the tests mentioned in longhorn/longhorn#6522 to pass.

Longhorn 6522 Signed-off-by: Eric Weber <eric.weber@suse.com>

ejweber · 2023-08-17T15:45:12Z

Tested on my own cluster since I haven't been able to figure out how to pass a -k <test> or <another_test> or <another_test> type argument in Jenkins (there's always an issue with tokenization).

9/10 tests passed on the first run. I tried the last one again and it passed on a second run. Not sure if that one is prone to flaking? But the logs did not indicate anything like the issue this PR is fixing.

-k test_single_replica_restore_failure or test_rebuild_with_inc_restoration or test_rebuild_with_restoration or test_dr_volume_with_restore_command_error or test_inc_restoration_with_multiple_rebuild_and_expansion

eweber@laptop:~/longhorn-manager> k logs longhorn-test -f
Defaulted container "longhorn-test" out of: longhorn-test, longhorn-test-report
============================= test session starts ==============================
platform linux -- Python 3.9.17, pytest-5.3.1, py-1.11.0, pluggy-0.13.1 -- /usr/bin/python3.9
cachedir: .pytest_cache
rootdir: /integration, inifile: pytest.ini
plugins: repeat-0.9.1, order-1.0.1
collecting ... collected 425 items / 415 deselected / 10 selected

test_ha.py::test_rebuild_with_restoration[s3] PASSED                     [ 10%]
test_ha.py::test_rebuild_with_restoration[nfs] PASSED                    [ 20%]
test_ha.py::test_rebuild_with_inc_restoration[s3] PASSED                 [ 30%]
test_ha.py::test_rebuild_with_inc_restoration[nfs] PASSED                [ 40%]
test_ha.py::test_inc_restoration_with_multiple_rebuild_and_expansion[s3] PASSED [ 50%]
test_ha.py::test_inc_restoration_with_multiple_rebuild_and_expansion[nfs] PASSED [ 60%]
test_ha.py::test_single_replica_restore_failure[s3] PASSED               [ 70%]
test_ha.py::test_single_replica_restore_failure[nfs] PASSED              [ 80%]
test_ha.py::test_dr_volume_with_restore_command_error[s3] FAILED         [ 90%]
test_ha.py::test_dr_volume_with_restore_command_error[nfs] PASSED        [100%]

=================================== FAILURES ===================================
________________ test_dr_volume_with_restore_command_error[s3] _________________

set_random_backupstore = None
client = <longhorn.Client object at 0x7f5784e4c7c0>
core_api = <kubernetes.client.api.core_v1_api.CoreV1Api object at 0x7f578506d790>
volume_name = 'longhorn-testvol-xuvgnb'
csi_pv = {'apiVersion': 'v1', 'kind': 'PersistentVolume', 'metadata': {'name': 'longhorn-testvol-xuvgnb-std'}, 'spec': {'access...eplicaTimeout': '30'}, 'volumeHandle': 'longhorn-testvol-xuvgnb-std'}, 'persistentVolumeReclaimPolicy': 'Delete', ...}}
pvc = {'apiVersion': 'v1', 'kind': 'PersistentVolumeClaim', 'metadata': {'name': 'longhorn-testvol-xuvgnb-std-pvc'}, 'spec':...ources': {'requests': {'storage': '1073741824'}}, 'storageClassName': '', 'volumeName': 'longhorn-testvol-xuvgnb-std'}}
pod_make = <function pod_make.<locals>.make_pod at 0x7f5784ee5c10>

    def test_dr_volume_with_restore_command_error(set_random_backupstore, client, core_api, volume_name, csi_pv, pvc, pod_make):  # NOQA
        """
        Test if Longhorn can capture and handle the restore command error
        rather than the error triggered the data restoring.
    
        1. Set a random backupstore.
        2. Create a volume, then create the corresponding PV, PVC and Pod.
        3. Write data to the pod volume and get the md5sum
           after the pod running.
        4. Create the 1st backup.
        5. Create a DR volume from the backup.
        6. Wait for the DR volume restore complete.
        7. Create a non-empty directory `volume-delta-<last backup name>.img`
           in one replica directory of the DR volume. This will fail the
           restore command call later.
        8. Write data to the original volume then create the 2nd backup.
        9. Wait for incremental restore complete.
           Then verify the DR volume is Degraded
           and there is one failed replica.
        10. Verify the failed replica will be reused for rebuilding
            (restore actually).
        11. Activate the DR volume and wait for it complete.
        12. Create PV/PVC/Pod for the activated volume.
        13. Validate the volume content.
        14. Verify Writing data to the activated volume is fine.
        """
        update_setting(client, common.SETTING_DEGRADED_AVAILABILITY, "false")
    
        std_volume_name = volume_name + "-std"
        data_path1 = "/data/test1"
        std_pod_name, std_pv_name, std_pvc_name, std_md5sum1 = \
            prepare_pod_with_data_in_mb(
                client, core_api, csi_pv, pvc, pod_make, std_volume_name,
                data_path=data_path1, data_size_in_mb=DATA_SIZE_IN_MB_1)
    
        std_volume = client.by_id_volume(std_volume_name)
        snap1 = create_snapshot(client, std_volume_name)
        std_volume.snapshotBackup(name=snap1.name)
        wait_for_backup_completion(client, std_volume_name, snap1.name)
        bv, b1 = find_backup(client, std_volume_name, snap1.name)
    
        dr_volume_name = volume_name + "-dr"
        client.create_volume(name=dr_volume_name, size=str(1 * Gi),
                             numberOfReplicas=3, fromBackup=b1.url,
                             frontend="", standby=True)
        wait_for_volume_creation(client, dr_volume_name)
        wait_for_volume_restoration_start(client, dr_volume_name, b1.name)
        wait_for_backup_restore_completed(client, dr_volume_name, b1.name)
    
        dr_volume = client.by_id_volume(dr_volume_name)
        # Will hack into the replica directory, create a non-empty directory
        # in the special path. (This path will be reserved for the restore.)
        # Then the following inc restore should fail.
        failed_replica = dr_volume.replicas[0]
        cmd = "mkdir -p " + "/host" + failed_replica.dataPath + "/volume-delta-" +\
              dr_volume.controllers[0].lastRestoredBackup + ".img/random-dir"
        exec_instance_manager(core_api,
                              failed_replica.instanceManagerName, cmd)
    
        data_path2 = "/data/test2"
        write_pod_volume_random_data(core_api, std_pod_name,
                                     data_path2, DATA_SIZE_IN_MB_1)
        std_md5sum2 = get_pod_data_md5sum(core_api, std_pod_name, data_path2)
        snap2 = create_snapshot(client, std_volume_name)
        std_volume.snapshotBackup(name=snap2.name)
        wait_for_backup_completion(client, std_volume_name, snap2.name)
        bv, b2 = find_backup(client, std_volume_name, snap2.name)
    
        # Wait for the incremental restoration triggered then complete.
        client.list_backupVolume()
        check_volume_last_backup(client, dr_volume_name, b2.name)
        wait_for_volume_restoration_start(client, dr_volume_name, b2.name)
    
        dr_volume = wait_for_volume_degraded(client, dr_volume_name)
        verified = False
        for r in dr_volume.replicas:
            if r.name == failed_replica.name:
>               assert not r['running']
E               assert not True

test_ha.py:1699: AssertionError
----------------------------- Captured stdout call -----------------------------
volume = {'accessMode': 'rwo', 'backendStoreDriver': 'v1', 'backingImage': '', 'backupCompressionMethod': 'lz4', 'backupStatus': [], 'cloneStatus': {'snapshot': '', 'sourceVolume': '', 'state': ''}, 'conditions': {'restore': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:29Z', 'message': '', 'reason': 'RestoreInProgress', 'status': 'True'}, 'scheduled': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:26Z', 'message': '', 'reason': '', 'status': 'True'}, 'toomanysnapshots': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:26Z', 'message': '', 'reason': '', 'status': 'False'}}, 'controllers': [{'actualSize': '169869312', 'address': '10.42.27.18', 'currentImage': 'longhornio/longhorn-engine:master-head', 'endpoint': '', 'engineImage': 'longhornio/longhorn-engine:master-head', 'hostId': 'eweber-v125-worker-e472db53-n8vj8', 'instanceManagerName': 'instance-manager-211d7cd4337ea60af17807834c0c7055', 'isExpanding': False, 'lastExpansionError': '', 'lastExpansionFailedAt': '', 'lastRestoredBackup': '', 'name': 'longhorn-testvol-xuvgnb-dr-e-abb5ac92', 'requestedBackupRestore': 'backup-b6c869cba426477e', 'running': True, 'size': '1073741824', 'unmapMarkSnapChainRemovedEnabled': False}], 'created': '2023-08-17 14:59:25 +0000 UTC', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataLocality': 'disabled', 'dataSource': '', 'disableFrontend': True, 'diskSelector': [], 'encrypted': False, 'engineImage': 'longhornio/longhorn-engine:master-head', 'fromBackup': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'frontend': '', 'kubernetesStatus': {'lastPVCRefAt': '2023-08-17T14:59:18Z', 'lastPodRefAt': '2023-08-17T14:59:18Z', 'namespace': 'default', 'pvName': '', 'pvStatus': '', 'pvcName': 'longhorn-testvol-xuvgnb-std-pvc', 'workloadsStatus': [{'podName': 'longhorn-testvol-xuvgnb-std-pod', 'podStatus': 'Running', 'workloadName': '', 'workloadType': ''}]}, 'lastAttachedBy': '', 'lastBackup': 'backup-b6c869cba426477e', 'lastBackupAt': '2023-08-17T14:59:18Z', 'migratable': False, 'name': 'longhorn-testvol-xuvgnb-dr', 'nodeSelector': [], 'numberOfReplicas': 3, 'offlineReplicaRebuilding': 'disabled', 'offlineReplicaRebuildingRequired': False, 'purgeStatus': [{'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'state': ''}, {'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'state': ''}, {'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'state': ''}], 'ready': False, 'rebuildStatus': [], 'recurringJobSelector': None, 'replicaAutoBalance': 'ignored', 'replicaDiskSoftAntiAffinity': '', 'replicaSoftAntiAffinity': 'ignored', 'replicaZoneSoftAntiAffinity': 'ignored', 'replicas': [{'address': '10.42.27.18', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-4c46e550', 'diskID': 'd859aad8-29ff-43a6-bb7e-f09ce6a2d93d', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-n8vj8', 'instanceManagerName': 'instance-manager-211d7cd4337ea60af17807834c0c7055', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'running': True}, {'address': '10.42.59.184', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-d446bea7', 'diskID': '960eccfd-bd02-47eb-995e-ab569503424c', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-9kz5b', 'instanceManagerName': 'instance-manager-de0a44c527b5c06489d7a009124e4fd3', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'running': True}, {'address': '10.42.86.170', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-ee250d88', 'diskID': '46d29fde-a8ad-463d-9e7b-c43de7ff8dce', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-xjmzr', 'instanceManagerName': 'instance-manager-c839c5bbab46d866e8380cbdb9460b4a', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'running': True}], 'restoreInitiated': True, 'restoreRequired': True, 'restoreStatus': [{'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': True, 'lastRestored': '', 'progress': 44, 'replica': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'state': 'in_progress'}, {'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': False, 'lastRestored': 'backup-b6c869cba426477e', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'state': 'complete'}, {'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': True, 'lastRestored': '', 'progress': 38, 'replica': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'state': 'in_progress'}], 'restoreVolumeRecurringJob': 'ignored', 'revisionCounterDisabled': False, 'robustness': 'healthy', 'shareEndpoint': '', 'shareState': '', 'size': '1073741824', 'snapshotDataIntegrity': 'ignored', 'staleReplicaTimeout': 0, 'standby': True, 'state': 'attached', 'unmapMarkSnapChainRemoved': 'ignored', 'volumeAttachment': {'attachments': {'volume-restore-controller-longhorn-testvol-xuvgnb-dr': {'attachmentID': 'volume-restore-controller-longhorn-testvol-xuvgnb-dr', 'attachmentType': 'volume-restore-controller', 'conditions': [{'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:29Z', 'message': '', 'reason': '', 'status': 'True'}], 'nodeID': 'eweber-v125-worker-e472db53-n8vj8', 'parameters': {'disableFrontend': 'true'}, 'satisfied': True}}, 'volume': 'longhorn-testvol-xuvgnb-dr'}}
volume = {'accessMode': 'rwo', 'backendStoreDriver': 'v1', 'backingImage': '', 'backupCompressionMethod': 'lz4', 'backupStatus': [], 'cloneStatus': {'snapshot': '', 'sourceVolume': '', 'state': ''}, 'conditions': {'restore': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:29Z', 'message': '', 'reason': 'RestoreInProgress', 'status': 'True'}, 'scheduled': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:26Z', 'message': '', 'reason': '', 'status': 'True'}, 'toomanysnapshots': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:26Z', 'message': '', 'reason': '', 'status': 'False'}}, 'controllers': [{'actualSize': '169869312', 'address': '10.42.27.18', 'currentImage': 'longhornio/longhorn-engine:master-head', 'endpoint': '', 'engineImage': 'longhornio/longhorn-engine:master-head', 'hostId': 'eweber-v125-worker-e472db53-n8vj8', 'instanceManagerName': 'instance-manager-211d7cd4337ea60af17807834c0c7055', 'isExpanding': False, 'lastExpansionError': '', 'lastExpansionFailedAt': '', 'lastRestoredBackup': '', 'name': 'longhorn-testvol-xuvgnb-dr-e-abb5ac92', 'requestedBackupRestore': 'backup-b6c869cba426477e', 'running': True, 'size': '1073741824', 'unmapMarkSnapChainRemovedEnabled': False}], 'created': '2023-08-17 14:59:25 +0000 UTC', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataLocality': 'disabled', 'dataSource': '', 'disableFrontend': True, 'diskSelector': [], 'encrypted': False, 'engineImage': 'longhornio/longhorn-engine:master-head', 'fromBackup': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'frontend': '', 'kubernetesStatus': {'lastPVCRefAt': '2023-08-17T14:59:18Z', 'lastPodRefAt': '2023-08-17T14:59:18Z', 'namespace': 'default', 'pvName': '', 'pvStatus': '', 'pvcName': 'longhorn-testvol-xuvgnb-std-pvc', 'workloadsStatus': [{'podName': 'longhorn-testvol-xuvgnb-std-pod', 'podStatus': 'Running', 'workloadName': '', 'workloadType': ''}]}, 'lastAttachedBy': '', 'lastBackup': 'backup-b6c869cba426477e', 'lastBackupAt': '2023-08-17T14:59:18Z', 'migratable': False, 'name': 'longhorn-testvol-xuvgnb-dr', 'nodeSelector': [], 'numberOfReplicas': 3, 'offlineReplicaRebuilding': 'disabled', 'offlineReplicaRebuildingRequired': False, 'purgeStatus': [{'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'state': ''}, {'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'state': ''}, {'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'state': ''}], 'ready': False, 'rebuildStatus': [], 'recurringJobSelector': None, 'replicaAutoBalance': 'ignored', 'replicaDiskSoftAntiAffinity': '', 'replicaSoftAntiAffinity': 'ignored', 'replicaZoneSoftAntiAffinity': 'ignored', 'replicas': [{'address': '10.42.27.18', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-4c46e550', 'diskID': 'd859aad8-29ff-43a6-bb7e-f09ce6a2d93d', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-n8vj8', 'instanceManagerName': 'instance-manager-211d7cd4337ea60af17807834c0c7055', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'running': True}, {'address': '10.42.59.184', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-d446bea7', 'diskID': '960eccfd-bd02-47eb-995e-ab569503424c', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-9kz5b', 'instanceManagerName': 'instance-manager-de0a44c527b5c06489d7a009124e4fd3', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'running': True}, {'address': '10.42.86.170', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-ee250d88', 'diskID': '46d29fde-a8ad-463d-9e7b-c43de7ff8dce', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-xjmzr', 'instanceManagerName': 'instance-manager-c839c5bbab46d866e8380cbdb9460b4a', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'running': True}], 'restoreInitiated': True, 'restoreRequired': True, 'restoreStatus': [{'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': False, 'lastRestored': 'backup-b6c869cba426477e', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'state': 'complete'}, {'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': False, 'lastRestored': 'backup-b6c869cba426477e', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'state': 'complete'}, {'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': True, 'lastRestored': '', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'state': 'in_progress'}], 'restoreVolumeRecurringJob': 'ignored', 'revisionCounterDisabled': False, 'robustness': 'healthy', 'shareEndpoint': '', 'shareState': '', 'size': '1073741824', 'snapshotDataIntegrity': 'ignored', 'staleReplicaTimeout': 0, 'standby': True, 'state': 'attached', 'unmapMarkSnapChainRemoved': 'ignored', 'volumeAttachment': {'attachments': {'volume-restore-controller-longhorn-testvol-xuvgnb-dr': {'attachmentID': 'volume-restore-controller-longhorn-testvol-xuvgnb-dr', 'attachmentType': 'volume-restore-controller', 'conditions': [{'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:29Z', 'message': '', 'reason': '', 'status': 'True'}], 'nodeID': 'eweber-v125-worker-e472db53-n8vj8', 'parameters': {'disableFrontend': 'true'}, 'satisfied': True}}, 'volume': 'longhorn-testvol-xuvgnb-dr'}}
volume = {'accessMode': 'rwo', 'backendStoreDriver': 'v1', 'backingImage': '', 'backupCompressionMethod': 'lz4', 'backupStatus': [], 'cloneStatus': {'snapshot': '', 'sourceVolume': '', 'state': ''}, 'conditions': {'restore': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:29Z', 'message': '', 'reason': 'RestoreInProgress', 'status': 'True'}, 'scheduled': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:26Z', 'message': '', 'reason': '', 'status': 'True'}, 'toomanysnapshots': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:26Z', 'message': '', 'reason': '', 'status': 'False'}}, 'controllers': [{'actualSize': '169869312', 'address': '10.42.27.18', 'currentImage': 'longhornio/longhorn-engine:master-head', 'endpoint': '', 'engineImage': 'longhornio/longhorn-engine:master-head', 'hostId': 'eweber-v125-worker-e472db53-n8vj8', 'instanceManagerName': 'instance-manager-211d7cd4337ea60af17807834c0c7055', 'isExpanding': False, 'lastExpansionError': '', 'lastExpansionFailedAt': '', 'lastRestoredBackup': '', 'name': 'longhorn-testvol-xuvgnb-dr-e-abb5ac92', 'requestedBackupRestore': 'backup-b6c869cba426477e', 'running': True, 'size': '1073741824', 'unmapMarkSnapChainRemovedEnabled': False}], 'created': '2023-08-17 14:59:25 +0000 UTC', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataLocality': 'disabled', 'dataSource': '', 'disableFrontend': True, 'diskSelector': [], 'encrypted': False, 'engineImage': 'longhornio/longhorn-engine:master-head', 'fromBackup': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'frontend': '', 'kubernetesStatus': {'lastPVCRefAt': '2023-08-17T14:59:18Z', 'lastPodRefAt': '2023-08-17T14:59:18Z', 'namespace': 'default', 'pvName': '', 'pvStatus': '', 'pvcName': 'longhorn-testvol-xuvgnb-std-pvc', 'workloadsStatus': [{'podName': 'longhorn-testvol-xuvgnb-std-pod', 'podStatus': 'Running', 'workloadName': '', 'workloadType': ''}]}, 'lastAttachedBy': '', 'lastBackup': 'backup-b6c869cba426477e', 'lastBackupAt': '2023-08-17T14:59:18Z', 'migratable': False, 'name': 'longhorn-testvol-xuvgnb-dr', 'nodeSelector': [], 'numberOfReplicas': 3, 'offlineReplicaRebuilding': 'disabled', 'offlineReplicaRebuildingRequired': False, 'purgeStatus': [{'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'state': ''}, {'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'state': ''}, {'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'state': ''}], 'ready': False, 'rebuildStatus': [], 'recurringJobSelector': None, 'replicaAutoBalance': 'ignored', 'replicaDiskSoftAntiAffinity': '', 'replicaSoftAntiAffinity': 'ignored', 'replicaZoneSoftAntiAffinity': 'ignored', 'replicas': [{'address': '10.42.27.18', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-4c46e550', 'diskID': 'd859aad8-29ff-43a6-bb7e-f09ce6a2d93d', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-n8vj8', 'instanceManagerName': 'instance-manager-211d7cd4337ea60af17807834c0c7055', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'running': True}, {'address': '10.42.59.184', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-d446bea7', 'diskID': '960eccfd-bd02-47eb-995e-ab569503424c', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-9kz5b', 'instanceManagerName': 'instance-manager-de0a44c527b5c06489d7a009124e4fd3', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'running': True}, {'address': '10.42.86.170', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-ee250d88', 'diskID': '46d29fde-a8ad-463d-9e7b-c43de7ff8dce', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-xjmzr', 'instanceManagerName': 'instance-manager-c839c5bbab46d866e8380cbdb9460b4a', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'running': True}], 'restoreInitiated': True, 'restoreRequired': True, 'restoreStatus': [{'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': False, 'lastRestored': 'backup-b6c869cba426477e', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'state': 'complete'}, {'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': False, 'lastRestored': 'backup-b6c869cba426477e', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'state': 'complete'}, {'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': True, 'lastRestored': '', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'state': 'in_progress'}], 'restoreVolumeRecurringJob': 'ignored', 'revisionCounterDisabled': False, 'robustness': 'healthy', 'shareEndpoint': '', 'shareState': '', 'size': '1073741824', 'snapshotDataIntegrity': 'ignored', 'staleReplicaTimeout': 0, 'standby': True, 'state': 'attached', 'unmapMarkSnapChainRemoved': 'ignored', 'volumeAttachment': {'attachments': {'volume-restore-controller-longhorn-testvol-xuvgnb-dr': {'attachmentID': 'volume-restore-controller-longhorn-testvol-xuvgnb-dr', 'attachmentType': 'volume-restore-controller', 'conditions': [{'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:29Z', 'message': '', 'reason': '', 'status': 'True'}], 'nodeID': 'eweber-v125-worker-e472db53-n8vj8', 'parameters': {'disableFrontend': 'true'}, 'satisfied': True}}, 'volume': 'longhorn-testvol-xuvgnb-dr'}}
volume = {'accessMode': 'rwo', 'backendStoreDriver': 'v1', 'backingImage': '', 'backupCompressionMethod': 'lz4', 'backupStatus': [], 'cloneStatus': {'snapshot': '', 'sourceVolume': '', 'state': ''}, 'conditions': {'restore': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:29Z', 'message': '', 'reason': 'RestoreInProgress', 'status': 'True'}, 'scheduled': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:26Z', 'message': '', 'reason': '', 'status': 'True'}, 'toomanysnapshots': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:26Z', 'message': '', 'reason': '', 'status': 'False'}}, 'controllers': [{'actualSize': '169869312', 'address': '10.42.27.18', 'currentImage': 'longhornio/longhorn-engine:master-head', 'endpoint': '', 'engineImage': 'longhornio/longhorn-engine:master-head', 'hostId': 'eweber-v125-worker-e472db53-n8vj8', 'instanceManagerName': 'instance-manager-211d7cd4337ea60af17807834c0c7055', 'isExpanding': False, 'lastExpansionError': '', 'lastExpansionFailedAt': '', 'lastRestoredBackup': '', 'name': 'longhorn-testvol-xuvgnb-dr-e-abb5ac92', 'requestedBackupRestore': 'backup-b6c869cba426477e', 'running': True, 'size': '1073741824', 'unmapMarkSnapChainRemovedEnabled': False}], 'created': '2023-08-17 14:59:25 +0000 UTC', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataLocality': 'disabled', 'dataSource': '', 'disableFrontend': True, 'diskSelector': [], 'encrypted': False, 'engineImage': 'longhornio/longhorn-engine:master-head', 'fromBackup': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'frontend': '', 'kubernetesStatus': {'lastPVCRefAt': '2023-08-17T14:59:18Z', 'lastPodRefAt': '2023-08-17T14:59:18Z', 'namespace': 'default', 'pvName': '', 'pvStatus': '', 'pvcName': 'longhorn-testvol-xuvgnb-std-pvc', 'workloadsStatus': [{'podName': 'longhorn-testvol-xuvgnb-std-pod', 'podStatus': 'Running', 'workloadName': '', 'workloadType': ''}]}, 'lastAttachedBy': '', 'lastBackup': 'backup-b6c869cba426477e', 'lastBackupAt': '2023-08-17T14:59:18Z', 'migratable': False, 'name': 'longhorn-testvol-xuvgnb-dr', 'nodeSelector': [], 'numberOfReplicas': 3, 'offlineReplicaRebuilding': 'disabled', 'offlineReplicaRebuildingRequired': False, 'purgeStatus': [{'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'state': ''}, {'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'state': ''}, {'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'state': ''}], 'ready': False, 'rebuildStatus': [], 'recurringJobSelector': None, 'replicaAutoBalance': 'ignored', 'replicaDiskSoftAntiAffinity': '', 'replicaSoftAntiAffinity': 'ignored', 'replicaZoneSoftAntiAffinity': 'ignored', 'replicas': [{'address': '10.42.27.18', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-4c46e550', 'diskID': 'd859aad8-29ff-43a6-bb7e-f09ce6a2d93d', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-n8vj8', 'instanceManagerName': 'instance-manager-211d7cd4337ea60af17807834c0c7055', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'running': True}, {'address': '10.42.59.184', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-d446bea7', 'diskID': '960eccfd-bd02-47eb-995e-ab569503424c', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-9kz5b', 'instanceManagerName': 'instance-manager-de0a44c527b5c06489d7a009124e4fd3', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'running': True}, {'address': '10.42.86.170', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-ee250d88', 'diskID': '46d29fde-a8ad-463d-9e7b-c43de7ff8dce', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-xjmzr', 'instanceManagerName': 'instance-manager-c839c5bbab46d866e8380cbdb9460b4a', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'running': True}], 'restoreInitiated': True, 'restoreRequired': True, 'restoreStatus': [{'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': False, 'lastRestored': 'backup-b6c869cba426477e', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'state': 'complete'}, {'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': False, 'lastRestored': 'backup-b6c869cba426477e', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'state': 'complete'}, {'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': True, 'lastRestored': '', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'state': 'in_progress'}], 'restoreVolumeRecurringJob': 'ignored', 'revisionCounterDisabled': False, 'robustness': 'healthy', 'shareEndpoint': '', 'shareState': '', 'size': '1073741824', 'snapshotDataIntegrity': 'ignored', 'staleReplicaTimeout': 0, 'standby': True, 'state': 'attached', 'unmapMarkSnapChainRemoved': 'ignored', 'volumeAttachment': {'attachments': {'volume-restore-controller-longhorn-testvol-xuvgnb-dr': {'attachmentID': 'volume-restore-controller-longhorn-testvol-xuvgnb-dr', 'attachmentType': 'volume-restore-controller', 'conditions': [{'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:29Z', 'message': '', 'reason': '', 'status': 'True'}], 'nodeID': 'eweber-v125-worker-e472db53-n8vj8', 'parameters': {'disableFrontend': 'true'}, 'satisfied': True}}, 'volume': 'longhorn-testvol-xuvgnb-dr'}}
volume = {'accessMode': 'rwo', 'backendStoreDriver': 'v1', 'backingImage': '', 'backupCompressionMethod': 'lz4', 'backupStatus': [], 'cloneStatus': {'snapshot': '', 'sourceVolume': '', 'state': ''}, 'conditions': {'restore': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:29Z', 'message': '', 'reason': 'RestoreInProgress', 'status': 'True'}, 'scheduled': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:26Z', 'message': '', 'reason': '', 'status': 'True'}, 'toomanysnapshots': {'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:26Z', 'message': '', 'reason': '', 'status': 'False'}}, 'controllers': [{'actualSize': '169873408', 'address': '10.42.27.18', 'currentImage': 'longhornio/longhorn-engine:master-head', 'endpoint': '', 'engineImage': 'longhornio/longhorn-engine:master-head', 'hostId': 'eweber-v125-worker-e472db53-n8vj8', 'instanceManagerName': 'instance-manager-211d7cd4337ea60af17807834c0c7055', 'isExpanding': False, 'lastExpansionError': '', 'lastExpansionFailedAt': '', 'lastRestoredBackup': 'backup-b6c869cba426477e', 'name': 'longhorn-testvol-xuvgnb-dr-e-abb5ac92', 'requestedBackupRestore': 'backup-b6c869cba426477e', 'running': True, 'size': '1073741824', 'unmapMarkSnapChainRemovedEnabled': False}], 'created': '2023-08-17 14:59:25 +0000 UTC', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataLocality': 'disabled', 'dataSource': '', 'disableFrontend': True, 'diskSelector': [], 'encrypted': False, 'engineImage': 'longhornio/longhorn-engine:master-head', 'fromBackup': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'frontend': '', 'kubernetesStatus': {'lastPVCRefAt': '2023-08-17T14:59:18Z', 'lastPodRefAt': '2023-08-17T14:59:18Z', 'namespace': 'default', 'pvName': '', 'pvStatus': '', 'pvcName': 'longhorn-testvol-xuvgnb-std-pvc', 'workloadsStatus': [{'podName': 'longhorn-testvol-xuvgnb-std-pod', 'podStatus': 'Running', 'workloadName': '', 'workloadType': ''}]}, 'lastAttachedBy': '', 'lastBackup': 'backup-b6c869cba426477e', 'lastBackupAt': '2023-08-17T14:59:18Z', 'migratable': False, 'name': 'longhorn-testvol-xuvgnb-dr', 'nodeSelector': [], 'numberOfReplicas': 3, 'offlineReplicaRebuilding': 'disabled', 'offlineReplicaRebuildingRequired': False, 'purgeStatus': [{'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'state': ''}, {'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'state': ''}, {'error': '', 'isPurging': False, 'progress': 0, 'replica': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'state': ''}], 'ready': False, 'rebuildStatus': [], 'recurringJobSelector': None, 'replicaAutoBalance': 'ignored', 'replicaDiskSoftAntiAffinity': '', 'replicaSoftAntiAffinity': 'ignored', 'replicaZoneSoftAntiAffinity': 'ignored', 'replicas': [{'address': '10.42.27.18', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-4c46e550', 'diskID': 'd859aad8-29ff-43a6-bb7e-f09ce6a2d93d', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-n8vj8', 'instanceManagerName': 'instance-manager-211d7cd4337ea60af17807834c0c7055', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'running': True}, {'address': '10.42.59.184', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-d446bea7', 'diskID': '960eccfd-bd02-47eb-995e-ab569503424c', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-9kz5b', 'instanceManagerName': 'instance-manager-de0a44c527b5c06489d7a009124e4fd3', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'running': True}, {'address': '10.42.86.170', 'backendStoreDriver': 'v1', 'currentImage': 'longhornio/longhorn-engine:master-head', 'dataPath': '/var/lib/longhorn/replicas/longhorn-testvol-xuvgnb-dr-ee250d88', 'diskID': '46d29fde-a8ad-463d-9e7b-c43de7ff8dce', 'diskPath': '/var/lib/longhorn/', 'engineImage': 'longhornio/longhorn-engine:master-head', 'failedAt': '', 'hostId': 'eweber-v125-worker-e472db53-xjmzr', 'instanceManagerName': 'instance-manager-c839c5bbab46d866e8380cbdb9460b4a', 'mode': 'RW', 'name': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'running': True}], 'restoreInitiated': True, 'restoreRequired': True, 'restoreStatus': [{'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': False, 'lastRestored': 'backup-b6c869cba426477e', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-26a6406d', 'state': 'complete'}, {'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': False, 'lastRestored': 'backup-b6c869cba426477e', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-40eeba6a', 'state': 'complete'}, {'backupURL': 's3://backupbucket@us-east-1/backupstore?backup=backup-b6c869cba426477e&volume=longhorn-testvol-xuvgnb-std', 'error': '', 'filename': 'volume-snap-c31e683d-7bd2-4252-aa98-1553ea37680c.img', 'isRestoring': False, 'lastRestored': 'backup-b6c869cba426477e', 'progress': 100, 'replica': 'longhorn-testvol-xuvgnb-dr-r-442ca947', 'state': 'complete'}], 'restoreVolumeRecurringJob': 'ignored', 'revisionCounterDisabled': False, 'robustness': 'healthy', 'shareEndpoint': '', 'shareState': '', 'size': '1073741824', 'snapshotDataIntegrity': 'ignored', 'staleReplicaTimeout': 0, 'standby': True, 'state': 'attached', 'unmapMarkSnapChainRemoved': 'ignored', 'volumeAttachment': {'attachments': {'volume-restore-controller-longhorn-testvol-xuvgnb-dr': {'attachmentID': 'volume-restore-controller-longhorn-testvol-xuvgnb-dr', 'attachmentType': 'volume-restore-controller', 'conditions': [{'lastProbeTime': '', 'lastTransitionTime': '2023-08-17T14:59:29Z', 'message': '', 'reason': '', 'status': 'True'}], 'nodeID': 'eweber-v125-worker-e472db53-n8vj8', 'parameters': {'disableFrontend': 'true'}, 'satisfied': True}}, 'volume': 'longhorn-testvol-xuvgnb-dr'}}
=============================== warnings summary ===============================
tests/test_ha.py::test_inc_restoration_with_multiple_rebuild_and_expansion[s3]
tests/test_ha.py::test_inc_restoration_with_multiple_rebuild_and_expansion[nfs]
  /usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
    return self.urllib3_response.getheaders()

-- Docs: https://docs.pytest.org/en/latest/warnings.html
===== 1 failed, 9 passed, 415 deselected, 2 warnings in 4199.95s (1:09:59) =====

-k test_dr_volume_with_restore_command_error[s3]

eweber@laptop:~/longhorn> k logs longhorn-test
Defaulted container "longhorn-test" out of: longhorn-test, longhorn-test-report
============================= test session starts ==============================
platform linux -- Python 3.9.17, pytest-5.3.1, py-1.11.0, pluggy-0.13.1 -- /usr/bin/python3.9
cachedir: .pytest_cache
rootdir: /integration, inifile: pytest.ini
plugins: repeat-0.9.1, order-1.0.1
collecting ... collected 425 items / 424 deselected / 1 selected

test_ha.py::test_dr_volume_with_restore_command_error[s3] PASSED         [100%]

================ 1 passed, 424 deselected in 233.28s (0:03:53) =================

engineapi/engine.go

PhanLe1010

LGTM

ejweber requested a review from a team as a code owner August 17, 2023 15:07

ejweber force-pushed the 6522-verify-replica-rebuild-fails branch from 71310d4 to 3f781c3 Compare August 17, 2023 15:12

Add replicaName argument to ReplicaRebuildVerify

bb7dc3e

Longhorn 6522 Signed-off-by: Eric Weber <eric.weber@suse.com>

ejweber force-pushed the 6522-verify-replica-rebuild-fails branch from 3f781c3 to bb7dc3e Compare August 17, 2023 15:17

This was referenced Aug 17, 2023

Add replica-instance-name argument to verify-rebuild-replica command longhorn/longhorn-engine#923

Merged

[BUG] Make restore volume degrade cause the volume restoration never complete longhorn/longhorn#6522

Closed

PhanLe1010 reviewed Aug 18, 2023

View reviewed changes

engineapi/engine.go Show resolved Hide resolved

PhanLe1010 approved these changes Aug 18, 2023

View reviewed changes

PhanLe1010 merged commit 59b4379 into longhorn:master Aug 18, 2023
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add replicaName argument to ReplicaRebuildVerify #2108

Add replicaName argument to ReplicaRebuildVerify #2108

ejweber commented Aug 17, 2023 •

edited

Loading

ejweber commented Aug 17, 2023

PhanLe1010 left a comment

Add replicaName argument to ReplicaRebuildVerify #2108

Add replicaName argument to ReplicaRebuildVerify #2108

Conversation

ejweber commented Aug 17, 2023 • edited Loading

ejweber commented Aug 17, 2023

PhanLe1010 left a comment

Choose a reason for hiding this comment

ejweber commented Aug 17, 2023 •

edited

Loading