test(support-bundle): should not timeout #1456

c3y1huang · 2023-07-11T04:52:07Z

longhorn/longhorn#6256

innobead · 2023-07-11T05:36:26Z

@yangchiu @khushboo-rancher review this.

cc @longhorn/qa

ref: 6256 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

chriscchien

LGTM

yangchiu · 2023-07-11T09:43:06Z

I tried to run the test case, but it failed with the following error:

        download_support_bundle(node_id, support_bundle_name, client)
>       wait_for_support_bundle_cleanup(client)

test_support_bundle.py:402: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

client = <longhorn.Client object at 0x7f5378b703d0>

    def wait_for_support_bundle_cleanup(client):  # NOQA
        ok = False
        for _ in range(RETRY_COUNTS):
            support_bundles = client.list_support_bundle()
            if len(support_bundles) == 0:
                ok = True
                break
    
            time.sleep(RETRY_INTERVAL)
>       assert ok
E       AssertionError

common.py:5547: AssertionError

https://ci.longhorn.io/job/public/job/v1.1.x/job/v1.1.x-longhorn-upgrade-tests-sles-arm64/169/

Do we need other modification to make it pass?

c3y1huang · 2023-07-11T10:00:37Z

I tried to run the test case, but it failed with the following error:

        download_support_bundle(node_id, support_bundle_name, client)
>       wait_for_support_bundle_cleanup(client)

test_support_bundle.py:402: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

client = <longhorn.Client object at 0x7f5378b703d0>

    def wait_for_support_bundle_cleanup(client):  # NOQA
        ok = False
        for _ in range(RETRY_COUNTS):
            support_bundles = client.list_support_bundle()
            if len(support_bundles) == 0:
                ok = True
                break
    
            time.sleep(RETRY_INTERVAL)
>       assert ok
E       AssertionError

common.py:5547: AssertionError

https://ci.longhorn.io/job/public/job/v1.1.x/job/v1.1.x-longhorn-upgrade-tests-sles-arm64/169/

Do we need other modification to make it pass?

Failing at wait_for_support_bundle_cleanup is expected at the current code bases. This happens because the support bundle download timed out, the lifecycle is not completed and the CR doesn't get cleaned up. This is a reproduction of the issue described in longhorn/longhorn#6256.

To have a PASS run requires the fixes:

Ref: test results

yangchiu

LGTM

khushboo-rancher · 2023-07-12T01:10:30Z

@mergify backport v1.5.x v1.4.x

mergify · 2023-07-12T01:10:54Z

backport v1.5.x v1.4.x

✅ Backports have been created

#1464 test(support-bundle): should not timeout (backport #1456) has been created for branch v1.5.x
#1465 test(support-bundle): should not timeout (backport #1456) has been created for branch v1.4.x

test(support-bundle): should not timeout (backport #1456)

yangchiu · 2023-07-13T01:30:31Z

Is it possible that this test case crash the pytest process on arm64 machine?

Like:
https://ci.longhorn.io/job/public/job/master/job/sles/job/arm64/job/longhorn-tests-sles-arm64/537/console
https://ci.longhorn.io/job/public/job/master/job/sles/job/arm64/job/longhorn-upgrade-tests-sles-arm64/534/console

https://ci.longhorn.io/job/public/job/v1.4.x/job/v1.4.x-longhorn-tests-sles-arm64/221/console
https://ci.longhorn.io/job/public/job/v1.4.x/job/v1.4.x-longhorn-upgrade-tests-sles-arm64/220/console

https://ci.longhorn.io/job/private/job/longhorn-tests-regression/4555/console

cc @c3y1huang @longhorn/qa

c3y1huang · 2023-07-13T01:48:19Z

Weird, so it's only happening in arm64. ~~I am not sure if it could be related to running out of disk space. Checking...~~

Not related to the disk space. The dummy file was created successfully.

c3y1huang · 2023-07-13T03:18:43Z

Weird, so it's only happening in arm64. I am not sure if it could be related to running out of disk space. Checking...

Observed in local cluster:

The dummy file was created successfully. Download started.

pytest got killed.

test_support_bundle.py::test_support_bundle_should_not_timeout ^[[A./run.sh: line 3:    42 Killed                  pytest -v "$@"

longhorn-test pod did not crash.

c3y1huang · 2023-07-13T03:24:17Z

~~Possibly related to the download timeout. Checking...~~

Not related. Pytest was killed before being timed out.

c3y1huang · 2023-07-13T04:10:12Z

The memory usage just before the pytest got killed:

MiB Mem : 7644.219 total,   93.559 free, 7395.656 used,  346.988 buff/cache

Comparing to the instance type we've used in amd64 vs arm64
amd64: t2.xlarge memory(16 GiB)
arm64: a1.xlarge memory(8 GiB)

The issue appears to be resolved when tested using an a1.2xlarge instance with 16 GiB of memory.

@yangchiu can you help to adjust the instance type for arm64 pipelines?

yangchiu · 2023-07-13T04:28:55Z

@yangchiu can you help to adjust the instance type for arm64 pipelines?

Created PR longhorn/infra#130 for this.

c3y1huang self-assigned this Jul 11, 2023

c3y1huang force-pushed the test-support-bundle-download-timeout branch from 5e0c967 to 261462e Compare July 11, 2023 04:53

c3y1huang mentioned this pull request Jul 11, 2023

[BUG] Unable to receive support bundle from UI when it's large (400MB+) longhorn/longhorn#6256

Closed

c3y1huang force-pushed the test-support-bundle-download-timeout branch from 261462e to fc4c774 Compare July 11, 2023 05:26

c3y1huang marked this pull request as ready for review July 11, 2023 05:29

c3y1huang requested a review from a team as a code owner July 11, 2023 05:29

test(support-bundle): should not timeout

a680283

ref: 6256 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

c3y1huang force-pushed the test-support-bundle-download-timeout branch from fc4c774 to a680283 Compare July 11, 2023 05:52

chriscchien reviewed Jul 11, 2023

View reviewed changes

yangchiu approved these changes Jul 11, 2023

View reviewed changes

khushboo-rancher approved these changes Jul 12, 2023

View reviewed changes

khushboo-rancher merged commit 9a437d5 into longhorn:master Jul 12, 2023

This was referenced Jul 12, 2023

test(support-bundle): should not timeout (backport #1456) #1464

Merged

test(support-bundle): should not timeout (backport #1456) #1465

Merged

khushboo-rancher added a commit that referenced this pull request Jul 12, 2023

Merge pull request #1464 from longhorn/mergify/bp/v1.5.x/pr-1456

b39809a

test(support-bundle): should not timeout (backport #1456)

khushboo-rancher added a commit that referenced this pull request Jul 12, 2023

Merge pull request #1465 from longhorn/mergify/bp/v1.4.x/pr-1456

a1269cb

test(support-bundle): should not timeout (backport #1456)

c3y1huang deleted the test-support-bundle-download-timeout branch July 12, 2023 01:29

yangchiu mentioned this pull request Jul 13, 2023

ci: use a1.2xlarge machine for arm64 longhorn/infra#130

Merged

roger-ryao mentioned this pull request Jan 18, 2024

[TEST] test_support_bundle.py test failed on AKS /GKE longhorn/longhorn#7722

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(support-bundle): should not timeout #1456

test(support-bundle): should not timeout #1456

c3y1huang commented Jul 11, 2023

innobead commented Jul 11, 2023

chriscchien left a comment

yangchiu commented Jul 11, 2023

c3y1huang commented Jul 11, 2023 •

edited

Loading

yangchiu left a comment

khushboo-rancher commented Jul 12, 2023

mergify bot commented Jul 12, 2023 •

edited

Loading

yangchiu commented Jul 13, 2023 •

edited

Loading

c3y1huang commented Jul 13, 2023 •

edited

Loading

c3y1huang commented Jul 13, 2023 •

edited

Loading

c3y1huang commented Jul 13, 2023 •

edited

Loading

c3y1huang commented Jul 13, 2023 •

edited

Loading

yangchiu commented Jul 13, 2023

test(support-bundle): should not timeout #1456

test(support-bundle): should not timeout #1456

Conversation

c3y1huang commented Jul 11, 2023

innobead commented Jul 11, 2023

chriscchien left a comment

Choose a reason for hiding this comment

yangchiu commented Jul 11, 2023

c3y1huang commented Jul 11, 2023 • edited Loading

yangchiu left a comment

Choose a reason for hiding this comment

khushboo-rancher commented Jul 12, 2023

mergify bot commented Jul 12, 2023 • edited Loading

✅ Backports have been created

yangchiu commented Jul 13, 2023 • edited Loading

c3y1huang commented Jul 13, 2023 • edited Loading

c3y1huang commented Jul 13, 2023 • edited Loading

c3y1huang commented Jul 13, 2023 • edited Loading

c3y1huang commented Jul 13, 2023 • edited Loading

yangchiu commented Jul 13, 2023

c3y1huang commented Jul 11, 2023 •

edited

Loading

mergify bot commented Jul 12, 2023 •

edited

Loading

yangchiu commented Jul 13, 2023 •

edited

Loading

c3y1huang commented Jul 13, 2023 •

edited

Loading

c3y1huang commented Jul 13, 2023 •

edited

Loading

c3y1huang commented Jul 13, 2023 •

edited

Loading

c3y1huang commented Jul 13, 2023 •

edited

Loading