test(robot): add drain cases from manual test cases

[chriscchien]

Which issue(s) this PR fixes:

Issue #9292

What this PR does / why we need it:

Implement test case in https://longhorn.github.io/longhorn-tests/manual/pre-release/node-not-ready/node-down/node-drain-deletion/ to automation test cases

Special notes for your reviewer:

Verified on local with v1.7.1

Additional documentation or context

N/A

[yangchiu]

Test steps of a robot test case should be self-explanatory, but the following steps are unclear in terms of what they are checking:

When Force drain node 2 and wait for 90 second dataEngine=${DATA_ENGINE}
And The drain process not completed
And Check instance-manager pod is running on node 2    dataEngine=${DATA_ENGINE}

Could these steps be made more readable to clarify what exactly is being validated?

[chriscchien]

Updated, removed dataEngine=${DATA_ENGINE} in When Force drain node 2 and wait for 90 second dataEngine=${DATA_ENGINE}and use data_engine = BuiltIn().get_variable_value("${DATA_ENGINE}") to get data engine setting

def get_instance_manager_on_node(node_name):
    data_engine = BuiltIn().get_variable_value("${DATA_ENGINE}")
    pods = get_all_pods_on_node(node_name)
    for pod in pods:
        labels = pod.metadata.labels
        if labels.get("longhorn.io/data-engine") == data_engine and \
           labels.get("longhorn.io/component") == "instance-manager":
            return pod.metadata.name
    return None

Test result

[yangchiu]

Is it necessary to introduce popen, process.poll(), process.terminate() and process.communicate()?

Can subporcess.check_output() with the timeout parameter accomplish the same goal?

ref: https://docs.python.org/3/library/subprocess.html#using-the-subprocess-module

[chriscchien]

Yes, in test case Setting Allow Node Drain with the Last Healthy Replica protects the last healthy replica with Pod Disruption Budget (PDB), fter change node-drain-policy from block-if-contains-last-replica to always-allow, it
need to validate if the drain process completed or not, currently I can not find a way to let subporcess.check_output() execute in background. thank you.

[yangchiu]

Is the following approach feasible?

def force_drain_node():
    subprocess.check_output(timeout=90)

Force drain node ${node_id} and expect failure
    Run Keyword And Expect Failure    force drain node

Force drain node ${node_id} and expect success
    force drain node

And Set setting node-drain-policy to block-if-contains-last-replica
...
...
When Force drain node 2 and expect failure
And Check instance-manager pod is running on node 2

When Set setting node-drain-policy to always-allow
And Force drain node 2 and expect success
And Check PDB not exist    

[chriscchien]

Hi @yangchiu, updated and test result

[yangchiu]

Is it possible to use subprocess.check_output() with the timeout parameter to eliminate the need for the explicit wait for ${duration} second?

[chriscchien]

In some scenario, the drain process need to running in background, using timeout seems force terminated the process.

[yangchiu]

Isn't wait for ${duration} second also an explicit timeout?