refactor lookup plugins using top down method #523

mikemorency · 2024-08-27T16:19:45Z

SUMMARY

This change rewrites the lookup plugin logic to use a top down approach to finding objects. The plugins will now step through an objects path and try to lookup each intermediate object until it gets to the final object.

The old method has a few issues:

The existing plugins use an incorrect search method which yield inconsistent results (as described in recursive_folder_or_rp_moid_search awaits objects_moid to be in order (but it isn't necessarily) #500)
They also cannot find resources with the same name, even if the paths given are unique. For example, if /datacenter/vms/my-folder and /datacenter/hosts/my-folder both exist, you can never find either one
Items with the same name will cause conflicts, even if those items have slightly different paths. For example: /datacenter/vms/foo and /datacenter/hosts/foo existing at the same time makes it difficult to search for either folder

Fixes:
#500
#445
#324

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

all lookup plugins

ADDITIONAL INFORMATION

The original top down approach is described in #445. I tried to get that version working but ran into a few issues.

softwarefactory-project-zuul · 2024-08-27T16:45:15Z

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/3689c867af674261b293d918e25ced00

✔️ ansible-test-cloud-integration-vmware-rest SUCCESS in 13m 56s
✔️ build-ansible-collection SUCCESS in 8m 54s
✔️ ansible-galaxy-importer SUCCESS in 4m 35s

…ookup-results

softwarefactory-project-zuul · 2024-08-27T17:43:08Z

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/a847d14ac38c4d77be7362d291c9c5cd

❌ ansible-test-cloud-integration-vmware-rest RETRY_LIMIT in 3m 49s
✔️ build-ansible-collection SUCCESS in 9m 05s
✔️ ansible-galaxy-importer SUCCESS in 3m 04s

softwarefactory-project-zuul · 2024-08-28T14:25:34Z

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/11f081e9fc794f9594328c77980c8664

❌ ansible-test-cloud-integration-vmware-rest RETRY_LIMIT in 3m 50s
✔️ build-ansible-collection SUCCESS in 8m 17s
✔️ ansible-galaxy-importer SUCCESS in 3m 46s

mikemorency · 2024-08-28T17:24:45Z

recheck

softwarefactory-project-zuul · 2024-08-28T17:49:10Z

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/131b3f1b8ce346c6a2c9e8071c29c759

✔️ ansible-test-cloud-integration-vmware-rest SUCCESS in 14m 22s
✔️ build-ansible-collection SUCCESS in 7m 54s
✔️ ansible-galaxy-importer SUCCESS in 5m 10s

mikemorency · 2024-08-30T12:30:52Z

Heres my before and after playbook. I tried to include it in the PR title comment but that seemed to make the Zuul checks fail. I think its because it tried to parse the ansible vars included in the playbook snippet 🤷

Test Playbook:

  tasks:
    - name: Lookup Datacenter
      ansible.builtin.debug:
        msg: "{{ lookup('vmware.vmware_rest.datacenter_moid', '/' + datacenter, **connection_args) }}"
    - name: Lookup Cluster
      ansible.builtin.debug:
        msg: "{{ lookup('vmware.vmware_rest.cluster_moid', '/' + datacenter + '/MyCluster', **connection_args) }}"
    - name: Lookup Cluster Contents
      ansible.builtin.debug:
        msg: "{{ lookup('vmware.vmware_rest.cluster_moid', '/' + datacenter + '/MyCluster/', **connection_args) }}"
    - name: Lookup Datastores
      ansible.builtin.debug:
        msg: "{{ item }} has MOID {{ lookup('vmware.vmware_rest.datastore_moid', '/' + datacenter + '/' + item, **connection_args) }}"
      loop:
        - datastore1
        - datastore2
        - storage-folder/datastoreNfs
        - hostA/datastore3
        - hostA/
    - name: Lookup Folders
      ansible.builtin.debug:
        msg: "{{ item }} has MOID {{ lookup('vmware.vmware_rest.folder_moid', '/' + datacenter + '/' + item, **connection_args) }}"
      loop:
        - vm/mmtest
        - vm/mmtest/
        - vm/mmtest/subfolder1
        - vm/mmtest/subfolder2
    - name: Lookup Networks
      ansible.builtin.debug:
        msg: "{{ item }} has MOID {{ lookup('vmware.vmware_rest.network_moid', '/' + datacenter + '/' + item, **connection_args) }}"
      loop:
        - Management Network
        - vMotion
        - dummy-networks/
        - dummy-networks/DPortGroup
    - name: Lookup Hosts
      ansible.builtin.debug:
        msg: "{{ item }} has MOID {{ lookup('vmware.vmware_rest.host_moid', '/' + datacenter + '/' + item, **connection_args) }}"
      loop:
        - 'hostA'
        - MyCluster/
        - MyCluster/hostA
    - name: Lookup VMs
      ansible.builtin.debug:
        msg: "{{ item }} has MOID {{ lookup('vmware.vmware_rest.vm_moid', '/' + datacenter + '/' + item, **connection_args) }}"
      loop:
        - "hostA/"
        - vCLS/
        - mm-test/mmtest
        - datastoreNfs
        - nested-vcenter-pool/
    - name: Lookup Resource Pools
      ansible.builtin.debug:
        msg: "{{ item }} has MOID {{ lookup('vmware.vmware_rest.resource_pool_moid', '/' + datacenter + '/' + item, **connection_args) }}"
      loop:
        - nested-vcenter-pool/
        - nested-vcenter-pool
        - MyCluster/

Old Output (v4.0.0)

PLAY [Test] *****************************************************************************************************************************************************************************************************************************************************
TASK [Lookup Datacenter] ****************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => {
    "msg": "datacenter-3"
}
TASK [Lookup Cluster] *******************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => {
    "msg": ""
}
TASK [Lookup Cluster Contents] **********************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => {
    "msg": ""
}
TASK [Lookup Datastores] ****************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => (item=datastore1) => {
    "msg": "datastore1 has MOID "
}
ok: [mikemorency] => (item=datastore2) => {
    "msg": "datastore2 has MOID "
}
ok: [mikemorency] => (item=storage-folder/datastoreNfs) => {
    "msg": "storage-folder/datastoreNfs has MOID datastore-3036"
}
ok: [mikemorency] => (item=hostA/datastore3) => {
    "msg": "hostA/datastore3 has MOID "
}
ok: [mikemorency] => (item=hostA/) => {
    "msg": "hostA/ has MOID "
}
TASK [Lookup Folders] *******************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => (item=vm/mmtest) => {
    "msg": "vm/mmtest has MOID group-v52464"
}
fatal: [mikemorency]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'vmware.vmware_rest.folder_moid'. Error was a <class 'ansible_collections.cloud.common.plugins.module_utils.turbo.exceptions.EmbeddedModuleUnexpectedFailure'>, original message: More than one object available: [<redacted>]."}
PLAY RECAP ******************************************************************************************************************************************************************************************************************************************************
mikemorency : ok=4    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

New Output

PLAY [Test] *****************************************************************************************************************************************************************************************************************************************************
TASK [Lookup Datacenter] ****************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => {
    "msg": "datacenter-3"
}
TASK [Lookup Cluster] *******************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => {
    "msg": "domain-c34"
}
TASK [Lookup Cluster Contents] **********************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => {
    "msg": [
        "domain-c34"
    ]
}
TASK [Lookup Datastores] ****************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => (item=datastore1) => {
    "msg": "datastore1 has MOID datastore-45"
}
ok: [mikemorency] => (item=datastore2) => {
    "msg": "datastore2 has MOID datastore-3081"
}
ok: [mikemorency] => (item=storage-folder/datastoreNfs) => {
    "msg": "storage-folder/datastoreNfs has MOID datastore-3036"
}
ok: [mikemorency] => (item=hostA/datastore3) => {
    "msg": "hostA/datastore3 has MOID datastore-3155"
}
ok: [mikemorency] => (item=hostA/) => {
    "msg": "hostA/ has MOID ['datastore-12645', 'datastore-12660', 'datastore-13341', 'datastore-18601', 'datastore-3036', 'datastore-3081', 'datastore-3155', 'datastore-45', 'datastore-48', 'datastore-67']"
}
TASK [Lookup Folders] *******************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => (item=vm/mmtest) => {
    "msg": "vm/mmtest has MOID group-v52464"
}
ok: [mikemorency] => (item=vm/mmtest/) => {
    "msg": "vm/mmtest/ has MOID ['group-v52685', 'group-v52686']"
}
ok: [mikemorency] => (item=vm/mmtest/subfolder1) => {
    "msg": "vm/mmtest/subfolder1 has MOID group-v52685"
}
ok: [mikemorency] => (item=vm/mmtest/subfolder2) => {
    "msg": "vm/mmtest/subfolder2 has MOID group-v52686"
}
TASK [Lookup Networks] ******************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => (item=Management Network) => {
    "msg": "Management Network has MOID dvportgroup-17756"
}
ok: [mikemorency] => (item=vMotion) => {
    "msg": "vMotion has MOID dvportgroup-17759"
}
ok: [mikemorency] => (item=dummy-networks/) => {
    "msg": "dummy-networks/ has MOID ['dvportgroup-52746', 'dvportgroup-52747']"
}
ok: [mikemorency] => (item=dummy-networks/DPortGroup) => {
    "msg": "dummy-networks/DPortGroup has MOID dvportgroup-52747"
}
TASK [Lookup Hosts] *********************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => (item=hostA) => {
    "msg": "hostA has MOID host-3152"
}
ok: [mikemorency] => (item=MyCluster/) => {
    "msg": "MyCluster/ has MOID ['host-12642', 'host-12657', 'host-3078', 'host-3152', 'host-36', 'host-40', 'host-64']"
}
ok: [mikemorency] => (item=MyCluster/hostA) => {
    "msg": "MyCluster/hostA has MOID host-3152"
}
TASK [Lookup VMs] ***********************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => (item=hostA/) => {
    "msg": "hostA/ has MOID ['vm-12749', 'vm-17649', 'vm-18084', 'vm-18263', 'vm-18695', 'vm-18697', 'vm-18698', 'vm-18699', 'vm-18704']"
}
ok: [mikemorency] => (item=vCLS/) => {
    "msg": "vCLS/ has MOID ['vm-18082', 'vm-18083', 'vm-18084']"
}
ok: [mikemorency] => (item=mm-test/mmtest) => {
    "msg": "mm-test/mmtest has MOID vm-11080"
}
ok: [mikemorency] => (item=datastoreNfs) => {
    "msg": "datastoreNfs has MOID vm-3035"
}
ok: [mikemorency] => (item=nested-vcenter-pool/) => {
    "msg": "nested-vcenter-pool/ has MOID ['vm-5063']"
}
TASK [Lookup Resource Pools] ************************************************************************************************************************************************************************************************************************************
ok: [mikemorency] => (item=nested-vcenter-pool/) => {
    "msg": "nested-vcenter-pool/ has MOID ['resgroup-5060']"
}
ok: [mikemorency] => (item=nested-vcenter-pool) => {
    "msg": "nested-vcenter-pool has MOID resgroup-5060"
}
ok: [mikemorency] => (item=MyCluster/) => {
    "msg": "MyCluster/ has MOID ['resgroup-3026', 'resgroup-35', 'resgroup-5060']"
}
PLAY RECAP ******************************************************************************************************************************************************************************************************************************************************
mikemorency : ok=9    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

changelogs/fragments/523-refactor-lookup-plugins.yml

bardielle · 2024-09-03T12:40:54Z

@mariolenz can you review this PR please?

plugins/plugin_utils/lookup.py

softwarefactory-project-zuul · 2024-09-03T12:55:24Z

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/4bf30e4698464cc6a85e1c61a41f3e80

✔️ ansible-test-cloud-integration-vmware-rest SUCCESS in 13m 23s
✔️ build-ansible-collection SUCCESS in 7m 49s
✔️ ansible-galaxy-importer SUCCESS in 4m 31s

softwarefactory-project-zuul · 2024-09-03T13:37:27Z

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/424d822b209848f9bdd7be7ab2c81756

✔️ ansible-test-cloud-integration-vmware-rest SUCCESS in 15m 57s
✔️ build-ansible-collection SUCCESS in 7m 47s
✔️ ansible-galaxy-importer SUCCESS in 4m 30s

softwarefactory-project-zuul · 2024-09-09T12:57:14Z

Build succeeded (gate pipeline).
https://ansible.softwarefactory-project.io/zuul/buildset/5886361bdf724eba95ad4a64eb305ea5

✔️ ansible-test-cloud-integration-vmware-rest SUCCESS in 11m 01s
✔️ build-ansible-collection SUCCESS in 8m 49s
✔️ ansible-galaxy-importer SUCCESS in 3m 01s

refactor lookup plugins using top down method

7622be6

lookup plugin lint fixes and adding method comments

9fde8ae

mikemorency marked this pull request as ready for review August 27, 2024 17:15

mikemorency requested review from mariolenz, machacekondra and bardielle August 27, 2024 17:15

Merge branch 'ansible-collections:main' into mm-bugfix/inconsistent-l…

8f374b7

…ookup-results

mikemorency closed this Aug 28, 2024

mikemorency reopened this Aug 28, 2024

bardielle reviewed Sep 1, 2024

View reviewed changes

changelogs/fragments/523-refactor-lookup-plugins.yml Outdated Show resolved Hide resolved

add issue links to changelog

d871593

mikemorency requested a review from bardielle September 3, 2024 12:32

bardielle approved these changes Sep 3, 2024

View reviewed changes

machacekondra reviewed Sep 3, 2024

View reviewed changes

plugins/plugin_utils/lookup.py Outdated Show resolved Hide resolved

machacekondra reviewed Sep 3, 2024

View reviewed changes

plugins/plugin_utils/lookup.py Outdated Show resolved Hide resolved

machacekondra approved these changes Sep 3, 2024

View reviewed changes

remove redundant checks and methods in api helper class

32834c5

mikemorency added the mergeit label Sep 9, 2024

softwarefactory-project-zuul bot merged commit 7ee65d2 into ansible-collections:main Sep 9, 2024
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor lookup plugins using top down method #523

refactor lookup plugins using top down method #523

mikemorency commented Aug 27, 2024 •

edited

Loading

softwarefactory-project-zuul bot commented Aug 27, 2024

softwarefactory-project-zuul bot commented Aug 27, 2024

softwarefactory-project-zuul bot commented Aug 28, 2024

mikemorency commented Aug 28, 2024

softwarefactory-project-zuul bot commented Aug 28, 2024

mikemorency commented Aug 30, 2024 •

edited

Loading

bardielle commented Sep 3, 2024

softwarefactory-project-zuul bot commented Sep 3, 2024

softwarefactory-project-zuul bot commented Sep 3, 2024

softwarefactory-project-zuul bot commented Sep 9, 2024

refactor lookup plugins using top down method #523

refactor lookup plugins using top down method #523

Conversation

mikemorency commented Aug 27, 2024 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

softwarefactory-project-zuul bot commented Aug 27, 2024

softwarefactory-project-zuul bot commented Aug 27, 2024

softwarefactory-project-zuul bot commented Aug 28, 2024

mikemorency commented Aug 28, 2024

softwarefactory-project-zuul bot commented Aug 28, 2024

mikemorency commented Aug 30, 2024 • edited Loading

bardielle commented Sep 3, 2024

softwarefactory-project-zuul bot commented Sep 3, 2024

softwarefactory-project-zuul bot commented Sep 3, 2024

softwarefactory-project-zuul bot commented Sep 9, 2024

mikemorency commented Aug 27, 2024 •

edited

Loading

mikemorency commented Aug 30, 2024 •

edited

Loading