iosxr_config appears to be stripping commas from jinja2 template

[calman-isu]

SUMMARY

I'm configuring a prefix-list via a jinja2 template, and when I have iosxr_config apply the template, I get "Invalid input detected at '^' marker". However, when I manually apply the template the router accepts the config. If I manually apply the config but remove the comma (,), I get the same error message the iosxr_config module throws.

I successfully use this same workflow for several other configuration sections, the only difference I can tell from this one is this one contains commas.

ISSUE TYPE

• Bug Report

COMPONENT NAME

iosxr_config

ANSIBLE VERSION

ansible 2.9.4
  config file = /home/myuser/ansible-backbone/ansible.cfg
  configured module search path = ['/home/myuser/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/myuser/.local/lib/python3.6/site-packages/ansible
  executable location = /home/myuser/.local/bin/ansible
  python version = 3.6.8 (default, Sep 26 2019, 11:57:09) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

CONFIGURATION

DEFAULT_CALLBACK_WHITELIST(/home/myuser/ansible-backbone/ansible.cfg) = ['yaml']
DEFAULT_FORKS(/home/myuser/ansible-backbone/ansible.cfg) = 8
DEFAULT_LOAD_CALLBACK_PLUGINS(/home/myuser/ansible-backbone/ansible.cfg) = True
DEFAULT_LOG_PATH(env: ANSIBLE_LOG_PATH) = /home/myuser/ansible.log
DEFAULT_STDOUT_CALLBACK(/home/myuser/ansible-backbone/ansible.cfg) = yaml
DEPRECATION_WARNINGS(/home/myuser/ansible-backbone/ansible.cfg) = True
HOST_KEY_CHECKING(/home/myuser/ansible-backbone/ansible.cfg) = False
PERSISTENT_COMMAND_TIMEOUT(/home/myuser/ansible-backbone/ansible.cfg) = 90
PERSISTENT_CONNECT_TIMEOUT(/home/myuser/ansible-backbone/ansible.cfg) = 115

OS / ENVIRONMENT

Red Hat Enterprise Linux Server release 7.8 (Maipo)

STEPS TO REPRODUCE

$ cat /tmp/iosxr-PS_ISU_TEST_IPV4.prefix-list
192.168.1.0/24,
10.10.10.0/24
end-set

- name: 20-configure-v4-prefix-set Apply VRF V4 prefix set
  cisco.iosxr.iosxr_config:
    lines: "{{ lookup('file', '/tmp/' + ansible_network_os + '-PS_ISU_' + item.name + '_IPV4.prefix-list' ) }}"
    replace: block
    match: strict
    comment: "Configured by Ansible play '{{ ansible_play_name }}'"
    parents: "prefix-set PS_ISU_{{ item.name }}_IPV4"
  with_items:
    - "{{ vrf }}"
  register: results
  tags:
    - prefix-set

EXPECTED RESULTS

RP/0/RP0/CPU0:nr-rtr-e63-mpls-pe-wifi3#show running-config prefix-set PS_ISU_TEST_IPV4
Mon Jun 22 16:29:16.620 CDT
prefix-set PS_ISU_TEST_IPV4
192.168.1.0/24,
10.10.10.0/24
end-set
!

ACTUAL RESULTS

ansible-playbook 2.9.4
config file = /home/myuser/ansible-backbone/ansible.cfg
configured module search path = ['/home/myuser/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/myuser/.local/lib/python3.6/site-packages/ansible
executable location = /home/myuser/.local/bin/ansible-playbook
python version = 3.6.8 (default, Sep 26 2019, 11:57:09) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
Using /home/myuser/ansible-backbone/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /home/myuser/itnet-ansible-inventory-prod/inventory as it did not pass its verify_file() method
script declined parsing /home/myuser/itnet-ansible-inventory-prod/inventory as it did not pass its verify_file() method
auto declined parsing /home/myuser/itnet-ansible-inventory-prod/inventory as it did not pass its verify_file() method
Parsed /home/myuser/itnet-ansible-inventory-prod/inventory inventory source with ini plugin
Loading callback plugin yaml of type stdout, v2.0 from /home/myuser/.local/lib/python3.6/site-packages/ansible/plugins/callback/yaml.py

PLAYBOOK: ncs-prefix-list.yml *******************************************************************************************************************************************************
Positional arguments: /home/myuser/ansible-backbone/ncs-prefix-list.yml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/home/myuser/itnet-ansible-inventory-prod/inventory',)
vault_password_files: ('/srv/ansible/.ansible_vault',)
forks: 8
1 plays in /home/myuser/ansible-backbone/ncs-prefix-list.yml

PLAY [Configure NCS routers] ********************************************************************************************************************************************************
META: ran handlers
Trying secret FileVaultSecret(filename='/srv/ansible/.ansible_vault') for vault_id=default
Trying secret FileVaultSecret(filename='/srv/ansible/.ansible_vault') for vault_id=default
Trying secret FileVaultSecret(filename='/srv/ansible/.ansible_vault') for vault_id=default

TASK [ncs-prefix-list : 20-configure-v4-prefix-set Apply VRF V4 prefix set] *********************************************************************************************************
task path: /home/myuser/ansible-backbone/roles/ncs-prefix-list/tasks/main.yml:2
File lookup using /tmp/iosxr-PS_ISU_TEST_IPV4.prefix-list as file
File lookup using /tmp/iosxr-PS_ISU_TEST_IPV4.prefix-list as file
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> attempting to start connection
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> using connection plugin network_cli
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> attempting to start connection
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> using connection plugin network_cli
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> local domain socket does not exist, starting it
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> control socket path is /home/myuser/.ansible/pc/583969ae82
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> local domain socket listeners started successfully
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> loaded cliconf plugin iosxr from path /home/myuser/.local/lib/python3.6/site-packages/ansible/plugins/cliconf/iosxr.py for network_os iosxr
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org>
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> local domain socket path is /home/myuser/.ansible/pc/583969ae82
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> local domain socket does not exist, starting it
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> control socket path is /home/myuser/.ansible/pc/9d4d868e38
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> local domain socket listeners started successfully
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> loaded cliconf plugin iosxr from path /home/myuser/.local/lib/python3.6/site-packages/ansible/plugins/cliconf/iosxr.py for network_os iosxr
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org>
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> local domain socket path is /home/myuser/.ansible/pc/9d4d868e38
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> ESTABLISH LOCAL CONNECTION FOR USER: myuser
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.6055048-83158123843394" && echo ansible-tmp-1592861900.6055048-83158123843394="echo /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.6055048-83158123843394" ) && sleep 0'
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> ESTABLISH LOCAL CONNECTION FOR USER: myuser
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.7736683-213160514981160" && echo ansible-tmp-1592861900.7736683-213160514981160="echo /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.7736683-213160514981160" ) && sleep 0'
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> Attempting python interpreter discovery
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> Attempting python interpreter discovery
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
Using module file /home/myuser/.ansible/collections/ansible_collections/cisco/iosxr/plugins/modules/iosxr_config.py
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> PUT /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/tmpbl6gob0h TO /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.6055048-83158123843394/AnsiballZ_iosxr_config.py
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c 'chmod u+x /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.6055048-83158123843394/ /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.6055048-83158123843394/AnsiballZ_iosxr_config.py && sleep 0'
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c '/usr/bin/python /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.6055048-83158123843394/AnsiballZ_iosxr_config.py && sleep 0'
Using module file /home/myuser/.ansible/collections/ansible_collections/cisco/iosxr/plugins/modules/iosxr_config.py
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> PUT /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/tmpbbn50p22 TO /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.7736683-213160514981160/AnsiballZ_iosxr_config.py
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c 'chmod u+x /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.7736683-213160514981160/ /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.7736683-213160514981160/AnsiballZ_iosxr_config.py && sleep 0'
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c '/usr/bin/python /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.7736683-213160514981160/AnsiballZ_iosxr_config.py && sleep 0'
<nr-rtr-e63-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c 'rm -f -r /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.7736683-213160514981160/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may not be related to the actual failure.
File "/tmp/ansible_cisco.iosxr.iosxr_config_payload_q7VvaB/ansible_cisco.iosxr.iosxr_config_payload.zip/ansible_collections/cisco/iosxr/plugins/module_utils/network/iosxr/iosxr.py", line 559, in load_config
label=label,
File "/tmp/ansible_cisco.iosxr.iosxr_config_payload_q7VvaB/ansible_cisco.iosxr.iosxr_config_payload.zip/ansible/module_utils/connection.py", line 185, in rpc
raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
failed: [nr-rtr-e63-mpls-pe-wifi3.tele.my.org] (item={'name': 'TEST', 'route_target': 999, 'v4_supernets': ['192.168.1.0/24', '10.10.10.0/24']}) => changed=false
ansible_facts:
discovered_interpreter_python: /usr/bin/python
ansible_loop_var: item
invocation:
module_args:
admin: false
after: null
backup: false
backup_options: null
before: null
comment: Configured by Ansible play 'Configure NCS routers'
config: null
exclusive: false
force: false
label: null
lines:
- 192.168.1.0/24
- |2-

    10.10.10.0/24
    end-set
  match: strict
  parents:
  - prefix-set PS_ISU_TEST_IPV4
  provider: null
  replace: block
  src: null

item:
name: TEST
route_target: 999
v4_supernets:
- 192.168.1.0/24
- 10.10.10.0/24
msg: |-
10.10.10.0/24
^
% Invalid input detected at '^' marker.
RP/0/RP0/CPU0:nr-rtr-e63-mpls-pe-wifi3(config-pfx)#
<nr-rtr-b31-mpls-pe-wifi3.tele.my.org> EXEC /bin/sh -c 'rm -f -r /home/myuser/.ansible/tmp/ansible-local-41798w1zx56vv/ansible-tmp-1592861900.6055048-83158123843394/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may not be related to the actual failure.
File "/tmp/ansible_cisco.iosxr.iosxr_config_payload_ioR6h7/ansible_cisco.iosxr.iosxr_config_payload.zip/ansible_collections/cisco/iosxr/plugins/module_utils/network/iosxr/iosxr.py", line 559, in load_config
label=label,
File "/tmp/ansible_cisco.iosxr.iosxr_config_payload_ioR6h7/ansible_cisco.iosxr.iosxr_config_payload.zip/ansible/module_utils/connection.py", line 185, in rpc
raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
failed: [nr-rtr-b31-mpls-pe-wifi3.tele.my.org] (item={'name': 'TEST', 'route_target': 999, 'v4_supernets': ['192.168.1.0/24', '10.10.10.0/24']}) => changed=false
ansible_facts:
discovered_interpreter_python: /usr/bin/python
ansible_loop_var: item
invocation:
module_args:
admin: false
after: null
backup: false
backup_options: null
before: null
comment: Configured by Ansible play 'Configure NCS routers'
config: null
exclusive: false
force: false
label: null
lines:
- 192.168.1.0/24
- |2-

    10.10.10.0/24
    end-set
  match: strict
  parents:
  - prefix-set PS_ISU_TEST_IPV4
  provider: null
  replace: block
  src: null

item:
name: TEST
route_target: 999
v4_supernets:
- 192.168.1.0/24
- 10.10.10.0/24
msg: |-
10.10.10.0/24
^
% Invalid input detected at '^' marker.
RP/0/RP0/CPU0:nr-rtr-b31-mpls-pe-wifi3(config-pfx)#

PLAY RECAP **************************************************************************************************************************************************************************
nr-rtr-b31-mpls-pe-wifi3.tele.my.org : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
nr-rtr-e63-mpls-pe-wifi3.tele.my.org : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

[mhofer117]

We are having the same / very similar issue. For us it fails as soon as there 2 or more sets (community, prefix, ...), even without commas. It can be reproduced with the following basic config:

community-set cust-id-test
  12:56
end-set
!
community-set services-testpf0000
  12:78
end-set
!

It always fails to execute the second end-set and stays inside the set context:

The full traceback is:
  File "/tmp/ansible_iosxr_config_payload_7byniuay/ansible_iosxr_config_payload.zip/ansible/module_utils/network/iosxr/iosxr.py", line 475, in load_config
    exclusive=exclusive, replace=replace, comment=comment, label=label)
  File "/tmp/ansible_iosxr_config_payload_7byniuay/ansible_iosxr_config_payload.zip/ansible/module_utils/connection.py", line 185, in __rpc__
    raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
fatal: [router1]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "admin": false,
            "after": null,
            "backup": false,
            "backup_options": null,
            "before": null,
            "commands": [
                "community-set cust-id-test",
                "  12:34",
                "end-set",
                "!",
                "community-set services-testpf0000",
                "  12:78",
                "end-set",
                "!"
            ],
            "comment": null,
            "config": null,
            "exclusive": false,
            "force": false,
            "host": null,
            "label": "UL902715781-1",
            "lines": [
                "community-set cust-id-test",
                "  12:34",
                "end-set",
                "!",
                "community-set services-testpf0000",
                "  12:78",
                "end-set",
                "!"
            ],
            "match": "none",
            "parents": null,
            "password": null,
            "port": null,
            "provider": null,
            "replace": "block",
            "src": null,
            "ssh_keyfile": null,
            "timeout": null,
            "username": null
        }
    },
    "msg": "show commit changes diff\r\n\r                                                ^\r\n% Invalid input detected at '^' marker.\r\nRP/0/RSP0/CPU0:router1(config-comm)#"
}

Ansible version: 2.9.18

Task example:

    - name: set facts
      local_action:
        module: set_fact
        configuration: "{{ lookup('file', 'config.txt').splitlines() }}"
        rolloutId: "{{ 999999999 | random(start=900000000) }}"
      delegate_to: localhost
      run_once: yes

    - name: apply config
      iosxr_config:
        commands: "{{ configuration }}"
        label: UL{{ rolloutId }}-1
        comment: null
        match: none
        replace: block
      register: result_config

It works when applying the same config with the iosxr_command module:

    - name: set facts
      local_action:
        module: set_fact
        configuration: "{{ lookup('file', 'config.txt').splitlines() }}"
        rolloutId: "{{ 999999999 | random(start=900000000) }}"
      delegate_to: localhost
      run_once: yes

    - name: prepare command list
      local_action:
        module: set_fact
        command_list: "{{ [ 'config' ] + configuration + [ 'commit label UL' + rolloutId + '-1', 'show configuration commit changes last 1', 'end' ] }}"
      delegate_to: localhost
      run_once: yes

    - name: apply config
      iosxr_command:
        commands: "{{ command_list }}"
      changed_when: True
      register: result_config

Already tried different timeouts, match and replace params to no avail.
What can I do to further debug this issue?

[oh-c]

I think I've been hitting the same issue. I'm updating prefix-sets on an IOS XR 7.1.3

Reading the debug output I think the reason is that the module never sends an "end-set" before doing "show commit changes diff"

Looking at the output below, I can see that the router is in "config-pfx" mode - and if I run the "show commit changes diff" manually on the device inside prefix-set update I get "Invalid inputet detected".

2021-04-28 21:29:12,157 p=46948 u=user n=ansible | response-3: b'RP/0/RSP1/CPU0:router(config-pfx)#'
2021-04-28 21:29:12,158 p=46948 u=user n=ansible | matched error regex (terminal_stderr_re) 'b'invalid input'' from response 'b"show commit changes diff\r\n\r                                                 ^\r\n% Invalid input detected at '^' marker.\r\nRP/0/RSP1/CPU0:router(config-pfx)#"'
2021-04-28 21:29:12,158 p=46948 u=user n=ansible | matched stdout regex (terminal_stdout_re) 'b'[\\r\\n]*[\\w+\\-\\.:\\/\\[\\]]+(?:\\([^\\)]+\\)){,3}(?:>|#) ?$'' from error response 'b"show commit changes diff\r\n\r                                                 ^\r\n% Invalid input detected at '^' marker.\
\nRP/0/RSP1/CPU0:router(config-pfx)#"'

[oh-c]

Digging a bit deeper. If I change the "show commit changes diff" to "show" in

cisco.iosxr/plugins/cliconf/iosxr.py

Line 177 in e92ab97

resp["show_commit_config_diff"] = self.get("show commit changes diff")

- I now get the error below.

It tries to commit the changes, without doing an end-set - and therefore commit will fail. Looks like a change in behaviour in IOS-XR 7.1 - I don't have an older version to check with.

matched cli prompt 'b'\r\n\r\n\rRP/0/RSP1/CPU0:router(config-pfx)#''
command: b'commit comment configured by iosxr_config'
 response-1: b'commit comment configured by iosxr_config\r\n\r'
 response-2: b"                                            ^\r\n% Invalid input detected at '^' marker.\r\n"

[oh-c]

@ashwini-mhatre and @NilashishC I see you're the last two comitters on https://github.com/ansible-collections/cisco.iosxr/blob/e92ab97616fa17b71afa149788d830b74be5182d/plugins/cliconf/iosxr.py - Can you confirm that the module is not doing "end-set" / "end" before trying to commit?

[ashwini-mhatre]

@oh-c I recently fix similar issue. following the the PR for the same #158 .

[lamehost]

It always fails to execute the second end-set and stays inside the set context:

Same problem here.

[Telstra-Elvis-Choi]

Do we have any update on this?