Encoding issues with UTF-8 files on z/OS for shell command

[danglunz]

Hi,

our latest z/OS system has been upgraded with Python 3.11 and IZOAU 1.2.3 and we have upgraded our ibm_zos_core collection to 1.6.0 from 1.4.0 (and even tried out the new 1.7.0-beta2) release.
Now we have encoding issues with a specific shell command. We are not sure if this is really related to the ibm_zos_core upgraded and if this is the correct place to ask for support, but as we haven't found any other place we try doing it here:

We have the following task that is being executed on the z/OS system (managed node/target):

    - name: Deploy broker
      shell: 
        cmd: ". /u/user/.profile && java -jar brokerhelper.jar -file /u/user/brokerhelper.xml"
        chdir: "/u/user"
        executable: "/bin/bash"

The file brokerhelper.xml has UTF-8 encoding on the file system and the utility brokerhelper.jar expects the file brokerhelper.xml to be in UTF-8.
The brokerelper.xml prolog contains the xml definition like this:
<?xml version="1.0" encoding="UTF-8"?>

However, when the shell command is being executed the file brokerhelper.xml is being converted by ansible to EBCDIC and the brokerhelper.jar gets that xml file in ebcdic encoding. We do not want that this happens.

We have tried out the following:

1. we have disabled pipeling in ansible.cfg and do not set the PYTHONSTDINENCODING variable.
2. We have tried setting the PYTHONSTDINENCODING with pipeling and this does also not help.
3. We have tried out to use zos_encode and convert the already stored as UTF-8 file on the z/OS system to UTF-8 again and it does make thins even worse:

      ibm.ibm_zos_core.zos_encode:
        src: "/u/user/brokerhelper.xml"
        encoding:
          from: "UTF-8"
          to: "IBM-1047"
        backup: yes  

Any ideas what might help?
Any help appreciated.
Thanks.

[richp405]

@rexemin

[ketankelkar]

You are correct, this is not related specifically to the ibm_zos_core collection. We do try to help out where we can with ansible community modules with z/OS targets because we are aware of the complexities there.

With the information provided here, I can make a few suggestions as to what might help, but we'd need a lot more information to really get a picture of what's going on.

The ansible community modules (eg builtin.shell) open files in binary mode, so there should be no converting happening within that module especially with the 3.11 version of python you've listed above. My first suggestion would be to take a closer look at that .profile file being invoked as the first command.

In the past, we've recommended using the builtin.raw module when dealing with encoding issues, however, those cases tended to be when output was being returned from the zOS target to the ansible controller.

It would help clarify for us what's happening if you could share more of your playbook, the environment variables you've configured for ansible and a verbose dump (-vvv) as well. Also, it would help if you shared how you are verifying the encoding of your target file. Nothing I'm seeing above would suggest to me that any kind of conversion is taking place from the ansible side.

I'm not at all familiar with the Java side of things, but would you be able to just switch the value of the encoding in that xml from utf-8 to ibm-1047 to make your app work? z/OS java would know how to interpret ibm-1047 too right?

[ddimatos]

For the record, this appears to be a cross posting to the Ansible community, for reference here.

[ddimatos]

Hello @danglunz , we will be closing this discussion as it has been over 45 days without activity. Our inactivity policy previously was 60 days and recently reduced to 45 days, thus we are closing this issue. Feel free to reopen the discussion.