Skip to content

Development Minutes

Jump to bottom Edit New page
Igor Todorovski edited this page Feb 13, 2025 · 202 revisions

Feb 13, 2025

Jan 30, 2025

{
  "files": [    {
      "path": "/usr/local/zopen/curl/bin/curl",
      "sha256": "a1b2c3d4e5f6...", 
      "mode": "755"
    },
    {
      "path": "/user/local/zopen/curl/lib/libcurl.so.4.8.0",
      "sha256": "d4e5f6..."
    },
    // Directory
    {
      "path": "/opt/zopen/curl/include/curl"
    },
  ],
  "executables": [
    "bin/curl",
    "bin/curl-config"
  ]
}

  • Notable tool updates:

    • Rsync updates - Resolved CVEs by updating to the latest version
      • Resolved issues related to syncing to/from non-z/OS/z/OS
    • GPG updates
    • zip/unzip/gzip accelerated

  • Public Facing Hardware integrated to CI/CD

  • Guild Meeting

  • Signing our git commits

  • Perl issues

Jan 16, 2025

Mentorship programs:

  • India (RVCE 3 students - Developing a data taxonomy for porting - underway, currently in education phase, understanding how builds work)
  • RCOS (Starting next week)
  • India (Starting in February - PES University - Porting an AI inferencing engine to z/OS)
  • OMP Mentorship program - starting in Summer

zopen discussion:

Next Guild Meeting:

Jan 2, 2025

  • Happy new year! 🎉
  • Changelog update
  • 2025 discussion

2024 - Year in review

  • Reached 200+ projects:
    • Keys new tools include: tmux, emacs, cppcheck and more
  • Officially joined the Open Mainframe Project
  • New contributors:
    • New university programs (Georgia Tech) - Kepler/Dian (zopen audit)
    • 21CS (Aiden - meson, conan)
    • Additional contributors from IBM (through JumpStart program)
    • Simplified contribution process (via issue)
  • Adoption: Grew from 3k to 5k installs per month
  • New functionality in zopen:
    • GPG integration for package signing and verification.
    • Pre-requisite support for packages.
    • Improved handling of conflicting tools
    • Beta Dataset I/O support in editors (Vim, Emacs).
    • zopen audit tool.
    • Improved checks (space, etc)

2024 - Retrospective

  • Difficulty attracting and maintaining new contributors
  • Maintaining consistent quality in the zopen package manager
    • Bugs found by the community, inconsistent error handling
  • Drop-off in zopen community engagement in 2H 2025
    • Fewer zopen guild calls, and overall engagement (discussions, discord)
  • Lack of clear documentation
    • zopen usage and configuration
  • Difference in behaviour between native tools and zopen community tools
  • Lack of z/OS access for new contributors
  • Migration from xlclang to clang - document transition

2025 - Goals

  • zopen:
    • Improving adoption:
      • Enabling support for airgapped environments with registry mirroring and local pax installs
    • Quality: additional unit and end to end testing for zopen
    • Adding support for an rpm package format
    • Inconsistent error handling:
      • Implementing consistent error handling
  • Engagement:
    • Increase the frequency and variety of community events (Guild meetings, OMP events)
    • Upload/improve documentation for new contributors
    • Consider incentives for new contributors
    • Public facing HW access for new contributors
    • Engaging with additional universities and OMP Internship program
  • Upstreaming
    • Monitor and track upstreaming success
    • CICD upstreaming
  • Leveraging AI
    • Can we use AI to improve zopen? or create tests?
    • Can we train an AI to assist in porting open source to z/OS?
  • Tools
  • Security:
    • Achieving OpenSSF badge certification - adding static scanning

2024 minutes

Dec 19

Dec 5

Nov 21

Nov 07

  • info on /usr merge - https://systemd.io/THE_CASE_FOR_THE_USR_MERGE/

  • Port libraries like simdutf8 to speed up dealing with codepages

  • Port systemd ?

  • Now that zopen is in OMP, for secure access, 2 (of many possible) options:

  • Website updates

  • Improving the process for contributing new tools

  • OMP Mentorship program

  • Updates:

    • GPG Integration: GPG is now integrated into the zopen package manager. All future builds will be signed and verified by zopen.
    • Pre-requisite Support: We've added support for managing package pre-requisites. This allows packages to target a specific OS level or feature (e.g., the /proc filesystem).
    • Man Page Conflict Resolution: We've introduced support for resolving conflicts with z/OS man pages. If you source your zopen-config with --override-zos-tools, zopen will prioritize its own man pages. If you omit or specify --nooverride-zos-tools, zopen will default to the z/OS tool man pages. zopen manpages that conflict will also be available in the prefixed form as described here.

  • meta package versioning - add changelog - can leverage Go, Rust packages.

Oct 10

September 26

September 12

August 29

August 14

Aug 1

image

July 18

  • Gary - iconv is the most important tool
  • OEF - add iconv?
  • Upcoming Guild Meeting - July 23 at 10am EST - to be scheduled
    • Zopen Audit
    • Zopen Config
    • New web pages
    • Demos
  • New Tools / Fixes
  • New Poll: https://app.sli.do/event/aCkEVLm7mZ6Cz5UMSYqfbQ
  • Another poll on access (to be released): https://docs.google.com/forms/d/e/1FAIpQLSduwzivmeNnjdMiQK-pOx4IEZKvLHWIyoY42pSahErm_A40Nw/viewform?usp=sf_link .
  • Meta packages
  • Enhanced Security and Verification
    • Signing and Verifying binaries: Integrating GPG in the zopen package manager to ensure the authenticity and integrity of binaries. (near term)
    • Securing Git Commits: Implementing GPG signing for git commits to verify the identity of the commit author and maintain the integrity of the codebase. (near term)
    • Source code static-scanning:  Investigating and implementing static scanning tools such as Cppcheck and CodeQL to enhance code quality and security. (mid term)
    • Enabling fuzz testing – Using Hongfuzz, a security-oriented, feedback-driven, and easy-to-use fuzzing tool, to detect and address vulnerabilities.

July 4

  • How we're handling packages that collide with z/OS UNIX tools under /bin/ #825
  • Meta packages
  • zopen poll
    • Considering providing a trusted mirror under ibm.com or ibm.net to avoid punching a firewall.
      • Gary: ibm.com preferred
    • Would open mainframe project hosted mirror be sufficient?
    • Would you prefer to mirror into a jump server that has access to the internet - vetting
  • Next Guild meeting potential topics:
    • Keplar from GT has agreed to demo the zopen audit enhancements - prefers to demo between July 7-20 - we'll aim to do the APAC and
    • What's to come (static scanning, vulnerability scanning, signing our assets with gpg, enabling mandatory code signing) - Haritha or I
    • Latest zopen updates - Russell?
    • Tool updates (boost, etc) - Haritha and I can cover this.
    • Go ports like Grafana and Prometheus

June 19

June 6

  • OMP and z/OS Open Tools name
  • Prometheus released, or VictoriaMetrics as an alternative,
  • node-exporter
  • Updates to core tools
    • Find/Grep (leaking open file descriptors)
  • Go tools
    • fx - A terminal JSON viewer + processor
    • s5cmd - A parallel S3 and local filesystem execution tool.
    • murex - A modern shell written in Go
    • grafana
  • New release of meta package?
    • zopen audit
    • zopen config
    • zopen-config fixes
    • zopen-config enhancements
  • remove meta release and point to metaport instead
    • simplifies maintenance
  • Best practices
    • Gary's conflicting tools: coretutils, diffutils, findutils, gawk, grep, libiconv, moreutils, sed, tar
  • OEF:
    • zstd, gzip considered, bzip2 for library compression
    • Lionel: no trademark in z/OS for IBM Open Enterprise Foundation for z/OS
  • Website additions
  • Fix https://github.com/ZOSOpenTools/meta/actions/workflows/enable_disabled_workflow.yml
    • Potential solution: run in Jenkins CI
    • Separate script for this, not in docupdate.sh
    • TODO: provide access for external users

May 23

  • Open Mainframe Project status
    • z/OS is trademarked. We cannot use z/OS or Z.
    • Name change and poll - starting with discord a filtered down list from discord https://discord.com/channels/880322471608344597/1144004073390100601/1243020969539272704 Chatting with our trademark team, because of the IBM mark 'z/OS', having the project name as 'z/OS Open Tools' is in violation of the trademark policies from IBM. We could adjust to something like 'Open Tools for z/OS', or even pick an entirely new name. Let me know what you'd prefer
  • Georgia Tech internships
  • Upcoming IBM Product (SMP/E deliverable of 8 tools)
    • Priorities from Gary: reliability, security, stability, performance
    • z/OS man does not support open source man pages - provide man in future versions
    • man-db should honour USS man pages - perhaps z/OS man should be enhanced to support both USS and groff format
      • could be handled through a converter, man-db can also be updated to understand the USS format
  • Grafana and Murex now available - Thanks Joon

Outstanding work:

  • Enhancements to zopen to be aware of IBM product
    • zopen install to check for IBM product in the file system (can be skipped with a toggle, --ignore-oef)
    • Lionel: do not add extra logic to consider OEF tools, users installing the tools should manage them themselves
  • Change comp_* to check_*
    • Enhance zopen to filter out comp_* and potentially change name from comp_ to check_
    • metadata change to be able to filter out in regular installs
  • Mechanism to toggle z/OS tools vs z/OS Open Tools (GNU Make vs /bin/make, GNU Awk vs /bin/awk)
  • Discuss: https://github.com/ZOSOpenTools/gawkport/issues/23
  • New zopen installer
  • z/OS system access

May 9

April 24

April 10

Recording pass: bGcyrzK3

  • Next guild meeting topics
    • V8 port
    • New zopen features
    • Vim LSP support
  • Go 1.22 on dev sandbox and CI machine:
    • /usr/lpp/IBM/cvg/v1r22/go/bin on dev machine
  • Murex - a shell written in Go
    • CI/CD - Joon will create the repo contents
  • Seeing this on v2r4 - limit it to run on v2r5 CI machines
/jenkins/workspace/Port-Build/godsect/godsect
go version go1.22.2 zos/s390x
CEE3728S The use of a function, which is not supported by this release of Language Environment was detected.
         From entry point _rt0_s390x_zos at compile unit offset +00000000266B57C4 at entry offset +000000000000137C at address 00000000266B57C4.
***ERROR: Make (minimal) failed. Log: /jenkins/workspace/Port-

March 28, 2024

March 14, 2024

Recording Pass: xZHMRsb7

February 29, 2024

View the Recording, Pass: NmnixjM5

February 15, 2024

Igor Todorovski's Personal Room-20240215 1415-1 pass: yRtNYaA7

February 1, 2024

Recording: Igor Todorovski's Personal Room-20240201 1601-1 6vDBvxwv

Add your topics here:

  • (Vignesh) For analytics & the OSV scanner, what if the users' systems didn't need to talk to a zopen site? Instead, we can add a verb, say zopen amisecure, and it'll generate a short block of text with tool & version numbers. Then, they paste it in a page in the zopen site and that calls the OSV API to get results. This'll allow us to cache status info of tool-version combos and do fewer calls overall. Can't remember... do we have more functionality than this planned for / currently done by the analytics?

    • https://google.github.io/osv-scanner/usage/#cc-scanning
    • Proposal: zopen audit tool (copied from npm)
      • Identifies all tools that have an open CVE .
      • zopen upgrade --vulnerable - updates vulnerable tools
      • Implementation:
        • Use osv-scanner as part of the CI/CD process to scan the source code and build a database of tool/version + vulnerabilities
        • We can then use zopen audit and/or a website to query for existing vulnearbilities

  • UTF8 Git support:

    • Draft PR: https://github.com/ZOSOpenTools/gitport/pull/108
    • __ae_autoconvert_state - does it persist on pthread create/spawn
      • It persists on spawn, but not persist on new threads. The environment variable is used to determine the auto convert state.
      • If we use setenv in zoslib, we can alleviate it.

  • Open Mainframe Project

    • Under evaluation (speaking with council reps)
      • Benefits: Bring in contributors from different organizations (interns), Drive broad industry awareness with the press and analysts, OMP takes all liability, they can fund the creation of a logo, paid zoom license/slack channel
      • Downsides: Requires yearly re-validation (typically a non-issue as long as you are active), meetings every other week (optional), share ownership with OMP, specific rules in place (code signing (cbttape does not code sign), licensing)

  • Meeting duration/timing/frequency:

    • Proposal to change the meeting to 1 hour earlier (for folks in India).
    • 2 hours earlier and every other week.

  • Guild Meeting - Feb. 7

Jan 25, 2024

Recording: Igor Todorovski's Personal Room-20240125 1618-1 pW6BR2Xh

Topics:

  • Git UTF8 encoding/file tag
  • System-wide UTF8 - As a bare minimum, which areas need to add support, to allow this optional z/OS-native UTF8 ecosystem to grow (added by Vignesh; not attending this one, just wanted to bring this up; delete this note later)

Git (UTF8)

z/OS Open Tools Git currently tags UTF-8 files as ASCII ISO8859-1 (819).

Problems:

  • Violates principle of least surprise - The file tag and encoded contents are not matching
  • 819 is not a complete subset of UTF-8 (e..g ¬ ), 819 characters can be represented as UTF-8.
  • working-tree-encoding=ISO8859-1 is considered the same as UTF8 (no conversion)

Proposal:

  • Switching default tag for UTF-8 encoded files to 1208
  • working-tree-encoding=ISO8859-1 will convert from UTF8 to ISO8859-1

Benefits:

  • Smoother transition for Rocket Git users
  • Consistency, no special case
  • Users can rely on .gitattributes or GIT_UTF8_CCSID environment variable to override default setting.
  • z/OS Open Tools programs already work with UTF-8 since _BPXK_AUTOCVT=ON is set (no translation occurs)

Problems:

  • Existing z/OS Git users may be affected by the change in default behaviour.
  • IF _BPXK_AUTOCVT=ALL is set, UTF8 to 819 (pccsid) will cause issues
  • Potential solution, set PCCSID to 1208

_BPXK_AUTOCVT

ON Activates the automatic file conversion of tagged files. This option affects conversion for I/O for regular, pipe, and character-special files that are tagged. OFF Deactivates the automatic file conversion of tagged files. OFF is the default. ALL Activates the automatic conversion of tagged files that are supported by Unicode Services. This option affects conversion for I/O for regular and pipe files that are tagged. Setting or unsetting ALL has no effect after translation for a file begins. If the conversion is between EBCDIC and ASCII, this option also affects conversion for I/O for character special files.

_BPXK_PCCSID

Identifies the program CCSID for the running thread or user. It can be used to override the internal default of 1047 (EBCDIC). Any value between 0 and 65535 can be assigned, but to avoid any subsequent errors, only values that are supported by Unicode Services should be used. Setting or unsetting this variable has no effect after translation for a file begins. When unset, the internal value of the program CCSID reverts to the default of 1047.

#include <stdio.h>
#include <stdlib.h>
#include <env.h>

#define BUFFER_SIZE 4096

int main() {
    char buffer[BUFFER_SIZE];
    
    setenv("_BPXK_PCCSID", "1208", 1);
    FILE *fin = fopen("codepage.txt","r");
    fgets(buffer, 100, fin);
    fprintf(stdout,"%s", buffer);
}
 вход
bash-5.2$ /bin/cat (PCSSID=1047) a.txt (1208)
cat: FSUM6180 file "a.txt": EDC5122I Input/output error.
bash-5.2$ vim a.txt
bash-5.2$ /bin/cat a.txt
💁👌🎍😍

TODOs:

  • unicode system services - requires additional set up
  • __ae_autoconvert_state - does it persist on pthread create/spawn
  • research into why outputting into tty does not display anything
  • _BPXK_TECHNIQUE=
  • _BPXK_TECHNIQUE
  • _BPXK_UNICODE_SUB
  • _BPXK_UNICODE_MAL
  • Pipes cannot be tagged as 1208 - double check
  • Why is there not function to set pccsid?
  • Support for other EBCDIC code pages 1140 or 037 (does it work with AUTOCVT=ALL)?

Jan 18, 2024

Attendees: Joon, CW, Anthony, John, Lionel, Haritha, Russell, Igor

Recording: Igor Todorovski's Personal Room-20240118 1601-1 GuKkEuE6

Agenda

image
  • Tools/Libraries being worked on:
    • tmux (Igor)
    • Boost (giridhar)
    • Terraform (Pradeep)
    • ccache (Sean)
    • caddy (Joon)
    • neofetch (Anthony)
    • z/OS Tools and Toys (Anthony)
    • Prometheus (zhilong)
    • fq
    • osv-scanner

Technical discussions

z/OS Open Tools Git currently tags UTF-8 files as ASCII ISO8859-1 (819).

Problems:

  • Violates principle of least surprise - The file tag and encoded contents are not matching
  • 819 is not a complete subset of UTF-8 (e..g ¬ ), 819 characters can be represented as UTF-8.

Proposal: Switching default tag for UTF-8 encoded files to 1208

Benefits:

  • Smoother transition for Rocket Git users
  • Consistency, no special case
  • Users can rely on .gitattributes or GIT_UTF8_CCSID environment variable to override default setting.
  • Z/OS Open Tools programs already work with UTF-8 since _BPXK_AUTOCVT=ON is set (no translation occurs)

Problems:

  • Existing z/OS Git users may be affected by the change in default behaviour.

_BPXK_AUTOCVT

ON Activates the automatic file conversion of tagged files. This option affects conversion for I/O for regular, pipe, and character-special files that are tagged. OFF Deactivates the automatic file conversion of tagged files. OFF is the default. ALL Activates the automatic conversion of tagged files that are supported by Unicode Services. This option affects conversion for I/O for regular and pipe files that are tagged. Setting or unsetting ALL has no effect after translation for a file begins. If the conversion is between EBCDIC and ASCII, this option also affects conversion for I/O for character special files.

_BPXK_PCCSID

Identifies the program CCSID for the running thread or user. It can be used to override the internal default of 1047 (EBCDIC). Any value between 0 and 65535 can be assigned, but to avoid any subsequent errors, only values that are supported by Unicode Services should be used. Setting or unsetting this variable has no effect after translation for a file begins. When unset, the internal value of the program CCSID reverts to the default of 1047.

Why UTF-8 tagged files are problematic:

Problems with _BPXK_PCCSID:

#include <stdio.h>
#include <stdlib.h>

#define BUFFER_SIZE 4096

int main() {
    FILE *inputFile, *outputFile;
    char inputFileName[100], outputFileName[100];
    char buffer[BUFFER_SIZE];
    size_t bytesRead;
    fprintf(stderr, "Before call\n");
    setenv("_BPXK_PCCSID", "1208", 1);
    fprintf(stderr, "After call\n");
}

Program will exit after setenv!

Jan 11, 2024

Attendees: Igor, Mike, Lionel, Erik (mfsysprog), Joon, Vignesh

Recording: Igor Todorovski's Personal Room-20240111 1600-1 mFn62EHp

Agenda

  • Happy new year!
  • Dev Meetings Purpose
    • Facilitate better communication, discuss and address contentious topics, get feedback on development activities, and steer the direction of z/OS Open Tools.
  • Intros
  • Upcoming Guild Meetings
    • APAC: Jan. 23 (7am EST)
    • Americas/Europe: Feb. 7 (11am EST)
  • What we're working on?
    • Evangelism:
      • Logo (identity)
      • LinkedIn Group?
      • Testimonials/Success Stories
      • Make z/OS a first class platform by providing real zHW
      • Social media?
    • Collecting z/OS Open Tools usage
    • Package manager improvements (local installs)
    • V8/gn/depot_tools
    • zoslib
    • currency
    • Upstreaming (Git)
  • Tools to consider for 2024:
image
  • Tools/Libraries being worked on:
    • tmux (Igor)
    • Boost (giridhar)
    • Terraform (Pradeep)
    • ccache (Sean)
    • gitlab (Joon)
    • caddy (Joon)
    • neofetch (Anthony)
    • z/OS Tools and Toys (Anthony)
    • Prometheus (zhilong)

Technical discussions

Git's handling of UTF-8 encoded files

  • z/OS Open Tools Git:
    • Currently tags UTF-8 files as ASCII ISO8859-1 (819)
    • Pros:
      • Works with ebcdic programs (like cat): 819 tagged files get auto-converted for EBCDIC programs with AUTOCVT=ON
      • Does not require setting AUTOCVT=ALL
    • Cons:
      • Violates principle of least surprise - The file tag and encoded contents are not matching
      • Some chars like ¬ have a different encoding in UTF-8 vs 819.
  • Rocket's Git
    • Currents tags UTF-8 files as UTF8 (1208)
    • Pros:
      • Encoding contents and file tag match. More intuitive
    • Cons:
      • UTF8 tagged files do not get auto-converted to EBCDIC for ebcdic programs (in AUTOCVT=ON)
      • Requires AUTOCVT=ALL which is not widely used
Add a custom footer
Add a custom sidebar
Clone this wiki locally