Skip to content

Conversation

@eric-downes
Copy link

After fiixng many things in the dependent libraries, I migrated the complete mcdp_posets module to Python 3 with the following changes:

  • Update metaclass syntax for Python 3 compatibility
  • Convert string formatting to f-strings throughout
  • Add hash methods for classes with eq methods
  • Fix iterator handling (e.g., zip() returns iterator in Python 3)
  • Replace deprecated time.clock() with time.process_time()
  • Add explicit imports for functions no longer built-in (functools.reduce)
  • Replace sys.maxint with sys.maxsize
  • Fix invalid escape sequences in docstrings
  • Update print statements in test files

Testing is blocked by dependencies requiring migration (mcdp_library module).

- Add Black formatter and configuration
- Add Flake8 linter with config
- Create pytest structure with basic tests
- Set up Travis CI configuration
- Improve README documentation
- Add pyproject.toml for modern packaging
- Create requirements-dev.txt for development dependencies
- Document modernization plans
- Include specific tasks and priorities
- Add references for implementation
- Mark completed modernization tasks
- Add detailed future implementation plan
- Organize improvements into phases
- Add specific next steps with priorities
- Create comprehensive migration strategy
- Document testing approach and recovery procedures
- Add step-by-step migration process
- Include common Python 2 to 3 migration issues
- Setup incremental testing procedures
- Create py_compatibility.py for cross-version compatibility
- Fix inspect.getargspec -> getfullargspec for Python 3
- Update string formatting to use f-strings
- Handle exception raising syntax changes
- Make dependencies soft-fail instead of hard-fail
- Update numpy.seterr() call syntax for Python 3
- Create and run basic import tests
- Add STRICT_DEPENDENCIES flag to toggle dependency checking behavior
- Update all dependency checks to use the flag for future strict mode
- Add fallback implementations for contracts and memoize_simple
- Use f-strings for string formatting
- Fix exception handling
- Clean up code for Python 3 compatibility
- Add Python 3 version of memoize_simple_py3.py
- Create indent_utils.py to avoid contracts dependency
- Update string_repr.py to work with Python 3
- Fix pickle and StringIO imports in debug_pickler.py
- Add compatibility code for Python 2/3 in memoize_simple_imp.py
- Add unit tests for memoize_simple functionality
- Add UTF-8 encoding for string to bytes conversion in hash functions
- Use f-strings instead of % formatting for format_list function
- Improve variable naming (l -> items) for better readability
- Add comprehensive docstrings
- Create and verify tests for all functions
- Use f-strings instead of % formatting
- Add comprehensive docstring for the duration_compact function
- Create tests to verify the functionality
- Fix edge cases in the tests to match actual behavior
- Add detailed progress report in py3_migrate_details.md
- Document all migrated files and their status
- Include recovery strategies and workarounds
- Update main migration plan with completed tasks
- Document next steps for continuing migration
- Fix PyContracts dependency through fork with Python 3.8+ fixes
- Update fileutils.py with proper string/bytes handling
- Enhance natsort.py with improved natural sorting
- Update safe_pickling.py with Python 3 pickle protocol support
- Improve safe_write.py with encoding support for text modes
- Update my_yaml.py to handle both old and new ruamel.yaml API
- Add comprehensive Python 3 compatibility tests
- Update migration documentation
- Update mcdp/development.py to import all_disabled directly
- Update string_repr.py to import indent directly
- Update debug_pickler.py to import describe_type directly
- Update mcdp_utils_misc/__init__.py to use memoize_simple_py3 directly
- Update migration documentation with progress on challenges
- Identify quickapp's imp dependency as a remaining issue
- Create patched version of compmake to replace imp.reload with importlib.reload
- Fix inspect.getargspec usage in compmake with inspect.getfullargspec for Python 3
- Fix escape sequence warnings in compmake
- Add setup_py3_deps.sh script to install patched dependencies
- Update migration documentation with progress on dependency issues
- Successfully run tests with patched dependencies
- Document the ZLogger import issue
- Explain the root cause and the impact on our migration
- Present multiple solution options
- Recommend keeping STRICT_DEPENDENCIES=False for now
- Provide implementation details for a future fix if needed
- Document changes made to PyContracts for Python 3.8+ compatibility
- Document changes made to compmake for Python 3.8+ compatibility
- Document changes and remaining issues for quickapp
- Include detailed explanations of the issues and their solutions
- Provide installation instructions for the patched dependencies
- Update timing.py to use time.process_time() instead of time.clock()
- Update string formatting to f-strings for better readability
- Fix collections import in locate_files_imp.py
- Enhance error handling in memos_selection.py
- Add comprehensive tests for all utility modules
- Update metaclass syntax for Python 3 compatibility
- Convert string formatting to f-strings
- Replace time.clock() with time.process_time()
- Fix for-else clause in space_meta.py
- Add clearer class docstrings
- Create migration progress tracking file

This commit updates the core infrastructure of the mcdp_posets module, including:
space_meta.py, space.py, poset.py, utils.py, and find_poset_minima directory.
- Migrate uppersets.py with proper f-string formatting
- Migrate rcomp.py with improved f-strings
- Migrate nat.py with sys.maxsize replacing sys.maxint
- Update progress tracking file with completed modules

These changes bring the core poset implementations into Python 3 compatibility
while preserving their mathematical semantics. The changes include string
formatting improvements, removal of Python 2 specific long type, and explicit
handling of reduce() which is no longer a built-in function in Python 3.
@eric-downes eric-downes changed the title Py3 update [WIP; do not merge] Py3 update Apr 10, 2025
@eric-downes
Copy link
Author

Started migrating dependent modules needed to properly test mcdp_posets:

    1. Fixed exception re-raising in several key modules:
       - mcdp_library/library.py
       - mcdp_lang/parse_interface.py
       - mcdp_lang/parse_actions.py
       - mocdp/comp/template_for_nameddp.py
   
   2. Added compatibility for collections.abc module in Python 3.12:
      - Replaced collections.MutableMapping with collections.abc.MutableMapping
      - Replaced collections.Sequence with collections.abc.Sequence
      - Added fallback imports for Python 3.11 and below
   
   3. Added fallback for nose.tools imports that rely on the removed imp module

@eric-downes
Copy link
Author

eric-downes commented Apr 10, 2025

1. Exception Handling Fixes

Fixed Python 2 style exception re-raising by updating:

  • Fixed raise e, None, traceback to Python 3's raise e.with_traceback(tb) in multiple files:
    • src/mcdp_library/library.py
    • src/mcdp_lang/parse_interface.py
    • src/mcdp_lang/parse_actions.py
    • src/mocdp/comp/template_for_nameddp.py

2. Collections Module Compatibility

Updated imports to support Python 3.12's removal of ABC classes from collections module:

  • Created compatibility imports for:
    • Sequence
    • MutableMapping
    • Mapping
    • Set
    • MutableSet
    • Iterable
  • Added the compatibility layer to multiple files:
    • src/mcdp_posets/poset_product.py
    • src/mcdp_lang/pyparsing_bundled.py

3. String/Bytes Handling

Fixed string vs. bytes handling for Python 3:

  • Updated the decode_identifier function in src/mcdp_lang/syntax.py to handle both Python 2 and 3
  • Created helper functions in pyparsing_compat.py to handle string encoding/decoding
  • Fixed parse_wrap function in src/mcdp_lang/parse_actions.py to handle Python 3 strings

4. Print Statement Conversion

  • Automatically fixed over 500 instances of Python 2 print statements to use Python 3's print function syntax
  • Created fix_print_statements.py script to automate this process

5. Pyparsing Compatibility Layer

Created a comprehensive compatibility layer to handle differences between pyparsing 2.x and 3.x:

  • Created src/mcdp_lang/pyparsing_compat.py which:
    • Tries to import from modern pyparsing 3.x first, then falls back to bundled version
    • Handles API differences between versions (camelCase vs snake_case)
    • Provides helper functions for common operations
    • Adds string/bytes conversion utilities
    • Fixed issues with the oneOf function to handle keyword parameters correctly

The library now attempts to use the installed pyparsing 3.x when available, falling back to the bundled version only when necessary.

Known Issues

  • The test case syntax_anyof.py still doesn't run due to a memoization issue with unhashable types.

@eric-downes
Copy link
Author

  1. Fixed Escape Sequences: Addressed invalid escape sequences in strings that would cause warnings or errors in Python 3, fixing 8
    files with issues like \i, \g, . etc.
  2. String Formatting: Converted 276 instances of old-style percent-formatting to f-strings for better readability and Python 3
    style.
  3. Integer Division: Fixed 39 instances of division that should use integer division (// instead of /) to maintain expected behavior
    in Python 3.
  4. Collections Module Imports: Created a script that fixes imports for the collections module, particularly for ABC classes moved to
    collections.abc in Python 3.10+.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant