Commits on Sep 13, 2024

1. TarOutput: Close collected files after reading them to free buffers …

   With tar output, TarOutput.addRealFile() did not close file objects.
   So in tar mode, the file objects of all captures regular files were
   not closed.
   
   As the fixed lines use `TarFile.addfile(tarinfo, open("fn"))`, the
   opened file object could have never been passed anywhere else.
   Thus, it is safe to fix this pattern using a context manager.
   
   Summary:
   
   - The changes to xen-bugtool are 4 safe lines to use `with open`
   - All other changes are improvements for the unit tests to cover it.
   
   Description:
   
   This was found by PYTHONDEVMODE=yes to trace unclosed resources.
   I set this up in .pre-commit-config.yaml, so while developing
   tests, I saw the warning while testing with pre-commit run -v.
   
   Fixes in this PR:
   
   - Use context managers to fix the locations which are easy to fix.
   - Extend the unit tests (those in master now) to cover both locations.
   - While updating those tests, improve them to check the file list of
     the ZIP and tar archives generated in this particular test case.
   - Configure GitHub CI to create source annotations for PYTHONWARNINGS
     which are issued by such cases and are shown in code review of PRs.
   
   Notes:
   
   In the course of a year, my development machine accumulated 24000
   files in /var/xapi/blobs (spread evenly over all subdirectories),
   so Python apparently does some garbage collection, but it's certainly
   not good to never close so many open file-like objects in Python.
   
   Closing them in time has the potential to reduce the memory usage
   of the xen-bugtool / xenserver-status-report process.
   
   There are more locations where Python 3.10 reports unclosed file
   objects, but it seems we may have to use enable tracemalloc as
   recommended by the issued PYTHONWARNINGS to find their source.
   
   Signed-off-by: Bernhard Kaindl <bernhard.kaindl@cloud.com>

   

   bernhardkaindl committed Sep 13, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 921927a
   • Browse repository at this point

   Copy the full SHA

   921927a View commit details

   Browse the repository at this point in the history