Migrate supportData to on-demand fetching with Pooch

.github/workflows/pull-request.yml

@@ -126,5 +126,7 @@ jobs:
         with:
           python-version: 3.13
           conan-args: --opNav True --allOptPkg --mujoco True --mujocoReplay True
+      - name: Pre-fetch supportData
+        run: bskLargeData
       - name: Build docs
         uses: ./.github/actions/docs

.pre-commit-config.yaml

@@ -10,3 +10,11 @@ repos:
       -   id: check-case-conflict
       -   id: check-merge-conflict
       -   id: debug-statements
+-   repo: local
+    hooks:
+      -   id: check-supportdata-registry
+          name: Ensure supportData registrySnippet.py is updated
+          entry: python Basilisk/utilities/supportDataTools/checkSync.py
+          language: python
+          files: ^supportData/
+          pass_filenames: false

MANIFEST.in

@@ -3,6 +3,7 @@ prune docs
 prune src/tests
 prune **/_UnitTest
 prune **/_Documentation
+prune supportData
 
 global-exclude *.pdf *.png *.jpg *.jpeg *.gif *.JPG *.svg *.psd
 global-exclude *.ipynb *.bib *.tex

docs/source/Install/pipInstall.rst

@@ -66,7 +66,7 @@ The main benefit of this approach will come in the future, when a set of pre-com
 allowing most users to easily ``pip install Basilisk`` without compilation, in the same way that packages like
 ``numpy``, ``scipy``, and ``pandas`` are available.
 
-To keep the wheel size smaller, the large BSK data files are not installed by default.  If the user
+To keep the wheel size smaller, the BSK data files are not installed by default.  If the user
 wants to use script that assumes they are included into the Basilisk python package, then go to the
 command line, change the current directory to be inside the environment where Basilisk was ``pip`` installed,
 and run the command::
@@ -76,8 +76,7 @@ and run the command::
 This command runs a python file stored in the ``src/utilities`` folder.
 The ``pip install`` process automatically
 creates this console command in the current python environment to call this python file.  The file
-directly downloads the missing large BSK data files and put them into the local Basilisk python
-package installation.
+directly downloads the missing BSK data files and put them into a local pooch cache.
 
 .. note::
 

docs/source/Support/Developer.rst

@@ -18,3 +18,4 @@ The following support files help with writing Basilisk modules.
    Developer/UnderstandingBasilisk
    Developer/migratingBskModuleToBsk2
    Developer/MigratingToPython3
+   Developer/addSupportData

docs/source/Support/Developer/addSupportData.rst

@@ -0,0 +1,144 @@
+.. _addSupportData:
+
+Adding Support Data Files
+=========================
+
+This page explains how to correctly add new files to the ``supportData``
+directory so they integrate with Basilisk's Pooch-based data management system
+and are versioned on GitHub.
+
+Basilisk **does not ship** supportData files inside the wheel.
+Instead, all files are registered and fetched on demand.
+
+When you add or update support data files, three major parts may need updating:
+
+1. ``supportData/`` directory (the files themselves)
+2. ``makeRegistry.py`` (to regenerate the MD5 registry)
+3. ``dataFetcher.py`` (to map the file into enums/category paths)
+
+
+----------------------------------------------------------------------
+Folder Layout
+----------------------------------------------------------------------
+
+Files must live inside:
+
+::
+
+    supportDataTools/<CategoryName>/<file>
+
+Where ``<CategoryName>`` must match one of the following:
+
+- ``AlbedoData``
+- ``AtmosphereData``
+- ``DentonGEO``
+- ``EphemerisData``
+- ``LocalGravData``
+- ``MagneticField``
+
+If you add a new sub-category, **you must also add it to**
+``dataFetcher.py`` (explained in Step 4).
+
+----------------------------------------------------------------------
+Step 1 — Add the File to supportData/
+----------------------------------------------------------------------
+
+Place your file under the appropriate category, e.g.:
+
+::
+
+    supportDataTools/AtmosphereData/NewMarsAtmosphere2025.csv
+
+Make sure the file is not covered by any ignore patterns
+(see ``makeRegistry.py``).
+
+----------------------------------------------------------------------
+Step 2 — Ensure makeRegistry.py will include the file
+----------------------------------------------------------------------
+
+The registry generator ignores certain patterns:
+
+.. code-block:: python
+
+    IGNORE_PATTERNS = (
+        "__pycache__",
+        ".pyc",
+        "__init__.py",
+        "*.bsp",
+    )
+
+If your file accidentally matches a pattern, remove or update the entry.
+
+----------------------------------------------------------------------
+Step 3 — Regenerate the registrySnippet.py file
+----------------------------------------------------------------------
+
+From the **project root**, run:
+
+::
+
+    python src/utilities/supportDataTools/makeRegistry.py > src/utilities/supportDataTools/registrySnippet.py
+
+This writes a dictionary of:
+
+- file to MD5 hash
+- symbols used by Pooch to verify downloads
+
+Commit the changed ``registrySnippet.py``.
+
+----------------------------------------------------------------------
+Step 4 — Update dataFetcher.py (Enums and Base Paths)
+----------------------------------------------------------------------
+
+In ``dataFetcher.py``, each category has:
+
+1. An Enum listing files
+2. A base path value
+3. A mapping in ``CATEGORY_BASE_PATHS``
+
+For example, adding ``NewMarsAtmosphere2025.csv`` requires:
+
+1. Add to the correct Enum:
+
+.. code-block:: python
+
+    class AtmosphereData(Enum):
+        NewMarsAtmosphere2025 = "NewMarsAtmosphere2025.csv"
+
+2. Ensure the base path exists:
+
+.. code-block:: python
+
+    ATMOSPHERE_DATA_BASE_PATH = "supportData/AtmosphereData/"
+
+3. Ensure the category name → base path map includes it:
+
+.. code-block:: python
+
+    CATEGORY_BASE_PATHS = {
+        "AtmosphereData": ATMOSPHERE_DATA_BASE_PATH,
+        ...
+    }
+
+
+----------------------------------------------------------------------
+External Data Sources
+----------------------------------------------------------------------
+
+Some kernel files are **not in the Git repo** and **should not be hashed**:
+
+.. code-block:: python
+
+    EXTERNAL_KERNEL_URLS = {
+        "supportData/EphemerisData/de430.bsp": "https://naif.jpl.nasa.gov/...",
+        ...
+    }
+
+These entries automatically override registry hashes:
+
+.. code-block:: python
+
+    for key in EXTERNAL_KERNEL_URLS:
+        REGISTRY[key] = None
+
+This prevents MD5 failures when NAIF updates files.

docs/source/Support/bskReleaseNotes.rst

@@ -26,6 +26,10 @@ Basilisk Release Notes
 
 Version |release|
 -----------------
+- Migrated supportData handling to a Pooch-based fetch system.
+- Removed supportData files from wheels and source distributions to reduce package size.
+- Added automatic MD5 registry generation via ``makeRegistry.py`` for versioned supportData.
+- Updated documentation to include workflow for adding new supportData files and regenerating the registry.
 - Removed automated version bumping on merges to develop. Version bumps will be handled manually moving forward.
 - Added SWIG as a python package dependency in ``requirements_dev.txt`` and updated installation instructions.
 - Updated CI to no longer automatically publish git tags and publish wheels to PyPI. This process will be handled manually.

examples/BskSim/models/BSK_Dynamics.py

@@ -142,8 +142,7 @@ def SetGravityBodies(self):
         self.moon = 2
 
         self.gravFactory.addBodiesTo(self.scObject)
-        self.gravFactory.createSpiceInterface(bskPath + '/supportData/EphemerisData/',
-                                              timeInitString,
+        self.gravFactory.createSpiceInterface(time=timeInitString,
                                               epochInMsg=True)
         self.epochMsg = self.gravFactory.epochMsg
 

examples/MultiSatBskSim/modelsMultiSat/BSK_EnvironmentEarth.py

@@ -21,12 +21,14 @@
 from Basilisk.simulation import ephemerisConverter, groundLocation, eclipse
 from Basilisk.topLevelModules import pyswice
 from Basilisk.utilities import macros as mc, simIncludeGravBody
+from Basilisk.utilities.supportDataTools.dataFetcher import get_path, DataFile
 
 bskPath = __path__[0]
 
 
 class BSKEnvironmentModel:
     """Defines the Earth Environment."""
+
     def __init__(self, SimBase, envRate):
         # Define empty class variables
         self.mu = None
@@ -40,7 +42,9 @@ def __init__(self, SimBase, envRate):
         processTasksTimeStep = mc.sec2nano(envRate)
 
         # Create task
-        SimBase.envProc.addTask(SimBase.CreateNewTask(self.envTaskName, processTasksTimeStep))
+        SimBase.envProc.addTask(
+            SimBase.CreateNewTask(self.envTaskName, processTasksTimeStep)
+        )
 
         # Instantiate Env modules as objects
         self.gravFactory = simIncludeGravBody.gravBodyFactory()
@@ -65,56 +69,70 @@ def SetGravityBodies(self):
         Specify what gravitational bodies to include in the simulation.
         """
         # Create gravity bodies
-        gravBodies = self.gravFactory.createBodies(['sun', 'earth', 'moon'])
-        gravBodies['earth'].isCentralBody = True
-        self.mu = self.gravFactory.gravBodies['earth'].mu
-        self.planetRadius = self.gravFactory.gravBodies['earth'].radEquator
+        gravBodies = self.gravFactory.createBodies(["sun", "earth", "moon"])
+        gravBodies["earth"].isCentralBody = True
+        self.mu = self.gravFactory.gravBodies["earth"].mu
+        self.planetRadius = self.gravFactory.gravBodies["earth"].radEquator
         self.sun = 0
         self.earth = 1
         self.moon = 2
 
         # Override information with SPICE
         timeInitString = "2021 MAY 04 07:47:48.965 (UTC)"
-        self.gravFactory.createSpiceInterface(bskPath + '/supportData/EphemerisData/',
-                                              timeInitString,
-                                              epochInMsg=True)
-        self.gravFactory.spiceObject.zeroBase = 'Earth'
+        self.gravFactory.createSpiceInterface(time=timeInitString, epochInMsg=True)
+        self.gravFactory.spiceObject.zeroBase = "Earth"
 
         # Add pyswice instances
-        pyswice.furnsh_c(self.gravFactory.spiceObject.SPICEDataPath + 'de430.bsp')  # solar system bodies
-        pyswice.furnsh_c(self.gravFactory.spiceObject.SPICEDataPath + 'naif0012.tls')  # leap second file
-        pyswice.furnsh_c(self.gravFactory.spiceObject.SPICEDataPath + 'de-403-masses.tpc')  # solar system masses
-        pyswice.furnsh_c(self.gravFactory.spiceObject.SPICEDataPath + 'pck00010.tpc')  # generic Planetary Constants
+        de430_path = get_path(DataFile.EphemerisData.de430)
+        naif0012_path = get_path(DataFile.EphemerisData.naif0012)
+        de403masses_path = get_path(DataFile.EphemerisData.de_403_masses)
+        pck00010_path = get_path(DataFile.EphemerisData.pck00010)
+        pyswice.furnsh_c(str(de430_path))  # solar system bodies
+        pyswice.furnsh_c(str(naif0012_path))  # leap second file
+        pyswice.furnsh_c(str(de403masses_path))  # solar system masses
+        pyswice.furnsh_c(str(pck00010_path))  # generic Planetary Constants Kernel
 
     def SetEpochObject(self):
         """
         Add the ephemeris object to use with the SPICE library.
         """
 
         # self.epochMsg = self.gravFactory.epochMsg
-        self.ephemObject.ModelTag = 'EphemData'
-        self.ephemObject.addSpiceInputMsg(self.gravFactory.spiceObject.planetStateOutMsgs[self.sun])
-        self.ephemObject.addSpiceInputMsg(self.gravFactory.spiceObject.planetStateOutMsgs[self.earth])
-        self.ephemObject.addSpiceInputMsg(self.gravFactory.spiceObject.planetStateOutMsgs[self.moon])
+        self.ephemObject.ModelTag = "EphemData"
+        self.ephemObject.addSpiceInputMsg(
+            self.gravFactory.spiceObject.planetStateOutMsgs[self.sun]
+        )
+        self.ephemObject.addSpiceInputMsg(
+            self.gravFactory.spiceObject.planetStateOutMsgs[self.earth]
+        )
+        self.ephemObject.addSpiceInputMsg(
+            self.gravFactory.spiceObject.planetStateOutMsgs[self.moon]
+        )
 
     def SetEclipseObject(self):
         """
         Specify what celestial object is causing an eclipse message.
         """
         self.eclipseObject.ModelTag = "eclipseObject"
-        self.eclipseObject.sunInMsg.subscribeTo(self.gravFactory.spiceObject.planetStateOutMsgs[self.sun])
+        self.eclipseObject.sunInMsg.subscribeTo(
+            self.gravFactory.spiceObject.planetStateOutMsgs[self.sun]
+        )
         # add all celestial objects in spiceObjects except for the sun (0th object)
         for item in range(1, len(self.gravFactory.spiceObject.planetStateOutMsgs)):
-            self.eclipseObject.addPlanetToModel(self.gravFactory.spiceObject.planetStateOutMsgs[item])
+            self.eclipseObject.addPlanetToModel(
+                self.gravFactory.spiceObject.planetStateOutMsgs[item]
+            )
 
     def SetGroundLocations(self):
         """
         Specify which ground locations are of interest.
         """
         self.groundStation.ModelTag = "BoulderGroundStation"
         self.groundStation.planetRadius = self.planetRadius
-        self.groundStation.specifyLocation(np.radians(40.009971), np.radians(-105.243895), 1624)
-        self.groundStation.minimumElevation = np.radians(10.)
+        self.groundStation.specifyLocation(
+            np.radians(40.009971), np.radians(-105.243895), 1624
+        )
+        self.groundStation.minimumElevation = np.radians(10.0)
         self.groundStation.maximumRange = 1e9
 
     # Global call to initialize every module