Relax Pydantic Version Requirements

[calebstewart]

Pydantic v2 has been around for nearly 2 years, and as such many things use the new version. tcex and the associated repositories related to it all have a strict <2.0.0 requirement on Pydantic, which is problematic when trying to utilize these libraries in a larger application or ecosystem which uses a newer version of Pydantic.

This PR and a collection of others I am in the process of making together aim to relax the version requirements for Pydantic by utilizing the pydantic.v1 namespace introduced in v1.10.17. This pyandic.v1 namespace allows users to utilize Pydantic as it was in v1 even when v2 is installed. It is explicitly listed as an option in the official Migration Guide.

I plan to place this same PR description across all repositories which I found that needed updating, so here's some overarching information about this process as I have been able to understand it:

1. There are three "primary" projects which define Python packages: tcex, tcex-cli, and tcex-app-testing.
2. The primary projects need their pyproject.toml updated to relax the requirements to >=1.10.17,<3.0.0.
3. Secondary projects are used as submodules within the primary projects. The submodule projects that have references to pydantic are (as far as I could find): tcex-app-config, tcex-app-playbook, tcex-util.
4. All projects (primary and secondary) need all references to importing pydantic updated to reference pydantic.v1.

This is all relatively straightforward. In the forks I have made, I've run the following straightforward one-liner to replace all from pydantic... lines appropriately:

find . -name '*.py' | while read filepath; do
  sed 's/^from pydantic/from pydantic.v1/g' -i "$filepath"
done

For repositories with submodules (the primary repositories mentioned above), I filtered out the submodule paths before making this replacement. After the secondary repositories are merged, the primary repository submodules will also need to be updated to account for these changes in the primary repositories as well.

I will be the first to admit that I am not the best person to test all of this as I'm not intimately familiar with tcex or all the in's and out's of your application(s). But I am familiar with Pydantic, and this version conflict is a bump in the road for me, so I'm happy to work through troubleshooting. I'm happy to discuss options and work with the team, but given that this was just a bunch of find-and-replace followed by careful coordination of submodules, I figured I'd at least get the ball rolling.

Once all six of the PRs are open, I'll go back through and make a comment on each to reference the others so they can be properly tracked/linked.

[calebstewart]

Related Open PRs:
ThreatConnect-Inc/tcex-app-config#15
ThreatConnect-Inc/tcex-app-playbook#9
ThreatConnect-Inc/tcex-app-testing#15
ThreatConnect-Inc/tcex-cli#19
ThreatConnect-Inc/tcex-util#11

[bsummers-tc]

Thanks for this well written PR Caleb. We have discussed this very update recently and planned to spend some time on it in the near future. Currently we are migrating from pylint, pydocstyle, and black to use ruff (https://astral.sh/ruff). We are also switching to uv (https://github.com/astral-sh/uv) for package management.

Please be patient as we work through other priorities before we get to this PR.

[calebstewart]

I understand conflicting priorities. Thanks for getting back to me. For the record, I was just doing some testing to validate these changes a bit as well. I replaced the relevant submodules with symlinks locally to my own forks like this:

rm -rf tcex/app/config tcex/app/playbook tcex/util
ln -s $(pwd)/../tcex-util ./tcex/util
ln -s $(pwd)/../tcex-app-config ./tcex/app/config
ln -s $(pwd)/../tcex-app-playbook ./tcex/app/playbook

This obviously breaks my git repository, but allows me to test with my forks without having to actually change the git submodules. Next, I setup a virtual environment and installed tcex:

python3 -m venv .venv
source .venv/bin/activate
pip install -e '.[dev,test]'

This installs the latest version of pydantic due to the loosened version constraints. When I ran this, I got version 2.10.6:

$ pip show pydantic
Name: pydantic
Version: 2.10.6
Summary: Data validation using Python type hints
Home-page: https://github.com/pydantic/pydantic
Author:
Author-email: Samuel Colvin <s@muelcolvin.com>, Eric Jolibois <em.jolibois@gmail.com>, Hasan Ramezani <hasan.r67@gmail.com>, Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>, Terrence Dorsey <terry@pydantic.dev>, David Montague <david@pydantic.dev>, Serge Matveenko <lig@countzero.co>, Marcelo Trylesinski <marcelotryle@gmail.com>, Sydney Runkle <sydneymarierunkle@gmail.com>, David Hewitt <mail@davidhewitt.io>, Alex Hall <alex.mojaki@gmail.com>, Victorien Plot <contact@vctrn.dev>
License:
Location: /home/caleb/git/tcex/.venv/lib/python3.12/site-packages
Requires: annotated-types, pydantic-core, typing-extensions
Required-by: tcex

I then executed the following test script:

#!/usr/bin/env python3
import os
from pathlib import Path
import json

from tcex import TcEx

tcex = TcEx(
    config={
        "tc_api_access_id": os.environ["TC_API_ACCESS_ID"],
        "tc_api_secret_key": os.environ["TC_API_SECRET_KEY"],
        "tc_api_path": os.environ["TC_API_URL"],
        "tc_log_path": str(Path.cwd()),
    },
)

group_id = 12345 # but with a real ID
group = tcex.api.tc.v3.group(id=group_id).get()
print(json.dumps(group.json(), indent=2))

This script worked, and then I installed Pydantic 1.10.21 to make sure it still worked with Pydantic v1:

$ pip show pydantic
Name: pydantic
Version: 1.10.21
Summary: Data validation and settings management using python type hints
Home-page: https://github.com/pydantic/pydantic
Author:
Author-email: Samuel Colvin <s@muelcolvin.com>
License: MIT
Location: /home/caleb/git/tcex/.venv/lib/python3.12/site-packages
Requires: typing-extensions
Required-by: tcex

Once again, the script worked. This is obviously not an exhaustive test, but it does at least show it's not completely broken.

I would have run the pytest tests, but frankly I don't know what they do and I am wary of running them against our account without fully understanding the potential ramifications.