feature: API checks: Check for types compatibility

[pawamoy]

Following the discussion in #75, we released a first version of the API breakage detection feature, which does not check if types (parameters types, return types) are compatible between the old and new code.
This issue is here to discuss about adding support for checking types compatibility.

It's already possible at runtime with beartype, so maybe we could use in the inspector. But for the visitor we need something static.

[pawamoy]

Moving this here from the README:

• List of things to consider for breaking changes
  • Changed type of parameter
  • Changed type of public module attribute
  • Changed return type of a public function/method
  • All of the previous even when parent is private (could be publicly imported or assigned somewhere),
    and later be smarter: public assign/import makes private things public!
  • Inheritance: removed, added or changed base that changes MRO

[pawamoy]

I wonder: could we somehow recurse into objects to check that they implement the same "protocols"? Example:

class Foo:
    def method1(self): ...
    def method2(self, param1: int, param2: int): ...

class Bar:
    def method2(self, param1: int, param2: int): ...

class Baz:
    def method1(self): ...
    def method2(self, param1: int): ...

class Qux:
    def method1(self): ...
    def method2(self, param1: str, param2: int): ...

class Quux:
    def method1(self): ...
    def method2(self, param1: int, param2: int): ...
    def method3(self): ...

def f_old(param: Foo): ...

def f_new(param: Bar): ...  # breaks: removed method1
def f_new(param: Baz): ...  # breaks: removed param2 on method2
def f_new(param: Qux): ...  # breaks: changed type of param1 on method2 (int -> str is a "known" breakage)
def f_new(param: Quux): ...  # doesn't break: added method3

I imagine a type_compatible(self, other) method on objects that uses next(find_breaking_changes(self, other)) and returns False if a breakage is yielded, True if StopIteration is raised.

[pawamoy]

Or maybe there's a way to hook into mypy...

[pawamoy]

We would need to match protocols in all expression parts.

In dict[MyStr, This] vs. MyDict[str, That]:

• dict and MyDict must match
• MyStr and str must match
• This and That must match

To compare objects, we'd have to dynamically load external objects (but only if they weren't already used). Examples:

• param: list[otherpkg.Thing] vs. param: Sequence[otherpkg.Thing]: here we see that otherpkg.Thing is the same object (canonical path) before and after, so we don't need to load it
• param: otherpkg.Thing vs. param: set[str]: here we'd need to load otherpkg.Thing
• param: set[str] vs. param: otherpkg.Thing: here we'd need to load otherpkg.Thing again

Or we could expect users to pre-load these modules and soft-fail otherwise.

By the way, how to compare set[str] to Thing? Should Thing inherit from a generic with str type? Or should all the equivalent parameters be typed str, and if they're typed T (type alias? how to check that?) then only check generics?

[bswck]

Perhaps we could use pycroscope? I've asked Jelle if he thinks it's possible.
pycroscope exposes some APIs and is a semi-static type checker (it reasons about runtime objects), so I think it's potentially just a matter of making griffe and pycroscope talk to each other.
As far as I remember, pycroscope has some limitations in type checker-specific things like type narrowing on control flow, but for our base use case it sounds terrifically good.

[bswck]

FYI, folks from typeshed kinda solved this with mypy_primer using heuristics. It generates nice comments like this:

• Make __exit__ return types of generator context managers specifiable python/typeshed#14439 (comment) (breaking)
• Fix regression in DiDegreeView python/typeshed#14732 (comment) (non-breaking)

It's a nice approach, just really slow and won't detect some things that would break outside of packages checked by primer (and primer does check loooots of packages).

[bswck]

By the way, how to compare set[str] to Thing? Should Thing inherit from a generic with str type? Or should all the equivalent parameters be typed str, and if they're typed T (type alias? how to check that?) then only check generics?

That's a complicated topic. We'd need to implement consistency and materialization to decide about assignability.

Note that we'd also have to check variance to know what should be assignable to what.

For example, foo() in all versions (before and after change) passes local type checking, but only the first change is compatible with 3rd party dependers:

-def foo(x: float) -> float:
+def foo(x: float) -> bool:
     return bool(x)

this one isn't:

-def foo(x: float) -> float:
+def foo(x: bool) -> float:
     return bool(x)

because callables are contravariant on the argument type and covariant on the return type, as nicely explained in https://peps.python.org/pep-0483/#covariance-and-contravariance.

[pawamoy]

Maybe I'm missing something but:

it imports the modules it type checks

...sounds like a no-go to me for static analysis in Griffe 🤔

[bswck]

Hm, I think I got confused. It's kind of unrealistic to do it with pycroscope when I think about it. pycroscope would need to import both packages in the same process, as Jelle noted.