Why is the variance of Generic Protocols not decided by the inputs?

[randolf-scholz]

There is/was a lot of debate about whether Mapping should be covariant in the Key/Value Type.

My question is: why don't the input types themselves decide this?

I.e. Mapping[key_co, value_co] would be covariant in both, Mapping[key_contra, value_contra] would be contra-variant in both, etc. Currently, this gives errors of the form error: Variance of TypeVar "key_co" incompatible with variance in parent type in mypy.

So, e.g. mypy won't let me make a class CoContraMapping(Mapping[key_co, value_contra]).

[carljm]

I rather think we should be moving in the opposite direction, where variance is automatically determined by the use of a type variable in context, rather than explicitly specified as part of the typevar itself. Whether a typevar can soundly be variant or contravariant is in general determined by where it appears in method signatures of the type: if it appears as an argument type, it cannot safely be variant; if it appears as a return type, it cannot safely be contravariant; if it appears as both, it must be invariant.

[erictraut]

@carljm, draft PEP 695 proposes to determine variance automatically based on usage. A proof of concept for PEP 695 is already implemented in pyright if you want to try it out.

@randolf-scholz, regardless of whether type parameter variance is specified explicitly (like it is in the Python type system today) or determined automatically from usage (as proposed in PEP 695), the variance for type parameters is fixed for a given class, and subclasses must honor that variance. In the case of Mapping (which isn't defined as a Protocol, incidentally), KT (corresponding to the key type) is invariant and VT_co (corresponding to the value type) is covariant. If you are subclassing from Mapping as in your example above, the subclass must honor this variance. It's OK for the subclass to impose tighter restrictions on the variance (e.g. use an invariant type parameter for the value key rather than a covariant type parameter), but it is not OK for it to use a less restrictive or incompatible variance. In your case, you're attempting to use a covariant key and a contravariant value. Both of these violate the variance of the type parameters in the Mapping class, so mypy is correct to flag this as an error.

[randolf-scholz]

I am starting to understand now. One example that has always bothered me is the following:

from typing import TypeVar, Mapping, Hashable

value_co = TypeVar("value_co", covariant=True)


def unpack(x: Mapping[Hashable, value_co]) -> list[value_co]:
    return list(x.values())


d: dict[str, int] = {"a": 1}
values: list[int] = unpack(d)  # ✘ expected Mapping[Hashable, int]

My thoughts now, after carljm's comment and re-reading the wikipedia article on covariance, it seems the following is true:

1. Function-Types should be contravariant in the inputs and covariant in the outputs.
2. When we pass an object, we really could consider it as passing the collection of bound functions.
3. Since for Mapping.__getitem__, keys appear as input and for Mapping.__iter__ , keys appear as output, Mapping needs to be invariant in keys.
   • However, values only ever appear as output, so Mapping can be covariant in the values
   • The MutableMapping class has additionally MutableMapping.__setitem__ that takes values as inputs, so it needs to be invariant in the values as well to be type-safe.
4. In many cases, functions that take Mapping inputs never use Mapping.__getitem__, but rather Mapping.keys, Mapping.values, Mapping.items, for which the keys appears as output.
   • So in these cases, Mapping may as well be covariant in the keys?
   • For the dict class it should be noted that d.values never calls d.__getitem__
5. One way to resolve the issue is to create an additional type variable HashableT

value_co = TypeVar("value_co", covariant=True)
HashableT = TypeVar("HashableT")
def unpack(x: Mapping[HashableT, value_co]) -> list[value_co]:
    return list(x.values())
d: dict[str, int] = {"a": 1}
values: list[int] = unpack(d) 

where now, we basically tell the typing system that unpack works for any HashableType, whereas before due to invariance it only with exactly Hashable. So in a sense, using the TypeVar is somewhat equivalent to adding an @overload for any possible Hashable type.

---

Is that roughly it?