Annotating __new__ return type in abstract base class.

[andy-slac]

I'm trying to make type annotation work with the __new__ method in an abstract base class, the code is along these lines:

from __future__ import annotations

from abc import ABC, abstractmethod


class Abc(ABC):

    def __new__(cls, config: str) -> Abc:
        if cls is Abc:
            if config.startswith("x:"):
                cls = ImplX
            elif config.startswith("y:"):
                cls = ImplY
            else:
                raise ValueError(f"Unexpected config {config}")
        return super().__new__(cls)

    @abstractmethod
    def method(self) -> None:
        raise NotImplementedError()


class ImplX(Abc):

    def __init__(self, config: str):
        print("Making ImplX instance")

    def method(self) -> None:
        pass

class ImplY(Abc):

    def __init__(self, config: str):
        print("Making ImplY instance")

    def method(self) -> None:
        pass

config = "x:/a/b/c"
impl = Abc(config)

(I know it's a bit unconventional, but for historical reasons we have an existing non-abstract class that we want to make abstract but keep the way client code makes instances).

This looks OK to me and it actually works, but mypy complains about the last line:

error: Cannot instantiate abstract class "Abc" with abstract attribute "method"  [abstract]

I guess mypy assumes that __new__ tries to make an instance of Abc instead of a sub-class of Abc (I'm not sure that __new__ even allowed to return an instance of abstract class).

I tried to workaround it by annotating __new__ with actual types (this is a sort of lie because actual type can be almost anything in a real use case):

    def __new__(cls, config: str) -> ImplX | ImplY:

and that causes this mypy error:

error: "__new__" must return a class instance (got "ImplX | ImplY")  [misc]

I managed to suppress the error if I use just a single concrete type on return annotation:

    def __new__(cls, config: str) -> ImplX:

(mypy then complains about return super().__new__(cls), but I can mask it).

The last workaround is not ideal, of course, as mypy now thinks that Abc(...) always returns ImplX, which is not true.

I wonder if I'm doing something completely wrong and seriously abusing __new__, or if this is a limitation of mypy. Does anyone know of a better approach to solve this?

This is with Python 3.11 and mypy 1.5.1.

[andy-slac]

And I think python/mypy#14106 is a bug report for the identical issue for older mypy version.

[hauntsaninja]

This is slightly unusual and I'm not sure mypy is wrong about the "instantiating abstract class" message. I'd consider setting --disable-error-code abstract.

must return a class instance definitely seems like a mypy limitation, in general, I don't like mypy's current handling of __new__, see also python/mypy#16020

[andy-slac]

Thanks, but I cannot disable abstract code globally, as we have a lot of other code that we do want to check.

I guess what I really want is for annotation like:

class Abc(ABC):
    def __new__(cls, ...) -> Abc:

to mean that __new__ returns an instance of the Abc's non-abstract subclass. I think it's obvious that it cannot return an instance of abstract class here.

I think that python/mypy#14106 correctly suggests that __new__ with abstract return type should be treated just like a regular method that returns an instance.

[erictraut]

It sounds like you don't want Abc to act like an abstract class, so I'm not sure why you are using ABC as a base class. Have you considered removing ABC and modeling it as a protocol?

from typing import Protocol

class BaseProto(Protocol):
    def method(self) -> None:
        ...

class Abc(BaseProto):
    def __new__(cls: type[BaseProto], config: str) -> BaseProto:  # type: ignore
        if cls is Abc:
            if config.startswith("x:"):
                cls = ImplX
            elif config.startswith("y:"):
                cls = ImplY
            else:
                raise ValueError(f"Unexpected config {config}")
        return super().__new__(cls)

    def method(self) -> None:
        raise NotImplementedError()

class ImplX(BaseProto):
    def __init__(self, config: str):
        print("Making ImplX instance")

    def method(self) -> None:
        pass

class ImplY(BaseProto):
    def __init__(self, config: str):
        print("Making ImplY instance")

    def method(self) -> None:
        pass

config = "x:/a/b/c"
impl = Abc(config)

Note that mypy still requires you to use a # type: ignore for the __new__ return type. Pyright does not.

[andy-slac]

No, this is definitely not what I want. The example that I posted is a valid Python code which works and does exactly what we need. The only question for us is how to make mypy errors go away.

The actual code base is a lot more complicated, those classes have many abstract methods and I do not want to add additional intermediate classes/protocols just to make mypy happy.