Intersphinx does not create references to external project when using import as alias

[photoniker]

Describe the bug

I'm creating a code docu using the docstring included in the code. In the conf.py file the intersphinx extension is included with mapping dicts to numpy, scipy, matplotlib, ...

E. g. When I import numpy as np or from typing import List, the external references do not exists in the build html docu as expected. It only works for standard python types.

I could not find anything in the Sphinx docu about this. Is this a bug or how must Sphinx be configured here?

How to Reproduce

$ git clone https://github.com/.../some_project
$ cd some_project
$ pip install -r requirements.txt
$ cd docs
$ make html SPHINXOPTS="-D language=de"
$ # open _build/html/index and see bla bla

Expected behavior

When I use absolute import like import numpy, import typing, the external docu is references in the html docu.

I would expected that the external docu references are created also for the import aliases.

Your project

--

Screenshots

No response

OS

Win

Python version

3.8

Sphinx version

1.44

Sphinx extensions

"sphinx.ext.autodoc",     "sphinx.ext.viewcode",     "sphinx.ext.todo",     "sphinx.addnodes",     "sphinx.ext.napoleon",     "sphinx_autodoc_typehints",     "sphinx.ext.intersphinx",     "sphinx.ext.mathjax",     "sphinx.ext.viewcode",

Extra tools

No response

Additional context

No response

[ProGamerGov]

Was a solution ever found for this issue?

[photoniker]

I didn't get any answer to this issue. So still open?!

[ProGamerGov]

We might have to implement it ourselves via a PR.

This looks like the code for the extension: https://github.com/sphinx-doc/sphinx/blob/5.x/sphinx/ext/intersphinx.py

[mhostetter]

I had this problem a while ago, which was why I subscribed to this thread. I found using type hints solves this problem. I don't know if it's worth it to you to switch, but... Also with proper type hints you can use the autodoc-process-signature callback to further modify the presentation of the type hint signatures.

Types in docstring

None of these are "clickable" except float.

def func_2(x, y):
    """
    This is the docstring of func_2.

    Parameters
    ----------
    x: np.ndarray
        The first argument.
    y: np.ndarray
        The second argument.

    Returns
    -------
    List[float]
        The return value.
    """
    pass

Types as type hints

All of these are "clickable".

def func(x: np.ndarray, y: np.ndarray) -> List[float]:
    """
    This is the docstring of func.

    Parameters
    ----------
    x
        The first argument.
    y
        The second argument.

    Returns
    -------
    The return value.
    """
    pass

Conf.py

intersphinx_mapping = {
    "python": ("https://docs.python.org/3", None),
    "numpy": ("https://numpy.org/doc/stable/", None),
}

autodoc_typehints = "both"
# autodoc_typehints_format = "short"  # This is handy too

You can optionally add the autodoc-process-signature callback if you'd like.

def autodoc_process_signature(app, what, name, obj, options, signature, return_annotation):
    # Do something
    return signature, return_annotation

def setup(app):
    app.connect("autodoc-process-signature", autodoc_process_signature)

[photoniker]

OK I think I see the difference now. When you use a docstring generator as you have it in some python IDEs, it typically generates the docstring with the type included as it is defined in the docstring conventions: Numpy Docstring Format.

def func(self, val: np.ndarray) -> List[float]:
    """docstring of func

    Args:
        val (np.ndarray): Input argument

    Returns:
        List[float]: return value
    """
    return

--> When you delete all typehints in the docstring, it works.

def func(self, val: np.ndarray) -> List[float]:
    """docstring of func

    Args:
        val: Input argument

    Returns:
        return value
    """
    return

Does that mean the link to the numpy doc is overwritten?

[mhostetter]

@photoniker how does this function render?

def func(self, val: np.ndarray) -> List[float]:
    """docstring of func

    Args:
        val (int): Input argument

    Returns:
        str: return value
    """
    return

[photoniker]

Seems to work with python build-in types

[mhostetter]

That's a bummer. It appears that the docstring types supersede the function's actual annotations. I would've hoped for the opposite.

There's clearly an issue (as you reported) with how Napoleon processes types from the docstring. The workaround I've used is to simply remove types from the docstring and use type hints.

[photoniker]

Removing the type hints in all my docstrings is quite a bit of work ;-)

[mhostetter]

Here's another workaround... You could attach to autodoc-process-docstring and find "np." and replace it with "numpy.". That would then add a link to numpy.ndarray in your docs.

[ProGamerGov]

Here's another workaround... You could attach to autodoc-process-docstring and find "np." and replace it with "numpy.". That would then add a link to numpy.ndarray in your docs.

Do you have an example for that? I'd like to ensure that nn.Module (torch.nn.Module) and F.interpolate (torch.nn.functional.interpolate) redirect to the appropriate PyTorch docs, without having to type out the full path.

[mhostetter]

Below is an example with OPs function. The same trick can be applied to nn.Module to torch.nn.Module.

Here's the full example project. foo2-example.zip

Before

Notice np.ndarray isn't linked.

def func(self, val: np.ndarray) -> List[float]:
    """docstring of func

    Args:
        val (np.ndarray): Input argument

    Returns:
        List[float]: return value
    """
    pass

After

Notice np.ndarray was converted to numpy.ndarray and is now linked.

#conf.py
def autodoc_process_docstring(app, what, name, obj, options, lines):
    for i in range(len(lines)):
        lines[i] = lines[i].replace("np.", "numpy.")
        lines[i] = lines[i].replace("List[", "~typing.List[")

def setup(app):
    app.connect("autodoc-process-docstring", autodoc_process_docstring)

Shorten links

If you add autodoc_typehints = "short" to conf.py and do things like lines[i] = lines[i].replace("np.", "~numpy.") the hyperlinks will be shortened like below.

[ProGamerGov]

@mhostetter Thank you!

Is there an easy way to do something similar in the doc string and parameter descriptions as well?

Like making this:

"""
:func:`F.interpolate`
"""

Equivalent to this:

"""
:func:`torch.nn.functional.interpolate`
"""

Or does your solution work on the entire doc string? If it works on the entire doc string, how can I only target the parameters?

[mhostetter]

Yes, the entire docstring.

def func(self, val: np.ndarray) -> List[float]:
    """docstring of func

    Here is a detailed docstring. You know :func:`F.interpolate` is a cool function.

    Args:
        val (np.ndarray): Input argument

    Returns:
        List[float]: return value
    """
    pass

# conf.py
intersphinx_mapping = {
    "python": ("https://docs.python.org/3", None),
    "numpy": ("https://numpy.org/doc/stable/", None),
    "torch": ("https://pytorch.org/docs/master/", None),
}

def autodoc_process_docstring(app, what, name, obj, options, lines):
    print(name)
    print("before:", lines)
    for i in range(len(lines)):
        lines[i] = lines[i].replace("np.", "numpy.")
        # lines[i] = lines[i].replace("np.", "~numpy.")  # For shorter links
        lines[i] = lines[i].replace("F.", "torch.nn.functional.")
        lines[i] = lines[i].replace("List[", "~typing.List[")
    print("after:", lines)

def setup(app):
    app.connect("autodoc-process-docstring", autodoc_process_docstring)

If you print the lines, you see exactly what is getting processed. If you only want to modify the types, you need to look for :type or :rtype at the beginning of a line.

reading sources... [ 50%] foo.func
foo.func
before: ['docstring of func', '', 'Here is a detailed docstring. You know :func:`F.interpolate` is a cool function.', '', ':param val: Input argument', ':type val: np.ndarray', '', ':returns: return value', ':rtype: List[float]', '']
after: ['docstring of func', '', 'Here is a detailed docstring. You know :func:`torch.nn.functional.interpolate` is a cool function.', '', ':param val: Input argument', ':type val: numpy.ndarray', '', ':returns: return value', ':rtype: ~typing.List[float]', '']

And it renders as below, with everything hyperlinked.

[ProGamerGov]

@mhostetter Ah, okay. Thanks for the help!

from typing import Callable
import torch
import torch.nn as nn
def test_func(x: torch.Tensor, test_module: nn.Module, test_fn: Callable) -> torch.Tensor:
    """
    Description of function.

    Args:

        x (torch.Tensor): The input tensor.
        test_module (nn.Module): The nn.Module  instance to run x through.
        test_fn (Callable): A callable class or function to pass x through.

    Returns:
        x (torch.Tensor): The output tensor.
    """
    return test_module(test_fn(x))

My code uses doc strings like the one above, so I think that I could do something like this then (including increasing the specificity by targeting values inside / outside brackets, potentially with regex):

def autodoc_process_docstring(app, what, name, obj, options, lines):
    parameters_docs = False:
    for i in range(len(lines)):
        if "Args:" in lines[i]:
            parameters_docs = True
        if not parameters_docs:
            continue
        lines[i] = lines[i].replace("nn.Module", "torch.nn.Module")
        lines[i] = lines[i].replace("Callable[", "~typing.Callable[")


def setup(app):
    app.connect("autodoc-process-docstring", autodoc_process_docstring)

[mhostetter]

You can't filter on "Args:". Sphinx converts the Google-style docstrings Args: and Returns: sections into rST syntax, which makes it even easier to filter. See my example in my previous post. You just need to look for :type or :rtype at the beginning of a line.

reading sources... [ 50%] foo.func
foo.func
before: ['docstring of func', '', 'Here is a detailed docstring. You know :func:`F.interpolate` is a cool function.', '', ':param val: Input argument', ':type val: np.ndarray', '', ':returns: return value', ':rtype: List[float]', '']
after: ['docstring of func', '', 'Here is a detailed docstring. You know :func:`torch.nn.functional.interpolate` is a cool function.', '', ':param val: Input argument', ':type val: numpy.ndarray', '', ':returns: return value', ':rtype: ~typing.List[float]', '']

You instead would want something like this:

    for i in range(len(lines)):
        if not (lines[i].startswith(":type") or lines[i].startswith(":rtype")):
            continue
        # Do stuff

[ProGamerGov]

@mhostetter You don't need to use autodoc_typehints = "short", and you can change the link to say whatever you want.

For example:

lines[i] = lines[i].replace("Callable", ":obj:`Callable <typing.Callable>`")

lines[i] = lines[i].replace("F.interpolate", ":obj:`F.interpolate <torch.nn.functional.interpolate>`")

[ProGamerGov]

I've discovered a new issue. When manually adding :obj: or :class: to the docstring types, they are no longer formatted as italics. Yet somehow Sphinx is able to implicitly hyperlink some of the types and italicize them when they are not manually specified with :obj: or :class:.

Is there anyway to fix this issue?

Edit:

# conf.py
extensions = [
    "sphinx.ext.autodoc",
    "sphinx.ext.napoleon",
    "sphinx.ext.intersphinx",
]

intersphinx_mapping = {
    "python": ("https://docs.python.org/3", None),
    "pytorch": ("https://pytorch.org/docs/stable", None),
}

def my_func(show_progress: bool = True) -> bool:
    """
    Args:

        show_progress (bool, optional): Displays the progress of computation.
    """
    return show_progress

This is an example of HTML code generated by Sphinx without explicitly defining the types:

<li><p><strong>show_progress</strong> (<a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.10)"><em>bool</em></a><em>, </em><em>optional</em>) – Displays the progress of computation.

And this is the same HTML, except for the bool variable being explicitly wrapped in :class:bool``:

<li><p><strong>show_progress</strong> (<a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.10)"><code class="xref py py-class docutils literal notranslate"><span class="pre">bool</span></code></a>, optional) – Displays the progress of computation.

The <em> and </em> tags are missing in the second example for some reason.

[flying-sheep]

Would have been nice if my PR had been merged two years ago, then we wouldn’t need workarounds: #9039

[HealthyPear]

Hello,

Not sure if my problem has the same root as yours, so before opening a duplicate issue I wanted to check.

In my project, I am using the traitlets library and basically aliasing their types in my package.

In their code, they do import typing as t and they call t.Any, t.Dict, etc...

then when I compile my docs I see warnings as

WARNING: py:class reference target not found: t.Any

Is this the same problem? Sphinx not able to "intersphinx" aliased modules?