Create Type Stubs for use with mypy

[mchlnix]

Mypy is throwing me an error, when trying to use Crypto.Cipher._mode_cbc.CbcMode in my own type stub files.

It would be great to have type stubs for this library, but I don't know how one would place them in a project, I just have them side by side with the normal .py files.

[mchlnix]

I'd love some feedback. Was this attempted before, or is it as "easy" as creating them and making a pull request to python/typeshed?

[mchlnix]

Works for me that way. I left out everything with an underscore, or which would give an NotImplementedError.

[Legrandin]

Just a quick note that I find this improvement (and static analysis in general) very valuable.

Right now I don't have enough time to provide you with some meaningful feedback (I would also want to play around with MyPy a big longer), but it seems to me you are heading toward the right direction with the PRs and I would be happy to merge in the end.

[mchlnix]

Can you give me a definition on "in the end"? Do you want to get more familiar with mypy or is there something I can/need to do?

[Legrandin]

Sorry for not coming back earlier on this.

The main question I have concerns the way to distribute stubs, also with respect to PEP561.
Right now, your patch does not package the type stubs, so one has to check out the git repo, and set MYPYPATH.

There are a few options:

1. Create a stub-only, separate package, like pycryptodome-stubs. Not ideal for maintenance.
2. Include stubs upstream into typeshed. I see this as not recommended, as it poorly scales.
   Also, there is already type information for the Crypto namespace, though generated from the old PyCrypto.
3. Embed type definition in the code as comments, and do not use stubs.
4. Ship stubs alongside source files - in the same directory under site-packages or dist-packages.
   This seemed the best option, but it doesn't appear to work (yet). mypy does not pick them up.

Any thoughts?

[mchlnix]

I didn't even think about the package side of things. The way I use them (after consulting with the mypy team) is using the MYPYPATH variable as you stated.

I'm not sure, what you mean that typeshed doesn't scaly well. As with the PyCrypto stubs already in the typeshed repository, that would be something to hash out. But as long as pycryptodome is compatible to Crypto shouldn't it be fine to just use the stubs for Crypto? That might even be wanted for legacy projects.

I have to say I started using Crypto, then switched to pycryptodome using the backwards compatibility and based on that I developed the stubs, so that might also be a problem, not sure.

I'm no expert on typing either, I basically read up on exactly what I need to know to make my code be able to be checked. But I think a good way would be to go on gitter and talk to the devs directly, about what is the best thing to do.

If all else fails, it can't hurt to put them into the repo and add something to the test suite, or just the Readme, mentioning that they're available.

[Legrandin]

OK, I think I managed to get option 4 above working (branch type_stubs), so I will go for that approach: stubs get automatically installed together with the library, there is no need to set MYPYPATH and stubs remain right next to the source files (good for maintainability).

But as long as pycryptodome is compatible to Crypto shouldn't it be fine to just use the stubs for Crypto?

pycryptodome is not entirely compatible with PyCrypto API: certain things (not many though) have been dropped on purpose and several new ones have been added. For that reason, the stubs in typeshed are not suited for supporting static analysis of pycryptodome-based libraries. However, the inline stubs (option 4) take priority over stubs in typeshed so everything will work fine - the two will not be in conflict.

[mchlnix]

Did you rewrite/rework them?

[mchlnix]

To the other point of you adding and dropping stuff from Crypto compatibility, I think the added stuff is not part of Crypto and shouldn't be available as type stubs there (that can mess up IDEs). They can be under pycryptodomex stubs with the rest, to keep them separate. And the stuff you dropped can't be used anyway, so no harm in them having types stubs that will never be used, right?

As long as you didn't change anything, which would need the old Crypto stubs to be changed.

That way I feel like both camps are getting something out of it.

But it's 11pm and I'm melting chocolate on the side, so maybe I'm missing something.

[Legrandin]

Did you rewrite/rework them?

Just moved them over (see git log).

I think the added stuff is not part of Crypto and shouldn't be available as type stubs there (that can mess up IDEs). They can be under pycryptodomex stubs with the rest, to keep them separate.
And the stuff you dropped can't be used anyway, so no harm in them having types stubs that will never be used, right?
As long as you didn't change anything, which would need the old Crypto stubs to be changed.
That way I feel like both camps are getting something out of it.

Anybody basing their app on the old PyCrypto library (despite its age and vulnerabilities) fshould rely on the stubs from the typeshed package (which is actually something that mypy does already automatically, no action required), as opposed to explicitly using stubs taken from this package.

Anybody using PyCryptodome (in either the Crypto or Cryptodome namespace) should better use the stubs you started creating (also something that does not require any extra action from the user, since mypy will automatically pick them up thanks to PEP561). The implicit and I think widespread expectation is that stubs match the specific source code they ship with: it's weird to include definition for code that has been removed long before or not include code that has been added and it is active in the library.

But it's 11pm and I'm melting chocolate on the side, so maybe I'm missing something.

Well, I'm missing that definitely!

[mchlnix]

I think I still don't get the problem you are seeing.

Suppose there are separate pycryptodome stubs, as if there never was a Crypto library and that gets added to typeshed. Users using your library from scratch will never see the old Crypto stubs.
Users using your library as a stand in would include it as Crypto and use the Crypto stubs (as happened to me). Except for the things you remove, they can use everything Crypto describes. The things you removed can't be used anyway, so there is no harm in them still being in the type stubs for Crypto and if they use a mishmash (like I was) they are doing it wrong and cannot expect there to be hybrid type stubs.

There may be the use case of Crypto not expecting bytearrays (not sure, but possible) but then again, if byte arrays were not supported by Crypto, using them with pycryptodome in Crypto mode should not be supported by the stubs in my opinion.

Am I right in thinking, that everyone not switching from Crypto would import your library as pycryptodomex? Because then I really don't see any conflict.

Also, since you just moved them, I hope you can find the time to look them over. I did my best, but only typed the "public" methods and variables and left out py3compat for example. And since I learned the typing system while creating the stubs, there might be some early mistakes/inaccuracies.

EDIT:
I guess what I'm proposing really only works if both versions of the stubs are in typeshed, since the way you are doing it now will probably supersede the difference in importing it as Crypto or pycryptodomex. If that is what you are refering to, then I can see what you mean now.