Use of CFFI next to ctypes

[juliomateoslangerak]

I'm working on the implementation of Meadowlark's SLM. I'm bringing what I did on Mick's original code to adapt the newer blink dlls. At the time, I pretty much had to adapt everything as the newer blink SDK was pretty different. I did all of that using CFFI and not ctypes. Why I do this is vague in my memory but I remember some things, including the own examples of Meadowlark, not working using ctypes.

In my opinion, CFFI is more convenient than ctypes as soon as you go beyond the classic "hello C world" example. In particular is going to extract of information from the header files, presumably overcoming poor documentation issues.

I'd suggest adding CFFI as a dependency to microscope

[carandraug]

How does this work? Do you need to include the original C headers to get it compiled at install time? Or at runtime? Or do we need to compile them ourselves and commit them to the repo? (if we need to compile them ourselves, how does it work between different CFFI versions)?

(to be honest I'm a bit wary. This feels too similar to swig which we purposely avoid because of past issues)

[iandobbie]

I don't know how it might work but I would assume you need a stub that talks to the underlying C functions which have a copyright that doesn't allow us to share names, calling parameters etc. Then we implement our own calling conventions on top of the underlying dll calls and have our own names, parameters etc so don't tread on their copyright.

Then anyone who wants to use it need to sign an MTA with Nikon to get the dll and download our blob that uses our interface which we then publish with no Nikon copyright. It's a horrid hack, likely to be very brittle and masses of work to hide the copyrighted API, for a piece of equipment you already paid for. Fundamentally, I think it is totally broken approach and pretty typical of proprietary hardware manufactures approach to try and get customer lock-in.

I don't think the market segment that this represents has any prospect of lock-in. It is a tiny fraction of their total market and a lot of builders utilize older, repurposed stands so they just use what they have around rather than buying stands for the systems in question.

[carandraug]

@iandobbie I think you're confusing this issue with #305 (the one about controlling a Nikon Ti). This is about using CFFI instead of ctypes (initially for Meadowlark's SLM but I guess the goal is to adopt CFFI instead of ctypes moving forward).

[juliomateoslangerak]

The idea is definitely not to replace ctypes with CFFI, but let them coexist. I think CFFI is a better and more modern tool than swig and it's build by the team developing Pypy.

As I understand, cause many of these things go beyond what my brain can reach, CFFI may provide some advantages, in some scenarios vs ctypes. Essentially CFFI can be used in 2 ways:

• ABI mode: More similar to the way ctypes works and the way I have implemented the SLM. In this mode, there is no compilation and every call to C goes through libffi (like ctypes). The advantage I see compared to ctypes is that there is much less boilerplate and you may get the definitions from the header file. This may make a difference with complex libraries.
• API mode: CFFI will indeed compile a C-extension for python and, with all the complexity it might bring in terms of deployment and support, this could make the code much faster. The compilation is at installation time. This improvement in speed only takes place for every call to C so, in our case it would not make sense in most applications.

In my case now I would only need for CFFI to be in the list of dependencies. I didn't test the API mode and I wonder if, in scenarios where we are calling a C library often, like is the case of the SLM to load a new image, that would make a difference in real case scenarios. What about cameras?

[carandraug]

The idea is definitely not to replace ctypes with CFFI, but let them coexist. I think CFFI is a better and more modern tool than swig and it's build by the team developing Pypy.

But then parts of Microscope uses ctypes and other parts use CFFI. Using two different libraries for the same job means that we need to know about both when developing Microscope which makes our maintenance work more complex.

[...] you may get the definitions from the header file.

But then we need to add the header files to our code? I'm not sure we can do it for most libraries. Even on the few cases where vendors distribute the c libraries for free, the header files tend to be on a separate SDK under some sort of agreement which prevents us from sharing them.

[carandraug]

But then parts of Microscope uses ctypes and other parts use CFFI. Using two different libraries for the same job means that we need to know about both when developing Microscope which makes our maintenance work more complex.

To expand on this, I'm not against adopting CFFI. I'm only against keeping both as the goal. If we decide that we want to try CFFI, then there will be a temporary phase where we use both to test it and once we decide that indeed CFFI is better for us, then we should drop ctypes.

I'm just not sure that we can adopt CFFI though because of the extra complications in terms of distributing the code or the extra compiling steps.

[juliomateoslangerak]

As I see it, there could be devices implemented using ctypes and others implemented using CFFI. I think it would be nice to have the option to use one or the other. I very well see cases where one would be more suited than the other.

The only situation where I see support might be an issue is when any of us is obliged to fix issues in the other platform that he never used before. Given that most of us are continuing the support in the stuff we developed originally, I don't think it is a major issue. This argument may be inverted too and say that having the option to use one or the other might attract contributors already familiar with either method. That is a weak argument too possible.

As for the distribution, there is no need to distribute header files but just provide the path to them in the host PC. There is some lack of elegance in this. As an alternative, the definitions may be included as a string. That would indeed require naming the same functions (I believe signature is not necessary) and eventually incur into licencing issues. However, that is not more sharing than using ctypes... am I right? I think there is the posibility to just declare the headers such as "include <blink_SDK>" but I didn't try this.

As whether compilation, it is not necessary unless you use the API mode. I don't think that there is a gain in performance that justifies compiling. I did a quick test and calling from Python simple C function returning an int executes 10E7 times in

• 1.3 secs using ctypes
• 0.9 secs using CFFI without compilation
• 0.73 secs using CFFI with pre-compilation
• 0.5 secs if that function was in python