[ENH] Faster and more flexible code, and code sharing for kernel tests

[adam2392]

Towards: #15

Changes proposed in this pull request:

• refactors code to setup for kcd test
• allows any of the pairwise kernel strings to be passed in from sklearn (which is significantly faster than using partial because sklearn optimizes the in-house kernels)
• also requires kernel functions to be a specific API, so it's easier to test, implement and document

This should all make implementation of the kcd test pretty straightforward

Before submitting

• I've read and followed all steps in the Making a pull request
  section of the CONTRIBUTING docs.
• I've updated or added any relevant docstrings following the syntax described in the
  Writing docstrings section of the CONTRIBUTING docs.
• If this PR fixes a bug, I've added a test that will fail without my fix.
• If this PR adds a new feature, I've added tests that sufficiently cover my new functionality.
• I have added a changelog entry succintly describing the change in this PR in the whats_new relevant version document.

After submitting

• All GitHub Actions jobs for my pull request have passed.

[codecov]

Codecov Report

Merging #19 (ed2ebe0) into main (1938c19) will increase coverage by 1.09%.
The diff coverage is 78.99%.

@@            Coverage Diff             @@
##             main      #19      +/-   ##
==========================================
+ Coverage   81.73%   82.83%   +1.09%     
==========================================
  Files           8        9       +1     
  Lines         345      402      +57     
  Branches       66       72       +6     
==========================================
+ Hits          282      333      +51     
- Misses         47       49       +2     
- Partials       16       20       +4     

Files Changed	Coverage Δ
pywhy_stats/kernels.py	81.81% <71.42%> (+16.60%)	⬆️
pywhy_stats/utils.py	62.16% <71.42%> (+5.64%)	⬆️
pywhy_stats/kernel_utils.py	75.00% <75.00%> (ø)
pywhy_stats/independence/kci.py	98.11% <92.30%> (ø)
pywhy_stats/api.py	93.93% <100.00%> (ø)
pywhy_stats/independence/fisherz.py	95.65% <100.00%> (ø)
pywhy_stats/independence/power_divergence.py	71.71% <100.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[adam2392]

Tldr re: #19 (comment)

I will put up a sep commit after Neurips rebuttal period is over to see if we can converge :)

[bloebp]

Can you investigate the failing checks?

[adam2392]

@bloebp lmk wdyt. I addressed the two issues:

1. use RNG, so the seed does not universally affect the tests
2. I refactored, but I think there might be still a discrepancy:

• on one hand pairwise_kernels can take a callable that is defined on (X, Y) 1D row vectors and will run a joblib parallelization
• on the other hand if you define your own kernel callable that can compute this in a vectorized fashion over all the samples, then you want to call it directly rather than joblib parallelize each row.

E.g.

    def my_custom_kernel(X):
        # X here is multidimensional, where the first dimension is continuous and the second categorical. We can handle
        # this by combining two kernels.
        return delta_kernel(X[:, 0].reshape(-1, 1)) * rbf_kernel(X[:, 1].reshape(-1, 1))

we want to call directly, but for the following, we would want to call it via pairwise_kernels.

def unvectorizable_pairwise_kernel(X, Y=None):
     ...

Should we have a rule that if the number of arguments in the kernel callable is exactly 1, then it is assumed to be some optimized kernel callable that is already vectorized and hence should be called directly?

Otoh, if 2 possible function arguments, then it is better to pass it through pairwise_kernels because it is assumed to be of the unvectorizable form that operates on a pair of samples.

[adam2392]

I addressed the issue above and just made it clear that we assume the following:

If the metric is a callable, it will have either one input for ``X``, or two inputs for ``X`` and
    ``Y``. If one input is passed in, it is assumed that the kernel operates on the entire array to compute
    the kernel array. If two inputs are passed in, then it is assumed that the kernel operates on
    pairwise row vectors from each input array. This callable is parallelized across rows of the input
    using :func:`~sklearn.metrics.pairwise.pairwise_kernels`. Note that ``(X, Y=None)`` is a valid input
    signature for the kernel function and would then get passed to the pairwise kernel function. If
    a callable is passed in, it is generally faster and more efficient if one can define a vectorized
    operation that operates on the whole array at once. Otherwise, the pairwise kernel function will
    call the function for each combination of rows in the input arrays.

[adam2392]

Xref: scikit-learn/scikit-learn#27123

[adam2392]

@bloebp I will let you merge if it looks good to you. I enabled auto-squash merge commit.

The last commit should fix the circleCI build.