Definition of estimators of entropy#
Let us define several estimators of entropy. We are going to use the GC -(Gaussian Copula), the KNN (k Nearest Neighbor) and the kernel-based -estimator.
+(Gaussian Copula), the KNN (k Nearest Neighbor), the kernel-based +estimator and the histogram estimator.# list of estimators to compare
metrics = {
"GC": get_entropy("gc"),
"Gaussian": get_entropy(method="gauss"),
+ "Histogram": get_entropy(method="histogram"),
"KNN-3": get_entropy("knn", k=3),
"Kernel": get_entropy("kernel"),
}
@@ -607,7 +608,7 @@ Entropy of data sampled from an exponential distributionplt.show()
Total running time of the script: (0 minutes 17.077 seconds)
+Total running time of the script: (0 minutes 20.368 seconds)
Download Jupyter notebook: plot_entropies.ipynb
Contents
Definition of estimators of entropy#
Let us define several estimators of entropy. We are going to use the GC -(Gaussian Copula), the KNN (k Nearest Neighbor), the kernel-based -and the Gaussian estimators.
+(Gaussian Copula), the KNN (k Nearest Neighbor), the kernel-based, the +Gaussian estimators and the histogram estimator.# list of estimators to compare
metrics = {
"GC": get_entropy("gc"),
"Gaussian": get_entropy("gauss"),
+ "Histogram": get_entropy("histogram"),
"KNN-3": get_entropy("knn", k=3),
"Kernel": get_entropy("kernel"),
}
@@ -582,7 +583,7 @@ Comparison of estimatorsplot(h_x, h_theoric, n)
Total running time of the script: (0 minutes 24.356 seconds)
+Total running time of the script: (0 minutes 32.217 seconds)
Download Jupyter notebook: plot_entropies_mvar.ipynb
Contents
Contents
Definition of entropy estimators#
We are going to use the GCMI (Gaussian Copula Mutual Information), KNN -(k Nearest Neighbor) and a Gaussian kernel-based estimator.
+(k Nearest Neighbor), a Gaussian kernel-based estimator and the histogram +estimator.# list of estimators to compare
metrics = {
"GCMI": get_entropy("gc", biascorrect=False),
"KNN-3": get_entropy("knn", k=3),
"KNN-10": get_entropy("knn", k=10),
"Kernel": get_entropy("kernel"),
+ "Histogram": get_entropy("histogram"),
}
# number of samples to simulate data
-n_samples = np.geomspace(100, 10000, 10).astype(int)
+n_samples = np.geomspace(20, 1000, 15).astype(int)
# number of repetitions to estimate the percentile interval
n_repeat = 10
@@ -500,14 +501,13 @@ Definition of entropy estimatorsax.set_ylabel("Entropy [bits]")
Entropy of data sampled from multinormal distribution#
-Let variables \(X_1,X_2,...,X_n\) have a multivariate normal distribution +
+Let variables \(X_1,X_2,...,X_n\) have a multivariate normal distribution \(\mathcal{N}(\vec{\mu}, \Sigma)\) the theoretical entropy in bits is defined by :
\[H(X) = \frac{1}{2} \times log_{2}({|\Sigma|}(2\pi e)^{n})\]+
# function for creating the covariance matrix with differnt modes
def create_cov_matrix(n_dims, cov, mode="dense", k=None):
"""Create a covariance matrix."""
@@ -581,7 +581,7 @@ Entropy of data sampled from multinormal distributionplt.show()
Total running time of the script: (22 minutes 53.991 seconds)
+Total running time of the script: (0 minutes 24.136 seconds)
Download Jupyter notebook: plot_entropy_high_dimensional.ipynb
Entropy of data sampled from multinormal distribution
Contents
Definition of MI estimators#
Let us define several estimators of MI. We are going to use the GC MI (Gaussian Copula Mutual Information), the KNN (k Nearest Neighbor) and the -kernel-based estimator and using the a binning approach.
+kernel-based estimator, using the a binning approach and the histogram +estimator. Please note that the histogram estimator is equivalent to the +binning, with a correction that relate to the difference between the Shannon +entropy of discrete variables and the differential entropy of continuous +variables. This correction in the case of mutual information (MI) is not +needed, because in the operation to compute the MI, the difference between +discrete and differential entropy cancel out.# create a special function for the binning approach as it requires binary data
mi_binning_fcn = get_mi("binning", base=2)
+
def mi_binning(x, y, **kwargs):
x = digitize(x.T, **kwargs).T
y = digitize(y.T, **kwargs).T
@@ -481,6 +488,7 @@ Definition of MI estimators"KNN-10": get_mi("knn", k=10),
"Kernel": get_mi("kernel"),
"Binning": partial(mi_binning, n_bins=4),
+ "Histogram": get_mi("histogram", n_bins=4),
}
# number of samples to simulate data
@@ -565,7 +573,7 @@ MI of data sampled from normal distributionplt.show()
Total running time of the script: (0 minutes 16.529 seconds)
+Total running time of the script: (0 minutes 20.639 seconds)
Download Jupyter notebook: plot_mi.ipynb
Contents
Definition of MI estimators#
-We are going to use the GCMI (Gaussian Copula Mutual Information) and KNN -(k Nearest Neighbor)
+We are going to use the GCMI (Gaussian Copula Mutual Information), KNN +(k Nearest Neighbor) and histogram estimator.
# list of estimators to compare
metrics = {
"GCMI": get_mi("gc", biascorrect=False),
"KNN-3": get_mi("knn", k=3),
"KNN-10": get_mi("knn", k=10),
+ "Histogram": get_mi("histogram", n_bins=3),
}
# number of samples to simulate data
-n_samples = np.geomspace(1000, 10000, 10).astype(int)
+n_samples = np.geomspace(20, 1000, 15).astype(int)
# number of repetitions to estimate the percentile interval
n_repeat = 10
@@ -584,7 +585,7 @@ MI of data sampled from splitted multinormal distributionplt.show()
Total running time of the script: (42 minutes 38.604 seconds)
+Total running time of the script: (0 minutes 30.050 seconds)