Skip to content

Commit

Permalink
Add new GCM model evaluation module
Browse files Browse the repository at this point in the history 
This module adds a new method for evaluating a fitted gcm. Here, we evaluate the performance of causal mechanisms, the underlying modeling assumptions (if possible), the goodness of the generated joint distribution and the graph structure. This utilizes some of the existing methods, but also introduces new ones.

This further adds a new user guide and notebook entries demonstrating the usage.

Part of introducing the module required to make some changes in other modules and implementatins, which are mostly fixes and improvements.

Signed-off-by: Patrick Bloebaum <bloebp@amazon.com>
  • Loading branch information
@bloebp
bloebp committed Nov 17, 2023
1 parent e846851 commit 1d3cd9a
Show file tree
Hide file tree
Showing 18 changed files with 1,601 additions and 94 deletions.
168 changes: 125 additions & 43 deletions docs/source/example_notebooks/gcm_basic_example.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,21 +51,34 @@
},
{
"cell_type": "markdown",
"source": [
"At this point we would normally load our dataset. For this introduction, we generate\n",
"some synthetic data instead. The API takes data in form of Pandas DataFrames:"
],
"id": "85ea234d",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%% md\n"
}
},
"id": "7ec5357c05109df9"
"source": [
"At this point we would normally load our dataset. For this introduction, we generate\n",
"some synthetic data instead. The API takes data in form of Pandas DataFrames:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "71c06991",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"import numpy as np, pandas as pd\n",
Expand All @@ -75,17 +88,20 @@
"Z = 3 * Y + np.random.normal(loc=0, scale=1, size=1000)\n",
"data = pd.DataFrame(data=dict(X=X, Y=Y, Z=Z))\n",
"data.head()"
],
]
},
{
"cell_type": "markdown",
"id": "b24c2de4",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
"name": "#%% md\n"
}
},
"id": "fe1b3eff33c5faf8"
},
{
"cell_type": "markdown",
"source": [
"Note how the columns X, Y, Z correspond to our nodes X, Y, Z in the graph constructed above. We can also see how the\n",
"values of X influence the values of Y and how the values of Y influence the values of Z in that data set.\n",
Expand All @@ -94,29 +110,43 @@
"models. Here, these mechanism can either be assigned manually if, for instance, prior knowledge about certain causal\n",
"relationships are known or they can be assigned automatically using the `auto` module. For the latter,\n",
"we simply call:"
],
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f2c342c9",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%% md\n"
"name": "#%%\n"
}
},
"id": "54e59a824d7fc2d9"
"outputs": [],
"source": [
"auto_assignment_summary = gcm.auto.assign_causal_mechanisms(causal_model, data)"
]
},
{
"cell_type": "markdown",
"id": "6f5d2959-342a-4a2f-af49-3ba0e698f607",
"metadata": {},
"source": [
"Optionally, we can get more insights from the auto assignment process:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4f1493d8-9a0b-4ed5-97be-7b5f39cbd7d4",
"metadata": {},
"outputs": [],
"source": [
"gcm.auto.assign_causal_mechanisms(causal_model, data)"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
},
"id": "ced986f744d8e333"
"print(auto_assignment_summary)"
]
},
{
"cell_type": "markdown",
Expand All @@ -130,19 +160,30 @@
{
"cell_type": "code",
"execution_count": null,
"outputs": [],
"source": [
"causal_model.set_causal_mechanism('X', gcm.EmpiricalDistribution())\n",
"causal_model.set_causal_mechanism('Y', gcm.AdditiveNoiseModel(gcm.ml.create_linear_regressor()))\n",
"causal_model.set_causal_mechanism('Z', gcm.AdditiveNoiseModel(gcm.ml.create_linear_regressor()))"
],
"id": "5a50a60a",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"id": "ae90eccaf892bb41"
"outputs": [],
"source": [
"causal_model.set_causal_mechanism('X', gcm.EmpiricalDistribution())\n",
"causal_model.set_causal_mechanism('Y', gcm.AdditiveNoiseModel(gcm.ml.create_linear_regressor()))\n",
"causal_model.set_causal_mechanism('Z', gcm.AdditiveNoiseModel(gcm.ml.create_linear_regressor()))"
]
},
{
"cell_type": "markdown",
"id": "e0d1cbd1-2544-4b18-a2c9-ddd59cd69f94",
"metadata": {},
"source": [
"Here, we set node X to follow the empirical distribution we observed (nonparametric) and nodes Y and Z to follow an additive noise model where we explicitly set a linear relationship."
]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -176,6 +217,46 @@
"Fitting means, we learn the generative models of the variables in the SCM according to the data."
]
},
{
"cell_type": "markdown",
"id": "766840ab-a353-4061-ae28-f9a9c451e490",
"metadata": {},
"source": [
"Once fitted, we can also obtain more insights into the model performances:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "671cd8d8-75b0-4019-b503-8aa83ad91615",
"metadata": {},
"outputs": [],
"source": [
"print(gcm.evaluate_causal_model(causal_model, data))"
]
},
{
"cell_type": "markdown",
"id": "176d918a-47ed-4440-a925-51f6c19281da",
"metadata": {},
"source": [
"This summary tells us a few things:\n",
"- Our models are a good fit.\n",
"- The additive noise model assumption was not rejected.\n",
"- The generated distribution is very similar to the observed one.\n",
"- The causal graph structure was not rejected."
]
},
{
"cell_type": "markdown",
"id": "67cfaf23-1b90-4124-84cb-39ea67401a22",
"metadata": {},
"source": [
"<div class=\"alert alert-block alert-info\">\n",
"Note, this evaluation take some significant time depending on the model complexities, graph size and amount of data. For a speed-up, consider changing the evaluation parameters.\n",
"</div>"
]
},
{
"cell_type": "markdown",
"id": "fc324e13",
Expand Down Expand Up @@ -205,28 +286,29 @@
},
{
"cell_type": "markdown",
"source": [
"This intervention says: \"I'll ignore any causal effects of X on Y, and set every value of Y\n",
"to 2.34.\" So the distribution of X will remain unchanged, whereas values of Y will be at a fixed\n",
"value and Z will respond according to its causal model."
],
"id": "9976c633",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%% md\n"
}
},
"id": "68a94da68fba303e"
"source": [
"This intervention says: \"I'll ignore any causal effects of X on Y, and set every value of Y to 2.34.\" So the distribution of X will remain unchanged, whereas values of Y will be at a fixed value and Z will respond according to its causal model."
]
},
{
"cell_type": "markdown",
"id": "d65e3319-58ea-4284-8810-c88b10d08448",
"metadata": {},
"source": [
"DoWhy offers a wide range of causal questions that can be answered with GCMs. See the user guide or other notebooks for more examples."
],
"metadata": {
"collapsed": false
},
"id": "1e7aed4828c843a9"
"<div class=\"alert alert-block alert-info\">\n",
"DoWhy offers a wide range of causal questions that can be answered with GCMs. See the user guide or other notebooks for more examples.\n",
"</div>"
]
}
],
"metadata": {
Expand All @@ -245,7 +327,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
"version": "3.9.16"
}
},
"nbformat": 4,
Expand Down
18 changes: 18 additions & 0 deletions docs/source/example_notebooks/gcm_counterfactual_medical_dry_eyes.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,24 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Optionally, we can now also evaluate the fitted causal model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(gcm.evaluate_causal_model(causal_model, medical_data))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This confirms the accuracy of our causal model.\n",
"\n",
"Now returning to our original problem, let's load Alice's data, who happens to have a rare allergy (Condition = 1)."
]
},
Expand Down
36 changes: 17 additions & 19 deletions docs/source/example_notebooks/gcm_draw_samples.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -104,9 +104,7 @@
}
},
"source": [
"If our modeling assumptions are correct, the generated data should now resemble the observed data distribution, i.e. the\n",
"generated samples correspond to the joint distribution we defined for our example data at the beginning. A quick\n",
"way to make sure of this is to estimate the KL-divergence between observed and generated distribution:"
"If our modeling assumptions are correct, the generated data should now resemble the observed data distribution, i.e. the generated samples correspond to the joint distribution we defined for our example data at the beginning. One way to make sure of this is to estimate the KL-divergence between observed and generated distribution. For this, we can make use of the evaluation module:"
]
},
{
Expand All @@ -124,25 +122,25 @@
},
"outputs": [],
"source": [
"gcm.divergence.auto_estimate_kl_divergence(data.to_numpy(), generated_data.to_numpy())"
"print(gcm.evaluate_causal_model(causal_model, data, evaluate_causal_mechanisms=False, evaluate_invertibility_assumptions=False))"
]
},
{
"cell_type": "markdown",
"id": "0ccbc2ad",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"id": "9d12f880-685b-4110-b83d-3d9837b3223e",
"metadata": {},
"source": [
"Here, we expect the divergence to be (very) small.\n",
"\n",
"**Note**: We **cannot** validate the correctness of a causal graph this way,\n",
"since any graph from a Markov equivalence class would be sufficient to generate data that is consistent with the observed one,\n",
"but only one particular graph would generate the correct interventional and counterfactual distributions. This is, seeing the example above,\n",
"X→Y→Z and X←Y←Z would generate the same observational distribution (since they encode the same conditionals), but only X→Y→Z would generate the\n",
"correct interventional distribution (e.g. when intervening on Y)."
"This confirms that the generated distribution is close to the observed one. "
]
},
{
"cell_type": "markdown",
"id": "f430827d-fc75-4c23-8d2d-3d98605dd927",
"metadata": {},
"source": [
"<div class=\"alert alert-block alert-info\">\n",
"While the evaluation provides us insights toward the causal graph structure as well, we cannot confirm the graph structure, only reject it if we find inconsistencies between the dependencies of the observed structure and what the graph represents. In our case, we do not reject the DAG, but there are other equivalent DAGs that would not be rejected as well. To see this, consider the example above - X→Y→Z and X←Y←Z would generate the same observational distribution (since they encode the same conditionals), but only X→Y→Z would generate the correct interventional distribution (e.g., when intervening on Y).\n",
"</div>"
]
}
],
Expand All @@ -162,9 +160,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
"version": "3.9.16"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
}
Loading

0 comments on commit 1d3cd9a

Please sign in to comment.