s d interpretability

https://www.kdnuggets.com/2018/12/machine-learning-explainability-interpretability-ai.html

dif between interpretability and explainability

Interpretability 
is about the extent to which a cause and effect can be observed within a system. 
Or, to put it another way, it is the extent to which you are able to predict what is going to happen, 
given a change in input or algorithmic parameters. 
It’s being able to look at an algorithm and go yep, I can see what’s happening here.

Explainability, 
meanwhile, is the extent to which the internal mechanics of a machine or deep learning system 
can be explained in human terms. 

It’s easy to miss the subtle difference with interpretability, but consider it like this: 
interpretability is about being able to discern the mechanics without necessarily knowing why. 
Explainability is being able to quite literally explain what is happening.

Techniques and methods for improving machine learning interpretability

While questions of transparency and ethics may feel abstract for the data scientist on the ground, there are, in fact, 
a number of practical things that can be done to improve an algorithm’s interpretability and explainability.

>Algorithmic generalization
The first is to improve generalization. This sounds simple, but it isn’t that easy.
When you think most machine learning engineering is applying algorithms in a very specific way 
to uncover a certain desired outcome, the model itself can feel like a secondary element - it’s simply a means to an end.
However, by shifting this attitude to consider the overall health of the algorithm, and the data on which it is running,
you can begin to set a solid foundation for improved interpretability.

-Pay attention to feature importance
This should be obvious, but it’s easily missed. Looking closely at the way the various features of your algorithm
have been set is a practical way to actually engage with a diverse range of questions, from business alignment to ethics.
Debate and discussion over how each feature should be set might be a little time-consuming,
but having that tacit awareness that different features have been set in a certain way is nevertheless an important step 
in moving towards interpretability and explainability.

-LIME: Local Interpretable Model-Agnostic Explanations
While the techniques above offer practical steps that data scientists can take,
LIME is an actual method developed by researchers to gain greater transparency on what’s happening inside an algorithm.
The researchers explain that LIME can explain “the predictions of any classifier in an interpretable and faithful manner, 
>>by learning an interpretable model locally around the prediction.”

What this means in practice is that the LIME model develops an approximation of the model 
by testing it out to see what happens when certain aspects within the model are changed. 
>>>Essentially it’s about trying to recreate the output from the same input through a process of experimentation.

-DeepLIFT (Deep Learning Important Features)
DeepLIFT is a useful model in the particularly tricky area of deep learning.
It works through a form of backpropagation:
it takes the output, then attempts to pull it apart
by ‘reading’ the various neurons that have gone into developing that original output.

Essentially, it’s a way of digging back into the feature selection inside of the algorithm (as the name indicates).

-Layer-wise relevance propagation
Layer-wise relevance propagation is similar to DeepLIFT, 
in that it works backwards from the output, identifying the most relevant neurons within the neural network until you return to the input (say, for example, an image). If you want to learn more about the mathematics behind the concept, this post by Dan Shiebler is a great place to begin.
Adding complexity to tackle complexity: can it improve transparency?

The central problem with both explainability and interpretability is that you’re adding an additional step in the development process. Indeed, you’re probably adding multiple steps. From one perspective, this looks like you’re trying to tackle complexity with even greater complexity.

-----------

"using deep networks for scientific discovery in physiological signals"

p.16
Building efficient interpretable systems 
is one of the major obstacles to deployment of machine learning models in healthcare (Tonekaboni et al., 2019). 
Some studies have attempted to interpret the decisions made by their black-box models. 
For 2D imaging data, Rajpurkar et al. (2017) >> inspect class activation mappings of their derived model
to gain trust in its predictions. 
For 1D ECG data, Goodfellow et al. (2018b) >> inspect neural activations 
and find correlates between decision rules of experts and deep networks.


-------------
Maithra Raghu and Eric Schmid


7. Interpretability Model Inspection and Representation Analysis

Many standard applications of deep learning (and machine learning more broadly) focus on 
>prediction — learning to output specific target values given an input.

Scientific applications, on the other hand, are often focused on 
>understanding — identifying underlying mechanisms giving rise to observed patterns in the data.
the ultimate goal remains to understand what attributes give rise to these observations.

To answer these kinds of questions, we can turn to interpretability techniques. 

Interpretability methods are sometimes equated to a fully understandable, step-by-step explanation of the model’s decision process. 
Such detailed insights can often be intractable, especially for complex deep neural network models. 

Instead, research in interpretability focuses on a much broader suite of techniques that can provide insights 
ranging from (rough) >>feature attributions — determining what input features matter the most, 
to >>model inspection — determining what causes certain neurons in the network to fire. 
In fact, these two examples also provide a rough split in the type of interpretability method.

--------
AI researchers want to study AI the same way social scientists study humans

Maybe we don’t need to look inside the black box after all. We just need to watch how machines behave, instead.

Karen Hao https://www.technologyreview.com/2019/04/29/135578/ai-researchers-want-to-study-ai-the-same-way-social-scientists-study-humans/

“We've developed scientific methods to study black boxes for hundreds of years now,
but these methods have primarily been applied to [living beings] up to this point,”
says Nick Obradovich, an MIT Media Lab researcher and co-author of a new paper published last week in Nature.
https://www.nature.com/articles/s41586-019-1138-y
“We can leverage many of the same tools to study the new black box AI systems.”

The paper’s authors, propose to create a new academic discipline called “machine behavior.”
It approaches studying AI systems in the same way we’ve always studied animals and humans: 
>>through empirical observation and experimentation.

How an agent—whether artificial or biological—behaves in its habitat, when coexisting in groups,
and when interacting with other intelligent agents.

“We’re seeing the rise of machines with agency (mmm),
machines that are actors making decisions and taking actions autonomously,”.
Thus they need to be studied “as a new class of actors with their own behavioral patterns and ecology.”

This doesn’t mean to suggest that AI systems have developed some kind of free will. 
(They certainly have not; they’re only glorified math models.)
But it is meant to move away from viewing AI systems as passive tools 
that can be assessed purely through their technical architecture, performance, and capabilities.
They should instead be considered as active actors 
that change and influence their environments and the people and machines around them.

Ultimately, they would study the emergent properties that arise from 
many humans and machines coexisting and collaborating together.

“We are all one giant human-machine system,” says Obradovich.
“We need to acknowledge that and start studying it that way.”

It’s important to note that most of these ideas aren’t new. 
Roboticists, for example, have long studied human-computer interaction.
And the field of science, technology, and society have what’s known as the “actor-network theory,”

Each discipline can take advantage of the other’s expertise
“We need the expertise of scientists from across all behavioral and computational disciplines,” Obradovich says.
“Figuring out how to live with machines is a problem too vast for any one discipline to solve alone.”

-------------
https://www.quantamagazine.org/how-artificial-intelligence-is-changing-science-20190311/

Yet not everyone feels that a lack of transparency is necessarily a problem.
Lenka Zdeborová, a researcher at the Institute of Theoretical Physics at CEA Saclay in France, 
>>points out that human intuitions are often equally impenetrable. 
You look at a photograph and instantly recognize a cat — “but you don’t know how you know,” she said. 
>>“Your own brain is in some sense a black box.”

And the struggle to describe what goes on inside the “mind” of a machine 
is mirrored by the difficulty we have in probing our own thought processes.

------------

George Dyson
Any system simple enough to be understandable
will not be complicated enough to behave intelligently, 
while any system complicated enough to behave intelligently 
will be too complicated to understand.

-----------

 -----Black box problem
" One solution is to build machine-learning systems that show their workings: so-called glassbox—as opposed to black-box—AI. 
 Glassbox models are typically much-simplified versions of a neural network in which it is easier to track
 how different pieces of data affect the model. It's a tricky balance, however. 
 Too much transparency can lead to information overload. 

Visualize this: Another approach is to include visualizations that show a few key properties of the model
and its underlying data. However it’s not clear if they really help.

So-called interpretable or explainable AI (XAI) aims to help people understand what features 
in the data a neural network is actually learning. 

In a new study, researchers found that people who used one of two popular interpretability tools didn’t actually 
understand them but trusted them anyway. In fact, use of the tools led to incorrect assumptions
about the data and the models, and instilled a false confidence that made participants more trigger-happy
about model deployment. This phenomenon is known as “automation bias”—and it could be increasing people’s chance of
using AI that’s inaccurate or biased. This is exactly the opposite effect of what XAI is trying to do in the first place.


- ------------
from mit review 

"Facebook has a new way to track the images used to train a model.
The technique involves making imperceptible tweaks to images to create a kind of watermark.
Every model makes its own distinct mark, and the mark doesn’t impair the image classifier’s overall accuracy. 
This allows researchers to more easily match up models with the images that were used to train them 
in large-scale systems with complicated data pipelines.

Facebook calls the technique “radioactive data” 
because it is analogous to the use of radioactive markers in medicine, which show up in the body under x-ray.
Highlighting what data has been used to train an AI model makes them more transparent.
It could help flag potential sources of bias 
or reveal when a data set was used without permission or for inappropriate purposes. " very cool!

----------------
on evolutionary computation
https://www.technologyreview.com/2018/07/18/104191/evolutionary-algorithm-outperforms-deep-learning-machines-at-video-games/

The evolved code has another advantage. Because it is small, it is easy to see how it works. 
By contrast, a well-known problem with deep-learning techniques is that it is sometimes impossible to know why 
they have made particular decisions, and this can have practical and legal ramifications.