s d examples

Def

**Understand**



      'Using deep networks for scientific discovery in physiological signals'

we offer a novel method of “removing” a hand-engineered feature from the network’s hypothesis space,
thus forcing it to try and learn representations which are different from known ones, 
as a method of scientific exploration.

We further examine how could our method be used as a tool for 
>> examining scientific hypotheses. 

We simulate this scenario by looking into the importance of eye movements in classifying sleep from EEG.
We show that our tool can successfully focus a researcher’s attention 


-----------


https://arxiv.org/abs/1808.06640

Adversarial Removal of Demographic Attributes from Text Data

(es interesante para probar hipotesis tambien porque puedes 
1. quitar human bias
2. probar un experimento sin una o muchas variables
3. simular como se comportaria/que aprendes de un fenomeno si le quitas una variable) 

----------

Quantum physicists like Roger Melko 
of the Perimeter Institute for Theoretical Physics and the University of Waterloo in Ontario 
have used neural networks to solve some of the toughest and most important problems in that field,
>>such as how to represent 
the mathematical “wave function” describing a many-particle system.

---------

Lee Cronin

More recently, Lee Cronin, a chemist at the University of Glasgow, has been using a robot to
>>>>>>>>>>>>>>>>>>>randomly mix chemicals, to see what sorts of new compounds are formed.

Monitoring the reactions in real-time with a mass spectrometer, a nuclear magnetic resonance machine,
and an infrared spectrometer,
>>>>the system eventually learned to predict which combinations would be the most reactive. 

Even if it doesn’t lead to further discoveries, Cronin has said, 
the robotic system could allow chemists to >>>>>speed up their research by about 90 percent.

------------------

Last year, another team of scientists at ETH Zurich used neural networks to 
  >>deduce physical laws from sets of data.

Their system, a sort of robo-Kepler, 
  >> rediscovered 
the heliocentric model of the solar system from records of the position of the sun and Mars in the sky, as seen from Earth,
and figured out the law of conservation of momentum by observing colliding balls. 

Since physical laws can often be expressed in more than one way, 
the researchers wonder if the system might offer 
  >> new ways — perhaps simpler ways — of thinking about known laws.

------

DeepChem

   Count the number of cells in a microscopy image

------

-Super-Resolution

A technique for transforming low resolution images to high resolution images. 

Scientific Examples: Super resolution is arguably even more useful for scientific settings
than standard natural image benchmarks. Two recent papers look at 
U-nets for super-resolution of fluorescence microscopy [245] (code: https://csbdeep.bioimagecomputing.com/) 
and electron microscopy [56]. 
Other examples include super resolution of chest CT scans [231] and Brain MRIs [31].

----------

https://deepmind.com/blog/article/FermiNet
FermiNet: Quantum Physics and Chemistry from First Principles

In an article recently published in Physical Review Research, we show how deep learning can help solve the fundamental equations of quantum mechanics for real-world systems. 
Not only is this an important fundamental scientific question, but it also could lead to practical uses in the future, allowing researchers to prototype new materials and chemical syntheses in silico before trying to make them in the lab. 


-------------------------------------------------------------------
-------------------------------------------------------------------



**Create**


Using generative modeling, 
astrophysicists could investigate how galaxies change 
when they go from low-density regions of the cosmos to high-density regions,
and what physical processes are responsible for these changes.
Schawinski
https://www.quantamagazine.org/how-artificial-intelligence-is-changing-science-20190311/


----------------
article https://phys.org/news/2020-05-artificial-intelligence-record-setting-catalyst-carbon.html
paper https://www.nature.com/articles/s41586-020-2242-8

Artificial intelligence helps researchers produce record-setting 
catalyst for carbon dioxide-to-ethylene conversion

Researchers at University of Toronto Engineering and Carnegie Mellon University are using (AI) 
>to accelerate progress in transforming waste carbon into a commercially valuable product with record efficiency.

They leveraged AI to 
   >> speed up the search for the key material 
in a new catalyst that converts carbon dioxide (CO2) into ethylene—a chemical precursor to a wide range of products,
from plastics to dish detergent.

The new catalyst is the first one for CO2-to-ethylene conversion to
   >>have been designed in part through the use of AI.
It is also the first experimental demonstration of the active learning approaches Ulissi has been developing.

"With other chemical reactions, we have large and well-established datasets listing the potential catalyst materials
and their properties," says Ulissi.
"With CO2-to-ethylene conversion, we don't have that, 
  >> so we can't use brute force to model everything. 
Our group has spent a lot of time thinking about creative ways
to find the most interesting materials."

The algorithms created by Ulissi and his team use a combination of machine learning models and active learning strategies 
  > to broadly predict what kinds of products a given catalyst is likely to produce, 
even without detailed modeling of the material itself.

They applied these algorithms for CO2 reduction to screen over 240 different materials,
  >> discovering 4 promising candidates that were predicted to have desirable properties 
over a very wide range of compositions and surface structures.

----------

 Deepchem
https://deepchem.readthedocs.io/en/latest/index.html

What are some of the things you can use DeepChem to do? Here’s a few examples:

    Predict the solubility of small drug-like molecules

    Predict binding affinity for small molecule to protein targets

    Predict physical properties of simple materials

    Analyze protein structures and extract useful descriptors
    
  ------
  
  raghu et al
  
-One recent scientific example is given by the use of simple MLPs for pharamaceutical formulation [256], 
developing variants of a drug that is stable and safe for patient use.

-style transfer [64]

-------------------------------------------------------------------
-------------------------------------------------------------------



**Automate**

Adam

Occasionally, grand claims are made regarding the achievements of a “robo-scientist.”
A decade ago, an AI robot chemist named Adam investigated the genome of baker’s yeast 
and worked out >>which genes are responsible for making certain amino acids.
(Adam did this by observing strains of yeast that had certain genes missing, 
and comparing the results to the behavior of strains that had the genes.) 
Wired’s headline read, “Robot Makes Scientific Discovery All by Itself.”

---

Eve - university of manchester

---




-------------------------------------------------------------------
-------------------------------------------------------------------


**Creativity and imagination 




https://openai.com/blog/dall-e/

DALL·E[1]

We decided to name our model using a portmanteau of the artist Salvador Dalí and Pixar’s WALL·E.
is a 12-billion parameter version of GPT-3 trained to generate images from text descriptions,
>> using a dataset of text–image pairs. 
We’ve found that it has a diverse set of capabilities, including creating anthropomorphized versions of animals and objects,
>> combining unrelated concepts in plausible ways, 
rendering text, and applying transformations to existing images.

Overview

Like GPT-3, DALL·E is a transformer language model. 
It receives both the text and the image as a single stream of data containing up to 1280 tokens,
and is trained using maximum likelihood to generate all of the tokens, one after another.

A token is any symbol from a discrete vocabulary;
for humans, each English letter is a token from a 26-letter alphabet.
DALL·E’s vocabulary has tokens for both text and image concepts. 
Specifically, each image caption is represented using a maximum of 256 BPE-encoded tokens with a vocabulary size of 16384,
and the image is represented using 1024 tokens with a vocabulary size of 8192.

This training procedure allows DALL·E to not only generate an image from scratch,
but also to regenerate any rectangular region of an existing image that extends to the bottom-right corner,
in a way that is consistent with the text prompt.

Drawing multiple objects

Simultaneously controlling multiple objects, their attributes, and their spatial relationships presents a new challenge.
For example, consider the phrase “a hedgehog wearing a red hat, yellow gloves, blue shirt, and green pants.” 
To correctly interpret this sentence, DALL·E must not only correctly compose each piece of apparel with the animal,
but also form the associations (hat, red), (gloves, yellow), (shirt, blue), and (pants, green) without mixing them up.[4]

This task is called variable binding, and has been extensively studied in the literature.17181920
We test DALL·E’s ability to do this for relative positioning, stacking objects, and controlling multiple attributes.

While DALL·E does offer some level of controllability over the attributes and positions of a small number of objects,
the success rate can depend on how the caption is phrased. 
As more objects are introduced, DALL·E is prone to confusing the associations between the objects and their colors,
and the success rate decreases sharply.

-Visualizing perspective and three-dimensionality

-Visualizing internal and external structure

-Inferring contextual details
Depending on the orientation of the capybara, it may be necessary to draw a shadow,
though this detail is never mentioned explicitly.
>We explore DALL·E’s ability to resolve underspecification in three cases: 
changing style, setting, and time; 
drawing the same object in a variety of different situations; 
and generating an image of an object with specific text written on it.

>>>> -Combining unrelated concepts

The compositional nature of language 
allows us to put together concepts to describe both real and imaginary things. 
We find that DALL·E also has the ability to combine disparate ideas to synthesize objects,
some of which are unlikely to exist in the real world. 
We explore this ability in two instances: transferring qualities from various concepts to animals,
and designing products by taking inspiration from unrelated concepts.

"an armchair in the shape of an avocado. an armchair imitating an avocado. "

-Animal illustrations

In the previous section, we explored DALL·E’s ability to combine unrelated concepts when generating images of real-world objects.
Here, we explore this ability in the context of >>>> art, for three kinds of illustrations:
anthropomorphized versions of animals and objects, animal chimeras, and emojis.

"an illustration of a baby daikon radish in a tutu walking a dog"

-Zero-shot visual reasoning

GPT-3 can be instructed to perform many kinds of tasks solely from a description 
and a cue to generate the answer supplied in its prompt, without any additional training. 
For example, when prompted with the phrase 
“here is the sentence ‘a person walking his dog in the park’ translated into French:”, 
GPT-3 answers “un homme qui promène son chien dans le parc.” This capability is called zero-shot reasoning.

Motivated by these results, we measure DALL·E’s aptitude for >>analogical reasoning problems 
by testing it on Raven’s progressive matrices, a visual IQ test that saw widespread use in the 20th century.

-Geographic knowledge

We find that DALL·E has learned about geographic facts, landmarks, and neighborhoods. 
Its knowledge of these concepts is surprisingly precise in some ways and flawed in others.

-Temporal knowledge

In addition to exploring DALL·E’s knowledge of concepts that vary over space, 
we also explore its knowledge of concepts that vary over time.
"a photo of a phone from the 20s"

Summary

DALL·E is a simple decoder-only transformer t
hat receives both the text and the image >>as a single stream of 1280 tokens—256 for the text and 1024 for the image—
and models all of them autoregressively. 
The attention mask at each of its 64 self-attention layers 
allows each image token to attend to all text tokens. 
DALL·E uses the standard causal mask for the text tokens, 
and sparse attention for the image tokens 
with either a row, column, or convolutional attention pattern, depending on the layer. 

Text-to-image synthesis 
has been an active area of research since the pioneering work of Reed et. al,
whose approach uses a GAN conditioned on text embeddings. 

we use CLIP to rerank the top 32 of 512 samples for each caption. 
This procedure can also be seen as a kind of language-guided search, 
and can have a dramatic impact on sample quality.

"an illustration of a baby daikon radish in a tutu walking a dog [caption 1, best 8 of 2048]" 

------

-------------------------------------------------------------------

------------------------------------------------------------------


Unstructured





-----------





--------------
from raghu et al   

>>  a lot of them are about science but not necessarily scientific discovery


     Multi-layer perceptrons

Scientific Examples 
One recent scientific example is given by the use of simple MLPs for pharamaceutical formulation [256], 
developing variants of a drug that is stable and safe for patient use.



      CNNs

-Image classification

Scientific Examples: Image classification has found many varied scientific applications, such as in 
analyzing cryoEM data [226] (with associated code https://github.com/cramerlab/boxnet). 
An especially large body of work has looked at medical imaging uses of image classification, specifically, 
using CNNs to predict disease labels. 

Examples range from ophthalmology [72], radiology (2D x-rays and 3D CT scans) [258, 5, 185],
pathology [135, 55], analyzing brain scans (PET, fMRI) [202, 45]. 
An excellent survey of the numerous papers in this area is given by [228].

-Object detection

Scientific Examples: Object detection has also gained significant attention across different scientific applications. 
It has been used in many medical settings to 
   >> localize features of interest, 
for example, tumor cells across different imaging modalities [125, 269] 
or fractures in radiology [199, 227].

-Semantic Segmentation and Instance Segmentation

Scientific Examples: Out of all of the different types of imaging prediction problems, 
segmentation methods have been especially useful for (bio)medical applications. 
Examples include segmenting brain MR images [156, 236], 
identifying key regions of cells in different tissues [254, 217] 
and even studying bone structure [129].

-Super-Resolution

A technique for transforming low resolution images to high resolution images. 

Scientific Examples: Super resolution is arguably even more useful for scientific settings
than standard natural image benchmarks. Two recent papers look at 
U-nets for super-resolution of fluorescence microscopy [245] (code: https://csbdeep.bioimagecomputing.com/) 
and electron microscopy [56]. 
Other examples include super resolution of chest CT scans [231] and Brain MRIs [31].

-Pose Estimation

Scientific Examples: Pose estimation has gained significant interest in neuroscience settings,
where videos of animals are recorded, and
   >>  automatically predicting poses in the image 
can help identify important behaviors. 

An example is given by [146, 147], with associated code http://www.mousemotorlab.org/deeplabcut.

Other Tasks with Convolutional Neural Networks

In the preceding sections, we have overviewed some of the most common tasks for which convolutional neural networks are used.
However, there are many additional use cases of these models that we have not covered, including
video prediction [57],
action recognition [52]
and style transfer [64]. 


     Graph Neural Networks
     
Many datasets, such as (social) network data and chemical molecules
have a graph structure to them, consisting of vertices connected by edges. 

Problems where the data has an inherent graph structure, 
>> and the goal is to learn some function on this graph structure
— either at the per vertex level or a global property of the entire graph.

There are also spatio-temporal graph neural networks 
— performing predictions on graph structures evolving over time.

Scientific Examples: Graph neural networks have been very popular for 
> several chemistry tasks,
such as predicting molecular properties [53, 93, 67, 103], 
determining protein interfaces [60, 229] 
and even generating candidate molecules [41, 21]. 

A useful library for many of these chemistry tasks is https://github.com/ deepchem,
which also has an associated benchmark task [249]. 

A detailed tutorial of different graph neural networks and their use in molecule generation can be seen at 
https://www.youtube.com/watch?v= VXNjCAmb6Zw.



      Neural Networks for Sequence Data

A very common attribute for data is to have a sequential structure. 
This might be frames in a video, amino acid sequences for a protein or words in a sentence. 

Developing neural network models to work with sequence data has been one of the most extensive areas of research 
in the past few years. A large fraction of this has been driven by progress on tasks in natural language processing,


-Language Modelling (Next Token Prediction)

takes as input the tokens of the sequence up to time/position t, 
and then uses these to predict token t + 1.

tries to predict the next word

also for capturing sequence dependencies in scientific applications, such as 
protein function prediction [77, 80], 
and using the hidden representations as part of a larger pipeline for protein structure prediction in AlphaFold 

-Sequence to Sequence

Transforming one sequence to another.
Encoder-decoder structure.
Found in some scientific applications as varied as biology [23] and energy forcasting [145]. 
Sequence to sequence models critically rely on a technique called attention.

-Question Answering

One other popular benchmark for sequence data has been question answering. 
Here, a neural network model is given a paragraph of text (as context) 
and a specific question to answer on this context as input. 
It must then output the part of the paragraph that answers the question.

Question answering critically relies on the neural network model 
understanding the relevance and similarity 
of different sets of sequences (e.g. how relevant is this part of the context to the question of interest?). 

This general capability (with appropriate reformulation) has the potential to be broadly useful,
both for determining similarity and relevance on other datasets,
and for question answering in specialized domains [61].


Models for sequence data 

-RNNs
RNNs have found several scientific applications for data with sequential structure, 
such as in genomics and proteomics [175, 132, 111].

-Attention
A significant problem in using RNNs and working with sequential data is the 
difficulty in capturing long range dependencies
(when tokens in the sequence that are very far apart from each other
must be processed together to inform the correct output). 

-Transformers
Which critically relies on attentional mechanisms (the paper is in fact titled Attention is All you Need.) 
During training transformers take in the entire sequence as input all at once, 
but have positional embeddings that respects the sequential nature of the data.

Scientific Examples: There have been several interesting examples of transformers used in scientific settings, such as
training on protein sequences to find representations encoding meaningful biological properties [195], 
protein generation via language modelling [142], 
bioBERT [121] for text mining in biomedical data (with pretrained model and training code), 
embeddings of scientific text [18] (with code https: //github.com/allenai/scibert) 
and medical question answering [237].


      
      
      Other Tasks with Sequence Data
      
A few other sequential data tasks that might be of interest are 

Structured prediction, where the predicted output has some kind of structure, 
from tree structures (in e.g. parsing) [28, 246] to short, executable computer program structure [271] 

Summarization, where passages of text are summarized by a neural network [130, 273]. 
We’ll also discuss word embeddings later in the survey.



----------------------------------------------------------------------------------------------------