Skip to content

Latest commit

 

History

History
288 lines (217 loc) · 17.8 KB

AutoML.md

File metadata and controls

288 lines (217 loc) · 17.8 KB
Raw

AutoML

AutoML is aimed to find the best algorithm for the specific task.

AutoML draws on many disciplines of machine learning, prominently including

  • Bayesian optimization;
  • Meta learning;
  • Transfer learning;
  • Regression models for structured data and big data;
  • Combinatorial optimization.

Bayesian optimization is introduced in Bayesian Learning. Meta learning will extends beyond the stacking generalization.

AutoML is regarded as an approach to general artificial intelligence. Different from the supervised machine learning, AutoML is to tune the hyper-parameters, which is handed tuned in usual algorithm. Evolutionary or genetic algorithms are applied to solve some unstructured problems.

Automated Feature Engineer

Meta-Learning: Learning to Learn

Learning to Optimize

We may find that the following facts:

Machine learning has enjoyed tremendous success and is being applied to a wide variety of areas, both in AI and beyond. This success can be attributed to the data-driven philosophy that underpins machine learning, which favours automatic discovery of patterns from data over manual design of systems using expert knowledge.

Yet, there is a paradox in the current paradigm: the algorithms that power machine learning are still designed manually. This raises a natural question: can we learn these algorithms instead? This could open up exciting possibilities: we could find new algorithms that perform better than manually designed algorithms, which could in turn improve learning capability.

For some optimization problems, we are lucky to obtain the closed form solver such as the projector or the proximity operator.

The iterative optimization methods such as stochastic gradient descent, ADMM map the current parameters to the new ones closer to the optimum. In terms of mathematics, these methods is a fixed point mapping: $$x^{(t+1)}=M(x^{(t)})$$ so that $f(x^{(t+1)})\leq f(x^{(t)})$ and $\lim_{t\to\infty}f(x^{(t)})=\min f(x)$under some proper conditions.

Learning-based methods

Learnable Optimization

Several recent studies build deep structures by unrolling a particular optimization model that involves task information, i.e., learning to optime.

Like gradient boost decision tree, we can optimize a cost function with a machine learning algorithms to fit the gradients- that is so-called gradient boost machine. In another hand, it is expected that machine learning could learn/approximate the ierative formula of any optimization algorithms.

learning-based iterative methods

Numerous tasks at the core of statistics, learning and vision areas are specific cases of ill-posed inverse problems. Recently, learning-based (e.g., deep) iterative methods have been empirically shown to be useful for these problems. Nevertheless, integrating learnable structures into iterations is still a laborious process, which can only be guided by intuitions or empirical insights. Moreover, there is a lack of rigorous analysis about the convergence behaviors of these reimplemented iterations, and thus the significance of such methods is a little bit vague. This paper moves beyond these limits and proposes Flexible Iterative Modularization Algorithm (FIMA), a generic and provable paradigm for nonconvex inverse problems. Our theoretical analysis reveals that FIMA allows us to generate globally convergent trajectories for learning-based iterative methods. Meanwhile, the devised scheduling policies on flexible modules should also be beneficial for classical numerical methods in the nonconvex scenario. Extensive experiments on real applications verify the superiority of FIMA.

Deep Unrolling

The move from hand-designed features to learned features in machine learning has been wildly successful. In spite of this, optimization algorithms are still designed by hand. In this paper we show how the design of an optimization algorithm can be cast as a learning problem, allowing the algorithm to learn to exploit structure in the problems of interest in an automatic way. Our learned algorithms, implemented by LSTMs, outperform generic, hand-designed competitors on the tasks for which they are trained, and also generalize well to new tasks with similar structure. We demonstrate this on a number of tasks, including simple convex problems, training neural networks, and styling images with neural art.

Transfer Learning

This half-day workshop (in conjunction with IEEE Big Data 2019) is a continuation of our past Big Data Transfer Learning (BDTL) workshops (1st BDTL, 2nd BDTL, 3rd BDTL) which will provide a focused international forum to bring together researchers and research groups to review the status of transfer learning on both conventional vectorized features and heterogeneous information networks.

AutoDL

NAS

Differentiable Neural Architecture Search

The algorithm is based on continuous relaxation and gradient descent in the architecture space. It is able to efficiently design high-performance convolutional architectures for image classification (on CIFAR-10 and ImageNet) and recurrent architectures for language modeling (on Penn Treebank and WikiText-2).

Optimization based Pruning for DNN Compression

AOGNet

An AOGNet consists of a number of stages each of which is composed of a number of AOG building blocks. An AOG building block splits its input feature map into $N$ groups along feature channels and then treat it as a sentence of $N$ words. It then jointly realizes a phrase structure grammar and a dependency grammar in bottom-up parsing the “sentence” for better feature exploration and reuse. It provides a unified framework for the best practices developed in state-of-the-art DNNs.

We first need to understand the underlying wisdom in designing better network architectures: It usually lies in finding network structures which can support flexible and diverse information flows for exploring new features, reusing existing features in previous layers and back-propagating learning signals (e.g., gradients).

Weight Agnostic Neural Networks

AutoNLP

Empirical Model Learning

Active Learning

Sequential Learning

A Python implementation of Online Sequential Extreme Machine Learning (OS-ELM) for online machine learning