-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathtong-ji-xue-xi-fang-fa-luo-ji-si-di-hui-gui-yu-zui-da-shang-mo-xing.html
executable file
·153 lines (144 loc) · 7.61 KB
/
tong-ji-xue-xi-fang-fa-luo-ji-si-di-hui-gui-yu-zui-da-shang-mo-xing.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
<!DOCTYPE html>
<html lang="zh">
<head>
<meta charset="utf-8" />
<title>《统计学习方法》-逻辑斯蒂回归与最大熵模型</title>
<link rel="stylesheet" href="https://bridgeqiao.github.io/blog/theme/css/main.css" />
<!--[if IE]>
<script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
</head>
<body id="index" class="home">
<header id="banner" class="body">
<h1><a href="https://bridgeqiao.github.io/blog/">Bridge Qiao's Blog </a></h1>
<nav><ul>
<li><a href="https://bridgeqiao.github.io/blog/category/computer-vision.html">Computer vision</a></li>
<li><a href="https://bridgeqiao.github.io/blog/category/fun.html">fun</a></li>
<li><a href="https://bridgeqiao.github.io/blog/category/linux.html">linux</a></li>
<li class="active"><a href="https://bridgeqiao.github.io/blog/category/machine-learning.html">Machine Learning</a></li>
<li><a href="https://bridgeqiao.github.io/blog/category/matlab.html">Matlab</a></li>
<li><a href="https://bridgeqiao.github.io/blog/category/suan-fa-ti.html">算法题</a></li>
</ul>
</nav>
</header><!-- /#banner -->
<section id="content" class="body">
<article>
<header>
<h1 class="entry-title">
<a href="https://bridgeqiao.github.io/blog/tong-ji-xue-xi-fang-fa-luo-ji-si-di-hui-gui-yu-zui-da-shang-mo-xing.html" rel="bookmark"
title="Permalink to 《统计学习方法》-逻辑斯蒂回归与最大熵模型">《统计学习方法》-逻辑斯蒂回归与最大熵模型</a></h1>
</header>
<div class="entry-content">
<footer class="post-info">
<span>二 23 一月 2018</span>
</footer><!-- /.post-info --> <p>逻辑斯蒂回归是一个比较经典的分类方法，最大熵模型是按照最大熵准则来进行分类的算法，两个都属于对数线性模型。</p>
<h2>逻辑斯蒂回归</h2>
<p>在二分类中的模型：
</p>
<div class="math">$$
\begin{aligned}
P(Y=1|X)&=\frac{exp(\omega X+b)}{1+exp(\omega X+b)}\\
P(Y=0|X)&=\frac{1}{1+exp(\omega X+b)}
\end{aligned}
$$</div>
<p>
这里<span class="math">\(X\in \boldsymbol{R}^n\)</span>是输入，<span class="math">\(Y\in \{0,1\}\)</span>是输出，比较两个条件概率的大小，将实例x分到概率值大的那一类。这个公式对于求解问题不是很直观，将分类的概率变成对数几率或<code>logit</code>函数：
</p>
<div class="math">$$
logit(p)=log\frac{p}{1-p}=\omega X+b，p表示事件发生的概率
$$</div>
<p>
这样的好处是直接对分布进行建模，避免了假设分布不准确所带来的问题。</p>
<p>采用极大似然法来估计<span class="math">\(\omega,b\)</span>：
</p>
<div class="math">$$
l(\omega,b)=\sum_{i=1}^{m}\ln p(y_i|x_i;w,b)
$$</div>
<p>
令<span class="math">\(\beta=(\omega;b),\hat{x}=(x;1)\)</span>，则<span class="math">\(\omega^Tx+b\)</span>可简写为<span class="math">\(\beta^Tx\)</span>，最后得到：
</p>
<div class="math">$$
l(\beta)=\sum_{i=1}^{m}(-y_i\beta^T\hat{x}_i+\ln(1+e^{\beta^T\hat{x}_i}))\\
\beta^*=\arg\min_\beta l(\beta)
$$</div>
<h2>最大熵模型</h2>
<p>求出给定数据集的联合分布<span class="math">\(f(x,y)\)</span>和边缘分布<span class="math">\(f(x)\)</span>。</p>
<p>最大熵模型定义了在给定输入变量<span class="math">\(x\)</span>时，输出变量y的条件分布：
</p>
<div class="math">$$
P(y|x,\theta)=\frac{e^{\theta f(x,y)}}{\sum_{y\in M}e^{\theta f(x,y)}}
$$</div>
<p>
<span class="math">\(f(x,y)\)</span>相当于约束条件，输入和输出的需要满足的条件，可以当作对输入和输出同时抽取的特征，分类效果还可以，但运行速度不理想。</p>
<p>在具体计算时，将条件分布转换成对偶问题进行的求解，过程比较复杂，感兴趣的可以看看《统计学习方法》。</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdn.bootcss.com/mathjax/2.7.4/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>
</div><!-- /.entry-content -->
</article>
</section>
<section id="extras" class="body">
<div class="blogroll">
<h2>blogroll</h2>
<ul>
<li><a href="http://getpelican.com/">Pelican</a></li>
<li><a href="http://python.org/">Python.org</a></li>
<li><a href="https://github.com">Github</a></li>
</ul>
</div><!-- /.blogroll -->
</section><!-- /#extras -->
<footer id="contentinfo" class="body">
<p>Powered by <a href="http://getpelican.com/">Pelican</a>. Theme <a href="https://github.com/blueicefield/pelican-blueidea/">blueidea</a>, inspired by the default theme.</p>
</footer><!-- /#contentinfo -->
</body>
</html>