-
Notifications
You must be signed in to change notification settings - Fork 1
/
atom.xml
191 lines (91 loc) · 226 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>Ryan961's Studio</title>
<icon>https://ryan961.github.io/icon.png</icon>
<link href="https://ryan961.github.io/atom.xml" rel="self"/>
<link href="https://ryan961.github.io/"/>
<updated>2024-01-19T16:00:00.000Z</updated>
<id>https://ryan961.github.io/</id>
<author>
<name>Ryan961</name>
</author>
<generator uri="https://hexo.io/">Hexo</generator>
<entry>
<title>Go Slices: internals and usage</title>
<link href="https://ryan961.github.io/2024/01/19/go-slice/"/>
<id>https://ryan961.github.io/2024/01/19/go-slice/</id>
<published>2024-01-19T16:00:00.000Z</published>
<updated>2024-01-19T16:00:00.000Z</updated>
<content type="html"><![CDATA[<blockquote><p><em>本文基于官方文档，并借鉴了大量讨论 slice 的文章来综述 slice 的底层结构和使用方法。你可以在文末找到这些文章的链接，欢迎阅读。</em></p></blockquote><h1 id="介绍"><a href="#介绍" class="headerlink" title="介绍"></a>介绍</h1><p>Go 的切片类型提供了一种处理类型化数据序列的便捷高效的方法。切片类似于其他语言中的数组，但具有一些不寻常的属性。</p><h1 id="Arrays"><a href="#Arrays" class="headerlink" title="Arrays"></a>Arrays</h1><p>切片类型是建立在 Go 数组类型之上的抽象，但它在日常编程中的使用并不多，此处仅作简单介绍。</p><ul><li>数组类型定义指定长度和元素类型。例如，类型 [4]int 表示一个由四个整数组成的数组。</li><li>数组的大小是固定的(<strong><em>数组的 len 和 cap 相等，就等于数据长度</em></strong>)，它的长度是其类型的一部分（[4]int 和 [5]int 是不同的、不兼容的类型）。</li><li>数组可以以通常的方式索引，因此表达式 s[n] 访问第 n 个元素，从零开始。</li><li>数组不需要显式初始化；数组的零值是一个随时可用的数组，其元素本身已为零</li></ul><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> sliceA [<span class="number">2</span>]<span class="type">int</span></span><br><span class="line">sliceB := [<span class="number">2</span>]<span class="type">int</span>{}</span><br><span class="line">sliceC := [...]<span class="type">int</span>{<span class="number">0</span>, <span class="number">0</span>}</span><br><span class="line"></span><br><span class="line">fmt.Println(sliceA, <span class="built_in">len</span>(sliceA), <span class="built_in">cap</span>(sliceA)) <span class="comment">// prints [] 0 0</span></span><br><span class="line">fmt.Println(sliceB, <span class="built_in">len</span>(sliceB), <span class="built_in">cap</span>(sliceB)) <span class="comment">// prints [] 0 0</span></span><br><span class="line">fmt.Println(sliceC, <span class="built_in">len</span>(sliceC), <span class="built_in">cap</span>(sliceC)) <span class="comment">// prints [] 0 0</span></span><br></pre></td></tr></table></figure><p><a href="https://go.dev/play/p/cIDpNTcNuTF">(Try it yourself)</a></p><h1 id="Slices"><a href="#Slices" class="headerlink" title="Slices"></a>Slices</h1><p>切片的类型规范为 []T，其中 T 是切片元素的类型。与数组类型不同，切片类型没有指定的长度。切片文字的声明方式与数组文字类似，只是省略了元素计数：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> sliceA []<span class="type">int</span></span><br><span class="line">sliceB := []<span class="type">int</span>{}</span><br><span class="line">sliceC := []<span class="type">int</span>{<span class="number">1</span>, <span class="number">2</span>}</span><br><span class="line">sliceD := sliceC[:<span class="number">1</span>]</span><br><span class="line">sliceE := sliceC[<span class="number">0</span>:<span class="number">1</span>:<span class="built_in">cap</span>(sliceC)]</span><br><span class="line">sliceF := sliceC[:<span class="number">0</span>]</span><br><span class="line">sliceG := <span class="built_in">make</span>([]<span class="type">int</span>, <span class="number">0</span>)</span><br><span class="line">sliceH := <span class="built_in">make</span>([]<span class="type">int</span>, <span class="number">1</span>)</span><br><span class="line">sliceI := <span class="built_in">make</span>([]<span class="type">int</span>, <span class="number">0</span>, <span class="number">1</span>)</span><br><span class="line">sliceJ := <span class="built_in">make</span>([]<span class="type">int</span>, <span class="number">1</span>, <span class="number">2</span>)</span><br><span class="line"></span><br><span class="line">fmt.Println(sliceA, <span class="built_in">len</span>(sliceA), <span class="built_in">cap</span>(sliceA)) <span class="comment">// prints [] 0 0</span></span><br><span class="line">fmt.Println(sliceB, <span class="built_in">len</span>(sliceB), <span class="built_in">cap</span>(sliceB)) <span class="comment">// prints [] 0 0</span></span><br><span class="line">fmt.Println(sliceC, <span class="built_in">len</span>(sliceC), <span class="built_in">cap</span>(sliceC)) <span class="comment">// prints [1 2] 2 2</span></span><br><span class="line">fmt.Println(sliceD, <span class="built_in">len</span>(sliceD), <span class="built_in">cap</span>(sliceD)) <span class="comment">// prints [1] 1 2</span></span><br><span class="line">fmt.Println(sliceE, <span class="built_in">len</span>(sliceE), <span class="built_in">cap</span>(sliceE)) <span class="comment">// prints [1] 1 2</span></span><br><span class="line">fmt.Println(sliceF, <span class="built_in">len</span>(sliceF), <span class="built_in">cap</span>(sliceF)) <span class="comment">// prints [] 0 2</span></span><br><span class="line">fmt.Println(sliceG, <span class="built_in">len</span>(sliceG), <span class="built_in">cap</span>(sliceG)) <span class="comment">// prints [] 0 0</span></span><br><span class="line">fmt.Println(sliceH, <span class="built_in">len</span>(sliceH), <span class="built_in">cap</span>(sliceH)) <span class="comment">// prints [0] 1 1</span></span><br><span class="line">fmt.Println(sliceI, <span class="built_in">len</span>(sliceI), <span class="built_in">cap</span>(sliceI)) <span class="comment">// prints [] 0 1</span></span><br><span class="line">fmt.Println(sliceJ, <span class="built_in">len</span>(sliceJ), <span class="built_in">cap</span>(sliceJ)) <span class="comment">// prints [0] 1 2</span></span><br></pre></td></tr></table></figure><p><a href="https://go.dev/play/p/DRTBGyrpbPq">(Try it yourself)</a></p><p><strong>// 注意：<code>slice</code> 只可以和 <code>nil</code> 做比较, <code>slice</code> 之间做比较编译期间就不会通过。</strong></p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">> fmt.Println(sliceA == sliceB)</span><br><span class="line">invalid operation: sliceA == sliceB (slice can only be compared to nil)</span><br></pre></td></tr></table></figure><h2 id="数据结构"><a href="#数据结构" class="headerlink" title="数据结构"></a>数据结构</h2><blockquote><p><em><a href="https://github.com/golang/go/blob/master/src/runtime/slice.go#L15">https://github.com/golang/go/blob/master/src/runtime/slice.go#L15</a></em></p></blockquote><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> slice <span class="keyword">struct</span> {</span><br><span class="line">array unsafe.Pointer</span><br><span class="line"><span class="built_in">len</span> <span class="type">int</span></span><br><span class="line"><span class="built_in">cap</span> <span class="type">int</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><ul><li><code>array</code> 是指向数组的指针。</li><li><code>len</code> 是切片中的元素数量（与调用 <code>len(s)</code> 获得的值相同）。</li><li><code>cap</code> 是切片的总容量（切片开头和分配内存末尾之间可以容纳的元素数量）。</li></ul><p>从 <code>slice</code> 的数据结构来看，它所存储的是数据数组的指针地址(<code>array</code>)。所以如果传递切片，就会创建一个指向原始数组 <code>array</code> 的一个新的 <code>slice</code> 切片。这样，对切片的操作就与对数组索引的操作有着相同的高效性，但这也造成了<strong>新切片元素的改动会同时影响到原始切片元素</strong>（这是因为 <code>array</code> 是相同的）。</p><h2 id="append-元素-扩容机制"><a href="#append-元素-扩容机制" class="headerlink" title="append 元素(扩容机制)"></a>append 元素(扩容机制)</h2><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">sliceA := []<span class="type">int</span>{<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>}</span><br><span class="line">sliceB := <span class="built_in">append</span>(sliceA, <span class="number">4</span>)</span><br><span class="line">sliceC := <span class="built_in">append</span>(sliceB, <span class="number">5</span>)</span><br><span class="line">sliceC[<span class="number">0</span>] = <span class="number">0</span></span><br><span class="line"></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceA)), sliceA) <span class="comment">// prints {0xc00009a000 3 3} [1 2 3]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceB)), sliceB) <span class="comment">// prints {0xc00009c000 4 6} [0 2 3 4]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceC)), sliceC) <span class="comment">// prints {0xc00009c000 5 6} [0 2 3 4 5]</span></span><br><span class="line"></span><br><span class="line">sliceD := <span class="built_in">append</span>([]<span class="type">int</span>{<span class="number">1</span>, <span class="number">2</span>}, <span class="number">3</span>)</span><br><span class="line">sliceE := <span class="built_in">append</span>(sliceD, <span class="number">4</span>)</span><br><span class="line">sliceF := <span class="built_in">append</span>(sliceD, <span class="number">5</span>)</span><br><span class="line">sliceF[<span class="number">0</span>] = <span class="number">0</span></span><br><span class="line"></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceD)), sliceD) <span class="comment">// prints {0xc0000aa020 3 4} [0 2 3]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceE)), sliceE) <span class="comment">// prints {0xc0000aa020 4 4} [0 2 3 5]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceF)), sliceF) <span class="comment">// prints {0xc0000aa020 4 4} [0 2 3 5]</span></span><br></pre></td></tr></table></figure><p><a href="https://go.dev/play/p/aM7wIBbQmBr">(Try it yourself)</a></p><h3 id="append-的两种场景"><a href="#append-的两种场景" class="headerlink" title="append 的两种场景"></a>append 的两种场景</h3><p><img data-src="https://cdn.jsdelivr.net/gh/ryan961/img-floder/blog/screenshot-20240121-190658.png" alt="screenshot-20240121-190658"></p><ul><li>当 append 之后的长度小于等于 cap，将会直接利用原底层数组剩余的空间。</li><li>当 append 后的长度大于 cap 时，则会分配一块更大的区域来容纳新的底层数组。</li></ul><p><strong>因此，为了避免内存发生拷贝，如果能够知道最终的切片的大小，预先设置 cap 的值能够获得更好的性能。</strong></p><h3 id="growslice-切片扩容"><a href="#growslice-切片扩容" class="headerlink" title="growslice 切片扩容"></a><a href="https://github.com/golang/go/blob/f19f31f2e7c136a8dae03cbfe4f8ebbb8b54569b/src/runtime/slice.go#L155">growslice</a> 切片扩容</h3><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// growslice allocates new backing store for a slice.</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// arguments:</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">//oldPtr = pointer to the slice's backing array</span></span><br><span class="line"><span class="comment">//newLen = new length (= oldLen + num)</span></span><br><span class="line"><span class="comment">//oldCap = original slice's capacity.</span></span><br><span class="line"><span class="comment">// num = number of elements being added</span></span><br><span class="line"><span class="comment">// et = element type</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// return values:</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">//newPtr = pointer to the new backing store</span></span><br><span class="line"><span class="comment">//newLen = same value as the argument</span></span><br><span class="line"><span class="comment">//newCap = capacity of the new backing store</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// Requires that uint(newLen) > uint(oldCap).</span></span><br><span class="line"><span class="comment">// Assumes the original slice length is newLen - num</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// A new backing store is allocated with space for at least newLen elements.</span></span><br><span class="line"><span class="comment">// Existing entries [0, oldLen) are copied over to the new backing store.</span></span><br><span class="line"><span class="comment">// Added entries [oldLen, newLen) are not initialized by growslice</span></span><br><span class="line"><span class="comment">// (although for pointer-containing element types, they are zeroed). They</span></span><br><span class="line"><span class="comment">// must be initialized by the caller.</span></span><br><span class="line"><span class="comment">// Trailing entries [newLen, newCap) are zeroed.</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// growslice's odd calling convention makes the generated code that calls</span></span><br><span class="line"><span class="comment">// this function simpler. In particular, it accepts and returns the</span></span><br><span class="line"><span class="comment">// new length so that the old length is not live (does not need to be</span></span><br><span class="line"><span class="comment">// spilled/restored) and the new length is returned (also does not need</span></span><br><span class="line"><span class="comment">// to be spilled/restored).</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">growslice</span><span class="params">(oldPtr unsafe.Pointer, newLen, oldCap, num <span class="type">int</span>, et *_type)</span></span> slice {</span><br><span class="line">oldLen := newLen - num</span><br><span class="line"><span class="keyword">if</span> raceenabled {</span><br><span class="line">callerpc := getcallerpc()</span><br><span class="line">racereadrangepc(oldPtr, <span class="type">uintptr</span>(oldLen*<span class="type">int</span>(et.Size_)), callerpc, abi.FuncPCABIInternal(growslice))</span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> msanenabled {</span><br><span class="line">msanread(oldPtr, <span class="type">uintptr</span>(oldLen*<span class="type">int</span>(et.Size_)))</span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> asanenabled {</span><br><span class="line">asanread(oldPtr, <span class="type">uintptr</span>(oldLen*<span class="type">int</span>(et.Size_)))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> newLen < <span class="number">0</span> {</span><br><span class="line"><span class="built_in">panic</span>(errorString(<span class="string">"growslice: len out of range"</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> et.Size_ == <span class="number">0</span> {</span><br><span class="line"><span class="comment">// append should not create a slice with nil pointer but non-zero len.</span></span><br><span class="line"><span class="comment">// We assume that append doesn't need to preserve oldPtr in this case.</span></span><br><span class="line"><span class="keyword">return</span> slice{unsafe.Pointer(&zerobase), newLen, newLen}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">newcap := nextslicecap(newLen, oldCap)</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> overflow <span class="type">bool</span></span><br><span class="line"><span class="keyword">var</span> lenmem, newlenmem, capmem <span class="type">uintptr</span></span><br><span class="line"><span class="comment">// Specialize for common values of et.Size.</span></span><br><span class="line"><span class="comment">// For 1 we don't need any division/multiplication.</span></span><br><span class="line"><span class="comment">// For goarch.PtrSize, compiler will optimize division/multiplication into a shift by a constant.</span></span><br><span class="line"><span class="comment">// For powers of 2, use a variable shift.</span></span><br><span class="line">noscan := et.PtrBytes == <span class="number">0</span></span><br><span class="line"><span class="keyword">switch</span> {</span><br><span class="line"><span class="keyword">case</span> et.Size_ == <span class="number">1</span>:</span><br><span class="line">lenmem = <span class="type">uintptr</span>(oldLen)</span><br><span class="line">newlenmem = <span class="type">uintptr</span>(newLen)</span><br><span class="line">capmem = roundupsize(<span class="type">uintptr</span>(newcap), noscan)</span><br><span class="line">overflow = <span class="type">uintptr</span>(newcap) > maxAlloc</span><br><span class="line">newcap = <span class="type">int</span>(capmem)</span><br><span class="line"><span class="keyword">case</span> et.Size_ == goarch.PtrSize:</span><br><span class="line">lenmem = <span class="type">uintptr</span>(oldLen) * goarch.PtrSize</span><br><span class="line">newlenmem = <span class="type">uintptr</span>(newLen) * goarch.PtrSize</span><br><span class="line">capmem = roundupsize(<span class="type">uintptr</span>(newcap)*goarch.PtrSize, noscan)</span><br><span class="line">overflow = <span class="type">uintptr</span>(newcap) > maxAlloc/goarch.PtrSize</span><br><span class="line">newcap = <span class="type">int</span>(capmem / goarch.PtrSize)</span><br><span class="line"><span class="keyword">case</span> isPowerOfTwo(et.Size_):</span><br><span class="line"><span class="keyword">var</span> shift <span class="type">uintptr</span></span><br><span class="line"><span class="keyword">if</span> goarch.PtrSize == <span class="number">8</span> {</span><br><span class="line"><span class="comment">// Mask shift for better code generation.</span></span><br><span class="line">shift = <span class="type">uintptr</span>(sys.TrailingZeros64(<span class="type">uint64</span>(et.Size_))) & <span class="number">63</span></span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line">shift = <span class="type">uintptr</span>(sys.TrailingZeros32(<span class="type">uint32</span>(et.Size_))) & <span class="number">31</span></span><br><span class="line">}</span><br><span class="line">lenmem = <span class="type">uintptr</span>(oldLen) << shift</span><br><span class="line">newlenmem = <span class="type">uintptr</span>(newLen) << shift</span><br><span class="line">capmem = roundupsize(<span class="type">uintptr</span>(newcap)<<shift, noscan)</span><br><span class="line">overflow = <span class="type">uintptr</span>(newcap) > (maxAlloc >> shift)</span><br><span class="line">newcap = <span class="type">int</span>(capmem >> shift)</span><br><span class="line">capmem = <span class="type">uintptr</span>(newcap) << shift</span><br><span class="line"><span class="keyword">default</span>:</span><br><span class="line">lenmem = <span class="type">uintptr</span>(oldLen) * et.Size_</span><br><span class="line">newlenmem = <span class="type">uintptr</span>(newLen) * et.Size_</span><br><span class="line">capmem, overflow = math.MulUintptr(et.Size_, <span class="type">uintptr</span>(newcap))</span><br><span class="line">capmem = roundupsize(capmem, noscan)</span><br><span class="line">newcap = <span class="type">int</span>(capmem / et.Size_)</span><br><span class="line">capmem = <span class="type">uintptr</span>(newcap) * et.Size_</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// The check of overflow in addition to capmem > maxAlloc is needed</span></span><br><span class="line"><span class="comment">// to prevent an overflow which can be used to trigger a segfault</span></span><br><span class="line"><span class="comment">// on 32bit architectures with this example program:</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// type T [1<<27 + 1]int64</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// var d T</span></span><br><span class="line"><span class="comment">// var s []T</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// func main() {</span></span><br><span class="line"><span class="comment">// s = append(s, d, d, d, d)</span></span><br><span class="line"><span class="comment">// print(len(s), "\n")</span></span><br><span class="line"><span class="comment">// }</span></span><br><span class="line"><span class="keyword">if</span> overflow || capmem > maxAlloc {</span><br><span class="line"><span class="built_in">panic</span>(errorString(<span class="string">"growslice: len out of range"</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> p unsafe.Pointer</span><br><span class="line"><span class="keyword">if</span> et.PtrBytes == <span class="number">0</span> {</span><br><span class="line">p = mallocgc(capmem, <span class="literal">nil</span>, <span class="literal">false</span>)</span><br><span class="line"><span class="comment">// The append() that calls growslice is going to overwrite from oldLen to newLen.</span></span><br><span class="line"><span class="comment">// Only clear the part that will not be overwritten.</span></span><br><span class="line"><span class="comment">// The reflect_growslice() that calls growslice will manually clear</span></span><br><span class="line"><span class="comment">// the region not cleared here.</span></span><br><span class="line">memclrNoHeapPointers(add(p, newlenmem), capmem-newlenmem)</span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line"><span class="comment">// Note: can't use rawmem (which avoids zeroing of memory), because then GC can scan uninitialized memory.</span></span><br><span class="line">p = mallocgc(capmem, et, <span class="literal">true</span>)</span><br><span class="line"><span class="keyword">if</span> lenmem > <span class="number">0</span> && writeBarrier.enabled {</span><br><span class="line"><span class="comment">// Only shade the pointers in oldPtr since we know the destination slice p</span></span><br><span class="line"><span class="comment">// only contains nil pointers because it has been cleared during alloc.</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// It's safe to pass a type to this function as an optimization because</span></span><br><span class="line"><span class="comment">// from and to only ever refer to memory representing whole values of</span></span><br><span class="line"><span class="comment">// type et. See the comment on bulkBarrierPreWrite.</span></span><br><span class="line">bulkBarrierPreWriteSrcOnly(<span class="type">uintptr</span>(p), <span class="type">uintptr</span>(oldPtr), lenmem-et.Size_+et.PtrBytes, et)</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line">memmove(p, oldPtr, lenmem)</span><br><span class="line"></span><br><span class="line"><span class="keyword">return</span> slice{p, newLen, newcap}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// nextslicecap computes the next appropriate slice length.</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">nextslicecap</span><span class="params">(newLen, oldCap <span class="type">int</span>)</span></span> <span class="type">int</span> {</span><br><span class="line">newcap := oldCap</span><br><span class="line">doublecap := newcap + newcap</span><br><span class="line"><span class="keyword">if</span> newLen > doublecap {</span><br><span class="line"><span class="keyword">return</span> newLen</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">const</span> threshold = <span class="number">256</span></span><br><span class="line"><span class="keyword">if</span> oldCap < threshold {</span><br><span class="line"><span class="keyword">return</span> doublecap</span><br><span class="line">}</span><br><span class="line"><span class="keyword">for</span> {</span><br><span class="line"><span class="comment">// Transition from growing 2x for small slices</span></span><br><span class="line"><span class="comment">// to growing 1.25x for large slices. This formula</span></span><br><span class="line"><span class="comment">// gives a smooth-ish transition between the two.</span></span><br><span class="line">newcap += (newcap + <span class="number">3</span>*threshold) >> <span class="number">2</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// We need to check `newcap >= newLen` and whether `newcap` overflowed.</span></span><br><span class="line"><span class="comment">// newLen is guaranteed to be larger than zero, hence</span></span><br><span class="line"><span class="comment">// when newcap overflows then `uint(newcap) > uint(newLen)`.</span></span><br><span class="line"><span class="comment">// This allows to check for both with the same comparison.</span></span><br><span class="line"><span class="keyword">if</span> <span class="type">uint</span>(newcap) >= <span class="type">uint</span>(newLen) {</span><br><span class="line"><span class="keyword">break</span></span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// Set newcap to the requested cap when</span></span><br><span class="line"><span class="comment">// the newcap calculation overflowed.</span></span><br><span class="line"><span class="keyword">if</span> newcap <= <span class="number">0</span> {</span><br><span class="line"><span class="keyword">return</span> newLen</span><br><span class="line">}</span><br><span class="line"><span class="keyword">return</span> newcap</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><img data-src="https://cdn.jsdelivr.net/gh/ryan961/img-floder/blog/20240121220913.png" alt="20240121220913"></p><p>slice 扩容过程的简化流程:</p><ol><li>检查新长度 newLen 是否超过旧 slice 的容量 oldCap，如果新长度 newLen 小于 0，则抛出错误。</li><li>如果所处理的元素类型的大小为零，返回长度和容量都和新长度 newLen 相同的 slice。</li><li>使用 nextslicecap(newLen, oldCap) 函数计算新的容量 newCap：<ul><li>首先，新的容量 newCap 初始化为旧的容量 oldCap 的二倍。</li><li>如果新长度 newLen 大于这个 doublecap，那么新长度 newLen 就会成为新的容量 newCap。</li><li>如果旧的 slice 容量 oldCap 小于阈值 256，则新的容量 newCap 就是 doublecap，即旧容量的二倍。</li><li>如果旧的 slice 容量 oldCap 大于阈值 256，那么新的容量 newCap 就会在旧的容量 oldCap 基础上增长约 25%，在增长过程中，只要新的容量 newCap 足以容纳新长度 newLen，增长过程就会停止。</li><li>在整个过程中，一旦 newCap 出现溢出，将直接将新长度 newLen 设为新容量 newCap。</li></ul></li><li>根据元素类型和新的容量 newCap，计算需要的内存大小，并在特殊情况下（例如元素类型大小为 1、指针大小以及 2 的幂次）做一些优化操作。在这个过程中，可能需要特别处理来避免计算溢出。</li><li>检查计算结果是否溢出或者超出最大分配大小，如果是则抛出错误。</li><li>为新的容量 newCap 分配新的内存。</li><li>清空新内存的未使用部分。</li><li>将旧 slice 的数据复制到新的内存中。</li><li>返回新的 slice，包括新内存的指针，新的长度 newLen 和新的容量 newCap。</li></ol><h2 id="截取-拷贝"><a href="#截取-拷贝" class="headerlink" title="截取(拷贝)"></a>截取(拷贝)</h2><p>切片表达式的签名：<code>s[start:end:cap]</code></p><p>基于切片 s 创建一个新的切片（可以包括原切片的所有元素），包含的元素从索引 <code>start</code> 处开始，到但不包括 <code>end</code> 索引处的元素，<code>cap</code> 是新创建子切片的容量，是可选的。如果 <code>cap</code> 省略，子切片的容量等于其长度。子切片的长度计算公式：<code>end - start</code>。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line">sliceA := []<span class="type">int</span>{<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>}</span><br><span class="line">sliceB := sliceA[:]</span><br><span class="line">sliceC := sliceA[:<span class="number">1</span>]</span><br><span class="line">sliceD := sliceA[<span class="number">1</span>:]</span><br><span class="line">sliceE := sliceA[:<span class="number">1</span>:<span class="number">2</span>]</span><br><span class="line"></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceA)), sliceA) <span class="comment">// prints {0xc00001a018 3 3} [1 2 3]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceB)), sliceB) <span class="comment">// prints {0xc00001a018 3 3} [1 2 3]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceC)), sliceC) <span class="comment">// prints {0xc00001a018 1 3} [1]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceD)), sliceD) <span class="comment">// prints {0xc00001a020 2 2} [2 3]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceE)), sliceE) <span class="comment">// prints{0xc00001a018 1 2} [1]</span></span><br><span class="line"></span><br><span class="line">sliceF := <span class="built_in">make</span>([]<span class="type">int</span>, <span class="built_in">len</span>(sliceA))</span><br><span class="line"><span class="built_in">copy</span>(sliceF, sliceA) <span class="comment">// a deep copy of sliceA</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceA)), sliceA) <span class="comment">// prints {0xc00001a018 3 3} [1 2 3]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceF)), sliceF) <span class="comment">// prints {0xc00001a030 3 3} [1 2 3]</span></span><br><span class="line"></span><br><span class="line">sliceG := <span class="built_in">make</span>([]<span class="type">int</span>, <span class="number">1</span>, <span class="number">2</span>)</span><br><span class="line">sliceH := modify(sliceG) <span class="comment">// sliceG[0] = 1</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceG)), sliceG) <span class="comment">// prints {0xc0001200a0 1 2} [1]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceH)), sliceH) <span class="comment">// prints {0xc0001200a0 1 2} [1]</span></span><br><span class="line"></span><br><span class="line">sliceI := append1(sliceG) <span class="comment">// append(sliceG, 4)</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceG)), sliceG) <span class="comment">// prints {0xc0001200a0 1 2} [1]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceI)), sliceI) <span class="comment">// prints {0xc0001200a0 2 2} [1 4]</span></span><br><span class="line"></span><br><span class="line">sliceJ := append2(sliceG) <span class="comment">// append(sliceG, 5, 6)</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceG)), sliceG) <span class="comment">// prints {0xc0001200a0 1 2} [1]</span></span><br><span class="line">fmt.Println(*(*slice)(unsafe.Pointer(&sliceJ)), sliceJ) <span class="comment">// prints {0xc000122020 3 4} [1 5 6]</span></span><br></pre></td></tr></table></figure><p><a href="https://go.dev/play/p/DAcDhGEBaW4">(Try it yourself)</a></p><ul><li>切片的截取是创建一个指向原始数组 <code>array</code> 的一个新的 <code>slice</code> 切片。新切片的 <code>cap</code> 最大不得超过原切片 <code>cap</code>（超过则会报错 <code>panic: runtime error: slice bounds out of range [::5] with capacity 3</code>）。<code>[:n]</code> 时，新切片 <code>cap</code> 等于原切片 <code>cap</code>, <code>[n:]</code> 新切片 <code>cap</code> 等于原切片 <code>cap</code> 减 n。</li><li>切片在传递的过程中，会创建一个指向原始数组 <code>array</code> 的一个新的 <code>slice</code> 切片，函数中对切片的修改<strong>可能会导致原切片的底层数组发生变化</strong>。而一旦切片发生扩容，则会对切片进行<strong>深度拷贝</strong>，深度拷贝会创建一个新的切片，并拷贝原切片的元素，新切片将拥有自己独立的元素副本。</li><li>可以使用 <code>func copy(dst, src []T) int</code> 函数对切片进行深度拷贝, dst 是目标切片，src 是源切片。两个切片必须有相同的元素类型 T。该函数返回复制的元素数量，<strong>取 dst 和 src 长度的最小值</strong>。注意，<strong>如果目标切片 dst 的长度小于源切片 src 的长度，则只会复制 src 的前 len(dst) 个元素</strong>。要深度复制整个切片，必须确保 dst 具有足够的容量以容纳 src 的所有元素。</li></ul><h2 id="range-遍历"><a href="#range-遍历" class="headerlink" title="range 遍历"></a>range 遍历</h2><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">sliceA := []<span class="type">int</span>{<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>}</span><br><span class="line"><span class="keyword">for</span> i := <span class="keyword">range</span> sliceA {</span><br><span class="line"> fmt.Printf(<span class="string">"sliceA[%d]: %d\n"</span>, i, sliceA[i])</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">sliceB := []<span class="type">int</span>{<span class="number">4</span>, <span class="number">5</span>}</span><br><span class="line"><span class="keyword">for</span> i, v := <span class="keyword">range</span> sliceB {</span><br><span class="line"> fmt.Printf(<span class="string">"sliceB[%d]: %d\n"</span>, i, v)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">sliceC := []<span class="type">int</span>{<span class="number">7</span>}</span><br><span class="line"><span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="built_in">len</span>(sliceC); i++ {</span><br><span class="line"> fmt.Printf(<span class="string">"sliceC[%d]: %d\n"</span>, i, sliceC[i])</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><a href="https://go.dev/play/p/eTmbqkAv5un">(Try it yourself)</a></p><h1 id="技巧与陷阱"><a href="#技巧与陷阱" class="headerlink" title="技巧与陷阱"></a>技巧与陷阱</h1><h2 id="slice-不是并发安全的数据结构"><a href="#slice-不是并发安全的数据结构" class="headerlink" title="slice 不是并发安全的数据结构"></a><code>slice</code> 不是并发安全的数据结构</h2><h2 id="将切片-slice-作为输入传递给-append-或函数后避免再次使用它"><a href="#将切片-slice-作为输入传递给-append-或函数后避免再次使用它" class="headerlink" title="将切片 slice 作为输入传递给 append 或函数后避免再次使用它"></a>将切片 <code>slice</code> 作为输入传递给 <code>append</code> 或函数后避免再次使用它</h2><blockquote><ul><li><a href="https://www.dolthub.com/blog/2023-10-20-golang-pitfalls-3/">They’re called Slices because they have Sharp Edges: Even More Go Pitfalls</a></li></ul></blockquote><p>对于初学者或不理解底层原理的用户而言，append 或传递后的 <code>slice</code> 变得不可预测（详见上面的 <a href=""><em>append 元素(扩容机制)</em></a>）。在日常场景中，避免再次使用可以减少很多奇奇怪怪的问题。</p><h2 id="预分配-slice-内存，避免-growslice-发生内存拷贝"><a href="#预分配-slice-内存，避免-growslice-发生内存拷贝" class="headerlink" title="预分配 slice 内存，避免 growslice 发生内存拷贝"></a>预分配 <code>slice</code> 内存，避免 <code>growslice</code> 发生内存拷贝</h2><blockquote><ul><li><a href="https://oilbeater.com/2023/07/19/pre-alloc-slice-for-golang/"><em>Golang 中预分配 slice 内存对性能的影响</em></a></li><li><a href="https://oilbeater.com/2024/01/09/alloc-slice-for-golang-2-md/"><em>Golang 中预分配 slice 内存对性能的影响（续）</em></a></li></ul></blockquote><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">sliceA := []<span class="type">int</span>{<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>}</span><br><span class="line">sliceB := <span class="built_in">make</span>([]<span class="type">int</span>, <span class="number">0</span>, <span class="built_in">len</span>(sliceA))</span><br><span class="line"><span class="keyword">for</span> _, v := <span class="keyword">range</span> sliceA {</span><br><span class="line">sliceB = <span class="built_in">append</span>(sliceB, v)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="切片复用，避免重复创建"><a href="#切片复用，避免重复创建" class="headerlink" title="切片复用，避免重复创建"></a>切片复用，避免重复创建</h2><blockquote><ul><li><a href="https://go.dev/wiki/SliceTricks">Go Wiki: SliceTricks #Filtering without allocating</a></li><li><a href="https://books.studygolang.com/advanced-go-programming-book/ch1-basic/ch1-03-array-string-and-slice.html">切片内存技巧</a></li></ul></blockquote><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">TrimSpace</span><span class="params">(s []<span class="type">byte</span>)</span></span> []<span class="type">byte</span> {</span><br><span class="line"> b := s[:<span class="number">0</span>]</span><br><span class="line"> <span class="keyword">for</span> _, x := <span class="keyword">range</span> s {</span><br><span class="line"> <span class="keyword">if</span> x != <span class="string">' '</span> {</span><br><span class="line"> b = <span class="built_in">append</span>(b, x)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> b</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>对于性能敏感的场景, 复用对象可以避免反复的对象创建的内存开销。另外还有使用 <code>sync.Pool</code> 以及使用俄罗斯大佬 (<code>fasthttp</code> 作者) 的 <a href="https://github.com/valyala/bytebufferpool">bytebufferpool</a> 将性能推到极限的做法（详见: <a href="https://oilbeater.com/2024/01/09/alloc-slice-for-golang-2-md/"><em>Golang 中预分配 slice 内存对性能的影响（续）</em></a>）。</p><h2 id="避免切片内存泄漏"><a href="#避免切片内存泄漏" class="headerlink" title="避免切片内存泄漏"></a>避免切片内存泄漏</h2><blockquote><ul><li><a href="https://books.studygolang.com/advanced-go-programming-book/ch1-basic/ch1-03-array-string-and-slice.html">避免切片内存泄漏</a></li><li><a href="https://mp.weixin.qq.com/s/p7CxNIk0YI_cS3pebUWAOg">深入探究 Go 中的 array 与 slice</a></li><li><a href="https://geektutu.com/post/hpg-slice.html#3-1-%E5%A4%A7%E9%87%8F%E5%86%85%E5%AD%98%E5%BE%97%E4%B8%8D%E5%88%B0%E9%87%8A%E6%94%BE">切片(slice)性能及陷阱</a></li><li><a href="https://github.com/golang/go/issues/63393">slices: have Delete and others clear the tail #63393</a></li></ul></blockquote><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> gopherRegexp = regexp.MustCompile(<span class="string">"gopher"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 泄漏场景</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">FindGopher</span><span class="params">(filename <span class="type">string</span>)</span></span> []<span class="type">byte</span> {</span><br><span class="line"> <span class="comment">// 读取大文件 1,000,000,000 bytes (1GB)</span></span><br><span class="line"> b, _ := ioutil.ReadFile(filename)</span><br><span class="line"> <span class="comment">// 只取一个 6 字节的子切片</span></span><br><span class="line"> gopherSlice := <span class="built_in">make</span>([]<span class="type">byte</span>, <span class="built_in">len</span>(<span class="string">"gopher"</span>))</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 深度拷贝</span></span><br><span class="line"> <span class="built_in">copy</span>(gopherSlice, gopherRegexp.Find(b...)</span><br><span class="line"> <span class="keyword">return</span> gopherSlice</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 正确场景</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">FindGopher</span><span class="params">(filename <span class="type">string</span>)</span></span> []<span class="type">byte</span> {</span><br><span class="line"> <span class="comment">// 读取大文件 1,000,000,000 bytes (1GB)</span></span><br><span class="line"> b, _ := ioutil.ReadFile(filename)</span><br><span class="line"> <span class="comment">// 只取一个 6 字节的子切片</span></span><br><span class="line"> gopherSlice := <span class="built_in">make</span>([]<span class="type">byte</span>, <span class="built_in">len</span>(<span class="string">"gopher"</span>))</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 深度拷贝</span></span><br><span class="line"> <span class="built_in">copy</span>(gopherSlice, gopherRegexp.Find(b...)</span><br><span class="line"> <span class="keyword">return</span> gopherSlice</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="for-i-0-i-len-slice-i-迭代，避免-range-值拷贝开销"><a href="#for-i-0-i-len-slice-i-迭代，避免-range-值拷贝开销" class="headerlink" title="for i := 0; i < len(slice); i++ { ... } 迭代，避免 range 值拷贝开销"></a><code>for i := 0; i < len(slice); i++ { ... }</code> 迭代，避免 <code>range</code> 值拷贝开销</h2><blockquote><ul><li><a href="https://www.calhoun.io/does-range-copy-the-slice-in-go/">Does Go’s range Copy a Slice Before Iterating Over It?</a></li><li><a href="https://geektutu.com/post/hpg-range.html">for 和 range 的性能比较</a></li></ul></blockquote><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">sliceA := []<span class="type">int</span>{<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>}</span><br><span class="line"><span class="keyword">for</span> k, v := <span class="keyword">range</span> sliceA {</span><br><span class="line">fmt.Println(k, v)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="built_in">len</span>(sliceA); i++ {</span><br><span class="line">fmt.Println(i, sliceA[i])</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>对于性能敏感的场景，避免使用 <code>range</code> 可以减少值拷贝开销。<code>range</code> 在迭代过程中返回的是迭代值的拷贝，如果每次迭代的元素的内存占用很低，那么 <code>for</code> 和 <code>range</code> 的性能几乎是一样，例如 <code>[]int</code>。但是如果迭代的元素内存占用较高，例如一个包含很多属性的 <code>struct</code> 结构体，那么 <code>for</code> 的性能将显著地高于 <code>range</code>，有时候甚至会有上千倍的性能差异。对于这种场景，建议使用 <code>for</code>，如果使用 <code>range</code>，建议只迭代下标，通过下标访问迭代值，这种使用方式和 <code>for</code> 就没有区别了。如果想使用 <code>range</code> 同时迭代下标和值，则需要将切片/数组的元素改为指针，才能不影响性能。</p><h1 id="推荐阅读"><a href="#推荐阅读" class="headerlink" title="推荐阅读"></a>推荐阅读</h1><ul><li><a href="https://github.com/golang/go/blob/master/src/runtime/slice.go#L15">https://github.com/golang/go/blob/master/src/runtime/slice.go</a></li><li><a href="https://go.dev/blog/slices-intro">Go Slices: usage and internals</a></li><li><a href="https://go.dev/wiki/SliceTricks">Go Wiki: SliceTricks</a></li><li><a href="https://ueokande.github.io/go-slice-tricks/">Go Slice Tricks Cheat Sheet</a></li><li><a href="https://github.com/golang/go/issues/63393">slices: have Delete and others clear the tail #63393</a></li><li><a href="https://www.calhoun.io/does-range-copy-the-slice-in-go/">Does Go’s range Copy a Slice Before Iterating Over It?</a></li><li><a href="https://www.dolthub.com/blog/2023-09-08-much-ado-about-nil-things/">Much Ado About Nil Things: More Go Pitfalls</a></li><li><a href="https://www.dolthub.com/blog/2023-10-20-golang-pitfalls-3/">They’re called Slices because they have Sharp Edges: Even More Go Pitfalls</a></li><li><a href="https://oilbeater.com/2023/07/19/pre-alloc-slice-for-golang/">Golang 中预分配 slice 内存对性能的影响</a></li><li><a href="https://oilbeater.com/2024/01/09/alloc-slice-for-golang-2-md/">Golang 中预分配 slice 内存对性能的影响（续）</a></li><li><a href="https://mp.weixin.qq.com/s/p7CxNIk0YI_cS3pebUWAOg">深入探究 Go 中的 array 与 slice</a></li><li><a href="https://mp.weixin.qq.com/s/uNajVcWr4mZpof1eNemfmQ">你真的了解 go 语言中的切片吗？</a></li><li><a href="https://books.studygolang.com/advanced-go-programming-book/ch1-basic/ch1-03-array-string-and-slice.html">Go 语言高级编程(数组、字符串和切片)</a></li><li><a href="https://draveness.me/golang/docs/part2-foundation/ch03-datastructure/golang-array-and-slice/#%E4%BD%BF%E7%94%A8%E4%B8%8B%E6%A0%87">Go 语言设计与实现(切片)</a></li><li><a href="https://geektutu.com/post/hpg-slice.html">Go 语言高性能编程(切片性能及陷阱)</a></li><li><a href="https://geektutu.com/post/hpg-range.html">Go 语言高性能编程(for 和 range 的性能比较)</a></li><li><a href="https://mp.weixin.qq.com/s/GUY8_pF9C7W-NCs1rB5KXg">Go 扒一扒 sort 工具包底层用的哪些排序算法？</a></li><li><a href="https://www.ardanlabs.com/blog/2023/08/golang-slices-binary-search.html">Slices Package: Binary Search</a></li><li><a href="https://www.ardanlabs.com/blog/2023/08/golang-slices-clip-clone-compact.html">Slices Package: Clip, Clone, and Compact</a></li><li><a href="https://www.ardanlabs.com/blog/2023/08/golang-slices-compare.html">Slices Package: Compare</a></li><li><a href="https://www.ardanlabs.com/blog/2023/09/golang-slices-contains-delete-equal.html">Slices Package: Contains, Delete, and Equal</a></li></ul><p><strong><em>More to come…</em></strong> :)</p>]]></content>
<summary type="html"><blockquote>
<p><em>本文基于官方文档，并借鉴</summary>
<category term="golang" scheme="https://ryan961.github.io/categories/golang/"/>
<category term="golang" scheme="https://ryan961.github.io/tags/golang/"/>
</entry>
<entry>
<title>Elasticsearch Notes</title>
<link href="https://ryan961.github.io/2024/01/05/elasticsearch-notes/"/>
<id>https://ryan961.github.io/2024/01/05/elasticsearch-notes/</id>
<published>2024-01-05T16:00:00.000Z</published>
<updated>2024-01-05T16:00:00.000Z</updated>
<content type="html"><![CDATA[<h2 id="Environment"><a href="#Environment" class="headerlink" title="Environment"></a>Environment</h2><ul><li><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/index.html">Elasticsearch</a> version: 7.x</p></li><li><p><a href="https://go.dev/doc/install">Go</a> version: >= 1.20 (下面示例中使用了 Go Generic 的工具类库 <a href="https://github.com/samber/lo">lo</a>、<a href="https://github.com/sourcegraph/conc">conc</a> … )</p><p>Go Elastic Client: <a href="https://github.com/olivere/elastic">olivere/elastic</a></p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"context"</span></span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line"></span><br><span class="line"> <span class="string">"github.com/olivere/elastic/v7"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">var</span> (</span><br><span class="line"> host = []<span class="type">string</span>{<span class="string">"http://192.168.16.28:9200"</span>}</span><br><span class="line"> user = <span class="string">"elastic"</span></span><br><span class="line"> password = <span class="string">"*****"</span></span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"> cli, err := NewClient(host, user, password)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="built_in">panic</span>(err)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> healthResponse, err := cli.CatHealth().Do(context.Background())</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="built_in">panic</span>(err)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> fmt.Println(healthResponse) <span class="comment">// [{1688526238 03:03:58 flower-es green 3 3 2135 1068 0 0 0 0 - 100.0%}]</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">NewClient</span><span class="params">(host []<span class="type">string</span>, user, password <span class="type">string</span>)</span></span> (*elastic.Client, <span class="type">error</span>) {</span><br><span class="line"> <span class="comment">// 当 es 服务器监听（publish_address）使用内网服务器 ip，而访问（bound_addresses）使用外网IP时，不要设置 client.transport.sniff 为 true。</span></span><br><span class="line"> <span class="comment">// 不设置 client.transport.sniff 时，默认为 false (关闭客户端去嗅探整个集群的状态)。因为在自动发现时会使用内网 IP 进行通信，</span></span><br><span class="line"> <span class="comment">// 导致无法连接到 es 服务器。因此此时需要直接使用 addTransportAddress 方法把集群中其它机器的 ip 地址加到客户端中。</span></span><br><span class="line"> <span class="keyword">return</span> elastic.NewClient(elastic.SetSniff(<span class="literal">false</span>), elastic.SetURL(host...), elastic.SetBasicAuth(user, password))</span><br><span class="line">}</span><br></pre></td></tr></table></figure></li></ul><h2 id="Respository"><a href="#Respository" class="headerlink" title="Respository"></a>Respository</h2><p><em><a href="https://github.com/ryan961/es-examples">https://github.com/ryan961/es-examples</a></em></p><h2 id="Examples"><a href="#Examples" class="headerlink" title="Examples"></a>Examples</h2><h3 id="Terms-Aggregation"><a href="#Terms-Aggregation" class="headerlink" title="Terms Aggregation"></a>Terms Aggregation</h3><blockquote><p><em><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/search-aggregations-bucket-terms-aggregation.html">Terms aggregation | Elasticsearch Guide [7.17] | Elastic</a></em></p></blockquote><h4 id="Kibana-Dev-Tools"><a href="#Kibana-Dev-Tools" class="headerlink" title="Kibana Dev Tools"></a>Kibana Dev Tools</h4><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestTerms</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"terms"</span>:{<span class="string">"field"</span>:<span class="string">"platform"</span>,<span class="string">"order"</span>:[{<span class="string">"_count"</span>:<span class="string">"desc"</span>}],<span class="string">"size"</span>:<span class="number">10</span>}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"*"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688313600</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1689004799</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestTerms_SubAggregation</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"aggregations"</span>:{<span class="string">"platform"</span>:{<span class="string">"terms"</span>:{<span class="string">"field"</span>:<span class="string">"platform"</span>}}},<span class="string">"terms"</span>:{<span class="string">"field"</span>:<span class="string">"appVersion"</span>,<span class="string">"order"</span>:[{<span class="string">"_count"</span>:<span class="string">"desc"</span>}],<span class="string">"size"</span>:<span class="number">10</span>}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"*"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688313600</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1689004799</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><h4 id="Go-Examples"><a href="#Go-Examples" class="headerlink" title="Go Examples"></a>Go Examples</h4><p><em><a href="https://github.com/ryan961/es-examples/blob/main/examples/search_aggs_terms_test.go">https://github.com/ryan961/es-examples/blob/main/examples/search_aggs_terms_test.go</a></em></p><h3 id="DateHistogram-Aggregation"><a href="#DateHistogram-Aggregation" class="headerlink" title="DateHistogram Aggregation"></a>DateHistogram Aggregation</h3><blockquote><p><em><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/search-aggregations-bucket-datehistogram-aggregation.html">Date histogram aggregation | Elasticsearch Guide [7.17] | Elastic</a></em></p></blockquote><h4 id="Kibana-Dev-Tools-1"><a href="#Kibana-Dev-Tools-1" class="headerlink" title="Kibana Dev Tools"></a>Kibana Dev Tools</h4><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestDateHistogram</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"date_histogram"</span>:{<span class="string">"calendar_interval"</span>:<span class="string">"1d"</span>,<span class="string">"field"</span>:<span class="string">"@timestamp"</span>,<span class="string">"time_zone"</span>:<span class="string">"Asia/Shanghai"</span>}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"*"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688313600</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1689004799</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestDateHistogram_SubAggregation</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"aggregations"</span>:{<span class="string">"dau"</span>:{<span class="string">"cardinality"</span>:{<span class="string">"field"</span>:<span class="string">"uuid"</span>}}},<span class="string">"date_histogram"</span>:{<span class="string">"calendar_interval"</span>:<span class="string">"1d"</span>,<span class="string">"field"</span>:<span class="string">"@timestamp"</span>,<span class="string">"time_zone"</span>:<span class="string">"Asia/Shanghai"</span>}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"*"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688313600</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1689004799</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestDateHistogram_RuntimeMappings</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"aggregations"</span>:{<span class="string">"regDay"</span>:{<span class="string">"aggregations"</span>:{<span class="string">"dau"</span>:{<span class="string">"cardinality"</span>:{<span class="string">"field"</span>:<span class="string">"uuid"</span>}}},<span class="string">"date_histogram"</span>:{<span class="string">"calendar_interval"</span>:<span class="string">"1d"</span>,<span class="string">"field"</span>:<span class="string">"register_time_date"</span>,<span class="string">"time_zone"</span>:<span class="string">"Asia/Shanghai"</span>}}},<span class="string">"date_histogram"</span>:{<span class="string">"calendar_interval"</span>:<span class="string">"1d"</span>,<span class="string">"field"</span>:<span class="string">"@timestamp"</span>,<span class="string">"time_zone"</span>:<span class="string">"Asia/Shanghai"</span>}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"_exists_:register_time"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688313600</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1689004799</span>}}}]}},<span class="string">"runtime_mappings"</span>:{<span class="string">"register_time_date"</span>:{<span class="string">"script"</span>:{<span class="string">"source"</span>:<span class="string">"if(doc['register_time'].size()!=0) {emit(doc['register_time'].value*1000);} else {emit(0);}"</span>},<span class="string">"type"</span>:<span class="string">"date"</span>}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><h4 id="Go-Examples-1"><a href="#Go-Examples-1" class="headerlink" title="Go Examples"></a>Go Examples</h4><p><em><a href="https://github.com/ryan961/es-examples/blob/main/examples/search_aggs_dateHistogram_test.go">https://github.com/ryan961/es-examples/blob/main/examples/search_aggs_dateHistogram_test.go</a></em></p><h3 id="Filters-Aggregation"><a href="#Filters-Aggregation" class="headerlink" title="Filters Aggregation"></a>Filters Aggregation</h3><blockquote><p><em><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/search-aggregations-bucket-filters-aggregation.html">Filters aggregation | Elasticsearch Guide [7.17] | Elastic</a></em></p></blockquote><h4 id="Kibana-Dev-Tools-2"><a href="#Kibana-Dev-Tools-2" class="headerlink" title="Kibana Dev Tools"></a>Kibana Dev Tools</h4><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestFilters</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"filters"</span>:{<span class="string">"filters"</span>:{<span class="string">"itemsLog"</span>:{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"LogType:itemsLog"</span>}},<span class="string">"loginLog"</span>:{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"LogType:loginLog"</span>}},<span class="string">"scoreLog"</span>:{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"LogType:scoreLog"</span>}}}}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"*"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688313600</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1689004799</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestFilters_SubAggregation</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"aggregations"</span>:{<span class="string">"count"</span>:{<span class="string">"cardinality"</span>:{<span class="string">"field"</span>:<span class="string">"uuid"</span>}}},<span class="string">"filters"</span>:{<span class="string">"filters"</span>:{<span class="string">"itemsLog"</span>:{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"LogType:itemsLog"</span>}},<span class="string">"loginLog"</span>:{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"LogType:loginLog"</span>}},<span class="string">"scoreLog"</span>:{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"LogType:scoreLog"</span>}}}}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"*"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688313600</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1689004799</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><h4 id="Go-Examples-2"><a href="#Go-Examples-2" class="headerlink" title="Go Examples"></a>Go Examples</h4><p><em><a href="https://github.com/ryan961/es-examples/blob/main/examples/search_aggs_filters_test.go">https://github.com/ryan961/es-examples/blob/main/examples/search_aggs_filters_test.go</a></em></p><h3 id="Composite-Aggregation"><a href="#Composite-Aggregation" class="headerlink" title="Composite Aggregation"></a>Composite Aggregation</h3><blockquote><p><em><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/search-aggregations-bucket-composite-aggregation.html">Composite aggregation | Elasticsearch Guide [7.17] | Elastic</a></em></p></blockquote><h4 id="Kibana-Dev-Tools-3"><a href="#Kibana-Dev-Tools-3" class="headerlink" title="Kibana Dev Tools"></a>Kibana Dev Tools</h4><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestComposite</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"composite"</span>:{<span class="string">"size"</span>:<span class="number">10000</span>,<span class="string">"sources"</span>:[{<span class="string">"customerId"</span>:{<span class="string">"terms"</span>:{<span class="string">"field"</span>:<span class="string">"customerId"</span>}}}]}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"*"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688054400</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1688745599</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestComposite_SubAggregation</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"aggregations"</span>:{<span class="string">"lastLevel"</span>:{<span class="string">"max"</span>:{<span class="string">"field"</span>:<span class="string">"topLevel"</span>}},<span class="string">"lastLoginIn"</span>:{<span class="string">"max"</span>:{<span class="string">"field"</span>:<span class="string">"actTime"</span>}}},<span class="string">"composite"</span>:{<span class="string">"size"</span>:<span class="number">10000</span>,<span class="string">"sources"</span>:[{<span class="string">"customerId"</span>:{<span class="string">"terms"</span>:{<span class="string">"field"</span>:<span class="string">"customerId"</span>}}}]}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"*"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688054400</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1688745599</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestComposite_DateHistogram</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"composite"</span>:{<span class="string">"size"</span>:<span class="number">10000</span>,<span class="string">"sources"</span>:[{<span class="string">"actDay"</span>:{<span class="string">"date_histogram"</span>:{<span class="string">"calendar_interval"</span>:<span class="string">"1d"</span>,<span class="string">"field"</span>:<span class="string">"@timestamp"</span>,<span class="string">"time_zone"</span>:<span class="string">"Asia/Shanghai"</span>}}},{<span class="string">"customerId"</span>:{<span class="string">"terms"</span>:{<span class="string">"field"</span>:<span class="string">"customerId"</span>}}}]}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"*"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688054400</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1688745599</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestComposite_DateHistogram_RuntimeMappings</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"runtime_mappings"</span>:{<span class="string">"register_time_date"</span>:{<span class="string">"type"</span>:<span class="string">"date"</span>,<span class="string">"script"</span>:{<span class="string">"source"</span>:<span class="string">"if(doc['register_time'].size()!=0)\n {emit(doc['register_time'].value*1000);}\n else{emit(0);}"</span>}}},<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"composite"</span>:{<span class="string">"size"</span>:<span class="number">10000</span>,<span class="string">"sources"</span>:[{<span class="string">"actDay"</span>:{<span class="string">"date_histogram"</span>:{<span class="string">"calendar_interval"</span>:<span class="string">"1d"</span>,<span class="string">"field"</span>:<span class="string">"@timestamp"</span>,<span class="string">"time_zone"</span>:<span class="string">"Asia/Shanghai"</span>}}},{<span class="string">"regDay"</span>:{<span class="string">"date_histogram"</span>:{<span class="string">"calendar_interval"</span>:<span class="string">"1d"</span>,<span class="string">"field"</span>:<span class="string">"register_time_date"</span>,<span class="string">"time_zone"</span>:<span class="string">"Asia/Shanghai"</span>}}},{<span class="string">"customerId"</span>:{<span class="string">"terms"</span>:{<span class="string">"field"</span>:<span class="string">"customerId"</span>}}},{<span class="string">"register_time"</span>:{<span class="string">"terms"</span>:{<span class="string">"field"</span>:<span class="string">"register_time"</span>}}}]}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"*"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1687968000</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1688659199</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestComposite_DateHistogram_Filters</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"aggregations"</span>:{<span class="string">"dau"</span>:{<span class="string">"aggregations"</span>:{<span class="string">"count"</span>:{<span class="string">"cardinality"</span>:{<span class="string">"field"</span>:<span class="string">"uuid"</span>}}},<span class="string">"filter"</span>:{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"LogType:loginLog"</span>}}},<span class="string">"itemNumbers"</span>:{<span class="string">"aggregations"</span>:{<span class="string">"sum"</span>:{<span class="string">"sum"</span>:{<span class="string">"field"</span>:<span class="string">"number"</span>}}},<span class="string">"filter"</span>:{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"LogType:itemsLog"</span>}}},<span class="string">"levelCount"</span>:{<span class="string">"filter"</span>:{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"LogType:scoreLog"</span>}}}},<span class="string">"composite"</span>:{<span class="string">"size"</span>:<span class="number">10000</span>,<span class="string">"sources"</span>:[{<span class="string">"actDay"</span>:{<span class="string">"date_histogram"</span>:{<span class="string">"calendar_interval"</span>:<span class="string">"1d"</span>,<span class="string">"field"</span>:<span class="string">"@timestamp"</span>,<span class="string">"time_zone"</span>:<span class="string">"Asia/Shanghai"</span>}}}]}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"*"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688054400</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1688745599</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// TestComposite_DateHistogram_Range</span></span><br><span class="line"><span class="variable constant_">GET</span> flower-business-*/_search</span><br><span class="line">{<span class="string">"aggs"</span>:{<span class="string">"aggs"</span>:{<span class="string">"aggregations"</span>:{<span class="string">"level"</span>:{<span class="string">"range"</span>:{<span class="string">"field"</span>:<span class="string">"topLevel"</span>,<span class="string">"ranges"</span>:[{<span class="string">"from"</span>:<span class="number">0</span>,<span class="string">"key"</span>:<span class="string">"0~500"</span>,<span class="string">"to"</span>:<span class="number">500</span>},{<span class="string">"from"</span>:<span class="number">501</span>,<span class="string">"key"</span>:<span class="string">"501~"</span>}]}}},<span class="string">"composite"</span>:{<span class="string">"size"</span>:<span class="number">10000</span>,<span class="string">"sources"</span>:[{<span class="string">"actDay"</span>:{<span class="string">"date_histogram"</span>:{<span class="string">"calendar_interval"</span>:<span class="string">"1d"</span>,<span class="string">"field"</span>:<span class="string">"@timestamp"</span>,<span class="string">"time_zone"</span>:<span class="string">"Asia/Shanghai"</span>}}}]}}},<span class="string">"query"</span>:{<span class="string">"bool"</span>:{<span class="string">"filter"</span>:[{<span class="string">"query_string"</span>:{<span class="string">"query"</span>:<span class="string">"LogType:loginLog"</span>}},{<span class="string">"range"</span>:{<span class="string">"@timestamp"</span>:{<span class="string">"format"</span>:<span class="string">"epoch_second"</span>,<span class="string">"from"</span>:<span class="number">1688054400</span>,<span class="string">"include_lower"</span>:<span class="literal">true</span>,<span class="string">"include_upper"</span>:<span class="literal">false</span>,<span class="string">"to"</span>:<span class="number">1688745599</span>}}}]}},<span class="string">"size"</span>:<span class="number">0</span>}</span><br></pre></td></tr></table></figure><h4 id="Go-Examples-3"><a href="#Go-Examples-3" class="headerlink" title="Go Examples"></a>Go Examples</h4><p><em><a href="https://github.com/ryan961/es-examples/blob/main/examples/search_aggs_composite_test.go">https://github.com/ryan961/es-examples/blob/main/examples/search_aggs_composite_test.go</a></em></p><h3 id="Update-By-Query"><a href="#Update-By-Query" class="headerlink" title="Update By Query"></a>Update By Query</h3><blockquote><p><em><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/docs-update-by-query.html">Update By Query API | Elasticsearch Guide [7.17] | Elastic</a></em></p></blockquote><h4 id="Kibana-Dev-Tools-4"><a href="#Kibana-Dev-Tools-4" class="headerlink" title="Kibana Dev Tools"></a>Kibana Dev Tools</h4><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="variable constant_">POST</span> flower-calcmeta/_update_by_query?ignore_unavailable=<span class="literal">true</span>&refresh=<span class="literal">true</span></span><br><span class="line">{</span><br><span class="line"> <span class="string">"query"</span>: {</span><br><span class="line"> <span class="string">"query_string"</span>: {</span><br><span class="line"> <span class="string">"query"</span>: <span class="string">"ad_id:6717889215 AND register_time:1677600000"</span></span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"script"</span>: {</span><br><span class="line"> <span class="string">"lang"</span>: <span class="string">"painless"</span>,</span><br><span class="line"> <span class="string">"source"</span>: <span class="string">""</span><span class="string">"ctx._source["</span>show<span class="string">"]=params["</span>show<span class="string">"];ctx._source["</span>active<span class="string">"]=params["</span>active<span class="string">"];ctx._source["</span>register<span class="string">"]=params["</span>register<span class="string">"];ctx._source["</span>register_active<span class="string">"]=params["</span>register_active<span class="string">"];ctx._source["</span>cost_active<span class="string">"]=params["</span>cost_active<span class="string">"];ctx._source["</span>cost_register<span class="string">"]=params["</span>cost_register<span class="string">"];"</span><span class="string">""</span>,</span><br><span class="line"> <span class="string">"params"</span>: {</span><br><span class="line"> <span class="string">"show"</span>: <span class="number">3914</span>,</span><br><span class="line"> <span class="string">"active"</span>: <span class="number">6</span>,</span><br><span class="line"> <span class="string">"register"</span>: <span class="number">1</span>,</span><br><span class="line"> <span class="string">"register_active"</span>: <span class="number">0.1667</span>,</span><br><span class="line"> <span class="string">"cost_active"</span>: <span class="number">5.53</span>,</span><br><span class="line"> <span class="string">"cost_register"</span>: <span class="number">33.18</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="Go-Examples-4"><a href="#Go-Examples-4" class="headerlink" title="Go Examples"></a>Go Examples</h4><p><em><a href="https://github.com/ryan961/es-examples/blob/main/gdt_fix_active/main.go">https://github.com/ryan961/es-examples/blob/main/gdt_fix_active/main.go</a></em></p><h3 id="Delete-By-Query"><a href="#Delete-By-Query" class="headerlink" title="Delete By Query"></a>Delete By Query</h3><blockquote><p><em><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/docs-delete-by-query.html">Delete by query API | Elasticsearch Guide [7.17] | Elastic</a></em></p></blockquote><h4 id="Kibana-Dev-Tools-5"><a href="#Kibana-Dev-Tools-5" class="headerlink" title="Kibana Dev Tools"></a>Kibana Dev Tools</h4><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="variable constant_">POST</span> flower-calcmeta/_delete_by_query</span><br><span class="line">{</span><br><span class="line"> <span class="string">"query"</span>: {</span><br><span class="line"> <span class="string">"query_string"</span>: {</span><br><span class="line"> <span class="string">"query"</span>: <span class="string">"ad_id:6717889215 AND register_time:1677600000"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="Go-Examples-5"><a href="#Go-Examples-5" class="headerlink" title="Go Examples"></a>Go Examples</h4><p><em><a href="https://github.com/ryan961/es-examples/blob/main/examples/deleteByQuery_test.go">https://github.com/ryan961/es-examples/blob/main/examples/deleteByQuery_test.go</a></em></p><h3 id="Scroll"><a href="#Scroll" class="headerlink" title="Scroll"></a>Scroll</h3><blockquote><p><em><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/scroll-api.html">Scroll API | Elasticsearch Guide [7.17] | Elastic</a></em></p></blockquote><h4 id="Go-Examples-6"><a href="#Go-Examples-6" class="headerlink" title="Go Examples"></a>Go Examples</h4><p><em><a href="https://github.com/ryan961/es-examples/blob/main/examples/scroll_test.go">https://github.com/ryan961/es-examples/blob/main/examples/scroll_test.go</a></em></p><h3 id="Script-Fields"><a href="#Script-Fields" class="headerlink" title="Script Fields"></a>Script Fields</h3><blockquote><p><em><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/search-fields.html#script-fields">Retrieve selected fields from a search | Elasticsearch Guide [7.17] | Elastic</a></em></p></blockquote><p>Example: Use scripted fields in Kibana to add day_x (days since unlock) field for unlock metrics.<br>Add path: Stack Management > Index Patterns > flower-business-* > Scripted fields > Add scripted field</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> (doc[<span class="string">"msg.code"</span>].<span class="title function_">size</span>() != <span class="number">0</span> && doc[<span class="string">"msg.code"</span>].<span class="property">value</span> == <span class="number">1040</span> && doc[<span class="string">"msg.eventName"</span>].<span class="property">value</span> == <span class="string">"unlockInstanceMap"</span>){</span><br><span class="line"> def now_ts = <span class="keyword">new</span> <span class="title class_">Date</span>().<span class="title function_">getTime</span>();</span><br><span class="line"> def now_inst = <span class="title class_">Instant</span>.<span class="title function_">ofEpochMilli</span>(now_ts);</span><br><span class="line"> <span class="title class_">ZonedDateTime</span> now = <span class="title class_">ZonedDateTime</span>.<span class="title function_">ofInstant</span>(now_inst,<span class="title class_">ZoneId</span>.<span class="title function_">of</span>(<span class="string">'Z'</span>));</span><br><span class="line"> <span class="title class_">ZonedDateTime</span> lockTime = <span class="title class_">ZonedDateTime</span>.<span class="title function_">ofInstant</span>(doc[<span class="string">'@timestamp'</span>].<span class="property">value</span>.<span class="title function_">toInstant</span>(), <span class="title class_">ZoneId</span>.<span class="title function_">of</span>(<span class="string">'Z'</span>));</span><br><span class="line"> def day = <span class="title class_">ChronoUnit</span>.<span class="property">DAYS</span>.<span class="title function_">between</span>(lockTime, now);</span><br><span class="line"> <span class="keyword">return</span> day</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="Kibana-Dev-Tools-6"><a href="#Kibana-Dev-Tools-6" class="headerlink" title="Kibana Dev Tools"></a>Kibana Dev Tools</h4><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line"><span class="variable constant_">POST</span> test/_doc</span><br><span class="line">{</span><br><span class="line"> <span class="string">"msg"</span>:{</span><br><span class="line"> <span class="string">"actionType"</span>:<span class="string">"compositeMerge"</span>,</span><br><span class="line"> <span class="string">"code"</span>:<span class="number">1040</span>,</span><br><span class="line"> <span class="string">"eventName"</span>:<span class="string">"merge"</span></span><br><span class="line"> },</span><br><span class="line"> <span class="string">"islandPassData"</span>:{</span><br><span class="line"> <span class="string">"activityId"</span>:<span class="number">1</span>,</span><br><span class="line"> <span class="string">"realOpenTime"</span>:<span class="number">1672023242702</span></span><br><span class="line"> },</span><br><span class="line"> <span class="string">"deviceId"</span>:<span class="string">"00000000-23c8-1dd0-0000-00005510f0e3"</span>,</span><br><span class="line"> <span class="string">"uuid"</span>:<span class="string">"4053c58e-7849-11ed-aa83-02734471da14"</span>,</span><br><span class="line"> <span class="string">"registerTime"</span>:<span class="number">1670649153</span>,</span><br><span class="line"> <span class="string">"unlockSkinAt"</span>:<span class="number">1671368118</span>,</span><br><span class="line"> <span class="string">"customerId"</span>:<span class="number">25851790</span>,</span><br><span class="line"> <span class="string">"@timestamp"</span>:<span class="string">"2023-03-07T21:09:03.000Z"</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="variable constant_">GET</span> test/_doc/mHBV3oYBGotJ2rqXyozh</span><br><span class="line">{}</span><br><span class="line"></span><br><span class="line"><span class="variable constant_">GET</span> test/_search</span><br><span class="line">{</span><br><span class="line"> <span class="string">"query"</span>: {</span><br><span class="line"> <span class="string">"match_all"</span>: {}</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"script_fields"</span>: {</span><br><span class="line"> <span class="string">"day_x"</span>: {</span><br><span class="line"> <span class="string">"script"</span>: {</span><br><span class="line"> <span class="string">"lang"</span>: <span class="string">"painless"</span>,</span><br><span class="line"> <span class="string">"source"</span>: <span class="string">""</span><span class="string">"</span></span><br><span class="line"><span class="string"> def now_ts = new Date().getTime();</span></span><br><span class="line"><span class="string"> def now_inst = Instant.ofEpochMilli(now_ts);</span></span><br><span class="line"><span class="string"> ZonedDateTime now = ZonedDateTime.ofInstant(now_inst,ZoneId.of('Z'));</span></span><br><span class="line"><span class="string"> ZonedDateTime lockTime = ZonedDateTime.ofInstant(Instant.ofEpochMilli(doc['islandPassData.realOpenTime'].value), ZoneId.of('Z'));</span></span><br><span class="line"><span class="string"> def day = ChronoUnit.DAYS.between(lockTime, now);</span></span><br><span class="line"><span class="string"> return day</span></span><br><span class="line"><span class="string"> "</span><span class="string">""</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="Script-Functions"><a href="#Script-Functions" class="headerlink" title="Script Functions"></a>Script Functions</h4><ul><li><a href="https://www.elastic.co/guide/en/elasticsearch/painless/master/painless-api-reference-shared-org-elasticsearch-script.html#painless-api-reference-shared-org-elasticsearch-script">Shared API for package org.elasticsearch.script | Painless Scripting Language [master] | Elastic</a></li><li><a href="https://www.elastic.co/guide/en/elasticsearch/painless/master/painless-api-reference-shared-java-time.html">Shared API for package java.time | Painless Scripting Language [master] | Elastic</a></li><li><a href="https://www.elastic.co/guide/en/elasticsearch/painless/master/painless-datetime.html">Using Datetime in Painless | Painless Scripting Language [master] | Elastic</a></li></ul><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// 将时间戳(s)转化成 ZonedDateTime 类型</span></span><br><span class="line"><span class="title class_">ZonedDateTime</span> lockTime = <span class="title class_">ZonedDateTime</span>.<span class="title function_">ofInstant</span>(<span class="title class_">Instant</span>.<span class="title function_">ofEpochSecond</span>(doc[<span class="string">'unlockSkinAt'</span>].<span class="property">value</span>), <span class="title class_">ZoneId</span>.<span class="title function_">of</span>(<span class="string">'Z'</span>));</span><br><span class="line"></span><br><span class="line"><span class="comment">// ms</span></span><br><span class="line"><span class="title class_">ZonedDateTime</span> lockTime = <span class="title class_">ZonedDateTime</span>.<span class="title function_">ofInstant</span>(<span class="title class_">Instant</span>.<span class="title function_">ofEpochMilli</span>(doc[<span class="string">'islandPassData.realOpenTime'</span>].<span class="property">value</span>), <span class="title class_">ZoneId</span>.<span class="title function_">of</span>(<span class="string">'Z'</span>));</span><br><span class="line"></span><br><span class="line"><span class="title class_">ZonedDateTime</span> lockTime = <span class="title class_">ZonedDateTime</span>.<span class="title function_">ofInstant</span>(doc[<span class="string">'@timestamp'</span>].<span class="property">value</span>.<span class="title function_">toInstant</span>(), <span class="title class_">ZoneId</span>.<span class="title function_">of</span>(<span class="string">'Z'</span>));</span><br><span class="line"></span><br><span class="line"><span class="title class_">ZonedDateTime</span> now = <span class="title class_">ZonedDateTime</span>.<span class="title function_">ofInstant</span>(<span class="title class_">Instant</span>.<span class="title function_">now</span>().<span class="title function_">toEpochMilli</span>(),<span class="title class_">ZoneId</span>.<span class="title function_">of</span>(<span class="string">'Z'</span>));</span><br></pre></td></tr></table></figure><p><strong><em>More to come…</em></strong> :)</p>]]></content>
<summary type="html"><h2 id="Environment"><a</summary>
<category term="golang" scheme="https://ryan961.github.io/categories/golang/"/>
<category term="elasticsearch" scheme="https://ryan961.github.io/categories/golang/elasticsearch/"/>
<category term="golang" scheme="https://ryan961.github.io/tags/golang/"/>
<category term="elasticsearch" scheme="https://ryan961.github.io/tags/elasticsearch/"/>
</entry>
<entry>
<title>Cleaning Up .zsh_history File with a Bash Script</title>
<link href="https://ryan961.github.io/2023/08/24/history-cleaner/"/>
<id>https://ryan961.github.io/2023/08/24/history-cleaner/</id>
<published>2023-08-24T16:00:00.000Z</published>
<updated>2023-08-24T16:00:00.000Z</updated>
<content type="html"><![CDATA[<h1 id="history-cleaner-sh"><a href="#history-cleaner-sh" class="headerlink" title="history_cleaner.sh"></a>history_cleaner.sh</h1><blockquote><p><a href="https://gist.github.com/ryan961/2baff64d162a946158405bb59bde1358">https://gist.github.com/ryan961/2baff64d162a946158405bb59bde1358</a></p></blockquote><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/bin/bash</span></span><br><span class="line"></span><br><span class="line">HISTFILE=<span class="variable">${HISTFILE:-"$HOME/.zsh_history"}</span></span><br><span class="line"></span><br><span class="line">verbose=<span class="literal">false</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="title">print_usage</span></span>() {</span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"Usage: <span class="variable">$0</span> [-f HISTFILE] [-n line1,line2,...] [-o offset] [-r regexp] [-e num] [-v]"</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"NOTE: -n, -r and -e options cannot be used together."</span></span><br><span class="line"> <span class="built_in">echo</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"Delete specific lines from .zsh_history file."</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">" -f HISTFILE Specify history file. Default is <span class="variable">$HISTFILE</span>."</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">" -n line1,line2,... Delete specified lines. Line number starts from 1."</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">" -o offset Delete lines before(-) or after(+) lines specified by -n."</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">" -r regexp Delete lines match regexp pattern."</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">" -e num Delete last num lines."</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">" -v Verbose output."</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">" -h Print this help."</span></span><br><span class="line"> <span class="built_in">echo</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># Parse flag</span></span><br><span class="line"><span class="keyword">while</span> <span class="built_in">getopts</span> <span class="string">":f:n:o:r:e:vh"</span> opt; <span class="keyword">do</span></span><br><span class="line"> <span class="keyword">case</span> <span class="variable">$opt</span> <span class="keyword">in</span></span><br><span class="line"> f) HISTFILE=<span class="variable">$OPTARG</span> ;;</span><br><span class="line"> n) lines_string=<span class="variable">$OPTARG</span> ;;</span><br><span class="line"> o) offset=<span class="variable">$OPTARG</span> ;;</span><br><span class="line"> r) regexp=<span class="variable">$OPTARG</span> ;;</span><br><span class="line"> e) last_lines=<span class="variable">$OPTARG</span> ;;</span><br><span class="line"> v) verbose=<span class="literal">true</span> ;;</span><br><span class="line"> h)</span><br><span class="line"> print_usage</span><br><span class="line"> <span class="built_in">exit</span> 0</span><br><span class="line"> ;;</span><br><span class="line"> \?)</span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"Invalid option: -<span class="variable">$OPTARG</span>"</span> >&2</span><br><span class="line"> print_usage</span><br><span class="line"> <span class="built_in">exit</span> 1</span><br><span class="line"> ;;</span><br><span class="line"> <span class="keyword">esac</span></span><br><span class="line"><span class="keyword">done</span></span><br><span class="line">IFS=<span class="string">','</span> <span class="built_in">read</span> -ra lines <<<<span class="string">"<span class="variable">$lines_string</span>"</span></span><br><span class="line"></span><br><span class="line">[[ <span class="variable">$verbose</span> == <span class="literal">true</span> ]] && <span class="built_in">echo</span> <span class="string">"Deleting from file: <span class="variable">$HISTFILE</span>"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Check HISTFILE exists</span></span><br><span class="line"><span class="keyword">if</span> [[ ! -f <span class="variable">$HISTFILE</span> ]]; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"History file <span class="variable">$HISTFILE</span> does not exist"</span> >&2</span><br><span class="line"> <span class="built_in">exit</span> 1</span><br><span class="line"><span class="keyword">fi</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Check be used together</span></span><br><span class="line"><span class="keyword">if</span> [[ (<span class="variable">${#lines[@]}</span> -gt 0 && (-n <span class="variable">$regexp</span> || -n <span class="variable">$last_lines</span>)) || (-n <span class="variable">$regexp</span> && -n <span class="variable">$last_lines</span>) ]]; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"Options -n, -r and -e cannot be used together"</span> >&2</span><br><span class="line"> <span class="built_in">exit</span> 1</span><br><span class="line"><span class="keyword">fi</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Verify input offset</span></span><br><span class="line"><span class="keyword">if</span> [[ -n <span class="variable">$offset</span> && <span class="variable">$offset</span> != 0 ]]; <span class="keyword">then</span></span><br><span class="line"> <span class="keyword">if</span> [[ <span class="variable">${#lines[@]}</span> == 0 ]]; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"You must specify line number with -n when using offset with -o."</span> >&2</span><br><span class="line"> <span class="built_in">exit</span> 1</span><br><span class="line"> <span class="keyword">fi</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> [[ <span class="variable">${#lines[@]}</span> -gt 1 ]]; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"You specify line number with -n (limit 1) when using offset with -o."</span> >&2</span><br><span class="line"> <span class="built_in">exit</span> 1</span><br><span class="line"> <span class="keyword">fi</span></span><br><span class="line"><span class="keyword">fi</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Delete specified lines</span></span><br><span class="line"><span class="keyword">if</span> [[ <span class="variable">${#lines[@]}</span> -gt 0 ]]; <span class="keyword">then</span></span><br><span class="line"> <span class="keyword">if</span> [[ -z <span class="variable">$offset</span> || <span class="variable">$offset</span> == 0 ]]; <span class="keyword">then</span> <span class="comment"># Delete specified lines</span></span><br><span class="line"> [[ <span class="variable">$verbose</span> == <span class="literal">true</span> ]] && <span class="built_in">echo</span> <span class="string">"Deleting lines: <span class="variable">${lines[*]}</span>"</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Verify line exist</span></span><br><span class="line"> <span class="keyword">for</span> line <span class="keyword">in</span> <span class="string">"<span class="variable">${lines[@]}</span>"</span>; <span class="keyword">do</span></span><br><span class="line"> <span class="keyword">if</span> [[ $(<span class="built_in">wc</span> -l <<span class="string">"<span class="variable">$HISTFILE</span>"</span>) -lt <span class="variable">$line</span> ]]; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"Line <span class="variable">$line</span> does not exist"</span> >&2</span><br><span class="line"> <span class="built_in">exit</span> 1</span><br><span class="line"> <span class="keyword">fi</span></span><br><span class="line"> <span class="keyword">done</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> ((index = <span class="number">0</span>; index < <span class="variable">${#lines[@]}</span>; index++)); <span class="keyword">do</span></span><br><span class="line"> line=$((lines[index] - index))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># Debug line content</span></span><br><span class="line"> delete_line=$(sed -n <span class="string">"<span class="variable">${line}</span>p"</span> <span class="string">"<span class="variable">$HISTFILE</span>"</span>)</span><br><span class="line"> [[ <span class="variable">$verbose</span> == <span class="literal">true</span> ]] && <span class="built_in">echo</span> <span class="string">"Deleting line [<span class="variable">${line}</span>]: <span class="variable">$delete_line</span>"</span></span><br><span class="line"></span><br><span class="line"> awk <span class="string">'NR != '</span><span class="variable">$line</span><span class="string">' {print}'</span> <span class="string">"<span class="variable">$HISTFILE</span>"</span> >tmp && <span class="built_in">mv</span> tmp <span class="string">"<span class="variable">$HISTFILE</span>"</span></span><br><span class="line"> <span class="keyword">done</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">else</span> <span class="comment"># Delete lines in range</span></span><br><span class="line"> <span class="keyword">if</span> [[ <span class="variable">$offset</span> -gt 0 ]]; <span class="keyword">then</span></span><br><span class="line"> first=<span class="variable">${lines[0]}</span></span><br><span class="line"> last=$((first + offset))</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> first=$((lines[<span class="number">0</span>] + offset))</span><br><span class="line"> last=<span class="variable">${lines[0]}</span></span><br><span class="line"> <span class="keyword">fi</span></span><br><span class="line"> [[ <span class="variable">$verbose</span> == <span class="literal">true</span> ]] && <span class="built_in">echo</span> <span class="string">"Deleting range: <span class="variable">$first</span> to <span class="variable">$last</span>"</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> ((i = first; i <= last; i++)); <span class="keyword">do</span></span><br><span class="line"> <span class="comment"># Debug line content</span></span><br><span class="line"> delete_line=$(sed -n <span class="string">"<span class="variable">${i}</span>p"</span> <span class="string">"<span class="variable">$HISTFILE</span>"</span>)</span><br><span class="line"> [[ <span class="variable">$verbose</span> == <span class="literal">true</span> ]] && <span class="built_in">echo</span> <span class="string">"Deleting line [<span class="variable">${i}</span>]: <span class="variable">$delete_line</span>"</span></span><br><span class="line"> <span class="keyword">done</span></span><br><span class="line"></span><br><span class="line"> awk <span class="string">'NR < '</span><span class="string">"<span class="variable">$first</span>"</span><span class="string">' || NR > '</span><span class="string">"<span class="variable">$last</span>"</span><span class="string">' {print}'</span> <span class="string">"<span class="variable">$HISTFILE</span>"</span> >tmp && <span class="built_in">mv</span> tmp <span class="string">"<span class="variable">$HISTFILE</span>"</span></span><br><span class="line"> <span class="keyword">fi</span></span><br><span class="line"> <span class="built_in">exit</span> 0</span><br><span class="line"><span class="keyword">fi</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Delete last n lines</span></span><br><span class="line"><span class="keyword">if</span> [[ -n <span class="variable">$last_lines</span> ]]; <span class="keyword">then</span></span><br><span class="line"> [[ <span class="variable">$verbose</span> == <span class="literal">true</span> ]] && <span class="built_in">echo</span> <span class="string">"Deleting last <span class="variable">$last_lines</span> lines"</span></span><br><span class="line"></span><br><span class="line"> total_lines=$(<span class="built_in">wc</span> -l <<span class="string">"<span class="variable">$HISTFILE</span>"</span>)</span><br><span class="line"> first=$((total_lines - last_lines))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> ((i = first; i < total_lines; i++)); <span class="keyword">do</span></span><br><span class="line"> delete_line=$(sed -n <span class="string">"<span class="variable">${i}</span>p"</span> <span class="string">"<span class="variable">$HISTFILE</span>"</span>)</span><br><span class="line"> [[ <span class="variable">$verbose</span> == <span class="literal">true</span> ]] && <span class="built_in">echo</span> <span class="string">"Deleting line [<span class="variable">${i}</span>]: <span class="variable">$delete_line</span>"</span></span><br><span class="line"> <span class="keyword">done</span></span><br><span class="line"></span><br><span class="line"> awk <span class="string">'NR < '</span><span class="string">"<span class="variable">$first</span>"</span><span class="string">' || NR > '</span><span class="string">"<span class="subst">$((total_lines-1)</span>)"</span><span class="string">' {print}'</span> <span class="string">"<span class="variable">$HISTFILE</span>"</span> >tmp && <span class="built_in">mv</span> tmp <span class="string">"<span class="variable">$HISTFILE</span>"</span></span><br><span class="line"> <span class="built_in">exit</span> 0</span><br><span class="line"><span class="keyword">fi</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Delete by regex</span></span><br><span class="line"><span class="keyword">if</span> [[ -n <span class="variable">$regexp</span> ]]; <span class="keyword">then</span></span><br><span class="line"> matches=$(grep -n -E <span class="string">"<span class="variable">$regexp</span>"</span> <span class="string">"<span class="variable">$HISTFILE</span>"</span> | <span class="built_in">cut</span> -d: -f1)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># shellcheck disable=SC2086</span></span><br><span class="line"> [[ <span class="variable">$verbose</span> == <span class="literal">true</span> ]] && <span class="built_in">echo</span> <span class="string">"Deleting lines match '<span class="variable">$regexp</span>':"</span> <span class="variable">$matches</span></span><br><span class="line"></span><br><span class="line"> i=0</span><br><span class="line"> <span class="keyword">for</span> line_number <span class="keyword">in</span> <span class="variable">$matches</span>; <span class="keyword">do</span></span><br><span class="line"> line=$((line_number - i))</span><br><span class="line"></span><br><span class="line"> delete_line=$(sed -n <span class="string">"<span class="variable">${line}</span>p"</span> <span class="string">"<span class="variable">$HISTFILE</span>"</span>)</span><br><span class="line"> [[ <span class="variable">$verbose</span> == <span class="literal">true</span> ]] && <span class="built_in">echo</span> <span class="string">"Deleting line [<span class="variable">$line_number</span>]: <span class="variable">$delete_line</span>"</span></span><br><span class="line"></span><br><span class="line"> awk <span class="string">'NR != '</span><span class="variable">$line</span><span class="string">' {print}'</span> <span class="string">"<span class="variable">$HISTFILE</span>"</span> >tmp && <span class="built_in">mv</span> tmp <span class="string">"<span class="variable">$HISTFILE</span>"</span></span><br><span class="line"></span><br><span class="line"> i=$((i + <span class="number">1</span>))</span><br><span class="line"> <span class="keyword">done</span></span><br><span class="line"> <span class="built_in">exit</span> 0</span><br><span class="line"><span class="keyword">fi</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Default print usage</span></span><br><span class="line"><span class="keyword">if</span> [[ <span class="variable">$#</span> -eq 0 ]]; <span class="keyword">then</span></span><br><span class="line"> print_usage</span><br><span class="line"><span class="keyword">fi</span></span><br></pre></td></tr></table></figure><p>This is a bash script to delete specific lines from <code>$HISTFILE</code> ( .zsh_history ) file.</p><p>It provides several options to specify which lines to delete:</p><ul><li>-n: Delete specified line numbers</li><li>-o: Delete a range of lines, used together with -n</li><li>-r: Delete lines matching a regular expression pattern</li><li>-e: Delete last N lines</li></ul><p>The script also supports:</p><ul><li>Specifying a different history file with -f</li><li>Print verbose output with -v</li><li>Print help message with -h</li></ul><p>The implementation uses awk to filter out target lines and redirect to a tmp file, then replace original file.</p><p>Some notes on usage:</p><ul><li>Line numbers start from 1</li><li>-n, -r and -e options cannot be used together</li></ul><p>This provides a flexible way to clean up .zsh_history by deleting unwanted lines.</p><p>Please let me know if you would like me to modify or expand the introduction.</p>]]></content>
<summary type="html"><h1 id="history-cleaner-sh"><a</summary>
<category term="shell" scheme="https://ryan961.github.io/categories/shell/"/>
<category term="shell" scheme="https://ryan961.github.io/tags/shell/"/>
<category term="linux" scheme="https://ryan961.github.io/tags/linux/"/>
<category term="mac" scheme="https://ryan961.github.io/tags/mac/"/>
</entry>
<entry>
<title>Kubernetes 集群搭建</title>
<link href="https://ryan961.github.io/2022/04/19/k8s-install/"/>
<id>https://ryan961.github.io/2022/04/19/k8s-install/</id>
<published>2022-04-19T16:00:00.000Z</published>
<updated>2022-04-19T16:00:00.000Z</updated>
<content type="html"><![CDATA[<span id="more"></span><h1 id="环境准备"><a href="#环境准备" class="headerlink" title="环境准备"></a>环境准备</h1><h2 id="主机"><a href="#主机" class="headerlink" title="主机"></a>主机</h2><blockquote><p>裸机安装至少需要两台机器（主节点、工作节点各一台)</p></blockquote><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">// System OS: Ubuntu 20.04.4 LTS</span><br><span class="line">k8s-01: ubuntu@172.16.20.33</span><br><span class="line">k8s-02: ubuntu@172.16.20.34</span><br><span class="line">k8s-03: ubuntu@172.16.20.35</span><br></pre></td></tr></table></figure><p>批量将本机的 pub key 复制到目标机器的 <code>~/.ssh/authorized_keys</code> :</p><p><code>copyKey</code></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/usr/bin/expect -f</span></span><br><span class="line"></span><br><span class="line"><span class="built_in">set</span> ip [lindex <span class="variable">$argv</span> 0]</span><br><span class="line"><span class="built_in">set</span> user [lindex <span class="variable">$argv</span> 1]</span><br><span class="line"><span class="built_in">set</span> password [lindex <span class="variable">$argv</span> 2]</span><br><span class="line"><span class="built_in">set</span> run 1</span><br><span class="line"><span class="built_in">set</span> <span class="built_in">timeout</span> 60</span><br><span class="line"></span><br><span class="line">spawn ssh-copy-id <span class="variable">$user</span>@<span class="variable">$ip</span></span><br><span class="line">expect {</span><br><span class="line"> <span class="string">"yes/no"</span> {</span><br><span class="line"> send <span class="string">"yes\r"</span></span><br><span class="line"> exp_continue</span><br><span class="line"> }</span><br><span class="line"> <span class="string">"password:"</span> {</span><br><span class="line"> send <span class="string">"<span class="variable">$password</span>\r"</span></span><br><span class="line"> exp_continue</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><code>copyKey.sh</code></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/bin/bash</span></span><br><span class="line"></span><br><span class="line">ips=<span class="string">"172.16.20.33 172.16.20.34 172.16.20.35"</span></span><br><span class="line">user=<span class="string">"ubuntu"</span></span><br><span class="line">password=<span class="string">"*"</span></span><br><span class="line">home_path=<span class="string">"/Users/artist"</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> [ ! -f <span class="variable">$home_path</span><span class="string">"/.ssh/id_rsa.pub"</span> ]; <span class="keyword">then</span></span><br><span class="line"> ssh-keygen -b 2048 -t rsa -f <span class="variable">$home_path</span><span class="string">"/.ssh/id_rsa"</span> -q -N <span class="string">""</span></span><br><span class="line"> <span class="comment">#cat $home_path"/.ssh/id_rsa.pub" >>$home_path"/.ssh/authorized_keys"</span></span><br><span class="line"><span class="keyword">fi</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> n <span class="keyword">in</span> <span class="variable">$ips</span>; <span class="keyword">do</span></span><br><span class="line"> ./copyKey <span class="string">"<span class="variable">$n</span>"</span> <span class="variable">$user</span> <span class="variable">$password</span></span><br><span class="line"> ssh <span class="variable">$user</span>@<span class="string">"<span class="variable">$n</span>"</span> <span class="string">"hostname -i"</span></span><br><span class="line"><span class="keyword">done</span></span><br><span class="line"></span><br></pre></td></tr></table></figure><h2 id="关闭-swap-内存"><a href="#关闭-swap-内存" class="headerlink" title="关闭 swap 内存"></a>关闭 swap 内存</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># k8s 较新版本都要求关闭 swap</span></span><br><span class="line">swapoff -a</span><br><span class="line"><span class="comment"># 重启</span></span><br><span class="line">reboot</span><br></pre></td></tr></table></figure><h2 id="每个节点分别设置主机名"><a href="#每个节点分别设置主机名" class="headerlink" title="每个节点分别设置主机名"></a>每个节点分别设置主机名</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 172.16.20.33</span></span><br><span class="line">hostnamectl set-hostname master</span><br><span class="line"><span class="comment"># 172.16.20.34</span></span><br><span class="line">hostnamectl set-hostname node1</span><br><span class="line"><span class="comment"># 172.16.20.35</span></span><br><span class="line">hostnamectl set-hostname node2</span><br></pre></td></tr></table></figure><h2 id="配置-hosts"><a href="#配置-hosts" class="headerlink" title="配置 hosts"></a>配置 hosts</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 所有节点都修改 hosts</span></span><br><span class="line"><span class="built_in">cat</span> <<<span class="string">EOF > /etc/hosts</span></span><br><span class="line"><span class="string">172.16.20.33 master</span></span><br><span class="line"><span class="string">172.16.20.34 node1</span></span><br><span class="line"><span class="string">172.16.20.35 node2</span></span><br><span class="line"><span class="string">EOF</span></span><br></pre></td></tr></table></figure><h2 id="设置时区"><a href="#设置时区" class="headerlink" title="设置时区"></a>设置时区</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">timedatectl set-timezone Asia/Shanghai</span><br><span class="line"><span class="comment"># 同时使系统日志时间戳也立即生效</span></span><br><span class="line">systemctl restart rsyslog</span><br></pre></td></tr></table></figure><h2 id="关闭防火墙和-selinux"><a href="#关闭防火墙和-selinux" class="headerlink" title="关闭防火墙和 selinux"></a>关闭防火墙和 selinux</h2><p>ubuntu 查看防火墙命令，ufw status 可查看状态，ubuntu20.04 默认全部关闭，无需设置。</p><h2 id="加载-br-netfilter-模块"><a href="#加载-br-netfilter-模块" class="headerlink" title="加载 br_netfilter 模块"></a>加载 br_netfilter 模块</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">cat</span> <<<span class="string">EOF > /etc/modules-load.d/k8s.conf</span></span><br><span class="line"><span class="string">overlay</span></span><br><span class="line"><span class="string">br_netfilter</span></span><br><span class="line"><span class="string">EOF</span></span><br><span class="line"></span><br><span class="line">modprobe overlay</span><br><span class="line">modprobe br_netfilter</span><br><span class="line"></span><br><span class="line"><span class="comment"># 查看 br_netfilter 模块是否已加载</span></span><br><span class="line">lsmod | grep br_netfilter</span><br></pre></td></tr></table></figure><h2 id="系统参数调整"><a href="#系统参数调整" class="headerlink" title="系统参数调整"></a>系统参数调整</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">cat</span> <<<span class="string">EOF > /etc/sysctl.d/k8s.conf</span></span><br><span class="line"><span class="string"># https://github.com/moby/moby/issues/31208</span></span><br><span class="line"><span class="string"># ipvsadm -l --timout</span></span><br><span class="line"><span class="string"># 修复ipvs模式下长连接timeout问题 小于900即可</span></span><br><span class="line"><span class="string">net.ipv4.tcp_keepalive_time = 600</span></span><br><span class="line"><span class="string">net.ipv4.tcp_keepalive_intvl = 30</span></span><br><span class="line"><span class="string">net.ipv4.tcp_keepalive_probes = 10</span></span><br><span class="line"><span class="string">net.ipv6.conf.all.disable_ipv6 = 1</span></span><br><span class="line"><span class="string">net.ipv6.conf.default.disable_ipv6 = 1</span></span><br><span class="line"><span class="string">net.ipv6.conf.lo.disable_ipv6 = 1</span></span><br><span class="line"><span class="string">net.ipv4.neigh.default.gc_stale_time = 120</span></span><br><span class="line"><span class="string">net.ipv4.conf.all.rp_filter = 0</span></span><br><span class="line"><span class="string">net.ipv4.conf.default.rp_filter = 0</span></span><br><span class="line"><span class="string">net.ipv4.conf.default.arp_announce = 2</span></span><br><span class="line"><span class="string">net.ipv4.conf.lo.arp_announce = 2</span></span><br><span class="line"><span class="string">net.ipv4.conf.all.arp_announce = 2</span></span><br><span class="line"><span class="string">net.ipv4.ip_forward = 1</span></span><br><span class="line"><span class="string">net.ipv4.tcp_max_tw_buckets = 5000</span></span><br><span class="line"><span class="string">net.ipv4.tcp_syncookies = 1</span></span><br><span class="line"><span class="string">net.ipv4.tcp_max_syn_backlog = 1024</span></span><br><span class="line"><span class="string">net.ipv4.tcp_synack_retries = 2</span></span><br><span class="line"><span class="string"># 要求iptables不对bridge的数据进行处理</span></span><br><span class="line"><span class="string">net.bridge.bridge-nf-call-ip6tables = 1</span></span><br><span class="line"><span class="string">net.bridge.bridge-nf-call-iptables = 1</span></span><br><span class="line"><span class="string">net.bridge.bridge-nf-call-arptables = 1</span></span><br><span class="line"><span class="string">net.netfilter.nf_conntrack_max = 2310720</span></span><br><span class="line"><span class="string">fs.inotify.max_user_watches=89100</span></span><br><span class="line"><span class="string">fs.may_detach_mounts = 1</span></span><br><span class="line"><span class="string">fs.file-max = 52706963</span></span><br><span class="line"><span class="string">fs.nr_open = 52706963</span></span><br><span class="line"><span class="string">vm.swappiness = 0</span></span><br><span class="line"><span class="string">vm.overcommit_memory=1</span></span><br><span class="line"><span class="string">vm.panic_on_oom=0</span></span><br><span class="line"><span class="string">EOF</span></span><br><span class="line"></span><br><span class="line">sysctl --system</span><br></pre></td></tr></table></figure><h2 id="启用-ipvs"><a href="#启用-ipvs" class="headerlink" title="启用 ipvs"></a>启用 ipvs</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#安装ipset及ipvsadm</span></span><br><span class="line">apt-get -y install ipset ipvsadm</span><br><span class="line"></span><br><span class="line"><span class="comment">#创建sysconfig/modules文件夹</span></span><br><span class="line"><span class="built_in">mkdir</span> -p /etc/sysconfig/modules/</span><br><span class="line"></span><br><span class="line"><span class="comment">#操作如下命令</span></span><br><span class="line"><span class="built_in">cat</span> <<<span class="string">EOF >/etc/sysconfig/modules/ipvs.modules</span></span><br><span class="line"><span class="string">#!/bin/bash</span></span><br><span class="line"><span class="string">modprobe -- ip_vs</span></span><br><span class="line"><span class="string">modprobe -- ip_vs_rr</span></span><br><span class="line"><span class="string">modprobe -- ip_vs_wrr</span></span><br><span class="line"><span class="string">modprobe -- ip_vs_sh</span></span><br><span class="line"><span class="string">modprobe -- nf_conntrack</span></span><br><span class="line"><span class="string">EOF</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 授权、运行、检查是否加载</span></span><br><span class="line"><span class="built_in">chmod</span> 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack</span><br></pre></td></tr></table></figure><h2 id="Master-主节点所需组件"><a href="#Master-主节点所需组件" class="headerlink" title="Master 主节点所需组件"></a>Master 主节点所需组件</h2><ul><li>containerd</li><li>kubectl 集群命令行交互工具</li><li>kubeadm 集群初始化工具</li></ul><h2 id="Node-工作节点需要组件"><a href="#Node-工作节点需要组件" class="headerlink" title="Node 工作节点需要组件"></a>Node 工作节点需要组件</h2><ul><li>containerd</li><li>kubelet 管理 Pod 和容器，确保他们健康稳定运行。</li><li>kube-proxy 网络代理，负责网络相关的工作</li></ul><h1 id="开始安装"><a href="#开始安装" class="headerlink" title="开始安装"></a>开始安装</h1><h2 id="安装-Container-Runtimes"><a href="#安装-Container-Runtimes" class="headerlink" title="安装 Container Runtimes"></a>安装 Container Runtimes</h2><p>容器引擎是 Kubernetes 最重要的组件之一，负责管理镜像和容器的生命周期。Kubelet 通过 Container Runtime Interface (CRI) 与容器引擎交互，以管理镜像和容器。</p><p>Kubernetes 在 1.24 版本中移除了 Dockershim，并从此不再默认支持 Docker 容器引擎，详情请参见 <a href="https://kubernetes.io/zh-cn/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/">Kubernetes 即将移除 Dockershim</a>。使用 Containerd 调用链更短，组件更少，更稳定，占用节点资源更少。</p><blockquote><p>关于 Containerd 和 Docker 容器引擎相关介绍可以查看以下文档：</p><ul><li><a href="https://developer.aliyun.com/article/751253?spm=5176.21213303.J_6704733920.7.525653c9WWxhBG&scm=20140722.S_community@@%E6%96%87%E7%AB%A0@@751253._.ID_community@@%E6%96%87%E7%AB%A0@@751253-RL_cri-LOC_main-OR_ser-V_2-P0_0">从零开始入门 K8s | 理解容器运行时接口 CRI</a></li><li><a href="https://cloud.tencent.com/document/product/457/35747">https://cloud.tencent.com/document/product/457/35747</a></li><li><a href="https://support.huaweicloud.com/usermanual-cce/cce_10_0462.html">https://support.huaweicloud.com/usermanual-cce/cce_10_0462.html</a></li></ul></blockquote><h2 id="安装-Containerd"><a href="#安装-Containerd" class="headerlink" title="安装 Containerd"></a>安装 Containerd</h2><p>官方文档: <a href="https://github.com/containerd/containerd/blob/main/docs/getting-started.md">https://github.com/containerd/containerd/blob/main/docs/getting-started.md</a></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 设置代理</span></span><br><span class="line"><span class="built_in">export</span> https_proxy=http://192.168.16.30:1087 http_proxy=http://192.168.16.30:1087 all_proxy=socks5://192.168.16.30:1087</span><br></pre></td></tr></table></figure><h3 id="Installing-containerd"><a href="#Installing-containerd" class="headerlink" title="Installing containerd"></a>Installing containerd</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">wget https://github.com/containerd/containerd/releases/download/v1.6.18/containerd-1.6.18-linux-amd64.tar.gz</span><br><span class="line"></span><br><span class="line">tar Cxzvf /usr/local containerd-1.6.18-linux-amd64.tar.gz</span><br><span class="line"></span><br><span class="line"><span class="comment"># 添加 containerd.service</span></span><br><span class="line"><span class="comment"># wget 命令: -P 指定不同目录保存下载的文件</span></span><br><span class="line">wget -P /etc/systemd/system https://raw.githubusercontent.com/containerd/containerd/main/containerd.service</span><br></pre></td></tr></table></figure><p><strong><em>修改配置:</em></strong></p><ol><li>生成默认配置</li></ol><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">mkdir</span> -p /etc/containerd/</span><br><span class="line"></span><br><span class="line">containerd config default > /etc/containerd/config.toml</span><br></pre></td></tr></table></figure><ol start="2"><li>修改 <code>CgroupDriver</code> 为 <code>systemd</code></li></ol><p>k8s 官方推荐使用 <code>systemd</code> 类型的 <code>CgroupDriver</code>。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]</span></span><br><span class="line"><span class="comment"># ...</span></span><br><span class="line"><span class="comment"># [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]</span></span><br><span class="line"><span class="comment"># SystemdCgroup = true</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 替换 config.toml 中 SystemdCgroup 值</span></span><br><span class="line">sed -i <span class="string">'s/SystemdCgroup = false/SystemdCgroup = true/g'</span> /etc/containerd/config.toml</span><br></pre></td></tr></table></figure><h3 id="Installing-runc"><a href="#Installing-runc" class="headerlink" title="Installing runc"></a>Installing runc</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">wget https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64</span><br><span class="line"></span><br><span class="line">install -m 755 runc.amd64 /usr/local/sbin/runc</span><br></pre></td></tr></table></figure><h3 id="Installing-cni-plugins"><a href="#Installing-cni-plugins" class="headerlink" title="Installing cni plugins"></a>Installing cni plugins</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">wget https://github.com/containernetworking/plugins/releases/download/v1.2.0/cni-plugins-linux-amd64-v1.2.0.tgz</span><br><span class="line"></span><br><span class="line"><span class="built_in">mkdir</span> -p /opt/cni/bin</span><br><span class="line"></span><br><span class="line">tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.2.0.tgz</span><br></pre></td></tr></table></figure><h3 id="Installing-nerdctl-crictl"><a href="#Installing-nerdctl-crictl" class="headerlink" title="Installing nerdctl && crictl"></a>Installing nerdctl && crictl</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># install `nerdctl`</span></span><br><span class="line">wget https://github.com/containerd/nerdctl/releases/download/v1.2.0/nerdctl-1.2.0-linux-amd64.tar.gz</span><br><span class="line"></span><br><span class="line">tar Cxzvf /usr/local/bin nerdctl-1.2.0-linux-amd64.tar.gz</span><br><span class="line"></span><br><span class="line"><span class="comment"># install `crictl`</span></span><br><span class="line">wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.26.0/crictl-v1.26.0-linux-amd64.tar.gz</span><br><span class="line"></span><br><span class="line">tar Cxzvf /usr/local/bin crictl-v1.26.0-linux-amd64.tar.gz</span><br></pre></td></tr></table></figure><p><em>可能遇到的问题：</em></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">root@master:~<span class="comment"># crictl images</span></span><br><span class="line">WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should <span class="built_in">set</span> the endpoint instead.</span><br><span class="line">E0223 14:52:26.053351 32562 remote_image.go:119] <span class="string">"ListImages with filter from image service failed"</span> err=<span class="string">"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory\""</span> filter=<span class="string">"&ImageFilter{Image:&ImageSpec{Image:,Annotations:map[string]string{},},}"</span></span><br><span class="line">FATA[0000] listing images: rpc error: code = Unavailable desc = connection error: desc = <span class="string">"transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"</span></span><br></pre></td></tr></table></figure><p><code>crictl</code> 依次查找容器运行时，当查找第一个 <code>unix:///var/run/dockershim.sock</code> 没有找到，所以报错了，需要手动指定当前容器运行时。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/</span></span><br><span class="line"><span class="built_in">cat</span> <<<span class="string">EOF > /etc/crictl.yaml</span></span><br><span class="line"><span class="string">runtime-endpoint: unix:///var/run/containerd/containerd.sock</span></span><br><span class="line"><span class="string">image-endpoint: unix:///var/run/containerd/containerd.sock</span></span><br><span class="line"><span class="string">timeout: 10</span></span><br><span class="line"><span class="string">debug: true</span></span><br><span class="line"><span class="string">EOF</span></span><br></pre></td></tr></table></figure><h3 id="Start-containerd-service"><a href="#Start-containerd-service" class="headerlink" title="Start containerd.service"></a>Start containerd.service</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">systemctl daemon-reload</span><br><span class="line"></span><br><span class="line">systemctl start containerd</span><br><span class="line">systemctl <span class="built_in">enable</span> containerd</span><br></pre></td></tr></table></figure><p><strong><em>Containerd 设置 http_proxy</em></strong></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">mkdir</span> -p /etc/systemd/system/containerd.service.d</span><br><span class="line"></span><br><span class="line"><span class="built_in">cat</span> <<<span class="string">'EOF'</span> > /etc/systemd/system/containerd.service.d/http-proxy.conf</span><br><span class="line">[Service]</span><br><span class="line">Environment=<span class="string">"HTTP_PROXY=http://192.168.16.30:1087"</span></span><br><span class="line">Environment=<span class="string">"HTTPS_PROXY=http://192.168.16.30:1087"</span></span><br><span class="line">Environment=<span class="string">"NO_PROXY=localhost,127.0.0.1"</span></span><br><span class="line">EOF</span><br><span class="line"></span><br><span class="line">systemctl daemon-reload</span><br><span class="line">systemctl restart containerd.service</span><br><span class="line"></span><br><span class="line"><span class="comment"># 查看是否生效</span></span><br><span class="line">systemctl show --property=Environment containerd</span><br></pre></td></tr></table></figure><h2 id="2-安装-kubeadm、kubectl-和-kubelet"><a href="#2-安装-kubeadm、kubectl-和-kubelet" class="headerlink" title="2. 安装 kubeadm、kubectl 和 kubelet"></a>2. 安装 kubeadm、kubectl 和 kubelet</h2><h3 id="配置国内阿里源"><a href="#配置国内阿里源" class="headerlink" title="配置国内阿里源"></a>配置国内阿里源</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">cat</span> <<<span class="string">EOF > /etc/apt/sources.list.d/kubernetes.list</span></span><br><span class="line"><span class="string">deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main</span></span><br><span class="line"><span class="string">EOF</span></span><br></pre></td></tr></table></figure><h3 id="添加证书"><a href="#添加证书" class="headerlink" title="添加证书"></a>添加证书</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -</span><br></pre></td></tr></table></figure><h3 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 查看 kubelet 版本列表: apt-cache madison kubelet</span></span><br><span class="line">apt-get update && apt-get install -y kubelet kubeadm kubectl</span><br><span class="line"></span><br><span class="line"><span class="comment"># 设置kubelet开机启动</span></span><br><span class="line">systemctl <span class="built_in">enable</span> kubelet</span><br></pre></td></tr></table></figure><h3 id="Master-节点部署"><a href="#Master-节点部署" class="headerlink" title="Master 节点部署"></a>Master 节点部署</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 查看k8s集群所需的镜像并进行提前下载</span></span><br><span class="line">kubeadm config images list</span><br><span class="line"></span><br><span class="line"><span class="comment"># 系统配置预检查</span></span><br><span class="line">kubeadm init phase preflight</span><br><span class="line"></span><br><span class="line"><span class="comment"># https://kubernetes.io/zh-cn/docs/reference/setup-tools/kubeadm/kubeadm-init/</span></span><br><span class="line">kubeadm init</span><br><span class="line"><span class="comment"># ......</span></span><br><span class="line"><span class="comment"># Your Kubernetes control-plane has initialized successfully!</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># To start using your cluster, you need to run the following as a regular user:</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># mkdir -p $HOME/.kube</span></span><br><span class="line"><span class="comment"># sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config</span></span><br><span class="line"><span class="comment"># sudo chown $(id -u):$(id -g) $HOME/.kube/config</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># Alternatively, if you are the root user, you can run:</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># export KUBECONFIG=/etc/kubernetes/admin.conf</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># You should now deploy a pod network to the cluster.</span></span><br><span class="line"><span class="comment"># Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:</span></span><br><span class="line"><span class="comment"># https://kubernetes.io/docs/concepts/cluster-administration/addons/</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># Then you can join any number of worker nodes by running the following on each as root:</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># kubeadm join 172.16.20.33:6443 --token rptcqb.flf9wkt1do06d2v4 --discovery-token-ca-cert-hash sha256:64ea9bc09fb80c4b7d81033e953e93f5173e99db6a655b88dfb69f166a0901a5</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 复制授权文件，以便 kubectl 可以有权限访问集群</span></span><br><span class="line"><span class="comment"># 如果你其他节点需要访问集群，需要从主节点复制这个文件过去其他节点</span></span><br><span class="line"><span class="comment"># 在其他机器上创建 ~/.kube/config 文件也能通过 kubectl 访问到集群</span></span><br><span class="line"><span class="built_in">mkdir</span> -p <span class="variable">$HOME</span>/.kube</span><br><span class="line"><span class="built_in">cp</span> -i /etc/kubernetes/admin.conf <span class="variable">$HOME</span>/.kube/config</span><br><span class="line"><span class="built_in">chown</span> $(<span class="built_in">id</span> -u):$(<span class="built_in">id</span> -g) <span class="variable">$HOME</span>/.kube/config</span><br><span class="line"></span><br><span class="line"><span class="comment"># 安装网络插件，否则 node 是 NotReady 状态（主节点跑）</span></span><br><span class="line"><span class="comment"># https://blog.csdn.net/ChaITSimpleLove/article/details/117809007</span></span><br><span class="line">kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml</span><br></pre></td></tr></table></figure><h3 id="Node-节点"><a href="#Node-节点" class="headerlink" title="Node 节点"></a>Node 节点</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 忘记了重新获取：kubeadm token create --print-join-command</span></span><br><span class="line">kubeadm <span class="built_in">join</span> 172.16.20.33:6443 --token rptcqb.flf9wkt1do06d2v4 --discovery-token-ca-cert-hash sha256:64ea9bc09fb80c4b7d81033e953e93f5173e99db6a655b88dfb69f166a0901a5</span><br></pre></td></tr></table></figure><h1 id="参考资料"><a href="#参考资料" class="headerlink" title="参考资料"></a>参考资料</h1><ul><li><a href="https://github.com/containerd/containerd/blob/main/docs/getting-started.md">https://github.com/containerd/containerd/blob/main/docs/getting-started.md</a></li><li><a href="https://k8s.huweihuang.com/project/runtime/containerd/install-containerd">https://k8s.huweihuang.com/project/runtime/containerd/install-containerd</a></li><li><a href="https://developer.aliyun.com/article/1038877">ubuntu 安装 Kubernetes-阿里云开发者社区</a></li><li><a href="https://lework.github.io/2019/10/01/kubeadm-install/">使用 kubeadm 安装 Kubernetes v15.4 ha 集群</a></li><li><a href="https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/">https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/</a></li><li><a href="https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/">https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/</a></li></ul>]]></content>
<summary type="html"><span id="more"></span>
<h1</summary>
<category term="kubenetes" scheme="https://ryan961.github.io/categories/kubenetes/"/>
<category term="linux" scheme="https://ryan961.github.io/tags/linux/"/>
<category term="kubenetes" scheme="https://ryan961.github.io/tags/kubenetes/"/>
<category term="ubuntu" scheme="https://ryan961.github.io/tags/ubuntu/"/>
</entry>
<entry>
<title>Tableflip + Systemd 实现 Go 应用热重启</title>
<link href="https://ryan961.github.io/2022/03/23/go-graceful-upgrades/"/>
<id>https://ryan961.github.io/2022/03/23/go-graceful-upgrades/</id>
<published>2022-03-23T16:00:00.000Z</published>
<updated>2022-03-23T16:00:00.000Z</updated>
<content type="html"><![CDATA[<p>对于一个常驻、高访问量的网络服务来说，在进行代码变更需要对进程进行升级/重启时，如何避免对正在通信的用户以及即将访问的用户造成影响便是一个难以忽视的问题，尤其是在用户量达到一定规模的时候。</p><p>本文将使用 <a href="https://github.com/cloudflare/tableflip">tableflip</a>并结合 <a href="https://github.com/cloudflare/tableflip#integration-with-systemd">systemd</a> 实现 Go 应用优雅的热重启。</p><span id="more"></span><h2 id="tableflip-简介"><a href="#tableflip-简介" class="headerlink" title="tableflip 简介"></a>tableflip 简介</h2><p><a href="https://github.com/cloudflare/tableflip">tableflip</a> 是 <a href="https://www.cloudflare.com/zh-cn/">cloudflare</a> 实现 Go 进程优雅重启的一个开源库，采用 <code>继承监听套接字</code> 方案，整体设计开放性足够，目前看起来是最好的一个实现。</p><p><code>tableflip</code> 的设计宗旨就是实现类似 nginx 的优雅热更新能力，包括：</p><ul><li>新进程启动成功后，老进程不会有资源残留</li><li>优雅的新进程初始化（新进程启动和初始化的过程中服务不会中断）</li><li>容忍新进程初始化的失败（如果新进程初始化失败，老进程会继续工作而不是退出）</li><li>同一时间只能有一个更新动作执行</li></ul><p><code>tableflip</code> 中的核心类型是 <code>Upgrader</code>，调用 <code>Upgrader.Upgrade</code> 会产生一个继承必要的 <code>net.Listeners</code><br>的新进程，并等待新进程发出表明其已成功完成初始化、退出或超时的信号。如果当前已有升级的任务在执行，则直接返回相应的错误。</p><p>当新进程启动成功后，调用 <code>Upgrader.Ready</code> 会清除无效的 fd 并向父进程发出初始化成功完成的信号，然后父进程就可以安心退出。至此，我们就完成了一次优雅的进程重启。</p><p><img data-src="https://cdn.jsdelivr.net/gh/ryan961/img-floder/blog/go-graceful-upgrades/tableflip_upgrade.png" alt="tableflip upgrade"></p><h2 id="代码示例"><a href="#代码示例" class="headerlink" title="代码示例"></a>代码示例</h2><p>源码地址：<a href="https://github.com/ryan961/go-exp/blob/main/reload/main.go">https://github.com/ryan961/go-exp/blob/main/reload/main.go</a></p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">upg, err := tableflip.New(tableflip.Options{PIDFile: <span class="string">"./reload.pid"</span>})</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"><span class="built_in">panic</span>(err)</span><br><span class="line">}</span><br><span class="line"><span class="keyword">defer</span> upg.Stop()</span><br><span class="line"></span><br><span class="line"><span class="comment">// 为了演示方便，为程序启动强行加入 1s 的延时，并在日志中附上进程 pid</span></span><br><span class="line">time.Sleep(time.Second)</span><br><span class="line">log.SetPrefix(fmt.Sprintf(<span class="string">"[PID: %d] "</span>, os.Getpid()))</span><br><span class="line"></span><br><span class="line"><span class="comment">// 监听系统的 SIGHUP 信号，以此信号触发进程重启</span></span><br><span class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line">sig := <span class="built_in">make</span>(<span class="keyword">chan</span> os.Signal, <span class="number">1</span>)</span><br><span class="line">signal.Notify(sig, syscall.SIGHUP, syscall.SIGUSR2, syscall.SIGTERM, syscall.SIGINT, syscall.SIGQUIT)</span><br><span class="line"><span class="keyword">for</span> si := <span class="keyword">range</span> sig {</span><br><span class="line"><span class="keyword">switch</span> si {</span><br><span class="line"><span class="keyword">case</span> syscall.SIGHUP, syscall.SIGUSR2:</span><br><span class="line"><span class="comment">// 核心的 Upgrade 调用</span></span><br><span class="line"><span class="keyword">if</span> err := upg.Upgrade(); err != <span class="literal">nil</span> {</span><br><span class="line">log.Println(<span class="string">"server upgrade failed:"</span>, err)</span><br><span class="line">}</span><br><span class="line"><span class="keyword">default</span>:</span><br><span class="line">log.Println(<span class="string">"shutdown server start:"</span>, si)</span><br><span class="line">upg.Stop()</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line">}()</span><br><span class="line"></span><br><span class="line"><span class="comment">// 注意必须使用 upg.Listen 对端口进行监听</span></span><br><span class="line">ln, err := upg.Listen(<span class="string">"tcp"</span>, fmt.Sprintf(<span class="string">":%d"</span>, endPoint))</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line">log.Fatalln(<span class="string">"Can't listen:"</span>, err)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 创建一个简单的 http server，/version 返回当前的程序版本</span></span><br><span class="line">mux := http.NewServeMux()</span><br><span class="line">mux.HandleFunc(<span class="string">"/version"</span>, <span class="function"><span class="keyword">func</span><span class="params">(rw http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line">log.Println(Version)</span><br><span class="line">rw.Write([]<span class="type">byte</span>(Version + <span class="string">"\n"</span>))</span><br><span class="line">})</span><br><span class="line">server := http.Server{</span><br><span class="line">Handler: mux,</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 照常启动 http server</span></span><br><span class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line">err := server.Serve(ln)</span><br><span class="line"><span class="keyword">if</span> err != http.ErrServerClosed {</span><br><span class="line">log.Println(<span class="string">"HTTP server:"</span>, err)</span><br><span class="line">}</span><br><span class="line">}()</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> err := upg.Ready(); err != <span class="literal">nil</span> {</span><br><span class="line"><span class="built_in">panic</span>(err)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">log.Printf(<span class="string">"http server listening %d"</span>, endPoint)</span><br><span class="line"></span><br><span class="line"><-upg.Exit()</span><br><span class="line"></span><br><span class="line"><span class="comment">// 给老进程的退出设置一个 30s 的超时时间，保证老进程的退出</span></span><br><span class="line">time.AfterFunc(<span class="number">30</span>*time.Second, <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line">log.Println(<span class="string">"Graceful shutdown timed out"</span>)</span><br><span class="line">os.Exit(<span class="number">1</span>)</span><br><span class="line">})</span><br><span class="line"></span><br><span class="line"><span class="comment">// 等待 http server 的优雅退出上面的代码实现了一个返回当前 version 的 http server，我们在启动过程中延迟 1s 以便观察升级过程中服务是否依旧可用。</span></span><br><span class="line">server.Shutdown(context.Background())</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>编译运行：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">go build -ldflags "-X main.Version=0.0.1" -o reload</span><br><span class="line">./reload</span><br></pre></td></tr></table></figure><p>使用 curl 模拟一些客户端请求（10 qps）：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">while true; do curl http://localhost:8080/version; sleep 0.1; done</span><br></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[PID: 3399216] 2022/03/23 22:21:31 http server listening 8080</span><br><span class="line">[PID: 3399216] 2022/03/23 22:23:48 0.0.1</span><br><span class="line">[PID: 3399216] 2022/03/23 22:23:48 0.0.1</span><br><span class="line">[PID: 3399216] 2022/03/23 22:23:48 0.0.1</span><br><span class="line">...</span><br></pre></td></tr></table></figure><p>然后，我们对应用进行了一些升级，将版本号修改为 0.0.2，并重新编译程序：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">go build -ldflags "-X main.Version=0.0.2" -o reload</span><br></pre></td></tr></table></figure><p>最后，尝试优雅的热重启：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">kill -HUP 3399216</span><br></pre></td></tr></table></figure><p>由此可见，客户端没有受到服务器端 reload 的影响。至此，进程优雅的热更新已完成。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">...</span><br><span class="line">[PID: 3399216] 2022/03/23 22:23:48 0.0.1</span><br><span class="line">[PID: 3399216] 2022/03/23 22:23:48 0.0.1</span><br><span class="line">[PID: 3401398] 2022/03/23 22:23:48 http server listening 8080</span><br><span class="line"><span class="meta prompt_">~/reload# </span><span class="language-bash">[PID: 3401398] 2022/03/23 22:23:48 0.0.2</span></span><br><span class="line">[PID: 3401398] 2022/03/23 22:23:48 0.0.2</span><br><span class="line">[PID: 3401398] 2022/03/23 22:23:48 0.0.2</span><br><span class="line">...</span><br></pre></td></tr></table></figure><h2 id="systemd-管理进程"><a href="#systemd-管理进程" class="headerlink" title="systemd 管理进程"></a>systemd 管理进程</h2><p>reload_go.service:</p><figure class="highlight toml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="section">[Unit]</span></span><br><span class="line"><span class="attr">Description</span>=Service using tableflip</span><br><span class="line"></span><br><span class="line"><span class="section">[Service]</span></span><br><span class="line"><span class="attr">WorkingDirectory</span>=/root/reload</span><br><span class="line"><span class="attr">ExecStart</span>=/root/reload/reload</span><br><span class="line"><span class="attr">ExecStop</span>=/bin/kill -TERM <span class="variable">$MAINPID</span></span><br><span class="line"><span class="attr">ExecReload</span>=/bin/kill -HUP <span class="variable">$MAINPID</span></span><br><span class="line"><span class="attr">PIDFile</span>=/root/reload/reload.pid</span><br><span class="line"><span class="attr">Type</span>=simple</span><br></pre></td></tr></table></figure><p>使用 <code>systemd</code> 管理进程：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line">root@:# systemctl start reload_go.service</span><br><span class="line">root@:# systemctl status reload_go.service</span><br><span class="line">● reload_go.service - Service using tableflip</span><br><span class="line"> Loaded: loaded (/etc/systemd/system/reload_go.service; static; vendor preset: enabled)</span><br><span class="line"> Active: active (running) since Thu 2022-03-24 21:04:07 CST; 6s ago</span><br><span class="line"> Main PID: 3929535 (reload)</span><br><span class="line"> Tasks: 7 (limit: 4619)</span><br><span class="line"> Memory: 1.4M</span><br><span class="line"> CGroup: /system.slice/reload_go.service</span><br><span class="line"> └─3929535 /root/reload/reload</span><br><span class="line"></span><br><span class="line">Mar 24 21:04:07 systemd[1]: Started Service using tableflip.</span><br><span class="line">Mar 24 21:04:08 reload[3929535]: [PID: 3929535] 2022-03-24 21:04:08 http server listening 8080</span><br><span class="line">root@:# systemctl reload reload_go.service</span><br><span class="line">root@:# systemctl status reload_go.service</span><br><span class="line">● reload_go.service - Service using tableflip</span><br><span class="line"> Loaded: loaded (/etc/systemd/system/reload_go.service; static; vendor preset: enabled)</span><br><span class="line"> Active: active (running) since Thu 2022-03-24 21:04:07 CST; 17s ago</span><br><span class="line"> Process: 3929585 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)</span><br><span class="line"> Main PID: 3929588 (reload)</span><br><span class="line"> Tasks: 10 (limit: 4619)</span><br><span class="line"> Memory: 1.8M</span><br><span class="line"> CGroup: /system.slice/reload_go.service</span><br><span class="line"> └─3929588 /root/reload/reload</span><br><span class="line"></span><br><span class="line">Mar 24 21:04:07 systemd[1]: Started Service using tableflip.</span><br><span class="line">Mar 24 21:04:08 reload[3929535]: [PID: 3929535] 2022/03/24 21:04:08 http server listening 8080</span><br><span class="line">Mar 24 21:04:23 systemd[1]: Reloading Service using tableflip.</span><br><span class="line">Mar 24 21:04:23 systemd[1]: Reloaded Service using tableflip.</span><br><span class="line">Mar 24 21:04:24 reload[3929588]: [PID: 3929588] 2022/03/24 21:04:24 http server listening 8080</span><br><span class="line">...</span><br></pre></td></tr></table></figure><p>至此，我们已基本完成使用 <code>systemd</code> 管理进程。</p><h2 id="判断-reload-是否成功"><a href="#判断-reload-是否成功" class="headerlink" title="判断 reload 是否成功"></a>判断 reload 是否成功</h2><p>根据前面的流程，我们已基本完成 reload + systemd 的需求。但实际在发布过程中，<code>reload</code> 成功与否 <code>systemd</code> 无法感知，必须人工介入查看进程状态才能获知。而判断 <code>reload</code> 成功与否的标志便是进程 <code>pid</code> 是否发生变化，所以我们加入 <code>reload.sh</code> 来判断 <code>pid</code> 是否变化：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta prompt_">#</span><span class="language-bash">!/bin/bash</span></span><br><span class="line"></span><br><span class="line">OLD_PID=$(cat $1)</span><br><span class="line">/bin/kill -HUP $OLD_PID</span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">等待进程 reload 成功</span></span><br><span class="line">sleep 10</span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">检查是否启动成功</span></span><br><span class="line">NEW_PID=$(cat $1)</span><br><span class="line"><span class="meta prompt_"></span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">旧进程退出, 新进程启动</span></span><br><span class="line">ps --pid $OLD_PID &>/dev/null</span><br><span class="line">OLD_EXIST=$?</span><br><span class="line">ps --pid $NEW_PID &>/dev/null</span><br><span class="line">NEW_EXIST=$?</span><br><span class="line"></span><br><span class="line">if [ $OLD_EXIST -eq 1 ] && [ $NEW_EXIST -eq 0 ]; then</span><br><span class="line"> exit 0</span><br><span class="line">else</span><br><span class="line"> exit -1</span><br><span class="line">fi</span><br></pre></td></tr></table></figure><p>更新 <code>reload_go.service</code> :</p><figure class="highlight toml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="section">[Unit]</span></span><br><span class="line"><span class="attr">Description</span>=Service using tableflip</span><br><span class="line"></span><br><span class="line"><span class="section">[Service]</span></span><br><span class="line"><span class="attr">WorkingDirectory</span>=/root/reload</span><br><span class="line"><span class="attr">ExecStart</span>=/root/reload/reload</span><br><span class="line"><span class="attr">ExecStop</span>=/bin/kill -TERM <span class="variable">$MAINPID</span></span><br><span class="line"><span class="attr">ExecReload</span>=/root/reload/reload.sh /root/reload/reload.pid</span><br><span class="line"><span class="attr">PIDFile</span>=/root/reload/reload.pid</span><br><span class="line"><span class="attr">Type</span>=simple</span><br></pre></td></tr></table></figure><p>模拟进程 <code>reload</code> 失败：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br></pre></td><td class="code"><pre><span class="line">root@:# systemctl start reload_go.service</span><br><span class="line">root@:# systemctl status reload_go.service</span><br><span class="line">● reload_go.service - Service using tableflip</span><br><span class="line"> Loaded: loaded (/etc/systemd/system/reload_go.service; static; vendor preset: enabled)</span><br><span class="line"> Active: active (running) since Thu 2022-03-24 21:43:15 CST; 2s ago</span><br><span class="line"> Main PID: 3935728 (reload)</span><br><span class="line"> Tasks: 7 (limit: 4619)</span><br><span class="line"> Memory: 1.3M</span><br><span class="line"> CGroup: /system.slice/reload_go.service</span><br><span class="line"> └─3935728 /root/reload/reload</span><br><span class="line"></span><br><span class="line">Mar 24 21:43:15 systemd[1]: Started Service using tableflip.</span><br><span class="line">Mar 24 21:43:16 reload[3935728]: [PID: 3935728] 2022/03/24 21:43:16 http server listening 8080</span><br><span class="line">root@:# systemctl reload reload_go.service</span><br><span class="line">root@:# systemctl status reload_go.service</span><br><span class="line">● reload_go.service - Service using tableflip</span><br><span class="line"> Loaded: loaded (/etc/systemd/system/reload_go.service; static; vendor preset: enabled)</span><br><span class="line"> Active: active (running) since Thu 2022-03-24 21:43:15 CST; 28s ago</span><br><span class="line"> Process: 3935781 ExecReload=/root/reload/reload.sh /root/reload/reload.pid (code=exited, status=0/SUCCESS)</span><br><span class="line"> Main PID: 3935789 (reload)</span><br><span class="line"> Tasks: 7 (limit: 4619)</span><br><span class="line"> Memory: 2.0M</span><br><span class="line"> CGroup: /system.slice/reload_go.service</span><br><span class="line"> └─3935789 /root/reload/reload</span><br><span class="line"></span><br><span class="line">Mar 24 21:43:15 systemd[1]: Started Service using tableflip.</span><br><span class="line">Mar 24 21:43:16 reload[3935728]: [PID: 3935728] 2022/03/24 21:43:16 http server listening 8080</span><br><span class="line">Mar 24 21:43:30 systemd[1]: Reloading Service using tableflip.</span><br><span class="line">Mar 24 21:43:31 reload[3935789]: [PID: 3935789] 2022/03/24 21:43:31 http server listening 8080</span><br><span class="line">Mar 24 21:43:40 systemd[1]: Reloaded Service using tableflip.</span><br><span class="line">root@:/etc/systemd/system# systemctl reload reload_go.service</span><br><span class="line">Job for reload_go.service failed.</span><br><span class="line">See "systemctl status reload_go.service" and "journalctl -xe" for details.</span><br><span class="line">root@:# systemctl status reload_go.service</span><br><span class="line">● reload_go.service - Service using tableflip</span><br><span class="line"> Loaded: loaded (/etc/systemd/system/reload_go.service; static; vendor preset: enabled)</span><br><span class="line"> Active: active (running) since Thu 2022-03-24 21:43:15 CST; 1min 43s ago</span><br><span class="line"> Process: 3936335 ExecReload=/root/reload/reload.sh /root/reload/reload.pid (code=exited, status=255/EXCEPTION)</span><br><span class="line"> Main PID: 3935789 (reload)</span><br><span class="line"> Tasks: 7 (limit: 4619)</span><br><span class="line"> Memory: 2.2M</span><br><span class="line"> CGroup: /system.slice/reload_go.service</span><br><span class="line"> └─3935789 /root/reload/reload</span><br><span class="line"></span><br><span class="line">Mar 24 21:43:31 reload[3935789]: [PID: 3935789] 2022/03/24 21:43:31 http server listening 8080</span><br><span class="line">Mar 24 21:43:40 systemd[1]: Reloaded Service using tableflip.</span><br><span class="line">Mar 24 21:44:39 systemd[1]: Reloading Service using tableflip.</span><br><span class="line">Mar 24 21:44:39 reload[3936338]: panic: reload.sh</span><br><span class="line">Mar 24 21:44:39 reload[3936338]: goroutine 1 [running]:</span><br><span class="line">Mar 24 21:44:39 reload[3936338]: main.main()</span><br><span class="line">Mar 24 21:44:39 reload[3936338]: /root/reload/main.go:23 +0x3e</span><br><span class="line">Mar 24 21:44:39 reload[3935789]: [PID: 3935789] 2022/03/24 21:44:39 server upgrade failed: child pid=3936338 exited: ex></span><br><span class="line">Mar 24 21:44:49 systemd[1]: reload_go.service: Control process exited, code=exited, status=255/EXCEPTION</span><br><span class="line">Mar 24 21:44:49 systemd[1]: Reload failed for Service using tableflip.</span><br><span class="line">...</span><br></pre></td></tr></table></figure><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>根据上述流程，我们已完成使用 <code>systemd</code> + <code>tableflip</code> 优雅重启进程的需求，并基本可以投入生产环境中使用。有需要也可以自行接入 <code>spug</code> 发布流程中。</p><p>源码地址：<a href="https://github.com/ryan961/go-exp/tree/main/reload">https://github.com/ryan961/go-exp/tree/main/reload</a></p><h2 id="参考资料"><a href="#参考资料" class="headerlink" title="参考资料"></a>参考资料</h2><ul><li><a href="https://github.com/cloudflare/tableflip">https://github.com/cloudflare/tableflip</a></li><li><a href="https://blog.cloudflare.com/graceful-upgrades-in-go/">https://blog.cloudflare.com/graceful-upgrades-in-go/</a></li><li><a href="https://gocn.vip/topics/qoYMdnTxo4">https://gocn.vip/topics/qoYMdnTxo4</a></li></ul>]]></content>
<summary type="html"><p>对于一个常驻、高访问量的网络服务来说，在进行代码变更需要对进程进行升级&#x2F;重启时，如何避免对正在通信的用户以及即将访问的用户造成影响便是一个难以忽视的问题，尤其是在用户量达到一定规模的时候。</p>
<p>本文将使用 <a href="https://github.com/cloudflare/tableflip">tableflip</a>并结合 <a href="https://github.com/cloudflare/tableflip#integration-with-systemd">systemd</a> 实现 Go 应用优雅的热重启。</p></summary>
<category term="golang" scheme="https://ryan961.github.io/categories/golang/"/>
<category term="golang" scheme="https://ryan961.github.io/tags/golang/"/>
<category term="reload" scheme="https://ryan961.github.io/tags/reload/"/>
<category term="systemd" scheme="https://ryan961.github.io/tags/systemd/"/>
</entry>
<entry>
<title>我的 macOS 软件清单</title>
<link href="https://ryan961.github.io/2022/03/19/my-mac-tools/"/>
<id>https://ryan961.github.io/2022/03/19/my-mac-tools/</id>
<published>2022-03-19T16:00:00.000Z</published>
<updated>2024-01-06T16:00:00.000Z</updated>
<content type="html"><![CDATA[<p>人生中的第一篇博客，无脑水一篇「我的 macOS 软件清单」～</p><span id="more"></span><h2 id="装机必备"><a href="#装机必备" class="headerlink" title="装机必备"></a>装机必备</h2><h3 id="常用软件"><a href="#常用软件" class="headerlink" title="常用软件"></a>常用软件</h3><ul><li><a href="https://mac.weixin.qq.com/?t=mac&lang=zh_CN">微信</a></li><li><a href="https://www.feishu.cn/">飞书</a></li><li><a href="https://telegram.org/">Telegram</a></li><li><a href="https://fanyi.youdao.com/index.html#/">网易有道翻译</a></li><li><a href="https://netnewswire.com/">NetNewsWire</a> 免费开源的老牌 RSS 阅读器。</li><li><a href="https://music.163.com/#/download">网易云音乐</a></li><li><a href="https://y.qq.com/download/mac.html?part=1&ADTAG=YQQ">QQ 音乐</a></li><li><a href="https://www.wps.com/">WPS Office</a></li><li><a href="https://www.macgpt.com/">MacGPT</a> Mac 端非常好用的 ChatGPT 应用。</li><li><a href="https://www.i4.cn/pro_pc.html">爱思助手</a> iPhone 管理助手。</li></ul><h3 id="浏览器"><a href="#浏览器" class="headerlink" title="浏览器"></a>浏览器</h3><ul><li>Safari 浏览器</li><li>Google Chrome</li></ul><h3 id="好用推荐"><a href="#好用推荐" class="headerlink" title="好用推荐"></a>好用推荐</h3><ul><li><a href="https://github.com/exelban/stats">stats</a> 查看系 (风) 统 (扇) 状 (转) 态 (速) 等各种系统指标的工具。</li><li><a href="https://github.com/Mortennn/Dozer">Dozer</a> Mac menu bar 隐藏工具。</li><li><a href="https://www.macbartender.com/Bartender4/">Bartender 4</a> 以二级菜单栏的形式隐藏管理 Mac 菜单栏图标，非常适合现在的刘海屏。</li><li><a href="https://github.com/lanceliao/china-holiday-calender">china-holiday-calender</a> 日历订阅，中国节假日、调休、补班日历信息。</li><li><a href="https://theunarchiver.com/">The Unarchiver</a> 解压缩工具。</li><li><a href="https://paper.meiyuan.in/">pap.er</a> 好用的 Mac 壁纸软件。</li><li><a href="https://apps.apple.com/cn/app/id1485844094">ishot</a> 截长图工具。</li><li><a href="https://github.com/AerialScreensaver/AerialCompanion">Aerial Companion</a> MacOS 版 Aerial 屏幕保护程序的配套应用程序，可进行安装和自动更新。</li><li><a href="https://apphousekitchen.com/">AlDente</a> 保持 MacBook Pro 电池健康度 100% 的小妙招（详见 <a href="https://www.v2ex.com/t/899220">保持 MacBook Pro 电池健康度 100%的实践经验分享</a>）。</li><li><a href="https://inputsource.pro/zh-CN">Input Source Pro</a> 自动切换输入法加上适时的提示。</li></ul><h3 id="效率工具"><a href="#效率工具" class="headerlink" title="效率工具"></a>效率工具</h3><ul><li><a href="https://apps.apple.com/cn/app/ohmystar/id1218642292?mt=12">OhMyStar</a> 管理和查找自己在 Github 上点的 Star。</li><li><a href="https://kapeli.com/dash">Dash</a> 开源项目文档查看工具。</li><li><a href="https://www.alfredapp.com/">Alfred</a> Mac 效率神器。</li><li><a href="https://github.com/yichengchen/clashX">ClashX</a> 那个啥科学上网，dddd（懂得都懂）。</li></ul><h2 id="开发环境"><a href="#开发环境" class="headerlink" title="开发环境"></a>开发环境</h2><h3 id="IDE-编辑器"><a href="#IDE-编辑器" class="headerlink" title="IDE/编辑器"></a>IDE/编辑器</h3><ul><li><a href="https://www.jetbrains.com/">JetBrains 全家桶</a> 搬砖主力工具，需要付费，也可以通过申请教育许可证、开发者认证等渠道激活。</li><li><a href="https://code.visualstudio.com/download">Visual Studio Code</a> 全能代码编辑器，远程神器！</li><li><a href="https://www.sublimetext.com/">Sublime Text</a> 代码 “记事本”，平时用做简单的代码或者数据编辑。</li></ul><h3 id="数据库"><a href="#数据库" class="headerlink" title="数据库"></a>数据库</h3><ul><li><a href="https://www.navicat.com.cn/">Navicat</a> MySQL、MongoDB 都能使用。</li><li><a href="https://github.com/uglide/RedisDesktopManager">RedisDesktopManager</a> Redis 可视化工具。</li><li><a href="http://www.sequelpro.com/">Sequel Pro</a> MySQL 图形客户端。</li><li><a href="https://www.mongodb.com/zh-cn/products/compass">MongoDB Compass</a> MongoDB 官方的图形用户界面。</li></ul><h3 id="虚拟机-容器"><a href="#虚拟机-容器" class="headerlink" title="虚拟机/容器"></a>虚拟机/容器</h3><ul><li><a href="https://www.parallels.com/">Parallel desktop</a> 虚拟机，比 Vmware 好用很多。</li><li><a href="https://hub.docker.com/">Docker</a> Docker，也是虚拟机。</li><li><a href="https://orbstack.dev/">OrbStack</a> mac 上最好用的 Docker 客户端（轻、快、低消耗），没有之一！</li></ul><h3 id="终端"><a href="#终端" class="headerlink" title="终端"></a>终端</h3><ul><li><a href="https://iterm2.com/">iTerm2</a> 终端软件，比 Mac 自带的好用多了。</li><li><a href="https://ohmyz.sh/">Oh My Zsh</a> 一键配置 Zsh 环境的脚本。</li><li><a href="https://brew.sh/">Homebrew</a> Mac 装各种工具用的。</li><li><a href="https://asdf-vm.com/">asdf</a> 巨好用的多版本管理工具。</li><li><a href="https://github.com/skywind3000/z.lua/">z.lua</a> 命令行快速跳转工具。</li></ul><h3 id="辅助工具"><a href="#辅助工具" class="headerlink" title="辅助工具"></a>辅助工具</h3><ul><li><a href="https://www.sourcetreeapp.com/">SourceTree</a> 图形界面管理 Git 仓库。</li><li><a href="https://www.postman.com/">Postman</a> 调试 API 用的 HTTP 客户端。</li><li><a href="https://www.charlesproxy.com/">Charles</a> 抓包工具。</li><li><a href="https://github.com/bloomrpc/bloomrpc">BloomRPC</a> 调试 Grpc 接口工具。</li><li><a href="https://github.com/oldj/SwitchHosts">SwitchHosts</a> 切换 Host 的工具。</li></ul><h2 id="杂项"><a href="#杂项" class="headerlink" title="杂项"></a>杂项</h2><h3 id="Chrome-plugins"><a href="#Chrome-plugins" class="headerlink" title="Chrome plugins"></a>Chrome plugins</h3><ul><li><a href="https://chrome.google.com/webstore/detail/proxy-switchyomega/padekgcemlokbadohgkifijomclgjgif">SwitchyOmega</a> 科学上网你懂的。</li><li><a href="https://www.octotree.io/">Octotree</a> 常逛 Github 的你，必备。可以将仓库代码梳理成一个树状文件列表，挂到页面左侧。</li><li><a href="https://chrome.google.com/webstore/detail/%E5%B9%BF%E5%91%8A%E7%BB%88%E7%BB%93%E8%80%85/fpdnjdlbdmifoocedhkighhlbchbiikl">广告终结者</a> 这个不用说了吧，清爽.</li><li><a href="https://chrome.google.com/webstore/detail/%E5%88%92%E8%AF%8D%E7%BF%BB%E8%AF%91/ikhdkkncnoglghljlkmcimlnlhkeamad">划词翻译</a> 学渣必备。</li><li><a href="https://immersivetranslate.com/">沉浸式翻译</a> 可以将网页内容转成双语翻译，允许自己指定翻译引擎。</li><li><a href="https://chrome.google.com/webstore/detail/ublacklist/pncfbmialoiaghdehhbnbhkkgmjanfhe">uBlacklist</a> 搜索结果中屏蔽特定的网站显示，清爽。</li><li><a href="https://1password.com/zh-cn">1Password</a> 密码本工具。</li><li><a href="https://chromewebstore.google.com/detail/modern-for-hacker-news/dabkegjlekdcmefifaolmdhnhdcplklo">Modern for Hacker News</a> Hacker News 网页 UI 插件.</li><li><a href="https://chromewebstore.google.com/detail/kbfnbcaeplbcioakkpcpgfkobkghlhen">Grammarly</a> 一个文法检查器和写作应用程序，它可以帮助用户提高写作技巧和文法水平。Grammarly 提供多种功能，如文法检查、拼写检查、标点检查、语气检测和自动引文等，用户可以根据自己的需要选择不同的功能。</li><li><a href="https://chromewebstore.google.com/detail/monica-chatgpt-4-%E9%A9%B1%E5%8A%A8%E7%9A%84-ai-c/ofpnmcalabcbjgholdjcjblkibolbppb">Monica</a> AI 全能助手，由 GPT-4 驱动。回答复杂问题，撰写邮件，阅读文章，智能搜索。随处可用。</li></ul><h3 id="JetBrains-plugins"><a href="#JetBrains-plugins" class="headerlink" title="JetBrains plugins"></a>JetBrains plugins</h3><ul><li><a href="https://plugins.jetbrains.com/plugin/12425-darktheme">DarkTheme</a></li><li><a href="https://plugins.jetbrains.com/plugin/13792--darkcode-theme">DarkCode Theme</a></li><li><a href="https://plugins.jetbrains.com/plugin/18922-gerry-themes">Gerry Themes</a></li><li><a href="https://plugins.jetbrains.com/plugin/13643-monokai-pro-theme">Monokai Pro Theme</a></li><li><a href="https://plugins.jetbrains.com/plugin/12995-trash-panda-theme">Trash Panda Theme</a></li><li><a href="https://plugins.jetbrains.com/plugin/16604-extra-toolwindow-colorful-icons">Extra ToolWindow Colorful Icons</a></li><li><a href="https://plugins.jetbrains.com/plugin/10469-carbon-now-sh">carbon-now-sh</a> 创建并共享源代码的精美图片(详见 <a href="https://carbon.now.sh/">https://carbon.now.sh/</a> )。</li><li><a href="https://plugins.jetbrains.com/plugin/17288-foldable-projectview">Foldable ProjectView</a> 折叠 .gitignore 里的文件夹。</li><li><a href="https://plugins.jetbrains.com/plugin/9861-git-commit-template">Git Commit Template</a></li><li><a href="https://plugins.jetbrains.com/plugin/7499-gittoolbox">GitToolBox</a></li><li><a href="https://plugins.jetbrains.com/plugin/12132-leetcode-editor">LeetCode Editor</a></li><li><a href="https://plugins.jetbrains.com/plugin/7017-plantuml-integration">PlantUML Integration</a></li><li><a href="https://plugins.jetbrains.com/plugin/10080-rainbow-brackets">Rainbow Brackets</a></li><li><a href="https://plugins.jetbrains.com/plugin/9836-randomness">Randomness</a> 插入随机数据。</li><li><a href="https://plugins.jetbrains.com/plugin/8579-translation">Translation</a></li><li><a href="https://plugins.jetbrains.com/plugin/18824-codeglance-pro">CodeGlance Pro</a> 在编辑器窗格中显示类似于 Sublime 中的缩小概览或小地图。</li><li><a href="https://plugins.jetbrains.com/plugin/20540-codeium-ai-autocomplete-and-chat-for-python-js-ts-java-go-">Codeium</a> AI Autocomplete and Chat for Python, JS, TS, Java, Go…</li><li><a href="https://plugins.jetbrains.com/plugin/22282-ai-assistant">AI Assistant</a></li><li><a href="https://plugins.jetbrains.com/plugin/17718-github-copilot">GitHub Copilot</a></li></ul><p><strong><em>More to come…</em></strong> :)</p>]]></content>
<summary type="html"><p>人生中的第一篇博客，无脑水一篇「我的 macOS 软件清单」～</p></summary>
<category term="mac tools" scheme="https://ryan961.github.io/categories/mac-tools/"/>
<category term="mac" scheme="https://ryan961.github.io/tags/mac/"/>
<category term="tools" scheme="https://ryan961.github.io/tags/tools/"/>
</entry>
</feed>