-
Notifications
You must be signed in to change notification settings - Fork 0
/
celeryyuan-li-he-shi-yong.html
273 lines (221 loc) · 15.2 KB
/
celeryyuan-li-he-shi-yong.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="author" content="CatsAction" />
<meta property="og:type" content="article" />
<meta name="twitter:card" content="summary">
<meta name="keywords" content="redis, geo, redis, " />
<meta property="og:title" content="Celery原理和使用"/>
<meta property="og:url" content="./celeryyuan-li-he-shi-yong.html" />
<meta property="og:description" content="前述 本篇讲述Celery的原理和基本使用 基本原理 Celery基于Broker传递执行任务，当tasks调用被调用时，先序列化，然后传递到Broker，worker取出来执行。 下面是celery基本的架构。 Broker通常是RabbitMQ和Redis 序列化 Celery通过Python的pickle模块或json序列化。 通过下面的例子简单看看Celery如何序列化Python对象的。 有两个Python模块，task和app。 ## task.py from celery import Celery app = Celery( 'demo', broker='redis://:@127.0.0.1:6379/1', backend='redis://:@127.0.0.1:6379/1') @app.task def add(x, y): return x + y # app …" />
<meta property="og:site_name" content="CatsAction's blog" />
<meta property="og:article:author" content="CatsAction" />
<meta property="og:article:published_time" content="2017-02-11T00:00:00+08:00" />
<meta name="twitter:title" content="Celery原理和使用">
<meta name="twitter:description" content="前述 本篇讲述Celery的原理和基本使用 基本原理 Celery基于Broker传递执行任务，当tasks调用被调用时，先序列化，然后传递到Broker，worker取出来执行。 下面是celery基本的架构。 Broker通常是RabbitMQ和Redis 序列化 Celery通过Python的pickle模块或json序列化。 通过下面的例子简单看看Celery如何序列化Python对象的。 有两个Python模块，task和app。 ## task.py from celery import Celery app = Celery( 'demo', broker='redis://:@127.0.0.1:6379/1', backend='redis://:@127.0.0.1:6379/1') @app.task def add(x, y): return x + y # app …">
<title>Celery原理和使用 ·
CatsAction's blog
</title>
<link href="//netdna.bootstrapcdn.com/twitter-bootstrap/2.3.2/css/bootstrap-combined.min.css" rel="stylesheet">
<link rel="icon" type="image/x-icon" href="./theme/image/logo.ico" />
<link rel="stylesheet" type="text/css" href="./theme/css/elegant.prod.css" media="screen">
<link rel="stylesheet" type="text/css" href="./theme/css/custom.css" media="screen">
</head>
<body>
<div id="content">
<div class="navbar navbar-static-top">
<div class="navbar-inner">
<div class="container-fluid">
<a class="btn btn-navbar" data-toggle="collapse" data-target=".nav-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</a>
<a class="brand" href="./"><span class=site-name>CatsAction's blog</span></a>
<div class="nav-collapse collapse">
<ul class="nav pull-right top-menu">
<li >
<a href=
.
>Home</a>
</li>
<li ><a href="./categories.html">Categories</a></li>
<li ><a href="./tags.html">Tags</a></li>
<li ><a href="./archives.html">Archives</a></li>
<li><form class="navbar-search" action="./search.html" onsubmit="return validateForm(this.elements['q'].value);"> <input type="text" class="search-query" placeholder="Search" name="q" id="tipue_search_input"></form></li>
</ul>
</div>
</div>
</div>
</div>
<div class="container-fluid">
<div class="row-fluid">
<div class="span1"></div>
<div class="span10">
<article itemscope>
<div class="row-fluid">
<header class="page-header span10 offset2">
<h1>
<a href="./celeryyuan-li-he-shi-yong.html">
Celery原理和使用
</a>
</h1>
</header>
</div>
<div class="row-fluid">
<div class="span8 offset2 article-content">
<h2>前述</h2>
<p>本篇讲述Celery的原理和基本使用</p>
<h2>基本原理</h2>
<p>Celery基于Broker传递执行任务，当tasks调用被调用时，先序列化，然后传递到Broker，worker取出来执行。</p>
<p>下面是celery基本的架构。</p>
<p><img alt="Redis Pub/Sub模式" src="./images/celery.png"></p>
<p>Broker通常是RabbitMQ和Redis</p>
<h2>序列化</h2>
<p>Celery通过Python的pickle模块或json序列化。</p>
<p>通过下面的例子简单看看Celery如何序列化Python对象的。</p>
<p>有两个Python模块，task和app。</p>
<div class="highlight"><pre><span></span><span class="c1">## task.py</span>
<span class="kn">from</span> <span class="nn">celery</span> <span class="kn">import</span> <span class="n">Celery</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Celery</span><span class="p">(</span>
<span class="s1">'demo'</span><span class="p">,</span>
<span class="n">broker</span><span class="o">=</span><span class="s1">'redis://:@127.0.0.1:6379/1'</span><span class="p">,</span>
<span class="n">backend</span><span class="o">=</span><span class="s1">'redis://:@127.0.0.1:6379/1'</span><span class="p">)</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">task</span>
<span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
<span class="k">return</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># app.py</span>
<span class="kn">from</span> <span class="nn">task</span> <span class="kn">import</span> <span class="n">add</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'start task'</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">add</span><span class="o">.</span><span class="n">delay</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">18</span><span class="p">)</span>
</pre></div>
<p>我们在终端执行<code>python app.py</code>，连接Redis我们可以看到多了一个celery key，它是Redis的list类型，后加入的放到最前面。查看某一项内容是这样的。</p>
<div class="highlight"><pre><span></span><span class="p">{</span><span class="err">\</span><span class="nt">"body\": \"W1syLCAxOF0sIHt9LCB7ImNhbGxiYWNrcyI6IG51bGwsICJlcnJiYWNrcyI6IG51bGwsICJjaGFpbiI6IG51bGwsICJjaG9yZCI6IG51bGx9XQ==\", \"content-encoding\": \"utf-8\", \"content-type\": \"application/json\", \"headers\": {\"lang\": \"py\", \"task\": \"test.add\", \"id\": \"fd93e14c-984b-4931-87f6-8547eb8f91e7\", \"shadow\": null, \"eta\": null, \"expires\": null, \"group\": null, \"retries\": 0, \"timelimit\": [null, null], \"root_id\": \"fd93e14c-984b-4931-87f6-8547eb8f91e7\", \"parent_id\": null, \"argsrepr\": \"(2, 18)\", \"kwargsrepr\": \"{}\", \"origin\": \"gen11663@localhost\"}, \"properties\": {\"correlation_id\": \"fd93e14c-984b-4931-87f6-8547eb8f91e7\", \"reply_to\": \"b99f0911-fce8-3812-8c71-26f80016b102\", \"delivery_mode\": 2, \"delivery_info\": {\"exchange\": \"\", \"routing_key\": \"celery\"}, \"priority\": 0, \"body_encoding\": \"base64\", \"delivery_tag\": \"d48c84e6-68d5-4403-9aaa-c40849c1d515\"}}"</span>
</pre></div>
<p>启动一个worker</p>
<div class="highlight"><pre><span></span><span class="n">celery</span> <span class="n">work</span> <span class="o">-</span><span class="n">A</span> <span class="n">task</span> <span class="o">-</span><span class="n">l</span> <span class="n">info</span>
</pre></div>
<p>可以看到结果被执行，查看Redis会看到有一个celery-task-meta-taskid的key，它是一个string，里面存放了task执行的结果和状态。</p>
<p>更多Celery的工作流程，查看<a href="https://www.cnblogs.com/ajianbeyourself/p/5551821.html">celery消息的编码和序列化</a>和<a href="https://www.cnblogs.com/cwp-bg/p/8759638.html">Python之Celery使用详解</a></p>
<p><strong>命令</strong></p>
<p><strong>实现机制</strong></p>
<hr />
<aside>
<nav>
<ul class="articles-timeline">
<li class="previous-article">« <a href="./rediszhi-shi-zheng-li.html"
title="Previous: Redis知识整理">Redis知识整理</a></li>
<li class="next-article"><a href="./redissui-bi.html"
title="Next: Redis随笔">Redis随笔</a> »</li>
</ul>
</nav>
</aside>
</div>
<section id="article-sidebar" class="span2">
<h4>Published</h4>
<time itemprop="dateCreated" datetime="2017-02-11T00:00:00+08:00"> 2
11, 2017</time>
<!---->
<h4>Category</h4>
<a class="category-link"
href="./categories.html#redis-ref">redis</a>
<h4>Tags</h4>
<ul class="list-of-tags tags-in-article">
<li><a href="./tags.html#geo-ref">geo
<span>2</span>
</a></li>
<li><a href="./tags.html#redis-ref">redis
<span>11</span>
</a></li>
</ul>
<h4>Stay in Touch</h4>
<div id="sidebar-social-link">
<a href="mailto:catsaction@126.com" title="My Email Address" target="_blank" rel="nofollow noopener noreferrer">
<svg xmlns="http://www.w3.org/2000/svg" aria-label="Mail" role="img" viewBox="0 0 512 512"><rect width="512" height="512" rx="15%" fill="#328cff"/><path d="m250 186c-46 0-69 35-69 74 0 44 29 72 68 72 43 0 73-32 73-75 0-44-34-71-72-71zm-1-37c30 0 57 13 77 33 0-22 35-22 35 1v150c-1 10 10 16 16 9 25-25 54-128-14-187-64-56-149-47-195-15-48 33-79 107-49 175 33 76 126 99 182 76 28-12 41 26 12 39-45 19-168 17-225-82-38-68-36-185 67-248 78-46 182-33 244 32 66 69 62 197-2 246-28 23-71 1-71-32v-11c-20 20-47 32-77 32-57 0-108-51-108-108 0-58 51-110 108-110" fill="#fff"/></svg>
</a>
<a href="https://github.com/boyaziqi" title="catsaction Github Repository" target="_blank" rel="nofollow noopener noreferrer">
<svg xmlns="http://www.w3.org/2000/svg" aria-label="GitHub" role="img" viewBox="0 0 512 512"><rect width="512" height="512" rx="15%" fill="#1B1817"/><path fill="#fff" d="M335 499c14 0 12 17 12 17H165s-2-17 12-17c13 0 16-6 16-12l-1-50c-71 16-86-28-86-28-12-30-28-37-28-37-24-16 1-16 1-16 26 2 40 26 40 26 22 39 59 28 74 22 2-17 9-28 16-35-57-6-116-28-116-126 0-28 10-51 26-69-3-6-11-32 3-67 0 0 21-7 70 26 42-12 86-12 128 0 49-33 70-26 70-26 14 35 6 61 3 67 16 18 26 41 26 69 0 98-60 120-117 126 10 8 18 24 18 48l-1 70c0 6 3 12 16 12z"/></svg>
</a>
<a href="https://www.linkedin.com/feed/" title="My LinkedIn" target="_blank" rel="nofollow noopener noreferrer">
<svg xmlns="http://www.w3.org/2000/svg" aria-label="LinkedIn" role="img" viewBox="0 0 512 512" fill="#fff"><rect width="512" height="512" rx="15%" fill="#0077b5"/><circle cx="142" cy="138" r="37"/><path stroke="#fff" stroke-width="66" d="M244 194v198M142 194v198"/><path d="M276 282c0-20 13-40 36-40 24 0 33 18 33 45v105h66V279c0-61-32-89-76-89-34 0-51 19-59 32"/></svg>
</a>
</div>
<h4>Friendship Links</h4>
<ul class="list-of-tags tags-in-article">
<li>
<a href="https://leetcode-cn.com/" title="力扣中国站点" target="_blank" rel="nofollow noopener noreferrer">
LeetCode
</a>
</li>
<li>
<a href="https://coolshell.cn/about" title="陈皓酷壳" target="_blank" rel="nofollow noopener noreferrer">
CoolShell
</a>
</li>
<li>
<a href="https://yikun.github.io/" title="姜毅坤博客" target="_blank" rel="nofollow noopener noreferrer">
Yikun
</a>
</li>
<li>
<a href="https://www.nowcoder.com/" title="牛客网" target="_blank" rel="nofollow noopener noreferrer">
牛客网
</a>
</li>
</ul>
</section>
</div>
</article>
</div>
<div class="span1"></div>
</div>
</div>
</div>
<footer>
<div>
<span class="site-name">CatsAction's blog</span> - It's a wonderful world
</div>
<div id="fpowered">
Powered by: <a href="http://getpelican.com/" title="Pelican Home Page" target="_blank" rel="nofollow noopener noreferrer">Pelican</a>
Theme: <a href="https://elegant.oncrashreboot.com/" title="Theme Elegant Home Page" target="_blank" rel="nofollow noopener noreferrer">Elegant</a>
</div>
</footer> <script src="//code.jquery.com/jquery.min.js"></script>
<script src="//netdna.bootstrapcdn.com/twitter-bootstrap/2.3.2/js/bootstrap.min.js"></script>
<script>
function validateForm(query)
{
return (query.length > 0);
}
</script>
<script>
(function () {
if (window.location.hash.match(/^#comment-\d+$/)) {
$('#comment_thread').collapse('show');
}
})();
window.onhashchange=function(){
if (window.location.hash.match(/^#comment-\d+$/))
window.location.reload(true);
}
$('#comment_thread').on('shown', function () {
var link = document.getElementById('comment-accordion-toggle');
var old_innerHTML = link.innerHTML;
$(link).fadeOut(200, function() {
$(this).text('Click here to hide comments').fadeIn(200);
});
$('#comment_thread').on('hidden', function () {
$(link).fadeOut(200, function() {
$(this).text(old_innerHTML).fadeIn(200);
});
})
})
</script>
</body>
<!-- Theme: Elegant built for Pelican
License : MIT -->
</html>