forked from ahangchen/dirtysalt.github.io
-
Notifications
You must be signed in to change notification settings - Fork 0
/
7-tips-for-improving-mapreduce-performance.html
50 lines (49 loc) · 3.75 KB
/
7-tips-for-improving-mapreduce-performance.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>7 Tips for Improving MapReduce Performance</title>
<meta name="generator" content="Org-mode" />
<meta name="author" content="dirtysalt" />
<link rel="shortcut icon" href="http://dirtysalt.info/css/favicon.ico" />
<link rel="stylesheet" type="text/css" href="./css/site.css" />
</head>
<body>
<div id="content">
<h1 class="title">7 Tips for Improving MapReduce Performance</h1>
<p>
<a href="http://blog.cloudera.com/blog/2009/12/7-tips-for-improving-mapreduce-performance/">http://blog.cloudera.com/blog/2009/12/7-tips-for-improving-mapreduce-performance/</a>
</p>
<ul class="org-ul">
<li>Configure your cluster correctly
<ul class="org-ul">
<li>文件系统取消 noatime 属性。</li>
<li>使用JBOD而不是用RAID或者是LVM，尤其是在TT和DN上。</li>
<li>mapred.local.dir and dfs.data.dir</li>
<li>If you find that a particular TaskTracker becomes blacklisted on many job invocations, it may have a failing drive.</li>
<li>If you see swap being used, reduce the amount of RAM allocated to each task in mapred.child.java.opts.</li>
</ul></li>
<li>Use LZO Compression</li>
<li>Tune the number of map and reduce tasks appropriately
<ul class="org-ul">
<li>distcp -Ddfs.block.size=$[256*1024*1024] /path/to/inputdata /path/to/inputdata-with-largeblocks 可以修改block size.</li>
</ul></li>
<li>Write a Combiner
<ul class="org-ul">
<li>A job performs aggregation of some sort, and the Reduce input groups counter is significantly smaller than the Reduce input records counter. 这个场景非常适合使用combiner. input records非常多但是groups非常少。</li>
<li>The number of spilled records is many times larger than the number of map output records as seen in the Job counters.</li>
</ul></li>
<li>Use the most appropriate and compact Writable type for your data</li>
<li>Reuse Writables 重用序列化对象
<ul class="org-ul">
<li>Add -verbose:gc -XX:+PrintGCDetails to mapred.child.java.opts. Then inspect the logs for some tasks. If garbage collection is frequent and represents a lot of time, you may be allocating unnecessary objects. 观察GC情况</li>
<li>it may not bring you a gain for every job, but if you’re low on memory it can make a huge difference. 重用序列化对象可以减少内存分配次数，显著改善GC带来的影响</li>
</ul></li>
<li>Use “Poor Man’s Profiling” to see what your tasks are doing</li>
</ul>
</div>
<!-- DISQUS BEGIN --><div id="disqus_thread"></div><script type="text/javascript">/* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * *//* required: replace example with your forum shortname */var disqus_shortname = 'dirlt';var disqus_identifier = '7-tips-for-improving-mapreduce-performance.html';var disqus_title = '7-tips-for-improving-mapreduce-performance.html';var disqus_url = 'http://dirtysalt.github.io/7-tips-for-improving-mapreduce-performance.html';/* * * DON'T EDIT BELOW THIS LINE * * */(function() {var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);})();</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><!-- DISQUS END --></body>
</html>