Skip to content

Commit d58e22e

Browse files
Wrong Split computation causing less number of container launch
1 parent 2e1af9b commit d58e22e

File tree

2 files changed

+6
-3
lines changed

2 files changed

+6
-3
lines changed

ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -359,8 +359,11 @@ public List<Event> initialize() throws Exception {
359359
splits[0] = new HiveInputFormat.HiveInputSplit(fileSplit, partIF);
360360
}
361361
} else {
362-
// Raw splits
363-
splits = inputFormat.getSplits(jobConf, numSplits.orElse(Math.multiplyExact(availableSlots, (int)waves)));
362+
// Raw splits.
363+
splits =
364+
inputFormat.getSplits(
365+
jobConf,
366+
numSplits.orElse((int) Math.min(availableSlots * waves, Integer.MAX_VALUE)));
364367
}
365368
// Sort the splits, so that subsequent grouping is consistent.
366369
Arrays.sort(splits, new InputSplitComparator());

ql/src/test/results/clientpositive/llap/mm_all.q.out

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1998,7 +1998,7 @@ Table Type: MANAGED_TABLE
19981998
Table Parameters:
19991999
COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
20002000
bucketing_version 2
2001-
numFiles 16
2001+
numFiles 27
20022002
numRows 500
20032003
rawDataSize 5312
20042004
totalSize #Masked#

0 commit comments

Comments
 (0)