Quantcast
Channel: Hortonworks » All Replies
Viewing all articles
Browse latest Browse all 3435

Reply To: Simple Hive query becoming slow with a lot of data

$
0
0

hm ok thank you!

I have 60 Map Tasks for 15GB and 427 for 100GB. And 16 Reducers and 111 for 100GB. Which i think is way too much but it seems like hive is not doing that partial aggregations within the big data since the reduce phase is really fast like seconds when i use combiner in my Mapreduce but takes forever with hive.

With Mapreduce i have like 32 Mappers and 1 Reducer for 15Gb and 212 Mappers and 1 reducer for 100Gb, i increased block size to 512MB because its faster.


Viewing all articles
Browse latest Browse all 3435

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>