How to Optimize YARN
Guide to tune and optimize YARN
You might have built your hadoop
cluster with super power hardware, but if you don’t tune YARN configurations
you are underutilizing your cluster. At this point I assume you are aware of
what is YARN, if not read through.
YARN – Yet another resource negotiator, this is Map-reduce version 2 with many new features such as dynamic memory assignment for mappers and reducers rather than having fixed slots etc.
What is Container?
Container represents allocated Resources
like CPU, RAM etc. It’s a JVM process, in YARN AppMaster, Mapper and Reducer
runs inside the Container.
What are these properties ? What can we tune ?
yarn.nodemanager.resource.memory-mb
yarn.scheduler.maximum-allocation-mb
yarn.scheduler.minimum-allocation-mb
mapreduce.map.memory.mb
mapreduce.reduce.memory.mb
For Hortonworks users:
For Cloudera Users:
Thanks for sharing,this blog makes me to learn new thinks.
ReplyDeleteinteresting to read and understand.keep updating it.
Android Training Institute in Chennai | Android Training Institute in anna nagar | Android Training Institute in omr | Android Training Institute in porur | Android Training Institute in tambaram | Android Training Institute in velachery