Hi, I manage a hadoop cluster that’s been installed using Ambari where we need to increase & decrease the memory & cpu resources available to node managers daily. The two parameters are
yarn.nodemanager.resource.memory-mb
yarn.nodemanager.resource.cpu-vcores
We would like to do this without having to restart the Resource Manager as that would kill any existing jobs.
We also would like to generate the yarn-site.xml files on all the nodes programmatically based on what other processes we want to run on each node that’s outside of Hadoop.
I have tested that yarn-site.xml can be changed and node managers can be restarted thus changing the resources used on them. The Resource Manager brings in the node with the modified resource allocations without any issues.
The problem is these values will be different from what’s in Ambari and if one of the admins were to restart node manger via Ambari the values stored in the postgres database are pushed out.
Ideally I would also like to run a script to update the postgres database values so that we don’t inadvertently push out the wrong config to a node manager.
Is there documentation available on how to edit and update the postgres database associated with Ambari ?
Or if someone has an alternative solution, I would be very interested in discussing it.
thanks