I am new to yarn, and i am developing a long running yarn application which when submitted should launch a “driver” and a “application master (AM from now on)”. Much like the way spark does it.
Job of AM :
1. Negotiate with Resource manager (RM from now on) for worker containers.
2. Manage container life cycle.
Job of Driver:
1. Driver to host a web UI.
2. Maintain a database of task submitted from the UI.
3. Broadcast the task to worker containers (without going through AM).
I have gone through a few example yarn applications ( Simple yarn application, distributed shell example yarn application), and both talk about launching a AM, which then launches worker containers assigning task to them.
My doubt is to how to design a Yarn application which can divide the job of AM in two parts of AM and Driver as mentioned above?