I ended up going the Cloudera Manager route since I was more familiar with it and hadn’t received any update on this thread, which made me nervous about moving forward with it. I was in a unique situation where I was expanding the cluster at the same time. Here’s what I did from a high level:
1. Use the new hosts as the base of a new cluster then decommission a few of my existing nodes so I still enough power to handle my existing load but such that I had roughly an even split between old nodes and new nodes.
2. Install CDH (or HDP if I had ended up using Ambari) on the “new” cluster.
3. Configure the system to be a mirror image of the old cluster. Use distcp to copy all HDFS files over, export all tables and import them into the new cluster, etc.
4. — Start Outage —
5. Cut the services over to the “new” cluster.
6. Verify the services can connect and data is available.
7. Perform an incremental update on the data.
8. — End Outage —
9. Migrate the rest of the nodes to the new cluster.
Again this likely won’t work for everyone since it would be rare to be doing a hardware upgrade at the same time but it worked perfectly for us. We’ve been running happily with a managed cluster for about a month now.