Further update:
What is happening is that the folder structure is getting written inside a virtual folder in the bucket. I am aware that s3 has no concept of folders but that is how it appears in file browser UI.
Once the job has failed, the path of the folder is <BUCKET_NAME>//2HBASE-SNAP-X . Notice the double forward slash there which is different to the job that is trying to write to <BUCKET_NAME>/2HBASE-SNAP-X
Why is this virtual folder being created and how can I get it to write to the correct path?