I could consistently reproduce this issue.
For some reason, as part tests/rename/10.t , mknod()-> glfs_h_mknod() fails with
EEXIST error. In NFS-Ganesha, on receiving this error, does a lookup of the object for which it receives ENOENT. Since it is in inconsistent state, it keeps retrying the fop resulting in hang on the client side.

This seem to be an issue with dht-tier. For every file creation, dht-tier seem to be creating linked files on cold tier first and then actual files on hot-tier. If for any reason actual file creation fails, linked files are not deleted from the cold tier (as can be seen below)
# ls -ltr /bricks/xfs_brick/b*/dir_tmp/fstest_3e00dbbaaeb95ee267e1a3b59a821b34/fstest_dc09c43ca69e030460a777fac05f25d7
/bricks/xfs_brick/b2-tier/dir_tmp/fstest_3e00dbbaaeb95ee267e1a3b59a821b34/fstest_dc09c43ca69e030460a777fac05f25d7:
total 0
/bricks/xfs_brick/b1-tier/dir_tmp/fstest_3e00dbbaaeb95ee267e1a3b59a821b34/fstest_dc09c43ca69e030460a777fac05f25d7:
total 0
/bricks/xfs_brick/b2/dir_tmp/fstest_3e00dbbaaeb95ee267e1a3b59a821b34/fstest_dc09c43ca69e030460a777fac05f25d7:
total 0
---------T 2 root root 0 Dec 7 13:18 fstest_cbf533920c1896845883fb493fce0d06
/bricks/xfs_brick/b1/dir_tmp/fstest_3e00dbbaaeb95ee267e1a3b59a821b34/fstest_dc09c43ca69e030460a777fac05f25d7:
total 0
---------T 2 root root 0 Dec 7 13:18 fstest_cbf533920c1896845883fb493fce0d06
This led to the inconsistent behaviour seen by the nfs-ganesha (mentioned in the above comment).
Exclusive file creation with the same file path shall result in ERR_EXISTS error since linked files are present and LOOKUP shall fail with EINVAL error since the actual files are missing.
]# ls fstest_3e00dbbaaeb95ee267e1a3b59a821b34/fstest_dc09c43ca69e030460a777fac05f25d7/fstest_cbf533920c1896845883fb493fce0d06
ls: cannot access fstest_3e00dbbaaeb95ee267e1a3b59a821b34/fstest_dc09c43ca69e030460a777fac05f25d7/fstest_cbf533920c1896845883fb493fce0d06: No such file or directory
# mkfifo fstest_3e00dbbaaeb95ee267e1a3b59a821b34/fstest_dc09c43ca69e030460a777fac05f25d7/fstest_cbf533920c1896845883fb493fce0d06
mkfifo: cannot create fifo `fstest_3e00dbbaaeb95ee267e1a3b59a821b34/fstest_dc09c43ca69e030460a777fac05f25d7/fstest_cbf533920c1896845883fb493fce0d06': File exists
This seem to be known issue with tiering. Request Rafi/Nithya to mark this bug duplicate (if any present already)