I have a project using Git where I've branched off of master to rename a directory.

The rename in the branch works as expected. When I switch back to the master branch the directory has its original name, but there's also an empty directory tree with the name that I changed it to in the branch.

Is this the expected behavior?
Am I missing a step?
Do I just need to delete these empty directory trees as they appear?

I know Git doesn't track empty directories and that may be a factor here.