Not only are dirt roads less maintened than paved roads - rarely ever are bushes cut back. Since everything grows very slowly in the desert the wood of any bush is extremely hard and produces nasty scratches on your paint - see image below. Folding back your mirror is a good first step.
As a general rule, it is best to stay as close to the road limitations on the driver's side.