The first time, he left after he found out Sami had tried to deceive him by changing Theo's paternity test. The second time, Sami was torn between Lucas and Brandon, so Brandon bowed out and left so lumi could create a stable family for Will.