Upon Emperor Otto's death in 973, Henry could rely on his ties to the South German duchies of Swabia and Bavaria as well as to the adjacent Kingdom of Burgundy. He installed his cousin Henry as Bishop of Augsburg, denying the investiture rights of Emperor Otto's son and successor Otto II. When his brother-in-law Duke Burchard III died without heirs, he raised claims to his Swabian duchy. However, Otto II enfeoffed his nephew Otto of Swabia; against the tenacious opposition of Burchard's widow Hadwig.

When in 983 Otto II suddenly died from malaria in Rome, Henry was released from captivity. He once again tried to usurp the German throne, when he abducted the infant Otto III and, according to the medieval chronicler Thietmar of Merseburg, had himself proclaimed King of the Romans at the graves of Emperor Otto I and King Henry the Fowler in Magdeburg and Quedlinburg. However, it turned out that he had lost the support of the German dukes and also was not able to oust Duke Henry the Younger from Bavaria.

Through the agency of Archbishop Willigis of Mainz, Henry in 985 finally submitted to Empress Theophanu and her mother-in-law Adelaide at an Hoftag assembly in Rohr. Although he failed in his attempt to gain control of Germany, he did regain Bavaria and in 989 also received the Carinthian duchy.