In 1882, after being sent to France for military study as military attaché, Terauchi was appointed to several important military posts. He was the first Inspector General of Military Education in 1898 and made that post one of the three most powerful in the Imperial Army. He was appointed as Minister of the Army in 1901, during the first Katsura administration. The Japanese victory in the Russo-Japanese War (1904–1905) occurred during his term as War Minister. After the war, he was ennobled with the title of danshaku (baron), and in 1911, his title was raised to that of hakushaku (count).

The annexation of Korea by Japan and subsequent policies introduced by the new government was highly unpopular with large segments of the Korean population, and Terauchi employed military force to maintain control. General Terauchi used the deep historical and cultural ties between Korea and Japan as justification for the eventual goal of complete assimilation of Korea into the Japanese mainstream. To this end, thousands of schools were built across Korea. Although this contributed greatly to an increase in literacy and the educational standard, the curriculum was centered on Japanese language and history, with the intent of assimilation of the populace into loyal subjects of the Japanese Empire.

Other of Terauchi's policies also had noble goals but unforeseen consequences. For example, land reform was desperately needed in Korea. The Korean land ownership system was a complex system of absentee landlords, partial owner-tenants, and cultivators with traditional but without legal proof of ownership. Terauchi's new Land Survey Bureau conducted cadastral surveys that reestablished ownership by basis of written proof (deeds, titles, and similar documents). Ownership was denied to those who could not provide such written documentation (mostly lower class and partial owners, who had only traditional verbal "cultivator rights"). Although the plan succeeded in reforming land ownership/taxation structures, it added tremendously to the bitter and hostile environment of the time by enabling a huge amount of Korean land to be seized by the government and sold to Japanese developers. He was created a Count in the Kazoku in 1911.

The Japanese Governor-General, Count Terauchi, is a very strong and able man, and under his administration many improvements have been made in Korea. This has not always been done without friction between the natives and their conquerors, it must be confessed, but the results are certainly astonishing. The government has been reorganized, courts have been established, the laws have been revised, trade conditions have been improved and commerce has increased. Agriculture has been encouraged by the opening of experiment stations, railroads have been constructed from the interior to the sea-coast, and harbours have been dredged and lighthouses erected. Japanese expenditures in Korea have amounted to twelve million dollars yearly.

In 1916, Count Terauchi became the 9th person to serve as Prime Minister of Japan. During the same year, he received his promotion to the largely ceremonial rank of Gensui (or Marshal). His cabinet consisted solely of career bureaucrats as he distrusted career civilian politicians. During part of his administration he simultaneously also held the post of Foreign Minister and Finance Minister.

The billiken doll, which was a Kewpie-like fad toy invented in 1908 and was very popular in Japan, lent its name to the Terauchi administration, partly due to the doll’s uncanny resemblance to Count Terauchi's bald head.

Duus, Peter. The Abacus and the Sword: The Japanese Penetration of Korea, 1895-1910 (Twentieth-Century Japan - the Emergence of a World Power. University of California Press (1998). ISBN 0-520-21361-0.