Earth elementals are found beneath Seers' Village during the Elemental Workshop I and Elemental Workshop II quests. There are several roaming around in the quest area, but you will cause one to appear by mining a rock in the earth room.
They are hardly affected by swords. Pickaxes, however, work excellently, as they are indeed made of rock.

There are two types of Earth elementals: the ones roaming around and the ones you can mine. The first type DO NOT drop elemental ores.