Your rig and the server, together. Well your rig seems barely meet the minimum requirement, and the server does lag a lot (At least for Gysahl server, we hit 3,300 online population before yesterday server crash). Didn't you run the benchmark?

I had a similar problem and thought I would buy a new PC to solve it, so I gave my old one to my daughter who is at college, she can play FFXIV fine in her house, same comp, same router and settings, same broadband provider, same everything that I can tell, but the difference in playability is quite marked. Why!?! So...maybe you need a new house just like me!?

Graphic lag and server lag are two very different things. If you use a skill and it takes 10 seconds for the skill to activate, that's server lag. If you walk into a room with 4 npcs and only 5 players, but it takes a minute for the npcs/players to pop up, that's graphic lag.

Download a program that will display your framerate in-game, such as the free version of FRAPS. Then just watch your framerate. If it is staying around ~30 then you are pretty good and you are mostly experiencing server lag. However, if your framerate is constantly dropping below 20 or even 10, then it is time to upgrade something.