Based on what you can see by going to the custom difficult settings, difficulty level affects four things

Starting resources

Number of neutral armies

Growth rate of neutral armies

Aggressiveness of AI

Each of these has 3 levels to it. Easy sets them all at the easiest level, normal at the middle, and hard sets them all at the hardest level. Custom difficulty allows you to combine various levels for each setting (so you could have aggressive AI, but reduce the growth rate of neutral armies).