The first EC album I bought was "Painted From Memory," which is totally awesome but maybe not the best way to get a feel for him overall. I'd recommed the two-disc best of. Now is a really great time to get into EC because he's doing a few re-releases a year, complete with a remastered original album, bonus disc and very in-depth liner notes. I wish more artists would do that kind of thing.

What dsmith said. Try to get his 2CD The Very Best Of (the one with the green Warholian cover, not the 1CD one with the purple cover). It has his best stuff from 1977 to 1998 on it, 42 tracks in total! Based on that you can find out which Elvis period suits your tastes best. He has released so many different albums that it's sometimes hard to believe they all come from the same artist.
If you do want to start with a non-compilation album here's a list of good ones:
- My Aim Is True (1977, by some considered one of the best debuts ever. However, it's without his regular backing band.)
- This Year's Model (1978, with The Attractions. Short, post-punkish songs with a lot of bile and bitterness)
- Armed Forces (1979, the songs are still short and bitter, but the music is a bit more elaborate)
- Get Happy!! (1980, extremely short songs (20 songs in less than 50 minutes!) and a bit of punkish soul)
- Trust (1981, my first EC album and he's getting a bit mellower)
- Imperial Bedroom (1982, very elaborate instrumentation and many consider this Elvis' best)
- King Of America (1986, his Americana record)
- Blood & Chocolate (1986, his back-to-basics record. Raw and bitter again due to his divorce)

I haven't heard his latest one yet (The Delivery Man), but the reviews I've read are positive. Also another note, as mentioned in the previous post, EC has been re-issuing his back catalog. Each album has been remastered and has a bonus disc with alternate takes, outtakes, demo's and live versions plus very in-depth liner notes written by Elvis himself. You should get these as they are great value for money. All of the albums I recommended have been re-released already, apart from King Of America (which will probably be re-released next year).

I hope this makes the choice a bit easier, even though I've listed quite a lot of albums. It's just that he's made so much great music. Many consider his first five albums (My Aim Is True to Trust) the best run of first five albums ever.