Combining a cardigan and a button up shirt is a really nice look that can be pulled off easily. My suggestion would be to wear a darker colored (maybe even earthy toned) cardigan over a colored shirt.
http://www.thesartorialist.com/