Should I even bother or leave it like this:
CFLAGS="-march=native -O2 -pipe"

Don't bother, and stick to -march=native.

and -march=native will revert automatically to the best suited -march, which will be core-avx-i.

Long answer:
the improvements between sandy bridge and ivy bridge mainly concern the avx (or sse) part of the cpu, which their vector processing unit for floating point. The only programms that make extensive use of that are scientific ones (number crunching), multimedia applications (steam conversion, image manipulation) and games (though they are rarely self-compiled).

If you are using one of those programs you might see a difference in speed, but it hard to say if that improvement is measurable, let alone noticeable for a human being (you need ~30% of speed improvement to be noticed as human)

HTH
V._________________read the portage output!
If my answer is too concise, ask for an explanation.