Quality/prestige of department vs. quality/prestige of university

I am talking about undergraduate; obviously the former is more important for graduate education. However, I have heard that that is not true for undergraduate education. That certainly makes sense if one in unsure of what to major in, or plans on working after college. My question is: if one is set on a major, and thinks that he will go to graduate school in that field, is the university or specific department more important?

For example, let's say I want to study physics, and I get into UCSB (top ~10 in physics, but not super prestigious overall) and Brown/Penn/Dartmouth (Ivy, but not particularly strong in physics). Which one would I choose?

Also, I've heard that department rankings are based on the graduate program and have little to do with the quality of undergraduate education. However, an undergraduate who wishes to go to graduate school will often do research and/or take a few graduate courses, which has to do with the graduate program and (I believe, correct me if I'm wrong) has an impact on applying to graduate school.