Living conditions:
Georgia Tech:
Atlanta is a big city in the southeast United States. It's hot, humid, and constantly trying not to get eaten by kudzu vines that were unwisely planted for animal fodder back in the 19th century. Cost of living is pretty reasonable, from what I've heard, and the graduate stipend should be sufficient if you're single and living with a few roommates, and it's probably okay if you're married without children. Being in the south, you'll have access to all the southern comfort foods that are guaranteed to make you gain fifty pounds overnight, but being in a big cosmopolitan city, you'll also have access to all sorts of cultural sophistication.
Ohio State:
Columbus is further north and is considered part of the midwest. You'll get warm, humid summers and moderately cold, snowy winters. The cost of living is probably pretty low; it's a smaller city than Atlanta (although by no means is it small), and it hasn't experienced the massive housing boom that has hit some other places in the US. It's probably not quite the cultural hotspot that Atlanta is, but there should be enough of the comforts of life for a physics student to be happy.

Reputations:
Overall, Georgia Tech is a more prestigious school than OSU (top 30 vs top 70 in the US), but they're pretty comparable in terms of physics (Georgia is #28 and OSU is #23 according to the US News rankings). More people have probably heard of OSU because they have a pretty strong football team, but that's not going to matter to physicists nearly as much. Georgia Tech has one of the biggest and most active gravity programs in the country at the moment, so they have that going for them. I don't know anything about astroparticle physics at OSU (I didn't apply there like I did Georgia Tech). I would ask professors you know in both fields about what they think.

So I should care about physics ranking of the university, even if it's overall ranking is lower? For example, I've noticed that Colorado Boulder is considered as one of the top programs in physics, although their overall ranking is not even in top 100. Does this mean it's better to do physics Ph.D in CU Boulder than, say New York University which has a better overall ranking?

Are you sure that Georgia Tech's gravity program is that strong? They have a very small group of three professors. I've investigated about them, and they don't seem to be publishing many papers with their graduate students, and their alumnus don't have many published papers, neither are they working or doing post-docs in top schools.
Thanks for the reply by the way, you were very helpful.

Their gravity program is a bit bigger than three professors. Even if it weren't, that's still a pretty sizable group for gravitational physics (if you include all of Chicago's cosmology people, they've got five professors in their top-notch gravity program, and a couple of them are older than dirt). They've got three professors listed specifically under "gravitational physics with gravitational waves," but they've got other gravity research (mostly in cosmology) going on and strong connections to LIGO. And all three of those professors listed under gravitational waves are very highly regarded in the field, at least going by citations.

There are a few things about physics graduates you should remember:

Your average tenured professor has about ten PhD students during his or her career.

The number of tenured physics faculty across the nation from year to year is more or less constant.

Somewhere around half of tenure-track faculty positions in the US (I don't know if it's for physics or just professorships in general) graduated from one of the Ivy Leagues, Stanford, MIT, Caltech, or a comparable university.

Until 2016 and the LIGO announcement, almost nobody cared about gravitational physics. According to my research advisor, he was one of around 5 GR hires in the entire nation when he was hired in the early 2000s.

To make a long story short, roughly 1 in 10 PhD students get tenure-track faculty positions, and maybe 1 in 20 for your typical GR student (if I had to guess). A similar number end up at national labs, and a lot of them will be working on classified or otherwise restricted projects that can't legally be published in typical scientific journals. Most of the rest end up in industry, where even physics-related positions aren't going to lead to many publications.

So I should care about physics ranking of the university, even if it's overall ranking is lower? For example, I've noticed that Colorado Boulder is considered as one of the top programs in physics, although their overall ranking is not even in top 100. Does this mean it's better to do physics Ph.D in CU Boulder than, say New York University which has a better overall ranking?

Yes. To go with your example of CU Boulder: they might not be known for much else outside of physics, but the only institution with a comparable AMO program, for example, is MIT. They've had multiple Nobel laureates on their faculty, and they're closely associated with NIST through JILA. On the other end of the stick, my undergraduate institution has top-notch business (particularly accounting, of all things) and law programs, but the graduate physics program isn't really very big or well-known.

[/quote]
So I should care about physics ranking of the university, even if it's overall ranking is lower? For example, I've noticed that Colorado Boulder is considered as one of the top programs in physics, although their overall ranking is not even in top 100. Does this mean it's better to do physics Ph.D in CU Boulder than, say New York University which has a better overall ranking?
[/quote]

Yes, exactly. Overall ranking of university is totally irrelevant with PhDs. Do pay attention to their specialties though. For example, the university of Rochester is one of the best in the countries for AMO, but is not good at all for HEP-th.