yep, the only point of the "swelling" is to verify that you've got viable yeast in the smack pack. If you're making a starter (which you should pretty much always do), you'll verify the viability with it instead so no need to wait for the swelling.