That is to say, are there cells that, between infancy and adulthood, get larger? Or is all growth done entirely via cell division? I'm wondering if it is safe to assume that the approximate number of cells per unit mass in a mammal will remain fairly constant throughout its lifespan.

Metaplasia - one type of specialised cell is replaced with another, usually more durable one, E.g. columnar epithelium in the respiratory tract of a smoker being replaced with squamous epithelium.

The first three responses to stress can change both cell size (auxetic response) and number (multiplicative response), meaning you can't really say that cells per unit mass stays constant throughout life.

Muscle tissue grows predominantly by hypertrophy, meaning that the muscle gains during puberty are likely to be more as a result of cells getting larger than the development of new ones.

Both auxetic and multiplicative growth occurs as the body develops. In addition to this, accretionary growth also occurs - where connective tissues such as bone and cartilage increase in size.

This is not my field so I am sure there are other examples, but certain neurons will definitely be larger in adulthood than in infancy. There are motor neurons that connect the spine to, for example, the toes. These will grow in length as an animal grows. So, in a human infant they will be a few centimeters long and can reach lengths of over a meter in an adult.