I also started with .22 rifles and now have 7 of them.
When I started to acquire some larger calibers I also had some concerns about recoil.

I had some bad initial experiences with a old Springfield 1903 in .30-06 but I found out that it had more to do with the narrow steel butt pad along with poor rifle placement at my shoulder and too loose a hold.

I couldn't begin to understand rifle placement when shooting a .22LR. You really learn why it's important if you shoot a .300 Win Mag and hold it as loosely as you hold a .22LR.

Once I got rifles with good butt pads, I found that the experience was based on the rifle placement and not the caliber and my concerns went away.
I did have to learn to manage recoil so the rifle stayed on target but that just came with practice.

Noise can be as much of the problem with flinching as the actual rifle movement. Make sure you use good hearing protection so the noise won't become the factor.

After many years, I now have bolt rifles in .223, .22-250, .270, .308, and .30.06 and semi autos in 7.62x39, .308 and .30-06.
The long actions bolt rifles, .270 and .30-06, recoil the most, but the recoil hasn't been a problem even as I am getting older.

In fact, rapidly approaching 70, I am shooting as well or better than I ever have.
I guess lots of practice really nelps.