There's one major danger of going one size lower.
Generally, frames vary according to three characteristics, with each measurement increasing as you go up the frame sizes:
- Seat tube height and therefore standover clearance
- Top tube length
- Handlebar height
If you go one size lower, the handlebars will be very low and you may not be able to raise them to a reasonable level. Especially with today's ahead stems, this is a major limitation. Also, in road bikes it's very trendy to have ridiculously low handlebars, so low nobody but a competitive cyclist is able to ride them. If you do purchase a drop bar bike, prefer to buy an endurance road bike, or a touring bike, or a gravel bike. Non-endurance road bikes are from hell.
When I purchase a bike (and I would never purchase a non-endurance road bike, but even then I find that on typical bikes such as gravel bikes or endurance road bikes the handlebars are too low on the stock installation), I usually purchase bit larger frame than what would be ideal to me. Because today's top tubes are not fully level but rather we have "compact" frame geometry, this means that standover clearance will be just barely enough, not too little. Then I swap the stem to a shorter one in order to eliminate the problem of having too long frame, and install it such that all spacers are below the stem and the angled stem is angled up.
I don't know where you are non-flexible. I'm also non-flexible, if I have to ride with a low handlebar every part of my body hurts. I have never found raising my leg over the saddle and top tube to be any limitation for me.
If you absolutely have to buy one size lower to be able to raise your leg over the bike (which I won't believe but then again we all are different), you almost surely need a 35-degree stem that you can angle up to eliminate the effects of too small frame. Here the problem is that the handlebars in the stock installation will be so low you have absolutely no way of raising the handlebars to any reasonable level. Thus, you are unable to estimate what length 35-degree stem you need, because the handlebars are too low and the riding position is therefore so different from your final riding position. Therefore, most likely you need to purchase two new stems: one 35-degree stem which you use as a trial stem, and then when you know if the stem is much too short, or little too short, or correct length, or little too long, or much too long, you buy the correct length 35-degree stem. If you're very lucky (chance maybe 20%) then the first 35-degree stem you buy was the optimal length.