Short version:
The David Gareja site straddles the border between Georgia and Azerbaijan. There was an episode in 2012 where the Azerbaijan authorities stopped tourists crossing to their side. The temporary resolution to that episode is still holding.
Your images demonstrate that you crossed the border, evidently without issue. The caves in your photographs are on the Azerbaijan side. I visited in 2016 without a problem despite meeting border guards from both sides.
Longer version:
The border between Georgia and Azerbaijan in this area follows the crest of a ridge; Georgia is on the northern (shallower) side, and Azerbaijan on the southern (steeper cliff-like) side. While these facts are not disputed, per this paper they do not define the border with enough precision to allow you to know when you cross the border. This has never been better defined because, as Gilles describes, the Georgians want the border moved such that the entire complex is in Georgian territory.
There is no man-made border demarcation at all at the site. There is a low fence that people assume marks the border, but it does not.
There are two monasteries at the site that people visit (ref: Lonely Planet for the area and Google Maps):
- Lavra Monastery is at the bottom of the hill on the Georgian side and is inhabited by Georgian monks.
- Udabno Cave Monastery is primarily located in caves close to the top of the steep southern slope, i.e. mostly on the Azerbaijan side of the border but always within about 50-100 metres of the border line (depending on precisely where the border is).
All visitors arrive from the Georgian side, there is no access from Azerbaijan.
In 2012 there was a brief episode where the Azerbaijanis stopped people crossing to see the caves on their side of the border. I don't think anyone was harmed, they were just turned back. The issue was resolved a coupled of weeks later. This agreement is holding and if it breaks down it is likely that news will come out quite quickly. That said, if visiting Davit Gareja it's worth being aware of the border situation and of the risk.
The UK Foreign Office says:
If you visit the Udabno caves at the Davit Gareja monastery site, take care not to cross the Georgia-Azerbaijan border, which is unmarked in this area.
I imagine other governments give similar advice. I'm not going to advise people to go against their government's advice, but personally I would be more worried about venomous snakes than rampaging Azerbaijani border guards.