How to prevent exiting camera viewpoint when rotating (but still allow pan and zoom)

Question

I am using reference cameras that have background images. I want to be able to prevent the view from exiting the camera viewpoint and reverting back to a free perspective camera.

I do not want to lock the view to the camera to move the camera (it is meant to stay where it is). I just want to be able to zoom + pan to get close to parts of the background image but disallow the rotating that exits the view. No combination of locking transforms or camera to view locks seem to do it.

Thanks in advance.