how to formulate this same problems as a transfer function or in the frequency domain? I mean I could compute the transfer function for the pendulum equation.

The range of motion involved in bringing a pendulum from down position to up position means that the systems needs to be treated as a non-linear system unlike the stabilise-in-the-up-position for angles near to the equilibrium case. So transfer function approaches may not exist as they are intended for linear systems.

I am not sure if a PID controller could find a suitable control law to bring the pendulum to the top?

IMO, this should be possible by intentionally designing a controller to make the closed system unstable for the down position. The oscillations of the the unstable system would grow (if designed properly) until the amplitude reaches 180 deg; i.e. up position.

But I am still not clear how this swing-up problem is modelled in the classical control sense.

This may be because of the range of motion prevents it from being represented as a linear system. I don't know if there exists any literature which tackles swing-up problem using classical methods.

how to formulate this same problems as a transfer function or in the frequency domain? I mean I could compute the transfer function for the pendulum equation.

The range of motion involved in bringing a pendulum from down position to up position means that the systems needs to be treated as a non-linear system unlike the stabilise-in-the-up-position for angles near to the equilibrium case. So transfer function approaches may not exist as they are intended for linear systems.

I am not sure if a PID controller could find a suitable control law to bring the pendulum to the top?

IMO, this should be possible by intentionally designing a controller to make the closed system unstable for the down position. The oscillations of the the unstable system would grow (if designed properly) until the amplitude reaches 180 deg; i.e. up position^{See comment below.}.

But I am still not clear how this swing-up problem is modelled in the classical control sense.

This may be because of the range of motion prevents it from being represented as a linear system. I don't know if there exists any literature which tackles swing-up problem using classical methods.