Most answers rightly point out that a force, in order to achieve what you are stipulating (it changes the direction of the object, not its speed), must be constantly (at each instant) perpendicular. This entails that the object is, due to the applied force, constantly changing its direction and the force is at the same also re-adjusting its own direction, so as to always be pointing in one which has no component in the object's ever-changing direction.
But I would add that you will not see this unless the force in question has a certain modulus, so that the magnitude of the acceleration that it causes is always that of uniform circular motion, i.e. $a = v^2/r$.
In practice you can achieve this by either constantly adjusting the magnitude of the applied force or initially fixing the velocity of the object.
The first thing is what happens when the object is on the end of a string attached to a pivot. In this case, it is held in circular motion by tension, which is (like normal force) a constraint force, meaning that it self-adjusts so as to achieve its aim: normal force has the appropriate modulus to avoid penetration of the object into the ground, while tension exerted by a string also self-adjusts its modulus so as to avoid that the object flies away from circular trajectory and self-adjusts its direction thanks to the pivot around which the string rotates.
The second thing is what happens with gravity. The modulus of the force of gravity is what it is, it does not self-adjust. But then if you fire a projectile from the top of a mountain and you want that it adopts a circular trajectory, following the curvature of the Earth, without ever falling closer to it or escaping away, you must imprint on it the adequate velocity, so that $v^2/r$ matches acceleration due to gravity.