I know what you mean, it is like a reflex reaction when someone tests your reflexes and taps under your kneecap and your leg jerks up. I notice that if I rest my palm on a table and lift my fingers then quickly tap my second or third finger on the table my pinky has a bit of a “knee jerk” reaction.
Since this is an involuntary thing what you need to do is train your pinky to voluntarily stay down. I have never practiced exercises for this but try holding the guitar and hold your fingers about 1/4” to 1/2” above a string. Without plucking, slowly try different finger combinations like 1-2-1-2 or 1-3-1-3 Do this very slowly at first, consciously trying to keep your pinky in place. I recommend to my students to put the preceding fingers down so when you go from 1 to 2 leave the 1 down and when you do from 1 to 3 put 2 down with the 3. If this is really hard at first try them with the pinky on the string at first to get the feel of it not moving when you manipulate fingers 2 and 3. Then gradually lift the pinky off the string and try again.
You can come up with longer combinations on your own, like 1-2-1-3 or 1-3-1-4 and gradually start adding the plucking when you make progress. Good luck!