For me, anyway, it's mainly practice and experience.
Practice, because your pitch recognition gets better and better. This can not be overemphasised. Sitting down and trying to work out how to play "twinkle twinkle little star" or "Oh Christmas tree" or something like that is incredibly valuable to your musical development. Translating familiar melodies (the more familiar and simpler the better) is a great place to start, in training your ears to relate your subjective experience of pitch and melody, to an objective understanding of what those notes actually are.
That's something that comes with practice; the more you use it, the better your pitch recognition is.
So for example, if you sing me any 2 notes in sequence, then I'll be able to tell you what the interval between them is. That comes with practice. If you play 8 random notes at me, one after the other, then I'll be able to play them back to you. That's practice. If I hear a song on the radio, I'll be able to play the melody back to you, more or less the first time. That's both practice and experience.
Practice because that gives you good pitch and interval recognition as I outlined above, but also experience, in that the more you play and work out music by ear, the more you start to recognise patterns that notes often fall into. On top of that, as well as hearing each note in relation to the one that came before it (which is what I'd be doing with a series of 8 random notes), you relate it to the music context it comes in. It's easier to work out things that fall into patterns you understand and recognise.
Put it this way, imagine you were trying to acquire the skill of copying photographs. You would obviously get better at painting in general over time, better at manipulating the brush, better at understanding how to produce the colour and lighting that you want with paint. But you would also get better at painting specific things, better at painting trees, better at painting water, better at painting sunlight etc. Because you'd be doing the same sorts of shapes each time. Reading books about how to paint trees certainly would help you progress faster, but most important would just be the practice! And of course, let's say I give you a photograph and you have to copy it from memory, without looking (which is more like what working out music is like, unless you are spamming the replay button like a m*****f***a). A painting of something you recognise, and know how to paint would be much easier to reproduce then something you'd never seen before!
So, to give a simple example, if you listen to a lot of blues music, you'll notice that there are certain licks that are often used to end a phrase, and certain chord changes and flourishes which pop up again and again, and with different little variations. So then when you go to play along with a blues song, when you hear a phrase that's the same, or similar, or related to something you already know well, you don't have to work it out from scratch because you already know what it is. More abstract from that, you start to notice more more generalised principles of what kinds of notes tend to go together in what context, and what those contexts all sound like, and so as well as recognising specific note patterns, you also recognise a sort of "framework" of possibilities based on what the music sounds like.
Formal music theory can help this of course, you might here a melody and go "oh that's an ascending major scale starting on the third, with a flat 7 passing tone" or whatever. But it still all comes down to playing real music by ear, and theory is only as good as how it relates to your actual subjective understanding of the music you're hearing and playing. Theory can then give you ways to abstract and analyse and make sense of that music, which is of course useful because it allows you to see even more patterns, some of which you might not have noticed on your own.
The real benefit of getting good at playing by ear, is that you get music theory lessons for free, all the time. Whenever you properly listen to music, and pay attention, you notice notes and chords as they come, and so it's impossible not to learn a little about ho music works just from hearing music played. Of course, I'm not saying you hear every single note of every complex piece of music, and the more complex and more unfamiliar music is, the less of it that you pick up just from listening, but of course, that's where the practice part comes in!