According to the rules of English grammar, in that sort of situation, a relative pronoun is always required, so dropping the "who" or "that" is technically not grammatically correct. However, in terms of regular usage, that sort of omission is actually not uncommon, and I don't think would actually sound particularly strange to anyone.
So, if you are trying to be as correct as possible (for example, in formal writing, or when speaking to an English teacher, etc) it would probably be best not to leave it out, but in general conversation it is probably perfectly fine.
Note that your reasoning for why it is required is not quite correct. Technically it is also required even if you're just talking about "a friend". The "of mine" doesn't change things here. It just happens that it's very common for people to omit the pronoun in a short sentence like "He is a friend I love", but they will more often include it in a more complicated phrase such as "He is a friend of mine whom I love" (I suspect mostly because the longer phrase is more complex so more helper words can make things easier for the listener to sort it out, and also being a longer phrase already, the addition of one more word isn't as big a deal).
Note also that, technically, if you want to be grammatically correct, it should be "He is a friend of mine whom I love", not "who" (since the phrase before the pronoun is acting as the direct object of "I love", not the subject, the correct pronoun is "whom" (object)). Also note, however, that in reality almost nobody is pedantic enough to make that distinction anymore, and lots of people don't even know the difference...
