There is no RAW for the targeting
Dude, you've been done wrong.
And not by your DM, but by the rules system itself. The Sanctuary spell reads simply enough:
Until the spell ends, any creature who targets the warded creature with an attack or a harmful spell must first make a Wisdom saving throw.
All we need to do to adjudicate this is figure out, did you (perhaps inadvertently) target the enemy when you targeted their armor? And that would be a simple question if 5e had rules about what constitutes spell targeting, but it doesn't. We know that Making an Attack requires selecting a target (creature, object, or location), and that is pretty straightforward. But there are many, many spells that are not attacks (which, like heat metal, don't involve attack rolls) and in these cases what they actually target is not clearly defined. Most rules interactions assume that we know what 'the target' of a spell is, and yet we don't actually know what counts as the target of a spell (I highly recommend you read the linked Q&A). The spell description does not explicitly state what the target of a spell is, and we thus have to infer it.
The classic example is the Area of Effect spell fireball, which says (emphases mine):
A bright streak flashes from your pointing finger to a point you choose within range and then blossoms with a low roar into an explosion of flame. Each creature in a 20-foot-radius sphere centered on that point must make a Dexterity saving throw. A target takes 8d6 fire damage on a failed save, or half as much damage on a successful one.
Most people would say that the target of the spell is 'a point in space' and that the caster targets that point. But the creatures that take damage are explicitly referred to as targets - did the caster target them as well? Or did the spell target them independently of the caster? Can I include a creature under sanctuary in the area of my fireball without having to save against sanctuary? Can I do so if I explicitly choose the point in space to be centered on the protected creature? Does it matter whether or not I know that the creature is included in the area of the spell? Answering these questions requires interpreting the rules, precisely because the fireball spell does not tell us explicitly what the target(s) of the spell are.
Your case with heat metal is similar (emphases mine):
Choose a manufactured metal object, such as a metal weapon or a suit of heavy or medium metal armor, that you can see within range. You cause the object to glow red-hot. Any creature in physical contact with the object takes 2d8 fire damage when you cast the spell. Until the spell ends, you can use a bonus action on each of your subsequent turns to cause this damage again.
Nowhere in the spell description are we told explicitly 'you target X and only X when you cast this spell'. Rather, we have to infer what the target is, which is the source of your conflict with your DM. Here, you and your DM both appear to have what is called 'motivated reasoning'. You want to damage the enemy, so you are reasoning that you chose the only target of the spell as the enemy's armor, not the enemy, and have thus not violated the Sanctuary. Your DM wants the enemy to not be damaged, so they look at your claim like we might look at someone who says 'yeah, I shot the gun, but I didn't kill that guy - the bullet did'. To your DM, choosing to target the armor after the sanctuary was up is the same as choosing to damage the enemy - it is targeting the enemy with damage.
Unfortunately, since the 5e rules in general, and the heat metal spell in particular, do not tell us what the target of the spell is, we have no way of knowing which of you two is 'right'. In a case of conflicting interpretations, we have to assume that the DM's interpretation is correct. You can argue the point with them, but ultimately they get to decide how their world works.
However, you don't 'lose' your whole spell
On the other hand, even if the sanctuary would protect your enemy from your application of heat damage, that does not end your spell.
Sanctuary says:
On a failed save, the creature must choose a new target or lose the attack or spell.
Your DM (perhaps because of their motivated reasoning) is assuming that "lose the...spell" means that if you fail the Wisdom save on any of your subsequent bonus actions to have the object radiate heat damage, that the entire spell itself is canceled. This would be true if the save was made as you cast the spell (and if the sanctuary actually prevented you from targeting the armor, see above). But in your case you have already cast the spell; it has already 'stuck'. Your bonus action is not trying to target the spell itself on the armor, rather it is trying to assign more damage coming from the object. Even assuming that this counts as targeting, it is only this one bonus action that would be lost, not the whole spell itself.
This can be seen from the spell description, where it says you lose 'the attack' or spell. If you attack with a sword, the sword does not disappear on a failed save, nor do you conclude 'I guess I can never use that sword again'. Rather, what is lost is that particular attack. If a monster with multiattack failed a save against sanctuary on their first attack, they would not lose all the subsequent attacks, just the first one, and they could try to attack the sanctuaried defender with their next attack (attempting another save).
We should not expect the protection to be stronger against spells than against weapon attacks when it is included in the same clause. If the spell you used was coming into being when it came up against a failed save, the whole spell would be lost, but if you have a pre-existing spell, like a pre-existing sword, you don't lose the spell itself, merely one of its "attacks" (or in your case, one chance to assign damage).
