Can you use a reaction while being brought to 0 HP?

Question

Our GM threw enemies at us that had a reaction allowing them to counterattack when attacked. (I don’t know the reaction’s wording; he almost never shares such details.) Upon our killing one of them, the GM said, “Oh, and on his way down he hits you for 9.” Do reactions work this way? I would have thought he’s already dead.

Answer 1

I will take a different approach here. But first, answering the actual question, based on the rules:

It depends on the specific wording of the creature's feature

But since you said your DM won't let you have that...

Yes, it's possible.

I mean, he is the DM. He is the one making the rules for your campaign, not the books. Even in the books, some reactions can be triggered after a hit, but before the damage is effectively considered - for example, Shield works like that.

Way more important than the rules

If you feel this is unfair or that the DM is somehow being abusive on his rulings, talk to him. Any action from a creature can be as the DM wishes, given some effort in writing the features, so arguing about rules is usually worthless - it's better to straight forward talk about what's annoying you.

Side note on experiences like this

I had players complaining about some features in my homebrew creatures and saying "Hey, if we can't do that, why can he?" - well, aside the obvious "the dragon can breath fire, you can't" (he wasn't a dragonborn ;P) answer, it was important to explain that NPCs/monsters work differently from PCs, although they follow the same general rules, and just like PCs that have some stuff that make them unique (class features, feats that they choose, etc), my creatures also have features and stuff that make them unique. I might make some mistakes balance-wise and put too much power in these features, it happens surely, but personally it's awkward when a player complains that my creatures can do stuff they can't - I'm not complaining my Kobold can't cast Divine Smite.

TL;DR

Yes, some reactions work like that. Even if they didn't, your DM can create a feature that works like that. Even if he couldn't, your problem probably is not with the rules themselves, so talk to him about the real stuff anyway.

Answer 2

It depends on the type of action

Readied Action

Reactions trigger after the corresponding effect. Being unconscious/dead means you cannot take actions or reactions.

Ready an Action

First, you decide what perceivable circumstance will trigger your reaction.

(Emphasis mine)

This seems to clearly state that, without a specific ability such as Uncanny Dodge or shield, the circumstance must be perceivable. Being hit, before taking damage, is not.

General Effect

In general, a reaction can occur after a hit, but before damage (although this is invalid for Readied Actions). Therefore, it is technically, by RAW, possible for your situation to happen.