This question is prompted by this answer, which argues that Animal Handling is a wrong skill for managing an INT 3 monstrosity, a Grick tamed by some goblins.
A more common example case of this might involve handling a Griffon mount.
If a player wants to handle (approach, pacify, tame) such a monstrous mount/beast of burden/pet, what ability check should a DM call for?
In your example of a Griffon, they have the following text in their decriptions:
Once trained, a griffon is a fierce and loyal steed. It bonds with one master for life, fighting to the death to protect that rider.
[Emphasis mine]
This suggests that a second person would not be able to really interact with even a trained griffon mount. These are not horses which will take any rider.
There are more monstrosities than any answer here can go through, and some may be tameable like a horse, but generally short of being the one that raised one from birth it is unlikely you can succeed at all.
I can't actually find the rule that says as much, but if there is no chance of success (or no interesting consequence of failure), don't roll.
As a DM you are free to override the default assumptions so maybe in your world a griffon will submit to anyone once it has been trained, and you need to work out how to do this.
Look at the list of skills and see if any might be useful. I am not going to list them all here, but the few that might be useful (in my opinion) are as follows:
Nature This sounds like it has potential uses, but the PHB suggests this is to recall lore about nature, so this is out.
Survival The PHB suggests this can be used to find signs of an owlbear, so clearly has some interaction with monstrosities, but only tracking, so this is out.
Persuasion Maybe you can talk to a monstrosity? Well this is for talking to people specifically, so again, by the PHB this is out.
Animal handling And this is where I have to disagree with the linked answer. The logic there is that a beast being an ordinary animal means that nothing else can be an animal, but that isn't correct. It leaves a whole classification of extraordinary animals to be classified, and those, at least some of those, could well be monstrosities.
If you rule out animal handling as being a useful skill here, then effectively you are saying there is no useful skill, but someone tamed that griffon in the first place, so there must be at least one useful skill. The trick isn't to try and rule every skill out based on a strict reading, but to find the most closely related skill - and that is animal handling.
I completely disagree with the suggestion that "animal handling" applies only to Beast-type creatures. In prior editions, that creature type was directly called "Animal", and they've renamed it to Beast specifically to remove the implication that this one creature type contains all "animals".
Furthermore, the actual description of the skill says quite the opposite:
Animal Handling. When there is any question whether you can calm down a domesticated animal, keep a mount from getting spooked, or intuit an animal's intentions, the DM might call for a Wisdom (Animal Handling) check. You also make a Wisdom (Animal Handling) check to control your mount when you attempt a risky maneuver. (Player's Handbook, p.176)
Specifically, animal handling applies to any attempt to calm or control a mount, and it makes no mention of that mount being a Beast or being domesticated. If you're on a mount, you use Animal Handling to control it, and that's as firm a ruling as any of the skill descriptions will give you.
Quite a few creatures called out as being commonly used as mounts have the Monstrosity type, including the Griffon and Hippogriff; and there are others like the pegasus and nightmare that are neither Beast nor Monstrosity. The creature type simply doesn't enter into it.
Does that mean you can use Animal Handling to train and control a grick? Well, not necessarily -- but you can't necessarily use Animal Handling to train and control a wolf either. The given use-cases were calming domestic animals, figuring out what an animal is thinking, and keeping your mount under control. None of those actually cover "make a non-domestic animal do what I want it to do", so we're in the realm of DM adjudication. It's up to the DM to decide when "animal handling" is a viable option for a proposed task. (For that matter, "teach my dog to sit and roll over on command" isn't specifically covered by the skill description, so even that would technically be a DM ruling.)
If you want to calm a griffon or spend downtime to tame a rust monster, I certainly can't see any skill check that would make more sense than Animal Handling, but it's entirely possible that at your table, your DM has another skill in mind, or thinks no skill applies to such tasks and wants a straight ability check. It's also entirely possible that your DM will simply rule that it's not possible to do these things, and they wouldn't be going against the rules as written. But that isn't unique to Monstrosities or based on creature type; a DM could easily rule that you can't calm an angry lion or tame a deer. These are simply tasks that don't fall under the specified list of what Animal Handling can do.
I disagree with the logic of the linked answer, which says that because 'all varieties of ordinary animals are beasts', anything which isn't a beast can't be an animal. This is an over-extension; a more accurate reading of these rules would be that 'anything which isn't a beast can't be an ordinary animal'. An owlbear, for example, is not an ordinary animal and is a monstrosity. Would an animal handling check have zero effect on a domesticated owlbear? Intuitively, I think not.
Ultimately, it's up to you as DM to choose the appropriate check and DC for any test which your characters wish to undertake. In the absence of a 'monstrosity handling' skill, I would use animal handling. Surely someone very good at handling animals is also relatively able to handle domesticated monstrosities? If you want to reflect the unordinary nature of the creature, consider a high DC of putting disadvantage on the check.
My argument above is along the lines of 'animal handling, seems, intuitively, to be the skill which most closely applies to this check'. You might rule, based on a valid reading of the description of 'animal handling' in the basic rules, that animal handling won't be any use here. In this case, you could rule that there is no skill proficiency which will help here, and that any check to handle a monstrosity is just straight Charisma or Wisdom with no proficiency modifiers applied.
The DM determines if an ability check can be made at all, and if so, the rules for ability checks on page 174 say:
For every ability check, the DM decides which of the six abilities is relevant to the task at hand and the difficulty of the task, represented by a Difficulty Class.
And it is again the DM's choice which skill to apply, if any:
Sometimes, the DM might ask for an ability check using a specific skill.
The DM can even use a different base ability for a skill if they use the variant rule (page 175), and that makes more sense to them.
Moreover, the description of the skills in not exclusive or limited to the listed examples, these are just example applications:
The skills related to each ability score are shown in the following list. (No skills are related to Constitution.) See an ability’s description in the later sections of this chapter for examples of how to use a skill associated with an ability.
If you want to use a skill, it is obivous that Animal Handling is by far best fit as a natural choice for handling any kind of animal-like creature, wether it is a stupid beast, or a stupid monstrosity. It's text on page 176 PHB explains that it for example represents proficiency with understanding animals, calming domesticated ones, and keeping mounts (not limited to animals) from getting spooked or getting them to do something risky:
Animal Handling. When there is any question whether you can calm down a domesticated animal, keep a mount from getting spooked, or intuit an animal's intentions (...) You also make a Wisdom (Animal Handling) check to control your mount when you attempt a risky maneuver.
Fantastic mounts like griffins are part of the game, so the examples even explicitly cite uses that would include monstrosities in that regard. And while they do not explictly list befriending or taming non-animal creatures of animal intelligence, they are the closest thing to that.
But you can also allow using other skills. For example, you could use Charisma (Intimidation) to cower the monstrosity into doing what you want, you could use Charisma (Persuasion) to handle a more intelligent monstrosity that is smart enough to get some basic ideas.
Or you could decide that no skill really applies as the task is outside of the training and experience that leads to the skill, and ask for a basic Charisma or Wisdom check.
