I admit that I'm a bit of a late-comer on the scene, having rejoined Puzzling after these sorts of puzzles started appearing, but I do believe there is a quality issue with the "What is an XYZ item™" series. The most obvious reason, of course, is that this type of puzzle is trivial to set up - pick a property, then just provide two lists of examples... right?
Unfortunately, what you want to do is usually not as simple as that, and here's my take. I've written this specifically for the Word™ series since they are the most common, but most of these tips easily apply to other variations as well.
Do use an adequate sample size
Make sure there's enough words to work out the property. Then maybe add a few more just in case. You're the one making the puzzle, so of course the property's going to seem obvious to you, but from the perspective of the solver it'll look like there's infinite possibilities at first glance.
Aside from difficulty reasons, having more examples also makes it more likely that people will find your intended answer. At least one puzzle has fallen victim to having a much simpler unintended answer which happened to fit, and that is less than ideal.
Don't pick examples only for the sake of having interesting definitions
A common theme I've seen is picking pairs of synonyms, one which fits the property and one which doesn't. When done poorly, this only serves to confuse and provides very little in the way of hints to the solver since the words on either side are drastically different in terms of spelling. Of course it's perfectly possible to use related word pairs to provide good clues, but make sure you actually think hard about how useful your chosen words work as clues, rather than picking whatever word pairs come to mind that happen to fit.
I prefer to think of these puzzles like playing The Witness (don't worry if you don't know what that is). The clues should guide the solver by providing adequate examples for testing and rejecting hypotheses, rather than solely for looking pretty at first glance.
Do clue obscure properties
If the property you've chosen is obscure, that doesn't necessarily mean you shouldn't use it. What it does mean, though, is you'll likely want to provide hints.
Here are some ways hints might be provided:
- Pick a good, meaningful name for the Word™, rather than something that just tangentially relates
- Provide a bit of backstory to hint at the relevant information
- Clue a message using the words chosen
- Add a very easy example so that people have an "in" - e.g. if 31415 was present in a Number™ puzzle, that would be a good way of making puzzlers think that pi is relevant
Note that when I say hints, I mean from the get-go. Sure, you can give hints as you go if nobody gets it immediately, but ideally the puzzle should be solvable when it's initially posted.
Don't pick arbitrary properties
In the previous point when I said obscure, I didn't mean arbitrary. Sure, you could do "Words with less than 3 consonants and with vowels in alphabetical order", but would you really like that if you were the one solving?
Think about whether you would appreciate the solution if you were the one on the solving end. Of course, opinions are subjective and there's no guarantee that everyone will like the same puzzles as you, but if you wouldn't appreciate the puzzle yourself when solving then that's a red flag.
Do sort the words if order is irrelevant
This one's just puzzling etiquette. If the order of the words is irrelevant, sort them by alphabetical order, or by length then alphabetical order. It may seem trivial, but it gives solvers one less thing to worry about, and when the clues are out of order it's usually an indication that something is up.
