For example, X is a random integer from 1 to 16. Now I get a piece of information: X is 3, 5, 9, or 14. This has 2 bits of information for the knowledge about X. But if the list of options is random enough, I'd have to list all four integers to describe this knowledge, which is much more than 2 bits. It may need more information, if the probability distribution is not uniform, and I attach a probability to each of the options.
Is there a term for this phenomenon?
In mathematics context, this is basically just saying, for a random variable X, there are infinite possibilities of Yi that I(X;Yi) is a given number, so if Y represents the pair <i,Yi> where i is also a random variable independent of everything else, I(X;Y) = I(X;Yi), but H(Y) could be made very high. Or for simplicity, let's ignore the first part, and say for random variables X and Y, H(Y) could be much higher than I(X;Y). This is a quite dull question in mathematics, because they are clearly different as we use different symbols to refer to them, and it is usually unambiguous which one we are talking about. But in philosophy, we could easily say "some knowledge about X", that could refer to either of them, that is the real information inside X, and the information inside X and of the approach to find X together. So I think something is required to address this problem.
In the example, the approach could be to categorize the values into 4 arbitrary lists of 4 elements each, and find which list it belongs. It has more information if we consider the approach not predecided, but chosen in a way using an extra random variable.
It's especially important in philosophy context because if we are discussing about the concept of knowledge, the approach is usually an open set, and there isn't a default one if nothing is specified.
