Note: While this answer works, HighDiceRoller's solution is a lot simpler, faster and more elegant. I encourage anyone reading this to upvote it.
Here's a simple recursive AnyDice function that should hopefully do what you want, and allow fairly easy modification to explore variations of this mechanic:
function: roll S:n successes after K:n on DIE:d in N:n rolls {
if S <= 0 { result: 1 }
if N <= 0 { result: 0 }
result: [
if DIE >= [dc after K successes]
then [roll S-1 successes after K+1 on DIE in N-1 rolls]
else [roll S successes after K on DIE in N-1 rolls]
]
}
This function makes use of two helper functions. One of them is the generic one from this answer, used to choose between two outcomes based on a die roll:
function: if COND:n then A else B {
if COND { result: A } else { result: B }
}
The other one is the custom dc after K successes function, which you can define (or even redefine several times in your program!) any way you like to test different kinds of DC adjustment mechanics.
For example, here's how to calculate the probability of getting 3 successes in 10 rolls using your mechanic, for various values of X, Y and M:
set "maximum function depth" to 20
X: 10
loop Y over {1,3,5} {
function: dc after K:n successes {
result: X + Y*K
}
loop M over {0..+3} {
output [roll 3 successes after 0 on d20+M in 10 rolls] named "DC = [X] + [Y]*successes, mod = [M]"
}
}
I originally wrote this answer before you clarified that you're talking about D&D 5th edition ability checks, where a success means rolling a result equal to or over the DC on d20+M, so I made the code generic enough to handle any kind of rolls, not just d20+constant. For example, if you want to see what happens when rolling with advantage, you can replace d20+M in the code above with 1@2d20+M (or 2@2d20+M for disadvantage).
The code does assume a roll-at-least-DC-to-succeed mechanic, but even that could be fairly easily changed by changing the DIE >= [dc after K successes] condition in the main function.
Note that if you increase the number of rolls to 20 or more, you will also need to increase the "maximum function depth" setting to something higher than it. (AnyDice used to give a helpful error message when that limit was exceeded, but it seems that now it just crashes. I should probably report that as a bug.)
The results are best viewed in summary mode, where you can simply read the probabilities off the "mean" table:
Ps. Note that, since this function calls itself twice for each roll (to compute the probabilities of reaching the goal, given that the current roll is either a success or a failure), its running time scales approximately in proportion to something like \$N \times 2^{\min(N,S)}\$, where \$N\$ is the maximum number of rolls and \$S\$ is the required number of successes. For \$N, S ≤ 10\$ this is quite manageable, but if you go up to, say, \$N = S = 15\$ or so, it's likely to time out.
In principle, it should be possible to improve this exponential runtime to something proportional to, say, \$N \times S\$, but that would require either memoizing the results of the recursive function calls (which AnyDice doesn't currently support) or manually implementing an equivalent dynamic programming algorithm (which should be technically possibly in AnyDice, but quite difficult and complicated). At that point, one might as well just rewrite it in Python.
(…or you could just do what HighDiceRoller did, which is honestly a lot simpler and more efficient than my solution.)
