If I've understood your description correctly, the solution is, as usual, to write a function. Specifically, the following function (or, rather, pair of functions) should do what you want:
\ This function returns +1 if A wins, -1 if B wins and 0 if it's a draw: \
function: ATTR_A:n versus ATTR_B:n {
result: [2d10 attr ATTR_A versus 2d10 attr ATTR_B] \ call helper function \
}
function: ROLL_A:s attr ATTR_A:n versus ROLL_B:s attr ATTR_B:n {
SUCC_A: ROLL_A <= ATTR_A
SUCC_B: ROLL_B <= ATTR_B
\ higher number of successes wins: \
if SUCC_A > SUCC_B { result: +1 }
if SUCC_A < SUCC_B { result: -1 }
\ if both have two successes, highest roll wins: \
if SUCC_A = 2 & ROLL_A > ROLL_B { result: +1 }
if SUCC_A = 2 & ROLL_A < ROLL_B { result: -1 }
\ if both have one success, higher successful roll wins: \
if SUCC_A = 1 & 2@ROLL_A > 2@ROLL_B { result: +1 }
if SUCC_A = 1 & 2@ROLL_A < 2@ROLL_B { result: -1 }
\ otherwise it's a draw: \
result: 0
}
You can use them e.g. like this:
A: 6
B: 5
output [A versus B] named "[A] vs. [B]"
With that out of the way, let me explain what this code does.
The first function is simply a convenience wrapper that calls the second one, passing it two 2d10 dice pools and the given target attributes. The "magic" here is that, when an AnyDice function expecting a sequence parameter (i.e. one marked with :s) is given a dice pool instead, AnyDice automatically calls the function for every possible (sorted) result of rolling the dice and tallies up the results. Inside the function, the result of the roll is "frozen" into a fixed sequence of numbers that we can examine and manipulate any way we want.
So, what does the second function do? First, it calculates the number of successes for each player by comparing the sequence of dice values they rolled with their target attribute. If one player gets more successes that the other, that player wins. (Note that setting result: in AnyDice immediately ends the function, returning the given result.)
Otherwise, if both players have the same number of successes, we look at the various tie-breaking rules.
The two-success case is easy to implement in AnyDice: just comparing two sequences with < or > does a lexicographic comparison, i.e. it first compares the first numbers in each sequence and, if they're the same, falls back to comparing the next numbers in each sequence instead. Since the sequences of dice rolls generated by AnyDice when a dice pool is "frozen" by passing it into a function are automatically sorted in descending order, this comparison does exactly what we want.
For the one-success case, we can make use of the fact that, for both players to have exactly one success, we know that their higher roll must've been over the target and the lower one must've succeeded. Thus, we can simply compare the lower (i.e. second) numbers in the two-number sequence rolled by each player.
Finally, if none of the tests above matched, then either both players failed both of their rolls or they got the same number of successes but tied on the tie-break comparison. Either way, we simply return 0 to indicate a tie.
Ps. I used the Python script from this answer and Google Sheets to draw a heat map and a line chart showing the probability of a player with attribute A winning against an opponent with attribute B:
One notable observation evident from these charts is that players with low to moderate attribute values have a non-negligible chance of tying even against an opponent with attribute 0, since a mutual double failure is always a tie.
In general, if both players have lowish attributes, your mechanic tends to produce a lot of ties, which could make the players feel like they're both ineffectually flailing at each other without actually accomplishing anything. Of course, this might or might not be a problem, depending on what kind of a feel you want low-level combat to have in your system.
Pps. Dale M had the clever idea of relabeling the dice so that anything above the target number is replaced by zero. Unfortunately their elegant solution seems to have a few bugs — like counting (4,4) as losing against (5,0), even though it has more successes — but the relabeling trick can be easily adapted into my code to simplify and make it more efficient:
\ This function returns +1 if A wins, -1 if B wins and 0 if it's a draw: \
function: ATTR_A:n versus ATTR_B:n {
\ make relabeled d10s where anything above the target becomes zero: \
DA: 2d{1..ATTR_A, 0:(10-ATTR_A)}
DB: 2d{1..ATTR_B, 0:(10-ATTR_B)}
\ call a helper function to freeze the rolls: \
result: [roll DA versus DB]
}
function: roll ROLL_A:s versus ROLL_B:s {
SUCC_A: ROLL_A > 0
SUCC_B: ROLL_B > 0
\ higher number of successes wins: \
if SUCC_A > SUCC_B { result: +1 }
if SUCC_A < SUCC_B { result: -1 }
\ otherwise do a lexicographic comparison: \
result: (ROLL_A > ROLL_B) - (ROLL_A < ROLL_B)
}
Now the first function has little more to do than before: before calling the second function, it also needs to construct the relabeled dice. On the other hand, the second function is considerably simplified: after comparing the number of successes (which we still need to do explicitly, to avoid the bug in Dale M's version) we can resolve all ties just by lexicographically comparing the two sequences of rolled numbers.
And (especially for low target attributes) it's faster, too, since AnyDice is smart enough to notice that several sides of the relabeled d10s are now identical, so it doesn't need to call the second function separately for each of those equivalent rolls.
