Ask users if data is True Positive or False Positive

Question

I'm currently working on a system that manages big data and has a classification AI. However, the AI is not yet optimally trained, and I would like to gather some True Positive (TP) and False Positive (FP) examples to further train the AI and increase it's precision.

Since the data belongs to the users, it means I need to validate with them if the AI's predictions are correct, and let them be the judge of that. Taking that into consideration, I was trying to think of a way to ask the users for feedback regarding the data, something like "Is this a [true statement]?". However, I don't want them to think that the AI has poor performance and that the processed data cannot be trusted.

How could I ask the user for this feedback? Should I be honest and upfront? If so, how minimal can this process be? I was thinking of just an approach along the lines of the question and two buttons, meaning yes or no.

Answer 1

Honesty definitely builds trust. The worst thing as far as building trust in a model/prediction is for it to be confident and wrong - either other situation (confident and right, or not confident and wrong) still builds trust.

Does your AI have a confidence in its classification? If that confidence is very high, don't show the user anything, or perhaps an advanced feature hidden behind a button or link for you and other power users to correct the data in any case.

If the confidence is low, hang a banner saying:

We're not sure this is correct - if there is a problem, please provide feedback [here]

In that way, most (hopefully!) pages don't have that banner, and users perceive the benefit of the automated processes. However, when there is an issue, the user sees that the AI knew there might be an error, which further builds trust.

Answer 2

Is there an option to gamify it?

Where you have a pop up in the bottom right corner that just says "challenge the AI" explaining that the AI will need constant improvement to make it even better then it already is (and therefore improve their own work as well).

When clicked you enter some kind of game where you just give them a few statements. The message on the pop up could read: "Challenge out AI in under two minutes time" or something similar. Once the 'game' starts, limit the statements to about 5 or so maximum. After they did the 5, thank them and give them two buttons: 1: Go back 2: Challenge the AI again (another 5 statements start)

If possible within the program you might want to consider building in some sort of reward system for this. Or even have a scoreboard with all users from the same firm. In that case, you will have to limit the number of times a user can play on a single day to 3 rounds of challenges or so.

This could all be done in a pop-up and doesn't necessarily have to take them to another screen. Meaning they can just do it after they finished a task and want to "relax a bit" before starting their next task in the program.

Hope this helps.