Introduce a better way to identify trusted users

I began noticing some very odd voting shortly after the new review system was implemented. Normally, when I see a completely craptastic post with several up votes, I run straight to our tools to see if I notice any suspicious patterns. After chasing half a dozen geese, I realized that it's just people blindly voting in review.

There's also more users that write .. less than awesome posts now reviewing .. less than awesome posts. So it's not always blind gamification, ignorance is partially to blame.

What remains is, those that can maintain high quality and do participate in review flag this stuff, and we take action. So a partial solution, or perhaps the means of getting data to come up with a solution is this:

Track the amount of negative actions a moderator or trusted (20k) user took on a post any user positively reviewed, by the user(s) that reviewed it.

This should quickly be able to tell us who is doing it wrong, which leads to a better idea of what (if anything) could be done about it. I really want better data on the problem before anything is done to correct it.

The idea is interesting, but the thing is, people actually vote on posts. People spend a great deal of time going to posts and they generally remember to upvote them if they're good; it's a pretty natural interaction on the web. Going to people's profiles after getting certain magic votes and giving special magic person votes? Let's sanity check that quick.

People don't visit profile pages often. (Limiting factor 1) Profile pages are really busy, so a new button would be hard to notice (limiting factor 2) and some major education stuff would be needed (limiting factor 3). To boot, only "trusted users" are allowed to vote (limiting factor 4). If there's only so many trusted users, it's really easy for the system to break down if just a few people don't vote. And what if we're talking about Stack Overflow? Suddenly tag partitions are a big problem; trusted users like John Skeet and Eric Lippert might be great at identifying Trusted users in C# (if they care), but they're unlikely to see trusted users in tags they don't use, or worse, ignore (limiting factor 5).

The suggested method of starting this off (moderators voting) sounds bad too, in no small part because of the potential for drama. Suddenly I, as a moderator, am supposed to decide who gets privileges on the site. I'm sure you see the potential problem there.

On top of all of the things that drastically limit the likelyhood anyone will actually vote here, suddenly we're voting on people, not posts. With the exception of moderator elections, that's a pretty big paradigm shift.

The problem really shouldn't be judging the quality of users directly and subjectively like this. If anything a user's actions should be judged. If a user consistently approves edits that are rejected or makes edits that are rolled back, maybe they should get a notification or maybe even a brief approval/edit ban. But the whole thing where an extremely limited set of people are expected to vote on an extremely limited set of people via an extremely rare action just seems like an extremely good way to make sure we just plain don't have anyone that can actually use these tools.

If there's a problem with how people use a tool, address that. That's why we have edit bans, require multiple approvals, multiple close votes, etc. Just making a system so that virtually no one can ever use it just makes tools useless and forces the burden back on moderators who are already spread thin. The whole point of community trust and community moderation abilities is to lighten the load of moderators and make moderation a thing everyone can take part of, not just the 0.0001% of people who actually recieved or cast these new magic votes.

I like the idea. As jokerdino says there ought to be some mitigating factors but I would like to address the potential problems. I think they can be, at least partially, solved by the bit you edited out about allowing everyone to vote and what remains, i.e. starting with a trusted circle.

To avoid the "circle jerk" phenomenon I would start, as you suggest, with the moderators. Give each moderator unlimited votes. Once a moderator has nominated you, you're automatically "trusted".

Every trusted user then gets a limited number of votes. For example, 12 to start and then one a month. The point of this is for it to be a low number. People should be jealous of their votes so they're less likely to use them unwisely. It would make sense for the first batch to have more votes to spread the love around a little more.

It takes X votes from trusted users in order for yourself to become trusted, say 2.

Then to up the chances of ensuring that all tags are covered you allow everyone to vote. To stop everyone becoming trusted you still have to be nominated by one trusted user. You also need Y votes from other users. In other words you become trusted if you have the following:

• Y votes from "ordinary" users and one vote from a trusted user.
• OR X votes from trusted users.
• OR 1 vote from a moderator.

There could be a little list somewhere of people who need a single vote from a trusted user in order to become "trusted".

A trusted user would also get 1 "untrust" user vote each month. It would require another Z votes from trusted users for someone to become no longer trusted. Moderators would be included in this but would have unlimited votes (no accusations of revenge attacks).

Where I would differ from you is that in time, I think it would be worth applying this to the review queues (in the future). I'm not saying that you have to be a trusted user to use them necessarily but that you would have to have been voted as a trusted user by a few not yet trusted users in order to use them.

Personally, I think this idea on a whole is misguided.

The purpose of reputation is to fairly and dispassionately measure "trust" for how SE defines it. With this system, we might as well scrap reputation entirely. And it sets up an awful precedent: if certain people don't like who is at the top of the proverbial "trust" meter, scrap it until we come up with a system that matches those people's truthy idea of who should be trusted.

But I wanted to highlight one part of your proposal:

As to how to get this whole thing started: at first, give, say, 50 trust votes to a limited circle. The ideal circle would be the people who have undergone the most scrutiny by the community, and are best equipped to have a good impression of users' activities: the moderators.

One of the great benefits of Stack Exchange and how it even bills itself is that the community runs the sites, not them. Why would you cede this amount of power to a set of people who are elected for life?

For that matter, any time the moderators are asked to do a little more, they push back saying that they don't have time or that they're overworked or that they're just volunteers. Why would any moderator agree to this plan? Having to be solely responsible for deciding who is trusted on a site? If a moderator is overworked or has better things to do, this seems like a real hassle.

I get that you don't like how the review queue is going; I really do. But effectively scrapping the reputation system and ceding a huge amount of power to a select group of people is not the solution.

Rather, the problems you describe can easily be attributed to two things:

• Tools that could use some improvement, because it's too easy to do bad stuff and too difficult to undo aforementioned bad stuff
• A largely disinterested user base who haven't, as yet, found a reason to take the review queues as seriously as people like you do

Instead of writing off the tools and creating a far smaller pool of users helping with community moderation, I'd much rather see the tools improve and ideas on how to get the community more engaged.

Trusted users are probably, and hopefully folk who have put in a certain degree of community involvement. Considering that one of the things that @Pekka is proposing for the "trusted user" status is having access to the review queue, could I propose that one of the criteria for this would be a reasonable number of helpful flags (say 50), and at least a 70-80% ratio of helpful to non helpful flags - this means the user takes the effort to flag (and 50 isn't that hard).

And while using mods to 'seed' trusted users is a good idea, I think allowing a mod veto/revocation of the trusted user status would be a good balance against it. A large number of negative flags (of unknown amount) by distinct users might be grounds for a review of the status (automating it somewhat).

What I find problematic is the blackbox nature of this. While it prevents the system from being gamed, it feels someone against the whole open nature of the site. Its something that needs to be balanced pretty well.

I would like to propose one alternate solution how about this instead of trusted user ?

Keep the review system, allow a user to review a post if he has earned a silver/gold badge related to a question. For ex: to review a C tagged question I need to have a C Silver badge. That way not everybody can review it, it puts additional restriction.

PS: Do let me know reason if you choose to downvote.

I really like the idea behind the suggestion but I would like to propose some minor alterations, which I guess would make the idea a teeny bit better.

• The trustees should not have a permanent seat like the moderators. This would help us control the damage in case any user goes rogue and undetected. Each user may have a term of 30 days or so with any significant lapses of judgment costing their position.

• Going with your original idea of allocating votes to already trusted users, any user who accumulates a certain number of anti-trust votes cannot be appointed as a trusted user for, say 6 months or so. If someone is not trusted by quite a number of people, that should be a good enough indicator.

• Given subtle hints that the community team is overwhelmed with monitoring all 300+ community moderators across the network, it might be wise not to grant insta-votes to trusted users. Instead two votes from any trusted user can be considered final, which would make the body of trusted users self-regulatory and a safer solution.