Art mentioned this in a comment:

However, while some people are more zealous than others in flagging

I dunno whether bad luck, or good that it occurred, this happened yesterday:

The red color marked user is a well known overzealous flagger. I've seen many of their spam flags being retracted just a few seconds after flagging and then using a NAA flag. The blue marked user is a highly respected member of Charcoal (or atleast, I respect them a lot), and who is quite good at taking decisions.

In this case, the overzealous user flagged the post almost immediately. A few seconds later, the trusted user naa- reviewed the smokey post. After that the first user retracted the flag immediately and added a naa flag.

This wouldn't be a problem usually, but it would make a difference if that user is given the task of being the 6th flagger on a post with 5 auto flags.

Some suggestions to prevent this:

• Check if a user has lots of invalidated feedback and flag it from their account, so that you can be sure that they are not the 6th flagger.
• Check if a user has lots of invalidated feedback and stop/scold/educate the user to be more careful.
• Use just 4 instead of 5, but this has the disadvantage (from the data) that it doesn't help much. So I'm not much attached to the "use 4 instead of 5" statement.

You people are quite brilliant at this, and can figure out a better way. My only request is to somehow make sure that a overzealous users isn't the deciding factor, if you are flagging 5.

Good luck!

Art mentioned this in a comment:

However, while some people are more zealous than others in flagging

I dunno whether bad luck, or good that it occurred, this happened yesterday:

The red color marked user is a well known overzealous flagger. I've seen many of their spam flags being retracted just a few seconds after flagging and then using a NAA flag. The blue marked user is a highly respected member of Charcoal (or atleast, I respect them a lot), and who is quite good at taking decisions.

In this case, the overzealous user flagged the post almost immediately. A few seconds later, the trusted user naa- reviewed the smokey post. After that the first user retracted the flag immediately and added a naa flag.

This wouldn't be a problem usually, but it would make a difference if that user is given the task of being the 6th flagger on a post with 5 auto flags.

Some suggestions to prevent this:

• Check if a user has lots of invalidated feedback and flag it from their account, so that you can be sure that they are not the 6th flagger.
• Check if a user has lots of invalidated feedback and stop/scold/educate the user to be more careful.
• Use just 4 instead of 5, but this has the disadvantage (from the data) that it doesn't help much. So I'm not much attached to the "use 4 instead of 5" statement.

You people are quite brilliant at this, and can figure out a better way. My only request is to somehow make sure that a overzealous users isn't the deciding factor, if you are flagging 5.

Good luck!