92

In the beginning of March, I shared a process by which posts that need staff attention could get escalated, and a plan to test it out between March 16 and April 30, 2020. As promised on that post’s timeline, I’m here to share with you results from this testing period, as well as the plan for this process going forward. Spoiler: we think it went well and wanna keep it moving forward ^_^

Stats from the testing period:

During the duration of the test, a total of 127 questions across all Meta sites got added to them. The tag was added by a staff member on 74 (~58%) of these, and the other 53 (~42%) had the tag added by a moderator.

The majority of questions came from MSE, followed by MSO, and the rest Metas from all over the network:

  • 52 (~41%) from MSE.
  • 40 (~31%) from MSO.
  • 35 (~28%) from other child Metas.

Most of the questions escalated were bug reports or feature requests, but there were posts of all types escalated to staff (note that tags are not mutually exclusive):

With regards to staff response time, a total of 68 questions (~54%) got some sort of response from staff after the tag was added. The average time elapsed between the tag getting added and any sort of response was ~7d 1h (min. of ~0.1h; max. of ~48d 5h; median of ~1d 4h). Of these (sets below aren’t mutually exclusive):

  • 19 questions (~28% of above; ~15% of total) got commented on by staff. The average time elapsed between the tag getting added and the comment getting posted was ~6d 3h (min. of ~1min; max. of ~48d 5h; median of ~1h).
  • 44 questions (~65% of above; ~35% of total) got answered by staff. The average time elapsed between the tag getting added and the answer getting posted was ~6d 11h (min. of ~0.1h; max. of ~42d 2h; median of ~1d 18h).
  • 61 questions (~90% of above; ~48% of total) got edited by staff. The average time elapsed between the tag getting added and the edit getting submitted was ~7d 18h (min. of ~0.1h; max. of ~43d 12h; median of ~21h).

Most responses ended up resulting in a corresponding tag edit, and 65 (~51%) questions got removed. 60 of these got removed by staff, the other 5 by a moderator. Of these:

  • 48 questions (~74% of above; ~38% of total) got added to them.
  • 1 question (~2% of above; ~1% of total) got added to them.
  • 4 questions (~6% of above; ~3% of total) got added to them.
  • 2 questions (~3% of above; ~2% of total) got added to them.
  • 3 questions (~5% of above; ~2% of total) got added to them.
  • 7 questions (~11% of above; ~6% of total) didn’t get a new status tag added.

It’s also worth noting that of the 68 posts that got a response from staff, 51 had added by staff, and the other 17 by mods:

  • 74 tagged by staff, 51 responded, 23 unresponded
  • 53 tagged by mods, 17 responded, 36 unresponded

Out of the many posts that got escalated through this process a few were worthy of highlighting, either because of the work put in by the users who asked the question, or because they got more than a run-of-the-mill response from staff:

Next steps:

Given the way the test went, we want to keep this process going. In the original post I mentioned that we’d set targets for how many posts staff can respond to, and how quickly we’d be able to do so. I also mentioned we’d work on reviewing the targets we set quarterly, to make sure they’re still appropriate. Given the stats shared above, for the rest of Q2 2020 we’re setting a target at trying to respond to 50% of Meta posts from across the network, within 2 weeks of getting added to it. There’s about 6 weeks before the end of Q2, which should give us a good window to see if there’s consistency between the testing period and this new period, and if these targets are indeed reasonable (fingers crossed!).

For this process to work for both the community and the company going forward, for as long as this process is in effect the CM Team will provide you with guidance on what posts to escalate every 2 months — as I did for the testing period — along with stats similar to the above for the preceding 2 months. That way, we can ensure that y’all escalate stuff that’s more relevant to our current projects, and thus more likely to be impactful. You can expect a post before the end of May, with guidance on what to escalate in June (update: since posted here); in the meantime the guidance from the testing period still applies, but we’d also like to see anything relating to our ongoing work on Review Queues. As I noted in the guidelines for the testing period, though, we’re not just looking for stuff relevant to current projects: the guidance I offered there for new posts should still hold going forward, regardless of what our teams are working on.

The ratio, shared above, of posts tagged by staff vs. mods as well as whether or not these get an answer isn’t quite what we’d expected: this can be because the tag gets added to posts relating to issues currently being worked on, which means a higher likelihood of getting a reply… but it can also be due to uncertainty on the mods’ (and/or the communities’) end on whether stuff should be escalated. Whichever the case, we’d like to see more stuff escalated by the community and the mods: as such, I’d like to invite the moderators to grab a CM in The Teachers’ Lounge when in doubt, or to ultimately err on the side of over-escalating — this may ultimately drive our numbers down a bit, but that’s a risk we’re willing to take in order to make sure the process is as clear from your end as it is from ours.

The guidance and targets mentioned above are useful for new posts, or posts relating to projects our teams are working on… but what about older stuff? We’re gonna take the time between now and the end of Q2 to create guidance and establish a separate target for escalating and responding to old posts that don’t relate to any ongoing projects. These are likely to get much slower responses, so it makes sense that they’d have a separate target just for them, along with guidance specific to them. Measuring these will also help add some depth to the stats we plan to share, as they’ll touch on longstanding bug reports or feature requests from our communities.

We’ve gone through several internal changes, and through them all our commitment to improving our responses to our communities has been a top priority. As we continue to improve on the ways we respond to our communities’ requests, it is our hope that you will see tangible progress taking place. As with the previous posts, feedback on how the process went from your end is welcome, as are requests for clarification.

Improve this question
11
  • 5
    Out of total 127 posts, how many posts are not responded (i.e neither change in status tag, nor commented, answered, edited etc.)?
    – Pandya
    Commented May 15, 2020 at 16:16
  • If this has not already been done, would it be possible to recalculate the means and medians using some sort of metric that accounts for outliers?
    – Exempt-Medic
    Commented May 15, 2020 at 16:31
  • 5
    @Medix2 Probably for this sort of data the median is a pretty good outlier-adjusted alternative for the mean.
    – Bryan Krause
    Commented May 15, 2020 at 16:42
  • 3
    That would be 59 posts (127 total posts - the 68 that got a response), @Pandya ;)
    – JNat StaffMod
    Commented May 15, 2020 at 17:04
  • 1
    Is there some closure to the questions you don't answer, like a comment saying you don't have the time? Or are things just left hanging until they're not? I added that tag to a question over two weeks ago and I wonder when I can consider it forgotten.
    – Joonas Ilmavirta
    Commented May 15, 2020 at 17:08
  • 3
    Nothing gets forgotten, @JoonasIlmavirta: it's all on a board, so it's a matter of being able to actually get to the unresponded items — that's another thing we're gonna focusing on 'til the end of Q2, as part of trying to set goals for stuff that's been posted or tagged before whatever 2-month period we're measuring
    – JNat StaffMod
    Commented May 15, 2020 at 17:12
  • @JNat Thanks, sounds very good! I know things can take time.
    – Joonas Ilmavirta
    Commented May 15, 2020 at 17:20
  • 6
    If I understand correctly posts could be nominated for getting the tag [status-review] by flagging the post for moderator attention with a custom reason. How many of these flags were raised? How many of the added tags were due to one of these flags, so how many flag were acted upon? In other words: was this mostly an effort from moderators and staff, or was the community the driving force?
    – Marijn
    Commented May 15, 2020 at 18:34
  • @Marijn: I imagine it's harder to systematically measure those things, given that such custom reasons may be phrased in a number of different ways (assuming the contents of such custom flags are even easily accessible/browsable).
    – V2Blast
    Commented May 16, 2020 at 9:50
  • @V2Blast I can image it is hard to get such measurements, but for me it would be the more interesting statistic. An estimate would also do (no/some/many/most/all of the 53 moderator tag edits were caused by flags), and also maybe a small survey among moderators on how many flags they approximately received in total.
    – Marijn
    Commented May 16, 2020 at 12:04
  • 1
    As noted, @Marijn, it's hard to get accurate numbers for these, but I queried the database for flags that either point to the guidelines post or mention "status-review" for the duration of the test: 39 such flags were raised in the whole network (28 on MSE, 3 on MSO, 8 on the rest of the Metas). Of these, 1 was self-cleared, 16 were marked as unhelpful, and 22 were marked as helpful — again, this isn't representative of how effective flagging was, since some flags can be marked as helpful even if they don't result in the tag getting added.
    – JNat StaffMod
    Commented Jun 2, 2020 at 19:09

7 Answers 7

Reset to default
42

It's inevitable and understandable that certain questions require more time/effort to respond to than others.

As we get further along in evaluating this process, I think those "forgotten?/old" questions will start to accumulate a bit, even if the system is working perfectly. Because of that...

We’re gonna take the time between now and the end of Q2 to create guidance and establish a separate target for escalating and responding to old posts that don’t relate to any ongoing projects.

...I'm very happy that it's already part of your plans for the next set of evaluations, and further I might even say excited by the idea that posts only ~2 months old will be considered "old" and deserving of a bit of extra attention.

Add to that:

this may ultimately drive our numbers down a bit, but that’s a risk we’re willing to take in order to make sure the process is as clear from your end as it is from ours

and there is a lot to like here. Thank you and keep up the great work.

Improve this answer
24

Thank you. Things obviously aren't perfect, but perfect isn't what we've been asking for - just a real, human attempt to be reachable and to follow up on things that you (the company) have said you'll do.

FWIW, this is the straw that's ending my boycott of SE, and I'm returning my username to normal. I'll still be wary, but it looks like there's an actual path forward now.

Improve this answer
20

I suppose I'm a bit bummed out at the results here. I will readily admit that I'm coming at this from a perspective of strong bias, but I feel like, for the first time, I can see what's going on when the community you're a part of or want to prop up isn't Stack Overflow.

That is to say, it ain't easy to get CM's attention. It is a precious commodity and I cannot even begin to fathom how this can improve the circumstance smaller sites are in when they really need CM guidance, input or attention.

My thoughts and feedback, having participated in this:

  • The goal is to reach 50% of engagement across all Meta sites. That means, across all 140+ sites with varying levels of activity and engagement, half of the questions that are flagged in this fashion are ever responded to. With the lion's share of attention spent on Stack Overflow, my biggest concern is that this skews heavily towards the largest few communities in the network, while the smaller ones aren't regarded at all.

    I would posit that communities which don't get attention could interpret this any way they wish, honestly...

    Potential remedy: correlate the responses made by CMs by site and by traffic and compare that to the number of actual requests made by sites. If there is a positive correlation to larger site and CM attention, one way to make everyone feel heard is to take attention away from sites which were heavily favored in previous rounds of this experiment, and shift it to smaller sites. My takeaway from expanding the communication is to make sure that everyone feels like CMs aren't completely unreachable, and this can provide some data points to help address this perspective.

  • Several posts were highlighted as exemplary, due to either the attention to detail put into the post itself (from either staff or inquisitor), or due to its timeliness and relevance to current world events (e.g. Covid-19). There is no obvious correlation as to what qualifies these as exemplary besides either their relationship to an existing effort on the roadmap, a bug, or the fact that it's timely (e.g. Covid).

    Potential remedy: publicly specify criteria which is meant to encourage or facilitate CM responses. A few of us who have had a few winters here will know the types of questions which aren't going to get anyone's time of day, but knowing what the CM team is looking for in terms of question age, structure, tone, scope, etc would help when deciding to flag a question for CM concern. I never understood what was meant by "older" stuff; maybe something from January was too old?

  • For the questions which weren't responded to, the feedback loop is entirely absent, and those who hold or have held convictions on how the CM team regards their community may be strengthened or affirmed in this situation, which is something I think you guys wanted to avoid.

    Potential remedy: reply to everything, even if it's something simple like, "Okay, we see this and we're discussing it internally. We can't promise we'll give you a response, but we've heard you out."

    Speaking personally from my participation in this, hearing something rather than nothing would have done a lot to assuage my concerns that this was either impossible, untenable or not a constructive use of my time.

  • In a manner of speaking, some of the posts which were highlighted seemed to dovetail into what was already being worked on, which can be...problematic. In my head, it means that the community wanted X and had opinions on how it could be solved, whereas the company was already building Y and might incorporate parts of X into it.

    To be fair, I don't think that one's easily solvable, but something that could help in a situation like that - if a post is highlighted that happens to dovetail with what you're already building - is to establish some goals and scope. The community wanting triage to be fixed is great. Saying you're working on fixing triage is fantastic. Be sure that the definition of "fixed" is widely understood, though.

Overall, though, the big thing that you guys don't want to start selling us on is hope. Don't set lofty aspirational goals that we can read publicly and feel fuzzy about, just for there to be enough fine print or nuance that means not everyone can appreciate the same warm and fuzzies.

Improve this answer
14
  • 6
    You bring up a lot of points and I appreciate that. Did you know there's over 170 sites now? Oof. As far as the idea that most responses were related to things we were already working on - absolutely - and that will always be the case... we release a feature and it has bugs, we fix the bugs while we're in the midst of it because they're things we missed. I think any company would do the same... they patch the stuff they just built. That doesn't mean we won't get to the older stuff eventually... but a bug that's been around for years and is an edge case is going to be lower priority.
    – Catija
    Commented May 16, 2020 at 4:33
  • 6
    You also talk a lot about the CMs here. Actually, most of the responses as far as I'm aware haven't been from CMs at all - we're managing this but this isn't limited to our work. Bugs get taken directly to the devs and we only respond in cases where they want our help. We've had Marketing respond to some things, Design others, SRE... etc. I haven't counted but I think the CM responses are in the minority. That said, we do respond on all sorts of sites. It's not included in this data but I just responded to one of these questions on the Latin site earlier this evening.
    – Catija
    Commented May 16, 2020 at 4:41
  • 10
    The limiting factor is less "we're focused on SO and the other huge sites" and more... "what can we actually answer?" - or at least that's been my experience. We can definitely do more to say "we see you, it'll take a bit to actually do this" though. There's also a ton of stuff that we could have status-reviewed to get our numbers up - I handled all sorts of stuff on our new Drones and Model Aircraft site over their private beta but none of that got counted. :) They don't have mods yet to add a status tag... and it seems odd to status-review tag synonymizations. :P
    – Catija
    Commented May 16, 2020 at 4:54
  • 7
    I just looked at the ones that are outstanding and in our bucket and, well... a lot of them are appropriate for CMs as the decision makers but we're hampered because the tooling requires code changes that we can't make. We're working on getting better access to some of this stuff so that we can make site setting changes and text changes that require building the site but we haven't had that since we lost Jon and Shog. Being able to make these small changes - like updating the text in the new question asking form - are great opportunities to help make small improvements to the network sites.
    – Catija
    Commented May 16, 2020 at 5:05
  • 4
    "Older" means stuff from January or 2019 or 2016 or whenever... stuff that the sites have been asking for but couldn't get our attention to and that were still needed. Sometimes these are relatively simple changes that just got overlooked because we didn't have a good way to hear about them and add them to a list for review. Sometimes they're huge asks that we'll likely have to say "no" to... but at least the site will have closure. I spent a non-insignificant amount of time complaining that we had status tags and didn't use them or just decline stuff that was out of scope... so maybe... ;)
    – Catija
    Commented May 16, 2020 at 5:09
  • 5
    And, if you really don't trust the tagging system, you can always ping me in chat or email me to get my attention. The thing is... and I'm speaking for myself here... It's a bit of a struggle to know how to respond to some posts... for various reasons... anything from "do I even know the answer to this?" to ... knowing whether I even can engage on something... and every now and then, there's the fear that I'm going to say something that people really don't like, honest or not. No one likes to dash someone's ideas... well, unless it's the 12th time it's been suggested and it needs to go away.
    – Catija
    Commented May 16, 2020 at 5:40
  • 5
    @Catija: Thanks for the detailed response! :)
    – V2Blast
    Commented May 16, 2020 at 9:52
  • @Catija: Thanks for the replies. I'm going to try to respond to them where appropriate but I'll only tag you once.
    – Makoto
    Commented May 17, 2020 at 3:48
  • 1
    Firstly - the main concern in terms of the responses being related to things already in flight is...probably coincidental. The new features which have bugs will definitely enjoy their time in the limelight since that's forefront at everyone's mind, and it's quicker to respond. Things like the review queues being fixed or approached again is convenient, but there's been a long discussion about that since it came out, honestly.
    – Makoto
    Commented May 17, 2020 at 3:50
  • In that vein it does mean that Stack Exchange has design ideas for it, and the community has already provided a plethora of feedback about it, and it could be the case that those overlap, but that's more my latter point - if goals are established as to what the fix is that Stack Overflow wants to build, irrespective of what that actually is, then that would enable better communication between the community and set expectations when someone says that a thing is being worked on.
    – Makoto
    Commented May 17, 2020 at 3:51
  • 1
    I am actually quite surprised that CMs haven't been involved in this. I don't understand why they wouldn't be, if nothing else but to facilitate the conversation. I would think that a workflow like this would be unfortunate and...inefficient...to escalate bugs, as well, since those are deserving of their own attention and workflow which I would want to consider completely independent of escalating things that need staff attention raised by communities. Sure, bugs need attention. But bugs need a very special kind of attention that isn't related to a site wanting to fix workflows.
    – Makoto
    Commented May 17, 2020 at 3:54
  • 1
    Not being able to answer questions...hmm...that's tricky. It's entirely fair to not be an SME in a subject matter, but I would hope that this sort of communication both exposes those circumstances and emphasizes the need to get in the know. Not knowing how to reply to certain questions should actually be an opportunity to engage and learn more about the problem domain, both from the perspective of the people highlighting it, and from the company.
    – Makoto
    Commented May 17, 2020 at 3:57
  • 1
    Can't say much about there being a lack of tooling. My only thought: it's tough to wash clothes without water or heat.
    – Makoto
    Commented May 17, 2020 at 4:00
  • 2
    Lastly, my real philosophy will be to strongly favor established communication conventions rather than...less established conventions. The company sets the rules of engagement, and if I abide by them I have a reasonable expectation that those rules are honored. Using side channels to communicate - as unfortunately necessary as that may become sometimes - is not ideal. We can't have a process which indicates that some forms of communication are more equal than others.
    – Makoto
    Commented May 17, 2020 at 4:02
15

Just learnt something, nice to know we can use that tag outside of MSE, added it to a request I did some time ago.

From a moderator's point of view, it's very useful as it leaves a trace to the community. Poking inside TL is nice to get a status feedback on a request done, and I always had one fast from there, but we can't always copy/paste the TL's answer on our meta for leaving feedback back to our community. It's more transparent that way for our site users base.

Improve this answer
8

for the rest of Q2 2020 we’re setting a target at trying to respond to 50% of Meta posts from across the network, within 2 weeks of getting added to it.

For the determination if this target is reached, what would count as a response?

  • A staff member posted a comment, even if it was a meaningless comment?
  • A staff member posted a meaningful comment?
  • A staff member posted an answer?
  • The tag was removed and/or replaced with another status tag?
Improve this answer
1
  • 1
    status-review getting replaced by another status tag and answers are clear responses, I'd say. The stats I shared measure any comment, regardless of length, meaningfulness, etc. — I recognize that's not ideal, but there's no easy way to query the network for "meaningful staff comments." That being said, the goal with responding is to get to a point where the status tag gets replaced — and that is, again, a much clearer sign that the post has been responded to.
    – JNat StaffMod
    Commented May 29, 2020 at 15:25
7

Asked in comment and on TL but didn't get reply. Therefore asking by adding an answer thinking it may work:

According to statistics, 68 questions of total of 127 are responded. It is mentioned that 35 (~28%) questions came from other child Metas.

I would like to know (out of 35) How many posts from child Metas (non-MSE non-MSO) are responded?

Improve this answer
2
  • That would be a total of 9 that got some sort of response — so ~26% of the child Meta posts that got escalated, and ~7% of the total 127 posts.
    – JNat StaffMod
    Commented May 19, 2020 at 10:07
  • @JNat Thanks for the response.
    – Pandya
    Commented May 19, 2020 at 10:28
3

Yay! I like when things are successful.

So happy to hear you guys plan to continue increasing your engagement with the community. Thank you so much for that. Keep up the good work.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .