140

Users have long been asking for and discussing the pros and cons of a feature to proactively inform the question asker when their question has been closed. Although the initial response many years ago was that after consideration we would not implement this feature (for a variety of reasons), following the release earlier this year of the follow post feature (which greatly expanded inbox messaging relating to post updates, and has been positively received by the community), we decided to revisit this feature. 

During our announcement about the first phase of Question Close updates, we included a notice about an experiment that we would be running related to Email and Inbox notifications for closed questions: 

This is launching on Monday (April 20) on Stack Overflow as an A/B test. This update sends question authors an inbox notification when their question is closed. The idea is that the inbox notification will guide users to their closed question and the post notice displayed there will encourage them to edit the question to improve it.

If there are no edits, the question remains closed, and hasn't been deleted within an hour of closure, we'll also send a follow-up email with guidance about what to do next.

It’s running as an A/B test so that we can measure how effective these notifications are in guiding users to edit their questions.

The experiment ran for approximately two months on Stack Overflow only. The goals of the experiment were to see how adding inbox notifications to the post author would affect:

  1. The percentage of closed posts edited by the author after it was closed. We were hoping to see a positive change here.
  2. Site satisfaction/user reaction. We were hoping that users would not have a negative reaction to the new notifications and emails.
  3. Determine if the custom email notifications were more effective than just the inbox notifications were for leading to more post owner edits after posts are closed.

Results

Total Edited Diff Reopened after edit Diff
No notification 43619 5464 (12.53%) 590 (1.35%)
Inbox only 296 41 (13.85%) 10.6% 9 (3.04%) 124.8%
Inbox and email 39428 5437 (13.79%) 10.1% 616 (1.56%) 15.5%
Inbox total 39724 5478 (13.79%) 10.1% 625 (1.67%) 16.3%

Relating back to the goals of the experiment:

  1. Percent of closed posts edited:
    1. As shown by the data, in the control group (no notification) 12.53% of posts were edited by the post owner.
    2. This can be compared to 13.79% of posts edited after inbox notifications were sent, an increase of over 10% in the percentage of posts edited (corresponding to our first goal). This indicates that the notifications and/or email are having a positive effect in getting post owners to reengage after their posts are closed.
    3. This also led to an increase in the number of posts reopened after edits (1.35% to 1.57%). While this was not itself a goal for the experiment (there is nothing in this feature that is designed to lead to higher quality edits that are more likely to lead to posts being reopened), it is nice to see and is worth noting.
  2. Community reaction: We did not discover any significant negative feedback about the new notifications, nor was there any unexplained drop in site satisfaction numbers during this time period which might be correlated with a negative reception to this experiment. 
  3. Inbox only vs email:
    1. The original experiment did not test this question well enough. The logic put into place was to send the email out one hour after the inbox notification went out (as long as no edits were made, the question was still closed, and had not been deleted).
    2. Additionally, the system default interval to receive an email notification for unread inbox messages is 3 hours. Since 99% of participants in the experiment received this custom email, there is no way for us to tell if the 3 hour interval email (which sends the standard inbox notifications, without any special formatting relating to post closure) would have been more or less effective.
    3. Because of these factors, no determination can be made as to the effectiveness of the emails (on their own).
    4. It is worth noting that the Click Through Rate on the emails was 12.4%, which is a relatively high click through rate.

Experiment, take 2

Based on what we saw, we decided to graduate the inbox notifications. The following is now live, network wide:

  • Post owner inbox notifications when their question is closed
  • Post follower inbox notifications when a question that they follow is closed
  • No notifications are sent to:
    • Anyone when the question is closed due to being migrated to another site (this has its own notification)
    • The OP or any follower who cast the last vote for closing the question (including when a question author self-dupes their own question)

As far as whether or not to graduate the email, we opened a new experiment in mid-November (that ran for approximately two months), in which we had three groups:

  1. Control: Inbox notification only, no email
  2. A: Inbox notification sent upon closure and email sent after one hour
  3. B: Inbox notification sent upon closure and email sent after one day

Final Result: Closed emails after one day

The second round of the experiment included over 87,000 closed posts.

Findings:

  • The email sent one hour after the post CloseDate had no significant effect on edit rates for those posts by post owner (8.50% edited with inbox only versus 8.64% with email).
  • The email sent after one day did have a significant effect on the edit rates by post owner: 3.86% (versus 2.08% of posts edited after one day in the inbox-only control group).

Based on these results, we have graduated the closed email experiment, sending emails to users one day after post close. This excludes posts where the post owner has already edited the post, where the post is deleted or reopened, or where the post owner has indicated that they do not want to receive emails of this nature.

Improve this question
32
  • 4
    “This can be compared to 13.79% of posts edited after inbox notifications were sent, an increase of over 10% in the percentage of posts edited (corresponding to our first goal).” - How exactly do you figure that it’s an increase of 10%? The difference between 12.53% and 13.79% is only a little over 1% increase.
    – Ramhound
    Commented Nov 17, 2020 at 10:23
  • 61
    @Ramhound 1 percentage point increase, but (13.79 - 12.53)/12.53 ~ 10.05% increase in the rate.
    – muru
    Commented Nov 17, 2020 at 10:24
  • 22
    I wish to add that while the effect on SO may be rather small, I expect this to have a much bigger impact on some other sites.
    – Wrzlprmft
    Commented Nov 17, 2020 at 10:26
  • 26
    @rene: Several reasons: A lower ratio of questions that are unsalvageable, fewer help vampires, people being more invested in their questions, more timely closure.
    – Wrzlprmft
    Commented Nov 17, 2020 at 10:40
  • 21
    @Wrzlprmft interesting theory (and I agree with your premise). I will try to dig into the data after the experiment to see how this plays out on SO vs non-SO sites.
    – Yaakov Ellis
    Commented Nov 17, 2020 at 10:42
  • 13
    @Ramhound: The probability of this difference occurring by chance is astronomically low (from scipy.stats import fisher_exact; fisher_exact([[5464,43619-5464],[5478,39724-5478]]) → 7·10⁻⁸). The more interesting question is whether the effect size was worth it.
    – Wrzlprmft
    Commented Nov 17, 2020 at 10:52
  • 6
    @YaakovEllis, Not directly, no. But if I've had a post closed and deleted, unless I have high enough rep, it disappears. I assume these notifications stay your inbox for atleast 6 months and include a link to the question? Wouldn't an indirect result of this be the ability to view the post past that 90 day cut-off?
    – ouflak
    Commented Nov 17, 2020 at 12:28
  • 5
    @Trilarion possible explanation: the email that goes out contains different instructions based on the close reason (similar to what is shown in the post notice). Perhaps the instructions given here lead to higher quality edits. But that is just conjecture.
    – Yaakov Ellis
    Commented Nov 17, 2020 at 12:30
  • 3
    @ouflak this is not changing anything about the ability to view close votes. Just adding a notification when the post is closed.
    – Yaakov Ellis
    Commented Nov 17, 2020 at 12:33
  • 3
    Can a 10k user screenshot Jeff Atwood's now-deleted original answer to the original question? I'd like to read about the thinking back then. This seems like such an essential feature.
    – pkamb
    Commented Nov 17, 2020 at 17:30
  • 6
    One of the other answers actually has nearly all of what Jeff says. i.sstatic.net/hycsI.png is what Jeff said, as he said it
    – Journeyman Geek
    Commented Nov 17, 2020 at 17:50
  • 3
    @Trilarion Jeff's only wrong once you drastically change the audience that SO tries to serve. For the audience being served/targeted at the time, Jeff was very much right.
    – TylerH
    Commented Nov 17, 2020 at 22:07
  • 7
    Are four significant digits (e.g., 12.53%) warranted? The sample size alone would suggest a relative accuracy of about 0.5% (not percentage points) at best.
    – This_is_NOT_a_forum
    Commented Nov 18, 2020 at 2:16
  • 4
    @JohnBollinger regarding the number of edits - yes, it was not so many more. But every little bit counts. More than that - this is a feature that has been requested by users for a long time, and given that we vastly expanded our notifications system in the last year (to include followers, reopen), we were interested in this as well. We did however want to make sure that it was pushing things in the right direction.
    – Yaakov Ellis
    Commented Nov 18, 2020 at 6:45
  • 4
    A percentage of a percentage increase is not a useful measure, usually only used when somebody wants to present something as performing better than it actually did.
    – OrangeDog
    Commented Nov 19, 2020 at 11:07

5 Answers 5

Reset to default
56

Update from Yaakov Ellis's comment:

These emails are only sent to users who have their email settings for Tips & Reminders set to On. Users who have this set to Off should not receive the email, even if they are in the experiment group.

This kind of solves my concern. I still think the one-hour waiting time is too "nagging", but this is still being experimented on.

Original answer:

The email should be an opt-in feature. There are users that do not want to receive almost any SE-related notifications outside of the SE inbox, and this should not be an exception.

The advice that comes with the email could be shown on the SE website after the user clicks a link (only visible to them) in the close notice.

If this is not an option, I suggest to at least make the one-hour wait time between closure and email considerably longer. A closed question on SE is in no way a matter so urgent that it needs to be attended to in one hour.

Improve this answer
16
  • 39
    "The email should be an opt-in feature" This!!!
    – leonheess
    Commented Nov 17, 2020 at 16:42
  • 7
    This new feature needs to be attached to email settings that are already available in preferences, or maybe a new unique one. It doesn't make sense to not make it toggleable like all other SE emails.
    – zcoop98
    Commented Nov 17, 2020 at 17:51
  • 4
    What does this have to do with the GDPR?
    – user2357112
    Commented Nov 17, 2020 at 23:25
  • I don't think the GDPR says anything that requires this feature to be opt-in. The closest thing is that there needs to be a legal basis for processing personal data used to send emails, but contrary to popular misconception, that legal basis does not have to be explicit consent in advance.
    – user2357112
    Commented Nov 17, 2020 at 23:36
  • 2
    There should be an opt-out option, but I'm pretty sure the GDPR doesn't require this to be opt-in.
    – user2357112
    Commented Nov 17, 2020 at 23:43
  • 6
    GDPR has absolutely nothing to do with this.
    – Asteroids With Wings
    Commented Nov 18, 2020 at 1:32
  • 1
    @AsteroidsWithWings Maybe not, but the GDPR-concerned user has something to do with it. :)
    – Conrado
    Commented Nov 18, 2020 at 3:44
  • 14
    An opt-in would guarantee that virtually no-one gets the emails. Users who are engaged enough to go opt into this would almost certainly be the sort of users who don't need it. These emails obviously shouldn't be sent to people who've opted out of the relevant categories in their email settings, but making it opt-in would defeat the user benefit of sending the email.
    – Ryan M
    Commented Nov 18, 2020 at 7:24
  • I just removed the "GDPR" part, as it might be wrong and is not really necessary anyway.
    – wimi
    Commented Nov 18, 2020 at 7:37
  • It should respect the overall email notification settings, of course.
    – NoDataDumpNoContribution
    Commented Nov 18, 2020 at 8:26
  • 1
    Well here in the Netherlands you are not allowed to send newsletter-type of emails without consent. That probably boils down from EU legislation. That consent NEEDS to be opt-in. It can't be the old fashion remove this checkmark or check this box to NOT receive the newsletter. It has to be click here to subscribe. However these emails are of a different type, so it might not be needed in this case.
    – Luuklag
    Commented Nov 18, 2020 at 8:43
  • 3
    These messages are considered Transactional Email under the terms of GDPR and therefore exempt from necessary opt in/out. It might be nice to allow the user to opt in/out but there is no legal requirement to do so
    – Chris Davies
    Commented Nov 19, 2020 at 9:41
  • 12
    These emails are only sent to users who have their email settings for Tips & Reminders set to On. Users who have this set to Off should not receive the email, even if they are in the experiment group.
    – Yaakov Ellis
    Commented Nov 19, 2020 at 11:01
  • @roaima "transactional email" are not a concept from GDPR. What you are probably referring to is the processing of user data on the basis of fulfilling a contract. This would typically include things like "please pick up your order" emails and the general processing of an address to deliver their order. This would not apply to the processing of data in this case as far as I can tell. Most likely this would fall under the processing of PIIs based on a legitimate interest, which is something the user can legally object to which is typically implemented as an opt-out.
    – David Mulder
    Commented Nov 21, 2020 at 17:53
  • @DavidMulder Law refers to this very phrase in Where is the line between transactional vs marketing emails with GDPR?
    – Chris Davies
    Commented Nov 21, 2020 at 18:11
9

"... Additionally, the system default interval to receive an email notification for unread inbox messages is 3 hours. Since 99% of participants in the experiment received this custom email, there is no way for us to tell if the 3 hour interval email (which sends the standard inbox notifications, without any special formatting relating to post closure) would have been more or less effective."

and

[...]

  1. A: Inbox notification sent upon closure and email sent after one hour
  2. B: ″ ″ one day.

I think if people are fairly active they may or may not react before one hour, and likely will before one day; whereas people who check in every few days or longer (the majority?) are basically in the same category (receive email (group 2 and 3), or not (group 1)).

My point: For the majority, an hour or a day are too similar an interval (too close together).

You could guess which group they might fall into (assuming a correct email address) by checking how often they visit - so perhaps group 3 would benefit from being sent after 3-5 days.

Improve this answer
1
  • 4
    In the last experiment, users received the special email before there was a chance at getting the inbox notification. So no way of knowing if maybe the inbox notification by itself could do the trick or not, since the quickest that that can come is within 3 hours. Extending the experiment to a day gives some differentiation there to see if the inbox email might do the trick. That said, considering changing this to a longer email to make it more obvious.
    – Yaakov Ellis
    Commented Nov 19, 2020 at 10:59
5
  • No notifications are sent to:
    • [...]
    • The user who cast the last vote for closing the question (including when a question author self-dupes their own question)

The wording of this is a bit confusing to me. I understand not sending a notification to OP when they vote to close their own question. However, the main, non-parenthetical part of the sentence sounds like you are sending a notification to the first and second (and 3rd/4th for non-SO sites) close voters when a question they voted on is closed, but surely this is not the case.

Assuming I'm correct there, I'm wondering why you don't just say no notifications are sent to:

the post owner when they cast the vote that closes their question/vote to close their own question

or something similar?

Improve this answer
8
  • 15
    If a user who followed the question cast a close vote on the question, and that vote is not the last vote, they will receive a notification. If they cast the final vote, then will not receive a notification.
    – Yaakov Ellis
    Commented Nov 17, 2020 at 14:43
  • 4
    @YaakovEllis So a user who casts a close vote, but does not follow the question, will not get notified?
    – TylerH
    Commented Nov 17, 2020 at 15:00
  • 5
    That is how I interpreted things from the start @TylerH. You need to be either OP or a follower of the Q to be notified.
    – Luuklag
    Commented Nov 17, 2020 at 15:21
  • 6
    @Luuklag correct
    – Yaakov Ellis
    Commented Nov 17, 2020 at 15:24
  • 2
    @Luuklag It's how I'm hoping it was meant to be interpreted but since it's unclear based on the wording I wanted some explicit confirmation, because the alternative would be, IMO, pretty bad.
    – TylerH
    Commented Nov 17, 2020 at 16:27
  • It would be very weird to send a notification about the closure that you were the last nail of.
    – Braiam
    Commented Nov 17, 2020 at 17:41
  • 1
    @Braiam I agree. That's not what this answer is about.
    – TylerH
    Commented Nov 17, 2020 at 22:05
  • 1
    For future readers - TylerH has since edited the wording of that bullet point in Yaakov's post to clarify this point accordingly, changing it to: "The OP or any follower who cast the last vote for closing the question". :)
    – V2Blast
    Commented Nov 24, 2020 at 0:36
3

Great to have this new feature! I've had countless of questions being closed silently, which is extremely annoying when waiting for answers to questions.

Giving a chance to the OPs to improve/edit/update/etc. their questions before the Stack Exchange system makes it impossible to other users to answer the questions is very important. In that spirit, I believe one should also receive a notification when one's question gets deleted.

Improve this answer
2
  • 6
    I'm in doubt on notifications upon question deletion. Yes I'm all about (increased) transparancy, but on the other hand it will lead to a lot of re-posting questions, which I see as a major drawback. But perhaps this hypothesis is best put to the test.
    – Luuklag
    Commented Nov 23, 2020 at 7:41
  • Could you point out a couple of these closed questions? I’m not following how you would miss a very important question being closed. Maybe SO works differently than other sites and I’m not understanding.
    – bmike
    Commented Nov 30, 2020 at 11:45
-12

"... Additionally, the system default interval to receive an email notification for unread inbox messages is 3 hours. —copied from https://meta.stackexchange.com/a/356767/163204 & Question closed notifications experiment results and graduation where I originally commented, thank you https://meta.stackexchange.com/users/282094/rob for the encouragement

Why "3 hours"? Is Stackcorp's bulk/mass emailing really the issue (server stress and spam complaints basically?)? I want instant and every notifcation, personally*. Can I please pay $5/mo. or buy credits for instant and you pay for longer timing if you prefer? Picking "three" hours for everyone is a lack of thought.

Programmatically a customizable form to allow the user to set the setting is optimal, and would lead the way as a role model for other websites suffering with the same systemic/formulaic poverty of thought.

*I can not trust the current Notification emails to be regular, I manually refresh SE.com constantly just to stay informed (clicking the menubar's uninformative red light sequare, again and again), the Android app is the most reliable notification I get yet it's ethereal (restart phone and poof, missed several mods made in under 30 minutes, again), versus a copy of email apparently timed currently for dealing with server stress instead of user stress.

Improve this answer
11
  • 4
    For instant notifications there is the inbox on the site.
    – Luuklag
    Commented Nov 18, 2020 at 15:45
  • @Luuklag I mentioned that being a problem in the astericks (*), did you read that line? I added the word "menubar" for that line/part to be clearer.
    – prosody-Gabe Vereable Context
    Commented Nov 18, 2020 at 15:46
  • 9
    I don't get what is unreliable about it. It works perfectly fine for me.
    – Luuklag
    Commented Nov 18, 2020 at 15:49
  • @Luuklag If you can stay loaded 100% of the time, have a 100% working internet connection, always have your browser working, then you are privileged to get all your notifications with that unfriendly barebone feature. You're privileged then, I am not able to trust that works like you. Privilege.
    – prosody-Gabe Vereable Context
    Commented Nov 18, 2020 at 15:50
  • 23
    I'm sorry you're struggling with the on site notifications but you're asking for something that'd be a huge change to our defaults for notifications. That's not a bad thing but would need to be a separate discussion since this question relates to one specific notification only.
    – Catija
    Commented Nov 18, 2020 at 17:28
  • 2
    I reorganized as another q at meta.stackexchange.com/questions/356817/…. Thank you @Catija for the encouragement, it's definitely not personal. I agree and realize the work is a burden of complications (why nobody wants to muddle O:], and I myself can not help code ASP.NET the current SE lang last I checked) as well as you. :) I value your time.
    – prosody-Gabe Vereable Context
    Commented Nov 18, 2020 at 18:54
  • 1
    I am sorry but you have like ~20 question and answers in 3 years. I don't see how notification is bothering you.
    – Equinox
    Commented Nov 25, 2020 at 10:53
  • @venky_ Please read me carefully here this comment: I asked 100 questions on Quora and had similar issues, my 20 questions here are the same issue. My problem is that notifications are NOT bothering me, I do not get ENOUGH notifications, there are not enough ways to ANALYZE notifications, if you are a mod you get a team w/ support to dive advanced ques of notifications, I get a red light in my menubar making a smartphone sound. I am asking to be bothered by deeper notifications, notificati for every event, as it happens, not grouped in a "Edited Digest of Notifications" every ~arbitrary hours.
    – prosody-Gabe Vereable Context
    Commented Nov 25, 2020 at 14:02
  • 1
    Ironically in my answer someone complained I posted too many questions. Whatever you do, someone will complain. I can see why some people may prefer having an email instantaneously. Amazing this got 15 downvotes and only 4 upvotes.
    – Franck Dernoncourt
    Commented Nov 26, 2020 at 9:48
  • Instant email notifications have the downside of looking like SPAM, which may lower the server reputation if they send tons of emails to a single person. Which may lead to messages being moved to the SPAM folder automatically, which may lead to missing the notifications you want to receive in the first place. Please keep that in mind. In the end, the email servers may be blacklisted and everything is marked as SPAM for everybody.
    – Ismael Miguel
    Commented Dec 2, 2020 at 11:44
  • @IsmaelMiguel That is why I asked to pay $9/month with my real name bank debit card and multiple 3PID providers, I would even go in for an interview to be fingerprinted if I could just get notifications for the record, recorded at the level of a high-level law firm requesting access to a single account, but for my own account, please. For once I am tired of being dealt with like I am Predator User Seeking Nuclear Fissle Material using Tor for every activity -- wold DKIM and Secure DNS be necessary if we designed for User Begging For Once To Just Give My Real Name with Their Fingerprint ID?
    – prosody-Gabe Vereable Context
    Commented Dec 8, 2020 at 23:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .