Limit the number of questions per site in the Hot Network Questions list

The biggest problem seems to be that the number of answers has such a big effect, no matter what the answer quality is. Deleting spam answers from a question should not drop its hotness, but now it does. In my opinion answers with zero score have a disproportionate effect on hotness.

This same mechanism is what promotes the code golf questions too much according to my observations. I think that the weight given to the number of answers is a significant cause of imbalance between the sites. Fixing this should improve the situation, but I have no hard data to back this up.

The original hotness formula (see here) contains

(MIN(AnswerCount, 10) * QScore) / 5 + AnswerScore

where the weight due to answer count is clear. I think something like

QScore * SUM(IndividualAnswerScore * Weight(AnswerNumber))

would work better. (Or maybe the first multiplication could be replaced with addition?) Here Weight(k) is some weight, for example Weight(k)=1/k. So if the question has score 10 and its three answers have scores 15, 6, and 3, the total score would be 10*(15/1+6/2+3/3)=190.

By choosing the weight differently you prioritize different shapes of answer score distribution. If the weight decays rapidly, you only look at the first few answers. If the weight is constant, you just sum up the answer scores. If the weight is constant for k<4 and zero from k=4 onwards, then you sum the scores of the first three answers. I prefer to have some decay; one answer of 10 points is worth more than ten answers with one point each. Having more upvoted answers increases hotness as before — with the exception of cutoffs when the weight is zero from some point onward.

Or, for a simpler alternative, drop the answer cap from 10 to 3 or something in the formula. To some extent this can be realized with suitable weights as mentioned above. I don't see the relevance of the number of answers after three or so. But I do prefer the weighted sum if feasible.

Here is the current HNQ, as a data point. I can generate an updated version of that at any time if needed.

The major concern I have with jumping to conclusions now is this is just one sample; PPCG could just be having a run lately, and since we're looking for problems (or in a "PPCG again??" mental state), we notice with a bit of confirmation bias. There might be some merit in the "too much weight on answer count" claim (note they're capped at 10, btw), but if that hypothesis is on the right track I think it's more than that: It's sites that attract a lot of high scored answers that gain a lot of weight here. So sites with that activity pattern would presumably tend to appear on the HNQ more. But we'd need to gather data to prove that activity pattern on these sites.

Also, there is a correlation between answer count and question score on average (all sites I checked are like this). Now this considers the average of all questions, so is a wider sample than the HNQ list and therefore may ignore some subtleties of HNQs, but that trend suggests the top half of that equation may increase more than intended, i.e. QScore and AnswerCount are, on average, redundant with eachother. Plus, they're multiplied with each other, so it's a bit of an exponential bias.

All that said, first we should gather samples. A lot of them. Over the course of a few months (some past samples do exist). Table the discussion until then. I'm working on an HNQ tracking app and I'll post it here when I do.

Personally, I don't think that adjusting the core formula is the correct approach. According to What are the criteria for questions to be selected for Hot Network Questions?:

Succeeding questions from the same site are penalized by increasing amounts. So, the first question from SO in the list gets multiplied by 1.0, the second by 0.98, the third by 0.96, etc).

And also of course there is:

We make a per-site traffic adjustment so SO does not dominate the entire list.

So if tweaks were to be made I'd say keep the core formula the same, but:

• Increase the successive question penalty, and
• Possibly tweak traffic adjustments, because it could just be that some sites have simply outgrown their current weights.

(Additionally, it might be worth considering making the successive question penalty a per-site parameter rather than a global parameter, so it could be tweaked for certain sites... but I'm not 100% convinced that's the right idea in the big picture sense, it's just a thought.)

By the way, re this comment:

Too many code golf questions get on HNQ without having many upvotes -- they get them by being on HNQ. The hotness formula uses MIN(AnswerCount, 10) * QScore, so having lots of answers effectively boosts the question score. And the + AnswerScore term lets a poorly voted question get by by having good or numerous answers. I think a better formula would also factor in close votes and downvotes directly.

I'm inclined to agree, at least my gut is, but we should consider this carefully, because there's a couple potential counterpoints here. In particular, playing a bit of devil's advocate on purpose just to make sure we're staying sane:

• "Too many code golf questions get on HNQ without having many upvotes -- they get them by being on HNQ." -- Well, on the other hand, that's equally true for any question on the HNQ list, not just PPCG. Also, the questions got there without having many upvotes, that is, they got there. So whatever happens afterwards may help to keep them there, but doesn't have impact on them getting there in the first place.

• "I think a better formula would also factor in close votes ..." -- Just FYI, close votes are "factored in" in a way: Closed questions are excluded from the HNQ list. So it's not a continuous factor, but as soon as the CV count reaches 5, the question is removed. So there's that.

I do not have enough information to make any conclusions or arguments in any direction, my intent is just to make sure we're considering this carefully.

As of March 11th, 2019,

Each site can only have a max of five questions on the HNQ list at any given time.

This is a big, much-requested change and we may reduce the number in the future even further. We're starting it higher than some might want (suggestions went as low as one per site) because sites that have had a lot of exposure through the HNQ may see a dramatic drop in visits, so we need to be careful to find the right number here and possibly do some testing at different levels.

This number is also configurable on a per-site basis, so if a site wants to reduce their HNQ footprint, we can lower it even more, even to zero if a site wishes to be excluded entirely. Sites will needs to go through a meta discussion before requesting this change and it will be up to the site itself to request a change rather than having the limit imposed upon it (unless we lower the maximum for the entire network). So for example, Stack Overflow can't vote to kick Movies & TV off the list entirely because they don't want to see spoilers for the last season of Game of Thrones, but Movies & TV can ask that fewer of their questions be shown so that they can devote sufficient time to those that are.

Here is the data I've collected. This updates every hour. Note that times are in BST (GMT+1).

It starts at 00:00 2017-06-06 GMT. As of writing this post, it has data for 210 hours (one hour the internet connection failed).

Here's a graph of the number of Code Golf questions on the front page of stackexchange.com (the top 30 HNQs):

Blue is the number for any given hour. Red is the average for all time and yellow is the rolling average for the past 15 hours.

Feel free to analyse the data in there - there's not much right now, but it's getting there.