I'm not getting reputation notifications

Couple of things blew up here and it took a little while to isolate and repair the fallout, so sorry for the delay.

We had a (successful!) SQL failover in the wee hours of this morning but, for the brief period of time that the application was failing over, a number of duplicate events were written to a service that we call the "aggregator". This service is responsible for taking events from the various sites across the Stack Exchange network and aggregating them to denormalized tables so that we can get performant queries for things that cross the whole network - things like global inbox messages, rep, achievements, etc.

Usually this would be a non-issue - we have de-duplication logic in those code paths and it has functioned just fine for years. However, there were a specific set of events that had, clearly, never been de-duped - they're relatively rare (happen once per user per day). De-duping takes the form of sorting and eliminating the duplicates using equality checks - unfortunately the code for checking equality in one of these events looked like this:

public bool Equals(object obj)
{
    if (!EqualityBase(obj, out VisitEvent other)) return false;

    return other.Equals(this) && other.UserHistoryId == UserHistoryId;
}

That call to other.Equals(this) was very rarely hit, but it clearly recurses infinitely, leading to a StackOverflowException. That exception kills the process and because of the way the aggregator service runs, it meant that whichever server picked up the work would then die off when it tried to pick up the work.

We have some poison message handling around this but it didn't kick in effectively here and we have a large backlog of events to get through now the bug is fixed. It's getting through them now though and we should be all set in the next hour or so.

UPDATE Annnnnd it’s done. Thanks for your patience folks!