So today something weird happened while I was doing my daily business on Physics.SE. I'm not too worried with my rep, but I love numbers and my rep has ended with a 6 for the last week or so. I'll not bore you with details, but essentially today I noticed out of the blue that I lost 2 rep arbitrarily. I check my rep-o-meter logs and they didn't say a user was removed or that I had been downvoted or anything. I didn't do any downvoting myself. None of my posts were deleted. In fact, there was no notification anywhere (not the top bar, not the rep page, nothing) that my rep had changed at all.
Thankfully I had my site's meta page open and hadn't refreshed it for an hour. It confirmed a rep ending with a 6 at most an hour before, but the main site showed rep ending with a 4 and no action I had taken or was notified of had caused this -2.
So I have to know, what can cause you to lose (or gain; let's make this more broadly applicable) rep without being notified at all? Does this happen often? Have we all lost scores of rep in unnoticed and unnotified -2's here and there?