This is a difficult situation, and I am not sure if I would choose the "really prestigious university" over a "brilliant postdoc mentor" (e.g. a post doctoral fellowship with absent mentors, even if in a prestigious university, can be a waste of time), but let's assume that it is your choice.
On one hand, postdoctoral fellowship are often used as "temporary" positions while the fellow applies for tenure-track positions, and it is in no way a "breach" of contract to interrupt a 2 year postdoctoral position in order to accept a tenure-track one. On the other hand, breaking a promise is never nice (and should never be easy), and leaving a postdoctoral position for another one is borderline.
A polite way to manage such a situation would be to include the professor who was offering you the first postdoctoral position into your decision process, by writing to him an email honestly describing the situation and asking for his advice. He might be able to point out arguments that you did not think about, and/or he might advise you himself to accept the new position. (As a bonus, if his answer does not show that he has your best interest in mind, then he was not the right mentor for you anyway.) In the end you will make your own decision (as it should be), but including the professor in the decision process make it a joint decision, showing respect.
As a (second) bonus, if the professor is brilliant and nice, do try to make plans for collaborating with him even if you are a postdoctoral fellow at another university (discuss it with the people who fund your postdoctoral fellowship if this collaboration will quite time from the work you do with/for them): for one, the postdoctoral period should be an occasion to collaborate as broadly as you can; and for two, the university of this professor might be where you will apply for jobs after your fellowship!
Hope it helps!
