I happened to have gone to a toppish school (look up my profile if you really care which one). Even though I didn't realize it at the time, there are distinct advantages from the mathematical environment there. But I also think you shouldn't base your decision on graduate schools purely on artifical rankings of prestige. If you are deciding between schools, you have to visit them to get a sense of the environment there.
Unlike the undergraduate degree, if you do a graduate degree at a reasonably strong place, you will be spending 4, 5, or maybe 6 years with a close-knit group of people that number from 20-30 to 100+. It is where you will get to know a lot of future colleagues in your field, it is where you pick up new ideas from passing discussions, and it is where you learn how to do research. And the most important factor that goes into your decision of which graduate school to go to is the people.
You need to find out if you can work with the faculty there: is there anyone there in your field? Is his or her interest something you want to study? Are they taking students? How do they treat their students? All these are things you should talk to people, especially current students, about when you visit.
You also need to find out if you have fellow students†: you don't want to be the only student in the department doing (for example) PDEs when everyone else does algebraic geometry. The students whose research interests are closely related to yours will be the ones you discuss most with (and see most of); they will be the ones you ask for help and they will be your future colleagues. Find out if they have informal student seminars. Arrange your trip so that you can sit in on some of them. Having a couple more senior students who are willing to give you sound advice when you are just starting out will be very useful.
If you really must make a decision without first hand experience visiting, then so be it: higher ranked schools tend to have a larger, more diverse pool of faculty. They also tend to attract the better students. So that's where I will put my money were I to bet blindly. But I cannot stress enough the importance of finding out whether the school and environment matches you first hand.
A little side remark about "more room psychologically": I would argue that that is actually counter-productive. When I was in graduate school there was significant amount of down time where the research just doesn't go well. Looking back, I am really glad to have had my large and rowdy office: at least one of us will have made some progress at any given time. And with seminars happening now and then, one can at least learn something useful through discussions when the thesis project is stuck. Were not for all these "distractions", I fear that the ennui caused by the glacial pace of my thesis research may have led to psychological problems.
† Added Oct 2011, partly in response to Isomorphism's comment below: of course, fellow PhD students are not the only people you can talk to. They are often the ones you feel most comfortable approaching. But you should certainly also factor in PostDocs at the institution when you make your decision, as long as you be proactive about approaching them for discussions.