Interesting thought. I will try to offer some factors to consider related to the approach you are suggesting.
There's a risk of spreading yourself too thin. All things held constant, I would expect it to be more difficult on average to build relationships with faculty at institutions other than your PhD "home base." The time and effort spent building these relationships would thus be re-routed from what should be your main thrust, i.e. developing a reputation as an excellent student in a place that actually voted to have you and invested resources in you.
Faculty at top institutions are extremely busy and inundated with queries from all sorts of places and people who want their time and attention on collaborations of various types. Successful faculty are very strategic about where they invest their energies (especially in Physics? sorry, bad pun). This partly explains the uphill battle presumed in point #1. They usually have plenty of students and postdocs of their own who want to "collaborate." The returns from such one-off partnerships with this or that individual who met them at this or that conference are dubious, and typically high-flying faculty end up collaborating on high-visibility/payoff projects. Unless your work will fall in that category, I think productive sustained collaboration with quality attention and care invested in the relationship on both sides is highly unlikely.
The only relatively common, accepted form of collaboration with faculty from other institutions that I am aware of is IF those individuals are members of your dissertation committee. Colleges (at least in the U.S.) often require that at least one committee member be from another department at the home institution. Along the same lines, it is typically viewed as OK if there is another, additional committee member from another institution, as long as their being on the committee is well justified, e.g. by filling a gap in expertise or being in a unique position to contribute something given their specific context/resources. In contrast, collaboration on side projects can cause some minor (which could become major if left unchecked) issues involving your Advisor wondering what the heck is taking up your time that you could be spending in his lab and writing his papers...
This being said, it is possible that in some disciplines such collaborations are more frequent and more accepted than in others, and may pay off in some ways. However, the though I would like to leave you with is to carefully reflect whether such collaboration will in fact help your career plans, over and above similar collaborations with faculty at your home institution. If you find yourself in a situation/project where the answer is a clear YES, well, you can at least give it a shot and see what happens.
However, if by looking over the fence (where the grass typically looks greener) you might end up missing some gems on your own lawn, I would say stick to the familiar environment and try to make the most of the opportunities under your nose.
Lastly, collaboration can take many forms. It may not be necessary to go it alone and try to carve out an independent project with someone from another university/lab. Being on a research team (as a Research Assistant) that has some external partnerships with teams at other institutions can be enough to develop some contacts and build some name recognition in those places. Being considered a valuable member of the research team at home, and being valued by your own Advisor can speak louder than trying to convince someone on the outside that you are wonderful and totally worth their attention. When you will be close to graduating, if your Advisor believes in your potential, s/he will go a long way to help you network with various other labs/teams that s/he has relationships with. I personally would recommend this more "organic" route.
The academe is a big boat that doesn't like to be rocked, and stability is generally considered an asset. Having a reputation for dedication to one or two large-scale, complex, team projects is probably going to help you more than having 10 separate fledgling solo projects/partnerships. (Note: If you were an undergrad with a major in Business/Entrepreneurship, this advice would be the opposite. But a PhD in Physics seems more aligned with the solid/long-term academic paradigm.) Whichever way you choose to play your cards, good luck!