The radiometric dating of meteorites tells you when certain radioactive species were incorporated into that body. It is extremely difficult to see how that material could have got there at a later time, so the age you get is the age of formation of that body.
Meteorites do not form in stars. They require considerably cooler conditions, but they also require a certain density to stick together. Those conditions are found in a protoplanetary disk. So you might well ask, well maybe the meteorites were formed around another star and then somehow they have ended up in orbit around our Sun.
This is quite unlikely. Very similar ages have been determined for some tens of meteorites and that age happens to be just a touch older than the oldest rocks that can be found on the Earth and the Moon. There is no big age spread among meteorites. That age also coincides with the age for the Sun deduced from physical models (though this age is less accurate because of some remaining uncertainties in the physics, but it is not hundreds of millions of years out).
The obvious inference is that these meteorites were among the first solid bodies formed in the protoplanetary disk that ultimately formed the Earth and Moon and hence represents a lower limit to the age of the solar system. It depends where you wish to start the clock; the formation of the first solid bodies seems as good a place as any and models for disk formation and evolution suggest that these solid bodies do form within about a million years of the molecular cloud that formed the Sun beginning to collapse.
