That is not the intent in the PHB
In the PHB, tools are described as follows (emphases mine):
A tool helps you to do something you couldn't otherwise do, such as craft or repair an item, forge a document, or pick a lock. Your race, class, background, or feats give you proficiency with certain tools. Proficiency with a tool allows you to add your proficiency bonus to any ability check you make using that tool.
It seems clear that if the tool helps you do something "you couldn't otherwise do", that is, that you couldn't do without the tool, then you need the tool to do it. Merely having the proficiency shouldn't allow you to do something that is not possible without the tool itself.
XGtE is less clear-cut
In addition to tool use checks, XGtE introduces the concept of tool use proficiency influencing the result of other checks, although you might still need the tool itself on hand.
Under "Skills" (78) (emphases mine):
Every tool potentially provides advantage on a check when used in conjunction with certain skills, provided a character is proficient with the tool and the skill.
Here the text also says that not only do you need the proficiency, but you need the tool itself on hand.
Xanathar's, however, also introduces the idea of "special use" properties for tools.
Under "Special Use" (78):
Proficiency with a tool usually brings with it a particular benefit in the form of a special use, as described in this paragraph.
You are correct that the 'special use' rules call out the need for proficiency, but not necessarily the tool itself. Is this, as you suggest, a specific-over-general exception to the rule that the physical tools are required? Or is it merely assuming that you are already following the rules for tool use proficiencies, which say that you do need the tools themselves?
The possibility that not every benefit has to be tied to the physical use of the tool itself is also hinted at in the description of Skills (78) (emphasis mine):
the system is mildly abstract in terms of what a tool proficiency represents; essentially, it assumes that a character who has proficiency with a tool also has learned about facets of the trade or profession that are not necessarily associated with the use of the tool.
Honestly, I don't know what RAW is here - but I would suggest that it doesn't really matter. Tool use is a pretty low-stakes aspect of the game, generally meant to be part of the exploration and social interaction pillars, where DMs generally have more room to decide what seems reasonable for the story, rather than adhering to the stricter rules that govern the more competitive combat pillar.
In the specific case of Weaver's tools that you mention, I think that playing it either way, that tools in hand are required or that they are not, strains verisimilitude in certain circumstances. I would favor simply having the DM decide whether or not tools were required depending on your actual use scenario.
XGtE (85) says about Weaver's tools:
Components. Weaver's tools include thread, needles, and scraps of cloth. You know how to work a loom, but such equipment is too large to transport.
If you insist that the "special use" rules are a specific-over-general exception that means you don't need tools, that does indeed make the special use repair function strange:
Repair. As part of a short rest, you can repair a single damaged cloth object.
How is your knowledge of weaving supposed to help you accomplish this, absent needle and thread?
On the other hand, insisting that you always need the tools leads to equally strange results, since Weaver's tools proficiency also gives you 'additional insight' when you make Arcana or History checks related to cloth objects, and Investigation checks related to woven items. Here, I think your knowledge alone would suffice when examining such items, and you and you would not actually need needle and thread.
As a DM, I would likely rule that you do need the tools whenever you are trying to do something, but not when you are trying to know something.
My recommendation would be to simply describe to the DM how your character is using their knowledge, or their tools, or both to accomplish some goal, and let them rule on the results of your actions, as always.
