The risk I see is that components I pick up at the other desk can get
static energy because of rolling my desk chair or walking to the desk
(and picking up the components on that desk)
Never pick up a component up that is static sensitive unless it is in its proper packaging or is placed on a conductive mat that you can initially discharge your finger/body to. While touching that mat you can pick up the component and any charge you might acquire on the return journey from that desk to the other desk is of no consequence providing to acclimatize your body potential to the target circuit you are going to put that component in.
This is all helped by the use of earthed wrist bands because it guarantees a degree of protection without having to think what you are doing. Removing the earthed wrist band means you have to think about what you are doing and, of course, doing things in the correct order is important.
What is unsalvageable is a component removed from its protective packaging and placed on a non-conducting surface. You cannot subsequently guarantee what charge that component has acquired so even if you use a wrist strap it doesn't offer much protection. Always handle components correctly and always place them on something that offers conduction even if it isn't earthed. While handling them always connect your body first to the conductive protection rather than the component.