The issue:
On my Windows 10 build 1909 PC, when the explorer icon is open it shows the icon of the current navigation pane folder on my taskbar. I did few thing like rebuilding icons cache, but that did nothing to solve the issue.
Evidently I managed to mess up Explorer's settings somehow, most probably when I used Winaero Tweaker(?) because I did not touch Explorer's settings in any other way. I noticed this about 10 day ago and found no solution so far. But it is very annoying.
I found similar question (Windows Explorer icon showing wrong on taskbar asked Mar 26 at 16:23 by Akudaijin), but 1st: it was for Windows 7, 2nd: it seems that it was never answered.